ICML 2025

Self-Improving Transformers Overcome Easy-to-Hard and Length
Generalization Challenges
Nayoung Lee * 1 Ziyang Cai * 1 Avi Schwarzschild 2 Kangwook Lee 1 Dimitris Papailiopoulos 1 3
Abstract
Large language models often struggle with length
generalization and solving complex problem in-
stances beyond their training distribution. We
present a self-improvement approach where mod-
els iteratively generate and learn from their own
solutions, progressively tackling harder problems
while maintaining a standard transformer archi-
tecture.
Across diverse tasks including arith-
metic, string manipulation, and maze solving,
our method enables models to solve problems far
beyond their initial training distribution—for in-
stance, generalizing from 10-digit to 100-digit
addition without apparent saturation.
We ob-
serve that filtering for correct self-generated exam-
ples leads to exponential improvements in out-of-
distribution performance across training rounds.
Additionally, starting from pretrained models sig-
nificantly accelerates this self-improvement pro-
cess for several tasks. Our results demonstrate
how controlled weak-to-strong curricula can sys-
tematically expand model capabilities while pre-
serving architectural simplicity.
1. Introduction
Despite the remarkable success of transformer-based lan-
guage models (Vaswani et al., 2017) across a wide range
of tasks, these models exhibit significant limitations in
length generalization—the ability to extrapolate to longer
sequences than those seen during training. Even in simple
algorithmic tasks such as arithmetic, standard transformer
models trained with autoregressive objectives struggle to
generalize to longer problem instances (Anil et al., 2022).
To address this, prior work has explored various modifica-
*Equal contribution
1University of Wisconsin-Madison
2Carnegie Mellon University
3Microsoft.
Correspondence
to:
Nayoung Lee <nayoung.lee@wisc.edu>,
Ziyang Cai
<zcai75@wisc.edu>.
Proceedings of the 42 nd Internatio...