ICML 2025

PENCIL: Long Thoughts with Short Memory
Chenxiao Yang 1 Nathan Srebro 1 David McAllester 1 Zhiyuan Li 1
Abstract
While state-of-the-art LLMs have demonstrated
great promise of using long Chains-of-Thought
(CoT) to boost reasoning, scaling it up to more
challenging problems is fundamentally limited by
suboptimal memory usage — intermediate com-
putations accumulate indefinitely in context even
no longer needed for future thoughts. We intro-
duce PENCIL, which incorporates a novel reduc-
tion mechanism into the autoregressive genera-
tion process that recursively clean up intermedi-
ate thoughts based on patterns learned from train-
ing. By alternately generating and erasing, PEN-
CIL can think deeper to solve harder problems
using shorter context and less computes. Empiri-
cally, for example, we demonstrate PENCIL with
a small 25M-parameter transformer and 2048 con-
text length solves Einstein’s puzzle — a task that
challenges much larger models like GPT-4. Theo-
retically, we prove PENCIL can perform universal
efficient computation by simulating any Turing
machines with optimal time and space complexity,
and thus can solve arbitrary computable tasks that
are otherwise intractable for vanilla CoT.
1. Introduction
Recently, there has been a surge of interest in reasoning
with Chain-of-Thought (CoT) (Wei et al., 2022) and gen-
erating longer thoughts at test-time to tackle larger-scale
and more complicated problems (OpenAI, 2024; Guo et al.,
2025; Snell et al., 2024; Muennighoff et al., 2025). CoT is
an iterative generation process: each intermediate reason-
ing step is appended to the current context and treated as
the input in subsequent reasoning. The context grows until
reaching a final answer. While such an iterative model is
theoretically powerful – capable, in principle, of tackling
many intricate problems given unlimited length (Merrill &
1Toyota Technological Institute at Chicago. Correspondence
to: Chenxiao Yang <chenxiao@ttic.edu>.
Proceedings of the 42 nd I...