TPTT: Transforming Pretrained Transformer into Titans
Authors: name: Fabien Furfaro
Published: 2025-06-21 10:06:07

Abstract: Recent advances in large language models (LLMs) have led to remarkable
progress in natural language processing, but their computational and memory
demands remain a significant challenge, particularly for long-context
inference. We introduce TPTT (Transforming Pretrained Transformer into Titans),
a novel framework for enhancing pretrained Transformer models with efficient
linearized attention mechanisms and advanced memory management. TPTT employs
techniques such as Memory as Gate (MaG) and mixed linearized attention (LiZA).
It is fully compatible with the Hugging Face Transformers library, enabling
seamless adaptation of any causal LLM through parameter-efficient fine-tuning
(LoRA) without full retraining. We show the effectiveness of TPTT on the MMLU
benchmark with models of approximately 1 billion parameters, observing
substantial improvements in both efficiency and accuracy. For instance,
Titans-Llama-3.2-1B achieves a 20% increase in Exact Match (EM) over its
baseline. Statistical analyses and comparisons with recent state-of-the-art
methods confirm the practical scalability and robustness of TPTT. Code is
available at https://github.com/fabienfrfr/tptt . Python package at
https://pypi.org/project/tptt/ .