InfiniPot: Infinite Context Processing on Memory-Constrained LLMs
Authors: name: Minsoo Kim; name: Kyuhong Shim; name: Jungwook Choi; name: Simyung Chang
Published: 2024-10-02 13:09:41

Abstract: Handling long input contexts remains a significant challenge for Large
Language Models (LLMs), particularly in resource-constrained environments such
as mobile devices. Our work aims to address this limitation by introducing
InfiniPot, a novel KV cache control framework designed to enable pre-trained
LLMs to manage extensive sequences within fixed memory constraints efficiently,
without requiring additional training. InfiniPot leverages Continual Context
Distillation (CCD), an iterative process that compresses and retains essential
information through novel importance metrics, effectively maintaining critical
data even without access to future context. Our comprehensive evaluations
indicate that InfiniPot significantly outperforms models trained for long
contexts in various NLP tasks, establishing its efficacy and versatility. This
work represents a substantial advancement toward making LLMs applicable to a
broader range of real-world scenarios.