TeleRAG: Efficient Retrieval-Augmented Generation Inference with
  Lookahead Retrieval
Authors: name: Chien-Yu Lin; name: Keisuke Kamahori; name: Yiyu Liu; name: Xiaoxiang Shi; name: Madhav Kashyap; name: Yile Gu; name: Rulin Shao; name: Zihao Ye; name: Kan Zhu; name: Stephanie Wang; name: Arvind Krishnamurthy; name: Rohan Kadekodi; name: Luis Ceze; name: Baris Kasikci
Published: 2025-02-28 11:32:22

Abstract: Retrieval-augmented generation (RAG) extends large language models (LLMs)
with external data sources to enhance factual correctness and domain coverage.
Modern RAG pipelines rely on large datastores, leading to system challenges in
latency-sensitive deployments, especially when limited GPU memory is available.
To address these challenges, we propose TeleRAG, an efficient inference system
that reduces RAG latency with minimal GPU memory requirements. The core
innovation of TeleRAG is lookahead retrieval, a prefetching mechanism that
anticipates required data and transfers it from CPU to GPU in parallel with LLM
generation. By leveraging the modularity of RAG pipelines, the inverted file
index (IVF) search algorithm and similarities between queries, TeleRAG
optimally overlaps data movement and computation. Experimental results show
that TeleRAG reduces end-to-end RAG inference latency by up to 1.72x on average
compared to state-of-the-art systems, enabling faster, more memory-efficient
deployments of advanced RAG applications.