AlayaDB: The Data Foundation for Efficient and Effective Long-context
  LLM Inference
Authors: name: Yangshen Deng, name: Zhengxin You, name: Long Xiang, name: Qilong Li, name: Peiqi Yuan, name: Zhaoyang Hong, name: Yitao Zheng, name: Wanting Li, name: Runzhong Li, name: Haotian Liu, name: Kyriakos Mouratidis, name: Man Lung Yiu, name: Huan Li, name: Qiaomu Shen, name: Rui Mao, name: Bo Tang

AlayaDB is a cutting-edge vector database system natively architected for
efficient and effective long-context inference for Large Language Models (LLMs)
at AlayaDB AI. Specifically, it decouples the KV cache and attention
computation from the LLM inference systems, and encapsulates them into a novel
vector database system. For the Model as a Service providers (MaaS), AlayaDB
consumes fewer hardware resources and offers higher generation quality for
various workloads with different kinds of Service Level Objectives (SLOs), when
comparing with the existing alternative solutions (e.g., KV cache
disaggregation, retrieval-based sparse attention). The crux of AlayaDB is that
it abstracts the attention computation and cache management for LLM inference
into a query processing procedure, and optimizes the performance via a native
query optimizer. In this work, we demonstrate the effectiveness of AlayaDB via
(i) three use cases from our industry partners, and (ii) extensive experimental
results on LLM inference benchmarks.