Title: HiFC: High-efficiency Flash-based KV Cache Swapping for Scaling LLM Inference
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

inference

long context

KV cache

flash cache

swapping

HiFC

DRAM-free

vLLM

paged attention

SSD

pSLC

GDS

cost efficiency
Abstract: Large‑language‑model inference with long contexts often produces key–value (KV) caches whose footprint exceeds the capacity of high‑bandwidth memory on a GPU. Prior LLM inference frameworks such as vLLM mitigate this pressure by swapping KV cache pages to host DRAM. However, the high cost of large DRAM pools makes this solution economically unattractive. Although offloading to SSDs can be a cost-effective way to expand memory capacity relative to DRAM, conventional frameworks such as FlexGen experience a substantial throughput drop since the data path that routes SSD traffic through CPU to GPU is severely bandwidth-constrained. To overcome these limitations, we introduce HiFC, a novel DRAM‑free swapping scheme that enables direct access to SSD-resident memory with low latency and high effective bandwidth. HiFC stores KV pages in pseudo-SLC (pSLC) regions of commodity NVMe SSDs, sustaining high throughput under sequential I/O and improving write endurance by up to 8$\times$. Leveraging GPU Direct Storage, HiFC enables direct transfers between SSD and GPU, bypassing host DRAM and alleviating PCIe bottlenecks. HiFC employs fine-grained block mapping to confine writes to high-performance pSLC zones, stabilizing latency and throughput under load. HiFC achieves inference throughput comparable to DRAM-based swapping under diverse long-context workloads, such as NarrativeQA, while significantly lowering the memory expansion cost of a GPU server system by 4.5$\times$ over three years.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents HiFC (High-efficiency Flash Cache), an efficient KV cache management solution for large language model inference. Addressing the high cost of DRAM-based swapping and poor performance of conventional SSD offloading, HiFC enables direct GPU-to-SSD data transfer bypassing host DRAM while leveraging SSD's pseudo-SLC region for optimized I/O performance. Experimental results demonstrate comparable inference throughput (TPS) to DRAM solutions with significantly reduced costs. The method seamlessly integrates with existing vLLM frameworks and proves effective for long-context and high-batch inference scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths  \nS1)HiFC provides a practical way to reduce the expensive memory costs of running large language models by effectively using SSDs as extended GPU memory.   \nS2)The system works smoothly with existing popular frameworks like vLLM without requiring changes to model architectures.\n\nWeaknesses  \nW1)The paper focuses more on system building than new AI algorithms, so I believe its practical performance should be the most important criterion for evaluating its contributions.  It only tests with one GPU and one SSD, while real applications often use multiple GPUs and SSDs. We don't know how well HiFC would work in these more common setups.  \nW2)The experiments only use one model (DS-Qwen-32B) and fixed-length inputs (like 5K/10K tokens). In real-world use, LLMs handle mixed workloads with varying context lengths. We don't know if HiFC works as well with different model sizes or dynamic input lengths that change during operation.  \nW3)The paper could be more convincing if it includes a comparison with InstInfer, especially in terms of cost, latency, and compatibility. Such a comparison would provide a more comprehensive evaluation of the strengths and weaknesses of the proposed method relative to existing solutions.\n\nQUESTIONS:\nPlease refer to Weaknesses for details. I would increase the score if these points are adequately addressed.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a flash memory swapping scheme which swaps KV cache pages into an SSD drive. It is designed for LLM inference in memory-constrained environments, where the memory required for KV cache activations is too large to fit in GPU and thus the KV cache needs to be swapped / offloaded to a device with higher memory capacity. They present SSD drive swapping as an alternative to standard methods (offloading to CPU DRAM) which can achieve improved TCO. Their approach uses GPU-Direct-Storage (GDS) to transfer data directly from the GPU to the SSD. It attains similar throughput with DRAM swapping strategies with substantial reduction of TCO relative to providing access to large CPU DRAM pools. They also leverage custom allocation strategies to optimize for reuse in order to enhance device durability by promoting sequential memory accesses.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- They present a DRAM-free strategy with substantial TCO reduction by exploiting flash memory for offloading.\n- GDS eliminates the bottleneck of PCIe transfers when offloading to SSD (and allows high I/O transfers)\n- Their utilization of single-level cell modes is clever for improving write endurance as well as speeding up I/O transfers (as well as their other optimizations in terms of coalescing memory operations)\n- They design a flash-aware allocation strategy to optimize for reuse patterns in order to minimize wear on the SSD devices\n- They provide an end-to-end vLLM integration to demonstrate the real-world feasibility of their method\n- They present a thorough evaluation of their method, including stress testing their system throughput with different input/output configurations and demonstrating the importance of their sequential access patterns.\n\nWeaknesses:\n- This work mainly focuses on throughput-oriented LLM inference. For latency-oriented serving systems, this type of swapping-based method may create unsuitable latency overheads.\n- The TCO analysis is not thoroughly justified - there should be citations provided for the price ranges / other components in Table 1. This is particularly critical since claims around reduced power consumption / TCO are central to the aims of the paper (which aims to provide similar performance for reduced cost).\n- As mentioned in the limitations section, there are potential systems engineering challenges with maintaining SSD-based systems due to increased failure rate.\n\nQUESTIONS:\n- Are the price trends for these technologies (DRAM versus SSD) such that the TCO benefits from this method will shrink, persist, or grow over the coming years?\n- Are there any sustainability concerns from repeatedly burning through flash drives (even if this has lower TCO)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes HiFC, a novel DRAM-free approach for managing KV cache in large language model inference by directly swapping KV cache blocks between GPU memory and NVMe SSD storage. The main contribution of this paper is achieving comparable throughput to DRAM-based approaches while reducing total cost of ownership (TCO).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The DRAM-free design is innovative, bypassing the traditional CPU-DRAM data path.\n- A direct GPU-to-SSD data path is a significant architectural advancement.\n- The system leverages GPU Direct Storage, which sees the long-context problem from a system perspective, helping the community see the problem from another perspective.\n- Integration with vLLM makes HiFC practical.\n\nLimitations and Weaknesses:\n- Limited Experimental Scope: Offloaded-based KV cache management is mainly designed to extend the memory capacity. We see no comparison with other offloaded-based KV cache management with a DRAM baseline. Would offloading to SSDs maintain a larger context/more KV-cache than offloading to DRAM? That is an intriguing question this paper didn't answer.\n- The main contribution of this paper is reducing the total cost of ownership (TCO) by 4.5× over three years. We buy into this claim point, and we need to point out HiFC is working on a hybrid system with GPU/DRAM/SSD. And in most cases of cloud renting, the cost of owning a GPU is ten to a hundred times higher than the cost of owning DRAM or SSD. For example, ownership of 3-year DRAM costs \\\\$ 614 but ownership of 1-month A100 would cost over \\$1000. Therefore, saving 4.5× ownership of KV-cache management would be negligible compared with the total cost of ownership of the GPU.\n\nOverall, we think the DRAM-free design is innovative, but also believe in the design principle “lex parsimoniae”. Specifically, the HiFC adds an additional entity without performance gain, and TCO can be covered by the cost of GPU ownership. My biggest concern of the work is this is a practical / system contribution to the community but I don't think the gain / setting is justified. Does this design have impact on other settings or address other practical problems?\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}

comments: SUMMARY:
This paper introduces a system-level optimization to efficiently store KV cache. The authors propose to bypass DRAM, and do direct GPU–SSD swapping using fast SSD technology and NVIDIA’s GPU-Direct Storage. Specifically, they use pSLC regions of SSDs and seemelessly integrate with vLLM using a block-based memory allocator and swap engine. The authors present comprehensive results to support the efficacy of the proposed method.

STRENGTHS AND WEAKNESSES:
Strengths
- Motivation is clear. DRAM offloading is expensive and leads to extra latency overhead.
- Idea is novel. The paper leverages specific regions of the SSD to optimize the cost, ensuring DRAM free SSD swapping.
- Results show upto 4.5x TCO reduction in 3 years.

Weaknesses
- It is not clear how the system will scale with multi-GPU system
- Results reported on only one model. It will be helpful if authors can share results on other LLMs.

QUESTIONS:
NA

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-20