HiFC: High-efficiency Flash-based KV Cache Swapping for Scaling LLM Inference
Authors: Inho Jeong
Sunghyeon Woo
Sol Namkung
Dongsuk Jeon
Venue: NeurIPS 2025 poster ()
Keywords: LLM
inference
long context
KV cache
flash cache
swapping
HiFC
DRAM-free
vLLM
paged attention
SSD
pSLC
GDS
cost efficiency
Primary area: deep_learning
TL;DR: HiFC swaps LLM KV caches directly between GPU and pSLC-SSD, matching DRAM-level throughput while eliminating DRAM and slashing cost five-fold.

Abstract: Large‑language‑model inference with long contexts often produces key–value (KV) caches whose footprint exceeds the capacity of high‑bandwidth memory on a GPU. Prior LLM inference frameworks such as vLLM mitigate this pressure by swapping KV cache pages to host DRAM. However, the high cost of large DRAM pools makes this solution economically unattractive. Although offloading to SSDs can be a cost-effective way to expand memory capacity relative to DRAM, conventional frameworks such as FlexGen experience a substantial throughput drop since the data path that routes SSD traffic through CPU to GPU is severely bandwidth-constrained. To overcome these limitations, we introduce HiFC, a novel DRAM‑free swapping scheme that enables direct access to SSD-resident memory with low latency and high effective bandwidth. HiFC stores KV pages in pseudo-SLC (pSLC) regions of commodity NVMe SSDs, sustaining high throughput under sequential I/O and improving write endurance by up to 8$\times$. Leveraging GPU Direct Storage, HiFC enables direct transfers between SSD and GPU, bypassing host DRAM and alleviating PCIe bottlenecks. HiFC employs fine-grained block mapping to confine writes to high-performance pSLC zones, stabilizing latency and throughput under load. HiFC achieves inference throughput comparable to DRAM-based swapping under diverse long-context workloads, such as NarrativeQA, while significantly lowering the memory expansion cost of a GPU server system by 4.5$\times$ over three years.

HiFC: High-efficiency Flash-based KV Cache Swapping for Scaling LLM Inference
Inho Jeong!. Sunghyeon 130
Sol Namkung!
Dongsuk Jeon't
'Scoul National University
SK hynix "NAVER Cloud
Abstract
Largellanguage-mdeel inference with long contexts often produces key-value (KV) caches whose footprint exceeds the capacity of high-bandwidth memory on a GPU. Prior LLM inference frameworks such vLLM mitigate this pressure by swapping KV cache pages to host DRAM. However, the high cost of large DRAM pools makes this solution economically unattractive. Although offloading to SSDs can cost-effective way to expand memory capacity relative to DRAM, conventional frameworks such as FlexGen experience substantial throughput drop since the data path that routes SSD traffic through CPU GPU is severely Sandwidth-constrained To overcome these limitations, introduce HiFC, novel DRAM-free swapping scheme that enables direct access to SSD-resident memory with low latency and high effective bandwidth. HiFC stores KV pages in pscudo- SLC (pSLC) regions of commodity NVMe SSDs, sustaining high throughput under sequential I/O and improving write endurance by up to 8x. Leveraging GPU Direct Storage, HiFC enables direct transfers between SSD and GPU, bypassing host DRAM and alleviating PCIe bottlenecks. HiFC employs fine-grained block mapping to confine writes to high-performance pSI.C zones, stabilizing latency and throughput under load. HiFC achieves inference throughput comparable DRAM- based swapping under diverse long-context workloads, such as NarrativeQA, while significantly lowering the memory expansion cost of GPU server system by 4.5 over three years.
1
Introduction
Large language models (LLMs) have demonstrated state-of-ther thr---r performance across wide range of tasks, including machine translation, code generation, and open- ween-domiin question answering 12 4 [. However, L.LM services are often hindered by significant memory requirements due to the storage of key-value (KV) caches 6
7 g [9. The size of this KV cache grows proportionally with both the number of layers and the input sequence length, leading to substantial memory usage in state-of-the-ar LLMs. This issue becomes particularly pronounced in long-context scenarios, where the cumulative KV cache can easily exceed the capacity aighbaandwitt memory (HBM) on GPU, resulting in out-of-memory errors or degraded throughput.
To overcome GPU HBM capacity limits, several LLM inference frameworks 10
swap or offload KV cache blocks to host DRAM. However, provisioning and maintaining large DRAM pools requires substantial capital investment, increases power consumption, and demands more intensive cooling, ultimately inflating the cost for long-context inference deployments.
Attempts leverage lower-cost, higher-capacity NVMe SSDs for KV cache offloading have also been explored recently 7 [0 [3) However, since GPUs lack a native NVMe interface, every
'Equal contribution. Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

DRAM Buffer
GPU(HBM) KV Cache Blooks
Performance and swap count comparison
Throughput (tokensis)
440
R
Reguest_
Swq 2
120
Sway count
120
Request
100
80
RaquesL.
HIFC KV Cache Blocks
Swap out Swen 0
HIF Swapping
40
DRAM swapping
30
Batch
(a) HiFC handles GPU-to-Flast KV cache eviction and prefetch operations without involving DRAM, enabling efficient block- level swap management during decoding.
(b) Comparisons of throughput and total swap counts between HiFC and DRAM swapping
systems under increasing hatch sizes.
Figure 1: HiFC workflow and performance. (a) HiFC decoding workflow: (1)-(3) prompts are tokenized and sequences are allocated KV-cache blocks HBM; (4) decoding produces additional KV entries; (5) under HBM memory pressure, the scheduler selects a victim sequence and swaps the victim sequence KV cache from HBM to Flash cache (FC); (6) the freed GPU KV blocks are immediately reallocated to the next active sequence; (7) decoding of other sequences proceeds without blocking. (b) Comparison of throughput and total swap counts between HiFC and DRAM swapping systems under increasing batch sizes, showing that HiFC achieves comparable performance to DRAM swapping.
Flash I/O transaction must traverse host DRAM and the PCIe fabric, creating a severe bandwidth bottleneck [33. Morcover, frequent, non-sequential writes increase Write Amplification (WA) [14],
which severely accelerates Flash wear and undermines device endurance under sustained operations.
In this paper. we propose HiFC (High-efficiency Flash Cache), novel DRAM- paradigm for LLM inference that reduces the memory expansion cost without sacrificing inference throughput. HiFC efficiently offloads the large volume of KV cache generated during LLM inference to Flash- based storage, enabling effective reclamation of GPU HBM and improving sequence-lewe parallelism.
Fig. Ia sequentially illustrates how HiFC is utilized when the HBM capacity is insufficient during the decoding phase. Specifically, HiFC leverages pseudo-SLC (pSLC) regions in Flash to deliver stable high throughput under sequential I/O workloads, while simultaneously improving write endurance, which is measured in total bytes written (TBW), by up to 8x. HiFC further employs GPU Direct Storage (GDS) (15) to bypass host memory and climinate PCIc bottlenecks. Finally, HiFC applies fine-ge anee-graied block mapping to confine writes to high-performance SLC zones. The proposed DRAM- free architecture achieves comparable throughput
when batch size increases (Fig, Tb) while reducing memory expansion cost by 4.5x, providing a scalable solution for demanding long-context inference benchmarks such LongBench [16].
The main contributions of this work are as follows:
1. DRAM-Free Design. We show that large-scale L.M inference can be executed without relying host DRAM buffers for KV cache swapping, significantly reducing both upfront memory provisioning costs and sustained system-level power consumption.
2. Optimized GPU-SSD Direct Swapping. By leveraging GDS and pSLC optimizations, we enable direct and efficient data transfers between GPU and a Flash-based KV cache, bypassing the bottlenecks of host CPU and DRAM-bart based buffering.
3. Seamless Integration with vLLM Framework. We demonstrate the successful integration of HiFC into the vLLM inference engine. Experimental results confirm that our method can be adopted in real-world frameworks without requiring architectural changes compromis- ing system stability, achieving a 4.5x lower memory expansion cost and accommodating 4x more requests without performance degradation compared to DRAM-based swapping.

---

2 Background
2.1 Importance of KV Cache Management in LLM Service
Inference in decoder-based LLMS consists of two distinct phases: the prefill phase and the decode phase [7]. In the prefill phase, the model processes the entire input prompt in parallel. At each self-attention layer, the key and value vectors for all prompt tokens cache. In the decode phase, the model generates one token at a time. For each decoding step, cached key and value vectors from prior tokens are reused to compute self-attention, enabling efficient auto-regressive generation without reprocessing the full sequence at every step.
stored, denoted as the K
While this cache-based mechanism significamtly improves computational efficiency during decoding, a substantial amount of memory is required to store the KV cache. Specifically, the memory footprint of the KV cache scales proportionally with the batch size, hidden size of the model, and the input sequence length. As a result, scenarios involving large batch sizes or long input contexts can lead to excessive memory usage and potential out-of- memory failures. For instance, serving the DeepSeek- RI model [5] with merely a few hundred tokens per request can consume hundreds of gigabytes of GPU memory. Therefore, efficient KV cache management is critical for enabling scalable and high-throughput LLM inference, particularly in long-context or multi-request serving environments.
2.2 Existing KV Cache Management Techniques
Early LLM serving systems managed the KV cache by pre-allocating a single contiguous GPU buffer sized for maximum context length of each request. Since real sequences are usually shorter, this approach suffers severe internal fragmentation; empirical reports show that only 20-40% of the reserved KV cache space contains useful data, while the rest is wasted [6]. Two complementary techniques have emerged to overcome this limitation:
Offloading moves model weights or KV tensors out of GPU HBM to lower-tier devices (e.g., host
DRAM or SSD) and keeps them there for many decoding steps IC IR 19 exemplifies this strategy: it treats GPU, CPU, and NVMe storage as a unified hierarchy, offloads seldom-used layers or KV blocks, and compresses them to 4 bits to amortize I/O cost. This allowed a single 16 GB GPU to run a 175B-parameter model at reasonable throughput. The trade-off is higher latency whenever disk access is frequent, which can be unacceptable for interactive workloads.
20 21 22]. FlexGen 7
Swapping, on the other hand, evicts and restores data on demand in small pages or sequence-level blocks 6 23 24). For instance, PagedAttention in vLLM 6 splits KV cache of each request into uniform blocks of 16-128 tokens. When HBM fills up, selected blocks are temporarily moved to DRAM and fetched back as soon as the request resumes. Uniform page size drives internal and external fragmentation below 5%, and the scheduler caps the swapped volume at the HBM cache size to bound latency. Consequently, vLLM can serve many concurrent requests with long contexts while maintaining low tail latency. In addition, vLLM integrates PagedAttention with block-level KV virtualization and a BlockTable to eliminate fragmentation [0O. During decoding, an LRU-plus priority policy selects victim sequences for asynchronous swap-out to DRAM without pausing GPU exccution (25). A prefetch queue performs swap-in with a single synchronization before kernel launch, enabling a pipelined
Saatinng-Sa---uu Sarup-OutSSwart Sw design that sustains high tensor utilization and throughput [26]. A broader overview of approaches using Computational Storage Devices (CSDs) [13] and techniques targeting larger-scale scenarios in KV cache management [2 28 is
provided in Appendix A
3 HiFC Architecture
3.1 Motivation and Design Challenges of DRAM-Free Systems
As discussed in Section 2.2 offloading KV caches to host DRAM or SSD 7 a common approach to alleviating GPU memory pressure during long-context LLM inference. While offloading KV cache blocks to host DRAM can mitigate the limited capacity of GPU HBM, this approach incurs high workloads such as long-sequence
operational cost and remains insufficient for memory-intenste processing or agentic AI, where a large amount of KV cache is required. Alternatively, offloading to commodity NVMe SSDs can reduce infrastructure cost and provide better memory scalability,
3

---

KV Block Manager
Scheduler
Worker(GPP)
Running
Cache Engine GPU-FC Swapping Engine
Swapped
DDR
DDR
HBM
HBM
GPU Block Allocator
CPU Block Allocator
FC Block Allocator
Swapt
SSD
(GDS)
(Flash cache)
Figure 2: HiFC extends Block Manager of vLLM by integrating FC block allocator, enabling direct GPU-SSD KV cache swapping without relying host DRAM. The scheduler manages KV block placement based on sequence execution states, and the CUDA-based cache engine leverages GDS to provide an optimized data path between SSD and GPU HBM.
enabling support for larger KV caches and longer contexts. However, this strategy introduces severe throughput degradation, as each I/O must still traverse host DRAM and the PCIe bus. Under bandwidth -imited conditions, this indirect data path significantly impairs decoding performance [13]
These limitations motivate the design of DRAM-free offloading architecture that bypasses interme- diate memory staging and enables direct, high- highhbandwidth data transfer between GPUs and SSDs. This architecture avoids cumulative latency and bandwidth bottlenecks inherent in existing approaches, while simplifying the overall system design and reducing operational cost.
However, realizing this architecture faces several critical technical challenges. First, SSDs typically offer lower I/O throughput than DRAM, particularly under fragmented or random access patterns, making it difficult to sustain high performance during inference. Second. Flash memory suffers from limited endurance; frequent small writes increase WA and accelerate device wear-out, degrading long-term reliability. Third, traditional offloading paths rely CPU DRAM intermediate buffer, which introduces additional PCLe transfers and redundant memory copies, leading to increased latency and inefficient resource utilization.
3.2 HiFC: High-efliciency DRAM-Free Flash Cache Architecture
In Section 3.1 we analyzed the necessity and challenges of building DRAM-free systems. In this section, propose HiFC, first DRAM-free design that eliminates the need for host DRAM in KV cache management. HiFC enables a scalable and cost-effective DRAM-free inference stack without compromising throughput or reliability. HiFC achieves these improvements through three tightly integrated components: (1) Flash cache (FC) block allocator for direct SSD-based KV cache allocation, (2) Flash-aware block management algorithm maximize the performance and lifespan of the Flash storage, and (3) GDS-accelerated cache engine for efficient KV cache swapping between GPU and SSD.
Flash Cache Block Allocator. HiFC extends the vLLM memory allocator by introducing FC blocks alongside the existing GPU and CPU blocks. During the decoding phase, when GPU memory exhausted, existing KV cache blocks
swapped out to FC blocks to free up memory. These FC blocks are managed by the FC block allocator, which applies block append policy to generate sequential I/O workloads the SSD. All blocks are of fixed size (e.g.. 32-128 tokens) maintain compatibility with block-level scheduling of vLLMM and minimize fragmentation.
Flash-Aware Block Management. Unlike DRAM or GPU memory, SSDs are highly sensitive to random access and block reuse patterns. To address this, the FC block allocator employs a Flash- aware block allocation strategy that ensures the sequential physical placement of KV blocks, thereby reducing internal fragmentation and minimizing WA. Specifically, the module allocates the blocks evicted Flash physical order and manages them with a simple append policy that avoids reusing stale logical addresses. This sequential access pattern improves throughput and significantly extends

---

SSD lifespan 129], achieving WA factor below 1.02, compared to 1.4 in conventional SSD usage scenarios. These results are validated in Section 5.5
GPU-Flash Cache Swapping Engine. To maximize throughput and eliminate intermediate buffer- ing, HiFC directly transfers GPU-resident KV tensors SSD using GDS [I5] with byte-level offsets following the scheme in Appendix B for efficient 1/O dispatch. To this end, we integrate Flash cache swapping functionality into the vLLM cache engine. To further enhance I/O efficiency, HiFC leverages multi-threaded I/O scheduling (up to 16 threads) [TO] and the direct reuse of tensors as 4KB-all -ll-aaigeed GDS buffers [30]. This design sustains throughput of over 4.7 GiB/s the pSLC region of SSD, ensuring reliable KV cache swapping operations. As result, KV swapping with HiFC achieves LLM inference performance comparable that of DRAM-based approaches, which is validated by our results in Section 5.1
HiFC Architecture and Operation. This combination of techniques establishes an efficient and scalable DRAM-free memory hierarchy, balancing cost, capacity, and performance for long-contex: LLM inference. At its core, HiFC replaces the CPU block allocator used by vLLM with dedicated Flash Cache block allocator, which incorporates Flash-aware block management. This enables a dynamic swapping mechanism: during the decoding stage, when GPU memory becomes scarce, the scheduler identifies a victim sequence group and instructs the worker to initiate swap-out. The cache engine of worker executes this operation, utilizing a custom CUDA kernel transfer KV cache data directly to SSD via GDS for maximum throughput. Once sufficient GPU memory is reclaimed, the scheduler directs the worker to swap the swapped sequence group back into GPU memory. As illustrated in Fig. 2 this entire data flow facilitates direct GPU-HiFC KV cache transfers, completely bypassing the host DRAM.
4 Memory Expansion Cost Estimation
To evaluate the long-term costteffeetive of different KV cache storage mediums, we compute the memory expansion cost over a 3-year deployment using Eq. (T), which based on the Total Cost of Ownership (TCO) model. The cost consists of two components: the capital expenditure (CapEx) required provision the storage capacity, and the operational expenditure (OpEx) determined by sustained power consumption, datacenter power usage effectiveness (PUE), and regional electricity cost, with all costs denominated in USD.
MemExpCost = CapEx + (Power 24 x 365 x Years x PUE x EnergyCose
1000
Table 1: Three-year memory expansion cost comparison across memory types.
Metric
DRAM
Enterprise SSD
HiFC (ours)
Saving
Capacity
128 GiB
1.92
1 TiB
CapEx
$433
$270
$118
3.6x
Power
64 W
8.2 W
W
12.8
3-year Cost
$614
$303.6
S136
4.5x
Table I summarizes the 3-year memory expansion cost for three KV cache storage configurations: DRAM (DDR4), enterprise-grate TLC-based SSD, and HiFC (pSLC SSD). The HiFC configuration achieves a 3.6x reduction in capital expenditure and 12.8x lower power consumption compared to DRAM. This results in a total memory expansion cost of only $1 $136, which is 4.5x lower than the DRAM-based approach. Further details, including projections for future price changes, are provided in Appendix [ Directly utilizing the TLC region of commercial SSDs introduces performance degradation and reduced endurance, leading to cost-inefficiencc. HiFC addresses this problem by selecting commercial SSDs that support a pSL. region and operating exclusively within that region, which constitutes 20% of the total capacity in the selected SSDs. This approach extends the drive's lifespan by up to 8x and ensures stable performance. The details of the lifespan are provided in Appendix D
5

---

Experiments
To ensure consistent and accurate performance benchmarking, we first selected the DeepSeek- H.-DistillQwen-333
(DS-Qwen-32B) model [S] 31 and conducted measurements under fixed configurations. To validate the robustness and general applicability of HiFC, we then extended our evaluation to include diverse models and datasets, with these results presented in Section 5.3 In particular, we leveraged the wicro-benchmarking utilities provided by vLLM framework to construct precise execution scenarios and evaluate inference performance quantitatively. The detailed hardware, software, and workload configurations for all experiments are provided Appendix
5.1 Performance and KV Cache Swapping Comparison
Throughput (tokens/s)
Throughput (tokens/s)
120
20
40
100
Swap count
Swap count
80
80
60
40
HFCC swapping
DRAM swapping
20
HICC
DRAM swap count
10000
Input sequence length
Output sequence length
(a) The output sequence length fixed at 1k.
(b) The input sequence length fixed at 1K.
Figure 3: Performance and swap count comparison of HiFC and DRAM swapping DS-Qwen- 32B with batch size of 10. (a) As input length grows, throughput for both methods decreases, while swap counts peak around 3k tokens. Both methods show very similar trends. (b) As output length grows, throughput decreases at longer sequence lengths (after 5k), while the swap count steadily increases.
We trigger swap operations by progressively increasing both input and output sequence lengths and compare their impact on system performance. Fig. presents a comparative analysis of KV cache swap performance between HiFC and DRAM under various input and output configurations, where the throughput achieved by both memory types remains comparable. In Fig, [al as the input length increases, more KV cache is generated during the prefill phase, but fewer sequences remain active in the decoding stage, reducing effective concurrency and thus throughput. This also leads to fewer swap events.
In contrast, Fig. 3b demonstrates that increasing output length expands KV cache during the decoding phase, allowing for higher concurrency initially. However, as output grows, swap activity surges, while throughput gradually decreases after 5k tokens. Even under this 1/O-heavy condition, HilC achieves near-identical throughput to DRAM, validating its ability to sustain high throughput while maintaining comparable latency, also confirmed by our detailed analysis Appendix
5.2 Scalability in Hardware Topologies
To evaluate hardware scalability, tested HiFC under various GPU-to-SSD ratios. In multi-GPU setups, the pSLC space for the KV cache is partitioned and assigned evenly to each GPU, mirroring the cache management strategy in vLLM. This approach supports flexible deployment topologies, including:
* 1:11 (GPU:SSD) A baseline configuration for standard workstations.
N:11 - Multiple GPUs sharing a single SSD, optimizing for cost-efficiency.
N:N Multiple GPUs paired with multiple SSDs (e.g.. in RAID configuration) to maximize both computational performance and I/O bandwidth.
We evaluated the scalability HiFC across different GPU-SSD topologies using the DeepSeek-RR R1
Distill- Llama-8B model two A.00 GPUs. The workload consisted of batch size of 200 from the

---

GovReport dataset (8k input length on average) with a fixed Ik output length. The request size was calculated based on an average context length 9k.
Table 2: Performance across different GPU-SSD topologies.
Setting
DRAM PSLC Size
Cached requests
Throughput (tokens/s)
GPU+DRAM
50 GiB
51
172
GPU+SSD (1:1)
200 GiB
206
182
GPU+SSD (2:1)
200 GiB
206 (103 + 103)
367
GPU+SSD (2:2)
400 GiB
412 (206 + 206)*
364
(*) Using Data Parallelism, each GPU processes an equal number of requests.
The results in Table demonstrate strong scaling capabilities of HiFC. First, it was observed that the HiFC configuration with GPU:SSD = 1:1 achieves comparable performance to the GPU+DRAM baseline. We then scaled the system using Data Parallelism (DP). In 2:1 configuration, the batch was split across both GPUs, achieving 2 performance increase relative to the baseline. Notably, no I/O bottleneck was observed. Finally, the 2:2 configuration, also operating in the DP mode, maintained this 2 performance gain while simultancously doubling the total cached request capacity. This N:N setup similarly demonstrated no I/O bottlenecks, confirming HiFC's ability to scale efficiently.
5.3 Robustness across Diverse Models and Datasets
Beyond hardware scaling, we validated performance of HiFC more realistic scenarios. While primary experiments used fixed model and context length for fair comparison, real-world deployments must handle diverse workloads. We therefore conducted additional tests using multiple popular open-source models (DeepSeek-Llama-8) [32], DeepSeek- 14B [33), and Mistral- 7B 344) and datasets with context lengths ranging from a few hundred to over 18k tokens.
As shown in Table 3 HiFC maintains performance similar DRAM-based swapping across all tested models and datasets. The measured throughput difference remains negligible, with HiFC performing within 1-2% of DRAM, and in some reliability and efficiency of HiFC in practical, diverse inference environments.
outperforming it. These results underscore the
Table 3: Measured throughput (tokens/s) diverse models and datasets.
Model
KV Swap
Qasper
GovReport (avg. 8.7k length)
NarrativeQA (avg. 18.4k length)
(avg. 3.6k length)
DS-Llama-SB
DRAM
302.0
172.0
95.1
HiFC
301.3
182.3
95.8
DS-Qwen-14B
DRAM
176.4
100.9
46.4
HiFC
175.3
103.5
46.6
Mistral-7B
DRAM HiFC
281.0
163.0
416.1*
275.2
162.0
406.8*
()) Some requests truncated due to Mistral-7B's 32k context-length limit.
5.4 Impacts of KV Swapping on Throughput
Fig. illustrates how the vLLM scheduler determines swap events based on the available GPU KV cache budget. We fixed the batch size to 20 and increased the number of concurrently pipelined sequences examine the emergence and impact of swap behavior. Experiments were conducted on an NVIDIA 1000 80 GiB GPU, where 90% of the memory (71 GiB) was allocated: 61 GiB to model weights, 0.97 GiB activation buffers, and the remaining 9.1 GiB KV cache.
Based on our theoretical model, when the context length is Sk tokens, the total KV cache footprint exceeds 9.1 GiB beyond 8 concurrent sequences, triggering Flash swap operations. As shown in Fig. swap events for both HiFC and the DRAM baseline begin precisely at this threshold. Although swap counts increase with additional sequences, throughput in both systems continues increase

---

25
Throughput (tokens/s)
1440
Throughput (tokens/s)
220
120
Sway count
Swap p count
100
80
80
20
Output TPS
20
Output TP5
HIFC swap count
DRAM swep count
Max number of sequences
Max number sequences
(a) HiPC swapping.
(h) DRAM swapping.
Figure 4: Performance and swap count trend of HiFC and DRAM swapping. Each sub-figure shows the average output throughput and the number of swapping event as the max number of sequences increases for the DS-Qwen-322B model, with context length 5k, batch size 20, block size = 32, and 100 GiB of dedicated swap space for KV cache for both methods.
before saturating at 12 sequences. This indicates that the swap overhead is effectively absorbed, demonstrating that using HiFC does not negatively impact inference performance compared to DRAM, as both methods become compute-bound, instead of 1/O-bound.
This observation explained by the increasing slack introduced by deeper sequence pipelining. Each additional sequence widens the pipeline's scheduling window, enabling the system to overlap HiFC I/O latency with computation. Appendix G analyzes this effect in detail, showing how pipeline- induced throughput gaps provide sufficient stall budget to hide swap delays, resulting in stable performance even under high swap frequency.
5.5
Validating KV Block Management
Allocation KV Block Numbers
KV Block Number
3000
Append
End number
2500
Left Append
Table 4: Achieving near-ideal WAF with HiFC swapping.
2000
1500
Metric
Value
WAF
1000
500
Swap out Data Units Written
+232.5 GiB
+487,595
1.02
Stert number
Swap Out Sequence
1000
3000
SSD Specification HiFC vs. SSD Spec.
1.40
-27%
Figure 5: Allocation patterns of KV blocks during swap-out operations under different append strate- gies.
Fig. 5) shows the index distribution of KV blocks during swap-out operations, illustrating how different free block allocation strategies affect 1/O behavior. When using the proposed append strategy, FC KV blocks are allocated in sequential manner, resembling a typical Least Recently Used (LRU) pattern. In contrast, the left-append strategy yields markedly different pattern: while the initial 500 swap-out events remain sequential, subsequent allocations become highly non-sequential. This transition reflects shift from sequential to random I/O access pattern.
Furthermore, Table compares the total write volume recorded during swap-out operations with the actual data written to the SSD as reported by the SMART (55) log. In our experiments. 487,595 data units (DUs) were written to the device, equivalent to 232.5 GiB (assuming 512kB per DU). This volume corresponds to a WA factor of 1.02. These findings underscore the importance of sequential block allocation not only for throughput but also reducing unnecessary Flash wear. The detailed equation is described in Appendix

---

Our evaluation of a commercial SSD quantifies the performance gap between sequential and random 1/O workloads. The sequential access patterns achieve significantly higher throughput, confirming that preserving the spatial locality of KV blocks is critical for maximizing HiFC's performance. The performance evaluation results are presented in Section 5.7
5.6 Impacts of Block Size on Swap Latency Hiding
KV swapping performance
Swap count comparison
Performance comparison
Throughput (tokens/s)
Throughput (GIB/S)
Swap count
120
203
1SC15
Block size (tokens)
Block size (tokens)
Block size (tokens)
(a) KV swap throughput (GiB/s)
(b) Swap
(c) End-to-end throughput
Figure 6: Block size analysis demonstrating that HiFC matches DRAM's end-to-end performance (c), the raw swap throughput gap (a) effectively amortized by concurrency.
Fig. 6 compares the impact of block size on KV cache swapping performance for HiFC and DRAM under a long-context workload (context length = 10k, batch size = 10). Fig. 6a shows that both HiFC and DRAM exhibit increasing swap throughput with larger block sizes, where DRAM consistently achieves more than twice the throughput of HiFC. This trend is consistent with prior work such as vLLM 6 which adopts 32-token blocks by default to balance swap cost and fragmentation under paged attention. However, HiFC still achieves nearly identical end-to-end throughput, shown in Fig. 6c
As shown in Fig. 6b the number of swap events is dictated by the workload and block size, resulting in an identical count for both HiFC and DRAM. This finding is crucial as it validates the end-to-end performance comparison in Fig. 6cc
We also observe that larger block sizes, such as 128 and 256, can induce more swap events than the 64-token sweet spot, suggesting redundant swapping.
Fig.66c [c]/ccoirmm that end-to-end throughput is improved modestly with larger block sizes, suggesting that coarser-graine grained swapping has minimal adverse effectt inference performance. This finding aligns with LMCache [25], which also showed that grouping adjacent tokens physically aligned blocks improves overall throughput without harming latency, especially under decoding workloads.
Overall, these results suggest that the swapping performance gap between HiFC and DRAM does not significantly affect either the swap frequency end-to-end inference performance, as latency is effectively amortized by concurrent sequence execution. By aligning the block size with the sweet spot observed in Fig. 6b (e.g., 64 tokens), we can maximize swap efficiency without requiring manual tuning for similar workloads.
5.7 1/O Throughput Comparisons
Table 5: Sustained GDS I/O throughput across workloads.
I/O Workload
LBA Range
Min (GiB/s)
Max (GiB/s)
Avg (GiB/s)
SEQ WRITE
100 GiB
4.341
4.724
4.715
SEQ READ
100 GiB
4.985
4.988
4.987
RND WRITE
100 GiB
1.092
2.703
1.617
SEQ WRITE
GiB
1.416
1.841
1.689
Table 5 shows SSD performance benchmarks using the gdsio tool
to evaluate the effects of different I/O types and LBA ranges. Sequential reads and writes within the PSL.C region delivered consistently high throughput, whereas random writes were significantly slower due to internal
15

---

mechanisms such as garbage collection and TLC migration. When the write workload exceeded the pSLC capacity, performance degraded significantly. These findings suggest that structuring KV cache access sequential I/O is essential for achieving iigh-eerfrmmnce Flash cache swapping. When the GPU's compute throughput is sufficiently high, the overhead of KV cache swapping can emerge as a bottleneck in the sequence pipelining process.
5.8 Scalability of KV Cache Initialization: HiFC vs. DRAM
Table Comparisons of LLM session initialization time.
KV Cache Size (GiB)
10
20
30
40
50
60
70
80
90
100
HiFC (8)
33.0
34.0
33.0
33.0
33.0
33.0
33.0
33.0
32.0
32,0
DRAM (s)
39.0
46.0
46.0
59.0
59.0
60.0
89.0
89.0
90.0
90.0
A (= DRAM-HiFC) (8)
12
13
26
26
27
56
56
58
58
HiFC significantly reduces the initialization time of LLM inference sessions by eliminating the use of DRAM for KV cache storage. Table 6 reports the LLM session initialization time across different KV cache sizes (ranging from 10 GiB to 100 GiB) for two memory configurations: HiFC and DRAM. HiFC shows consistently low initialization latency around 32-34 seconds regardless of the KV cache size. In contrast, DRAM- particularly beyond 60 GiB, reaching up to 90 seconds 100 GiB. Notably, HiFC's performance advantage becomes more pronounced at larger cache sizes: at 100 GiB, the time gap between DRAM and HiFC reaches 58 seconds, resulting in 2.81 X speedup (90.0s
initialization time increases significantly with cache size,
32.0s). This gap arises because HiFC directly utilizes pre-generated Flash cache files, whereas the DRAM baseline must allocate tensors for every KV block upon each initialization. These findings highlight the scalability advantage of Flash-based KV cache initialization, making it compelling solution for large-batch or long-context LLM inference scenarios.
6 Limitation and Future Plan
While HiFC eliminates the need for DRAM and achieves cost-efficient scalability through direct GPU-SSD I/O, several limitations remain. First, in short-context or latency-sensitive workloads, Flash access latency may introduce delays in token processing. Second, in multi-GPU settings, efficient bandwidth scheduling for shared Flash cache becomes increasingly critical avoid contention. Third, maintaining consistent SSD performance often requires domain-specifie expertise in system configuration and filesystem tuning, which may hinder practical deployment. As future work, we plan to explore hybrid Flash-DRAM caching schemes that maintain stable performance across diverse workloads, and to share resources and know-how for SSD optimization.
Conclusion
In this work, we introduced HiFC, novel DRAM-free architecture designed efficiently provide KV cache memory expansion for large-context LLM inference. HiFC replaces conventional DRAM-basee swapping with a GDS-enabled SSD utilizing its pseudo-SLC region, thereby creating a direct GPU- to-SSD data transfer pathway that eliminates intermediate host memory bottlenecks. Our experiments demonstrated that HiFC maintains inference throughput comparable traditional DRAM swapping methods. In addition, our extended validation confirms this advantage holds across diverse models, datasets, and even in multi-GPU scaling scenarios, highlighting HiFC's robustness and general applicability. Furthermore, for long-context and multi-batch scenarios, vLLM's 's sequence pipelining scheduler and HiFC's optimized Flash cache worked in concert to effectively manage KV cache swaps, thereby minimizing overhead. Simultaneously, substituting 128 GiB of DRAM with 1 TiB pSLC-configured SSD resulted in an approximately fivefold reduction in the three-year memory expansion cost. Overall, HiFC provides an economically viable and efficient pathway to scalable LLM inference, proving particularly beneficial for large-context and large-batch inference scenarios.
10

---

Acknowledgments and Disclosure of Funding
This work was supported in part by the Institute of Information and Communications Technology Planning and Evaluation under Grant RS-2021-1121 Grant UITP-2025-RS-2222-005566
and Grant RSS20024-00347394 and in part by the National Research Foundation of Korea under Grant RS-2024-00408040 and Grant RS-2022-001444
The authors would like to thank SK hynix for financial support during Inho Jeong's graduate studies and NAVER Cloud for valuable discussions.
References
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- tion Processing Systems (NIPS), pages 5998-6008, 2017.
[2] Justin Zhao, Timothy Wang, Wael Abid, Geoffrey Angus, Arnav Garg, Jeffery Kinnison, Alex Sherstinsky, Piero Molino, Travis Addair, and Devvret Rishi. Lora land: 310 fine-tuned Ilms that rival gpt-4, A technical report. CoRR, abs/2405.00732, 2024.
[3] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, et al. The llama herd of models. CoRR, abs/2407.21788, 2024.
[4] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, et al. Mixtral of experts. CoRR, abs/2401 004088, 2024.
[5] DeepSeek-Al, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. CoRR, abs/2501.12948, 2025.
[6] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Jason Flinn, Margo I. Seltzer, Peter Druschel, Antoine Kaufmann, and Jonathan Mace, editors, Proceedings of the 29th Symposium on Operating Systems Principles, SOSP 2023, Koblenz, Germany, October 23-26, 2023, pages 61 1-262. ACM, 2023.
[7] Sheng Shen, Zhenyu Zheng, Ji Lin, Yujia Qin, Yuxiang Hu, Jie Tang, and Song Han. Flexgen: tligh-throughput generative inference of large language models with limited gpu memory. In Proceedings of the 2023 ACM International Conference on Machine Learning (ICML), 2023.
[8] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark W. Barrett, Zhangyang Wang, and Beidi Chen. H2O: heavy-hitter oracle for efficient generative inference of large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023.
[9] June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, and Dongsoo Lee. No token left behind: Reliable KV cache compression via importance-awas aware mixed precision quantization. CoRR, 005/2402.18066 2024.
[10] Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng
Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, and Olatunji Ruwase. Deepspeed inference: Enabling efficient inference of transformer models at unprecedented scale. arXiv preprint, arXiv:2207.00032. 2022.
[11] Jianbo Wu, Jie Ren, Shuangyan Yang, Konstantinos Parasyris, Giorgis Georgakoudis, Ignacio Laguna, and Dong Li. Lm-offload: Performance model-guided generative inference of large language models with parallelism control. 2024.
[12] Cheng Luo, Zefan Cai, Hanshi Sun, Jinqi Xiao, Bo Yuan, Wen Xiao, Junjie Hu, Jiawei Zhao, Beidi Chen, and Anima Anandkumar. Headinfer: Memory-efficient L.LM inference by head- wise offloading. CoRR, abs/2502. 12574, 2025.

---

[13] Xiurui Pan, Endian Li, Qiao Li, Shengwen Liang, Yizhou Shan, Ke Zhou, Yingwei Luo, Xiaolin Wang, and Jie Zhang. Instinfer: In-storage attention offloading for cost-effective long-context IIm inference. https://arxiv..rrg abs/ 2409 04992 2024.
[14] Nitin Agrawal, Vijayan Prabhakaran, Ted Wobber, John Davis, Mark Manasse, and Rina Panigrahy. Design tradeoffs for ssd performance. In USENIX Annual Technical Conference, pages 57-70, 2008.
[15] NVIDIA Corporation. Nvidia gpudirect storage. https:// nwidia. com/blog/ spudirect-storagee 2021. Accessed: 2025-05-14.
[16] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. Longhench: A bilingual, multitask benchmark for long context understanding, 2023.
[17] Ajay Patel, Bryan Li, Mohammad Sadegh Rasooli, Noah Constant, Colin Raffel, and Chris Callison-Burch. Bidirectional language models are also few-shot learners. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[18] Xuanlin Jiang, Yang Zhou, Shiyi Cao, Ion Stoica, and Minlan Yu. NEO: Saving gpu memory crisis with epu offloading for online LLM inference, 2024.
[19] Yi Xiong, Hao Wu, Changxu Shao, Ziqing Wang. Rui Zhang, Yuhong Guo, Junping Zhao, Ke Zhang, and Zhenxuan Pan. Layerkv: Optimizing large language model serving with layer-wise KV cache management, 2024.
[20] Keivan Alizadeh, Seyed Iman Mirzadeh, Dmitry Belenko, S. Khatamifard, Minsik Cho, Carlo C. Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. LLM in a flash: Efficient large language model inference with limited memory. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 12562-12584, Bangkok, Thailand, 2024. Association for Computational Linguistics.
[21] Aditya Dhakal, Pedro Bruel, Gourav Rattihalli, Sai Rahul Chalamalasetti, and Dejan Milojicic. LLM serving with efficient KV-cache management using triggered operations. In Cray User Group Conference (CUG), 2024.
[22] Yupeng Tang, Runxiang Cheng, Ping Zhou, Tongping Liu, Fei Liu, Wei Tang, Kyoungryun Bae, Jianjun Chen, Wu Xiang, and Rui Shi. Exploring CXL-based KV cache storage for LLM serving. NeurlPS 2024 ML. for Systems Workshop, 2024.
[23] Foteini Strati, Sara Mcallister, Amar Phanishayee, Jakub Tarnawski, and Ana Klimovic. Déjävu: KV-ccche streaming for fast, fault-tolerant generative LLM serving, 2024.
[24] Jie Ye, Bogdan Nicolae, Anthony Kougkas, and Xian-He Sun. Uncover the overhead and resource usage for handling KV cache overflow in LLM inference. In Supercomputing '24 Poster Proceedings, 2024.
[25] Xiang Fu, Yao Zhang, Haoran Zhu, et al. Lmcache: Generative inference with disk-efficient caching for large language models. arXiv preprint urXiv:2309.00278 2023.
[26] Shou Xu, Siyuan Luo, Jiasi Li, et al. Flashinfer: Efficient token-by-token inference of large language models with cache optimization, 2023.
[27] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: Trading more storage for less computation KVCache-centric architecture for serving LLM chatbot. In 23rd USENIX Conference on File and Storage Technologies (FAST 25). USENIX Association, 2025.
[28] Ruihan Li, Shizhen Liu, Yuwei Yao, Xiang Huang, Wencong Wang, Felix Yu, and Ang Zhao. Cachedattention: Cost-effective multi-turn Ilm serving via hierarchical ky cache, arXiv preprint 1973 2024.
12

---

[29] Weimin Yao, Tao Xie, Fan Wang, and Zili Sun. Warm: write-hotness aware retention manage- ment for flash-based storage systems. 2014 IEEE 30th Symposium on Mass Storage Systems and Technologies (MSST), pages 1-10. IEEE, 2014.
[30] Yu Chen, Jian Tang, and Yang Liu. Efficient graph data loading via gpu direct storage for large-scale gnn training. In Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025. To appear.
[31] DeepSeek AL -iisseck-r--isttill-----e Qwen 32B, 2024. Accessed: 2025-05-15.
https::/huggingffcccc.o// http://nnaacc.oo/eeeeeeee.ee ww/.copseeeeeeaeeeeeeepeeee -ii
DeepSeek-R1-DD
[32] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open Foundation and Fine-Tuned Chat Models, 2023.
[33]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen Technical Report, 2023.
[34] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B, 2023.
[35] NVM Express, Inc. NVM Express Base Specification Revision 2.0. Technical Standard, 2021.
[36] Bin Gao et Cost-efficient large language model serving for multi-turn conversations ith cachedattention. In 2024 USENIX Annual Technical Conference (USENIX ATC 24), 2024.
13

---

A Comparisons with Recent Related Works
A.1 Comparison to InstInfer
While our main paper discusses related work in the context of general KV cache offloading strategies, this section provides a detailed comparison with InstInfer [13), a concurrent work that also leverages NAND Flash to extend the KV cache. While both HiFC and InstInfer share the goal of moving beyond DRAM, they are built on fundamentally different design principles. Instlnfer a hardware- specialized solution that relies Computational Storage Drive (CSD) to perform in-storage attention processing, thereby alleviating PCIe bottlenecks. In contrast, HiFC a general software solution designed for broad adoption, utilizing commodity NVMe SSDs and GPUs within the popular vLLM framework to maximize cost-efficiency and system compatibility. We summarize the key distinctions across three axes: cost, performance characteristies, and system compatibility.
A.1.1 Cost and Accessibility
The primary difference in cost stems from the required hardware. HiFC is designed to work with ubiquitous, off-the-shelf NVMe SSDs, whereas InstInfer necessitates specialized CSDs. As shown in T this leads to significant disparity in initial hardware investment and accessibility, making HiFC more economically viable solution for immediate, large-scale deployment.
Table 7: Cost and hardware comparison between HiFC and InstInfer.
Feature
HiFC
InstInfer
Required Hardware
Commodity NVMe SSD
CSD
Example Device
NVMe Gen4 ITB SSD: $113
SmartSSD 4TB (used): ~S2,526
Implication
Low cost and easy deployment
Higher initial cost and hardware de- pendency
A.1.2 Latency and Throughput Optimization
HiFC and InstInfer employ different techniques to manage I/O and optimize performance (Table B). HiFC focuses on hiding I/O latency through vLLM's pipeline-awai scheduling and maximizes bandwidth using GDS, which creates a direct data path between the GPU and SSD. InstInfer, leveraging the computational capabilities of CSDs, processes attention mechanisms directly on storage device to reduce the volume of data transferred over the PCIe bus. While InstInfer can achieve high throughput by fully exploiting the CSD's internal bandwidth, its performance tightly coupled to this specialized hardware. HiFC's performance scales with SSD bandwidth and GPU DMA engines. A key operational difference is that HiFC swaps KV cache blocks only when HBM capacity is exhausted, whereas in InstInfer's design, the entire KV cache resides the CSD during inference.
common components such as
Table 8: Comparison of performance optimization strategies.
Feature
HiFC
InstInfer
I/O Optimization
GPU Direct Storage (GDS) Pipeline-aware scheduling
In-storage SparF Attention
Flash page-level data layout
KV Cache Reduction
Standard methods (GQA, FP8)
SparF Attention (reduces transfer size)
Performance Gains
Comparable DRAM (~2%% diff.)
Up to 11.1x throughput Flex- Gen (with CSD)
Swapping Behavior
Swaps only when HBM is full
Entire KV cache resides in CSD
14

---

A.1.3 System Compatibility and Generality
HiFC is architected as software-level solution that integrates seamlessly into the widely-used vLLM framework. This approach ensures broad compatibility with existing models, hardware (any GDS-supported NVIDIA GPU and NVMe SSD), and deployment pipelines. InstInfer, being hardware-dependen requires a more specialized stack, including modified version of the Flex Gen framework and specific drivers for CSDs (Table Consequently, HiFC offers more general and readily deployable path to scaling inference services, whereas InstInfer provides powerful but specialized alternative for environments where CSDs available.
Table Comparison system compatibility and ease of deployment. HiFC
Feature
InstInfer
Framework
Built vLLM
Modified FlexGen (CSD-specific)
Hardware Req.
GDS CDS-supported GPU NVMe SSD
CSD + compatible PCIe infrastructure
Deployment
Integrates into existing GPU services
Requires specialized hardware setup
Generality
software solution
Specialized hardware-software co- design
In conclusion, while InstInfer demonstrates excellent performance through hardware acceleration, its adoption is constrained by the cost and availability of specialized CSDs. HiFC, contrast, delivers cost-effective, hardware-agnostic, and easily deployable software solution that makes large-context LLM inference practical for a wide range of existing and future systems.
A.2 Comparison with Other Offloading Systems: AttentionStore and Mooncake
Beyond InstInfer, other recent works such as AttentionStore [36 and Mooncake 27 also address KV cache scaling by offloading external memory. While sharing the high-level objective of efficient KV cache management, their architectural approaches and target use-cases differ significantly from those of HiFC.
AttentionStore proposes a DRAM/SSD caching hierarchy optimized for multi-turn conversational workloads, where DRAM serves as the primary cache minimize latency-sente -ensiivvee SSD access. Mooncake introduces a disaggregated architecture for KV cache sharing across a cluster, primarily targeting GPU efficiency in large-batch offline inference scenarios.
In contrast, HiFC is designed as DRAM--fre online inference solution. It leverages GPU Direct Storage (GDS) to create a direct data path between the GPU and a commodity SSD, a design choice that maximizes I/O bandwidth and fully utilizes the SSD''s capacity. By tightly integrating with vLLM's pipeline scheduler, HiFC effectively hides the I/O latency of swapping, achieving DRAM- compaabaea performance at fraction of the cost. Table 10 summarizes the key architectural differences.
Table 10: Architectural comparison with AttentionStore and Mooncake.
Feature
AttentionStore
Mooncake
HiFC (ours)
Target Workload
Multi-turn conversa-
Long-context, large- batch, offline service
Long-context, large- batch, online service
tion, online service
Memory Hierarchy
GPUS-DRAMMSSDDS]
Multi-node: (GPU-DRAM-SSD)
GPUs-SSDs (DRAM- free)
Latency Handling
Uses DRAM as pri- mary cache to minimize I/O overhead from SSD access.
Employs caching and batching to mitigate re- mote memory access la- tency.
Uses vLLM's scheduler hide SSD I/O latency, achieving DRAM-level performance.
15

---

B Determining Flash Cache Byte Offsets
The effectiveness of the append FC block allocation strategy is empirically evaluated in Section 5.5 which demonstrates improved sequential I/O behavior under this design. To enable such behavior, logical construct for
the FC byte offset should be carefully determined. While FC blocks managing the KV cache, actual storage on SSD requires computing the byte-level offset within the Flash cache file. On the GPU, KV blocks are 4KB-aligned tensors. Aligning these logical blocks with the SSD's physical 4KB LBA (Logical Block Address) format is highly beneficial for I/O performance and device lifespan. Therefore, our allocation strategy (HiFCC is designed enforce this 4KB alignment. To minimize runtime computation overhead, we precompute and store a lookup table of K KV-type-specific offsets for each layer. These offsets are then passed directly to the GDS API for I/O dispatch. The equations below represent how those offsets are obtained.
B.1 Variable Definitions
L: number of attention layers
B: number of FC blocks per KV type per layer
H: number of KV heads
S: block size (tokens)
D: head size
T: data type byte size (e.g.. 2 for FP16)
B.2 Byte Offset Calculation Equations
Single block size (bytes).
block_size_bytes x
Layer offset (bytes). This the starting offset for a given layer layer_offsee = (xxxxxxxx x (2 B x block_size_byiie bytes), where 0 S 1
Intra-layer KV type offset (bytes). This is the relative offset for the Key or Value block region within a layer.
key_type horpe__ffftttt- = k x (B x block_ size bytes), where k = 0 value_type_offfet = (B block_size_bytes) where v
(5)
Total byte offset per layer. This calculates the final absolute byte offset for the start of the Key/Value cache region for specific layer.
KeyOffset()) layer_offset + key_type_offset ValueOffset()) layer_offise + value ttppttttpe__feeeeee_pe
(6)
As a concrete example, we apply the equations above the DeepSeek Qwen-32B model with parameters: L 64, B = 3200, H = 8, S = 128, D 128, and T = 2. Table []] shows the computed actual byte offsets (which correspond to KeyOffset(l) and ValueOffiet(() and their corresponding values in GiB for selected layers.
Table 11: FC byte offsets for selected layers.
Layer
Key Block Offset (bytes)
Value Block Offset (bytes)
Start Offiset (GiB)
0
838860800
0.00
1677721600
2516582400
1.56
2
3355443200
4194304000
3.12
62
(04018739200
104857600000
96.88
63
105696460800
106535321600
98.44
16

---

C Memory Expansion Cost Analysis and Future Projections
To substantiate the paper's cost reduction claims, this section provides detailed memory expansion cost analysis with explicit citations and examines projected market trends to evaluate the long-term economic viability of the HiFC architecture.
C.1 Updated 3-Year Memory Expansion Cost Analysis (H2 2025)
For enhanced clarity and reproducibility, we have re-evaluated all cost parameters using updated market data from the second half of 2025; the original paper used data from HI 2025. The updated 3-year memory expansion cost comparison is presented in Table [2 All cited prices are derived from publicly available sources ensure the reproducibility of the analysis (e.g., Memory,NET, Amazon, and the 2024 US Data Center Energy Usage Report).
Table 12: Updated 3-year memory expansion cost comparison based H2 2025 market data.
Metric
DRAM (DDR4)
TLC SSD (Gen4)
HiFC (Gen4)
Capacity (GiB)
128
1920
1024
Price per GiB ($)
5.06
0.20
0.10
CapEx ($)
647
384
102
Power Draw (W)
8.2
PUE
1.3
1.3
1.3
Energy Cost ($/kWh)
0.1
0.1
0.1
3-year Cost ($)
729
465
120
Previous Comparison Updated Comparison
x4.5
x3.9
1.0
x6.1
x3.9
1.0
The updated analysis reveals that the 3-year memory expansion cost gap has widened, with the DRAM-based solution now being 6. x more expensive than HiFC (previously x. This increase mainly due recent DRAM price hikes, further strengthening the cost advantage of SSD-based HiPC.
C.2 Projected Memory Types and Price Trends (2025-2027)
We further analyzed projected price trends for key components to determine if HiFC's cost benefits would shrink, persist, or grow. Table 13 summarizes these projections.
Table 13: Price trends for key hardware components.
Component
2025 (Current)
2027 (Proj.)
Price Trend
GPU (GB200)
$30K-$35K $1,050-$1.200
$20K-$25K $800-$950
Remains high due to demand
DDR5 DRAM (128 GiB)
Decrease adoption grows
Data Center SSD
$350- $420
$250-$300
Steady decline with oversupply
pSLC SSD (I TB)
$95-$120
$75-$100
Remains cost-effective
High-Perf. SSD
$3,000+
$1,500-$2,000
Decrease as GDS adoption grows
The market analysis indicates that HiFC's cost advantage likely to become even more significant over the coming years. While next-generation GPUs require expensive DDR5 memory, its price is projected to decrease more slowly than that of data center SSDs. HiFC capitalizes on this trend by using cost-effective SSDs for temporary KV cache, avoiding high-cost memory.
The updated market data confirms that HiFC's cost claims remain valid and that its cost benefits are even more pronounced in the current market environment. Future price projections reinforce this conclusion, demonstrating that HiFC provides a practical, economical, and sarward-looking solution for large-scale LLM inference deployments.
17

---

D Total Bytes Written (TBW) Enhancement via pSLC SSD
consumer-grade NVMe Gen4 SSD employs a dynamic pSLC caching mechanism to accelerate write throughput. The total available SLC cache size is approximately 200 GiB. In this experiment, we consider constrained usage pattern where only a fixed 200 GiB logical block range within the dynamic SLC cache is used exclusively for sequential write and read operations. Since the precise SSD lifetime depends on product-specife warranty policies and variations in internal technology. this analysis uses program/erase (P/E) cycles to provide an approximate lifespan estimation.
Official Endurance. According to the datasheet, the SSD-A 1 TB model guarantees 750 TBW (terabytes written) over its lifetime. This figure is based on the assumption of uniform usage across all NAND blocks in TLC mode, with an endurance of approximately 3,000 (P/E) cycles per cell.
SLC Mode Endurance. In pseudo-SLC mode, only I bit per cell is programmed, which significantly improves endurance approximately 30,000 P/E cycles per cell:
ESLC 30,000,
ErLC 3,000.
(
Let Cuse = 200 GiB be the capacity the fixed SLC region used. Then, the theoretical maximum TBW under this scenario is:
TBW = EStC X Cuse 30,000 x 200 GiB = 6,000 TiB.
This calculation assumes an ideal WA factor of 1. As demonstrated in Section 5.5) HiFC's sequential append strategy achieves a WA close to 1.0, so it is omitted from this theoretical calculation.
Comparison to Official TBW. When compared to the vendor-guarann guaranteed endurance:
TBW SLC
6,000
x
(10)
TBW official
750
This indicates that the SSD can withstand 8x more data writes in an SLC-only usage pattern than under the default TLC-mode assumption, assuming minimal cache folding and ideal sequential I/O behavior.
Table 14: Endurance comparison across different NAND usage modes.
Usage Mode
P/E Cycles
TBW (TiB)
TBW Multiplier
Official Spec (TLC, full drive)
3,000
750
1x
Theoretical Max (TLC, full drive)
3,000
3,000
4x
SI.C-only (200 GiB region)
30,000
6,000
8x
These findings confirm that selectively operating within the SLC cache region of the SSD-A signifi- cantly improves write endurance. The result is particularly useful for long-lived AI inference caches and KV swap storage systems with highly localized access patterns.
The SLC-Only endurance mode shown in Table 14 directly incorporated into the HiFC configuration. enabling significantly higher write durability. This configuration is analytically modeled in Eq. (I) of Section and serves the basis for the memory expansion cost estimation results summarized in Table By leveraging the high-cycle SLC region for KV swap traffic, HiFC achieves sustained endurance gains critical for long-lived inference workloads.
D.1 Empirical Endurance Analysis and Lifetime Validation
To assess the long-term viability of HiFC, we conducted an endurance test to predict the operational lifetime of the underlying SSD. The primary metric for SSD longevity is Total Bytes Written (TBW), which specifies the total amount of data that can be written to the drive before it is likely to fail.
D.1.1 Experimental Setup and Lifetime Formula
The endurance test was conducted using the following configuration:
18

---

GPU: NVIDIA A100 (80 GiB)
SSD for HiFC: ITB (pSLC mode, 200 GiB capacity, 6,000 TBW rating)
Dataset: GovReport (avg. 8.7k sequence length)
Output length: 1k tokens
The predicted lifetime in years is calculated based on the drive's rated PSLC TBW and the empirically
measured daily write volume, as shown in Equation
PSL.C TBW (TiB)
Lifetime (years) Total KV Written per Day (TiB) x 365
(11)
D.1.2 Results and Key Findings
The key metrics derived from the endurance test are presented in Table 15
Table 15: SSD endurance test results and lifetime prediction under sustained workload.
Metric
Value
Swap-out count (writes)
28
Avg. KV size (MiB) KV size per test (GiB)
984
26
Total test time (s)
1,097
Tests per day (extrapolated)
78
Total KV written per day (TiB)
1.98
PSLC TBW rating (TiB)
6,000
Predicted Lifetime (years)
8.3
The analysis predicts that under our sustained test conditions, the SSD utilized by HiFC would last for approximately 8.3 years. This operational lifespan significantly exceeds the typical expected service life of GPUs used for LLM inference, which is often estimated to be around 3 years. This result implies that for every three GPU replacement cycles, the SSD would only need to be replaced once. Therefore, HiFC not only provides a cost-effective solution for memory expansion but also demonstrates exceptional long-term endurance, ensuring that the storage component does not become a frequent point of failure or replacement in a production environment.
19

---

E
Experimental Environments
E.1
Performance and KV Cache Swapping Test Environment
Table 16: Hardware and software configuration used in our experiments.
System & Hardware
Server / CPU
Dell PowerEdge R750xa, 2 Intel Xeon Silver 4310
GPU
A100 GiB)
Memory
256 GiB DDR4
OS
Ubuntu 22.04.5 LTS
System Storage Flash cache
Samsung PM893, 7.68 TB SATA eSSD SSD, TB NVMe Gen4 (pSLC cache: 200 GiB)
Software & LLM Model
Baseline
CUDA Toolkit
12.3
PyTorch
2.5.1
GDS Release
1.8.12
L.M Model
DeepSeek-Rt--isiill---eee--
E.2 Diverse Models and Datasets Test Environment
Table 17: Experimental setup for mixed workload validation.
Item
Specification
Models
DeepSeck-Llama-SB. DeepSeck-Qwen- 14B, Mistral-7B
Datasets
Qasper (avg. 3.6k), GovReport (avg. 8.7k),
NarrativeQA (avg. 18.4k)
Block size
64 tokens
Output length
1k
E.3
Validation Environment for WAF
Table 18: Validation environment and test workloads.
Model & HiFC setting
Model
DeepSeek-B Biieeek-----isttii L--Distil--Wwer Qwen-32B 10,000 tokens
Max Model Length
HiFC Space
100.0 GiB
Number of SSDs
Test workload
Input sequence length
5,000 tokens
Ouput sequence length
5,000 tokens
Request
50
20

---

Latency Impact Analysis of HiFC Swapping
critical consideration for swapping- based approaches is the potential introduction of unacceptable 1/O latency, which could be problematic in aatency-eenitive serving systems. This section evaluates HiFC's swapping mechanism under heavy request load quantify its impact on end-to-end throughput and request latency.
As with any swapping mechanism, moving KV cache from GPU HBM another memory tier introduces some overhead. In vLLM, this is handled through two primary strategies: Recompute, which avoids 1/O but incurs additional computation, and Swapping. To maintain low latency, the vLLM scheduler preemptively sacrifices a few requests to free HBM, thereby preventing overall service delays. HiFC adopts this same scheduling logic, allowing hide most of the swap latency through pipeline overlapping.
F1 Experimental Setup
The evaluation conducted using the following configuration:
GPU: NVIDIA A100 (80 GiB)
Model: DeepSeck-R1 -1--Dstill-LLlmaa
HBM KV size: 4 GiB (starvation condition for recompute and swapping)
Dataset: THUDM/LongBennhhggr eearcc//tgvorpootteeeeeee preport_e (avg. 8K sequence length)
SSD for HiFC: 1TB (pSLC 200 GiB)
F22
Evaluation Results
We compared HiFC against GPU-only baseline and a DRAM-baseh swapping system. The GPU- only configuration quickly ran into Out-of-Memory (OOM) emors under the immense request load, highlighting the necessity of memory expansion strategy. The performance comparison detailed in Table 19
Table 19: Performance comparison of memory management strategies under a heavy, latency-sensitive
request load.
GPU only
DRAM-based
HiFC
Metric
Recompute
Swapping
Swapping
HBM for KV cache (GiB)
4
4
KV cache size (GiB)
4 (OOM)
50
200
Output tokens/s
179
172
182
Avg. latency (s/request)
5.5
5.8
5.4
F.3
Observations and Conclusion
The results in Table 19 shows that HiFC provides superior performance, achieving 5% higher throughput (182 vs. 172 tokens/s) and 7% lower average latency (5.4 s vs. 5.8 s) compared to the DRAM baseline. This is possible because HiFC successfully hides swap latency through vLLM's VO-computation overlapping and avoids the DRAM contention bottleneck (a source of tail latency) by using dedicated GDS 1/O path. These findings provide strong evidence that HiFC's design highly effective for latency-sensitive inference, preventing SSDs from becoming a performance bottleneck.
21

---

G Pipeline-Aware Model of Swap Overhead
Pipeline-indueed throughput gap. Under deep pipelining, the realised throughput TPS often We define the per-second throughput deficit
falls short of the ideal value deell RUN TPSsinglo.
ATPS
deee
TPStest
0).
(12)
Let Tiat be the mean per-token pipeline latency (e.g., 45 ms in our measurements). The deficit translates into a stall budget that the pipeline can tolerate without additional loss of throughput:
I gap
ArPs
Tia-
(13)
Flash-swap latency.
Swapping Nwwp victim blocks of Bsize tokens each incurs
Nswap Bgize Stok
swwp
Tix
(14)
BW Flash
where Tix is constant software overhead, the memory footprint per token, and BW the sequential bandwidth of the SSD.
Visibility criterion.
Swap overhead is visible only if exhausts the stall budget: visible
swap
gap
(otherwise hidden).
(15)
Empirical verification. Table 20 summarises representative run with a 32-MiB block size (128 tokens). Although 29-block swap adds 266 ms of Flash latency, the pipeline already exhibits 46 tokens s - deficit, yielding Tgp = 2.07s (Eq. (I3);; inevitably, the swap latency absorbed and no additional performance drop is observed.
Table 20: Pipeline gap-based swap-visibility analysis.
Timestamp
RUN
ATPS
Tep (ms)
Tswap (ms)
Eq. (15)
Visible
22:26:45
0
0
false
No
22:27:21
12
540
false
No
22:27:45
11
44
1 9800
false
No
22:27:51
13
46
2070
266
false
Hidden
This pipeline-award model generalises the sequence criterion and explains why Flash swapping remains performance neetraa under deep batching and long-conted workloads.
22

---

H Analysis of Write Amplification Factor (WAF)
We analyze the Write Amplitication Factor (WAF) of HiFC to evaluate its impact on SSD endurance. WAF is a critical metric for Flash-based storage, defined as the ratio of data physically written to the NAND Flash memory the data written by the host system. lower WAF signifies greater I/O efficiency and contributes to longer device lifespan.
We calculate the empirical WAF using metries from the device's SMART (Self-Monitoring, Analysis, and Reporting Technology) logs, as summarized in Table 21] The host writes correspond to the "Swap out" volume, while the NAND writes are derived from "Data Units Written," which the device reports in 512 kB units.
Table 21: Host write volume actual NAND write volume during the vLLM swapping test.
Source Metric
Value
vLLM (Host Writes)
Block Size
32 MB
Swap Out Count
7,440
Swap Out Size
232.5 GiB
SSD SMART (NAND Writes)
Data Units Written (Endy*
238,896,067
Data Units Written (Start)*
238,408,472
Data Units Written (Diff)
487,595
Total NAND Write Size
232.5 GiB
The WAF is computed by dividing the total bytes written to NAND by the total bytes written from the host, shown in Eq. (16).
WAF NAND Writes (B) 487, 595 x 512 x 1, 1,000
Host Writes (B) 7,440 x 32 10242 & 1.02
(16)
The resulting WAF of 1.02 exceptionally low, indicating that for every 1.02 bytes written to the physical NAND Flash. 1.00 byte was requested by the host. This value is 27% lower than the SSD's own specification of 1.4, which represents typical WAF for random read/write workloads. This demonstrates that the large, sequential write patterns generated by HiFC are highly beneficial for the endurance and longevity of the underlying Flash storage.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the claims made, including the contributions made in the paper and important assumptions and limitations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper discusses several limitations of the proposed HiFC system. In par- ticular, the effectiveness of the Flash cache depends on the SSD's endurance characteristics, including susceptibility to internal operations such as garbage collection and TL-C-t--SSLCC migration. The experiments were conducted under controlled SSD configurations with pSLC mode, and real-world deployments may encounter variance in I/O throughput and durability. Additionally, while the evaluation demonstrates HiFC's effectiveness across several models (including DS- DS-Qwen-32BB DS-Llama-SB,, DS-Qwen- 14B, and Mistral-7B). further validation on more diverse and significantly larger-scale architectures (e.g., 100B+ parameters) would strengthen its generalizability.e and larger model scales would strengthen generalizability.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
24

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: All mathematical formulations and assumptions for performance modeling are clearly stated and derived in the main text.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper provides detailed hardware, software, and benchmarking configura- tions needed to reproduce the core results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
25

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the complete code and benchmark scripts necessary to accurately reproduce our experimental results. The raw experimental data and logs presented in this paper are also included.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://niis.cc)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wttpccccppppppp....e.eeee..o.cpt weeiii/gguies///ee /ooeeeS/emmomdmmiiicaa ssiinni asionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All experiments include full details on block sizes, token lengths, models, and system parameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
26

---

Answer: [NA]
Justification: Statistical significance is not applicable as the study is based on deterministic system performance benchmarks rather than stochastic processes.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2- 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етгог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper specifies the hardware configuration, including GPU/CPU specs
and memory, for all experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 'tnnt make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: This work complies with the NeurIPS Code of Ethics throughout the design, implementation, and evaluation process.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
27

---

10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses both the positive impacts of improving LLM scalability and the implications of hardware-dependen cost trade-offs.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
No high-risk models or datasets are released; safeguards are not applicable.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All reused datasets and frameworks are properly credited with appropriate licensing.
28

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the assets creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new datasets or pretrained models.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No human participants or crowdsoureing were involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
29

---

Justification:
This study does not involve human subjects and does not require IRB approval.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Large Language Models (LLMs) were used during the writing and editing process to refine the presentation technical content. However, they were not used to generate, simulate, analyze core methodological components or experimental results.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy h for what should or should not be described.
30


Reviews:
[{"review_id": "NBxCD5zsVY", "reviewer": "Reviewer_Jq6w", "summary": "This paper introduces a system-level optimization to efficiently store KV cache. The authors propose to bypass DRAM, and do direct GPU–SSD swapping using fast SSD technology and NVIDIA’s GPU-Direct Storage. Specifically, they use pSLC regions of SSDs and seemelessly integrate with vLLM using a block-based memory allocator and swap engine. The authors present comprehensive results to support the efficacy of the proposed method.", "questions": "NA", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n- Motivation is clear. DRAM offloading is expensive and leads to extra latency overhead.\n- Idea is novel. The paper leverages specific regions of the SSD to optimize the cost, ensuring DRAM free SSD swapping.\n- Results show upto 4.5x TCO reduction in 3 years.\n\nWeaknesses\n- It is not clear how the system will scale with multi-GPU system\n- Results reported on only one model. It will be helpful if authors can share results on other LLMs.", "was_revised": true, "final_justification": "I maintain my original score", "rebuttal": "We appreciate the reviewer’s thorough assessment of our work and the constructive suggestions provided. Our detailed responses to the raised points are given below.\n\n**Q1.1.** How will the system scale in a multi-GPU environment?\n\n**A1.1.** We thank the reviewer for raising the important question about HiFC’s scalability in multi-GPU environments.\nTo address this concern, we have extended **HiFC to operate efficiently on multi-GPU and multi-SSD configurations** and conducted additional evaluations to verify its behavior under these setups.\n\nHiFC supports three deployment topologies:\n\n- **1:1 (GPU:SSD)** – baseline configuration commonly used in workstation setups\n- **N:1** – **multiple GPUs sharing a single SSD** to minimize storage cost\n- **N:N** – **multiple GPUs connected to multiple SSDs** (with RAID) to maximize both performance and bandwidth\n\n**Experimental Configuration**\n\n| Item | Specification |\n| --- | --- |\n| **GPU** | A100 (80GiB) × 2 |\n| **HBM KV cache size** | 4 GiB (starvation condition) |\n| **DRAM KV cache size** | 50 GiB |\n| **SSD for HiFC** | 1TB (pSLC 200 GiB) × 2 |\n| **Model** | DeepSeek-R1-Distill-Llama-8B |\n| **Dataset** | THUDM/LongBench/gov_report_e (avg. 8K length) |\n\n**Extended Evaluation Results**\n\n| Setting | DRAM / pSLC Size | TPS (tokens/s) | Requests (avg. 8K) |\n| --- | --- | --- | --- |\n| **GPU:DRAM** | 50 GiB | 172 | 51 |\n| **GPU:SSD = 1:1** | 200 GiB | **182** | 206 |\n| **GPU:SSD = 2:1** | 200 GiB | **367** | 206 (103 + 103) |\n| **GPU:SSD = 2:2** | 400 GiB | **364** | 412 (206 + 206) |\n\n**Observations**\n\n- The **1:1** configuration already provides **5% higher TPS** than DRAM while expanding KV cache capacity.\n- In the **2:1** configuration, **throughput nearly doubles (~2×)** as the number of GPUs increases because swap I/O collisions on the SSD are rare.\n- In the **2:2** configuration, **throughput also achieves a near-2×** increase while additionally supporting a higher number of requests.\n- In multi-GPU configurations, **each GPU is allocated the same pSLC region for KV cache**, following vLLM’s cache management policy.\n- The maximum request capacity scales with the number of SSDs. Choose SSD count based on service scale.\n\n**Conclusion**\n\nThese experiments demonstrate that **HiFC scales robustly in multi-GPU and multi-SSD environments**, achieving near-linear throughput scaling without sacrificing memory capacity.\n\n---\n\n**Q1.2.** Can the authors provide results on other LLMs to demonstrate broader applicability?\n\n**A1.2.** We thank the reviewer for asking about HiFC’s applicability to different LLMs and workloads. HiFC was developed with practical deployment scenarios in mind and has been tested with diverse datasets during development. In the original paper, we intentionally used a fixed model and input/output lengths to provide a fair and controlled comparison against DRAM-based swapping. **To further demonstrate generality, we conducted additional evaluations using several widely adopted open-source models (DeepSeek-Llama-8B, DeepSeek-Qwen-14B, and Mistral-7B**) across datasets with context lengths ranging from a few hundred tokens to over 18K tokens. All experiments were performed under the same GPU/SSD configuration as in the paper.\n\n**Experimental Setup**\n\n| Item | Specification |\n| --- | --- |\n| **GPU** | A100 (80GB) |\n| **HBM KV cache size** | 4 GiB (starvation condition) |\n| **KV swap memory** | DRAM (DDR4) / SSD (GEN4) |\n| **Datasets** | **Qasper** (avg. 3.6K), **GovReport** (avg. 8.7K), **NarrativeQA** (avg. 18.4K) |\n| **Block size** | 64 tokens |\n| **Output length** | 1K |\n\n**Extended Evaluation Results**\n\n| Model | KV Swap | Qasper | GovReport | NarrativeQA | Observation |\n| --- | --- | --- | --- | --- | --- |\n| **DS-Llama-8B** | DRAM | 302.0 | 172.0 | 95.1 |  |\n|  | **HiFC** | **301.3** | **182.3** | **95.8** | Avg. diff. ~2% |\n| **DS-Qwen-14B** | DRAM | 176.4 | 100.9 | 46.4 |  |\n|  | **HiFC** | **175.3** | **103.5** | **46.6** | Avg. diff. ~1% |\n| **Mistral-7B** | DRAM | 281.0 | 163.0 | 416.1* | NarrativeQA limited to <32K |\n|  | **HiFC** | **275.2** | **162.0** | **406.8*** | Avg. diff. ~2% |\n\n**Key Observations**\n\n- Across all tested models and datasets, **the performance gap between DRAM and HiFC remains within ~2%**, confirming negligible overhead.\n- These findings align with the results presented for DS-Qwen-32B in the paper.\n- The 32K context limit of Mistral affected some NarrativeQA requests, but the overall trend remained consistent.\n\n**Conclusion**\n\nThe additional experiments demonstrate that **HiFC consistently delivers DRAM-comparable throughput** across multiple LLM architectures and dynamic context lengths, while achieving a **4.5× reduction in total cost of ownership** by replacing expensive DRAM with a low-cost pSLC SSD. This confirms that HiFC is broadly applicable and effective for long-context LLM inference in realistic deployment scenarios.\n\n---\n\nWe have implemented these features and will release the full code and results on GitHub before the camera-ready version.\n\nWe appreciate the reviewers’ time and consideration in reviewing our rebuttal."}, {"review_id": "jO80yfGeq1", "reviewer": "Reviewer_9KLT", "summary": "This paper proposes HiFC, a novel DRAM-free approach for managing KV cache in large language model inference by directly swapping KV cache blocks between GPU memory and NVMe SSD storage. The main contribution of this paper is achieving comparable throughput to DRAM-based approaches while reducing total cost of ownership (TCO).", "questions": "See above", "limitations": "yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The DRAM-free design is innovative, bypassing the traditional CPU-DRAM data path.\n- A direct GPU-to-SSD data path is a significant architectural advancement.\n- The system leverages GPU Direct Storage, which sees the long-context problem from a system perspective, helping the community see the problem from another perspective.\n- Integration with vLLM makes HiFC practical.\n\nLimitations and Weaknesses:\n- Limited Experimental Scope: Offloaded-based KV cache management is mainly designed to extend the memory capacity. We see no comparison with other offloaded-based KV cache management with a DRAM baseline. Would offloading to SSDs maintain a larger context/more KV-cache than offloading to DRAM? That is an intriguing question this paper didn't answer.\n- The main contribution of this paper is reducing the total cost of ownership (TCO) by 4.5× over three years. We buy into this claim point, and we need to point out HiFC is working on a hybrid system with GPU/DRAM/SSD. And in most cases of cloud renting, the cost of owning a GPU is ten to a hundred times higher than the cost of owning DRAM or SSD. For example, ownership of 3-year DRAM costs \\\\$ 614 but ownership of 1-month A100 would cost over \\$1000. Therefore, saving 4.5× ownership of KV-cache management would be negligible compared with the total cost of ownership of the GPU.\n\nOverall, we think the DRAM-free design is innovative, but also believe in the design principle “lex parsimoniae”. Specifically, the HiFC adds an additional entity without performance gain, and TCO can be covered by the cost of GPU ownership. My biggest concern of the work is this is a practical / system contribution to the community but I don't think the gain / setting is justified. Does this design have impact on other settings or address other practical problems?", "was_revised": true, "final_justification": "I think during the rebuttal the authors couldn't address the concerns I raised limited scope + the cost concern I raised. Therefore, I decided to keep the original rating.", "rebuttal": "We appreciate the reviewer’s thorough assessment of our work and the constructive suggestions provided. Our detailed responses to the raised points are given below.\n\n**Q2.1.** Is the paper missing a fair comparison with other DRAM-based offloading methods (FlexGen, DeepSpeed, etc.)?\n\n**A2.1.** We thank the reviewer for raising the concern about the lack of explicit comparison with other DRAM-based offloading methods. Existing offload-based frameworks, including **FlexGen**[6] and **DeepSpeed**[9], rely on host DRAM to store KV cache, but their behavior differs from that of **vLLM**[5].\n\n**Offloading vs. Swapping: Key Differences**\n\n| Aspect | **Offloading (FlexGen, DeepSpeed)** | **Swapping (vLLM, HiFC)** |\n| --- | --- | --- |\n| **Primary Storage** | Host DRAM  / SSD(always used for KV cache) | GPU HBM (primary) + DRAM / SSD (only when HBM is insufficient) |\n| **When Data Moves** | KV cache always stored in DRAM, fetched on demand | KV cache stored in HBM until space runs out, then swapped out |\n| **Performance Impact** | High I/O overhead due to frequent host DRAM access | Latency mostly hidden by scheduler (decoding continues) |\n| **HiFC Enhancement** | Not applicable | Uses SSD + GDS to replace DRAM, reducing cost and expanding capacity |\n- **FlexGen** and **DeepSpeed** **store KV cache in host DRAM** continuously and **fetch it directly during attention.**\n- **vLLM** keeps KV cache in GPU HBM and **only swaps out to host DRAM** when necessary, hiding most latency through its scheduler.\n\n**We would like to emphasize that a direct, one-to-one comparison between the offloading and swapping techniques is inherently difficult, since they rely on different inference frameworks.** Even when using the same GPU only, the performance of vLLM and DeepSpeed can differ substantially, as each framework employs fundamentally different methods for attention processing and scheduling. **Therefore, we compared the DRAM‐based swapping technique implemented in vLLM with our HiFC method, which is also implemented in vLLM.** By keeping the framework constant, **we isolated the effect of storage type (DRAM vs. SSD), eliminating confounding factors** and clearly demonstrating the benefits of HiFC, which builds on vLLM while replacing DRAM with SSDs to achieve comparable latency at a significantly lower cost.\n\n---\n\n**Q2.2.** Does HiFC enable maintaining larger KV cache and context than DRAM-based offloading?\n\n**A2.2.** We thank the reviewer for this important question. **Yes, HiFC enables maintaining a significantly larger KV cache and supports longer contexts compared to DRAM-based offloading.** HiFC builds on vLLM’s swapping mechanism but replaces host DRAM with SSDs, which greatly expands storage capacity while keeping latency low. Under the **same TCO as DRAM, HiFC can store over 6.2× more KV cache.**\n\n**Memory Scaling for KV cache**\n\nTable below compares request capacity and relative TCO across different memory configurations.\n\n| Item | Dataset Name / Data Size |\n| --- | --- |\n| **Model** | DeepSeek-R1-Distill-Llama-8B |\n| **Datasets** | GovReport (avg. 8.7K) |\n| **Avg. KV cache size**  | 984 MiB |\n- **Requests = Memory Size / avg. KV cache size**\n\n| **Multi setting** | **Memory Size** | **Requests (avg. 8K)** | **Requests Ratio** | **3-year TCO (relative)** |\n| --- | --- | --- | --- | --- |\n| **GPU HBM** | 80 GiB | 82 | 0.6× | – |\n| **Host DRAM** | 128 GiB | 132 | 1 | 4.5× |\n| **GPU - SSD x 1** | 1 TiB (pSLC 200 GiB) | 206 | **1.5×** | 1× |\n| **GPU - SSD x 2** | 2 TiB (pSLC 200 GiB ×2) | 412 | **3.1×** | 2× |\n| **GPU - SSD x 4** | 4 TiB (pSLC 200 GiB ×4) | 824 | **6.2×** | 4× |\n\n---\n\n- For the cost of DRAM, operators can provision **four SSDs**, enabling up to **6.2× more requests compared to DRAM**\n- The pSLC region of a 1 TiB SSD (200 GiB) can be configured as a **800 GiB pSLC** with multiple SSDs in **RAID0**.\n- With vLLM’s flexible allocation, HiFC fully exploits this expanded Flash cache without requiring model modifications.\n\nIn practice, configuring **four SSDs** in HW/SW RAID0 and registering the Flash cache file in HiFC enables **seamless scaling**, allowing service providers to handle **longer contexts** and **more requests** while maintaining **significantly lower costs** than DRAM-based offloading.\n**Conclusion**\n\nThese results demonstrate that **HiFC not only matches DRAM-based approaches in performance but also provides a much larger KV cache capacity at lower cost**, directly addressing the reviewer’s concerns on scalability and impact.\n\n---\n\n**Q2.3.** Is the 4.5× TCO saving from HiFC significant when GPU ownership costs dominate the overall expenses in cloud environments?\n\n**A2.3.** We agree with the reviewer that in cloud environments, **GPU rental costs overshadow DRAM or SSD ownership costs**. Under such conditions, our 4.5× TCO savings for KV cache management may appear less significant compared to the high monthly cost of GPUs.\n\nHowever, HiFC targets a **different deployment model**: **LLM service providers operating on-premise GPU infrastructure**. In these environments, operators must optimize **memory expansion costs** rather than GPU rental fees when scaling to handle larger KV caches and longer contexts.\n\n- **For cloud users**, scaling typically means **renting more GPU instances**, where costs are dictated by the provider’s pricing model.\n- **For on-premise providers**, scaling can be achieved by **adding low-cost SSDs** to existing servers, avoiding expensive GPU upgrades.\n- SSDs are **simple to deploy**, can **reuse existing hardware**, and enable **quick scaling without downtime**.\n\nIn this context, HiFC provides a valuable alternative: rather than purchasing an additional **$14,000 A100 GPU**, an operator can **expand KV cache capacity** with a **$100 SSD**, achieving substantial scalability at minimal cost.\n\n**Cost Comparison (3-Year Ownership vs. Cloud Rental)**\n\n| **Resource** | **Ownership (3-Year)** | **Cloud Rental (1 Month)** | **Remark** |\n| --- | --- | --- | --- |\n| **GPU (A100)** | ≈ **$14,000** | ≈ **$1,000+** | Cloud rental can exceed $12,000/year |\n| **DRAM (128GB ECC DDR4)** | ≈ **$600** | Included in GPU instance | Ownership cost is relatively low |\n| **SSD (1TB pSLC)** | ≈ **$100** | Included in GPU instance | Easy to add/scale in on-premise setups |\n\n**Key Takeaways**\n\n- **For cloud services**, GPU rental dominates total costs, making memory-related savings less impactful.\n- **For infrastructure owners**, **HiFC** provides a **highly cost-efficient scaling method**, replacing costly GPU upgrades with inexpensive SSDs.\n- HiFC allows **seamless KV cache expansion** within vLLM without architectural changes, making it a **practical and economically meaningful solution** beyond the context of cloud rentals.\n\n**Conclusion**\n\nWhile HiFC’s TCO savings may be negligible in cloud rental scenarios, **it becomes highly significant for service providers managing their own GPU infrastructure**. The ability to expand memory using commodity SSDs instead of costly GPUs provides a compelling advantage in real-world deployments.\n\n---\n\n**Q2.4.** Does the DRAM-free HiFC design deliver meaningful benefits and practical impact beyond the added SSD component, given that GPU ownership costs dominate the overall TCO?\n\n**A2.4.** We thank the reviewer for this important question. **Yes, HiFC provides meaningful benefits and practical impact that go far beyond simply adding an SSD, even when GPU ownership costs dominate overall TCO.** HiFC was specifically designed to address the high cost and limited capacity of DRAM in KV cache management, offering a scalable and cost-effective solution without compromising performance.\n\n **Summary of HiFC’s Contributions**\n\n| **Aspect** | **HiFC’s Contribution** |\n| --- | --- |\n| **Cost Efficiency Beyond GPU Ownership** | While GPU rental costs dominate in cloud settings, HiFC targets **on-premise service providers**, delivering **4.5× lower KV cache management cost** than DRAM solutions and avoiding costly GPU upgrades. |\n| **Practical System Benefits** | HiFC enables operators to **expand KV cache capacity using inexpensive SSDs**, effectively removing memory bottlenecks in long-context inference. |\n| **Applicability to Other Settings** | HiFC supports **multi-GPU and multi-SSD configurations** with near-linear throughput scaling. It integrates seamlessly with **vLLM** without requiring model changes, and installation details are provided in ***HiFC_README.pdf*** within the supplementary material. |\n\n **Additional Remarks**\n\nBeyond cost savings, **HiFC introduces a new design principle for KV cache scaling**, utilizing commodity SSDs to overcome memory limits without architectural modifications. It extends the widely adopted **vLLM** framework, making it **immediately usable** by both researchers and practitioners. The full implementation will be released as open source to encourage adoption.\n\nIn real-world deployments, instead of upgrading to an additional $14,000 A100 GPU (80 GiB), **operators can simply add a $100 SSD (1 TiB), expanding KV cache capacity by up to 12.8×** with minimal modifications to their systems.\n\n **Conclusion**\n\nThese results show that HiFC is **not just a DRAM replacement using SSDs**, but a **practical, scalable, and cost-efficient solution** to the KV cache scalability problem. We believe HiFC makes a **meaningful contribution** to enabling **economical, high-performance** long-context LLM inference, directly addressing the reviewer’s concerns regarding its impact and applicability.\n\n---\n\nWe have implemented these features and will release the full code and results on GitHub before the camera-ready version.\n\nWe appreciate the reviewers’ time and consideration in reviewing our rebuttal."}, {"review_id": "pqbIZCTe6Z", "reviewer": "Reviewer_bZj1", "summary": "This paper presents a flash memory swapping scheme which swaps KV cache pages into an SSD drive. It is designed for LLM inference in memory-constrained environments, where the memory required for KV cache activations is too large to fit in GPU and thus the KV cache needs to be swapped / offloaded to a device with higher memory capacity. They present SSD drive swapping as an alternative to standard methods (offloading to CPU DRAM) which can achieve improved TCO. Their approach uses GPU-Direct-Storage (GDS) to transfer data directly from the GPU to the SSD. It attains similar throughput with DRAM swapping strategies with substantial reduction of TCO relative to providing access to large CPU DRAM pools. They also leverage custom allocation strategies to optimize for reuse in order to enhance device durability by promoting sequential memory accesses.", "questions": "- Are the price trends for these technologies (DRAM versus SSD) such that the TCO benefits from this method will shrink, persist, or grow over the coming years?\n- Are there any sustainability concerns from repeatedly burning through flash drives (even if this has lower TCO)?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- They present a DRAM-free strategy with substantial TCO reduction by exploiting flash memory for offloading.\n- GDS eliminates the bottleneck of PCIe transfers when offloading to SSD (and allows high I/O transfers)\n- Their utilization of single-level cell modes is clever for improving write endurance as well as speeding up I/O transfers (as well as their other optimizations in terms of coalescing memory operations)\n- They design a flash-aware allocation strategy to optimize for reuse patterns in order to minimize wear on the SSD devices\n- They provide an end-to-end vLLM integration to demonstrate the real-world feasibility of their method\n- They present a thorough evaluation of their method, including stress testing their system throughput with different input/output configurations and demonstrating the importance of their sequential access patterns.\n\nWeaknesses:\n- This work mainly focuses on throughput-oriented LLM inference. For latency-oriented serving systems, this type of swapping-based method may create unsuitable latency overheads.\n- The TCO analysis is not thoroughly justified - there should be citations provided for the price ranges / other components in Table 1. This is particularly critical since claims around reduced power consumption / TCO are central to the aims of the paper (which aims to provide similar performance for reduced cost).\n- As mentioned in the limitations section, there are potential systems engineering challenges with maintaining SSD-based systems due to increased failure rate.", "was_revised": true, "final_justification": "The author rebuttal has addressed each of the concerns I raised as well as the questions I had. As such, I will stick with my original rating (Accept).", "rebuttal": "We appreciate the reviewer’s thorough assessment of our work and the constructive suggestions provided. Our detailed responses to the raised points are given below.\n\n**Q3.1.** Will the swapping-based approach used in HiFC introduce unacceptable latency overheads in latency-oriented serving systems?\n\n**A3.1.** We truly appreciate the reviewer’s insightful question regarding whether HiFC’s swapping approach could introduce problematic latency in latency-sensitive serving systems. This is an important aspect, and we carefully examined it during our evaluation.\n\nWe assumed GPU VRAM alone couldn’t handle the immense request load.\n\nAs with any swapping mechanism, moving KV cache from GPU HBM to another memory tier introduces some overhead. In **vLLM**, this is handled through two strategies:\n\n- **Swapping**, which adds I/O overhead during KV cache transfer.\n- **Recompute**, which avoids I/O but incurs additional computation.\n\nTo keep latency under control, the **vLLM scheduler** selectively sacrifices a few requests to free up HBM, preventing overall service delays. HiFC adopts the same scheduling logic, and our results show that it maintains **end-to-end throughput** and **request latency** comparable to DRAM-based swapping.\n\n**Experimental Setup**\n\n- GPU: A100 (80GiB)\n- HBM KV cache size: 4 GiB (starvation condition)\n- DRAM KV cache size: 50 GiB\n- SSD for HiFC: 1TB (pSLC 200 GiB)\n- Model: DeepSeek-R1-Distill-Llama-8B\n- Dataset: THUDM/LongBench/gov_report_e (avg. 8K length)\n\n**Evaluation Results**\n\n|  | **GPU only** | **DRAM-based** | **HiFC** |\n| --- | --- | --- | --- |\n| **OOM Prevention** | Recompute | Swapping | Swapping |\n| **HBM for KV cache (GiB)** | 4 | 4 | 4 |\n| **KV cache size (GiB)** | 4 | 50 | 200 |\n| **Memory bandwidth (GB/s)** | 1,935 (spec) | 28 (estimated) | 7 (spec) |\n| **Output tokens/s** | 179 | 172 | 182 |\n| **Avg. latency (s/request)** | 5.5 | 5.8 | 5.4 |\n| **Requests (avg. 8K, 0.97 GiB)** | 4 | 52 | 206 |\n\n**What We Observed**\n\n- When HBM is limited, both swapping and recompute mechanisms are triggered as expected.\n- HiFC successfully **hides SSD swap latency** through pipeline overlap, minimizing the impact on service performance.\n- Interestingly, HiFC delivered **5% higher throughput** than DRAM-based swapping, likely because DRAM contention can increase tail latency in some cases.\n- Overall, HiFC achieved **latency very close to DRAM** while supporting **4× more requests**.\n\n**Conclusion**\n\nOur evaluation suggests that HiFC’s design effectively prevents SSD latency from becoming a bottleneck. By leveraging vLLM’s scheduling and overlapping techniques, HiFC hides most of the swap cost and preserves low latency. These findings give us confidence that HiFC can be applied to **latency-sensitive inference scenarios** without introducing major concerns.\n\n---\n\n**Q3.2.** Are the price ranges and other components used in the TCO analysis, particularly those in Table 1, sufficiently justified with citations given their critical role in supporting the paper’s cost reduction claims?\n\n**A3.2.** We apologize for not including explicit citations for the price ranges and components in our original TCO analysis. As cost and energy claims are central to the paper, we re-evaluated all parameters using updated market data. The original paper used prices from the **first half of 2025**, while the table below reflects data from the **second half of 2025** with added references for clarity.\n\n**Updated 3-years TCO Table**\n\n| **Metric** | **DRAM (DDR4-3200)** | **TLC SSD (GEN4)** | **HiFC (pSLC SSD, GEN4)** |\n| --- | --- | --- | --- |\n| **Capacity (GiB)** | 128 | 1920 | 1024 |\n| **Price_per_GiB ($)** | 5.06 | 0.20 | 0.10 |\n| **CapEx ($)** | 647 | 384 | 102 |\n| **Power Draw (W)** | 24 | 8.2 | 5 |\n| **PUE (ratio)** | 1.3 | 1.3 | 1.3 |\n| **Energy Cost ($/kWh)** | 0.1 | 0.1 | 0.1 |\n| **3-year TCO ($)** | 729 | 465 | 120 |\n| Previous ×4.5 | ×6.1 | ×3.9 | 1.0 |\n\n**Reference**\n\n- Memory, SSD: Memory.NET / Amazon\n- Power cost, PUE: 2024 US Data Center Energy Usage Report\n\n**Key Findings**\n\n- The updated analysis shows that the **3-year TCO gap has widened**: DRAM is now **6.1×** more expensive than HiFC (previously 4.5×).\n- This increase is mainly due to **recent DRAM price hikes**, further strengthening the cost advantage of SSD-based HiFC.\n- All cited prices are derived from publicly available sources, ensuring reproducibility of the analysis.\n\n**Conclusion**\n\nWe regret not including these references in the original submission. The updated data confirms that our TCO claims remain valid and that the cost benefits of HiFC have become even more pronounced in the current market environment. This reinforces the paper’s main argument that HiFC provides **a practical and economically significant solution** for large-scale LLM inference deployments.\n\n---\n\n**Q3.3.** What are the potential systems engineering challenges in maintaining SSD-based systems, particularly regarding increased failure rates?\n\n**A3.3.** We thank the reviewer for raising concerns about maintaining SSD-based systems. SSDs in HiFC are managed like standard GPU server drives with **NVMe logs** for monitoring, and failures have minimal impact as drives are easily replaced. Using **pSLC regions**, HiFC also ensures long SSD lifespan.\n\n**NVMe Error Management and Monitoring Technologies**\n\n- **SMART Log**: Shows health data (temperature, errors, remaining life) for early failure detection.\n- **Error Log**: Records I/O and PCIe errors to identify and troubleshoot faults.\n- **Self-Test Log**: Runs diagnostics to detect hidden issues before they impact performance.\n- **Telemetry Log**: Collects detailed metrics (I/O, wear) for tuning and reliability checks.\n\n**Conclusion**\n\nHiFC adds no extra risk beyond standard SSD-managed GPU servers. Using NVMe monitoring and quick drive replacement, **system stability is maintained**, and SSD-related engineering challenges remain minimal.\n\n---\n\n**Q3.4.** Are the price trends for these technologies (DRAM versus SSD) such that the TCO benefits from this method will shrink, persist, or grow over the coming years?\n\n**A3.4.** We apologize for not including explicit citations for the price ranges and components in our original TCO analysis. We appreciate the reviewer’s observation, as cost and energy claims are central to our work and require clear justification.\n\nFor this rebuttal, we have re-examined **current market prices (H2 2025)** and projected future trends for GPUs, DRAM, and SSDs. The updated values confirm that the **cost gap between DRAM and SSD has widened**, further strengthening HiFC’s economic advantage.\n\n**Projected 3-Year Price and TCO Trends (2025–2027)**\n\n| **Component** | **2025 (Current)** | **2026 (Projection)** | **2027 (Projection)** | **3-Year TCO Trend** |\n| --- | --- | --- | --- | --- |\n| **GPU (GB200)** | $30K – $35K | $25K – $30K | $20K – $25K | Remains high due to demand for cutting-edge AI training |\n| **DDR5 DRAM** (128 GiB) | $1,050 – $1,200  | $900 – $1,000 | $800 – $950 | **Decrease** as DDR5 adoption grows |\n| **Data center SSD** (1.92 TB) | $350 – $420  | $300 – $350 | $250 – $300 | **Steady decline** with NAND oversupply |\n| **pSLC SSD** (1 TB) | $95 – $120  | $85 – $110 | $75 – $100 | Remains cost-effective for HiFC |\n| **High-Perf. SSD** | $3,000+  | $2,000 – $2,500 | $1,500 – $2,000  | **Decrease** as GPU-direct SSD adoption grows |\n\nThanks to the reviewer’s question, we re-examined DRAM/SSD price trends and found **SSDs’ TCO advantage has increased due to rising DRAM costs**. While GPUs like **Blackwell** require expensive **DDR5**, HiFC uses SSDs only for temporary KV cache, avoiding high-cost memory. Future high-performance SSDs will further enhance offloading, and HiFC is already compatible via GDS.\n\n**Conclusion:**\n\nThe updated market data confirms that HiFC’s **TCO benefits are not only valid but are likely to become even more significant** over the coming years, reinforcing its practical value for large-scale LLM deployments.\n\n---\n\n**Q3.5.** Are there any sustainability concerns from repeatedly burning through flash drives (even if this has lower TCO)?\n\n**A3.5.** We appreciate the reviewer’s question on SSD sustainability, as endurance affects both cost and environmental factors. HiFC ensures **long-term reliability** by using only the **pSLC region**, which has over **10×** the endurance of TLC NAND. In our 1 TB SSD, the 200 GiB pSLC region supports **6,000 TB writes**—**8×** the rated 750 TBW. HiFC also uses **sequential I/O** to reduce internal GC operations, further extending lifespan.\n\n**Experimental Setup**\n\n- GPU: A100 (80GB)\n- SSD for HiFC: 1TB (pSLC 200 GiB)\n- Datasets: GovReport (avg. 8.7K)\n- Output length:  1K\n\n**Predicted SSD Lifetime Based on Testing and Lifetime Calculation Formula**\n\n- Total KV written per day (TiB) = KV size per test (GiB) × Tests per day\n- Lifetime (years) = pSLC TBW (TiB) / (Total KV Written per Day (TiB) × 365)\n- **Lifetime** (years) = 6000 / (1.98 × 365) ≈ 8.3\n\n| **Metric** | **HiFC**  |\n| --- | --- |\n| **Swap-out count (writes)** | 28 |\n| **Avg. KV size (MiB)** (8.7K length) | 984 |\n| **KV size per test (GiB)** | 26 |\n| **Total test time (s)** | 1,097 |\n| **Tests per day** | 78 |\n| **Total KV written per day (TiB)** | 1.98 |\n| **pSLC TBW** | 6,000 |\n| **Predicted Lifetime (years)** | **8.3** |\n\n**Key Findings**\n\n- Under our test conditions, HiFC’s SSD would last **up to 8.3 years**, far exceeding typical service lifetimes.\n- For comparison, reports such as **Tom’s Hardware** estimate the expected service life of GPUs used for LLM inference at **around 3 years**.\n- This means the SSD would only need to be replaced **once** while the GPU may undergo **three replacements** in the same period.\n\n---\n\nWe have implemented these features and will release the full code and results on GitHub before the camera-ready version.\n\nWe appreciate the reviewers’ time and consideration in reviewing our rebuttal."}, {"review_id": "8DnMrf9Rxp", "reviewer": "Reviewer_Gyax", "summary": "This paper presents HiFC (High-efficiency Flash Cache), an efficient KV cache management solution for large language model inference. Addressing the high cost of DRAM-based swapping and poor performance of conventional SSD offloading, HiFC enables direct GPU-to-SSD data transfer bypassing host DRAM while leveraging SSD's pseudo-SLC region for optimized I/O performance. Experimental results demonstrate comparable inference throughput (TPS) to DRAM solutions with significantly reduced costs. The method seamlessly integrates with existing vLLM frameworks and proves effective for long-context and high-batch inference scenarios.", "questions": "Please refer to Weaknesses for details. I would increase the score if these points are adequately addressed.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths  \nS1)HiFC provides a practical way to reduce the expensive memory costs of running large language models by effectively using SSDs as extended GPU memory.   \nS2)The system works smoothly with existing popular frameworks like vLLM without requiring changes to model architectures.\n\nWeaknesses  \nW1)The paper focuses more on system building than new AI algorithms, so I believe its practical performance should be the most important criterion for evaluating its contributions.  It only tests with one GPU and one SSD, while real applications often use multiple GPUs and SSDs. We don't know how well HiFC would work in these more common setups.  \nW2)The experiments only use one model (DS-Qwen-32B) and fixed-length inputs (like 5K/10K tokens). In real-world use, LLMs handle mixed workloads with varying context lengths. We don't know if HiFC works as well with different model sizes or dynamic input lengths that change during operation.  \nW3)The paper could be more convincing if it includes a comparison with InstInfer, especially in terms of cost, latency, and compatibility. Such a comparison would provide a more comprehensive evaluation of the strengths and weaknesses of the proposed method relative to existing solutions.", "was_revised": true, "final_justification": "The paper propose an efficient KV cache management solution for large language model inference. Addressing the high cost of DRAM-based swapping and poor performance of conventional SSD offloading. The rebuttal addressed most of my concerns.", "rebuttal": "We appreciate the reviewer’s thorough assessment of our work and the constructive suggestions provided. Our detailed responses to the raised points are given below.\n\n**Q4.1.** How does HiFC perform and scale in realistic multi-GPU and multi-SSD deployments?\n\n**A4.1.** We appreciate the reviewer’s insightful comment regarding the scalability of HiFC. As stated in our paper’s limitations, we have extended HiFC to support **multi-GPU and multi-SSD** configurations and confirmed its scalability with new experiments.\n\nHiFC supports the following topologies:\n\n- **1:1 (GPU:SSD)** – baseline configuration for workstation-level applications\n- **N:1** – multiple GPUs sharing one SSD, minimizing SSD cost\n- **N:N** – multiple GPUs with multiple SSDs (RAID-enabled), maximizing both GPU performance and SSD bandwidth\n\n**Experimental Setup**\n\n| Item | Specification |\n| --- | --- |\n| GPU | A100 (80GB) × 2 |\n| HBM KV cache size | 4 GiB (for HBM starvation condition) |\n| DRAM KV cache size | 50 GiB |\n| SSD for HiFC | pSLC 200 GiB × 2 |\n| Model | DeepSeek-R1-Distill-Llama-8B |\n| Dataset | THUDM/LongBench/gov_report_e (avg. 8K length) |\n\n**Extended Experimental Validation**\n\n| Setting | DRAM / pSLC Size | Requests (avg. 8K) | TPS (tokens/s) |\n| --- | --- | --- | --- |\n| GPU:DRAM | 50 GiB | 51 | 172 |\n| GPU:SSD = 1:1 | 200 GiB | 206 | 182 |\n| GPU:SSD = 2:1 | 200 GiB | 206 (103 + 103) | **367** |\n| GPU:SSD = 2:2 | 400 GiB | 412 (206 + 206) | **364** |\n\n**Key Findings:**\n\n- HiFC with a **1:1** configuration already outperforms DRAM-based swapping by **5% TPS** while providing much larger KV cache capacity.\n- **2:1 scaling** achieves **~2× throughput**, as swap overlaps rarely collide on a single SSD.\n- **2:2 scaling** maintains the **2× throughput increase** while expanding request capacity.\n- For multi-GPU setups, the pSLC space for KV cache is evenly assigned to each GPU, consistent with the cache management strategy used in vLLM.\n- Adding more SSDs increases request capacity. Select SSDs according to LLM size.\n\n**Conclusion:**\n\nThese results confirm that HiFC scales effectively across multi-GPU and multi-SSD setups, preserving both throughput and memory capacity advantages.\n\n---\n\n**Q4.2.** Does HiFC maintain its performance across different model sizes and dynamic, mixed-length workloads?\n\n**A4.2.** We appreciate the reviewer’s question regarding HiFC’s performance under mixed workloads and varying model sizes. HiFC was designed with real-world deployment in mind and has been evaluated on diverse datasets during development. The paper used a fixed model and input/output length to ensure a fair comparison with DRAM-based swapping. For additional verification, we conducted new tests using multiple popular open-source models (DeepSeek-Llama-8B, DeepSeek-Qwen-14B, and Mistral-7B) and datasets with **context lengths ranging from a few hundred to over 18K tokens**. These tests were run with the same GPU/SSD configuration as in the paper.\n\n**Experimental Setup**\n\n| Item | Specification |\n| --- | --- |\n| **GPU** | A100 (80GB) |\n| **HBM KV cache size** | 4 GiB (for HBM starvation condition) |\n| **KV swap memory** | DRAM (DDR4) / SSD (GEN4) |\n| **Datasets** | Qasper (avg. 3.6K), GovReport (avg. 8.7K), NarrativeQA (avg. 18.4K) |\n| **Block size** | 64 tokens |\n| **Output length** | 1K |\n\n**Extended Experimental Validation**\n\n| Model | KV Swap | Qasper | GovReport | NarrativeQA | Observation |\n| --- | --- | --- | --- | --- | --- |\n| **DS-Llama-8B** | DRAM | 302.0 | 172.0 | 95.1 |  |\n|  | HiFC | 301.3 | 182.3 | 95.8 | Avg. diff. ~2% |\n| **DS-Qwen-14B** | DRAM | 176.4 | 100.9 | 46.4 |  |\n|  | HiFC | 175.3 | 103.5 | 46.6 | Avg. diff. ~1% |\n| **Mistral-7B** | DRAM | 281.0 | 163.0 | 416.1* | NarrativeQA only <32K |\n|  | HiFC | 275.2 | 162.0 | 406.8* | Avg. diff. ~2% |\n\n**Key Findings**\n\n- Across all tested models and datasets, the **TPS difference between DRAM and HiFC remained within ~2%**.\n- Results are consistent with those reported for DS-Qwen-32B in the paper.\n- Mistral’s 32K limit affected some NarrativeQA requests, but performance trends remained similar.\n\n**Conclusion:**\n\nThese additional experiments confirm that **HiFC maintains DRAM-comparable throughput** across diverse models and dynamic context lengths. Therefore, HiFC is robust enough to replace DRAM in long-context LLM inference services under realistic mixed workloads.\n\n---\n\n**Q4.3.** How does HiFC compare to InstInfer in terms of cost, latency, and compatibility?\n\n**A4.3.** We thank the reviewer for suggesting a comparison with **InstInfer**, an excellent work that leverages Computational Storage Devices (CSDs) to optimize KV cache management and reduce I/O bottlenecks. Both InstInfer and HiFC aim to extend KV cache to NAND Flash, but they adopt fundamentally different approaches:\n\n- **InstInfer** is a **hardware-specialized solution** using CSDs to process KV cache with in-storage attention, eliminating PCIe bottlenecks.\n- **HiFC** is a **software-general solution** built on the widely used **vLLM** framework, utilizing commodity GPUs and SSDs to achieve cost efficiency and compatibility.\n\nThe differences in **cost**, **latency/throughput**, and **compatibility** are summarized below.\n\n**Cost Comparison**\n\n| Feature | HiFC | InstInfer |\n| --- | --- | --- |\n| **Required hardware** | Commodity NVMe SSD (pSLC) | Computational Storage Device (CSD) |\n| **Example device** | NVMe GEN4 1TB pSLC SSD: **$136** | SmartSSD(CSD) 4TB (reseller): **$2,526** |\n| **Cost ratio** | **≈18× cheaper**  | Higher initial cost due to specialized hardware |\n\nInstInfer requires CSD hardware (e.g., Daisyplus OpenSSD or SmartSSD), which significantly increases the initial cost. In contrast, HiFC uses readily available pSLC SSDs, resulting in a much lower TCO and easier deployment.\n\n**Latency / Throughput Comparison**\n\n| Feature | HiFC | InstInfer |\n| --- | --- | --- |\n| **I/O Optimization** | GPU Direct Storage (GDS), pSLC SSD focus, pipeline-aware scheduling | In-storage SparF Attention, flash page-level data layout |\n| **KV Cache Compression** | GQA and FP8 quantization **reduce KV cache size by up to 4×** | SparF Attention **reduces KV cache transfer size by up to 8×** |\n| **Performance Dependency** | Scales with **SSD bandwidth** and **GPU DMA performance (GDS)** | Depends on **CSD internal flash bandwidth** and **PCIe P2P DMA efficiency** |\n| **Measured Gains** | Comparable to DRAM with <2% latency difference; TPS scales with SSD/GDS | Up to **11.1×** throughput over FlexGen (with CSD) |\n| **Swapping Behavior** | KV cache is swapped only when HBM is insufficient | All KV cache resides in CSD during inference |\n\nHiFC hides swap latency via pipeline scheduling and uses GDS to minimize CPU/DRAM involvement. InstInfer achieves higher gains when fully utilizing CSD’s internal bandwidth but is more hardware-dependent.\n\n**Compatibility**\n\n| Feature | HiFC | InstInfer |\n| --- | --- | --- |\n| **Framework** | Built on **vLLM** (fully compatible with major models and options) | Modified **FlexGen** (CSD-specific) |\n| **Hardware Requirements** | Works with any **NVMe SSD** on NVIDIA GPUs supporting **GDS** | Requires **CSD** and compatible PCIe switches/drivers |\n| **Deployment** | **Easily integrates** into existing GPU inference services without requiring specialized hardware | **Specialized setup** required |\n| **Generality** | **General solution** for LLM inference service scaling | **Specialized solution** limited to CSD-enabled environments |\n\nInstInfer demonstrates strong scalability but relies on specialized hardware and custom software. HiFC, as a software-level solution, is broadly compatible with existing LLM frameworks and hardware, making it easier to deploy at scale.\n\n**Conclusion**\n\nThe additional experiments show that **HiFC achieves throughput on par with DRAM across** diverse LLM architectures and varying context lengths, while reducing the **total cost of ownership by 4.5×** through the use of an inexpensive pSLC SSD instead of costly DRAM. These findings highlight HiFC’s practicality and cost-efficiency, demonstrating its suitability for long-context LLM inference in real-world deployments.\n\n---\n\nWe have implemented these features and will release the full code and results on GitHub before the camera-ready version.\n\nWe appreciate the reviewers’ time and consideration in reviewing our rebuttal."}]

Author rebuttal:

Meta-review: This paper presents a system-level optimization for efficient KV cache storage and received mixed reviews. Three reviewers were very positive, while one expressed concerns about the practical applicability of the approach. The AC acknowledged these limitations but considered the work interesting and worthy of publication at NeurIPS, ultimately recommending acceptance.
Decision: Accept (poster)