Title: MPCache: MPC-Friendly KV Cache Eviction for Efficient Private LLM Inference
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Model

KV Cache Compression

Sparse Attention

Private Inference

Multi-party Computation
Abstract: Private large language model (LLM) inference based on secure multi-party computation (MPC) achieves formal data privacy protection but suffers from significant latency overhead, especially for long input sequences. While key-value (KV) cache eviction and sparse attention algorithms have been proposed for efficient LLM inference in plaintext, they are not designed for MPC and cannot benefit private LLM inference directly. In this paper, we propose an accurate and MPC-friendly KV cache eviction framework, dubbed MPCache, building on the observation that historical tokens in a long sequence may have different effects on the downstream decoding. Hence, MPCache combines a look-once static eviction algorithm to discard unimportant KV cache and a query-aware dynamic selection algorithm to activate only a small subset of KV cache for attention computation. MPCache further incorporates a series of optimizations for efficient dynamic KV cache selection, including MPC-friendly similarity approximation, hierarchical KV cache clustering, and cross-layer index-sharing strategy. Extensive experiments demonstrate that MPCache consistently outperforms prior-art KV cache eviction baselines across different generation tasks and achieves 1.8 ~ 2.01x and 3.39 ~ 8.37x decoding latency and communication reduction on different sequence lengths, respectively.


Peer reviews:
comments: SUMMARY:
The paper proposes MCPCache, a novel KV cache eviction strategy for use in LLM inference when performed via MPC. They thoroughly review and compare the state of the art and justify why their strategy works.

STRENGTHS AND WEAKNESSES:
It's a good paper, with a clear framework for why it is designed the way it is, and an explanation of how speed is improved via the KV eviction strategy.  

Stregths:
- Clear explanation of why the strategy works
- The extensive experiments demonstrate significant improvements over existing methods. MPCACHE achieves a reduction of up to 2.01x latency and 8.37x communication. 

Weaknesses:
- No Explicit Limitations Section: The paper does not have a dedicated "Limitations" section, and the authors state in the checklist that the paper has no limitations. MPC fundamentally has limitations due to communication complexity and isn't widely used in AI deployments for that reason.
-More generally, it's unclear with these speedups if anyone would be willing to accept the complexity of MPC inference.
- Assumption of Architectural Knowledge: The security model assumes that all parties are aware of the model architecture and the number of pruned tokens. This is fine and aligns with past literature, but it forgoes significant privacy.

QUESTIONS:
Please address how the speed tradeoffs here fit into the limitations of the broader approach.

LIMITATIONS:
No, see above.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors mainly address the high latency issue in securely performing KV cache eviction. To reduce the cost of similarity computation, they propose cache clustering, where similarity is calculated at the cluster level and refined progressively in a coarse-to-fine manner. To improve effectiveness, instead of using the mean of each cluster, they measure similarity using the maximum and minimum key values within the cluster. Additionally, they further enhance efficiency by exploiting the similarity of eviction patterns across adjacent layers, allowing similarity computation to be performed only on selected layers. Experiments conducted on various datasets and architectures demonstrate that the proposed method is both more efficient and effective compared to existing methods.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n* The authors propose a novel and efficient method using cache clustering.\n* They conduct extensive experiments to demonstrate the effectiveness of the proposed method.\n\n---\n\n###  Weaknesses\n* The proposed method appears to achieve approximately a 2× speedup in Figure 10. However, according to Figure 3(a), static eviction also shows no performance degradation up to 60%. Therefore, when compared to static eviction methods applied at a 60% rate, the proposed method might offer similar performance but little to no advantage in terms of speed improvement, since static eviction approaches are more efficient than dynamic eviction. Therefore, a comparison with static eviction at a 50–60% ratio in terms of both latency and performance is necessary.\n* Since the method requires additionally storing the minimum and maximum values for each cluster, it is expected to incur higher memory usage. In large models with long sequence lengths, this overhead could become significant. It would be helpful to include a comparison of memory consumption to assess this impact.\n* While the improvement in latency over existing methods is understandable, the reason for the performance improvement is not convincing at a high level. It would be helpful if the authors could provide a high-level analysis explaining why the proposed method outperforms the dynamic KV cache baseline in terms of effectiveness.\n\nQUESTIONS:\n* The reduction in communication cost is significant, but the corresponding latency acceleration appears to be relatively limited. Could the authors provide an analysis explaining this discrepancy?\n* Figures 7 and 8 appear after Figure 9 and are placed far from the text where Figure 7 is first mentioned. It would be better to relocate them closer to their first reference for improved readability.\n\nLIMITATIONS:\nThe paper does not explicitly address its limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an MPC-friendly KV cache eviction framework. MPC is a cryptography scheme that allows multiple parties to work together on the computation with leaking the plaintext. However, some operations are not efficient in MPC in terms of computation latency and communication, such as comparison and non-linear operations. The existing KV cache eviction on plaintext computation is not efficient in MPC. This paper studies the KV cache eviction in the context of MPC. The proposed scheme combines static eviction and dynamic selection. It reduces the expensive max and similarity approximation operations needed, and also has optimizations like hierarchical clustering and cross-layer index-sharing strategy. The evaluation shows the proposed eviction outperforms the baseline KV cache eviction mechanism in both model performance and latency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n+ (Significance and Originality) The first study on KV cache eviction in the MPC framework. This work will make the decoder models more feasible in MPC. I like the proposed schemes that reduce the number of max operations.\n+ (Quality) The proposed optimizations are derived from observations in detailed performance profiling. The paper is well-written and supported by solid experiments.\n\nWeaknesses\n- (Quality) The paper can provide more details on the performance improvement due to each of the optimizations. The current results do not give insights into the impact of each optimization.\n- (Quality) The evaluation metrics and how it is measured are not stated.\n- (Quality) The evaluation does not test under different network setups.\nIf the authors cannot evaluate the framework under different network settings, the computation and communication breakdown (i.e., percentage of time in communication vs. computation, or are they pipelined?) should be provided at least. \n\n- (Clarity) Some figures miss necessary legends. \n-- For example, Figure 1 did not mark what each color means in the breakdown. \n-- In Figure 1(d), the KV cache choice of the eviction baseline [47] can be biased. Table 3 shows that baseline [47] has worse latency performance than baseline InfLLM.\n\n-- It should also be stated clearly that F1 score is used as the model performance metric.\n\nQUESTIONS:\n* Please explain the color coding in Figure 1.\n\n* What is the implementation like in Figure 1(b) and (c)? Is KV cache used? If so, what KV cache eviction algorithm is used?\n\n* Is the latency in Table 3 per inference or for X number of queries? How the numbers are measured are not clearly stated in the paper not the evaluation.\n\n* Can you provide a further breakdown in \"model\" Figure 11? Earlier in the paper, you mentioned that softmax accounts for the majority of the cost. How about after your KV cache optimizations?\n\n* Can you provide insights on what the performance will be like if the networking has longer latency or bandwidth?\n\nLIMITATIONS:\nI agree with the authors that the proposed scheme is general. However, the resulting MPC is still slow for practical use. A discussion on MPC for decoder models can be added.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a framework for properly integrating KV caching into MPC-based LLM inference to reduce inference costs. To do this, they use an MPC-friendly similarity approximation, hierarchical KV cache clustering, and a cross-layer index-sharing strategy to reduce the overhead of dynamic token selection. Their experimental results demonstrate strong performance and efficiency results compared to existing KV cache schemes.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Tackles an important problem of revising the KV cache scheme to be more MPC-friendly\n- The writing style of the paper is great, with clear descriptions about the problem setup, motivational observations, and the proposed method\n- Lots of figures are utilized in the main text that help provide insight into important claims\n- The ideas proposed in the MPC-friendly dynamic KV cache selection section of the paper seem very interesting and innovative\n- Code is open-source and available for reproducibility \n- Experimental evaluations are comprehensive, on both performance and efficiency, with the proposed method comparable or outperforming existing static and dynamic KV cache methods\n\nWeaknesses:\nI have no major complaints about the work. I only have some minor complaints that may need to be addressed:\n- A legend is needed for Figure 1, as there are colors used in some of the graphs, and I do not know what they correspond to. \n- The descriptions for the figures and tables should be more comprehensive and self-contained. For example, for Figure 9 and Table 3, I do not know what models are being evaluated on and what performance metrics are being used. \n- For the tables, it would be visually easier to bold face the numbers that are best amongst the others\n- It would be insightful to include a non-KV cache baseline for both the performance and efficiency baselines\n\nQUESTIONS:\n- Is obersvation 1 a general observation, not specific to the MPC setting?\n- Is this work the first to address optimization in MPC-based LLM inference with KV cache?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

comments: SUMMARY:
The paper presents MPCache, a kKV cache eviction framework optimized for multi-party computation (MPC)-based private LLM inference. It combines static eviction during the prefill stage to prune unimportant tokens. Also, MPCache is incorporated with dynamic, query-aware cluster-based selection to further reduce the KV cache at each decoding step.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The paper introduces a hybrid static + dynamic KV eviction strategy tailored to MPC constraints
2. This is a training-free model that can be directly applied on existing LLMs

Weaknesses:
1. Optimizations are largely heuristic and empirical, lacking formal approximation bounds or convergence analysis.
2. Generally speaking, the experiment setup is not sufficient. For example, efficiency section $5.3$ Fig.10, only MHA models like Llama-2 are used, which gives a limited demonstration of how efficient MPCache is on SOTA models. 
3. In the section $4.4$, the authors missed a citation when discussing index sharing across adjacent layers. This finding has been extensively discussed in the paper: [https://arxiv.org/abs/2410.05076]{https://arxiv.org/abs/2410.05076}.

QUESTIONS:
See weaknesses.

LIMITATIONS:
See weaknesses.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03