Title: AegisGuard: RL-Guided Adapter Tuning for TEE-Based Efficient & Secure On-Device Inference
Venue: NeurIPS 2025
Accepted: yes
Keywords: Trusted Execution Environment

TEE

TEE-Shielded DNN Partition

Model Stealing Attack

Split inference
Abstract: On-device large models (LMs) reduce cloud dependency but expose proprietary model weights to the end-user, making them vulnerable to white-box model stealing (MS) attacks. A common defense is TEE-Shielded DNN Partition (TSDP), which places all trainable LoRA adapters (fine tuned on private data) inside a trusted execution environment (TEE). However, this design suffers from excessive host-to-TEE communication latency. We propose AegisGuard, a fine tuning and deployment framework that selectively shields the MS sensitive adapters while offloading the rest to the GPU, balancing security and efficiency. AegisGuard integrates two key components: i) RL-based Sensitivity Measurement (RSM), which injects Gaussian noise during training and applies a lightweight reinforcement learning to rank adapters based on their impact on model stealing; and (ii) Shielded-Adapter Compression (SAC), which structurally prunes the selected adapters to reduce both parameter size and intermediate feature maps, further lowering TEE computation and data transfer costs. Extensive experiments demonstrate that AegisGuard achieves black-box level MS resilience (surrogate accuracy around 39%, matching fully shielded baselines), while reducing end-to-end inference latency by 2–3× and cutting TEE memory usage by 4× compared to state-of-the-art TSDP methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is to introduce AegisGuard, a fine tuning and deployment framework that selectively shields the MS sensitive adapters while\noffloading the rest to the GPU, balancing security and efficiency. The main contribution includes: 1. the proposed solution reduces the communication workload between TEE and GPU for TSDP solutions through an RL-based sensitivity measurement approach for on-device LM inference. 2. the framework compresses intermediate feature maps to reduce the communication cost after deployment. The authors have provided comprehensive experiments for deployed LMs, demonstrating the inference efficiency, robust security against model stealing attack, and minimal accuracy loss of proposed AegisGuard.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths \n1. This paper presents this fine tuning and deployment framework, which have shown its efficiency and effeteness. In order to reduce the communication cost, the framework adopts a selective shielding strategy, where only the most sensitive adapters are executed inside the TEE, while less sensitive ones are offloaded to the untrusted GPU. Furthermore, structurally prunes are also introduced to further\nlowering TEE computation and data transfer costs\n2. The submission demonstrates a strong technical foundation with a robust experimental design and clear evidence supporting the claims presented in the paper. The experiments include a comprehensive evaluation for the proposed framework.\n3. The paper is well-structured and clearly written, making it accessible to a broader audience.\nWeaknesses\n1. Other than the efficiency of selective shielding strategy, more impact investigate of shielded adapter compression part could help the design more convincing, including the tradeoff between accuracy and latency, how much contribution from this specific part alone, etc.\n\nQUESTIONS:\n1. Is there latency overhead for the sparse metrics operations after structural pruning? What is the tradeoff and benefit from the compression?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes AegisGuard, a fine-tuning framework that balances efficiency and sensitivity. Unlike traditional approaches, which guard LoRA adapters in TEE incurring extra latency, AegisGuard uses RL-based sensitivity measurement to figure out which adapter is more sensitive, and reduce the TEE execution overhead via pruning and compression. Evaluation shows great reduction in latency and memory while maintaining model accuracy as well as model stealing resilience.\n\nSTRENGTHS AND WEAKNESSES:\n\\+ Neat idea and great execution\n\n\\+ TEE execution overhead is a pain point to solve, and the solution is simple yet elegant\n\n\\+ Nice evaluation on inference latency, defense effectiveness, model accuracy, as well as memory results in appendix\n\n\\- From figure 1, transfer latency seems the biggest bottleneck. In the evaluation results, it would’ve been nice to articulate the reduction, breaking down how much is attributed to transfer, how much to TEE execution, etc. The TEE execution reduction should also highlight the effect of pruning. Combining both, a comprehensive ablation study on latency would serve the paper well.\n\n\\- There are many approaches in analyzing layer/adapter sensitivity. It would have been nice to have a discussion of the design philosophy of the chosen approach. It would also be nice to have a comparison, just on this component, with other selection approaches. \n\n\\- Format:  Font in figure 1 is too small. In Figure 2, the flow of the upper half is a bit messy. It would be helpful to rearrange the layout for better readability. Table 1, please add units (i.e. ms vs sec) to avoid confusion\n\nQUESTIONS:\nHow is the impact of pruning on accuracy compensated by the next step of fine-tuning? This part is a bit counterintuitive. If it's because pruning is losing very little, then where is the limit? How much more pruning can be done to actually affect downstream accuracy? This is a complex interplay, with the selection of adapters. Unfortunately, the current evaluation, even appendix, couldn't capture this interplay well.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}

comments: SUMMARY:
This paper proposes AegisGuard, a framework that enhances the security and efficiency of deploying large lora models on user devices by selectively shielding only the most privacy-sensitive LoRA adapters inside TEEs while offloading less sensitive adapters to the GPU. Specifically, it introduces a reinforcement learning-based sensitivity measurement that uses Gaussian noise to rank adapter sensitivity to model stealing attacks and a shielded adapter compression (SAC) module to prune sensitive adapters. It reduces computation and communication costs. Experiments on LLaMA-7B and ViT show that AegisGuard maintains black-box level security, whiling significantly reducing inference latency and lowering TEE memory usage.

STRENGTHS AND WEAKNESSES:
**Strength**

+) The paper is well-written and easy to follow

+) The overall design is technically sound and reasonable to me.

+) It is a good idea to co-design the lora layers and secure inference platform.


**Weakness**

-) Limited scope to adapter-based fine-tuning, specifically LoRA.

-) The sensitivity estimation method may not be robust.

-) Some evaluation results are not clear to me (see questions)

QUESTIONS:
a) My main concern is the robustness of the layer sensitivity estimation, which is the core of the proposed method. The intuition was 'perturbing sensitive parameters leads to a larger loss perturbation', however, it also depends on the chosen input? How can we show the proposed method is robust to the chosen samples?

b) In Table 2, why ViT-Base C100 Shield-LoRA (19.25) was not the best (colored rectangle) but AegisGuard (22.67)? Same questions for LLaMA-7B.

LIMITATIONS:
Limited scope to adapter-based LoRA fine-tuning might be a limitation.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-04