AegisGuard: RL-Guided Adapter Tuning for TEE-Based Efficient & Secure On-Device Inference
Authors: CHE WANG
Ziqi Zhang
Yinggui Wang
Tiantong Wang
Yurong Hao
Jianbo Gao
Tao Wei
YANG CAO
Zhong Chen
Wei Yang Bryan Lim
Venue: NeurIPS 2025 poster ()
Keywords: Trusted Execution Environment
TEE
TEE-Shielded DNN Partition
Model Stealing Attack
Split inference
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We propose a noval fine tuning framework that uses reinforcement learning and adapter compression to selectively shield model layers in TEEs, which achieves strong model protection with 2–3× efficiency gains and no accuracy loss.

Abstract: On-device large models (LMs) reduce cloud dependency but expose proprietary model weights to the end-user, making them vulnerable to white-box model stealing (MS) attacks. A common defense is TEE-Shielded DNN Partition (TSDP), which places all trainable LoRA adapters (fine tuned on private data) inside a trusted execution environment (TEE). However, this design suffers from excessive host-to-TEE communication latency. We propose AegisGuard, a fine tuning and deployment framework that selectively shields the MS sensitive adapters while offloading the rest to the GPU, balancing security and efficiency. AegisGuard integrates two key components: i) RL-based Sensitivity Measurement (RSM), which injects Gaussian noise during training and applies a lightweight reinforcement learning to rank adapters based on their impact on model stealing; and (ii) Shielded-Adapter Compression (SAC), which structurally prunes the selected adapters to reduce both parameter size and intermediate feature maps, further lowering TEE computation and data transfer costs. Extensive experiments demonstrate that AegisGuard achieves black-box level MS resilience (surrogate accuracy around 39%, matching fully shielded baselines), while reducing end-to-end inference latency by 2–3× and cutting TEE memory usage by 4× compared to state-of-the-art TSDP methods.

AegisGuard: RL-Guided Adapter Tuning for TEE-Based Efficient & Secure On-Device Inference
Che Wang Zhang", Yinggui Wang², Tiantong Wang", Yurong Hao?, Jianbo Gao Tao Wei Yang Cao Zhong Chen Wei Yang Bryan Lim?
School of Computer Science, Peking University?.
Nanyang Technological University². University of Illinois Trrbana-Champaie Beijing Jiaotong University Ant Group", Institute of Science Tokyo chewangestu pku. edu.
Abstract
On-device large models (LMs) reduce cloud dependency but expose proprietary model weights to the end-user, making them vulnerable to white-box model steal- ing (MS) attacks. common defense is TEE-Shielded DNN Partition (TSDP), which places all trainable LoRA adapters (fine tuned on private data) inside a trusted execution environment (TEB). However, this design suffers from excessive host-to-TEE communication latency. We propose AegisGuard, a fine tuning and deployment framework that selectively shields the MS sensitive adapters while offloading the rest to the GPU, balancing security and efficiency. AegisGuard in- tegrates two key components: i) RL-based Sensitivity Measurement (RSM), which injects Gaussian noise during training and applies lightweight reinforcement learning to rank adapters based on their impact on model stealing; and (ii) Shielded- Adapter Compression (SAC), which structurally prunes the selected adapters reduce both parameter size and intermediate feature maps, further lowering TEE computation and data transfer costs. Extensive experiments demonstrate that AegisGuard achieves black-box level MS resilience (surrogate accuracy around 39%, matching fully shielded baselines), while reducing end-to-end inference latency by 2-3x and cutting TEE memory usage by 4x compared to state-of-the-art TSDP methods.
Introduction
The rapid development of large models (LMs) 6 10
attracting attention from both academia and industry has enabled efficient on-device LM inference,
28
Particularly with parameter-efficinn fine-tuning (PEFT; e.g., Low-Rank Adaptation [5] LMs can quickly adapt to various downstream tasks. A popular solution to provide LM-based services is deploy proprietary LMs fine-tuned on private data to the users' devices. However, such deployments introduce serious security risks, as attackers with visibility of model parameters can exploit exposed components (white-box accessible to the user) to effectively perform Model Stealing attacks [26), replicating and redistributing proprietary model capabilities by surrogate model.
39
36 to enhance the security of on-device inference. TSDP partitions the model into two parts and deploy them across TEE and untrusted GPU for accelerating the efficiency of trusted inference. The goal of TSDP is to reduce the workload in TEE by outsourcing the computation-intensiv layers untrusted GPUs, premised on model security. Compared to cryptographic solutions like multi-party computation (MPC)
Existing works proposed TEE-Shielded DNN Partition (TSDP) 45
Work done at NTU
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

and homomorphic encryption (5)) TEEs introduce less overhead and thus are more practical in real-world LMs deployment.
Shield-LoRA
Senultivlly Score obtatted RL Rl-eaaed Senativity Meveormment (RS8)
AegisGuard
TEE
LoRA
LoRA
LoRA +-1
Latency Brskidow
T&E
1.00
LoRA.R
Transter:
6.146
Layor
Layer
Leyar
Layor
Layer
Layer
Senaltvity
High
LoRA
LORA
GPU
Shieided Adapter
Compreseion (BAC)
Figure 1: Shield-LoRA (Left), shields all adapters in TEEs. AegisGuard (Right), selectively shields LoRA n, the higher sensitive adapter in TEE to reduce data transfer and computation costs.
straightforward solution that uses TSDP to protect LM to shield LoRA adapters (trained by private data) in TEE and leave the base model layers (publicly available) offloaded on GPU (i.c.. Shield-LoRA), as shown the left part Fig. []] However, this solutions face a new challenge: frequent data transfer between adapters (in TEE) and base models (on GPU). According to preliminary evaluation (16 batch size) on LLaMA-7B inference, the data transfer (14.91s) takes over 60% of the total inference latency (21 21.38s). Existing TSDP work mainly focuses how to select
critical model weights to shield 49 47 or how to obfuscate offloaded weights [45] 34 Thus, how
reduce the communication cost is still an open problem for TSDP in LM scenario.
To address this challenge, we propose AegisGuard, a novel fine-tuning and deployment framework that minimizes communication overhead while maintaining strong model protection against MS attacks. The key insight behind AegisGuard that not all LoRA adapters contribute equally to privacy leakage. The sensitivity of adapter (defined as how much its exposure helps the performance model stealing attack) varies across layers. This motivates a selective shielding strategy, where only the most sensitive adapters executed inside the TEE, while less sensitive ones are offloaded to the untrusted GPU.
One challenge is how to evaluate the adapter sensitivity. Directly evaluating the sensitivity via MS attacks is prohibitively costly. We instead propose lightweight approach, i) Reinforcement Learning-based Sensitivity Measurement (RSM), which estimates each adapter's privacy risk by combining reinforcement learning with Gaussian noise perturbations. More sensitive adapters are then prioritized for fine-tuning and selected for shielding; ii) Shielded Adapter Compression (SAC), which structurally compresses both parameter size and intermediate feature map dimensions of adapters identified by RSM, improving execution efficiency within the TEE.
Extensive experiments demonstrate Aegi GGuard outperforms existing approaches, achieving 2-3x inference latency speedup towards Shield-LoRA, AegisGuard provides an effective defense against MS attack. The attack accuracy is averagely 39. 1,, similar the black box protection (accuracy of 38. 38.9%). AegisGuard introduces negligible accuracy drop by 0. 1.1.%
The contributions of this paper are follows:
We propose to reduce the communication workload between TEE and GPU for TSDP solutions by an RL-based sensitivity measurement approach for on-device LM inference.
We design AegisGuard, fine-tuning framework that measures the adapter's sensitivity and compresses intermediate feature maps to reduce the communication cost after deployment.
We conduct comprehensive experiments for deployed LMs, demonstrating superior inference effi- ciency, robust security against model stealing attack, and minimal accuracy loss of AegisGuard.
2
Related works
2.1
TEE
Trusted Excution Environments (TEEs) is a secure space that isolated from other parts of the host machine [. We follow prior work to regard TEE secure and computation-limin limited area

---

TEEs, such as ARM TrustZone M. Intel SGX D. provide hurdware-enforeed isolated enclaves designed to secure sensitive data and computations against adversarial access. Recent works have explored utilizing TEEs to protect model security on users' devices against possibly malicious device owners 49 of TEE-base defenses is to downgrade the accurate and efficient white-box model stealing attacks to inaccurate and slow black-box attacks.
(without GPU) on a malicious adversary host machine 47 49
23 The goal
2.2 TEE-Shielded DNN Partition (TSDP)
Due to the limited computational capability of TEE, it is challenging to protect the entire model
using TEE 36 47 To address this issue, researchers propose TEB-Shielded DNN Partition (TSDP)
solutions that utilize co-located untrusted GPUs to accelerate the computation phase [49 47 14 34
TSDP solutions partition the model into parts. One is critical and computationally lightweight part, and the other is a computationally intensive part. The TEE shields the first part, and the second part is offloaded GPUs.
Limitation of existing TSDP Although TSDP can mitigate the computation overhead by utilizing GPU accelerators, existing solutions face another bottleneck: the cost of data transfer between TEE and GPU. According to the measurement of the latest TSDP work, the time of data transfer can take much as 35% of the total inference latency [47]. It is because the DNN layers are heterogencously deployed across TEE and GPU, and TSDP solutions have to transfer the internal feature map between TEE and GPU. The core goal of this paper is to reduce the communication workload by minimizing the number of shielded adapters in TEES while not compromising on security.
Gray-box Model Stealing (MS) Attack. The goal of a MS attack is to steal the functionality of victim model by acquiring a surrogate model with a similar performance to the victim model [27]. The attack against the TSDP models is gray-box model stealing because TSDP offloads some computation to the untrusted GPU. The adversary can utilize computation operators (e.g., model weights) to obtain more information than is available through black-box protection (shielding the whole model in TEE). The goal of TSDP is to provide a black-box-leve protection against MS attack in this gray-box setting 34
45
2.3 Large Model Compression
Recent studies have validated the existence of sparse sub-networks in Transformer models, achieving significant compression (e.g., over 50% parameter reduction) with minimal accuracy degradation 20 43 19 [3]. While some compression methods like Singular Value Decomposition (SVD) (37 can effectively reduce model size. they do not compress the intermediate feature dimensions. Because TSDP deploys an LM across TEE and GPU, it needs to transfer intermediate features between TEE and GPU. The feature dimension determines the amount of data communication and affects the efficiency of TSDP. Thus, structural pruning(43) [33] applied during fine-tuning provides more hardware- friendly compression: it directly removes redundant parameters (e.g., attention heads or ffn layers) as long as output dimensions, thereby generating sparse matrices that align with the computation and memory optimizations in TEEs.
3 Design of AegisGuard
3.1 Threats model
The model owner is the defender, who deploys a fine-tuned model user device for on-device inference. The deployed model has two components: i) an open-source pretrained backbone such as LLaMA, and ii) LoRA adapters fine-tuned using private data. The device owner is an honest-but-curious attacker that follows the correct inference protocol, but attempt to perform a MS attack, in which a surrogate model trained to approximate the victim model's 's behavior using only the exposed components. The defender uses a TEE (e.g., Intel SGX) to shield selected LoRA adapters while offloading the rest the model to co-located GPU. We assume the TEE is secure and cannot be compromised. However, the attacker can observe: i) all model weights and parameters that are offloaded to the GPU, and ii) inference outputs from the full model. Our defense objective to minimize the performance of the surrogate model while maintaining low inference latency and accuracy for the legitimate user.
We adopt gray-box security model consistent with prior TSDP literature [4
34 33 45
3

---

Our threat model focuses on protecting the model's proprietary fine-tuning knowledge against parameter-extraction and functional imitation attacks. We do not address input reconstruction or training data inversion attacks based on feature leakage from unshielded components. These are orthogonal threats, and we note that complementary techniques such as differentially private training, input anonymization, or secure enclaved preprocessing may be integrated with AegisGuard to provide stronger end-to-end protection.
3.2 Problem formulation
AegisGuard aims to achieve three-fold goal during the inference phase: security, efficiency, and accuracy. Security means that AegisGuard should defend against gray-box MS attacks [47] and minimize the surrogate model's performance. Efficiency means AegisGuard should reduce the overall inference latency by minimizing the amount of communication and computation the slow TEE. Accuracy means kegisGuard should maintain comparable accuracy with the model trained by standard fine-tuning. To better understand the AegisGuard approach, we define some symbols.
Let M denote the base model, with Dtrain representing training datasets. Similar to existing PEFT work [5], AegisGuard trains set adapters parameterized as AW = BA. We set a decoder layer as the unit. Let S = [81, ....... 8n, denote the sensitivity scores for layers, and A [01, (2,-..; 0t) represent policy actions of RSM, denotes selecting sensitive layers as trainable each step, where t step number. The objective of our approach can be formulated as:
F arg aarrmmn min LFr(MM( AW) + arg min (As:(AW;;)) + arg max JS AW; Devasni A) (1) [AW]
AW
s
The first term LFT minimizes task-specific loss through gradient- fine-tuning. The second term Taxc jointly optimizes higher sensitive adapter sparsity by pruning parameters based on sensitivity score S, achieving dynamic model compression. The third term JRSM maximizes privacy protection via RL. updates by evaluating sensitivity via noise perturbations to adapters and uses policy actions A to sample and train these sensitive adapters more progressively.
3. Score Update
RSM
R AL;
Layer
Layer
Layer
Layer
Layer
2. Layer-Wise Sensitivity Estimation
w/-
Nolsed
LoRA matrix
Goussian Noise
AL
Layer Sensitivity
Task Loss
Score
Layer 2
Layer Sampling
Layer 4
Inputs
Layer
Layer 2
Layer 3
Layer 4
Layer 5
PEFT
SAC
Trainable
Sensitive
Frozen
Insensitive
LoRA matrix
Public weights
Figure 2: Fine-tuning of AegisGuard, integrates two main modules: RL-based Sensitivity Measure- ment (upper) and Shielded Adapter Compression (lower).
3.3 Overview
Fig. illustrates the design of AegisGuard, which comprises two modules: RL-based Sensitiv- ity Measurement (RSM) and Shielded Adapter Compression (SAC). The RSM outputs S, which represents the privacy sensitivity of each layer, while the SAC outputs compressed sensitive adapters. The two modules are integrated to the fine-tuning phase to dynamically estimate layer-wise sensitivity and compress the sensitive layers. Specifically, during fine-tuning phase, sGuard conducts RSM and SAC regular intervals to monitor the change in layer-wise sensitivity and adjust

---

it accordingly. RSM consists of three stages, which follows the RL pipeline: policy action, reward feedback, and state update B0]. The first stage in Fig samples sensitive layers based on updated sensitivity scores. The second stage (2 in Fig 2 independently evaluates the sensitivity of layers through gaussian noise. The third stage (3 in 2 is to compute the rewards for each evaluated
layer, and then update corresponding sensitivity scores. SAC only prunes the adapter weights of active layer (sampled by RSM), with the pruning ratio dynamically adjusted according to its updated sensitivity score. The pseudocode algorithm of RSM and SAC is presented in Appendix. A
3.4 RL-baased Sensitivity Measurement
To better search sensitive layers, we model it as a contextual multi-armed bandit problem which learns to select candidate higher sensitive layers for TEE protection hased on continuously up- dated sensitivity scores S. The measurement consists of three stages: (1) sensitive layer selection (pol- icy actions), (2) layer sensitivity estimation (reward feedback), (3) sensitivity score update (state update). An episode of the measurement is conducted at every from fine-tuning steps (e.g.. 10 steps). We provide convergence analysis of layer selection based fine tuning strategy in Appendix
40
① Sensitive Layer Selection. Given model M with adapters, AegisGuard regards the sensitivity scores as environmental states. The scores are initialized to Sg = [o, ..... Oj. space A constitutes sequence of decisions a, that select subset of sensitive layers in each step for training. We utilize a uniform distribution combined with scores S to sample candidate sensitive layers. The policy action π, is formally defined follows:
The policy action
U (0, sigmoid(s:))
(2)
where U represents a weighted random sampling in the range of [0, ((s)). For sampling. the layer selection probabilities = {pI, Po. Pa are generated from n,. We select N₂ highest scoring layers as trainable for the next fine-tuning steps. After one episode (from steps of fine tuning), AegisGuard injects noise to the weights of the selected N; layers to quantify their sensitivity. However, due to the depth of LMs and the inaccuracy of initial layer selection, a group of adjacent layers may be identified as sensitive. Empirically, consolidating sensitive layers into contiguous sequences tends to cause unstable training. Therefore, we partition the total layers into {g: ) Ni groups and select & layers from each group. The candidate model for next fine tuning step is:
1, i
0 0= Po-
0,
otherwise
where f is the selection strategy that i-th layer I, would be chosen trainable layer if pa larger than threshold Pg- in corresponding group, then evaluate its sensitivity in current episode.
@ Layer Sensitivity Estimation. The layer sensitivity is estimated through noise perturbation for each layer's s adapters. In RSM episode, we inject Gaussian noise e into the low-rank matrix A E gdaa dd of each layer independently. Then, we conduct forward computations on the noised model evaluate sensitivity by measuring the change of output losses. This design is motivated by differential privacy (DP), indicating that sensitive parameters correlate with private data patterns 488 Per- turbing sensitive parameters leads larger loss tur- bulence. Fig. 3 shows the loss change after perturbing different layers on ViT-Base. The y-axis is the layer in- dex and x-axis shows loss change. We can observe that the layer's influence on the model loss varies by a large margin. Aegiscuard aims to utilize the correlation to select the sensitive layers during RSM episode.
Nositvv
L10
Layer Index
L4
-0.1
+0.1
Loss Change (x10)
Figure 3: Loss change when perturbing dif- ferent layers.
Then, the loss changes are used to guide the next action. The goal is to gather as much private information as possible in the current sensitive layers. Formally, the noise perturbation operation for the adapter can be expressed as:
xW + xB A rW + x(A+ ((0,000)0))) B, e~ e~ NO
(4)
5

---

N denotes gaussian noise, with mean of 0 and variance o². The variance is based on the standard deviation the adapters to ensure that the noise only functional for the adapter's weights but not impact the base model's weights, which is demonstrated by the weight distribution of base model and adapters presented in Appendix D.3. The i-th layers' change of loss defined as:
AC 1 B
E
[():)))) ((m;; 4,),
where the batch size of train data Dirain. I' denotes noise perturbation of i-th layer's adapter. The layer-wise reward is computed according to the loss change of injected noise in Eq. aas
N N
R = sgn(ACi)
(6)
R denotes the reward for i-th layer at step l. sgr(-) is the indicator function and sgn( (AC) is the sensitivity signal. Since the loss change would be small, sensitive signals while subtracting mean to stabilize reward across layers. Then, AC; is greater than zero, the layer is considered to be more sensitive to noise. Otherwise, the layer is considered to be less sensitive. The sensitive layers will be rewarded, whereas the less sensitive layers will be penalized.
exponential to amplify highly
Sensitivity Score update. The update of sensitivity score is based on the reward feedback of policy action a, shown in Eq|o, We only update layers that are trainable in current episode and do not update frozen layers. Otherwise, frozen layers may consistently increase or decrease the sensitivity scores without any training. Specifically, the update rule for the sensitivity score is formulated as follows:
8 8, + r/ p sigmoid(ss;) (1- sigmoid(s:)) if
where is the leamnin rate of reinforcement learning, controlling the convergence of sensitivity scores. After identifying sensitive layers, they will be selectively deployed TEEs for model protection.
3.5 Shielded Adapter Compression
Besides the frequent data transfer between TEEs and GPUs, reducing the amount of transmitted data and computation costs of TEES can also mitigate the inference latency. Therefore, we utilize structural pruning to compress shielded adapters' feature dimensions. In the following, we briefly introduce gradient-guide guided weight pruning mechanism and describe how we utilize it to dynamically compress the adapter of sensitive layers combined with RSM module.
LoRA Importance Estimation. According to prior work, the importance of the LoRA matrix element B,,,, can be quantified by measuring the loss impact after setting BA -WW,, 43 For an input x and the ground-truth prediction y, the importance of BA,,, can be given as:
20
li [C(x, y, W + B- A) f(x, v, W B- ABAA, WW,,))
where L denotes the loss of model. We utilize the first-ordes order Taylor expansion to approximate the importance F., in gradient-based manner, in order to minimize the expensive computations of weight matrix. The importance of parameters in LoRA matrices is formulated as:
OL DB,,
OL OC (W,, + (BA) A, + B.: A YB,, DA
Oc
f. = 1( -100,,
(9)
Shielded Adapter pruning. Since only selected layers are learnable in an RSM episode, gradients are unavailable for other frozen adapter parameters, making it challenging to directly apply Eq. in in dynamic model fine-tuning. Furthermore, pruning during the initial training phase may lead to bias and performance degradation, as all layers exhibit relatively low and random sensitivity to private data. Estimating weight importance and performing pruning with limited training data also increase the inaccuracies. Therefore, we propose dynamic pruning strategy that is compatible with our

---

sensitivity measurement module, ensuring both structural pruning compatibility and training stability, Formally, we design a dynamic pruning ratio for adapters in i-th layer as:
Rioal R
(1 (1-a+a. -
ratio; (1) (T- sigmoid(ss)
(10)
where T the total number of fine-tuning steps, è is the current fine tuning step, Reotal is overall prune ratio, and R2 is cumulative prune ratio. is hypper-aarmmetr control the pruning rate. å used to prevent division by zero. In this way; the layer-wise pruning ratio will progressively increase in linear manner. This design aims prune relatively more weights at the later training phase.
In addition, the prune is executed at head-level. The head importance H; is computed by averaging importance of weight elements in this head via moving average H; 3H; + (1 (1 H to mitigate
bias caused by single batch of data. Afterwards, weight mask is computed by ranking the head importance that partial lowest importance head would be set to O:
0. if H3 E rank (H'; (ratio,t)) -Nhead))
mask-
(11)
L otherwise.
where Nhead is the total number of heads. Afterwards, the following forward computation, the masked adapter would be calculated as:
f= xW + xBA @ mask,
(12)
where @ denotes the Hardamard product. The module reduces intermediate feature maps throughput for data transfer and mitigates computation costs by compressing the parameters in TEEs.
Experiments
To demonstrate the effectiveness of AegisGuard, in this section, we study four research questions:
RQI: How is the on-device efficiency of AegisGuard compared with prior work?
RQ2: How the defense effectiveness against model stealing attacks?
RQ3: Does AegisGuard introduce much accuracy degradation?
RQ4: How does the sensitivity vary across different layers?
4.1
Setup
Models and Datasets. We select large models from different domains and sizes, including generative
models (OPT-2,7B [44), LLaMA-7B (35)) and vision transformer (ViT-Base, (i-Laaree-14(2)) We
employ LoRA [5] for pframetee--fficient fine-tuning, For the dataset, we CommonSense [6]
to fine-tune generative models. We evaluate the model performance using six popular question-
answering benchmarks: ARC-Challenge and ARC-Easy (7. HellaSwag [41], OBQA (21). PIQA (5), and WinoGrande [29]. For ViT models, we use six diverse datasets: CIFAR1O, CIFAR 100, [18], UTKFace [46). MNIST 88) GTSRB [32] and SUN397 (8). Following prior works 44 4.
CIFAR and UTKFFac to evaluate the defense effectiveness against MS attack for fair comparison. Due to space constraints, we provide more details (such attack pipeline, evaluation metries, and hyperparameters) in Appendix
Implementation. We implement all code in PyTorch 2.5.1. The fine-tuning is conducted on server with one NVIDIA A6000 GPUs. For inference, we follow existing work 47 to build a prototype framework on PC with an Intel SGX enclave (SDK 2.6. GCC 7.5), and an NVIDIA RTX4090D 24GB GPU to evaluate the on-device LM performance.
Baselines. We select four baselines. Two are fate--ofthhe-rs TSDP solutions, two are bound
baselines. The SOTA TSDP solutions are TEESlice [47 and Phantom 4, which are published at top-tier conferences. The two bound baselines are No-Shield and Shield-LoRA, following [47]. No-Shield deploys the entire model to the untrusted GPU, which has the highest efficiency (all on GPU) but the lowest security (no TEE). Shield-LoRA refers to deploying all adapters in the TEE,
7

---

rather than the GPU. This baseline offers the highest security (all in TEE) but the lowest efficiency (no GPU). Besides, No-Shield and Shield-LoRA also have the highest accuracy because they directly apply LoRA fine-tuning to the base model and do not compress the model weights.
4.2 Inference Efficiency
Table I: Inference overhead real device. We report inference latency of AegisGuard compared with two baselines and Shield-LoRA setting (slowest). indicate lowest inference overhead
Inference Costs (ViT-Base)
Inference (LLaMA-77)
Time Shrelé-LoRA
Ptantom
TDBSScce
AeganGuard
Shield
(s)
Shield-LoRA
TBBSlice
Aegistuard
No Shield
5.73
TEE
12:34
4:25
8:93
4.31
2.54
Tranderr
13665
13667
11:02
Trans
11.96
9.03
End-to-End
32.77 (1.00x)
23.95
25
15 (2.16))
6.45
End-w--End
21.38 (1.00x)
13.43
11.78
6:42 (3.35g
0.96
AegisGuard shows the best efficiency in inference latency. It presents the real-device efficiency of ViT-Base and LLaMA-7B. [ shows the latency of GPU computation, TEE computation, data transfer, and the end-to-end latency. The end-to-end latencies of AegisGuard are 15.14ms 14ms and 6.42s for ViT-Base and L.aMA-7B, respectively. AegisGuard outperforms TEESlice and Phantom by an average of 1.60x. Compared to Shield-LoRA, AegisGuard accelerates the latency by reducing the number of adapters shielded in TEEs for minimizing communication and computations costs. Besides, the advantage of AegisGuard is larger when the model size increases. Compared to Shield-LoRA, AegisGuard accelerates by x.33 on LLaMA-7B, higher than ViT-Base (2.16x). We also measure the memory cost of Aegi eGgir. AegisGuard has average x.00 less parameters (54.96MB for LLaMA 7B, 0.88MB for ViT-Base) in TEEs than baselines. The memory results in Appendix D.2
The improvement of AegisGuard comes from TEE computation and data transfer. All methods have similar computation costs in GPUs, about 6 milliseconds for ViT-Base and 0.2 to 0.3 seconds for LLaMA-7B. For ViT-Base, AegisGuard selectively deploys only four sensitive adapters in TEE, whereas TEESlice places nine adapters. Thus, TEESlice exhibits more computation and data transfer costs. Phantom needs to insert three redundant adapters into models to obfuscate the real execution path. The MUX operation of Phantom is in TEEs choose the correct path in each layer based on the user's key. Thus, Phantom has a larger data transfer costs than AegisGuard.
4.3 Defense Effectiveness
Table 2: Performance of MS attack accuracy. A lower attack accuracy (surrogate model) represents a higher defense effectiveness. indicates the best defensive capability measured by the surrogate model's 's accuracy.
Model Stealing Altack Base)
Stealing Attack (LLaMA
Dataert
No-Shiold
Plantom
TEFSSice
Aegiafuurd
Shickd-LaRA
Tattastt
No-Shield
Phuntom
TFFSlio
AngiaGuard
Shicld-ooR
C10
57.97
36.19
41.89
40.70
42.75
ARC-E
65:00
57.43
39387
40.12
39,00
C100
86.07
50.70
28.18
22.67
10.25
PIDA
77.85
69.37
60.08
60:45
61.31
UTKFaae
75.01
27.57
18.95
19.10
20.80
WinoGrande
68.75
60.48
50.73
$1.48
50.27
Average
86.35
38.15
27.97
27.49
27.59
Avarage
23.21
62.42
50.52
50.68
$0.39
AegisGuard shows the comparable defensive effectiveness against Black-box setting after de- ployment. Table 2 shows the performance of the MS attack on ViT-Base and LLaMA-7B. The accuracy of the stolen surrogate model evaluates the defense performance. A lower accuracy repre- sents better MS attack defense. The last row shows the average accuracy and the value in parentheses is the performance gain over the black-box baseline (lowest attack accuracy). AegisGuard can effec- tively reduce the MS attack accuracy. In all cases, the attack accuracy of Aegi degirGuard are similar to the black-box baseline (1.00x higher than black-box). AegisGuard has a better protection than Phantom (1.38x lower) and similar performance TEESlice, which also provides black-box-leved protection. The comparatively high MS attack accuracy of Phantom is because its adapter parameters are exposed in an untrusted GPU and difficult to find the optimal obfuscation forward path for LMs. Thus, the adversary can recover certain amount of model functionality. TEESlice offers black-box- level protection because shields all weights that are updated by private data in TEE. However, as shown in Table I TEESlice has a (1.5-2.0)x higher latency than AegisGuard because AegisGuard only shields partial highly sensitive layers in TEEs. We also present randomly selecting shielded layers experiments, which requires approximate 2x layers than AegisGuard in Appendix D.4
8

---

4.4
Accuracy
Table 3: Accuracy of AegisCuard on nlp and ev tasks. downstream task accuracy (1.00x) with Shield-LoRA fine-tuning. RL denotes RL-based sensi- tivity measurement module. SAC means Shielded Adapter Compression.
indicates AegisGuard has similar
Model
Variant
CIFAR-10
CIFAR-100
UTKFace
MNIST
GTSRB
Sun397
Avg.
Shield-LoRA/haatoon
97.41 47.47
85.96
75.03
98.90
93.35
53.37
84.01
TEESlice
95.73
82.21
73.87
96.28
90.36
5/.28
82.12
Shield-LoRA-RI
97.35
85,80
74.92
98.72
93.21
53.20
83.87
ViT-Base
Shield- LoRA-SAC(30%)
96.12
84.77
74.61
97.41
92.89
52.48
83.05
AegiaGuard
97.97
86.07
75.01
98.25
93.20
53.76
84.04
Model
Variant
ARC-C
ARC-E
HellaSwag
OBQA
PIQA
WinoGrande
Avg.
Shield-LoRA/Phamnee
41.43
72.39
45.67
77.92
TEESlice
39.96
63.98
71.82
43.18
75.27
66.28
60.08
Shield-LoRA-RL. Shield-LoRA-SAC(ME
41.32
65.18
72.25
45.79
78.05
68.48
61.85
LLaMA-7B
39.55
62.95
69.46
42.56
72,80
65.72
58.81
AegisGuard
41.89
65.40
73.04
45.40
77.85
68.75
62.06
AegisGuard causes negligible accuracy fluctuation in downstream tasks. One concern of AegisGuard is that it may introduce too much accuracy drop than the standard fine-tuning with additional modules, Table shows the accuracy of AegisGuard compared to standard fine-tuning over six nlp and tasks, well ablation studies. For all cases, AegisGuard is comparable to stan- dard fine-tuning that the average accuracy fluctuation of LLaMA-7B and ViT-Base is approximately 0. 1.%.
According to the ablation study, each module in AegisGuard has no obvious negative impact on performance. It is because the RL-based sensitivity measurement and fine-tuning steps are performed iteratively. And this module only designed to affect sampling behavior. The adapter compression has a slight impact performance, when prune 30% weights in Shield-LoRA. But there has less impact on AegisGuard, cause only partial shielded adapters are pruned and its slight negative impact can also be elimiated by next fine tuning epochs. We also conduct experiments in other large models (e.g.. OPT-2.7B, ViT-Large), and the results are provided Appendix D.I
4.5 Sensitivity Visualization of Layers
CIFAR-100 (VIT-Base)
Fig. 4 shows the sensitivity comparison across differ- ent layers of two models: ViT-Base on CIFAR100 (upper figure) and LLaMA-7B on CommenSense (lower figure). ViT-Base shows higher sensitivity in deeper layers and middle layers. For LLaMA-7BB the sensitive layers are distributed across the model from shallow layers to deep layers. For example, layer 2, and 28 have the highest sensitivity score. There- fore, fixed scheme that only shields shallow or deep layers is insufficient for comprehensive model pro- tection. This observation demonstrate the necessary of a dynamic layer selection scheme to identify and protect the sensitive layers.
10
Layer
CommenSense (LLaMA-7B)
Sensitivyy
10
15
20
25
30
Layer idx
Figure 4: Layer-wise Sensitivity Scores
5 Conclusion
In this paper, we propose a noval fine tuning framework, namely AegisGuard. constructs hybrid LoRA-LM architecture for efficient and security on-device LM inference. We introduce RL-based Sensitivity Measurement, to proactively search sensitive layers and fine tune more steps on them. We propose Shielded Adapter Compression that reduce the intermediate feature maps and enhance the efficiency of computation TEEs. Comprehensive experiments across various large models and tasks showcases high-efficiency, high-security protection of on-device LM inference by AegisGuard.

---

Acknowledge.
Ziqi Zhang and Jianbo Gao are corresponding authors. This research is supported by Primary Re- search & Development Plan of Jiangsu Province (BE2023025, BE2023025-5), National Natural Science Foundation of China (62202011, 1, 62172010). This work also supported by Ant Group. The Singapore authors are supported by the NTU startup grant and the RIE2025 Industry Alignment Fund-Industry Collaboration Projects (IAF-ICP) (Award 12301E0026) as well as its Japan-Singapore Joint Call: JST-A 2024 (Project ID: R2416IR 139), administered by S The Japan author is supported by JSPS KAKENHI JP23K24851, JST PRESTO JPMJPR23P5, JST CREST JPMJCR21M2, JST NEXUS JPMJNX25C4. Any opinions, findings, and conclusions recom- mendations expressed in this material are those of the author(s) and do not reflect the views of Sponsors.
10

---

References
[1] Tiago Alves. Trustzone: Integrated hardware and software security. Information Quarterly, 3:18-24, 2004.
[2] Apple. New MacBook ple Intelligence.
Pro
features
M4
family of chips and Ap-
https://www apple. ccwp://gineeeerooo/222//////////
-app--iipppaaa--pee--e-eeee---ee new-macbooks www.ereee----aaaaeeeeeee -f and wwwpdddapp---eeelllleedeeeeeeeee inplle-iitelllgennee/
2024. Accessed: 2025-04-01.
[3] Kai Arulkumaran, Mare Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6):26-38, 2017.
[4] Juyang Bai, Md Hafizul Islam Chowdhuryy, Jingtao Li, Fan Yao, Chaitali Chakrabarti, and Deliang Fan. Phantom: rrivacy-presevving deep neural network model obfuscation in heteroge- neous tee and gpu system. In Proceedings of the USENIX Security Symposium '25. Zenodo, 2025.
[5] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 1432-7439) 2020.
[6] Jiasi Chen and Xukan Ran. Deep learning with edge computing: review. Proceedings of the IEEE, 107(8))665-111) 1674, 2019.
[7] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick. and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05455 2018.
[8] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist handwritten letters. In 2017 international joint conference on neural networks (IJCNN), pages 2921-2926 IEEE, 2017.
[9] Victor Costan and Srinivas Devadas. Intel sgx explained. Cryptology ePrint Archive, 2016.
[10] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
[11] Minghao Guo, Zhao Zhong, Wei Wu, Dahua Lin, and Junjie Yan. Irlas: Inverse reinforcement learning for architecture search. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9021-9029, 2019.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paitern recognition, pages 770-778, 2016.
[13] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun. Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruming and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241):1-124, 2021.
[14] Jiahui Hou, Huiqi Liu, Yunxin Liu, Yu Wang, Peng-Jun Wan, and Xiang- Li. Model protection: Real-time privacy- preeacy- inference service for model privacy at the edge. IEEE Transactions on Dependable and Secure Computing, (9(6))4270-4284 2021.
[15] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li. Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
[16] Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Lee. Llm-adapters: An adapter family for earfmeter-efficient fine-tuning of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5254-5276, 2023.

---

[17] Md Shihabul Islam, Mahmoud Zamani, Chung Hwan Kim, Latifur Khan, and Kevin W Hamlen. Confidential execution of deep learning inference at the untrusted edge with arm trustzone. In Proceedings of the Thirteenth ACM Conference on Data and Application Security and Privacy, pages 153-164, 2023.
[18] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[19] Xiuqing Lv, Peng Zhang, Sunzhu Li, Guobing Gan, and Yueheng Sun. Lightformer: Light- weight transformer using svd-based weight transfer and parameter sharing. In Findings of the Association for Computational Linguistics: ACL 2023, pages 10323-10335, 2023.
[20] Xinyin Ma, Gongfan Fang, and Xinchao Wang. language models. Advances in neural information processing systems, 36:21702-21722 2023.
On the structural pruning of large
[21] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity new dataset for open book question answering. arXiv preprint arXiv:1809.02730 2018.
[22]
MLC LLM. Mlc Ilm: Universal IIm deployment engine with ml compilation. https: //11m mlc. ai/#android, 2024. Accessed: 2025-04-01.
[23] Fan Mo. Ali Shahin Shamsabadi, Kleomenis Katevas, Soteris Demetriou, Ilias Leontiadis, Andrea Cavallaro, and Hamed Haddadi. Darknetz: towards model privacy at the edge using trusted execution environments. In Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services, pages 161-174, 2020.
[24] P Molchanov, S Tyree, T Karras, T Aila, and J Kautz. Pruning convolutional neural networks for resource efficient inference. In 5th International Conference on Learning Representations, ICLR 2017-Conference Track Proceedings, 2019.
[25] Yu Nesterov. Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM Journal on Optimization, 22(2):341-362, 2012.
[26] Daryna Oliynyk, Rudolf Mayer, and Andreas Rauber. I know what you trained last summer: A survey on stealing machine learning models and defences. ACM Computing Surveys, 55(14s):1- 41, 2023.
[27] Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Knockoff nets: Stealing functionality of black-box models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4954 4963, 2019.
[28] Konstantinos Roumeliotis and Nikolaos D Tselikas. Chatgpt and open-ai models: A prelimi- nary review. Future Internet. 19(6):192, 2023.
[29] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communications of the ACM, 64(9):99-106. 2021.
[30] Ashish Kumar Shakya, Gopinatha Pillai, and Sohom Chakrabarty. Reinforcement learning algorithms: A brief survey. Expert Systems with Applications, 231:120495, 2023.
[31] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 10099.1560 1556, 2014.
[32] Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural networks, 32:323-332, 2012.
[33] Yu Sun, Gaojian Xiong, Jianhua Liu, Zheng Liu, and Jian Cui. Tsqp: Safeguarding real-time inference for quantization neural networks on edge devices. In 2025 IEEE Symposium on Security and Privacy (SP), pages 1-1. IEEE Computer Society, 2024.
12

---

[34] Zhichuang Sun, Ruimin Sun, Changming Liu, Amrita Roy Chowdhury, Long Lu, and Somesh Jha. Shadownet: secure and efficient on-device model inference system for convolutional neural networks. In 2023 IEEE Symposium on Security and Privacy (SP), pages 1596-1612.
IEEE, 2023.
[35] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint (t..iv:2302.1377... 2023.
[36] Florian Tramer and Dan Bonch. Slalom: Fast, verifiable and private execution of neural networks in trusted hardware. In International Conference on Learning Representations.
[37] Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-Ilm: Truncation-awart singular value decomposition for large language model compression. arXiv preprint arXiv:2403.07377, 2024.
[38] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3492. IEEE, 2010.
[39] Xiaomi. Mace: Mobile ai compute engine.
https://github.com com http://thhbb.co//iiaaa////meee 2024.
Accessed: 2025-04-01.
[40] Kai Yao, Penglei Gao, Lichun Li, Yuan Zhao, Xiaofeng Wang, Wei Wang, and Jianke Zhu. Layer-wise importance matters: Less memory for better performance in parameter-efficiee fine- tuning of large language models. In Findings of the Association for Computational Linguistics: EMNIP 2024, pages 1977-1992, 2024.
[41] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800, 2019.
[42] Wenxuan Zeng. Meng Li, Wenjie Xiong, Tong Tong, Wen-jie Lu, Jin Tan, Runsheng Wang, and Ru Huang. Mpevit: Searching for accurate and efficient mpe-friendly vision transformer with heterogeneous attention. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 5052-5063, 2023.
[43] Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, and Bohan Zhuang. Loraprune: Structured pruning meets low-rank preprint arXiv:2305. 2023.
fine-tuning arXiv
[44] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint wwXiv:2255000666 2022.
[45] Zheng Zhang, Na Wang, Ziqi Zhang, Yao Zhang, Tianyi Zhang, Jianwei I Liu, and Ye Wu. Group- cover: A secure, efficient and scalable inference framework for on-device model protection based on tees. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
[46] Zhifei Zhang, Yang Song, and Hairong Qi. Age progression/regresion by conditional adver- sarial autoencoder. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5810-5818, 2017.
[47] Ziqi Zhang, Chen Gong, Yifeng Cai, Yuanyuan Yuan, Bingyan Liu, Ding Li, Yao Guo, and Xiangqun Chen. No privacy left outside: On the (in-) security of tee-shielded dnn partition for on-device ml. In 2024 IEEE Symposium on Security and Privacy (SP), pages 3327-3345. IEEE, 2024.
[48] Maxim Zhelnin, Viktor Moskvoretskii, Egor Shvetsov, Egor Venediktov, Mariya Krylova, Aleksandr Zuev, and Evgeny Burnaev. Gift-sw: Gaussian noise injected fine-tuning of salient weights for 1ms. arXiv preprint 2408 2024.
13

---

[49] Tong Zhou, Yukui Luo, Shaolei Ren, and Xiaolin Xu. Nnsplitter: an active defense solution for dnn model via automated weight obfuscation. In International Conference on Machine Learning, pages 42614-42624, PMLR, 2023.
[50] Itamar Zimerman, Moran Baruch, Nir Drucker, Gilad Ezov, Omri Soceanu, and Lior Wolf. Converting transformers to polynomial form for secure inference over homomorphic encryption. In Forty-first International Conference Machine Learning, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: Yes, the main claims made in the abstract and introduction include our primary contributions and the assumptions underlying our scenarios.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes].
Justification: Yes, we discuss the scenarios in which our approach is applicable and the limitations we assumed in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [Yes].
Justification: Yes, we provide the convergence analysis of AegisGuard for its randomized selection fine tuning strategy in Section. with detailed analyses provided in Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes].
Justification: Yes, we describe our method concept in detail, providing overview diagrams pendix
(Figure 2) pseudocode in Algorithm A specific implementations in experiments (Ap-
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
16

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes].
Justification: Yes, provide the implementation details in appendix, and artifacts released code address presented abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
publicc/guudes//odeSattttt httpsi//auaaaeeeeddeeeaaaasaaaaa www.aaammmiioooooooooooooooeeiie
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips /nips.cc/puut
guides/ PwoddSSumiissiooooo.....ooo
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes].
Justification:
Yes, we provide all the training and test details Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes].
Justification: Yes, provide the standard error of mean for LLMs and ViT models. All results are presented in Appendix D.I
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes].
Justification:
Yes, we provide the computer resources Section
We also provide
communication and memory costs in Appendix D.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https:
Answer: [Yes].
Justification: Yes, our research conforms to the NeurlPS Code of Ethics in every respect. We have thoroughly reviewed the guidelines and ensured that our work adheres to the ethical standards set forth.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes].
Justification: Yes, our work secures LMs inference by using TEEs to protect core adapter layers while reducing TEE-GPU communication. This enables privacy-preserving tamper- resistant AI services on-device, benefiting sensitive domains like healthcare and finance.
18

---

However, improved model confidentiality may hinder transparency, auditing, or accountabil- ity, and could be misused conceal harmful models. TEE encapsulation also raises trust concerns in critical applications. Future integration with auditing and attestation mechanisms is important. Overall, our method enhances security and efficiency, we provide the attack results in Appendix D.4
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: The research described in our paper does not involve data or models that have high risk for misuse, and thus does not require specific safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual- use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes].
Justification: Yes, all assets used in our paper are properly credited, and the license and terms of use are explicitly mentioned. Detailed citations, including version numbers and
URLs, are provided in the main content Section 4.1
19

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
20

---

This work does not involve human subjects crowdsourced data collection. It focuses on the design and evaluation of system for secure and efficient LLM inference using Trusted Execution Environments.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]-
Justification: LLMs were only used for writing and language editing purposes, which do not affect the core methodology, scientific rigor, or originality of the research.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM for what should or should be described.
21

---

A Pseudocode algorithm of AegisCuard
Algorithm 1 AegisGuard: RL-based Sensitivity Measurement & Shielded Adapter Compression
1: Initialize W with LoRA adapters AW B A
2: Set S for n layers
3: Set T' as total steps, rrm episode steps, Eprn prune interval steps
4: Initialize т, A [ao]
5: for 1 + T do
6:
Update AB, A, 4 AB, A-11
7: if f. mod f=mm==0 0 then
8:
for i-th layer in all layers do
9:
Noise injection: fi xW; ; xAA +)
10:
Layer Evaluation: AL
// loss change after noise perturbation
Compute Rewards: R
12:
Update Sensitivity Scores: 8 Rg sigmoid(s:) (1 sigmoid(s;))
end for
14:
Sampling a,: Select subset of sensitive layers 4 via
Update: A + at
16:
Partial selected sensitive layers I fine tuning.
end
18:
if t. mod ppm == 0 then
19:
for layer I, do Calculate prune ratio: ratio,(t)
20:
21:
Apply in matrix B, A: mask = 0 <- Rank(H, (I) Nhead)) end for end if
22:
24:
end for
Return W with compressed AW and S(T)
Here we present the pseudocode of AegisGuard, for better understanding of entire framework of Section and Fig.
B Convergence Analysis
In order theoretical prove the effectiveness of AegisGuard (especially the RSM module), provide the convergence analysis of randomized layer gradient descent algorithm, with reference to (25, extending it to setting where at iteration k. the total sampling mass Ci - Σt= Pt may vary, but lower-bounded by C2 2 > Under block-wise Lipschitz smoothness and a fixed step size < max, we prove almost-sure convergence to stationary points with an O(1/k) rate, and linear convergence when the objective function f satisfies the Polyak-tojasiewice (PI.) condition.
Let E R be partitioned into L disjoint blocks (i.e., layers), such that 0 = iteration k, each block i is selected with probability r\ ' 0, such that Σ p.. Cg. Let S denote the random set selected indices. For each E Sk. the update rule is given by:
At
11 o vif(((k))
while the other blocks remain unchanged. Assumption 1: Block-wise smoothness. There exist constants L, > 0 such that
IV.1 V.S(O)II S | for all 0.h.
Assumption 2: Step size. A fixed step size 7 satisfies < max. L.
Assumption 3: Uniform mass bound. There exists constant C' > 0 such that C₁ 2 C' for all k.
22

---

Assumption 4: Sblyak-Lojasiwiice Polyak- Lojasiewwiiiiizzz condition:
condition.
The objective function
satisfies the
2 2
where f is the minimum of f.
Gradient Update and Function Value Bound The update of block i by A ==)).f(00) results in the following bound for the objective function value:
f(e + U0A ( & f(0) ml
2fr & f(0) - "f V.f(0)]
Expected Function Value Change next step is:
Conditioned on 0s, the expectation of the function value the
E(f(0++1) 0k) Ca)S(Ot) + Σ s(0, + U,
S ((0n) " Σ X V (0+))1
C V [(0x)
((w)
Rearranging and taking the full expectation:
nC
E[f(0x+1).
2
Summation Over Iterations
Summing from h: to N gives:
C N
E
V f(O)) f(6)) f(0N) S f(6)) -f"
S0:
E f oc,
which implies that E (0+))] as co. standard supermartingale argument then implies that ((())) - almost surely.
Convergence Rate Define:
NN min uSkeN
From the finite sum above, we have:
2(f(0)) - fr)
N8 <
n/C
k=
which implies that 8N O(LNN)
23

---

Linear Convergence Under the Polyak-Lojasiewiez Condition condition, we have:
Under the Polyak Lojasseewii
B1 [r] (ls(ON) 14- n0 IE11 [[
S (1 (nCC)B[(((R)) f"),
which implies global linear convergence the minimum f"
In conclusion, the randomized layer selection method for gradient descent converges almost surely to stationary points with O(1/k) rate, and linearly under the Polyak-Lojasiewvn condition.
C Experiment Configuration
In this part, we provide the hyperparameter used in our experiments, including Reinforcement Learing in RSM, and fine tuning. We provide the model stealing attack pipeline and setup. All artifacts will be released through public GitHub repository upon paper acceptance, ensuring full transparency and replicability of our research.
C.1 Parameter Efficient Fine Tuning
During adapter fine-tuning for general tasks, primarily apply LoRA adapters the multi-head attention and feedforward layers, specifically to the query, key, value, and dense components. The exact placement of LoRA modules depends on the model's performance on task-specific datasets. Our goal is to minimize the number of trainable parameters while maintaining competitive accuracy. For instance, in relatively simple tasks such as CIFAR-I0. LoRA is only applied the query and value projections. In contrast, for more challenging NLP tasks, we insert LoRA adapters into the query, key, value, and dense components of each layer. The rank hyperparameter of LoRA chosen from 16, 32, 64, depending on the size of the model and the dataset.
For dynamic pruning, set the total pruning ratio with 20% and 50% to balance performance and efficiency. The pruning frequency is set to 20 steps, and the warm-up ratio is set to 0.1.
We use a micro-batch size is 8.32 and respectively. The training epochs of nlp tasks are setting in range of [2,5]. For vision tasks, the epoch is setting from 6 to 20 depends on the task complexity. We employ the AdamW optimizer, experimenting with a range of learning rates: [2e-5,4e-5,24 4,,44,22-4,,--44
as 8,1 for LLaMA and ViT,
C.2 Reinforcement Learning in RSM
Several hyperparameters are involved in this phase. The first is gaussian noise, which is set within the range of 0.01 to 0.1 for vision models. For LLaMA and OPT models, however, smaller noise value is required, as a larger gaussian noise may lead to loss vanishing or explosion due to the depth of layers. The sampling group size is fixed at 3. The reinforcement learning (RL) steps are scheduled every 20 fine-tuning steps, meaning that the RL procedure is executed once after every 20 steps of standard fine-tuning. The parameter N, is set to 40% of the total number of layers. The learning rate for the reinforcement learning process is selected in the range of to 8, depending on the convergence behavior of the Sensitivity score distributions.
C.3 Model Stealing Attack
Model Stealing Attack aims create "knockoff" or surrogate model that replicates the functionality of a victim model by adversaries, using only black-box access to query the victim model. The attack process typically involves initializing a surrogate model using publicly available mreitrained model, sampling images/text from an predictions, and then retraining the surrogate model on these collected prediction pairs to mimic the victim models' behavior. The effectiveness of model stealing is evaluated by measuring the surrogate models' accuracy on the victim model's private test dataset, indicating how well the surrogate model performs after being trained on the adversary's constructed dataset.
dataset to query the victim model and collect
24

---

To perform fair comparison, we employ standard query- beery-based stealing techniques where the attacker trains model from of collected data labeled by the partially -siielddd victim model. Query- based MS attack has been widely adopted in literature 47 4 27 For surrogate model initialization, we assume that adversaries have knowledge of the LM architectures, since many public I.Ms are open- sourced. Therefore, adversaries can compare differences between their local models and publicly available models to identify the trained parameters.
Model Stealing Attack Setup. Model stealing attacks aim to extract surrogate model that replicates the functionality of the victim model. To simulate this, we adopt the same architecture the victim model and fine-tune it using limited query-based training dataset, which consists of approximately 1% of the full dataset. This setting has been shown to be realistic and practical in real-world scenarios [47]. Moreover, AegisGuard follow the same fine-tuning configurations as baselines for fair comparison. During this process, we shield approximately 40% the total adapters based sensitivity scores, and reinitialize these adapters with random weights before retraining, The learning rate and number of training epochs are set to be consistent with those used in the initial fine-tuning phase.
D Supplementary Experimental Results
D.1 Ablation Study
Table 4: NLP Model Performance Across Tasks (Accuracy %)
Model
Variant
ARC-C
ARC-E
HellaSwag
OBQA
PIQA
WinoGGrande
Average
Shield-LoRAPhaniin
37.99
51.94
51.95
32.80
72.03
61.16
51.65
TEESlice
36.12
49.37
50.31
30.85
71.49
60.06
49.90
OPT-2.7B
Shicld-LoRA+RLeen.
37.95
52.41
51.75
33.92
71.65
59.70
51.56
Shicl+-LoRA+SAC(3555)
34.15
48.19
48.50
31.72
67.90
57.95
48.07
AegisGuard
38.07
52:27
51.62
33.80
71.82
59.82
52.40
Shield-LoRA
41.43
65.06
72.39
45.67
77.92
68.61
61.85
TEESlice
39.96
63.98
71.82
43.18
75.27
66.28
60.08
41.32
65.18
72.25
45.79
78.05
68.48
61.85
IJama-BB
Shics--oRA+SAAC(OSS)
39.55
62.95
69.46
42.56
72.80
65.72
58.84
AegisGuard
41.89
65.40
73.04
45.40
77.85
68.75
62.06
Table 5: Vision Model Performance Across Tasks (Accuracy %
Model
Variant
CIFAR-10
CIFAR-100
UTKFace
MNIST
GTSRB
Sun397
Average
LoA//Phatom
97.47
85.96
75.03
98.90
93.35
53.37
84.01
TEESlice
95.73
82,21
73.87
96.28
90.36
54.28
82.12
Shield-Loke LoRARRI -I.I.... wonn
97.35
85.80
74.92
98.72
93.21
53.20
83.87
ViT-Base
96.12
84.77
74.61
97.41
92.89
52.48
83.05
AegisGuard
97.97
86.07
75.01
98.25
93.20
53.76
84.04
Shicld-LoRW/Phantee
98.58
91.97
88.38
98.46
94.79
63.81
89.33
TEBSlice
95.76
89.37
85.46
95.58
91.84
62.05
86.68
Mield-LoRA+RLec. Shicld-LORASSAC(BRSS
98.41
91.78
88.25
98.19
94.62
63.55
89.13
Vi--Large .arge
97.11
90.62
97.12
92.95
62.54
87.81
AegisGuard
99.01
91.05
88.25
98.78
95.35
63.97
89.40
In this section, we compare the performance of AegisGuard, Phantom, TEESlice, and Shield-LoRA methods. The results are presented in Tanle[aand Table Phantom employs a pest-selection strategy; thus, its model performance are identical to the Shield-LoRA method, which uses standard fine- tuned LoRA-LMS without additional modifications. TEESlice exhibits slightly inferior performance compared to other methods due to the removal of several adapters aimed at minimizing the number of adapters shielded within TEEs. Although TEESlice can be theoretically applied LM scenarios, its practicality is limited, the adapter removal is passive and only small number of layers can
25

---

be removed while satisfying accuracy loss constraints. Ablation studies have demonstrate that each module has no obvious negative impact on overall model performance, except for the pruning ratio. Besides, due to the size of Table, we shows the standard error of NLP models in Table 6
Table 6: Model Performance Across Tasks (Accuracy %) with Standard Error of the Mean
Model
ARC-C
ARC-E
HellaSwag
OBQA
PIQA
WinoGrande
Average
OPT-2.7B
Aegiscuard
38.07 + 1.35
52.27 1.02
51.62 + 0.48
33.80 2.13
71.82 + 1.01
59.82 1.37
52.40 + 1.22
AngisGuard
41.89 +
1.01
73.04
2.00
77.85 +
68.75 1.29
62.07
Model
CIFAR-10
CIFAR-100
UTKFace
MNIST
GTSRB
Sun397
Average
ViT-Base
Aegistuard
97.97 + 0.20
86.07 0.55
75.01 + 1.03
98.25 0.52
93.20 + 0.45
53.76 1.21
84.04 1.11
ViT-Large
AegisGuard
19.01 +: 0.20
91.05 0.56
88.25 =: 1.02
98.78 0.41
95.35 = 0.31
63,97 + 1.36
89.40 + 1.02
D.2
Utility Analysis
Table 7: Memory Consumption Comparison Across Models and Methods, Bold denotes the minimal
memory consumption
Models
Methods
#Public Params
#Trinable Params
#Params in TEE
TEE Memory
Shield-LoRA
6.79B
56.10M
16.
224.4 MB
LLaMA 7B
Phantom
6.79B
72.93M
16.83M
67.32 MB
(r=32)
TEESlice
6.79B
44.88M
44.88M
179.52 MB
(ratio=30%)
AegisGuard
6.79B
56. 36M
13.74M
54.96 MB (4.08
Loi
2.708
47.19M
47.19M
188.76 MB
OPT 2.78 (r=32)
Phantom
2.70B
58.99M
11.80M
47.20 MB
TEESlice
2.70B
37.75M
37.75M
151.00 MB
(ratio=30%)
AegisGuard
2.70B
47.19M
11.56M
46.24 MB (4.08 x)
Shield-LoRA
86.71M
0.88M
0.88M
3.52 MB
ViT-Base
Phantom
86.71M
1.32M
0.44M
1.76 MB
(r=16)
TEESlice
86.71M
0.62M
0.62M
2.48 MB
(ratio=30%)
AegisGuard
86.71M
0.88M
0.22M
0.88 (4.00
Shield-LoRA
305.74M
2.36M
2.36M
9.44 MB
ViT-Large
Phantom
305.74M
3.54M
1.18M
4.72 MB
(r=16)
TEESlice
305.74M
1.65M
1.65M
6.60 MB
(ratio=30%)
AegisGuard
305.74M
2.36M
0.58M
2.32 MB (4.06
In this section, we present the memory consumption of Aegi suurr compared with baseline methods. Obviously, AegisGuard achieves minimal memory occupation in TEEs due to adapter compression and only partial layers shielded in TEEs. AegisGuard achieves averagely 4.00x less memory consumption than baselines.
Table 8: Inference and Computation Overhead Evaluation of ViT-Large and OPT-2.7B. lowest inference overhead
indicate
Inference Costs (VIT-Largs)
Infoence Costs (OPT:2.7B)
Time (ms)
Shiilld LoRA
Phantoes
TDESiiee
AogteDuard
No Shield
Time
Shield LoRA
Phantom
TECSicee
Aegiebuard
No Shield
GPU
23.56
0.36
$37.40
4.03
359
Total
35 (1. (1.00x)
678.47
61685
377.11 (2:111122000 (2.04x)
32.21
Total
6.0 (1.00x)
4.22
4.88
(.49
0.17
We also evaluate the inference and computation overhead for ViT-Large and OPT-2.7B, in order to demonstrate AegisGuard can be applied in different type of models. As shown in Table 8 AegisGuard achieves the best efficiency both in ViT-Large and OPT-2.7B, accelerating 2.04x and 4.05 respectively. Additionally, we provide the transfer intermediate features maps of NL. models (OPT-2.7B, L.LMM-7B), their feature dimension are 4096. The results are presented in T
26

---

Table 9: Comparison of transfer flops/size between original and compressed hidden dimensions (30%)
across different batch sizes. The original hidden size for LLMs are all 4096.
Batch Size
Original Size (MB)
Compressed Size (MB)
Original Flops
Compressed Flops
16
0.25
0.17
65,536
45,875
32
0.5
0.35
131,072
91,750
64
0.7
262,144
183,500
128
2
1,4
524,288
367,001
Base Vasel Weint tton
ASsuter WHidd Devinstion
390
190CE
3250
10K0
-235
-000
0.00
<is
536
<00
-00
0.11
035
Figure 5: Weight Value Distribution Comparison between Base Model and LoRA Adapter
D.3 Gaussian Noise Analysis
Fig. shows the weight magnitude distribution of base model ViT-Base and LoRA Adapter, which is W and B A. Obviously, the weight of W has larger value than fine-tuned adapters. Therefore, the noise setting based on adapters' weights has a negligible impact on base model weights.
In Fig 6 we present all sensitivity scores of ViT-Base for all layers under various vision datasets. sensitivity scores, which demonstrates
The results present that different layer has domain- shielding fixed number of layers are impractical for real on device LM protection.
D.4 Evaluation of Surrogate Model
As shown in Table 10 we present the MS attack performance of ViT-Large and OPT-2.7B, order to demonstrate AegisGuard can also apply to different LM architectures. The results demonstrate AegisGuard has comparable defensive effectiveness against black box level protections. But for ViT- Large, the surrogate model has better attack performance than ViT-Base, cause the base model (open- sourced weights) already has good generalization ability in classify CIFAR and UTKFace datasets.
Table 10: Performance of MS attack accuracy. lower attack accuracy (surrogate model) represents a higher defense effectiveness. indicate the best defensive capability.
Stealing (Wit-Large
Model Stealing Altick (CPFT:2.78)
Datavet
No-Shield
Phantom
TEESiice
Angletuuard
Shield-LoRA
Dutaset
No- Shield
Phontom
TEESioe
AngleGuard
Stield-LoRA
C10
99.01
96. 4%
94.12
93.58
93.47
ARC-E
$2.27
45.38
38.01
37.12
36,39
C100
91
48.63
27.25
26.85
27.01
PIQA
71,83
61.83
50.27
$0.30
$1.47
UTKFaaee
$8.25
67.37
$4.79
$5.38
49.67
WinoCrrande
39.82
50.34
45.23
45.74
45.81
Average
92.77
70.83
58.72
58.67
56.72
Average
61.30
$2.53
44.50
44.42
44.56
Additionally, we conduct experiments of selecting random number of layers on standard fine tuned ViT-Large model. The results are shown in Fig Obviously, standard fine tuned ViT-Large requires more shielded layers to protect its privacy. Averagely, the least number of shielded layers should
27

---

GTSRB
UTKFace
Sun397
Sensitivity
Sensitivity
Sensitivity
5
10
5
10
5
10
Layer idx MINST
Layer idx CIFAR-10
Layer idx CIFAR-100
Sensitivity
Sensitivity
Sensitivity
5
10
0
5
10
0
5
10
Layer idx
Layer idx
Layer idx
Figure 6: Sensitivity scores distribution of all layers for VTT-Base with six vision task benchmarks
greater than 15 while AegisGuard could achieve black-box level protection with approximately 1/3 shielded layers.
CIFAR-10
CIFAR-100
UTKFace
100
Random
Random
Random
Accuracy (%6
15
16
21
24
21
24
12
Shielded Layers
Shielded Layers
Shielded Layers
Figure 7: Accuracy comparison of Random Shielded Layers Selection and AegisGuard
Implementation of Baselines
Given the lack of previous works' implementation and experiments on TEE shielded fine tuned LMs, especially with low-rank adaptation, there is no ready-to-use tools for baseline evaluation. To provide a comprehensive demonstration the effectiveness of AegisGuard, we adapt most recent TEE-shielded Deep Neural Networks methods for evaluation.
E.1
Phantom Description
Phantom is a recent method proposed at USENIX Security 2025, which leverages reinforcement learning-based architecture search combined with top-K layer-wise sensitivity analysis to identify the most sensitive layers for inserting redundant modules. These redundant layers are designed degrade model performance for unauthorized users, while maintaining full functionality for authorized users who possess valid decryption keys. During inference, authorized users must verify their keys within
28

---

Trusted Execution Environment (TEE) to obtain correct predictions; otherwise, the model's forward pass becomes corrupted, leading erroneous outputs.
However, Phantom tailored specifically for models trained from scratch and cannot be directly applied large models (LMs) fine-tuned with Low-Rank Adaptation (LoRA). To enable fair comparison, we assume that adversaries lack knowledge of the actual forward computation path of a LoRA-based LLM. Accordingly, we propose insert a subset of redundant LoRA adapters into the model. Our objective is to maximize the loss of LoRA-fine-tune redundant adapters. Following the experimental setup in Phantom, we insert redundant adapters into the three most sensitive layers, as determined by layer-wise sensitivity analysis. The corresponding results are presented in Appendix D.I]
models in the presence of these
E.2 TEESlice Description
Although TÉESlice are specifically designed for models, such as ResNet-18(12), VGG[3I], it has similar model partition mechanism adapter-based fine tuning. TEESlice consists of two stages: model slice extraction and hybrid model deployment. The slice extraction stage automatically finds the "sweet spot" by minimizing the utility cost while maintaining accuracy and security. TEESlice trains Maense from the public backbone and then prunes Mdense to get Maparse. In the hybrid model deployment stage, Maparee deployed across the TEE and GPU. Model slices are deployed inside TEEs, whereas the other part of the backbone are offloaded on GPUs.
In experiments, regard lora adapter model slices proposed in TEESlice. The objective of this method is dropping as more as adapters with minimal accuracy loss following the iterative slice prune mechanism TEESlice. However, the pre-defined threshold 5 defined 1% governs the tolerable accuracy loss during pruning is too small for LMs to drop adapters. Then we set threshold 5% tolerable accuracy loss. The main difference between TEESlice and AegisGuard that our method actively selects model layers with higher sensitivity, intentionally gathering more private information into the layers should be deployed within the TEEs. In contrast, TEESlice adopts passive selection strategy, merely determining whether removing certain layer would negatively impact the final performance. The experimental results and analysis are presented Appendix D.]
F Limitations
Although the architecture presented in this paper achieves competitive performance in terms of inference efficiency, defensive effectiveness, and accuracy, it has the following limitations. First, even though we compare AegisGuard with most of-he-tt baselines (TEESlice (47), Phantom[4], published in top-tier security conferences. We do not include methods from leading Al conferences. There is an ICML 2024 paper, namely GroupCover{{5 (55, but it targets convolutional layers and cannot be applied to the Transformer models used in our setting. Second, the baselines focus on smaller models such ResNet18 [22 and VGG16 [31] whereas evaluate on large-scale models. Although their techniques could in principle be adapted to larger architectures, they may not achieve the same level of effectiveness in that context.
29


Reviews:
[{"review_id": "QSV32gxsQu", "reviewer": "Reviewer_RwdN", "summary": "This paper proposes AegisGuard, a framework that enhances the security and efficiency of deploying large lora models on user devices by selectively shielding only the most privacy-sensitive LoRA adapters inside TEEs while offloading less sensitive adapters to the GPU. Specifically, it introduces a reinforcement learning-based sensitivity measurement that uses Gaussian noise to rank adapter sensitivity to model stealing attacks and a shielded adapter compression (SAC) module to prune sensitive adapters. It reduces computation and communication costs. Experiments on LLaMA-7B and ViT show that AegisGuard maintains black-box level security, whiling significantly reducing inference latency and lowering TEE memory usage.", "questions": "a) My main concern is the robustness of the layer sensitivity estimation, which is the core of the proposed method. The intuition was 'perturbing sensitive parameters leads to a larger loss perturbation', however, it also depends on the chosen input? How can we show the proposed method is robust to the chosen samples?\n\nb) In Table 2, why ViT-Base C100 Shield-LoRA (19.25) was not the best (colored rectangle) but AegisGuard (22.67)? Same questions for LLaMA-7B.", "limitations": "Limited scope to adapter-based LoRA fine-tuning might be a limitation.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n\n+) The paper is well-written and easy to follow\n\n+) The overall design is technically sound and reasonable to me.\n\n+) It is a good idea to co-design the lora layers and secure inference platform.\n\n\n**Weakness**\n\n-) Limited scope to adapter-based fine-tuning, specifically LoRA.\n\n-) The sensitivity estimation method may not be robust.\n\n-) Some evaluation results are not clear to me (see questions)", "was_revised": true, "final_justification": "After reading the rebuttal, i maintain my scoring of weak accept.", "rebuttal": "We thank the reviewer RwdN for the valuable feedback on our paper. Our responses to the comments are below.\n\n**Question 1**: The robustness of the layer sensitivity estimation.\n\n**Answer 1:**\nOur layer sensitivity estimation method is robust to the choice of input samples, as shown by its convergence behavior, stability across random seeds, performance when experimented on different domain datasets, and its design based on parameter-space perturbations. We support this claim in the following:\n\n+ **The entire RL-guided fine-tuning is a convergence process that runs over many random batches, not a single sample**. The overall objective of our method is to fine-tune a large model that would converge to an optimal loss with a fixed number of layers. Note that during our designed RL-guided fine-tuning process, the model is not just exposed to a single random batch, but multiple random batches throughout. Additionally, our weighted layer selection sampling mechanism is designed to differentiate layer wise sensitivity with more sensitive layers (those with higher sensitivity scores) being fine-tuned more frequently. Therefore, if our estimation were highly sensitive to the batch composition, the selected sensitive layers would fluctuate throughout training, leading to instability or divergence. In contrast, we observe that the final adapter selection converges, resulting in successful downstream task performance. This confirms that sample-specific variance is smoothed out through repeated exposure and weighted selection during RL optimization.\n\n+ **Layer sensitivity is stable or converges across different datasets**. The results presented in Appendix D.3, Figure 6, also demonstrate this point. The sensitive layer selection process is stable and shows convergence in experiments conducted on domain-specific tasks across different random seeds. This shows that our method is robust to different task domains. \n\n+ **Our perturbation is injected into LoRA weights rather than inputs, and is larger than the perturbations injected into inputs in previous works**. Since the core design concept of our method is to disrupt the impact of specific LoRA layer weights rather than subtly perturbing inputs to enhance training robustness, we can inject large noise into it without considering model performance. This observation is drawn from the analysis of weight magnitude distribution presented in Figure 5 of the Appendix. LoRA is designed to introduce small, low-rank updates to the base model, and its parameter magnitudes are significantly smaller. This ensures that our perturbation can be large to effectively diminish the contribution of the fine-tuned LoRA layer while preserving the functionality of the original base model. Thus, the influence of the perturbed layer can be isolated and evaluated robustly, without being influenced by the chosen samples.\n\n**Question 2**: \nIn Table 2, why ViT-Base C100 Shield-LoRA (19.25) was not the best (colored rectangle) but AegisGuard (22.67)? Same questions for LLaMA-7B.\n\n**Answer 2**\nIn Table 2, Shield-LoRA is not a baseline defense solution so we don't highlight Shield-LoRA in all settings. It is only a reference point (security upper-bound of black-box setting) for comparison. The colored rectangle is only for the best-performing defense among the baselines and the proposed solution.\n\n\n**Limitation 3**: \nThe limited scenario of adapter-based LoRA fine-tuning.\n\n**Answer 3**\nWe focus on LoRA as it is the most commonly used parameter efficient fine-tuning method currently. Especially in the privacy field, many works[a][b] design their privacy methods based on LoRA-based fine-tuning scenarios. Additionally, many industrial on-device practices such as Apple[d] and Nvidia[e] have also adopted LoRA as a mainstream fine-tuning method. For example, Apple developed a new framework using LoRA adapters that incorporates a mixed 2-bit and 4-bit configuration strategy to maintain model quality. Nvidia developed the NeMo framework that mainly supports dynamic multi-LoRA inference, enabling simultaneous inference requests with different LoRA models.\n\nBesides, compared with normal adapters, LoRA contains less trainable parameters and are thus more compatible with TEE contraints (memory and computation constraints[c]). When we adopt TEE to provide black-box level security for LLM fine-tuned with private or sensitive data, efficiency is also a key objective. LoRA achieves this by significantly reducing the number of trainable parameters through low-rank matrix decomposition, compared with traditional adapter-based techniques. It not only improves computational efficiency but also achieves strong performance in modern applications, making it well-suited for TEE-based secure fine-tuning. As a result, in our setting (where resource efficiency and security are critical), LoRA is a practical choice.\n\n\n[a].Sun, Youbang, et al. \"Improving LoRA in Privacy-preserving Federated Learning.\" The Twelfth International Conference on Learning Representations(ICLR2024). \n\n[b].Luo, Zihao, et al. \"Privacy-Preserving Low-Rank Adaptation Against Membership Inference Attacks for Latent Diffusion Models.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 6. 2025.\n\n[c].Sun, Zhichuang, et al. \"Shadownet: A secure and efficient on-device model inference system for convolutional neural networks.\" 2023 IEEE Symposium on Security and Privacy (SP). IEEE, 2023.\n\n[d].Apple. \"Introducing Apple’s On-Device and Server Foundation Models\"\n\n[e].NVIDIA. \"Parameter-Efficient Fine-Tuning with NVIDIA NIM for LLMs\""}, {"review_id": "l7cqdfKBpx", "reviewer": "Reviewer_16rH", "summary": "This paper proposes AegisGuard, a fine-tuning framework that balances efficiency and sensitivity. Unlike traditional approaches, which guard LoRA adapters in TEE incurring extra latency, AegisGuard uses RL-based sensitivity measurement to figure out which adapter is more sensitive, and reduce the TEE execution overhead via pruning and compression. Evaluation shows great reduction in latency and memory while maintaining model accuracy as well as model stealing resilience.", "questions": "How is the impact of pruning on accuracy compensated by the next step of fine-tuning? This part is a bit counterintuitive. If it's because pruning is losing very little, then where is the limit? How much more pruning can be done to actually affect downstream accuracy? This is a complex interplay, with the selection of adapters. Unfortunately, the current evaluation, even appendix, couldn't capture this interplay well.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "\\+ Neat idea and great execution\n\n\\+ TEE execution overhead is a pain point to solve, and the solution is simple yet elegant\n\n\\+ Nice evaluation on inference latency, defense effectiveness, model accuracy, as well as memory results in appendix\n\n\\- From figure 1, transfer latency seems the biggest bottleneck. In the evaluation results, it would’ve been nice to articulate the reduction, breaking down how much is attributed to transfer, how much to TEE execution, etc. The TEE execution reduction should also highlight the effect of pruning. Combining both, a comprehensive ablation study on latency would serve the paper well.\n\n\\- There are many approaches in analyzing layer/adapter sensitivity. It would have been nice to have a discussion of the design philosophy of the chosen approach. It would also be nice to have a comparison, just on this component, with other selection approaches. \n\n\\- Format:  Font in figure 1 is too small. In Figure 2, the flow of the upper half is a bit messy. It would be helpful to rearrange the layout for better readability. Table 1, please add units (i.e. ms vs sec) to avoid confusion", "was_revised": true, "final_justification": "The author response addressed most of concerns. The reviewer maintain favorably position on the review", "rebuttal": "Thanks for providing valuable feedback on our paper. Our responses to the comments are below.\n\n**Question 1**: How is the impact of pruning on accuracy compensated by the next step of fine-tuning? If it's because pruning is losing very little, then where is the limit? How much more pruning can be done to actually affect downstream accuracy?\n\n**Answer 1**: \nThe accuracy drop of pruning can be compensated by the next step of fine tuning is because we adopt a **mixed strategy of pruning and fine-tuning**. Pruning is periodically applied during the fine-tuning process, and the pruned weights are permanently removed (i.e., not reconnected). After each pruning step, the remaining weights continue to be updated, progressively learning more task-relevant information during subsequent training. This allows the model to gradually adapt to the reduced parameter space and partially recover accuracy. \n\nIn contrast to traditional structural pruning methods that prune uniformly across all layers [a], our method adopts a dynamic and selective pruning strategy, focusing primarily on the shielded layers, those would execute inside the TEE. Consequently, **the total number of pruned parameters is relatively small compared to conventional pruning methods**, which helps preserve downstream task accuracy.\n\nTo better understand the pruning limits, we conduct additional experiments under various pruning ratios (e.g., 20%, 30%, and 50%) on ViT-Base and report the corresponding interplay and downstream accuracy. \n\n| Model     | Variant                  | CIFAR-10 | CIFAR-100 | UTKFace | MNIST | GTSRB | Sun397 | Avg.  |\n|-----------|--------------------------|----------|-----------|---------|--------|--------|--------|--------|\n|           | Shield-LoRA(baseline)    | 97.47    | 85.96     | 75.03   | 98.90  | 93.35  | 53.37  | 84.01 |\n|           | AegisGuard(20%)          | 97.83    | 86.20     | 75.21   | 98.68  | 93.51  | 53.54  | 84.16 |\n|  ViT-Base | AegisGuard(30%)          | 97.97    | 86.07     | 75.01   | 98.25  | 93.20  | 53.76  | 84.04 |\n|           | AegisGuard(50%)          | 95.21    | 79.84     | 70.27   | 95.02  | 89.65  | 50.43  | 80.07 |\n\nAs shown above, AegisGuard's performance remains stable at 20% and 30% compared with baseline, but it has a obvious degrade (around 4% drop) at 50% pruning ratio. These results help delineate the boundary of effective pruning in our setting.\n\n**Weakness 2**： Ablation study of latency overhead.\n\n**Answer 2**: \nWe provide a breakdown of inference latency components (GPU, TEE, and Transfer) in Table 1 of Section 4.2. For example, in the case of ViT-Base, the GPU processing time is 5.94 ms, TEE execution takes 4.31 ms, and data transfer costs 4.89 ms per batch. These results confirm that compared to Shield-LoRA, AegisGuard achieves 2-3 $\\times$ reductions in TEE execution and transfer overhead. \n\nTo further quantify the effect of pruning within the TEE, we include an ablation study as shown below to illustrate the execution cost of ViT-Base. This experiment measures the TEE execution time (averaging 10 batches) under different pruning ratios, demonstrating how pruning directly reduces the secure enclave’s computation cost. We will put this ablation study into appendix in the next version.\n\n| Pruning Ratio (%) | TEE Execution Time (ms) | Relative Reduction|\n|-------------------|--------------------------|------------------|\n| 0%               | 6.01                     | 1.00×            |\n| 20%               | 4.82                     | 1.25×            |\n| 30%               | 4.31                     | 1.39×            |\n| 50%               | 3.26                     | 1.84×            |\n\n**Weakness 3**: The discussion of the design philosophy of the chosen approach compared to other methods.\n\n**Answer 3**: \nOne representative approach is gradient-based methods[b][c], but they can not be applied to our scenario to estimate layer/adapter sensitivity. It is because these methods estimate the sensitivity based on layer/adapter gradients and generally assume that all layers are updated simultaneously during training. \n\nIn our setting, we adopt a dynamic fine-tuning strategy, where only a subset of layers/adapters are activated and updated at each training step, that the rest are frozen. This strategy is to identify which layers/adapters are more sensitive, and can better absorb task-specific features. As a result, gradient-based sensitivity estimation becomes infeasible, since gradients are not available for the frozen layers at any estimation step. \n\nTo address this limitation, we propose a perturbation-based sensitivity estimation method. This approach estimates a layer's influence by observing the loss change when a perturbation is applied to its parameters. It is straightforward to implement in the dynamic fine-tuning setting, computationally efficient. Additionally, to mitigate variance from single batche samples, we perform sensitivity estimation at fixed intervals (e.g., every 20 steps), ensuring more stable and robust evaluations. Overall, our design philosophy prioritizes and can better compatibility with our dynamic fine-tuning strategy.\n\n**Weakness 4**: Format issues\n\n**Answer 4**: Thank you for pointing out the format issues. We will revise the formatting of fonts, as well as improve the layout and flow of figures and tables to enhance the readability and presentation quality in the next version.\n\n\n[a]. Ma, Xinyin, Gongfan Fang, and Xinchao Wang. \"Llm-pruner: On the structural pruning of large language models.\" Advances in neural information processing systems 36 (2023): 21702-21720\n\n[b]. Sahoo, Sabyasachi, et al. \"A layer selection approach to test time adaptation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 19. 2025.\n\n[c]. Lee, Yoonho, et al. \"Surgical Fine-Tuning Improves Adaptation to Distribution Shifts.\" The Eleventh International Conference on Learning Representations."}, {"review_id": "LH0GknIRBU", "reviewer": "Reviewer_PYU5", "summary": "This paper is to introduce AegisGuard, a fine tuning and deployment framework that selectively shields the MS sensitive adapters while\noffloading the rest to the GPU, balancing security and efficiency. The main contribution includes: 1. the proposed solution reduces the communication workload between TEE and GPU for TSDP solutions through an RL-based sensitivity measurement approach for on-device LM inference. 2. the framework compresses intermediate feature maps to reduce the communication cost after deployment. The authors have provided comprehensive experiments for deployed LMs, demonstrating the inference efficiency, robust security against model stealing attack, and minimal accuracy loss of proposed AegisGuard.", "questions": "1. Is there latency overhead for the sparse metrics operations after structural pruning? What is the tradeoff and benefit from the compression?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths \n1. This paper presents this fine tuning and deployment framework, which have shown its efficiency and effeteness. In order to reduce the communication cost, the framework adopts a selective shielding strategy, where only the most sensitive adapters are executed inside the TEE, while less sensitive ones are offloaded to the untrusted GPU. Furthermore, structurally prunes are also introduced to further\nlowering TEE computation and data transfer costs\n2. The submission demonstrates a strong technical foundation with a robust experimental design and clear evidence supporting the claims presented in the paper. The experiments include a comprehensive evaluation for the proposed framework.\n3. The paper is well-structured and clearly written, making it accessible to a broader audience.\nWeaknesses\n1. Other than the efficiency of selective shielding strategy, more impact investigate of shielded adapter compression part could help the design more convincing, including the tradeoff between accuracy and latency, how much contribution from this specific part alone, etc.", "was_revised": true, "final_justification": "With the rebuttal and discussions, I will keep the initial rating as my recommendation.", "rebuttal": "We very thank the reviewer PYU5 for the valuable feedback on our paper. Our responses to the comments are below.\n\n**Question 1**: Is there latency overhead for the sparse metrics operations after structural pruning?\n\n**Answer 1**: \n**No, there is no significant latency overhead introduced by sparse matrix operations in our method**. This is because we apply structural pruning, specifically by removing attention heads, rather than performing unstructured pruning at the level of individual elements. As a result, the remaining weights can still be represented and computed using dense matrix operations, without introducing sparsity. Therefore, our method preserves the efficiency of dense computation and does not suffer from the irregular memory access patterns or hardware inefficiencies commonly associated with sparse operations.\n\n**Question 2**: What is the tradeoff and benefit from the compression?\n\n**Answer 2**: \nThe tradeoff of compression is that it requires longer finetuning time before model deployment, it's about 1.5 $\\times$ fine tuning time costs than normal fine tuning without compression. However, this process only occurs once and does not introduce repetive overhead after deployment. It does not introduce overhead to the online inference process.\n\nThe benefit is a lower computation workload in TEE in the online phase. We provide a breakdown of inference latency components (GPU, TEE, and Transfer) in Table 1 of Section 4.2. For example, in the case of ViT-Base, the GPU processing time is 5.94 ms, TEE execution takes 4.31 ms, and data transfer costs 4.89 ms per batch. These results confirm that compared to Shield-LoRA, AegisGuard achieves 2-3 $\\times$ reductions in TEE execution and transfer overhead. Additionally, we also added the TEE computation costs reduction of pruning is presented below separately. The results are conducted on the ViT-Base model due to rebuttal time constraints. This experiment measures the TEE execution time (averaging 10 batches) under different pruning ratios, demonstrating how pruning directly reduces the secure enclave’s computation cost. We will put this ablation study into appendix in the next version.\n\n| Pruning Ratio (%) | TEE Execution Time (ms) | Relative Reduction|\n|-------------------|--------------------------|------------------|\n| 0%               | 6.01                     | 1.00×            |\n| 20%               | 4.82                     | 1.25×            |\n| 30%               | 4.31                     | 1.39×            |\n| 50%               | 3.26                     | 1.84×            |"}]

Author rebuttal:

Meta-review: This manuscript proposes AegisGuard, a lightweight mechanism for protecting on-device models in TEEs against model stealing attacks. The claimed contributions are: (1) selective protection based on an RL-based layer-wise sensitivity measure, and (2) model compression techniques tailored to adapters deployed within TEEs. Empirical evaluation shows that AegisGuard is resilient to black-box attacks while reducing inference latency by up to 4x times.

The AC recommends therefore, "accept as a poster".

All reviewers were positive about AegisGuard, particularly its focus on addressing the inference latency of existing protection mechanisms and its comprehensive evaluation. The manuscript is also clearly written and easy to follow. The AC agrees: this work demonstrates how an ML-level framework can complement system-level protection to mitigate latency challenges.

The AC also agrees with several limitations raised during review. (1) The impact of the proposed protection mechanism is somewhat limited, as the solution focuses primarily on LoRA. The contribution would have been stronger had it extended to full fine-tuning, which is standard in fine-tuning for constructing chat models. (2) The paper lacks a comparison or discussion of alternative approaches to estimate sensitivity (or alternative metrics for evaluating layer vulnerability to model stealing attacks). These issues do not block acceptance, but they suggest that the impact and audience for this work may be narrower.

**Revision guidance.** For the camera-ready version, the AC recommends including any additional experiments in the Appendix, while restricting main-body revisions to minimal changes (e.g., typos, clarifications) so the scope does not exceed shepherding.
Decision: Accept (poster)