LoRO: Real-Time on-Device Secure Inference for LLMs via TEE-Based Low Rank Obfuscation
Authors: Gaojian Xiong
Yu Sun
Jianhua Liu
Jian Cui
Jianwei Liu
Venue: NeurIPS 2025 poster ()
Keywords: secure inference
large language models
trusted execution environments
intellectual property protection
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We analyze and compromise existing TEE-based secure inference methods by our Model Stealing attacks with Prior, and propose a novel secure inference method named LoRO, which is efficient, secure and accurate.

Abstract: While Large Language Models (LLMs) have gained remarkable success, they are consistently at risk of being stolen when deployed on untrusted edge devices. As a solution, TEE-based secure inference has been proposed to protect valuable model property. However, we identify a statistical vulnerability in existing protection methods, and furtherly compromise their security guarantees by proposed Model Stealing Attack with Prior. To eliminate this vulnerability, LoRO is presented in this paper, which leverages dense mask to completely obfuscate parameters. LoRO includes two innovations: (1) Low Rank Mask, which uses low-rank factors to generate dense masks efficiently. The computing complexity in TEE is hence reduced by an exponential amount to achieve inference speed up, while providing robust model confidentiality. (2) Factors Multiplexing, which reuses several cornerstone factors to generate masks for all layers. Compared to one-mask-per-layer, the secure memory requirement is reduced from GB-level to tens of MB, hence avoiding the hundred-fold latency introduced by secure memory paging. Experimental results indicate that LoRO achieve a $0.94\times$ Model Stealing (MS) accuracy, while SOTA methods presents $3.37\times$ at least. The averaged inference latency of LoRO is only $1.49\times$, compared to the $112\times$ of TEE-shielded inference. Moreover, LoRO results no accuracy loss, and requires no re-training and structure modification. LoRO can solve the concerns regarding model thefts on edge devices in an efficient and secure manner, facilitating the wide edge application of LLMs.

LoRO: Real-Time on-Device Secure Inference for LLMs via TEE-Based Low Rank Obfuscation
Gaojian Xiong, Yu Sun; Jianhua Liu, Jian Cui, Jianwei Liu School of Cyber Science and Technology, Beihang University, China (xionggaojian, sunyv, jason_hua, cuijianw, liujianweil (awpweeeuuuua..ee edu cn
Abstract
While Large Language Models (LLMs) have gained remarkable success, they are consistently at risk of being stolen when deployed on untrusted edge devices. As a solution, TEE-based secure inference has been proposed protect valuable model property. However, we identify statistical vulnerability in existing protection methods, and furtherly compromise their security guarantees by proposed Model Stealing Attack with Prior. To eliminate this vulnerability, LoRO is presented in this paper, which leverages dense mask completely obfuscate parameters. LoRO includes two innovations: (1) Low Rank Mask, which uses low-rank factors to gen- erate dense masks efficiently. The computing complexity in TEE hence reduced by exponential amount to achieve inference speed up, while providing robust model confidentiality. (2) Factors Multiplexing, which reuses several cornerstone factors to generate masks for all layers. Compared to one-mask-per-layer the secure memory requirement reduced from GB-level to tens MB, hence avoid- ing the hundred-fold latency introduced by secure memory paging. Experimental results indicate that LoRO achieve a 0.94x Model Stcaling (MS) accuracy, while SOTA methods presents 3.37x at least. The averaged inference latency of LoRO is only 1.49x, compared to the 112x of TEE-shielded inference. Moreover, LoRO results accuracy loss, and requires LoRO can solve the concerns regarding model thefts on edge devices in an efficient and secure manner, facilitating the wide edge application of LLMs.
re-training and structure modification.
Introduction
Large Language Models have demonstrated considerable ability in various domains [ However, the predominant cloud- based deployment exposes critical limitations: users have to tolerate additional network latency and upload their private data. In domains requiring real sime responsiveness and private data protection, e.g. autonomous driving [2] and personal smart agent [3), there is growing demand to deploy LLMS on user's edge devices with accelerators [4] 5). Alarmingly, on-device models are usually white-box vulnerable to be stolen 7 Since L.LMs are extremely expensive to train, it is expected that users can benefit from edge inference an efficient but black-box manner, i.c. enjoy high-quality and fast inference with knowledge about private parameters.
Trusted Execution Environment (bse) secure inference 9
13 [4 have been proposed to safeguard valuable on-device models against theft. Advanced TEE-based methods obfuscate and offload model parameters to Rich Execution Environment (REE) for acceleration using GPUs. The de-obfuscation keys are shielded in TEE to restore the inference results accurately. Compared cryptograph- based secure inference that relies on Homomorphic Encryption [I5] or Multi-Party Computation [16] TEE-based methods offer more practical solutions due to their real-time performance
14 and elimination network bandwidth occupation.
"Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Table 1: Model Stealing Accuracy (4) against proposed LoRO and SOTA TEE-hased secure inference methods. The averaged accuracy of each defense relative to black-box is reported in the last row. The best performance is in green, and the worst is in red.
Model
Model Size
Dataset
No-Shieli
TLG
ShadowNed [1]
Magnitude [
SOTER [00]
Ours
Black Dox
Food101
81.57%
81.51%
71.90%
80.05%
75.52%
16.05%
10.28%
CIFARIDO
89.86%
89.675
79.27%
87.28%
88.305%
15,03%
14.57%
SQLAD
81.84%
47.44%
$4.045
50.33 32%
45.51%
6.35%
7.01
RoBERTa
Base
MRPC
87.99%
69.39%
70.01%
86.88%
82.00 22.06%
31.61%
33.90%
97.78%
91 91.42%
73.43%
69.45%
85.33%
84
$2.84%
79.435%
84.00%
45.18%
44.02%
SQuAD
83.15%
49.21%
37.32%
19. 19.8
12.75 12.79%
10.50%
7.36%
BART
Large
MRPC
SST-2
$7.015
90,909
75.93%
68. 89%
55.53%
91
79. 96%
82 11%
31,61%
67.995
68.
31,84%
MNLI
$3.15%
86.05%
79.50%
81.62%
85.19%
49.90%
49,96%
1.5B
SQuAD
68.15%
29. 33.3%%
21.22%
26.86%
24.05%
5.07%
11.20%
Qwen2
3B
GSMSK
70.96%
40.1356
28.27%
34, 1556
22.83%
1.70%
5,85%
Spider
17.99%
13.55%
10.15%
9.69%
2.32%
2,50%
1,5H
SQJAD
$0.205
30.625
23.53%
28 72.7%%
26 80%
7.90%
9.73%
LLaMA3
38
GSMBK
53.125%
14.57%
12.31%
11.68%
12.47%
1 1.56%
2.31%
8B
Spider
34.62%
12.99%
10.70%
9.15%
11.94%
3.90%
3.82%
Average
7.03
4.12x
3.37x
3.85x
3.75x
0.94x
1.00x
Nevertheless, we identify a fundamental statistical vulnerability in the state-of-the-art protection methods, and undermine their security guarantees. As shown in Figure 2 since LLMs typically undergo retraining from public available pretrained models, there remains a statistical similarity in parameters or intermediate results [33]. Regretfully, current protection schemes fail to effectively conceal this inherent correlation. Based this vulnerability, we propose Model Stealing attack with Prior knowledge (MSP), which enables adversary to approximate the de-obfuscate keys using public prior knowledge, and recover private parameters through minimal tne-tuning. As demonstrated in Table D our MSP attack can achieve a high accuracy near original model, indicating that LLMs protected by existing methods could still be stolen. The details of MSP is introduced in Section 4
To address aforementioned problem, we present LoRO (Low-Rand Now-Rank-Obfusee Obfuscation), an efficient secure inference framework. To eliminate statistical vulnerability, dense masks are leveraged to completely obfuscate private parameters. But two challenges are raised by dense mask: (CI) high computing complexity in TEE and (C2) GB-level secure memory requirement, which can cause unacceptable hundred-foo fold latency. To solve (CI), Low Rank Mask is designed to reduce the computing complex- ity significantly. We employ low-rank factors to randomly generate dense masks, which reduces complexity from ((n²) ) of full matrix multiplication to O(n"), enabling real-time inference with- out compromising security. To solve (C2), Factor Mulriplexing is designed to optimize the secure memory requirement. Factor Multiplexing generates dense masks through efficient random linear combinations of several cornerstone factors. This optimizes pne-mask-per-layer to only several comerstone factors. For 7B models, the secure memory can be reduced from 1.02 GB to only 26 MB.
Experiments conducted on Intel SGX and Arm TrustZone platforms to evaluate proposed LoRO, focusing on three aspects: (1) For model confidentiality, LoRO effectively defends against Model Stealing (MS) attacks with near black-box performance (0. (0.94x), outperforming existing methods that exhibit 3.37x x at least. (2) For inference efficiency, LoRO's 's lightweight design maintains only a 1.49x inference latency average (compared to REE inference), achieving significant improvement over existing approaches the reaches 112x at least. (3) For model accuracy. LoRO introduce zero accuracy loss. Moreover, no training effort and architecture modification is required.
Proposed LoRO effectively protect edge-deployed LLMs as black-box model, while providing real- time and accurate inference service. Moreover, LoRO is plug-and-play without modifying the model structure or retraining. We believe that LoRO can solve the concerns regarding model thefts on edge devices, and facilitate the wide application of LLMs.
The code for implementing LoRO is available in hatp://gtttabccmm///llaboommmmm Our contri- butions could be summarized as follows:
We reveal a statistical vulnerability in existing TEE-based secure inference methods. Based on this, we propose Model Stealing attack with prior, which empirically demonstrates the SOTA methods inadequately deliver their security guarantee when protecting LLMs.

---

Table 2: Representative methods of TEE-based secure inference compatible with LLMs. O means
no access to REF, means only part of parameters accelerated by REF, and means full acceleration.
REE
No Retraining
No Architecture
Resistance to MSP attacks
Obfuscation
Category
Method
Acceleration
Requirement
Modification
Scheme
Ml.Capuue
O
None
Full Shielding
Penetralium
None
Partial Shielding
AegisDNN
0
None
TEESlice
1
X
V
None
Magnitude
Additive Multiplicative Add & Mal
SOTER
De-Obfuscation
Nha
Shielding
NNSplitter
Addiúive
Permutation
LoRO
Additive
We present LoRO, TEE-based efficient secure inference framework, which consists of two components: (1) Low Rank Mask to shield valuable LLMs from thefts, while providing real- time and accurate inference performance
edge. (2) Factor Multiplexing climinate the intensive secure memory requirement of protecting LLMs, which avoids the hundred- latency introduced by frequent secure memory paging.
Experiments on both Intel SGX and ARM TrustZone demonstrate that MS attack accuracy is reduced to black-box level (0.94x) from existing x. Moreover, LoRO introduces only 1.49x inference latency, compared the 112x of TEE-shielded inference. Notably, LoRO introduces no accuracy loss, and requires no re- re-training or architecture modification.
2 Related Work
2.1 Trusted Execution Environment
Trusted Execution Environments (TEEs) employ memory isolation to ensure the confidentiality and integrity of application inside. Prominent edge TEE implementations include Intel SGX [17] and Arm TrustZone [8]. However, edge TEEs are originally designed for small critical tasks, e.g. key exchange. They typically equip with limited computational resources and secure memory capacities restricted to 128 MB. Deploying LLMs within TEEs induces excessive secure memory paging, which necessitates additional cryptographic operations maintain data security. Consequently, inference latency could escalate by hundred-fold [99 20 21, which is unacceptable to users.
2.2 TEE-based Secure Inference
TEE-Based Secure Inference aims to protect edge-deployed valuable models from being stolen. Advanced methods leverage TEE to safeguard eritical neural network components, e.g. selective parameter subsets and de-obfuscation keys. The computation-intensit major parameters [2 are offloaded REE, enabling access to hardware accelerators (GPU/NPUs) for accelerations. Without the secret parts in TEE, adversary can only access the model with performance degraded to nearly random guessing. Compared cryptographic-base approaches, TEE-based solutions demonstrate superior practicality by reducing the inference latency and eliminating the network occupation. As shown in Table existing methods can be categorized follows:
Full shielding. Some research focus shielding the whole inference process in TEE. As typical method, MLCapsule (23) decrypts model layer-by-layer in TEE avoid secure memory paging. Similarly, Penetralium 20 breaks down the models in parts by novel adaptive decomposition algorithm, achicving efficient memory allocation. Such methods protect all parameters TEE, preventing any possible privacy leakage. However, the absence of REE acceleration would result in hundred-fold inference latency, rendering they impractical for real-time tasks.
25 26 have proposed partial model shielding strategies. For example, AegisDNN [27] designs dynamic programming algorithm
Partial shielding. To facilitate REE acceleration, several studies 24

---

Model Manufacturer
Obfiscated
Permulation
Samilaniy
Hungarian Algorithm
Key: x
y
Partilly Recovered Model
Private Datesel
Clusters
LLM
Deploy
Additive Mask
Small Durtset
Obfissatted Model
Q
Gradients Ancenoly
Sclectively Fine Tune
Edge REE
Edge
Location
Small
Datioel
Liftloed
Model Secret
Multiplicative Noise
Var (W)
Key:
Obfixcatted
...
Ohfinecated
Eatimane
Fino-Tune
Model De-Offusctiinn
Knockoff
(a) Edge Deployment Scenario
(b) Pipeline of Model Stealing with Prior
Figure 1: (a) Edge Deployment Scenario (b) Pipeline of proposed Model Stealing attack with Prior.
identify and protect only the most critical layers. Likewise, TEESlice 13 suggests to freeze
the backbone, confining all privacy in shielded model slices. These approaches achieve a security- efficiency balance small models, but face severe latency delay in LLMs.
De-obfuscation shielding. Recent studies have increasingly focused on parameter obfuscation methods. The parameters are obfuscated and offloaded to REE for acceleration [12 28 Subse- quently, intermediate results are de-obfuscated in TEE using shielded keys. According to obfuscation mechanism, current methods can be classified into three categories: permutation [14], additive mask
[ 12 and multiplicative noise [TO] [1] Since the major computation (commonly over 95% [1]
is offloaded to REE, obfuscation-bast solutions provide remarkable inference efficiency.
We introduce the protection mechanism of each methods, and analyze their vulnerability in Section
3
Threat Model
Edge environment. As shown in Figure 1 (a), we consider edge environment, where edge devices are equipped with TEE and accelerators. The TEE considered absolutely secure, preventing any data leakage and integrity compromising, Side channel attacks (29 30]. which could cause privacy leakage TEE, is not considered in this study.
Adversary. The adversaries try to achieve a surrogate model that exhibits similar performance to the edge-deployed LLMs. These adversaries possess powerful capabilities, controlling the entire environment external to the TEE, including the operating system and hardware. They are also well-versed in artificial intelligence. To enhance MS attacks, adversaries model architecture from information available in the REE and utilize publicly available models from the Internet. A small dataset (about 10% size of training set) is held by adversaries to launch attacks.
deduce the complete
Design goal. The goal of model manufacturer is to protect the edge-deployed LLMs from thefts, while providing fast and accurate inference service to users. To achieve this goal, the following objectives should be met simultaneously:
Model Confidentiality: Without authorization to TEE, adversaries cannot obtain the param- eters of deployed LLMs through MS attacks analysis attacks. The ideal situation that models are protected as black-box.
Inference Efficiency: Inference latency of deployed models should be reduced as possible. The ideal situation that inference latency is comparable to that in REE.
The model accuracy should be maintained to provide high-quality service. The ideal situation is that no loss in accuracy is introduced.
Model Accuracy:

---

High Simiartt
Acomoly Large Gndiezt
(a) TLG.
(b) Magnitude.
(c) Proposed LoRO.
Figure 2: T-SNE visualization of parameters on representative methods. Each point represents the column vector from weight matrix. Number in (a).c) indicates the cosine similarities, and line in (c),(c means the shift distance of obfuscated parameters.
4 Model Stealing Attack with Prior
In this section, we introduce how MSP is designed in detail. For each methods category, we first analyze the potential risk, and then introduce our attack pipeline. Finally, experimentally evaluate our attack performance. The overall pipeline of MSP attack is depicted in Figure (b).
4.1 Permutation Methods.
Risk analysis. This approach leverages permutation to obfuscate model parameters [4]. Given permutation matrix म, in which elements E {0, 1) and πΛ² I, the input and parameters W of linear layer are obfuscated as follows: x' = 2 W = x W. Although the parameters are obfuscated, the output y is still accurate, since y x W xuz T xW.
However, a significant drawback of this approach lies the shared obfuscation key between in- termediate results and model parameters. Once adversaries successfully restore the intermediate results, they can directly recover the plain-text model parameters. Furthermore, permutation- based obfuscation fails conceal the statistical distribution of columns in the intermediate results. As shown in Figure (a), adversaries can exploit the high distribution similarity between private and public models to estimate and then de-obfuscea model without training effort.
Attack pipeline. Based on the risk analyzed above, we first estimate the permutation key based on the intermediate results distribution. Specifically, we formulate the attack as an optimal matching problem for maximal similarity of intermediate results columns. First, a batch of input x, is randomly chosen, and sent to both public pretrained model layer W, and obfuscated layer W' to get batched output y/p W, and y xr W' Given the columns of intermediate results, similarity matrix computed over the columns of y and y' as follows:
5= Si cosine ciiinn__sillarttys... V/,) 0S S columns 1],
(I)
where Yi and y represent the i-th column of and y' respectively, which is averaged across the batch. Then, based S, we leverage Hungarian Algorithm (31) resolve the optimal column matching. The permutation key π can be nearly perfectly recovered, and utilized to de-obfuscate parameters. Finally, perform fine-tuning using the limited dataset, to correct potential minor mismatches.
4.2 Additive Mask Methods.
Risk analysis. These methods apply additive mask parameters W' W + and apply One-Time-Pad (OTP) o on input: x' x + o. The results y is de-obfuscated in TEE as:
x W
Fr
(oW + or) = wW,
REE
TEE
Pre-Computed
where xW' is accelerated in REE, ar is securely computed in TEE, and oW +or is pre-computed and stored. Notably, 2r holds the same heavyweight complexity with zW, which results in unacceptable latency in TEE. To overcome this, existing methods leverage sparse mask to downgrade the
5

---

complexity. For instance, Magnitude [9] only obfuscate 1 % largest weights, and NNSplitter 12 selects extremely sparse parameters protect by reinforcement learning.
However, under the constraint of mask sparsity, only limited number of parameters are obfuscated. To reduce the model accuracy to random guessing, the magnitude of changes in the obfuscated parameters is significantly large [28] 321. Based on this insight, we observe a gradient anomaly phenomenon: compared the normal counterparts in the same layer, obfuscated parameters exhibit substantially larger gradient values on the training data. As shown in Figure 2 (b), obfuscated parameters can be easily located, and then selectively fine-tuned for recovery.
Attack pipeline. Based on the identified gradient anomaly phenomenon, we initially localize the obfuscated parameters. By iterating through the attacker's small dataset, we identify the top 5% of parameters with the largest averaged gradient obfuscated parameters. Subsequently, these identified parameters are set to be trainable, while the remaining parameters are frozen, enabling selective fine-tuning, Empirical observations reveal that the localization accuracy is not entirely precise, as small subset of obfuscated parameters still exhibits minimal variation. Consequently, few additional rounds of full fine-tuning are ultimately required.
4.3 Multiplicative Noise.
Risk analysis. These methods apply multiplicative factor to obfuscate parameters. For instance, SOTER [10] obfuscate layer using one factor W ,W, and de-obfuscate the results as: y - - 1- xW" =W. Similarly, ShadowNet [I] applies one factor for each columns.
However, as indicated in 13. multiplicative noise struggles to conceal the statistical distribution of parameters. The value of applied factor have significant impact on the variance parameters. As result, the adversary can leverage the variance of public models to estimate the applied factor, and then de-obfuscate the parameters.
Attack pipeline. Given the variance var(W) of public models, and the variance var(W"), we follow [13 tofirst estimate multiplicative noise as: μ vaa((W")//ar(()) Then we de-obfuscate parameters using μ', and fine-tune the model with adversary's 's small dataset.
4.4 Attack Results
As demonstrated in Table 1 we conduct experimental evaluations regarding the attack performance of proposed MSP. estimate strongest adversary, the highest result five individual experiments reported. Notably, the MS accuracy against existing methods reaches average of 3.37x to 4.12x, even comparable to the original model some scenarios, indicating that current approaches fail to safeguard high-value LLMs from thefts.
5 TEE-Based Low Rank Obfuscation
In this section, we introduce LoRO in detail. As shown in Figure it consists of two components: Factor Multiplexing to reuse the cornerstone factors in deployment stage, and Low Rank Mask to safeguard the valuable LLM from thefts in inference stage. These two components are described respectively as follows, and we provide persuade code in Appendix
5.1 Low Rank Mask
dilemma. Edge TEEs face significant resource constraints, particularly when protecting high-complexity computations, which can introduce a hundred-fold increase in inference latency [8] (2) Consequently, the core of TEE-based secure inference lies in maximizing inference efficiency while maintaining model confidentiality. On the one hand, full shielding methods can provide robust security, but the unacceptable latency renders them impractical. On the other hand, while existing methods offer substantial efficiency improvements, they fail to fully obscure statistical distribution (as detailed in Section 4. This compromise from confidentiality to efficiency significantly increase the risk of model theft, leading them vulnerable to proposed MSP attack. Hence, it's challenge achieve robust confidentiality in an efficient way.

---

Pre-Computed
LIT Keys
Restose
Renslt
Permuratio
Result Restore
X
OTP
Penuua dien
Aptrvatin
Adil
OTP
OTP
Nurm
D
W
We
Scaled Product Attention
Connerstone Factors
D2
Officd
D
W2
Wos
Accelerator
Adverssry
Multiplasing
Wg
Only Obfusested Parameters
Deploy ment
(b) Infereace Stage
Lightweight Operation in TEE
Hearyweight Inference in REE
Edge TEE
Fdge
Shickled Socrets
Figure 3: Illustration of proposed LoRO, which incorporates two stages: (a) Deployment Stage: to pre-computing keys and obfuscate parameters in TEE. (b) Inference Stage: inference with obfuscated parameters in REE and restore the results in TEE.
Our solution. To address the dilemma introduced above, our solution to obfuscate parameters using dense additive mask, which can fully obscure the statistical distribution and provide high security [3]. As shown in Figure 3 we focus on protecting parameters of linear layer. LoRO is compatible popular LLMs since they are almost based linear layers. Given the dense mask D, the parameter W is obfuscated as W = W + D. The inference results is described as follows:
xW
DD xW. TEE
REE
However, one main limitation is that the dense mask is too heavyweight to be shielded. Given the shape in n] of W. dense mask holds a high computation complexity O(n²). This is same with full shielding methods, which could result in an unacceptable latency in TEE. To tackle this problem and achieve real-time efficiency, our key insight is to generate dense mask using low-rank factors Given two low-rank factors Buxd Adxa and generated dense mask D BA, the inference results are re-formulated as follows:
y
WW =BA xW.
RBE
TEE
In conclusion, proposed methods address the iifiiiincy-confidentaaiii dilemma in following aspects. For efficiency, the complexity in TEE reduced from complex O(n²) to O(n² x d) = =Onn", where d Hence, inference efficiency is significantly improved. For confidentiality, each elements in W are obfuscated randomly. As shown in Figure since statistical distribution are fully obscured, adversaries cannot access W without obtaining shielded D)
Intermediate results protection. To protect masks from analysis attacks, the intermediate results should also be protected. For the input z and of linear layer, OТP is employed as: x' Same to Equation 2 the applied noise can be accurately removed. Specifically, for the scaled dot production of attention module [3), since OTP is not applicable 34 (as detailed in Appendix D leverage permutation matrix æ to protect intermediate results as follows:
Q'K"T Vk
QKT Vk
= k VV y/ = softmax
V msoftmax
V
(5)
where k is the hyper-paramen that has been set in model training stage. To accommodate model variants, the potential RoPE (35] and other activation function computed TEE. Then result is restored in TEE as: y n" y'y Notably, since our secret / is independent from the risk we discovered in TLG (detailed Section avoided. Due to page limitation, more details of design and paramteerssssers can be found in Appendix A

---

we aim to 5.2 Factor Multiplexing
Limited secure memory challenge. Edge TEEs are constrained by limited secure memory. typically capped at 128 MB [17 36). However, to shield pair of low-rank factors per layer for 7B LLMs, the secure memory could reach approximately 1.02 GB. The introduced memory paging issue leads to unacceptable increase in inference latency and potential security risk [37]. severely compromising the practicality. Hence, critical challenge lies in efficiently protecting low-rank factors within such restricted secure memory.
Our solution. Factor Multiplexing proposed address this challenge. Our insight lies in the random combination of several limited low-rank factors, which enables reusing factors to save secure memory. Instead of preparing pair of factors per layer, we randomly choose from corne- stone factors and generate dense mask. As shown in Figure (a), given a cornerstone factors set [A, A,, B1, ... ...BB). Ba), J. a small set S S {AI,..., An) x (B,,... B... is randomly whesen,where the [S] m. Then mask D) is generated as follows:
D= E aBjAi.
(6)
(A,,B))P
where is the random weight. In this way, only the cornerstone factors are required in memory. instead of the heavyweight one mask per layer. In real-time scenario, is set to two for best efficiency, and the low computation complexity maintained. Moreover, Factor Multiplexing enables updating the obfuscated parameters using new masks periodically. The secure memory requirement of one-mask mee-mask--er-layye per-layer and Factor Multiplexing is reported Appendix B
Experiments
In this section, LoRO is evaluated experimentally. According to design goal (in Section 3 answer the following questions:
RQI: How is the defense effectiveness of LoRO? (in Section 6.2
RQ2: What is the inference speed of LoRO compared to existing methods? (in Section 6.3
RQ3: What is the model accuracy of LoRO compared to original models? (in Section 6.4
6.1 Experiment Setup.
Benchmarks.
Proposed LoRO is evaluated on several famous benchmarks, including SQuAD
(reading comprehension) 38 GSM8k (mathematics) (39), Spider (code generation) 44 and repre-
sentative standard GLUE 41 The accuracy is reported as metric. For generation tasks, only exact matched answer is considered correct. We also evaluate LoRO on Computer Vision tasks, including and Food101
CIFAR100 42
4
Models.
Representative models of various scales and structure are selected, including RoBERTa BART [46) VIT 47
45
Qwen
48 and LLaMA 49 50
Hardware devices. Experiments regarding model confidentiality and accuracy conducted server equipped with two NVIDIA RTX 4090 GPUs. Inference efficiency are evaluated on two TEE platforms. For TrustZone, NVIDIA Jetson Orin NX board equipped with 6-core ARM Cortex-A78AE CPU and 1024-core NVIDIA Ampere architecture GPU is employed, and OP-TEE (5I) is leveraged as TEE OS. For Intel SGX, laptop equipped with Intel Core 19- 108858 CPU and Quadro T2000 GPU is adopted, and Gramine-SGr. [52] is the basic TEE OS.
Due to page limitation, more details and experiments can be found in Appendix A and
6.2 Model Confidentiality
As shown in Table T we report the MSP accuracy against LoRO and various defense methods. The significantly high MS accuracy (3.75x to 4.12x) demonstrate the vulnerability of current defense

---

Mi.rpp Seadap
Dererelz soTER
num
MRD 12
122.8x
322
jar
210 19
Latevey (laneyya.gg.
RoBERTa-B
BART-L
Qwen2-3B
L1MMA32-1B
Qwen2-7B
2L-MA3
Figure 4: Inference latency of various methods SGX environment. All results are normalized to REE inference latency (red line, xx). We depict in log scale for better visualization.
ere 12
MLCopaule
Seandah
DulaeTx SOTEE
172 47
209.00
229 AC
1oL
Zar
Lakenyy (Log
Lnay Iia
Part
10
RoBERTa-B
BART-L
Qwen2-1B
LL.MA332-BB
Qwen2-7B
Figure 5: Inference latency of various methods on TrustZone environment. All results are normalized to REE inference latency (red line, 1x). We depict in log10 scale for better visualization.
approaches to our proposed MSP attack. Notably, adversary can achieve near-complete recovery in several scenarios. In contrast, LoRO maintains robust defense performance of only 0,94x, comparable to the 1.00x baseline in black-box settings. These results indicate that LoRO provides effective protection on LLMs against MS attempts.
Answer to RQ1: Proposed LoRO effectively downgrades private LLMs to black-box, pro- viding robust security valuable intellectual property protection.
6.3 Inference Efficiency
Experiments conducted popular TEEs evaluate the inference efficiency of LoRO, including Intel SGX and ARM TrustZone. As depicted in Figure and Figure proposed LoRO introduces only 1.49x latency demonstrates that our methods provides real-time inference and is practical for real-world application. Notably, as shown in table with the raise of model scale, the computing complexity in TEE is significantly increased due to larger intermediate size, resulting higher latency.
average, compared the 10x 250.39x of existing methods. This
Answer to RQ2: Compared the averaged 112.10x of existing methods, proposed LoRO introduces only 1.49x latency, achieving real-time inference efficiency.
6.4 Model Accuracy
As shown in Table 3, we report the model accuracy of original model, proposed LoRO method and the obfuscated model exposed in REE. Only 0.01% drop is observed in BART on MRPC, which caused by float precision error. is demonstrated that the high accuracy of original model is maintained. Moreover, the obfuscated model is completely downgraded to random guessing level.
Answer to RQ3: Proposed LoRO introduces no accuracy loss, maintaining the high model accuracy. The obfuscated model exposed in REE is downgraded to unusable.

---

Table 3: Model accuracy of original model (Ori.), LoRO and obfuscated model in REE (Obf.).
SQuAD
MRPC
SST-2
MNLI
GSM8K
Spider
Ori.
81.84%
47.99%
93.58%
85.33%
RoBERTa
LoRO (t)
81.84%
87 57.99%
93.58%
85.33%
Obf.
0.64%
31.61%
50.91%
35.45%
Ori. 83.15%
87.01%
95.30%
88.15%
BART
LoRO (t)
83.15%
87 .0000%
95.30%
88.15%
Obf.
0.76%
31.60%
49.08%
35.45%
Ori.
48.45%
70.96%
17.99%
Qwen2
LoRO (f)
48.45%
70.96%
17.99%
Obf. (L)
0.00%
0.00%
0.00%
Ori.
49.20%
53. 13%
34.62%
LLaMA3
LoRO (1)
49.20%
53.12%
34.62%
Obf. (.)
0.00%
0.00%
0.00%
7
Discussion
Side channel Attacks. Side channel attacks [29 30 can cause leakage from TEE potentially, Fortunately, our LoRO is compatible with existing advanced defense schemes [33 mitigate such attacks. Hence, defending against side channel attacks is considered a sole research domain regarding TEE security, and is out of our scope.
Some advanced GPU provides Confidential Computing 55 56]. Nevertheless, the substantial cost of GPUs with confidential computing (CC) capabilities presents significant barrier to their widespread adoption in edge computing environments. More affordable alternatives, including Intel Software SGX or ARM TrustZone integrated with dedicated AI acceleration hardware, are expected to remain the predominant solution for edge devices in the foreseeable future. Consequently, our investigation into secure inference for edge computing systems does not currently incorporate Trusted GPUs within its scope.
Trusted GPU.
8 Conclusion
In this paper, we first identify a statistical vulnerability in existing TEE-based secure inference methods, and break their security by designed Model Stealing attack with Prior. To protect on-device ILMs,, LoRO is proposed, which incorporates two components: (1) Low Rank Mask to completely obfuscate model parameters, while achieving fast inference via lightweight low-rank computing in TEE. Moreover, no accuracy loss is introduced, and no retraining and model modification is required. (2) Factor Multiplexing to significantly reduce the secure memory requirement, avoiding additional latency introduced by secure memory paging. Proposed LoRO downgrade on-device LLMs black-box level (0.94x MS accuracy), while providing real-time inference service (only 1.49x latency, compared to the 112x of existing SOTA). LoRO can solve the concern regarding theft on untrusted edge devices, hence greatly facilitating the wide adoption of LLMs.
Acknowledgement
We would express our gratitude to all anonymous reviewers and chairs for their valuable dedica- tion. This research supported by the Natural Science Foundation of Hangzhou (under Grant 2025SZRJJ1400) and the CCF- PCF-Phytium Fund (under Grant CCF-Phytium 202306).
10

---

References
[1] Yupeng Chang, Xu Wang. Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):1 1:3):1-5,,20 2024.
[2] Zhenjie Yang. Xiaosong Jia, Hongyang Li, and Junchi Yan. LIm4drive: A survey of large language models for autonomous driving. arXiv preprint arXhv:2311.010433 2023.
[3] Yuanchun Li. Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, al. Personal Ilm agents: Insights and survey about the capability, efficiency and security. arXiv preprint
2024.
[4] Tianxiang Tan and Guohong Cao. Deep learning on mobile devices through neural process- ing units and edge computing. In IEEE INFOCOM 2022-IEEE Conference Computer Communications, pages 1209-1218. IEEE, 2022.
[5] Yu Sun, Jianhua Liu, Gaojian Xiong, Qinglin Song, Jianwei Liu, Gang Wang, and Rui Wang. Towards trusted 6g mobile edge computing: A secure batch large language models deployment framework. IEEE Transactions on Mobile Computing, 2025.
[6] Zhichuang Sun, Ruimin Sun, Long Lu, and Alan Mislove. Mind your weight (s): A large-scale study on insufficient machine learning model protection in mobile apps. In 30th USENIX security symposium (USENIX security 21), pages 1955-1972, 2021.
[7] Pengeheng Ren, Chaoshun Zuo, Xiaofeng Liu, Wenrui Diao, Qingchuan Zhao, and Shanqing Guo. Demistify: Identifying on-device machine learning models stealing and reuse vulnera- bilities in mobile apps. In Proceedings of the 46th IEEF/ACM International Conference on Software Engineering, pages 1-13, 2024.
[8] Yu Sun, Gaojian Xiong, Jianhua Liu, Zheng Liu, and Jian Cui. Tsqp: Safeguarding real-time inference for quantization neural networks on edge devices. In 2025 IEEE Symposium on Security and Privacy (SP), pages 1-1. IEEE Computer Society, 2024.
[9] Jiahui Hou, Huiqi Liu, Yunxin Liu, Yu Wang, Peng-Jun Wan, and Xiang- Yang Li. Model protection: Real-time preeev-ire inference service for model privacy at the edge. IEEE Transactions Dependable and Secure Computing, 12)6:427744484 2021.
[10] Tianxiang Shen, Ji Qi, Jianyu Jiang, Xian Wang, Siyuan Wen, Xusheng Chen, Shixiong Zhao, Sen Wang, Li Chen, Xiapu Luo, et al. {SOTER]: Guarding black-box inference for general neural networks at the edge. In 2022 USENIX Annual Technical Conference (USENIX ATC 22), pages 723-738, 2022.
[11] Zhichuang Sun. Ruimin Sun, Changming Liu, Amrita Roy Chowdhury, Long Lu, and Somesh Jha. Shadownet: secure and efficient on-device model inference system for convolutional neural networks. In 2023 IEEE Symposiun on Security and Privacy (SP), pages 1596-1612. IEEE, 2023.
[12] Tong Zhou, Yukui Luo, Shaolei Ren, and Xiaolin Xu. Nnsplitter: an active defense solution for dnn model via automated weight obfuscation. In International Conference on Machine Learning, pages 42614 42624. PMLR, 2023.
[13] Ziqi Zhang, Chen Gong, Yifeng Cai, Yuanyuan Yuan, Bingyan Liu, Ding Li, Yao Guo, and Xiangqun Chen. No privacy left outside: On the (in-) security of tee-shielded dnn partition for on-device ml. In 2024 IEEE Symposium Security and Privacy (SP), pages 3327-3344 IEEE, 2024.
[14] Qinfeng Li, Zhiqiang Shen, Zhenghan Qin, Yangfan Xie, Xuhong Zhang, Tianyu Du, Sheng Cheng, Xun Wang, and Jianwei Yin. Translinkguard: Safeguarding transformer models against model stealing in edge deployment. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 1479-34488 2024.

---

[15] Ran BaaaddBBcchrch,, Nathan Dowlin, Kim Laine, Kristin Lauter, Michael Nachrig, and John Wernsing. Cryptonets: Applying neural networks encrypted data with high throughput and accuracy. In International conference on machine learning, pages 201-210. PMLR, 2016.
[16] Deevashwer Rathee, Mayank Rathee, Nishant Kumar, Nishanth Chandran, Divya Gupta, Aseem Rastogi, and Rahul Sharma. Cryptflow:: Practical 2-party secure inference. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, pages 925-342, 2020.
[17] Victor Costan and Srinivas Devadas. Intel sgx explained. Crypiology ePrint Archive, 2016.
[18] Sandro Pinto and Nuno Santos. Demystifying arm trustzone: A comprehensive survey. ACM computing surveys (CSUR), 51(6): 1-36, 2019.
[19] Yuepeng Li, Deze Zeng, Lin Gu, Quan Chen, Song Guo, Albert Zomaya, and Minyi Guo. Efficient and secure deep learning inference in trusted processor enabled edge clouds. IEEE Transactions on Purallel and Distributed Systems, 33(12):4311443 2022.
[20] Mengda Yang, Wenzhe Yi, Juan Wang, Hongxin Hu, Xiaoyang Xu, and Ziang Li. Penetral- ium:
and memory-efficient neural network inference at the edge. Future Generation Computer Systems, 156:30-41, 2024.
[21] Yu Sun, Gaojian Xiong, Xiao Liu, and Yan Li. A survey on trusted execution environment based secure inference. Netinfo Security, 24(12):1799-18) 1818, 2024.
[22] Yu Sun, Gaojian Xiong, Xianxun Yao, Kailang Ma. and Jian Cui. Gi-pip: Do we require impractical auxiliary dataset for gradient inversion attacks? In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4675- 4679. IEEE, 2024.
[23] Lucjan Hanzlik, Yang Zhang, Kathrin Grosse, Ahmed Salem, Maximilian Augustin, Michael Backes, and Mario Fritz. Micapsule: Guarded offline deployment of machine learning as a service. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 3300-3309, 2021.
[24] Alexander Schlögl and Rainer Böhme. ennclave: Offline inference with model confidentiality. In Proceedings of the 13th ACM Workshop on Artificial Intelligence and Security, pages 93-104,
2020.
[25] Tarek Elgamal and Klara Nahrstedt. Serdab: An iot framework for partitioning neural networks computation across multiple enclaves. In 2020 20th IEEE/ACM International Symposium Cluster, Cloud and Internel Computing (CCGRID), pages 519-528. IEEE, 2020.
[26] Fan Mo, Ali Shahin Shamsabadi, Kleomenis Katevas, Soteris Demetriou, Ilias Leontiadis, Andrea Cavallaro, and Hamed Haddadi. Darknetz: towards model privacy at the edge using trusted execution environments. In Proceedings of the 18th International Conference on Mobile Systems, Applications, and Services, pages 161-174, 2020.
[27] Yecheng Xiang, Yidi Wang, Hyunjong Choi, Mohsen Karimi, and Hyoseung Kim. Aegisdnn: Dependable and timely execution of dnn tasks with sgx. 2021 IEEE Real-Time Systems Symposium (RTSS), pages 68 81. IEEF, 2021.
[28] Zheng Zhang, Na Wang, Ziqi Zhang, Yao Zhang, Tianyi Zhang, Jianwei Liu, and Wu. Group- cover: A secure, efficient and scalable inference framework for on-device model protection based on tees. In Forty-first International Conference on Machine Learning, 2024.
[29] Alexander Nilsson, Pegah Nikbakht Bideh, and Joakim Brorsson. A survey of published attacks on intel sgx. arXiv preprint arXiv:2006. / 13598, 2020.
[30] Jo Van Bulck, Frank Piessens, and Raoul Strackx. Nemesis: Studying microarchitectural timing leaks in rudimentary cpu interrupt logic. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pages 178-195, 2018.
12

---

[31] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
[32] Jingtao Li, Adnan Siraj Rakin, Yan Xiong, Liangliang Chang, Zhezhi He, Deliang Fan, and Chaitali Chakrabarti. Defending bit-flip attack through dnn weight reconstruction. In 2020 57th ACM/IEEE Design Automation Conference (DAC), pages 1-6, IEEE, 2020.
[33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[34] Ding Li, Ziqi Zhang, Mengyu Yao, Yifeng Cai, Yao Guo, and Xiangqun Chen. Teeslice: Protecting sensitive neural network models trusted execution environments when attackers have pre-trained models. ACM Transactions on Software Engineering and Methodology, 2024.
[35] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063. 2024.
[36] Wei Zheng, Ying Wu, Xiaoxue Wu, Chen Feng, Yulei Sui, Xiapu Luo, and Yajin Zhou. A survey of intel sgx and its applications. Frontiers of Computer Science, 15:1-15, 2021.
[37] Yunkai Bai, Peinan Li, Yubiao Huang, Shiwen Wang, Xingbin Wang, Dan Meng, and Rui Hou. Secpaging: Secure enclave paging with hardware-enforoed protection against controlled- channel attacks. In Proceedings of the 61st ACM//EEE Design Automation Conference, pages 1-6, 2024.
[38] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint (rXiv:1606.052550, 2016.
[39] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems, 2021. URL htlps://rxxiv org/abs/21 10.1116810 9, 2021.
[40] Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang. Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint (00vv:/809.00855
2018.
[41] Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 1804.0046 2018.
[42] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[43] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool, Food-101 101 - mining discriminative components with random forests. In European Conference on Computer Vision, 2014.
[44] Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacI-HH.T, volume 1, page 2. Minneapolis, Minnesota, 2019.
[45] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1977:1990 2019.
[46] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequenc pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.134611 2019.
[47] Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv: 2020.
[48] Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuan- jun L.v, Jinzheng He, Junyang Lin, Qwen2-audio technical report. arXiv preprint arXiv:2407.1075, 2024.
13

---

[49] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems, 2021. URL htpp::/arxxiv. org/abs/21 10.1110.114100 14168, 9, 2021.
[50] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama
3 herd of models. arXiv preprint arXiv:2407:21778, 2024.
[51] Linaro. Op-tee trusted os. https:://github.coomm//
Sateeeepptee_eem 2024.
[52] Chia-Che Tsai, Donald E Porter, and Mona Vij. {Graphene-SGXX: A practical library {OS} for unmodified applications on {SGX}. In 2017 USENIX Annual Technical Conference (USENIX ATC 17), pages 645-658, 2017.
[53] Ghada Dessouky, Tommaso Frassetto, and Ahmad-Reza Sadeghi. (HybCache): Hybrid (Side- Channel-Resilient) caches for trusted execution environments. In 29th USENIX Security Symposium (USENIX Security 20), pages 451 451-468, 2020.
[54] Sajin Sasy, Sergey Gorbunov, and Christopher W Fletcher. Zerotrace: Oblivious memory primitives from intel sgx. In 25th Annual Network and Distributed System Security Symposium, NDSS 2018. The Internet Society, 2018.
[55] Yongqin Wang, Rachit Rajat, Jonghyun Lee, Tingting Tang, and Murali Annavaram. Fastrack: Fast io for secure ml using gpu tees. arXiv preprint vrXiv:2410.152444 2024.
[56] Jianwei Zhu, Hang Yin, Peng Deng, Aline Almeida, and Shunfan Zhou. Confidential computing on nvidia hopper gpus: A performance benchmark study. arXiv preprint (XXi::4409003992 2024.
[57] Yugeng Liu, Rui Wen, Xinlei He, Ahmed Salem, Zhikun Zhang, Michael Backes, Emiliano De Cristofaro, Mario Fritz, and Yang Zhang. (ML-Doctor): Holistic risk assessment of inference attacks against machine learning models. In 3/st USENIX Security Symposium (USENIX Security 22), pages 4525 4542,, 2022.
[58] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
[59] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
[60] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. International conference on machine learming, pages 5156-5165. PMLR, 2020.
14

---

Table 4: Hlyee-ppaameeter of knockoff attacks.
RoBERTa
BART
Qwen2
I.aMA3
Epoch
Learning Rate
Epoch
Learning Rate
Epoch
Learning Rate
Epoch
Learning Rate
TLG
1
3e-5
2
3e-5
2
5e-5
2
5e-5
ShadowNet
2
le-5
3
le-5
4
le-5
4
le-5
SOTER
2
lc-5
lc-5
4
lc-5
4
lc-5
Magnitude
3
5e-5
le-5
5
3e-5
5
3e-5
Table 5: Detailed hyper-parameters of LoRO.
RoBERTa
BART
Qwen2
LLaMA3
Model Size
Base
Large
1.5B
3B
7B
1.5B
3B
8B
Rank
12
16
24
30
24
30
36
A
More Details
Attack details. We introduce attack details here. For each method, the adversary first reason the model structure from exposed model parts in REF, and get public pretrained model from Internet.
Then, the adversary conducts model de-obfuscation (as shown in Figure D.. In this stage, the batch size we leveraged to attack permutation methods is 16, and the inputs are randomly selected from datasets. Notably, empirically find the batch size can't 't be too large, since the averaged intermediate results would be very similar and difficult to matched. For additive noise, we iterate all training data in small dataset to accumulate the gradients. We freeze major parameters, and only 5% largest parameters are fine-tuned for 2 epoches. For multiplicative noise methods, our methods is as same as 13, and other details need to be clarified. The knockoff setting of each method shown in Table 4 The Shadow Net is considered as the combination of additive mask and multiplicative noise. We follow 13 to first remove the mask in REE, and then estimate the multiplicative key.
In the final knockoff stage, we develop our code based on the ML-Doctor [57]. Since model have been approximately recovered before, and only 10% of dataset are leveraged to conduct attacks, only train models with a low learning-rane as 5e-4 in 2 epoches to avoid over-fitting. The results reported in Table is taken from the best of five individual experiments.
LoRO details. The details and apper-parameters of LoRO is introduced here. All the random noise and low-rank factors randomly sampled from Gaussian Noise. We set the m 2 to achieve fast inference speed. According to LoRA [58], the rank d is able to effectively adapt the model to specific domain, thought the d is enough to obfuscate models well. The detailed hyper- prpee-pprameters of our experiments shown in T
Implementation optimization. For TrustZone, we mainly conduct two optimizations. First, take inspiration from ShadowNet (TT) we design matrix multiplication based on NEON instruction set accelerate computation in TEE. This achieves about 3x speed up when the dimension of matrix is over 768. Second, to reduce the data transfer time between REE and TEE, we allocate a pinned shared memory between them as data exchange zone. While this may raise inconvenience in coding, but the transfer latency is significantly reduced. Notably, it is necessary to compile the OP-TEE with larger secure memory (at least 128MB) since LLMs are significantly large. For methods like MLCapsule [33] the required memory could reach tens of GB.
Compatibility for small models. Our LoRO designed to obfuscate linear parameters (includ- ing both linear and convolutional layers). Previous work targeting small models (9] also focuses on protecting linear layers. Hence, proposed LoRO inherently compatible to small models, where the lincar operations major the parameters and inference computing.
10
11
13
27
15

---

Table 6: Latency breakdown of LoRO on Intel SGX.
RoBERTa-B
BART-L
Qwen2-3B
LLaMA3,2-38
Qwen2-7B
LLaMA3.2-8B
TEE 45.72%
47.20%
38.71%
29.92%
40.57%
37.82%
REE Transfer
37.82%
37.03%
55.62%
56.09%
43.93%
49.20%
16.45%
15.76%
15.66%
13.98%
15.50%
12.96%
Table 7: Accuracy of LoRO on MATH.
Table Accuracy of LoRO on ViT.
Model
Category
MATH
Food101
CIFAR100
Gemma 9B
Ori.
36.63%
Ori.
81.57%
89.86%
Gemma 9B
LoRO (1)
36.62%
VIT
LoRO (1)
81.57%
89.85%
Gemma 9B
Obf. (L)
0.00%
Obf. (4)
0.99%
1.00%
B More Experiments on LoRO
Latency breakdown. As presented in Table we analyze the latency distribution among TEE, REE, and data transfer. For smaller LLMs such as RoBERTa-B and BART-L, the computational efficiency is primarily limited by TEE processing and data transfer latency. However, in larger models like LLaMA and Qwen, the REE latency becomes dominant due to the high dimension full matrix computations. This clearly demonstrates the lightweight advantage achieved by LoRO.
Accuracy of LoRO on VIT. We also test the accuracy of LoRO on ViT, as shown in Table B Only small drop of 0.01 % is observed on CIFARIOO 100, which is caused by float error. This demonstrate the high accuracy of LoRO on vision tasks.
Accuracy of LoRO on MATH. LoRO is also evaluated on challenging dataset MATH [59] As shown in the Table LoRO maintains near-identical accuracy, with the 0.01% % drop attributable to floating-poin point error. This can be solved in 32-bit models. Crucially, the obfuscated model becomes unusable (0% accuracy), confirming our method's 's robustness to adversary.
Secure memory requirement. As shown in Table 9 we report the secure memory requirement of preserving secrets in TEE. Compared one-mask-per-layy paradigm, proposed Factor Multiplexing effectively reduce the secure memory to MB-level compatible to edge TEE. The high latency and potential risk introduced by secure memory page is hence avoided.
More latency comparison. As shown in Table 10 LoRO's 's latency outperforms most of existing methods. For comprehensive comparison,
only report the de-obfuscation latency of SOTER. While TLG performs similarly to LoRO in inference efficiency, TLG cannot provide black-box security (as demonstrated in Section We also notice that MLCapsule suffers critical slowdowns from the secure paging overhead, and Magnitude is slow since the sparse computation in TEE is too heavyweight for large scale LLMs.
C More experiments on MSP
MSP under low auxiliary data. In Section 3 about 10% of training data is assumed to be held by the adversary launch MSP attacks. The 10% data assumption allows design defenses robust against a powerful adversary.
We evaluate the attack performance under low data ratio (1%-10% of SQuAD Qwen2). As shown in Table [T] our LoRO consistently surpasses black-box performance even at 1% data volume, while existing SOTA TLG fails to provide adequate protection. Even under low data ratio conditions, the proposed MSP poses a serious threat to existing methods.
MSP on CV models. We also evaluate Model Stealing with Prior (MSP) experiments ViT and ResNet18. For model prior knowledge, we used model trained on CIFARTO 10. We did not adapt TLG to CNNs. As shown in the table 12 MSP remains effective for CV models like VIT and ResNet18.
16

---

Table 9: Secure memory requirement of Factor Multiplexing and naive one-mask-per-layee
RoBERTa
BART
Qwen-7B
LLaMA-8B
One-Mask-Per-Layee
232 MB
594 MB
1.02 GB
1.14 GB
Factor Multiplexing
3.84 MB
11.88 MB
26.30 MB
28.28 MB
Table 10: More experiments on latency comparison.
Model
MLCapsule
SOTER ShadowNet
Magnitude
TLG
LoRO
RoBERTa-B
54.08
1.34x
2.35
8.92
1.10x
1.12x
BART-L
95.17x
1.62x
2.48x
10.33
1.14x
1.18x
Qwen2-7B
321.58x
2.01x
3.60x
35.84x
1.59x
1.64x
LaMA3.2-8B
358.60x
2.10x
3.75x
37.67x
1.70x
1.77x
D Limitation of ОТР on Attention
We analyze why the OTP cannot be applied secure intermediates of Attention module here. During forward inference, Attention block requires to compute QK When OTP are applied to secure Q and K as Q' Q Og. K' K + og, the multiplication can be formulated follows:
QTkT= = + Qof + toKK" +0,o
(7)
The term 0,%% can be pre-computed and removed must be computed in a real-time manner. Since o, and Oμ shares the same shape of Q and K, applying ОTР will introduce unacceptable computation in TEE. Due to the limited computational resources in TEE, this will lead to prohibitive inference latency. This issue is also exists in protecting V.
lightweight manner, the term Qo K
The TOSEM version of TEESlice 34 suggests that this issue can be solved by using Linear Attention [60]. But the Linear Attention may have a bit influence model performance, and is still not widely adopted in popular LLMs. In this paper, we propose to use permutation to secure the intermediate results of Attention Block, while the remained parts should be protected by ОТP.
E Algorithm Description
For better comprchension, we present the persuade code of deployment stage in Algorithm ! and the inference stage in Algorithm 2
17

---

Table 11: MSP performance under low auxiliary data.
Dataset Ratio
1%
3%
5%
10%
Average
TLGG 14.20%
19.78%
24.95%
29.33%
2.92x
LoRO
4.08%
4.22%
4.38%
5.07%
0.63x
Black-Box
4.37%
6.95%
8.40%
11.20%
1.00x
Table 12: MSP performance on Computer Vision models.
Model
Dataset
TLG
SOTER
Magnitude
ShadowNet
Black-Box
ViT-Base
CIFAR 100
89.67%
88.30%
87.28%
79.27%
14.57%
ViT-Base
Food101
81.51 %
75.52%
80.05%
71.90%
10.28%
ResNet18
CIFAR100
N/A
67.30%
73.55%
65.42%
14.74%
Algorithm 1: LoRO Deployment Stage.
Input: Original Parameters
Output: Obfuscated Parameters W
Key, Secret, W + [::::
B, A Rondom oise();
> Randomly generate cornerstone low-rank factors
3 i. <--00 0:
len len(W):
5 while < len do
a, 1__index, A index = RandomIndex();
b Randomly choose factors Parameters obfuscation
W W + BBBBBBindddee A
Secret [| = B[B_indez(i]] A A[A_index((]], ar):
Keyli = Precomputee(ecret()))
Precompute de-obfuscation keys.
10 end
W to(RE));
Offlood obfuscated parameters REE > Protect secrets and keys in TEE
12 Secret.to(TEE);
(eyEt)(TEE);
return W":
Algorithm LoRO Inference Stage,
Input: Inference Input
Output: Inference Output y
Key, Secret 4- TEE_ Preparee)
2 W' + REE Preparee;
b Load Memory
i <- 0:
len = len(W):
while i < len do
GREE = Inference x.
Inference with obfuscated parameters REE
УTEE Low Rank Inference(Seccred)) x.oo(EE))))
D De-obfuscation in TEE > Result restore TEE
x yREE TEE) + yTEE:
if = len then
10
(REEE
Return result when finish
11
break:
end
13
next is scaled dot attention then permutation(z, Key(i):
Intermediate result protection
else 15
OTP(a, Keyli));
Intermediate result protection
end 17
18 end
return y:
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main contributions have been concluded in the final of Introduction (Section I).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Limitaions have been discussed in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
Our paper does not include any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have included all the required information.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification: Our code is available in http://gthbbccm///loooommm//// datasets are publicly available.
and all
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
public/guides httpee/eeeooeeeeeeeessssoooooon.
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https wwwgguddddeeeuuubbeeeee
//nips.co
ssiniolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: All details have been introduced in Section Section Appendix A
Section and
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Since our work is about model property protection, error bars are hard to reflect the best ability of adversaries. Instead, we report the highest results in five individual experiments, which is more meaningful.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have included the computing environments in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We follow the NeurIPS code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work focuses on protecting model property, and has no negative societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No data models released.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the existing assets in this work publicly available for research usage. The license can all be found in huggingface.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets are released.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing nor research with human subjects are involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No crowdsoureing nor research with human subjects are involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Focusing on LLM models protection, We have not involved LLMs as compo- nents in our methods.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
25


Reviews:
[{"review_id": "bA3KBFfBRb", "reviewer": "Reviewer_PcW2", "summary": "This paper proposes LoRO, which leverages dense mask to completely obfuscate parameters for TEE-based secure LLM. LoRO includes two innovations: (1) Low Rank Mask, which uses low-rank factors to generate dense masks efficiently. The computing complexity in TEE is hence reduced by an exponential amount to achieve inference speed up, while providing robust model confidentiality. (2) Factors Multiplexing, which reuses several cornerstone factors to generate masks for all layers. Experiments are conducted on multiple models and tasks to prove the effectiveness of LoRO.", "questions": "Please see the weakness above.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n+ (Novelty) This paper propose novel low rank obfuscation, which leverages low rank mask and factor multiplexing to generate dense masks efficiently.\n\n+ (Contribution) This paper targets an important research field and proposes promising solutions. \n\nWeaknesses:\n\nMy major concern of this paper is about the writing clarity. \n\n- Table 1 is not well illustrated. Please add reference to other works. How 'Average' is calculated? If the best performance is in green, some data points of black-box should be in free right, e.g., 10.28%? How model stealing accuracy is defined?\n\n- Lack of background on prior works. What are their mechanism? Which work is considered as SOTA before LoRO and why? Some terms are not defined before using, e.g., TLG.\n\n- Lack of clarification of the adversary. In line 122, adversary is defined as 'parties attempting to steal valuable LLMs'. However, what does steal LLMs refer to? Does it refer to recover the exact parameters stored in TEE? Or it refers to achieve a surrogate LLM which has similar performance? It is assumed that adversary can obtain 10% training data. How could this happen? If TEE is used for securing the model, a more general case is that the model is trained on private data. If the model is purely trained on public data, people are more willing to open source the model nowadays rather than use TEE.\n\n- The multiplicative noise method and corresponding attack is not referred or explained in the paper. However, they appear in Figure 2.\n \n- Typo in line 167: xW' is accelerated in REE -> x'W' is accelerated in REE. Line 169: sparse mask r_s -> sparse mask r\n\n- Should equation 3 be y = xW'-xD? if it is x'W', equation 3 will be y = x'W'-xD-o(W+D), similar to equation 2, right?", "was_revised": true, "final_justification": "All my questions have been addressed. I've raised my score accordingly.", "rebuttal": "Thank you for your valuable and insightful feedback. We sincerely appreciate your recognition of the novelty and contribution of our work. Due to space constraints in the initial submission, we were unable to include all details, leading to some clarity problems. We will carefully address all your comments. Below, we organize your questions and provide our responses point by point:\n\n>**Q1:** \n(1.1) Table 1 is not well illustrated. Please add reference to other works.\n(1.2) How 'Average' is calculated?\n(1.3) If the best performance is in green, some data points of black-box should be in free right, e.g., 10.28%?\n(1.4) How model stealing accuracy is defined?\n\n**R1:** \nThanks for your valuable comments.\n\n (1.1) We will **add correct reference** to the table, and **add citations** for each baseline method in Table 1 as follows: TransLinkGuard [13], ShadowNet [10], Magnitude [8], SOTER [9]. We will also **clarify in caption that \"No-Shield\" denotes scenarios without any defense mechanism, and \"Black-Box\" represents the hypothetical ideal security scenario** where all model parameters remain completely inaccessible to the adversary.\n\n (1.2) **The 'Average' value is calculated by taking the mean of the ratio between each method's Model Stealing (MS) accuracy and the Black-Box MS accuracy across all benckmarks.** We will provide a detailed explanation of this calculation in the caption.\n\n (1.3) We believe there might be a small typo (\"free\" likely intended as \"green\"). If we have misunderstood, please let us know. The \"Black-Box\" column represents the MS accuracy achieved under the assumption that the adversary has zero access to model parameters. **It is not a specific defense method but rather the security baseline we aim for. Performance comparisons are strictly between our method and the SOTA defense methods listed in (1.1).** This table design convention is also adopted in prior works [7][12][13].\n\n(1.4) Model Stealing (MS) accuracy refers to the test accuracy of the model obtained by the adversary using the \"Model Stealing with Prior\" attack proposed in this paper. We will explicitly define this term in the caption of Table 1.\n\n>**Q2:** Lack of background on prior works. \n(2.1) What are their mechanism? \n(2.2) Which work is considered as SOTA before LoRO and why? \n(2.3) Some terms are not defined before using, e.g., TLG.\n\n**R2:**\n Thanks for your insightful comments.\n\n (2.1) The core mechanisms of the baseline defense methods (TransLinkGuard, ShadowNet, Magnitude, SOTER) are introduced and analyzed in Section 4 ('Model Stealing Attack with Prior') and Appendix A under 'Risk Analysis', where we further discuss their vulnerabilities. **We will ensure this context is clearly linked to or reiterated in the Related Work section.**\n\n (2.2) Currently, **TransLinkGuard (TLG) is considered the SOTA defense specifically designed for LLM protection in TEE-based inference.** However, our analysis revealed security vulnerabilities in TLG. The other works (ShadowNet, Magnitude, SOTER) were originally designed for smaller models (e.g., CNNs). For a comprehensive comparison, we adapted and implemented these methods on LLMs in our experiments. We will add this clarification regarding the SOTA baselines in the experimental setup section.\n\n (2.3) We agree that \"TLG\" was not defined upon first use. TLG stands for TransLinkGuard. We will ensure this method name is clearly defined upon its first mention in the Related Work section.\n\n>**Q3:** Lack of clarification of the adversary. \n(3.1) In line 122, adversary is defined as 'parties attempting to steal valuable LLMs'. However, what does steal LLMs refer to? Does it refer to recover the exact parameters stored in TEE? Or it refers to achieve a surogate LLM which has similar performance? \n(3.2) It is assumed that adversary can obtain 10% training data. How could this happen? If TEE is used for securing the model, a more general case is that the model is trained on private data. If the model is purely trained on public data, people are more willing to open source the model nowadays rather than use TEE.\n\n**R3:**\nThanks for your valuable comments. \n\n (3.1) The adversary's goal is to **achieve a surrogate LLM that exhibits similar functionality and performance to the target model**, not necessarily to recover the exact parameters stored within the TEE. We acknowledge the lack of clarity and **will explicitly define this objective (functional equivalence via a surrogate model) in the Threat Model section**.\n\n (3.2) The assumption of the adversary possessing 10% of the training data is **adopted from established advanced TEE-based secure inference literature ([9,12,31,54]) and represents a widely considered, strong adversarial capability within this research community.** \n \n We assume the model is trained on private data, which necessitates protection via TEE. The 10% data assumption allows us to design defenses robust against a powerful adversary. **If the adversary possesses less than 10% data, the defense effectiveness of our proposed LoRO would be even stronger.**  \n \n To address your concern, we conduct the following experiments:\n\n **1. Evaluate attack performance under low-data ratio** (1%-10% of SQuAD on Qwen2). As shown below, our LoRO consistently surpasses black-box performance even at 1% data volume, while existing SOTA TLG fails to provide adequate protection. We will clarify the threat model and incorporate these results into the paper.\n\n| Dataset Ratio | TLG    | LoRO  | Black-Box |\n|:-------------:|:------:|:-----:|:---------:|\n| 1%            | 14.20% | 4.08% | 4.37%     |\n| 3%            | 19.78% | 4.22% | 6.95%     |\n| 5%            | 24.95% | 4.38% | 8.40%     |\n| 10%           | 29.33% | 5.07% | 11.20%    |\n| Average       | 2.92x  | 0.63x | 1.00x     |\n\n **2. Evaluate attack performance under data with distribution gap**. (10% of *half* the CIFAR100 classes) to conduct attacks, LoRO maintains security while TLG exhibits significant leakage:\n\n| Model    | Dataset  | Auxiliary Dataset          | TLG    | LoRO   | Black-Box |\n|----------|----------|----------------------------|--------|--------|-----------|\n| ViT-Base | CIFAR100 | 10% of Half classes of CIFAR100 | 61.96% | 13.50% | 10.28%    |\n| Average  | -        | -                          | 6.03x  | 1.31x  | 1.00x     |\n\n We will clarify the threat model and incorporate these comprehensive sensitivity analyses.\n\n>**Q4:** The multiplicative noise method and corresponding attack is not referred or explained in the paper. However, they appear in Figure 2.\n\n**R4:**\n Thank you for catching this omission. We apologize for the lack of explanation in the main text. Due to page limitations, **the details of attacks against the multiplicative noise defense method were moved to Appendix A ('Attack against Multiplicative Noise').** We will contain corresponding text into the main text.\n\n>**Q5:** Typo in line 167: xW' is accelerated in REE -> x'W' is accelerated in REE. Line 169: sparse mask r_s -> sparse mask r\n\n**R5:** \n Thank you very much for identifying these typos! We sincerely appreciate your careful reading and meticulous review. **We will carefully correct these errors (xW' to x'W' in line 167 and r_s to r in line 169)** and conduct a thorough proofreading to eliminate any other typos throughout the manuscript.\n\n>**Q6:** Should equation 3 be y = xW'-xD? if it is x'W', equation 3 will be y = x'W'-xD-o(W+D), similar to equation 2, right?\n\n**R6:**\n Thank you for this valuable observation. You are absolutely correct. **We will revise Equation 3 to $y = x'W' - xD - o(W + D)$, and revise Equation 4 to $y = x'W' - xBA - o(W + BA)$,** ensuring alignment with Equation 2. We appreciate you pointing this out and will check all equations for consistency and accuracy.\n\nFinally, we would like to express our gratitude again. We have benefited a lot from your professional feedback. We hope that all the writting details have been clarified, and look forward to your reply!"}, {"review_id": "GosMH56sSg", "reviewer": "Reviewer_eg2Y", "summary": "This paper focuses on the problem of model stealing attacks that can occur when running LLMs on untrusted devices such as edge devices. This paper first proposes an MSP attack, indicating that the existing protection method using TEE (Trusted Execution Environment) suffers from statistical vulnerabilities and may be exploited by the attacker. Therefore, this paper proposes a new method LoRO, which completely fuzzes the model parameters through the dense masking mechanism, including using low rank factors to generate dense masks, improving inference efficiency, and sharing low rank factors across multiple layers to reduce memory consumption.", "questions": "This paper is well-organized, and the writing and charts are very clear and easy to understand, and the Threat Model is also described clearly. LoRO is simple but effective, with low performance loss, and does not require model structure modification or retraining. Although this method increases the memory requirements of the device, this paper also proposes Factor Multiplexing to reduce the memory. But I still have some concerns:\n\n1. In the pipeline of MSP attack, the attacker used a small dataset for fine-tuning. Whether this part of the small dataset is a subset of figure1 (a) Private Dataset or a dataset that is completely another domain.\n2. In Figure 2, it is unclear what each point in the t-SNE visualization represents. Are they column vectors from the weight matrices of each model layer? I believe the authors should clarify this explicitly in the figure caption.\n3. Consider that Factor Multiplexing reduces storage pressure, it will introduce new computational or runtime costs. The paper should provide a more detailed analysis of this trade-off.\n4. In Table 3, LoRO shows almost no performance drop across several datasets, such as SQuAD and GSM8K. However, I wonder how well it would perform on more challenging benchmarks, for example, the MATH dataset [1].\n5. Since LoRO places key components inside the TEE, side-channel attacks are a very realistic threat in deployment. Any such leakage could directly expose the shared low-rank basis factors used for parameter obfuscation.\n\nOverall, I think this motivation is sufficient and the proposed method is also very effective.\n\nMinors:\n\n1. It should be clearly written in Table1's caption that this is the result of evaluation under the MSP attack.\n\nReference:\n\n1. Dan Hendrycks et al. “Measuring Mathematical Problem Solving With the MATH Dataset.” NeurIPS 2021.", "limitations": "Missing discussion of Side Channel Attacks", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n- Trendy topic\n- Well-organized paper\n- Easy to follow\n\nWeaknesses:\n\n- More clarifications are needed", "was_revised": true, "final_justification": "I will keep the original rating.", "rebuttal": "We sincerely appreciate your generous praise of our work and your invaluable expertise. Your insightful comments have greatly enhanced the quality of our paper. Below are our point-by-point responses:\n\n> **Q1:** In the pipeline of MSP attack, the attacker used a small dataset for fine-tuning. Whether this part of the small dataset is a subset of figure1 (a) Private Dataset or a dataset that is completely another domain.\n\n**R1:** \nThank you for raising this important point. The small dataset is a **subset of the private dataset** (Fig. 1a). This assumption aligns with established practices in advanced TEE-based secure inference research [7,9,12,13,31]. \n\nTo address your concern, we conduct the following experiments:\n\n**1. Evaluate attack performance under low-data ratio** (1%-10% of SQuAD on Qwen2). As shown below, our LoRO consistently surpasses black-box performance even at 1% data volume, while existing SOTA TLG fails to provide adequate protection. We will clarify the threat model and incorporate these results into the paper.\n\n| Dataset Ratio | TLG    | LoRO  | Black-Box |\n|:-------------:|:------:|:-----:|:---------:|\n| 1%            | 14.20% | 4.08% | 4.37%     |\n| 3%            | 19.78% | 4.22% | 6.95%     |\n| 5%            | 24.95% | 4.38% | 8.40%     |\n| 10%           | 29.33% | 5.07% | 11.20%    |\n| Average       | 2.92x  | 0.63x | 1.00x     |\n\n**2. Evaluate attack performance under data with distribution gap**. (10% of *half* the CIFAR100 classes) to conduct attacks, LoRO maintains security while TLG exhibits significant leakage:\n\n| Model    | Dataset  | Auxiliary Dataset          | TLG    | LoRO   | Black-Box |\n|----------|----------|----------------------------|--------|--------|-----------|\n| ViT-Base | CIFAR100 | 10% of Half classes of CIFAR100 | 61.96% | 13.50% | 10.28%    |\n| Average  | -        | -                          | 6.03x  | 1.31x  | 1.00x     |\n\nWe will clarify the threat model and incorporate these comprehensive sensitivity analyses.\n\n> **Q2:** In Figure 2, it is unclear what each point in the t-SNE visualization represents. Are they column vectors from the weight matrices of each model layer? I believe the authors should clarify this explicitly in the figure caption.\n\n**R2:** \nThank you for this valuable observation. **Each point in Fig. 2 represents t-SNE visualizations of column vectors from a weight matrix layer.** We will explicitly state this in the figure caption to improve clarity.\n\n> **Q3:** Consider that Factor Multiplexing reduces storage pressure, it will introduce new computational or runtime costs. The paper should provide a more detailed analysis of this trade-off.\n\n**R3:** \nWe appreciate your suggestion on analyzing trade-offs. As noted, Factor Multiplexing introduces minimal computational overhead:\n\nFor an $n \\times n$ parameter matrix, it adds only $\\mathcal{O}(n)$ operations – negligible compared to $\\mathcal{O}(n^3)$ matrix computations. **Our measurements show only <3% runtime increase.** This efficiency enables our method's practical deployment, and we will strengthen this analysis in Section 5.2.\n\n> **Q4:** In Table 3, LoRO shows almost no performance drop across several datasets, such as SQuAD and GSM8K. However, I wonder how well it would perform on more challenging benchmarks, for example, the MATH dataset [1].\n\n**R4:** \nThank you for suggesting this valuable evaluation. **On the challenging and well-designed MATH dataset, we evalute the accuracy of LoRO on Gemma 9B (4-shot)**:\n\n| Model    | Category       | MATH Accuracy |\n|:--------:|:--------------:|:-------------:|\n| Gemma 9B | Original       | 36.63%        |\n| Gemma 9B | LoRO (↑)       | 36.62%        |\n| Gemma 9B | Obfuscated (↓) | 0.00%         |\n\nAs shown in the Table above, **LoRO maintains near-identical accuracy** (36.62% vs. 36.63%), **with the 0.01% drop attributable to floating-point error** (this can be solved in 32-bit models). Crucially, the obfuscated model becomes unusable (0% accuracy), confirming our method's robustness to adversary.\n\nWe will include this benchmark into our manuscript.\n\n> **Q5:** Since LoRO places key components inside the TEE, side-channel attacks are a very realistic threat in deployment. Any such leakage could directly expose the shared low-rank basis factors used for parameter obfuscation.\n\n**R5:** \nWe deeply appreciate this security-focused feedback. **We will discuss on side channel attacks as follows:**\n\nSide channel attacks [26,27] can cause leakage from TEE pontentially. Fortunately, our LoRO is compatible with existing advanced defense schemes [50,51] to mitigate such attacks. Hence, defending against side channel attacks is considered a sole research domain regarding TEE security.\n\nThis is included in the Appendix B in our initial submission.\n\n**Notably, our threat model follows TEE-based inference literature [7,9,10,12,13,31] where TEE integrity is assumed.** We will consider to defend against side channel attacks in the furture.\n\nOnce again, we are truly grateful for your generous praise and constructive feedback, which have significantly strengthened our work. Should you have further questions, we would be honored to provide additional clarifications."}, {"review_id": "tWitDkbUvj", "reviewer": "Reviewer_Wwxk", "summary": "This paper presents a secure inference framework for LLMs using TEE. It employs additive masking and a one-time pad for matrix obfuscation, incorporates a permutation matrix to protect scaled dot-product operations, and applies factor multiplexing to optimize memory usage. Experimental results show strong resistance to model-stealing attacks and reduced latency compared to existing approaches.", "questions": "1. How is the MSP attack implemented for LoRO?\n2. How is ShadowNet’s obfuscation broken?\n3. Since LoRO uses the same additive masking technique described in Section 4.2, does it suffer from the same vulnerabilities?\n4. How does LoRO ensure the correctness of outsourced computation?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Analyzes the risks of current methods and proposes a new attack to demonstrate them.\n- Evaluation and visualization effectively demonstrate both security and efficiency.\n- Provides a thorough analysis and comprehensive summary of existing work.\n\nWeaknesses:\n- The MSP attack requires extensive prior knowledge. While this assumption is reasonable for LLMs—since most are open-source and fine-tuned from publicly available models—it may not hold for small models and ViTs. These models often lack such vulnerabilities, making the success of the MSP attack on them questionable.\n\n- The proposed LoRO appears to be a combination of several existing techniques: the low-rank mask resembles the use of Freivalds’ algorithm in Slalom [a], the permutation matrix is directly adopted from TransLinkGuard [13], and the factor multiplexing is similar to the cornerstone fingerprints in SOTER [9].\n\n- LoRO lacks result verification for outsourced computation, a feature provided by methods like TEESlice [12, 31] and SOTER [9].\n\n- Some attack details are missing. In Section 4, the paper outlines the attack pipeline for three types of obfuscation methods but lacks specific details on how ShadowNet’s obfuscation is broken—particularly since it employs both multiplicative and additive masking on weights. The explanation in Appendix A is also unclear. Additionally, it is not sufficiently explained how the proposed LoRO is attacked. Since the attacker cannot recover the exact weights, is the attack conducted using techniques from TransLinkGuard [13]? \n- Some evaluations are missing. In Section 6.3, the paper compares efficiency with a few full shielding methods and SOTER. While comparing with full shielding approaches is reasonable, since they offer similar black-box security and the performance gains of LoRO are somewhat expected. The evaluation only includes SOTER, overlooking other methods listed in Table 1. Additionally, it does not compare with TEESlice, which is also applicable to LLMs [31].\n\n[a] Tramer, Florian, and Dan Boneh. \"Slalom: Fast, verifiable and private execution of neural networks in trusted hardware.\" arXiv preprint arXiv:1806.03287 (2018).\n\nSome Mistakes and Suggestions:\n- In Equations 3 and 4, the variable should be x instead of x', as x' is not introduced until line 223.\n- Missing citations. In Tables 1 and 2, it would be better to include citations for the baseline methods directly in the tables to improve readability, especially for newcomers to the field. Some abbreviations, such as TLG for TransLinkGuard, differ from the original paper titles. While defining such abbreviations is acceptable, proper citations are still necessary.\n- Incorrect highlighting. In Table 1, it states “the worst is in red,” but several red-highlighted values are not the highest. For example, in the RoBERTa model on the SST-2 dataset, SOTER’s 93.20% is higher than the red-highlighted 91.42%. Similarly, for the BART model: on the MRPC dataset, SOTER’s 82.11% is higher than the red 79.96%; on the SST-2 dataset, SOTER’s 94.57% is higher than the red 91.05%.\n- Incorrect table references. In lines 44, 186, 279, and 500, “Table 1” is incorrectly linked to Section 1 instead of jumping to the actual table. Additionally, in lines 282 and 519, “Table C” is mentioned, which appears to be a mistake—it likely refers to “Table 6”, but is incorrectly linked to Table 5.", "was_revised": true, "final_justification": "The author rebuttal fully addressed my concerns:\n\n1. Regarding novelty, the authors clearly articulate the differences between the proposed LoRO and previous works, demonstrating its novelty and effectiveness.\n2. For baselines, the authors provide additional experiments showing its superior efficiency compared to prior methods.\n3. On verification for outsourced computation, the authors conducted further evaluations.\n\nWith all concerns addressed, I raise my score to Accept.", "rebuttal": "We sincerely thank you for your insightful feedback and constructive critiques. Your appreciation for our theoretical analysis is deeply encouraging. Below are our point-by-point responses to your concerns (some weaknesses are addressed alongside related questions):\n\n> **W1:** The MSP attack requires extensive prior knowledge. While this assumption is reasonable for LLMs—since most are open-source and fine-tuned from publicly available models—it may not hold for small models and ViTs. These models often lack such vulnerabilities, making the success of the MSP attack on them questionable.\n\n**R1:** Thank you for this valuable observation. We have now supplemented our evaluation with **Model Stealing with Prior (MSP) experiments on ViT and ResNet18 (small model)**. For model prior knowledge, we used model trained on CIFAR10. Due to time constraints during rebuttal response, we did not find a perfect way to adapt TLG to CNNs.\n\n| Model| Dataset  | TLG    | SOTER  | Magnitude | ShadowNet | Black-Box |\n|-|----------|--------|--------|-----------|-----------|-----------|\n| ViT-Base | CIFAR100 | 89.67% | 88.30% | 87.28%    | 79.27%    | 14.57%    |\n| ViT-Base | Food101  | 81.51% | 75.52% | 80.05%    | 71.90%    | 10.28%    |\n| ResNet18 | CIFAR100 | N/A    | 67.30% | 73.55%    | 65.42%    | 14.74%    |\n\nAs shown in the table above, **MSP remains effective for small models like ViT and ResNet18**. This is not surprising, since prior work [7,12] have attacked small models successfully. Our contribution lies in **extending this attack to more challenging LLMs via rigorous statistical analysis**.\n\n> **W2:** The proposed LoRO appears to be a combination of several existing techniques: the low-rank mask resembles the use of Freivalds’ algorithm in Slalom [a], the permutation matrix is directly adopted from TransLinkGuard [13], and the factor multiplexing is similar to the cornerstone fingerprints in SOTER [9].\n\n**R2:** We appreciate your careful analysis. While surface similarities exist, LoRO’s core mechanisms differ fundamentally:  \n\n1. **Freivalds vs. Our Low-Rank Mask**. As shown in the Table below, the objective and mechnism of two methdos are different. Actually, we are inspired of Parameter-Efficient-Fine-Tune(PEFT) [b] to develop the idea of low-rank mask.\n   \n| Technique | Objective| Mechnism |\n|-|-|-|\n| Freivalds' algorithm | Verify the integrity of outsourced computation |$P (A\\times B \\times r=C\\times r \\| A\\times B \\neq C) < \\epsilon$, where $\\epsilon$ is very small. Hence verify the result $C=A\\times B ?$ efficiently. |\n| Low-Rank Mask| Obfuscate LLM parameters | $W'=W+BA$ to obfuscate $W$|\n\n2. **TLG Permutation vs. Our Permutation**. TransLinkGuard [13] uses permutation to obfuscate parameters as $W'=\\pi^T W$. As analyzed in Section 4.1, it is vulnerable to intermediate-result analysis, which can cause leakage of $\\pi$ and then de-obfuscate $W'$. In contrast, our permutation **only protects Attention Block intermediate results as $Q=\\pi^T Q$** (similar to OTP but adapted for transformers). Crucially, **LoRo secures parameters via Low-Rank Mask as $W'=W+BA$, preventing recovery even if $\\pi$ are compromised.**\n\n3. **SOTER Fingerprints vs. Factor Multiplexing**. While both use linear combinations, their purposes and mechnism diverge:  \n   - SOTER/TSQP [7,9]: Generate challenge-response pairs for *computation integrity*, based on the associativity.\n   - Our approach: Minimize secure memory overhead for **Low-Rank Mask storage**, and is not related to associativity.\n\n**In conclusion, the shared mathematical concept serves for distinct security objectives, and is different in mechnism.**\n\n> **W3 & Q4:** (W3) LoRO lacks result verification for outsourced computation, a feature provided by methods like TEESlice [12, 31] and SOTER [9]. (Q4) How does LoRO ensure the correctness of outsourced computation?\n\n**R3:** Thank you for highlighting this. While LoRO focuses on TEE-based obfuscation, it **fully supports two established integrity methods: Challenge-Response**[7,9] and **Freivalds' algorithm[a,12,31]**:  \n\nTo solve your concern, **we evaluate the Challenge-Response Method [7,9] on LoRO**. Experimental results show that LoRO achieves **100% fault injection detection accuracy under INT-8, with no false positive.** Under FLOAT-32, we observe only one false positive, which can be resolvable via error-threshold tuning.\n\n\n> **W4 & Q1 & Q2:**  (W4) Some attack details are missing. In Section 4, the paper outlines the attack pipeline for three types of obfuscation methods but lacks specific details on how ShadowNet’s obfuscation is broken—particularly since it employs both multiplicative and additive masking on weights. The explanation in Appendix A is also unclear. Additionally, it is not sufficiently explained how the proposed LoRO is attacked. Since the attacker cannot recover the exact weights, is the attack conducted using techniques from TransLinkGuard [13]? (Q1) How is the MSP attack implemented for LoRO? (Q2) How is ShadowNet’s obfuscation broken?\n\n**R4:** We appreciate this request for clarification and will introduce them as follows: \n\n1. **Attack on ShadowNet.** ShadowNet’s obfuscation: $W'=W\\times \\lambda+F$. Since $F$ is dense ($F$ of Magnitude is sparse), the introduced computation in TEE is costly, so it’s offloaded to REE. Following the TEESlice [12, 31], we first remove the mask $F$, and then estimate $\\lambda$ via variance analysis (Appendix A). Finally,  we fine-tune the model with a small dataset.\n\n2. **Attack on LoRO.** Since LoRO exhibit no real parameters, we evaluate three attack strategies:  \n   - (1) **LoRA-like fine-tune**: Initialize parameters using obfuscated weights, and optimize $BA$ through small datasets to achieve model stealing [b].\n   - (2) **REE-parameter initialization**: We directly use REE parameters to initialize model and conduct fully finetune.\n   - (3) **Black-box**: Standard black-box Knockoff attack with random initialization.\n  \n   Experimental results show that **(1) outperforms (2)**, so we report the result of (1) and (3) in Table 1.\n\n>**W5:** Some evaluations are missing. In Section 6.3, the paper compares efficiency with a few full shielding methods and SOTER. While comparing with full shielding approaches is reasonable, since they offer similar black-box security and the performance gains of LoRO are somewhat expected. The evaluation only includes SOTER, overlooking other methods listed in Table 1. Additionally, it does not compare with TEESlice, which is also applicable to LLMs [31].\n\n**R5:** Thank you for this suggestion. We now provide comprehensive efficiency comparisons and discuss TEESlice:  \n\n**Efficiency Comparison.** Due to code availability, initial submission only compared with SOTER. We have now implemented other baselines on SGX. As shown in the Table below, LoRO’s latency is comparable to TLG, while TLG cannot provide black-box security. Since OTP is not applicable, we remove it from attention block protection in magnitude and shadownet. We also notice that other methods suffer critical slowdowns since:  \n  - MLCapsule: Secure paging overhead  \n  - SOTER: Activation computation \n  - Magnitude: Sparse computation in TEE is too heavyweight for large scale LLMs\n\n|Model|MLCapsule|SOTER|ShadowNet|Magnitude|TLG|LoRO\n|-|-|-|-|-|-|-|\n|RoBERTa-B|54.08x|1.34x|2.35x|8.92x|1.10x|1.12x|\n|BART-L|95.17x|1.62x|2.48x|10.33x|1.14x|1.18x|\n|Qwen2-7B|321.58x|2.01x|3.60x|35.84x|1.59x|1.64x|\n| LaMA3.2-8B|358.60x|2.10x|3.75x|37.67x|1.70x|1.77x|\n\n\n**Discussion on TEESlice.** While TEESlice [12] pioneered model stealing defenses on small models, its journal version [31] requires **replacing transformer attention with linear attention** for OTP compatibility. This introduces accuracy drops and expensive retraining costs, making it **incompatible with mainstream LLMs (e.g., LLaMA, Qwen)**. So it's **difficult for us to compare with TEESlice on mainstream LLMs**. In contrast, LoRO preserves model architecture and offers broader applicability.\n\n>**W6:** Typos and writting mistakes.\n\n**R6**. We sincerely appreciate your meticulous proofreading. We have corrected:  \n1. Eq. 3 revised to $y = x'W' - xD - o(W + D)$ (with $x'$ defined in text)  \n2. Eq. 4 revised to $y = x'W' - xBA - o(W + BA)$  \n3. Added citations in Table 1: TLG [13], ShadowNet [10], Magnitude [8], SOTER [9]  \n4. Revised Table 1 highlights  \n5. Fixed references to Table 1/Appendix C  \n\nWe are conducting additional rounds of proofreading to eliminate residual errors.\n\n> **Q3** Since LoRO uses the same additive masking technique described in Section 4.2, does it suffer from the same vulnerabilities?\n\n**R7**: Thank you for this important question. We will analyze the security in detail:\n\nThe additive mask: $W'=W+D$ itself is considered secure，as the OTP introduced by Slalom [a] is stil secure nowadays. But de-obfuscate the results in TEE requires to compute $xD$, which is equivalent to full inference in TEE. To avoid the high latency, existing methods introduce:\n\n- Magnitude: sparsifies $D$, which introduces gradient anomaly.\n- ShadowNet: offload $D$ to REE for acceleration.\n\nProposed LoRO dose not expose $D$ to REE, and apply dense mask on all parameters. Hence, LoRO dose not suffer from the same vulnerabilities.\n\n> We are profoundly grateful for your time and expertise. Your feedback has significantly strengthened our work. All revisions above will be incorporated into the manuscript. Should any concerns remain, we would be honored to address them.\n\nReference\n\n[a] Tramer, Florian, and Dan Boneh. \"Slalom: Fast, verifiable and private execution of neural networks in trusted hardware.\" arXiv preprint arXiv:1806.03287 (2018).\n\n[b] Hu E J, Shen Y, Wallis P, et al. Lora: Low-rank adaptation of large language models[J]. ICLR, 2022, 1(2): 3."}, {"review_id": "y1tcNolD7l", "reviewer": "Reviewer_tpNN", "summary": "This paper focuses on the risk of model theft faced by large language models when deployed on untrustworthy edge devices. To address this growing problem, the authors propose a novel secure reasoning framework, LoRO, which is built on top of the Trusted Execution Environment (TEE) and aims to provide a language model reasoning solution for edge devices that combines real-time, high-precision, and strong protection of model intellectual property.", "questions": "Please refer to the three issues outlined in the Weaknesses section.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The problem addressed in the paper - deploying LLM safely and efficiently on edge devices - is one of the core challenges facing the AI field today. It has important practical significance and application prospects.\n- The author demonstrates excellent logical organization and clarity of expression in writing the paper. The article is well structured and able to lead the reader to a smooth understanding of the research motivation and technical realization of the paper.\n\nWeaknesses\n- The effectiveness of MSP attacks largely hinges on the statistical similarity between publicly available pretrained models and the target private model. However, the degree of this similarity—and its consistency across different model architectures, tasks, and fine-tuning strategies—can vary significantly. A more in-depth discussion on the origins and extent of this similarity would strengthen the paper.\n- While the low-rank design offers clear benefits in terms of efficiency, it also raises a critical concern: could this architectural choice inadvertently open up vulnerabilities that are specific to low-rank structures?\n- In the evaluation setup of MSP attacks, the small dataset available to the attacker—reportedly around 10% of the original training set—serves as a critical factor influencing the attack's effectiveness. However, the paper lacks a sensitivity analysis on how the performance of the attack varies with respect to the size, quality, and relevance of this dataset to the target model’s data.", "was_revised": true, "final_justification": "Thank you for helping me resolve the issue — I have revised the score to Accept.", "rebuttal": "We sincerely appreciate your recognition of our work and your positive remarks regarding our writing organization. Your insightful feedback is highly valuable to us. We now address your concerns point-by-point below:\n\n>**Q1:** The effectiveness of MSP attacks largely hinges on the statistical similarity between publicly available pretrained models and the target private model. However, the degree of this similarity—and its consistency across different model architectures, tasks, and fine-tuning strategies—can vary significantly. A more in-depth discussion on the origins and extent of this similarity would strengthen the paper.\n\n**R1:** We deeply appreciate this valuable perspective. While the concept of leveraging statistical similarity for attacks **was initially proposed in TEESlice [12]**, that work focused exclusively on small CV models and **did not extend attacks to LLMs**. Modern LLMs architectures (typically Transformer variants) exhibit well-documented high similarity in both parameter and representation spaces [7, Ref 1]. This phenomenon is particularly pronounced in proprietary domain fine-tuned models [Ref 2]. **Based on these observations, this work represents the first successful extension of such attacks to LLMs, revealing critical security vulnerabilities in existing TEE-based secure inference.**\n\nNotably，our MSP attack only requires parameter distribution to estimate TEE-shielded model secrets. To address your concern directly, we intentionally attacked RoBERTa using BERT as the pretrained model (RoBERTa's training data significantly exceeds BERT's, with typical parameter similarity around 40%). As shown in the table below, SOTA TLG remains vulnerable to MSP attacks, while our proposed LoRO achieves performance comparable to black-box settings:\n\n| Model   | Dataset | TLG    | LoRO   | Black-Box |\n|:-------:|:-------:|:------:|:------:|:---------:|\n| BERT    | MRPC    | 67.01% | 31.61% | 33.90%    |\n| BERT | SST-2   | 88.35% | 73.43% | 69.45%    |\n| BERT | MNLI    | 81.74% | 45.18% | 44.02%    |\n| Average | - | 1.70x  | 1.01x  | 1.00x |\n\nWe will incorporate this analysis and experimental validation into the paper.\n\n>**Q2:** While the low-rank design offers clear benefits in terms of efficiency, it also raises a critical concern: could this architectural choice inadvertently open up vulnerabilities that are specific to low-rank structures?\n\n**R2:** Thank you for highlighting this important consideration. LoRO's security is ensured through two complementary mechanisms:\n\n1. **Parameter Security**: Our model obfuscation follows $W'=W+D$, where the TEE-shielded mask $D=BA$ is additive noise. The adversary only observes $W'$ on the REE side. Recovering the additive factors from $W'$ alone is extremely challenging—**equivalent to solving the cocktail party problem [Ref 3] with minimal priors**. This additive protection underpins Slalom's security [Ref 4], which remains unbroken to date. \n   \n2.  **Statistical Security**: Unlike existing methods whose sparse $D$ creates detectable gradient anomoly, our dense $D$ matrix applies random obfuscation across all parameters. This prevents adversaries from extracting meaningful information about the secret $BA$ through gradient analysis.\n\nBoth theoretical analysis and empirical results confirm LoRO's robustness. We will expand this security discussion in the paper.\n\n>**Q3:** In the evaluation setup of MSP attacks, the small dataset available to the attacker—reportedly around 10% of the original training set—serves as a critical factor influencing the attack's effectiveness. However, the paper lacks a sensitivity analysis on how the performance of the attack varies with respect to the size, quality, and relevance of this dataset to the target model’s data.\n\n**R3:** We appreciate this thoughtful suggestion and acknowledge the importance of dataset sensitivity. \n\nThe assumption of the adversary possessing 10% of the training data is **adopted from established advanced TEE-based secure inference literature ([9,12,31,54]) and represents a widely considered, strong adversarial capability within this research community.** \n\nTo address your concern, we conduct the following experiments:\n\n**1. Evaluate attack performance under low-data ratio** (1%-10% of SQuAD on Qwen2). As shown below, our LoRO consistently surpasses black-box performance even at 1% data volume, while existing SOTA TLG fails to provide adequate protection. We will clarify the threat model and incorporate these results into the paper.\n\n| Dataset Ratio | TLG    | LoRO  | Black-Box |\n|:-------------:|:------:|:-----:|:---------:|\n| 1%            | 14.20% | 4.08% | 4.37%     |\n| 3%            | 19.78% | 4.22% | 6.95%     |\n| 5%            | 24.95% | 4.38% | 8.40%     |\n| 10%           | 29.33% | 5.07% | 11.20%    |\n| Average       | 2.92x  | 0.63x | 1.00x     |\n\n**2. Evaluate attack performance under data with distribution gap**. (10% of *half* the CIFAR100 classes) to conduct attacks, LoRO maintains security while TLG exhibits significant leakage:\n\n| Model    | Dataset  | Auxiliary Dataset          | TLG    | LoRO   | Black-Box |\n|----------|----------|----------------------------|--------|--------|-----------|\n| ViT-Base | CIFAR100 | 10% of Half classes of CIFAR100 | 61.96% | 13.50% | 10.28%    |\n| Average  | -        | -                          | 6.03x  | 1.31x  | 1.00x     |\n\nWe will clarify the threat model and incorporate these comprehensive sensitivity analyses.\n\nWe are profoundly grateful for your invaluable insights, which have significantly strengthened our work. Should any remaining concerns arise, we would be grateful for the opportunity to address them.  \n\n**Paper-external References:**  \n\n[Ref 1] \nYuantao Zhang et.al, A Perplexity and Menger Curvature-Based Approach for Similarity Evaluation of Large Language Models, 2025\n\n[Ref 2] \nMax Klabunde et.al, Towards Measuring Representational Similarity of Large Language Models, 2023\n\n[Ref 3] \nBarry Arons, A Review of The Cocktail Party Effect, 1992\n\n[Ref 4] \t\nSlalom: Fast, verifiable and private execution of neural networks in trusted hardware, 2019, ICLR"}]

Author rebuttal:

Meta-review: (a) Summary of the Paper:
The paper addresses the concern of model theft on untrusted edge devices, which is a key barrier to the wider adoption of LLMs. Unlike existing TEE-based secure inference methods that generally suffer from statistical vulnerabilities, the authors propose LoRO, a method designed to protect on-device LLMs. LoRO incorporates two main components:
  (i) Low-Rank Masking – completely obfuscates model parameters while enabling faster inference through low-rank computation in TEE. The approach introduces no accuracy loss and requires neither retraining nor model modification.
  (ii) Factor Multiplexing – mitigates additional latency caused by secure memory paging by reducing the secure memory footprint.
With these techniques, LoRO effectively downgrades on-device LLMs to black-box level (achieving 0.94× MS accuracy) while still delivering real-time inference performance (1.49× latency, compared to 112× in existing SOTA methods).

(b) Strengths:
The reviewers highlighted several strengths of the paper:
   -Novelty of the proposed approach (Reviewer PcW2).
   -Logical flow and clarity of the presentation (Reviewer eg2Y).
   -Strong theoretical analysis and comprehensive literature summary (Reviewer Wwxk).
  -Clear and logical organization of the paper (Reviewer tpNN).

(c) Weaknesses:
The main concerns raised by reviewers include:
-Limited experiments on challenging datasets.
-The need for additional technical clarity in some parts.
-Require additional experiments under critical or adversarial settings.
- One of the reviewer also raised a few issues related to the writing. 

(d) Justification for Decision:
Three reviewers recommended acceptance and one suggested borderline accept. The paper address a timely problem of secure and efficient on-device LLM inference.  The proposed method demonstrates both novelty and practicality, with theoretical and empirical evidence. Overall, the strengths of the  paper clearly outweigh the weaknesses, and I recommend acceptance.

(e) Rebuttal and Discussion Summary:
During the rebuttal, the authors addressed the concerns raised by the reviewers with additional analysis and experiments. All four reviewers agreed that their concerns had been adequately addressed and indicated a preference for acceptance. The Area Chair concurs with the reviewers’ assessment.

The authors are expected to incorporate all additional experiments and clarifications provided during the rebuttal into the revised manuscript, as promised. They should also fix the minor writing issues noted by reviewers and open-source the code as stated in the manuscript.
Decision: Accept (poster)