Title: LoRO: Real-Time on-Device Secure Inference for LLMs via TEE-Based Low Rank Obfuscation
Venue: NeurIPS 2025
Accepted: yes
Keywords: secure inference

large language models

trusted execution environments

intellectual property protection
Abstract: While Large Language Models (LLMs) have gained remarkable success, they are consistently at risk of being stolen when deployed on untrusted edge devices. As a solution, TEE-based secure inference has been proposed to protect valuable model property. However, we identify a statistical vulnerability in existing protection methods, and furtherly compromise their security guarantees by proposed Model Stealing Attack with Prior. To eliminate this vulnerability, LoRO is presented in this paper, which leverages dense mask to completely obfuscate parameters. LoRO includes two innovations: (1) Low Rank Mask, which uses low-rank factors to generate dense masks efficiently. The computing complexity in TEE is hence reduced by an exponential amount to achieve inference speed up, while providing robust model confidentiality. (2) Factors Multiplexing, which reuses several cornerstone factors to generate masks for all layers. Compared to one-mask-per-layer, the secure memory requirement is reduced from GB-level to tens of MB, hence avoiding the hundred-fold latency introduced by secure memory paging. Experimental results indicate that LoRO achieve a $0.94\times$ Model Stealing (MS) accuracy, while SOTA methods presents $3.37\times$ at least. The averaged inference latency of LoRO is only $1.49\times$, compared to the $112\times$ of TEE-shielded inference. Moreover, LoRO results no accuracy loss, and requires no re-training and structure modification. LoRO can solve the concerns regarding model thefts on edge devices in an efficient and secure manner, facilitating the wide edge application of LLMs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the risk of model theft faced by large language models when deployed on untrustworthy edge devices. To address this growing problem, the authors propose a novel secure reasoning framework, LoRO, which is built on top of the Trusted Execution Environment (TEE) and aims to provide a language model reasoning solution for edge devices that combines real-time, high-precision, and strong protection of model intellectual property.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The problem addressed in the paper - deploying LLM safely and efficiently on edge devices - is one of the core challenges facing the AI field today. It has important practical significance and application prospects.\n- The author demonstrates excellent logical organization and clarity of expression in writing the paper. The article is well structured and able to lead the reader to a smooth understanding of the research motivation and technical realization of the paper.\n\nWeaknesses\n- The effectiveness of MSP attacks largely hinges on the statistical similarity between publicly available pretrained models and the target private model. However, the degree of this similarity—and its consistency across different model architectures, tasks, and fine-tuning strategies—can vary significantly. A more in-depth discussion on the origins and extent of this similarity would strengthen the paper.\n- While the low-rank design offers clear benefits in terms of efficiency, it also raises a critical concern: could this architectural choice inadvertently open up vulnerabilities that are specific to low-rank structures?\n- In the evaluation setup of MSP attacks, the small dataset available to the attacker—reportedly around 10% of the original training set—serves as a critical factor influencing the attack's effectiveness. However, the paper lacks a sensitivity analysis on how the performance of the attack varies with respect to the size, quality, and relevance of this dataset to the target model’s data.\n\nQUESTIONS:\nPlease refer to the three issues outlined in the Weaknesses section.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a secure inference framework for LLMs using TEE. It employs additive masking and a one-time pad for matrix obfuscation, incorporates a permutation matrix to protect scaled dot-product operations, and applies factor multiplexing to optimize memory usage. Experimental results show strong resistance to model-stealing attacks and reduced latency compared to existing approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Analyzes the risks of current methods and proposes a new attack to demonstrate them.\n- Evaluation and visualization effectively demonstrate both security and efficiency.\n- Provides a thorough analysis and comprehensive summary of existing work.\n\nWeaknesses:\n- The MSP attack requires extensive prior knowledge. While this assumption is reasonable for LLMs—since most are open-source and fine-tuned from publicly available models—it may not hold for small models and ViTs. These models often lack such vulnerabilities, making the success of the MSP attack on them questionable.\n\n- The proposed LoRO appears to be a combination of several existing techniques: the low-rank mask resembles the use of Freivalds’ algorithm in Slalom [a], the permutation matrix is directly adopted from TransLinkGuard [13], and the factor multiplexing is similar to the cornerstone fingerprints in SOTER [9].\n\n- LoRO lacks result verification for outsourced computation, a feature provided by methods like TEESlice [12, 31] and SOTER [9].\n\n- Some attack details are missing. In Section 4, the paper outlines the attack pipeline for three types of obfuscation methods but lacks specific details on how ShadowNet’s obfuscation is broken—particularly since it employs both multiplicative and additive masking on weights. The explanation in Appendix A is also unclear. Additionally, it is not sufficiently explained how the proposed LoRO is attacked. Since the attacker cannot recover the exact weights, is the attack conducted using techniques from TransLinkGuard [13]? \n- Some evaluations are missing. In Section 6.3, the paper compares efficiency with a few full shielding methods and SOTER. While comparing with full shielding approaches is reasonable, since they offer similar black-box security and the performance gains of LoRO are somewhat expected. The evaluation only includes SOTER, overlooking other methods listed in Table 1. Additionally, it does not compare with TEESlice, which is also applicable to LLMs [31].\n\n[a] Tramer, Florian, and Dan Boneh. \"Slalom: Fast, verifiable and private execution of neural networks in trusted hardware.\" arXiv preprint arXiv:1806.03287 (2018).\n\nSome Mistakes and Suggestions:\n- In Equations 3 and 4, the variable should be x instead of x', as x' is not introduced until line 223.\n- Missing citations. In Tables 1 and 2, it would be better to include citations for the baseline methods directly in the tables to improve readability, especially for newcomers to the field. Some abbreviations, such as TLG for TransLinkGuard, differ from the original paper titles. While defining such abbreviations is acceptable, proper citations are still necessary.\n- Incorrect highlighting. In Table 1, it states “the worst is in red,” but several red-highlighted values are not the highest. For example, in the RoBERTa model on the SST-2 dataset, SOTER’s 93.20% is higher than the red-highlighted 91.42%. Similarly, for the BART model: on the MRPC dataset, SOTER’s 82.11% is higher than the red 79.96%; on the SST-2 dataset, SOTER’s 94.57% is higher than the red 91.05%.\n- Incorrect table references. In lines 44, 186, 279, and 500, “Table 1” is incorrectly linked to Section 1 instead of jumping to the actual table. Additionally, in lines 282 and 519, “Table C” is mentioned, which appears to be a mistake—it likely refers to “Table 6”, but is incorrectly linked to Table 5.\n\nQUESTIONS:\n1. How is the MSP attack implemented for LoRO?\n2. How is ShadowNet’s obfuscation broken?\n3. Since LoRO uses the same additive masking technique described in Section 4.2, does it suffer from the same vulnerabilities?\n4. How does LoRO ensure the correctness of outsourced computation?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the problem of model stealing attacks that can occur when running LLMs on untrusted devices such as edge devices. This paper first proposes an MSP attack, indicating that the existing protection method using TEE (Trusted Execution Environment) suffers from statistical vulnerabilities and may be exploited by the attacker. Therefore, this paper proposes a new method LoRO, which completely fuzzes the model parameters through the dense masking mechanism, including using low rank factors to generate dense masks, improving inference efficiency, and sharing low rank factors across multiple layers to reduce memory consumption.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Trendy topic\n- Well-organized paper\n- Easy to follow\n\nWeaknesses:\n\n- More clarifications are needed\n\nQUESTIONS:\nThis paper is well-organized, and the writing and charts are very clear and easy to understand, and the Threat Model is also described clearly. LoRO is simple but effective, with low performance loss, and does not require model structure modification or retraining. Although this method increases the memory requirements of the device, this paper also proposes Factor Multiplexing to reduce the memory. But I still have some concerns:\n\n1. In the pipeline of MSP attack, the attacker used a small dataset for fine-tuning. Whether this part of the small dataset is a subset of figure1 (a) Private Dataset or a dataset that is completely another domain.\n2. In Figure 2, it is unclear what each point in the t-SNE visualization represents. Are they column vectors from the weight matrices of each model layer? I believe the authors should clarify this explicitly in the figure caption.\n3. Consider that Factor Multiplexing reduces storage pressure, it will introduce new computational or runtime costs. The paper should provide a more detailed analysis of this trade-off.\n4. In Table 3, LoRO shows almost no performance drop across several datasets, such as SQuAD and GSM8K. However, I wonder how well it would perform on more challenging benchmarks, for example, the MATH dataset [1].\n5. Since LoRO places key components inside the TEE, side-channel attacks are a very realistic threat in deployment. Any such leakage could directly expose the shared low-rank basis factors used for parameter obfuscation.\n\nOverall, I think this motivation is sufficient and the proposed method is also very effective.\n\nMinors:\n\n1. It should be clearly written in Table1's caption that this is the result of evaluation under the MSP attack.\n\nReference:\n\n1. Dan Hendrycks et al. “Measuring Mathematical Problem Solving With the MATH Dataset.” NeurIPS 2021.\n\nLIMITATIONS:\nMissing discussion of Side Channel Attacks", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes LoRO, which leverages dense mask to completely obfuscate parameters for TEE-based secure LLM. LoRO includes two innovations: (1) Low Rank Mask, which uses low-rank factors to generate dense masks efficiently. The computing complexity in TEE is hence reduced by an exponential amount to achieve inference speed up, while providing robust model confidentiality. (2) Factors Multiplexing, which reuses several cornerstone factors to generate masks for all layers. Experiments are conducted on multiple models and tasks to prove the effectiveness of LoRO.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n+ (Novelty) This paper propose novel low rank obfuscation, which leverages low rank mask and factor multiplexing to generate dense masks efficiently.\n\n+ (Contribution) This paper targets an important research field and proposes promising solutions. \n\nWeaknesses:\n\nMy major concern of this paper is about the writing clarity. \n\n- Table 1 is not well illustrated. Please add reference to other works. How 'Average' is calculated? If the best performance is in green, some data points of black-box should be in free right, e.g., 10.28%? How model stealing accuracy is defined?\n\n- Lack of background on prior works. What are their mechanism? Which work is considered as SOTA before LoRO and why? Some terms are not defined before using, e.g., TLG.\n\n- Lack of clarification of the adversary. In line 122, adversary is defined as 'parties attempting to steal valuable LLMs'. However, what does steal LLMs refer to? Does it refer to recover the exact parameters stored in TEE? Or it refers to achieve a surrogate LLM which has similar performance? It is assumed that adversary can obtain 10% training data. How could this happen? If TEE is used for securing the model, a more general case is that the model is trained on private data. If the model is purely trained on public data, people are more willing to open source the model nowadays rather than use TEE.\n\n- The multiplicative noise method and corresponding attack is not referred or explained in the paper. However, they appear in Figure 2.\n \n- Typo in line 167: xW' is accelerated in REE -> x'W' is accelerated in REE. Line 169: sparse mask r_s -> sparse mask r\n\n- Should equation 3 be y = xW'-xD? if it is x'W', equation 3 will be y = x'W'-xD-o(W+D), similar to equation 2, right?\n\nQUESTIONS:\nPlease see the weakness above.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}