Archilles' Heel in Semi-open LLMs: Hiding Bottom against Recovery Attacks

Authors: Hanbo Huang, Yihan Li, Bowen Jiang, Ruoyu Sun, Lin Liu, Zhuotao Liu, Bo Jiang, Shiyu Liang
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: semi-open model, closed-sourcing approach

Abstract:
To address privacy concerns with large language models, industrial users request local fine-tuning and deployment, but unencrypted models risk theft. Hardware-based security provides protection but is constrained by secure memory, leading to semi-open configurations. Semi-open models balance security and customization by keeping key layers closed-source within a secure environment while allowing others to be fine-tuned, but closed-source layers are susceptible to recovery attacks. In this paper, we explore the design of semi-open models with fewer closed-source layers, aiming to increase customizability while ensuring resilience to recovery attacks. We analyze the contribution of closed-source layer to the overall resilience and theoretically prove that in a deep transformer-based model, there exists a transition layer such that even small recovery errors in layers before this layer can lead to recovery failure. Building on this, we propose \textbf{SCARA}, a novel approach that keeps only a few bottom layers as closed-source. SCARA employs a fine-tuning-free metric to estimate the maximum number of layers that can be publicly accessible for customization. We apply it to five models (1.3B to 70B parameters) to construct semi-open models, validating their customizability on six downstream tasks and assessing their resilience against various recovery attacks on sixteen benchmarks. We compare SCARA to baselines and observe that it generally improves downstream customization performance and offers similar resilience with over \textbf{10} times fewer closed-source parameters. We empirically investigate the transition phenomenon and analyze the effectiveness and limitations of our scheme.

Decision: withdrawn
Reviewer scores: 5, 3, 6, 8

Reviews and discussion:
<thread>
creation date:
1729814123666

thread_type:
official_review

<thread_object>
creator: reviewer_ag9z

<content>
summary:
This paper presents SCARA, a method to identify decoder layers to hide in decoder-only LLMs. The authors provide theoretical proof of transition layers, where early errors are amplified in subsequent layers. They introduce RD to assess post-recovery performance when specific layers are hidden. Experiments show that SCARA, by hiding only a few layers, achieves a recovery ratio close to baselines while maintaining customization performance similar to fully open approach. The experiments also confirm the existence of transition layers in the models.

strengths:
1. The paper is well-written, featuring thorough experiments and clear explanations from both theoretical and empirical perspectives.
2. The overall layout is visually pleasing, and the figures are diverse and effectively illustrate the content, aiding readers' understanding.
3. It proposes a straightforward and effective method for constructing a semi-open model by hiding only a few layers, while achieving baseline-level resilience and customization performance comparable to the fully open setting.
4. The insight regarding the existence of a transition layer contributing to resilience is particularly compelling, with a detailed theoretical explanation that enhances understanding.
5. The authors provide comprehensive empirical validation across multiple architectures and benchmarks, covering models of various sizes (1.3B-70B), and testing customizability and recovery performance on several benchmarks. They also conducted experiments on recovery datasets of different sizes, demonstrating sufficient experimental rigor.
6. The authors proposed additional enhancements to the original baseline, strengthening the protection of baseline SAP and highlighting SCARA’s effectiveness in preserving resilience.
7. The authors empirically validated their theory of the transition layer’s existence and pointed out that smaller models exhibit transition layers earlier than larger models.
8. The authors clearly identified the limitations of the SCARA method, noting its ineffectiveness on small models (OPT-350M) and its inability to defend against other adversary attacks.
9. The proposed SCARA algorithm has clear practical applications, offering a viable solution for enhancing the customizability of semi-open models while preserving comparable resilience.

weaknesses:
1. One mathematical notation in Section 4.2 is unclear. The loss function $\ell$ for RD(I) is not specified, making it confusing.
2. Figures 1 and 2 have minimal captions and small text, reducing readability and limiting their ability to convey insights.

questions:
1. Is the "attack datasets" mentioned in Figure 2 the same as the "recovery datasets" discussed later in the paper?
2. Could you clarify the formula and the loss function used for RD(I)?
3. Could you clarify how fully-closed and semi-open models differ in practice?
4. Could you explain more about the distinctions between FT-all, FT-closed, and SEM in Section 5.1?
5. Can the row and column headers in the tables be made clearer by avoiding abbreviations?
6. Could you explain more about potential future work that could be included in the paper?
7. Could the authors clarify what the value 0.00 represents in Table 1 and Table 2?
8. The authors discussed the impact of datasets of different lengths on the effectiveness of SCARA in the experimental section, but these datasets did not appear in the setup. Could the authors provide a detailed introduction to the composition of these datasets?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730674602024

thread_type:
official_review

<thread_object>
creator: reviewer_1rg5

<content>
summary:
This paper studies the problem of how to design semi-open models (i.e., models whose weights are only partially open-sourced) that can simultaneously also be resilient to recovery attacks. The paper finds that a transition layer exists, such that even small recovery errors in layers before this layer can lead to recovery failure. Building on these insights, the paper proposes an approach called SCARA that keeps only a few bottom layers as closed-source. With this new approach, the paper shows that it is possible to improve downstream customization performance while maintaining similar resilience.

strengths:
1. The paper is well-written, and the presentation is clear and clean. 

2. The approach is well motivated --- it first starts from an empirical observation that closed-sourcing the first two layers offers significantly greater resilience than the last two, while the model shares similar customizability under the two cases. This implies that close sourcing the later layers may be the optimal solution for keeping the resistance to recovery attacks. The paper subsequently further formally establishes this finding with rigorous theoretical analysis, showing the existence of a transition layer such that even small recovery errors in layers before this layer can lead to recovery failure. This also intuitively makes sense --- when the attacker is asked to recover the earlier layers as opposed to the later layers, the errors in early layers will be amplified by later layers. This asymmetry is natural. 

3. Based on this insight, the paper also proposes an effective approach for the selectively closed-sourcing model to defend against recovery attacks. The experiment results support the effectiveness of the approach. 

Overall, the paper is nicely done.

weaknesses:
One outstanding weakness of this paper is that the threat model considered may not be practically relevant. It seems the authors coined the semi-open model's scenario, that seems not really exist in the real world. 

Currently, the most common setups are either open-source or closed-source. For close-source context, when developers do want their users to customize their models, the standard practice is to deploy fine-tuning APIs (e.g., https://platform.openai.com/docs/guides/fine-tuning) rather than partially open-source a part of the model. It seems to make no sense to only open-source the first few layers of a model to enable customization. Because the customization anyway still needs the involvement of the closed-source developers --- so they can fine-tune and connect the first few layers and the later layers to really deploy the model. Then, why not just close-source all weights and directly ask the users to upload custom data, and then the closed-source developers fine-tune and deploy the model for the users, like what is being done in fine-tuning APIs? 

I worry that if not developers will do the partial open-sourcing like the authors of this paper consider, then the problem itself may not hold.

questions:
Can the authors explain and clarify why the semi-open models are practically relevant?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730754522390

thread_type:
official_review

<thread_object>
creator: reviewer_kkbu

<content>
summary:
The paper proposes a method for identifying which layers in the model that, if recovered by an adversary in an iterative layer recovery process, will make subsequent layers easier to recover.

strengths:
The evaluation seems comprehensive. It was easy to follow the problem setup and the method.

weaknesses:
I question the threat model that the authors are introducing. I don't think there's any chance of stealing an LLM through any method that exists no matter how semi-closed/semi-open it is. The only methods that have been proposed that can do something like this specifically target the embedding layer.

It seems like the main insight of the paper, that hiding the earlier layers in the model is more impactful than hiding later layers because if an attacker wants to recover the model they'll pay an accuracy error scaling in the depth of the model past the layer they haven't yet recovered, is trivial. If you asked someone who had never heard anything about this literature of hiding layers, whether they should hide the first block or the last block, I'm certain everyone would choose to hide the first block. There's plenty of work already showing that later layers are more or less redundant and don't learn anything new. This is because attention heads in block N have the ability to learn Nth order interactions, but for N > 2, these interactions typically don't get learned and the attention heads just degenerate [1].

The actual implementation of the method is not sophisticated. It just takes this straightforward insight and turns it into a metric. But that metric is itself just "what happened if I closed the first N layers of the model" and then returns the first one that passes some threshold of difficulty.

It doesn't seem like the evaluation is really fair. The authors evaluate against SEM. But SEM just wants to recover the embedding and the authors are trying to show what happens if they hide the early parts of the network. This seems like an indication that this isn't a particularly realistic threat model.

[1] https://arxiv.org/abs/2404.08634

questions:
n/a
</content>
</thread_object>

</thread>
<thread>
creation date:
1730776422699

thread_type:
official_review

<thread_object>
creator: reviewer_d5gh

<content>
summary:
The paper introduces SCARA, a selective closed-sourcing approach for designing semi-open large language models (LLMs) that enhance customizability while maintaining resilience against recovery attacks. The authors develop an algorithm that strategically keeps only a few bottom layers closed-source, ensuring model flexibility without compromising security. They theoretically demonstrate a "transition layer" within deep transformer models, showing that recovery errors in layers before this point lead to recovery failure, while errors in later layers have a limited impact. SCARA estimates the optimal number of layers to hide using a novel metric based on initial recovery loss, bypassing the need for fine-tuning. The method is applied to five models ranging from 1.3B to 70B parameters, tested across six downstream tasks and sixteen recovery benchmarks. Results show that SCARA improves downstream performance while requiring over ten times fewer closed-source parameters than baselines, achieving improvements, especially in domain-specific tasks like Financial, with 30% higher performance on Llama2-70B. SCARA maintains comparable resilience against recovery attacks.

strengths:
1. The paper introduces SCARA, a method that selectively closes only the bottom layers of semi-open large language models (LLMs) to enhance customizability while maintaining resilience against recovery attacks.
2. It provides a theoretical analysis of the existence of a transition layer in transformer-based models.

weaknesses:
1. **Unclear Motivation for Semi-Open Models:** The market is dominated by closed-source models and fully open-source models. If customization needs are already addressed by existing fine-tuning services provided for closed-source models (e.g., API-based fine-tuning on closed models like GPT-4), it would be insightful to understand the specific motivations and advantages driving the development of a semi-open architecture. 
2. **The threat model is not clear.** The threat model concerning recovery attacks on semi-open LLMs is insufficiently defined. The paper does not clearly specify the adversary's capabilities, such as the extent of access to the model's architecture, parameters, or outputs. This lack of clarity makes it challenging to assess the effectiveness of the proposed SCARA method in mitigating such threats.
3. **Insufficient Details on SCARA's Implementation:** The description of SCARA's methodology is vague, particularly regarding the fine-tuning-free metric used to determine which layers to keep closed-source. The paper does not provide a clear explanation of how this metric is calculated, the data required, or the computational resources involved etc.
4. **Evaluation minors:** While the authors present experimental results across multiple models and tasks, the evaluation lacks depth. The paper does not offer a comprehensive analysis of SCARA's performance compared to existing methods, nor does it explore potential trade-offs between customizability and security.

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731912582032

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response by Authors (1/2)

comment:
Dear area chair and reviewers,

We sincerely thank the reviewers for their time and valuable comments. Overall, the reviewers appreciated our innovative theoretical insights (d5Gh, 1rG5, ag9Z), particularly the identification of transition layers (ag9Z). All reviewers also commended the comprehensive evaluation across various model sizes and benchmarks, as well as the clear and effective presentation of our findings (kKBu, ag9Z, 1rG5). Additionally, some reviewers highlighted the effectiveness of SCARA (1rG5) and the potential impact of our work on future research directions (ag9Z).

We acknowledge the reviewers' concerns regarding the real-world applicability of semi-open models and the details of our threat model. In this general response, we aim to clarify our motivation and provide a more detailed explanation of the threat model and the adversary’s capabilities. In the individual responses, we address each specific comment and question raised. We hope these clarifications and responses adequately address the reviewers’ concerns.

**Q1:** **Are semi-open models widely used? what is the advantage of these semi-open models? What is the motivation of designing semi-open models?**

**R1:** Companies like OpenAI offer fine-tuning APIs [1] for closed-source models (e.g., GPT-4), while META open-sources models like the Llama series [2] for customization. Semi-open models, which combine closed-source embeddings (e.g., OpenAI's text-embedding-ada-002 [3] , Cohere's embeddings [4], or Google Vertex AI [5]) with open-source modules (e.g., LlamaIndex [6], Haystack [7]), have gained popularity for tasks like search [8], recommendation systems [9], anomaly detection [10], and classification [11]. This hybrid approach benefits both vendors and users. Vendors retain control over proprietary components, generate revenue, and reduce the computational demands associated with fine-tuning for downstream tasks [12-14]. Users, in turn, can customize open-source modules offline to better optimize performance for specific tasks [15-16].

Despite their success in tasks like search and recommendation systems, semi-open models face challenges with more complex tasks, such as deep reasoning, knowledge-intensive operations, and problem-solving in code or math. To address these challenges, we propose a novel semi-open framework that enhances performance on complex tasks while ensuring protection against recovery attack. Our approach partitions a pretrained LLM into closed-source and open-source components, fully leveraging the model's capabilities for handling complex tasks. We theoretically prove that errors in the early decoder layers significantly impact performance, while later layers are less critical. Based on this insight, we introduce SCARA, a design that closed-source key early layers while allowing users to customize open modules. To optimize this design, we propose a fine-tuning-free metric, ``recovery difficulty'', to determine the optimal partition point. This framework provides a balance between high customizability and strong resilience against recovery attacks, advancing the capabilities of semi-open models.

We have revised the first paragraph in the introduction to provide more application of semi-open models.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732723735312

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response 2 by Authors: More Elaboration on Motivation

comment:
We sincerely thank all reviewers for their constructive comments on the semi-open paradigm. We would also like to take this opportunity to further elaborate on the scenarios where semi-open models are widely deployed.

Many vendors now offer large language models (LLMs) that come with high training costs but deliver exceptional performance across various industries [1-2]. Industrial users, such as healthcare organizations, financial institutions, and government agencies, often request these models to be customized with private data, fine-tuned, and deployed locally in on-premises environments [3-5]. However, deploying models without encryption exposes their architecture and parameters, making them susceptible to model stealing attacks that retrieve parameters from CPU, RAM, and vRAM, posing significant risks to intellectual property [5-9]. To mitigate these risks, vendors use hardware-based security techniques like Trusted Execution Environments (TEEs) and encrypted inference, which conceal the model weights and restrict unauthorized access during inference [10-11]

Despite their benefits, TEEs have significant limitations, such as restricted secure memory (e.g., 128MB–256MB in Intel SGX)[7,14], which is insufficient for the gigabytes of memory required by large language models [12,14,15]. As a result, only a portion of the model's layers can be secured within the environment, leaving the remainder exposed. This creates a **semi-open** model setting that balances the need for security and customization.

However, this semi-open approach introduces a tradeoff. Securing or "**closed-sourcing**" more parameters limits the scope of fine-tuning on private data, reducing the model's ability to adapt to specific needs. Conversely, exposing or "**open-sourcing**" more parameters increases vulnerability to model extraction or distillation attacks, where attackers can query the closed-source module, construct input-output pairs, and train a mimic model that replicates its functionality. In this paper, we address this challenge by exploring methods to determine which parts of the model should be closed-sourced in a secure environment, enabling effective fine-tuning while safeguarding against extraction risks.

We have revised the abstract and introduction and hope this updated explanation effectively clarifies our approach and addresses the concerns about the motivation for designing semi-open models.

[1] https://openai.com/index/hello-gpt-4o/

[2] GPT-4 Technical Report https://arxiv.org/pdf/2303.08774

[3] How to run LLMs locally: Hardware, tools and best practices https://www.techtarget.com/searchEnterpriseAI/tip/How-to-run-LLMs-locally-Hardware-tools-and-best-practices

[4] Locally Run Large Language Models May Help Preserve Patient Privacy https://www.techtarget.com/healthtechanalytics/news/366590151/Locally-Run-Large-Language-Models-May-Help-Preserve-Patient-Privacy

[5] Securing AI Model Weights https://www.rand.org/pubs/research_reports/RRA2849-1.html

[6] Deepsniffer: A dnn model extraction framework based on learning architectural hints https://dl.acm.org/doi/pdf/10.1145/3373376.3378460

[7] SoK: All You Need to Know About On-Device ML Model Extraction - The Gap Between Research and Practice https://www.usenix.org/system/files/usenixsecurity24-nayan.pdf

[8] DNN model architecture fingerprinting attack on CPU-GPU edge devices.  https://ieeexplore.ieee.org/abstract/document/9797366

[9] Deepsteal: Advanced model extractions leveraging efficient weight stealing in memories https://arxiv.org/pdf/2111.04625

[10] Shadownet: A secure and efficient on-device model inference system for convolutional neural networks. https://arxiv.org/pdf/2011.05905

[11] No privacy left outside: On the (in-) security of tee-shielded dnn partition for on-device ml.  https://arxiv.org/pdf/2310.07152

[12] CoreGuard: Safeguarding Foundational Capabilities of LLMs Against Model Stealing in Edge Deployment. https://arxiv.org/pdf/2410.13903

[13] Open-Source Solutions for Running LLMs Offline: Benefits, Pros and Cons, and Should You Do It? Is it the Time to Have Your Own Skynet? https://medevel.com/running-llms-offline-benefits-should-you-do-it-1300/

[14] A Fast, Performant, Secure Distributed Training Framework For LLM. https://ieeexplore.ieee.org/abstract/document/10446717

[15] TEESlice: Protecting Sensitive Neural Network Models in Trusted Execution Environments When Attackers have Pre-Trained Models. https://arxiv.org/pdf/2411.09945
</content>
</thread_object>

</thread>


Metareview: