Alignment-Aware Model Extraction Attacks on Large Language Models

Authors: Zi Liang, Qingqing Ye, Yanyun Wang, Sen Zhang, Yaxin Xiao, RongHua Li, Jianliang Xu, Haibo Hu
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Model Extraction Attack, Large Language Models, Alignment

Abstract:
Model extraction attacks (MEAs) on large language models (LLMs) have received increasing attention in recent research. However, existing attack methods typically adapt the extraction strategies originally developed for deep neural networks (DNNs). They neglect the underlying inconsistency between the training tasks of MEA and LLM alignment, leading to suboptimal attack performance. To tackle this issue, we propose Locality Reinforced Distillation (LoRD), a novel model extraction algorithm specifically designed for LLMs. In particular, LoRD employs a newly defined policy-gradient-style training task that utilizes the responses of victim model as the signal to guide the crafting of preference for the local model. Theoretical analyses demonstrate that i) the convergence procedure of LoRD in model extraction is consistent with the alignment procedure of LLMs, and ii) LoRD can reduce query complexity while mitigating watermark protection through exploration-based stealing. Extensive experiments on domain-specific extractions validate the superiority of our method in extracting various state-of-the-art commercial LLMs.

Decision: reject
Reviewer scores: 5, 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1730012878330

thread_type:
official_review

<thread_object>
creator: reviewer_5msi

<content>
summary:
This paper proposes a new model extraction attack (MEA) algorithm, named LoRD. The authors claim that existing MEA methods suffer from not taking the preference alignment process into consideration during stealing. The authors try to use the victim model's response as the guidance to help select the local chosen and rejected responses (also as the optimization target in some cold start cases and regularization terms). The authors believe their loss design can make the attack more efficient and resistant to text watermark defenses.

strengths:
1. The studied problem is practical and meaningful. The motivation is good and reasonable.

2. The paper is well-written, the tables and figures are well displayed. I thank the authors for their great efforts on well organizing their manuscript.

3. I like the analysis about Figure 5, which shows some novel and interesting insights.

weaknesses:
I have some concerns about current submissions. I hope the authors could address my questions below. I may adjust my scores based on the authors' responses.

1. First, I should suggest the authors to carefully read and follow the Author Guide in the website to place the Ethical Statement (and optionally the Reproducibility Statement) after main text before references, rather than in the Appendix.

2. Regarding the method part, in Line 210-211, the authors states that "indicate whether a selected sentence is locally isotropic to the victim model’s response... in the current optimization step". However , from Figure 3 and Eq. (8) we can see that the victim model's response $y_{vic}$ is not used in deciding chosen ($y^{+}$) and rejected ($y^{-}$) responses and in objective loss function $L_{obj}$ (unless in the cold start case). So I am wondering how the victim model's response can guide the preference alignment of the local/target model?

3. Regarding the form of objective function in Eq. (8), it seems to be very similar to SimPO [1] loss (without some regularizations). So I am wondering why do authors not try this straightforward idea to perform model stealing: sample two responses from local model, prompt the victim model to directly decide the chosen one and rejected one, then places them into the Eq. (8).

4. Regarding the regularization term, it has the same form of the objective function but places $y^{+}$ with $y_{vic}$ in the denominator. I think the function of this regularization function if to consider the victim model's response the chosen response and directly distill the knowledge of victim model into the local model. So why call it the regularization function?

5. I do not fully understand the analysis about Query Efficiency in Section 4.2. I am confused why the ideal query times for LoRD can be reduced to $O(V^{NQ}\times C)$.


6. The Watermark Resistance part is interesting and reasonable. But I think selecting vocabulary-splitting based watermarking method [2] is inappropriate (as we can see, the p-values of MLE are already very high), the authors should choose backdoor-based watermarking methods [3,4], which would make the results more convincing.

7. The experimental results in Table 1 show limited improvement over baseline MLE. 


[1] Meng, Yu, Mengzhou Xia, and Danqi Chen. "Simpo: Simple preference optimization with a reference-free reward." 

[2] Kirchenbauer, John, et al. "A watermark for large language models." ICML 2023

[3] Gu, Chenxi, et al. "Watermarking pre-trained language models with backdooring."

[4] Li, Peixuan, et al. "Plmmark: a secure and robust black-box watermarking framework for pre-trained language models." AAAI 2023

questions:
There are some typos or presentation errors:

(1) In Eq. (4), $y_{i, <j}$ should be bolden.

(2) In Eq. (6), why $\hat{y}$ in the first term but $y$ in the second term.

(3) Figure 2, Step 4, should "and" be "or" (according to the last line in Page 4)?

(4) In Appendix, there are a lot of misuses of ```\citet``` (should be ```\citep```).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730459497574

thread_type:
official_review

<thread_object>
creator: reviewer_anx7

<content>
summary:
This paper designs a new model extraction attack targeting LLMs. The method innovatively uses reinforced distillation, allowing a local model to more quickly and accurately learn the victim model’s knowledge. Moreover, thanks to reinforcement learning, the local model does not learn the watermark embedded in the victim model. The authors conducted extensive experiments to verify the effectiveness of this method.

strengths:
- The paper is well-written with a clear structure and rich content, making it easy to follow.
- The authors designed a new reinforcement learning-based distillation method called Locality Reinforced Distillation (LoRD), achieving better results in LLM model extraction problems.
- The method can steal commercial models under a fully black-box threat model, making it highly practical.
- Unlike supervised learning-based methods (MLE), LoRD does not imitate the victim’s generation as labels, so it does not replicate possible watermarks in the victim model.
- LoRD’s learning approach improves the way LLMs are extracted, thereby reducing the cost of queries.
- Although the method is not highly effective on every task, the authors have deeply explained the reasons behind these issues.
- As an attack method against LLMs, the authors responsibly discussed ethical concerns and provided some possible defense strategies.

weaknesses:
- The method is a domain-specific model extraction method; the authors should clarify this in the introduction section.
- The design of the method includes some thresholds. Although the authors provided specific values, they did not carefully introduce the impact of these parameters and whether attackers need to set different thresholds for different local and victim models.
- In Equation 8, the authors removed a term but did not explain the deeper reasons and impacts.

questions:
- In Equation 8, does removing P_{y_{vic}} have negative effects?
- In Equation 9, if using y+ instead of y-, what differences and impacts would there be?
- For some commercial models that do not provide generation probabilities, how effective is this method?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730538105065

thread_type:
official_review

<thread_object>
creator: reviewer_bcxr

<content>
summary:
This paper proposes a new model stealing attack, particularly geared towards target LLMs that are aligned via RLHF.

strengths:
1. The premise of the paper seems promising and does try to fill in an important gap in the knowledge about model stealing attack against modern LLM, particularly targeting the RLHF process. This seems like an interesting problem with potential impact.

weaknesses:
1. **Lack of rigor and explanation in the derivation of the loss functions**
    1. L228: I'm quite lost here about why this is the right derivation of $R_{\theta_\phi}$ from Eq. 6. I'm not an expert in RL so I may just miss something obvious, but I'd like to see the full derivation somewhere in the paper.
    2. L239: These design decisions seem rather unprincipled to me. The scaling term is simply dropped. To make a claim that the algorithm still works as intended after the approximation, I'd like to see an ablation study to validate this point.
    3. L248 (”requires an extra exponential…”): I'm a bit confused by this claim. I'm not sure why just one more exponentiation would noticeably increase the runtime to the point that it is a consideration for the attacker.
    4. L250 (”only the logarithmic part of KL divergence is employed”): Again, this seems quite unprincipled. What effect does it have? Any ablation study?
    5. L251: What are the "selected tokens" in this case?
    6. Eq 9: Here, the KLD term is not even applied to the current and the initialized models. It is only on the current model with two different outputs (they are not distributions). It is not KLD anymore. I do not understand why it is motivated by the usual KLD term or whether they even serve the same purpose.
    7. L264 (”Finally, we wrap L_LoRD with a sigmoid…”): Why is this necessary?
2. **Lack of rigor in theoretical analysis in Section 4**
    1. The theoretical analysis unfortunately lacks any rigor or real purpose in the paper. Proposition 1 is not a well-defined mathematical statement. While the proofs in the appendix are "not wrong," they add no information and do not support this proposition. Proposition 2 also does not support the proposed method.
    2. Section 4.2 also lack mathematical rigor, and the statements are handwavy. For example, the analysis on the number of queries for both MLE and LoRD seems to lack any derivation or source.
3. **Experiments**
    1. Why not evaluate the alignment of the model since the attack tries to imitate the RLHF process which is mostly used for safety training? If we are simply evaluating specific downstream tasks or knowledge, then using MLE to steal the model seems perfectly fine, and there is no need to use any alignment technique.
    2. The empirical results overall are relatively weak; LoRD almost has no improvement over the MLE baseline.
4. **Presentation**
    1. Figure 3: I'm not entirely sure what this figure is trying to communicate or add more information beyond the text or Figure 2. I might just be missing the point here.
    2. L205 (second paragraph of Section 3.1): This entire paragraph just dives into the technical design of the algorithm. I think it might be a good idea to just explain the intuition or the design choices in words before providing all the details.
    3. Table 1: I believe there are too many unnecessary numbers in this table. For example, perhaps only report F1 instead of precision and recall?

### Nitpicks

1. L30 (”ChatGPT cha (2024)”): There seem to be multiple typos on the citations throughout the paper.
2. There is a mistake where the authors cite references in the wrong format without parentheses, i.e., using `\citet` instead of `\citep` when using `natbib`. This happens so often that it slightly disrupts the reading.

questions:
--
</content>
</thread_object>

</thread>
<thread>
creation date:
1730574235114

thread_type:
official_review

<thread_object>
creator: reviewer_xic1

<content>
summary:
The paper discusses the vulnerabilities of large language models (LLMs) to model extraction attacks (MEAs). The authors propose a Locality Reinforced Distillation (LoRD) method via introducing the reinforcement learning procedures. LoRD costs less query times and mitigates watermark protection. Extensive experiments demonstrate the effectiveness of LoRD in extracting commercial LLMs. The paper also provides a theoretical analysis, discussing why LoRD can achieve stronger watermark resistance and higher query efficiency than existing methods.

strengths:
1. The paper might be the first work to steal models by considering the alignment procedure of LLMs with RL. 
2. Extensive and comprehensive experiments.
3. The paper provides theoretical analysis on the consistency of model performance.

weaknesses:
Refer to the questions. 
The experimental results are interesting. A more detailed analysis could be beneficial.

questions:
1. Could you provide a more detailed analysis of Figure 6: Comparison of watermark resistance? According to Eq. 11, resistance across different \lambda values should be consistent. However, the results in Figure 6 show some inconsistencies in performance.

2. How does the choice of local model impact the final results? Since the goal of this paper is to steal the alignment capacity of a large commercial LLM, the capability of the foundational local model should be critical. Have you experimented with other models besides Llama3-8B?

3. In Table 1, for "Data to Text: E2E NLG Dušek et al. (2020) with 64 query samples," the stolen model (+LoRD) outperforms the victim model. Could this be due to overfitting? Could you analyze this further?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732330888879

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Thank you to all the reviewers for your valuable feedback. We have thoroughly revised our work based on your comments and provided detailed, point-by-point responses to your questions.

Please do not hesitate to reach out if you have any further questions or concerns. Thanks!

For your reference, the original version of the submission is available for comparison at: https://anonymous.4open.science/r/LoRD-MEA-1EF2/v1.pdf
</content>
</thread_object>

</thread>
<thread>
creation date:
1733310242605

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the Review [1/2]

comment:
We sincerely thank all reviewers for their valuable time and efforts during both the review and rebuttal periods. For the convenience of the reviewers' discussion and the Chairs' assessment, we provide a summary of the reviews below.

# Summary of the Weaknesses and the Concerns

We have made every effort to address the concerns raised by the four reviewers. Among them, two reviewers indicated that their concerns have been resolved, one reviewer highlighted one major concern that remains unresolved, and one reviewer did not provide any responses.

We have categorized these concerns into three main areas: (1) missing necessary experiments, (2) misunderstandings of the paper, and (3) requests for further discussion and additional experiments.

## Missing Necessary Experiments

- **Safety Alignment Extraction**: Reviewer Bcxr and Reviewer aNx7 suggested the need for a safety alignment extraction mechanism beyond domain-specific stealing. We agree with this necessity and have included alignment extraction experiments for two tasks in the revised version of the paper.

- **Ablation Study**: The reviewers expressed interest in an additional ablation study to validate the intuitions behind our loss function design. In response, we revised the methodology section of the paper and included an ablation study to address these concerns.

**Status**: Reviewer aNx7 acknowledged that these revisions well addressed their concerns, while Reviewer Bcxr has not provided feedback yet.

## Misunderstanding of the Paper

Several misunderstandings about the paper were identified, and we have sought to clarify them during the rebuttal:

1. **Impacts of the Choice of the Local Model**: We clarified that relevant experiments have already been presented in the Appendix.

2. **Limited Improvement to MLE**: We highlighted experimental results in Tables 1, 2, and 5, as well as Figures 7 and 8, to demonstrate that the improvements are substantial rather than limited.

3. **"Lack of Rigor and Explanation"** in the Design of Loss Functions and Theoretical Analysis: (i) We noted that many current works in the field of LLM+RL also lack rigor and explanation. In contrast, our study goes beyond intuition and empirical experiments by offering some in-depth theoretical explanations and analyses. (ii) We provided additional theoretical analysis in the revised version. (iii) We improved the readability of the paper based on prior studies.

**Status**: Point 1 was accepted by Reviewer Xic1, and Points 2 and 3(ii) were accepted by Reviewer 5Msi. The remaining points are still awaiting responses from Reviewer Bcxr.

## Discussion

1. **Abnormal Experimental Results**: Reviewer Xic1 raised concerns regarding abnormal points in the watermark resistance experiments (Figure 6) and the D2T experiments (Table 1). In response, we provided explanations and supplemented additional experiments to support our claims.
2. **Methodology Discussion**: Reviewer 5Msi proposed some interesting exploration of the methodology, particularly regarding the importance of the regularization term and the **rationale for not using direct prompt-based feedback**. We addressed these points by explaining our design choices and analyzing potential drawbacks of obtaining binary feedback using direct-intent prompts, which considers three core factors: stealthiness, query efficiency, and complexity.
We also supplemented two groups of experiments to support our analysis.
3. **Model-Level Watermarks**: We expanded the discussion in our paper on the potential utility of model-level watermarks for defending against our proposed method, addressing relevant concerns.

**Status**: Points 1 and 3 have been addressed. For Point 2, although our explanation with three reasons did not fully satisfy Reviewer 5Msi, the additional empirical attempt was provided as further clarification.


Additionally, the reviewers identified presentation issues in the paper, such as the incorrect placement of the "Ethical Statement" section. We appreciate their careful review and have revised the paper accordingly, which has been immensely helpful.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces an RL-based method called Locality Reinforced Distillation (LoRD) to reduce query complexity of LLM-targeted model extraction attacks. While the method itself is promising, reviewers identified major issues, including unclear presentation and insufficient experiments. Despite the authors' detailed rebuttal, the reviewers concluded that the current version does not meet ICLR's standards. I encourage the authors to continue refining their work for future submissions.