Decoupled Alignment for Robust Plug-and-Play Adaptation

Authors: Haozheng Luo, Jiahao Yu, Wenxin Zhang, Jialong Li, Jerry Yao-Chieh Hu, Xinyu Xing, Han Liu
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Jailbreak Aligner, Memory Editing

Abstract:
We introduce a low-resource safety enhancement method for aligning large language models (LLMs) without the need for supervised fine-tuning (SFT) or reinforcement learning from human feedback (RLHF).  
Our main idea is to exploit knowledge distillation to extract the alignment information from existing well-aligned LLMs and integrate it into unaligned LLMs in a plug-and-play fashion. Methodology, we employ delta debugging to identify the critical components of knowledge necessary for effective distillation. On the harmful question dataset, our method significantly enhances the average defense success rate by approximately 14.41\%, reaching as high as 51.39\%, in 17 unaligned pre-trained LLMs, without compromising performance.

Decision: reject
Reviewer scores: 3, 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1730180397491

thread_type:
official_review

<thread_object>
creator: reviewer_z3rx

<content>
summary:
This paper proposes DAPA, an LLM safety enhancement method without SFT/RLHF. This is performed by first identifying the "alignment knowledge" stored in the MLP module of an aligned model, and transfer the knowledge to unaligned models. Empirically, the defense success rate was improved, and the utility was not sacrificed.

strengths:
The tackled problem, LLM safety and jailbreak, is timely. The proposed delta debugging algorithm for finding alignment knowledge within the MLP module is novel, and the ablation study as well as utility test is comprehensive.

weaknesses:
The main weaknesses I consider are as follows.
- DAPA requires an aligned LLM. Although claimed as "a low-resource safety enhancement method for aligning LLMs", the preliminary of DAPA algorithm for aligning an unaligned LLM is **the very existence** of an aligned LLM. This makes the stated contribution vague since it cannot serve as a substitution to SFT/RLHF, and thus cannot reduce the resource used for alignment or be an "economic solution to LLM safety enhancement".
- The safety test is weak. Throughout the paper, the defense success rate is measured using AdvBench **directly**. However, given the rapid development of LLM safety (or specifically, jailbreak), there are a suite of jailbreak adversaries proposed and been collected by some benchmark projects (e.g., [1] [2]). Thus, it is insufficient to claim safety improvement without testing the model performance against **any** of these representative adversaries.
- The claim of "the alignment knowledge is stored in the Feed-Forward Network (FFN) layers" lacks supportive evidence. This claim is deduced in the paper via Figure 3, where only **a single harmful prompt** is utilized to compare the influence of the MLP module and the attention module across layers. More quantitative results are needed to support this result.

[1] HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
[2] JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models

questions:
My questions to authors are as follows.
- Among the unaligned models, there are several base candidates that have not been tuned to follow instructions. Have you launched additional instruction following on non-harmful datasets in addition to the relative memory substitution described in section 4.1? If not, the result with base models will be hard to compare against instruction-tuned ones.
- When extending Figure 3 to incorporate multiple harmful prompts, is it still true that the MLP's influence is the strongest?
- Can DAPA maintain its benefit when encountering more advanced jailbreak adversaries?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703423428

thread_type:
official_review

<thread_object>
creator: reviewer_eaef

<content>
summary:
This paper proposes a novel low-resource safety enhancement method for LLM by firstly locating and then editing the specific knowledge, without the need of SFT or RLHF. Experiments on several datasets prove the effectiveness.

strengths:
1. Clear writing and easy to follow.

2. Comprehensive experimental results demonstrate the effectiveness of the proposed method.

3. In my opinion, it is interesting and somewhat reasonable to enhance the safety capabilities of LLMs by identifying and editing unethical knowledge, with the help of another safer LLM.

weaknesses:
1. In my opinion, the identification and editing of unethical knowledge within LLMs need to be conducted with great caution. The internal behavior of LLMs may be difficult to measure and analyze accurately, and different knowledge measurement tools may yield varying conclusions about the areas where knowledge is stored. Maybe this could affect the foundational research of this paper, although the conclusions drawn in terms of methodological design and experimentation are somewhat reasonable.  Have you considered using multiple knowledge measurement tools to cross-validate the findings?


2. Modifying knowledge in LLMs raises another issue: it may impact the LLM's ability to respond to general questions. The experimental section of the paper also supports this problem that editing unsafe knowledge can affect its normal response capabilities. Although the decreasing in general capabilities is not significant in experiments, in my opinion, the relationship between the two may not be a trade-off: enhancing safety capabilities does not necessarily require a decrease in general capabilities.

3. The safety knowledge alignment used in this paper depends on a safer teacher LLM from the same family. So, which model will teach these teacher LLMs to be safer? Furthermore, I believe that teaching LLMs to recognize harmful prompts and learn when to refuse to respond may be a more effective way to enhance their safety capabilities.

4. In my view, the ideas and motivations of this paper have some similarities with unlearning, such as in [1-3], both focusing on how to erase specific knowledge. It is necessary to include some discussion on the connection and difference between knowledge unlearning and editing.

[1] Safe Unlearning: A Surprisingly Effective and Generalizable Solution to Defend Against Jailbreak Attacks

[2] Towards Safer Large Language Models through Machine Unlearning

[3] Large Language Model Unlearning

questions:
1. In the absence of a safer teacher model, how can effective model alignment be achieved? Are there alternative sources of knowledge or alignment strategies?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731025481987

thread_type:
official_review

<thread_object>
creator: reviewer_ft2p

<content>
summary:
In this paper, the authors are trying to address a challenging task which is to enhance the safety alignment of large language models without SFT and RLHF. The proposed method(called DAPA), unlike previous methods, only requires limited extra resources by employing a knowledge distillation approach. Two detection/alignment modules, i.e. MLP alignment and Gate alignment, are applied to store the alignment knowledge and check the model's output. The proposed method achieves competitive results with several large base models on the tested datasets.

strengths:
In general, the paper is well-written and easy to follow.

Unlike previous methods which require costly SFT or RLHF, the proposed method can be an additional module as a "plug-and-play approach" to various large language models with extra low-cost resources required. 

The proposed method is tested on models from different families, which shows the flexibility of DAPA that can potentially adapt to other models.

The use of knowledge distillation for alignment is very interesting.

In this paper, the authors present extensive testing with well-documented evaluation metrics across various models and achieve very competitive results. The ablation studies are also well set.

weaknesses:
The paper has tested the proposed DAPA mostly on small/mid-sized models (up to 13B models), there is uncertainty around its performance and resource efficiency when scaled to the large LLM models.

The proposed method introduces layer-specific edits without necessarily clarifying the deeper alignment process which may lead to alignment outcomes that are difficult to interpret or predict. This may make it hard to understand how the proposed module changes specific model behaviors.

As the ethical guidelines and alignment standards often evolve, I wonder if the continuous updates or recalibration of the alignment modules will become difficult, especially with large models.

questions:
Please check the aforementioned issues in "Weaknesses" section. Some additional questions:

I wonder if the proposed method can be easily extended to large multimodal models.

Does the proposed method have potential interpretability?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731102774053

thread_type:
official_review

<thread_object>
creator: reviewer_grxu

<content>
summary:
This paper proposes an interesting idea of isolating the weights of LLMs that impact the safety alignment the most. It then applies these weights to unaligned models to improve their safety alignment. Various experiments demonstrate the effectiveness of the proposed method.

strengths:
1. The paper presents an interesting idea of identifying the important weights from the teacher models for safety alignment. By simply replacing student's weights with these weights from the teacher, we can achieve significant DSR improvements with minimum effort.
2. The paper introduces a simple algorithm based on previous works.
3. The experiments conducted are extensive including comparison with lots of models from different model family.

weaknesses:
1. The paper **doesn't follow the ICLR formatting guideline** by significantly changing the margins. It affects the readability of this paper.
2. The presentation of the paper is poor, e.g. some table and figure titles are too lengthy such as table 2 and 3 and figure 6. Lots of the content should be put into the main text instead of title. I don't know if this is caused by changing the margin or the authors intend to put them in the title.
3. Unnecessary highlight of the figure and table titles. This paper uses lot of bold and highlight which are not necessary.
4. While the authors tried to get more space by changing the margins, they split the tables which should be merged into one into multiple tables such as table 6 and 7, which occupies more spaces. 
5. The contribution is not enough. The main contribution of the paper is a naive search algorithm based on a calibration set. The results are likely highly dataset dependent. There is an experiment in the ablation study to test it on a different dataset, but it's not enough to justify the generability of the proposed method.
6. The evaluation dataset is only 128 prompts. It's very likely the results are overfit to the dataset or calibration set.
7. There is no comparison with the teacher model. E.g in table 1 and 2, after replacing the layers, have we achieved similar or same performances with the teacher model? If not, what's the gap there?
8. Although I understand the motivation of the authors, I don't the palpability of this method from the data provided by the authors. E.g. how is the proposed method compared to just simply fine-tuning using PEFT. Given the number of samples are small, it's possible that fine-tuning with PEFT is more efficient and effective. It's hard to tell from the provided data.
9. The paper claims to that the gate projection has the most significant impact, followed by up projection. But judging from figure 4, it looks like the down projection has a bigger impact than up projection, is this a typo?

Overall, I think this paper presents some interesting and promising findings, but it requires significant more data points to support the generability of the proposed idea.

questions:
Please see my comments in the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732062777108

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Rebuttal/Revision Response

comment:
Dear Reviewers,

We thank the reviewers for their insightful questions and reviews. We truly appreciate your time and effort dedicated to improving our work.

This work proposes a low-resource safety enhancement method for aligning large language models without the need of large computational resources and datasets, with knowledge distillation and memory editing ( Sec 2).
-   Comparing with traditional alignment method, it achieves substantial safety performance improvements with limited resources.
-   It is validated experimentally (Sec4) and interpretability (Appendix F) .
-   Consequently, it provides a robust methodology in the era of large foundation models for more economic alignment without sacrificing performance.
   
We have done all the experiments suggested and answered all the questions. All modifications are marked in red color.

These revisions include additional explanations, paragraphs and sections to help readers understand the proposed method, and additional experiments to highlight the advantages of it. Most importantly, the new advanced jailbreak attack and multimodal model experiment result have been added to further classify the efficiency of DAPA in alignment. For the convenience of reviewers, [this response](https://openreview.net/forum?id=lwTTZkDWoT&noteId=tDdIs4boAA) provides a high-level overview of our contributions.

 1. **New JailbreakBench Dataset experiment results in Appendix G.7** `reviewer GrXu` `reviewer Z3rx`

	-  Conducted an ablation study to evaluate 100 out-to-date harmful questions from JailbreakBench.
	    
	-   **Results**: Observed a 3.06% DSR improvement, confirming the method's robustness against challenging datasets.
	    

2. **New Advanced Jailbreak Adversary Testing in Appendix G.9**. `reviewer Z3rx`
	
	- Conducted an ablation study to test to Llama-2-7B models with GPTFuzzer  
	- **Results**: DSR increased significantly, from 19% to 31%, showcasing effectiveness against sophisticated jailbreak adversarial. These results confirm that our method is dataset-agnostic and demonstrates robust performance on diverse and advanced evaluation scenarios.

3. **New Experiment on LLaVA-1.5-7B Multimodal Model in Appendix G.8**. `reviewer F2TP`  
	- Conducted an ablation study to evaluate the HarmBench dataset with teacher model llava-1.6-vicuna-7b.  
	- **Results**: Achieved a remarkable 24.27% DSR improvement. This  indicates our method can successfully extend to Large Multimodel Models.

3. **New Experiment on LLaMA3-70B Model in Appendix G.10.** `reviewer F2TP`

	-   Conducted an ablation study to apply DAPA to Hermes-3-Llama-3.1-70B-Uncensored using Meta-Llama-3-70B-Instruct as the teacher model with the Advbench dataset.
	-   **Results**: Observed a 10% improvement in DSR, validating DAPA’s effectiveness on Larger Language Models (70B) and Large Multimodal Models.
	    

4. **New Experiment on Fine-tuned Large Language Model in Appendix G.11.** `reviewer Z3rx`

	-   Conducted an ablation study on the ShareGPT unfiltered dataset, fine-tuning the Llama-2-7B model using the Llama-2-7B-chat template with QLoRA.
	    
	-   **Results**: After applying DAPA alignment, the Defense Success Rate (DSR) improved significantly, increasing from 10.16% to 18.40%, highlighting DAPA’s effectiveness in enhancing alignment on fine-tuned models.
	    

5. **New Memory Space Interpretability Analysis with ROME in Appendix F** `reviewer EaeF` `reviewer F2TP`
	
	-   Conducted an ablation study to visualize four additional harmful prompts to analyze memory space usage.
	    
	- **Results**: Confirmed that MLP layers predominantly store alignment knowledge and revealed consistent storage patterns across different prompt types. These enhancements strengthen the evidence for the method's scalability, robustness, and interpretability across a wide range of models and datasets.
	
6. **New Memory Space Analysis in Appendix G.12** `reviewer Z3rx`

	-   Conducted an ablation study to provide additional visualization results of memory space.
	    
	- **Results**:  Hidden states in the middle layers significantly influence model output, with MLP layers exerting a higher indirect effect compared to attention layers. These findings are consistent with observations in Figure 3, reinforcing the critical role of middle-layer dynamics in model performance

-   **Revised Table 2 Caption:** Shorten the caption for Table 2. `reviewer GrXu`
    
-    **Revised Table 3:** Shorten the caption for Table 3. `reviewer GrXu`
    
-    **Revised Figure 6**: Shorten the caption for Figure 6. `reviewer GrXu`
    
-    **Revised Table 6**: Shorten the caption for Table 6. `reviewer GrXu`
    
-    **Revised Table 10**: Shorten the caption for Table 10. `reviewer GrXu`
    
-    **Revised Table 15**:Shorten the caption for Table 15. `reviewer GrXu`
    
-    **Revised Sec 4.4**: Put the HarmfulQA results in appendix G.1. and add more description of the ablation study `all reviewers`
</content>
</thread_object>

</thread>
<thread>
creation date:
1732689123435

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Rebuttal/Revision Response - Continued

comment:
We thank `Reviewer Z3rx` for providing valuable suggestions on additional experiments, which helped enhance the competitiveness of our paper. As a result, we have updated our paper with a new version. In case the reviewer requires a new categorisation for our paper revision, we have posted the updated revision here.

**Revised Appendix Mapping:**
- **New HarmBench Dataset experiment results in Appendix G.8**

	- Conducted an ablation study to evaluate 321 out-to-date harmful questions from HarmBench.

	- **Results**: Observed a 4.16% DSR improvement, confirming the method's robustness against challenging datasets.
- **New Advanced Jailbreak Adversary Testing GCG in Appendix G.11**.  
    
    -   Conducted an ablation study to test to Llama-2 family models with GCG
    -   **Results**: DSR increased significantly, from 29.2% to 38.83%, showcasing effectiveness against sophisticated jailbreak adversarial. These results confirm that our method is dataset-agnostic and demonstrates robust performance on diverse and advanced evaluation scenarios.

- **New Advanced Jailbreak Adversary Testing AutoDAN in Appendix G.12**.  
    
    -   Conducted an ablation study to test to Llama-2 family models with AutoDAN
    -   **Results**: DSR increased significantly, from 18.69% to 30.1%, showcasing effectiveness against sophisticated jailbreak adversarial. These results confirm that our method is dataset-agnostic and demonstrates robust performance on diverse and advanced evaluation scenarios.

- **Revise Advanced Jailbreak Adversary Testing AutoDAN in Appendix G.10**.  
    
    -   Conducted an ablation study to test to Llama-2 family models with GPTFuzzer
    -   **Results**: DSR increased significantly, from 7.97% to 24.59%, showcasing effectiveness against sophisticated jailbreak adversarial. These results confirm that our method is dataset-agnostic and demonstrates robust performance on diverse and advanced evaluation scenarios.

-  **New Memory Space Analysis in Appendix G.15**  
    
    -   Conducted an ablation study to provide additional visualization results of the memory space, incorporating 12 new visualizations in the updated version. 
        
    -   **Results**: Hidden states in the middle layers significantly influence model output, with MLP layers exerting a higher indirect effect compared to attention layers. These findings are consistent with observations in Figure 3, reinforcing the critical role of middle-layer dynamics in model performance.

- **New Experiment on LLaVA-1.5-7B Multimodal Model in Appendix ~~G.8~~ G.9.**
- **New Advanced Jailbreak Adversary Testing in Appendix ~~G.9~~ G.10.**
- **New Experiment on LLaMA3-70B Model in Appendix ~~G.10~~ G.13.**
- **New Experiment on Fine-tuned Large Language Model in Appendix ~~G.11~~ G.14.**
- **New Memory Space Analysis in Appendix ~~G.12~~ G.15.**
- **New Appendix ~~G.14~~ G.16**: Example of DAPA on LLava-1.5 model.
- **New Appendix ~~G.13~~ G.17**: Model name and corresponding labels.

Thank you again for the review. We are writing to kindly ask if you have any additional comments regarding our response. In addition, if our new experiments address your concern, we would like to kindly ask if you could consider raising the score.
</content>
</thread_object>

</thread>


Metareview: