Decoupled Alignment for Robust Plug-and-Play Adaptation

### 1 Introduction

We introduce Decoupled Alignment for Robust Plug-and-Play Adaptation (termed Dapa), a low-resource safety enhancement method for aligning large language models (LLMs).
Dapa aligns unaligned Large language models (LLMs) with ethical guidelines even without supervised fine-tuning (SFT) or reinforcement learning from human feedback (RLHF).

This innovation is practically urgent and important.
LLMs have been widely adopted in various applications recently, demonstrating their ability to generate high-quality human-like texts (Team et al., ; Touvron et al., ; Ivison et al., ).
However, the security of these models has become a significant concern due to the potential risks of generating harmful content (Wu et al., ; Yu et al., , ; Chao et al., ; Deng et al., ).
To align the LLMs with ethical guidelines, researchers have developed various methods to enhance their safety.
For example, the Llama-2-Chat (Touvron et al., ) and Gemma-it (Team et al., ) models have been extensively fine-tuned to improve their alignment performance.
However, these methods often require extensive computational resources or manual red-teaming, which can be costly and time-consuming (Team et al., ; OpenAI, ; Bai et al., ; Ganguli et al., ).
Thus, most of the LLMs finetuned from the pre-trained models by third-party developers do not undergo the alignment process (Xu et al., ; Chiang et al., ; Ivison et al., ), leaving them vulnerable to generating harmful content by users with malicious intent.
To combat these issues,
we seek motivations from knowledge distillation technologies (Xu et al., ; Hahn and Choi, ), where a teacher model’s knowledge is transferred to a student model.

Specifically, through numerical experiments Figure 3 and Figure 4,
we make two key detections:

- •
MLP Alignment. the alignment knowledge is stored in the Feed-Forward Network (FFN) layers, also referred to as MLP layers,
- •
Gate Alignment.
Further,
the gate layer within MLP layers is the key component that determines whether the model’s output is aligned with ethical guidelines.

Focusing on these two key components, we propose to use memory editing technology (Meng et al., , ) to transfer alignment knowledge from one aligned LLM to another unaligned LLM within the same family.
Surprisingly, by incorporating the aligned gate layers into unaligned LLMs, we dramatically improve their alignment performance.
This technique differs from other alignment strategies in two aspects:
computational and manual resources for model alignment through fine-tuning or reinforcement learning (Dai et al., ).
This work provides a timely, robust and economic solution to LLM safety enhancement, aiming for more efficient and low-cost alignment.

Figure: Figure 1: The Alignment Example of The Dapa on The Chinese-Alpaca-7B Model.
Refer to caption: x1.png

###### Contribution.

We present a low-resource safety enhancement method for aligning LLMs.

- •
Methodologically,
we propose Dapa (as shown Figure 1) to enhance the security of LLMs against harmful prompts.
Dapa utilizes memory editing technology to identify the memory space responsible for alignment performance.
Specifically,
we present a delta debugging-based search algorithm to address the challenge of pinpointing the memory space (gate layers) responsible for alignment performance.
This allows us to locate the alignment-related modules for memory editing via knowledge distillation.
Then,
we perfrom surgery migrating alignment-related modules from one aligned model to unaligned model to achieve cheap yet effective satefy enhancement.
As a result, Dapa achieves near-optimal alignment performance for LLMs without requiring extensive computational resources or fine-tuning.
- •
Empirically, we apply Dapa to 17 models from three popular LLM families (LLama2, Mistral, and Gemma), achieving significant safety enhancements with minimal computational effort (adapting at most 8.11% model parameters).
Specifically, the Dapa-aligned models show an average 14.41% increase in Defense Success Rate (DSR), with one of Gemma’s models achieving a dramatic increase of 51.39%.
Surprisingly, we achieve this while largely maintaining the core functionalities of the models. We evaluate the models’ performance using various assessment techniques, including cosine similarity scores, model perplexity, few-shot prompting, and Chain-of-Thought (CoT) methodologies.
The average degradation in perplexity is only 1.69, and the average drop in model reasoning ability is only 2.59%.

###### Organization.

§ 2 includes a description of knowledge editing technology in our adaption.
§ 3 includes the algorithm to search critical memory space responsible for model alignment
§ 4 includes the numerical evaluation results to show the efficiency of our adaption.
§ 5 includes a conclusion and a discussion of future directions.

#### Related Works

###### Model Alignment.

With the developments of the LLMs (Team et al., ; Touvron et al., ; Bai et al., ), concerns about their security have become significant (Weng and Wu, ; Yu et al., ). Among these concerns, the potential risks of generating harmful content have attracted the most attention.
To counteract the potential risks of harmful outputs, developers often engage in fine-tuning during safety training to decrease the likelihood of such outputs (Wu et al., ; Touvron et al., ; Ganguli et al., ; Bai et al., ).
Current methods for safeguarding LLMs against jailbreak attacks include Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), and Supervised Fine-Tuning (SFT) (Rafailov et al., ; Peng et al., ; Ouyang et al., ).
However, these fine-tuning methods are both slow and costly.
Many practitioners are exploring ways to lower the expenses associated with alignment fine-tuning (Wang et al., ; Yao et al., ), yet costs remain substantial.

###### Memory Editing.

Knowledge editing focuses on altering specific behaviors of LLMs (Huang et al., ; Meng et al., , ), and can be divided into three primary paradigms (Yao et al., ). The first paradigm edits the memory during the inference stage (Wei et al., ; Zheng et al., ; Mitchell et al., ), employing memory retrieval or in-context learning for modifications. The second paradigm adjusts model parameters and structures during the training stage (Meng et al., , ). The third paradigm utilizes associative memory models such as the Modern Hopfield Network (Hu et al., , , ; Wu et al., , ; Hu et al., ; Ramsauer et al., ) to edit model memory effectively. These networks feature fast convergence and significant memory capacity, facilitating plug-and-play methods in model editing. Subsequent efforts utilize knowledge editing to detoxify LLMs. (Wang et al., ) explores the use of contextual semantics to allocate memory space, employing memory editing techniques to adjust the relevant memory areas. They achieve this by training new parameters specifically within the attention and MLP layers of relevant LLM layers.
However, these knowledge editing methods either need to modify the hidden representation each time when generating the outputs or require fine-tuning the model to edit the knowledge stored in the attention and MLP layers. Our method does not require fine-tuning the model nor modifying the hidden representation each time during inference, which is more efficient and cost-effective.

### 2 Memory Editing

To locate the association of ethical memory in the parameters of an autoregressive LLM, we begin by analyzing and identifying the specific hidden states that have the strongest correlation with ethical memory.

Figure: Figure 2: The Architecture of Transformer Models. We describe the architecture of Transformer models utilized by state-of-the-art LLMs such as Llama (Touvron et al., ) and Gemma (Team et al., ). Many of these models employ activation functions like SwiGLU (Shazeer, ) or GELU (Hendrycks and Gimpel, ) in their MLP layers. Each Transformer block combines an attention mechanism with MLP layers (comprising Up, Gate, and Down modules). Our figure illustrates the transition of the model’s hidden representation from the previous state to the next state.
Refer to caption: x2.png

An autoregressive transformer model $G$ is a type of language model that generates text by predicting the next token in a sequence given the previous tokens.
For each input $\mathbf{X}$, the model maps the input to a sequence of tokens $\mathbf{S}=[\mathbf{s}_{1},\mathbf{s}_{2},\dots,\mathbf{s}_{T}]\in\mathbf{X}$ , which are then fed into the transformer model.
Within the transformer model, the $i$-th tokens are first embedded into a sequence of hidden states $h_{i}^{(l)}$.
The final output $y=\text{decode}(h_{T}^{(L)})$ is generated by the decoder layer from the last hidden state.

In Figure 2, we visualize the internal computation of $G$ as a grid of hidden states $h_{i}^{(l)}$.
Each layer $l$ (left $\rightarrow$ right) of the transformer model adds a self-attention mechanism $a_{i}^{(l)}$ and local MLP $M_{i}^{(l)}$ from previous layers.
Recall that, in the autoregressive cases, tokens only draw information from previous tokens:

$$ $\displaystyle\mathbf{h}_{i}^{(l)}$ $\displaystyle=\mathbf{h}_{i}^{(l-1)}+\mathbf{a}_{i}^{(l)}+\mathbf{M}_{i}^{(l)} ,\quad\mathbf{a}_{i}^{(l)}=\text{attn}^{(l)}(\mathbf{h}_{1}^{(l-1)},\dots, \mathbf{h}_{T}^{(l-1)})$ $\displaystyle\mathbf{M}_{i}^{(l)}$ $\displaystyle=\mathbf{W}_{\text{down}}^{(l)}\sigma(\mathbf{W}_{\text{gate}}^{( l)}\gamma(\mathbf{a}_{i}^{(l)}+\mathbf{h}_{i}^{(l-1)})\cdot\mathbf{W}_{\text{ up}}^{(l)}\gamma(\mathbf{a}_{i}^{(l)}+\mathbf{h}_{i}^{(l-1)})$ $$

Each layer’s MLP is a three-layer neural network parameterized by $\mathbf{W}_{\text{up}}$, $\mathbf{W}_{\text{gate}}$, and $\mathbf{W}_{\text{down}}$, along with a SwiGLU (Shazeer, ) or GELU (Hendrycks and Gimpel, ) activation function in several popular LLMs, such as LLama (Touvron et al., ), Gemma (Team et al., ).
For further background on transformers, we refer to (Vaswani et al., ).

###### Storage of Alignment Knowledge.

We first use knowledge editing technology (Meng et al., ) to identify where the alignment knowledge is stored in the model. We use one unethical question as a prompt to Llama-2-7B-chat.
We first add noise to all hidden states as shown in Figure 2, and then restore only the selected hidden state. We then measure the difference in output probability between the corrupted run (adding noise to all hidden states) and the corrupted run with one hidden state restored, referred to as the indirect effect of the selected hidden state. The higher the indirect effect, the more critical the hidden state is to the model’s output probability. We iteratively apply this process to all hidden states to identify the hidden states that have the most significant impact on the model’s output probability, and show the results in Figure 3. We could observe that the hidden states in the middle layers of the model have the most significant impact on the model’s output, and the MLP layers have a higher indirect effect than the attention layers. This aligns with the findings in (Meng et al., ). We show more numerical results in Appendix. The results confirm that the alignment knowledge is mainly stored in the middle MLP layers of the model.

Figure: Figure 3: Visualizing Attention, MLP, and All Modules on Memory Space. We visualize the influence of unethical prompt tokens on the results using the aligned LLama-2-7B-chat model to identify memory space. This includes examining the effects on attention, MLP, and all modules.
Refer to caption: x3.png

To better understand the impact of each module in the MLP layers towards the alignment knowledge, we customize the knowledge editing technology (Meng et al., ) to visualize the indirect effects of different MLP modules: the gate, up, and down projections.
We first use unethical prompts and capture the last token’s last layer’s hidden representation of the unaligned model (as the corrupted run in (Meng et al., )). Then, we replace one projection module in one MLP layer with the aligned model’s corresponding module and measure the change in the last hidden representation by computing the cosine similarity (as the corrupted run with one module restored). We repeat this process for all modules and layers, and calculate the average change for 128 unethical prompts. The results are shown in Figure 4. We observe that the gate projection has the most significant impact on the model’s last token hidden representation, followed by the up projection. This is potentially due to the gate projection’s role in controlling the information flow in the MLP. Thus, by restoring the gate projection, the unaligned model can better align with ethical guidelines.

Figure: Figure 4: Impact of Different MLP Modules on Hidden Representation. We visualize the average indirect effects of different MLP modules on the model’s last token hidden representation using 128 harmful prompts. Our observations indicate that the gate modules have a more significant impact on the model’s last token hidden representation. Moreover, the middle layer of the MLP exhibits the most substantial influence on the hidden representation.

### 3 Delta Debugging

Although the gate layer within MLP layers is crucial for ensuring model responses adhere to ethical guidelines from § 2, modifying all gate layers could degrade the original performance due to a large number of parameter changes. We propose a strategy to efficiently identify the optimal memory space for targeted modifications, enhancing alignment while preserving performance.

We incorporate delta debugging (Zeller and Hildebrandt, ) in our strategy. Delta debugging is a systematic approach that automates the debugging process by identifying the smallest set of changes responsible for a program’s failure. It methodically reduces the set of changes, testing progressively smaller subsets until pinpointing the precise cause of the failure. In Dapa, we consider it a program failure when LLMs provide an unethical response to an unethical question. To demonstrate how delta debugging works in Dapa, let $\mathbf{S}\in\mathbb{S}$ be a memory space where $\mathbb{S}$ is the universe memory of all MLP modules.
A policy is defined by the function $\pi:\mathbb{S}\rightarrow\{0,1\}$, where if $\pi(\mathbf{S})=1$, it indicates that the memory space $\mathbf{S}$ is beneficial for enhancing alignment, and if $\pi(\mathbf{S})=0$, it indicates that the memory space $\mathbf{S}$ does not contribute to improving alignment. Given an aligned model memory space $\mathbf{S}$ and policy $\pi$, we aim to find the smallest memory space $\mathbf{S}^{*}\in\mathbb{S}$ in the aligned model which can most efficiently improve the unaligned ability to defend the jailbreak. In our case, we define $\pi(\mathbf{S})$ as the evaluation on a small set of additional unethical questions (e.g., 5% of preserved data). If the model provides ethical responses to all these questions, we set $\pi(\mathbf{S})=1$; otherwise, $\pi(\mathbf{S})=0$.

We next briefly describe the delta debugging process in our aligner, as shown in Algorithm 1. Given the input memory space of aligned model $\mathbb{S}$, number of partition $n=2$ and a list of memory space set $L$
of $\mathbb{S}$. we first split the memory space into $n$ partitions. We then check if there exists a partition $s_{i}$ such that $\pi(s_{i})=1$. If such a partition exists, we update the memory space to $s_{i}$ and update $n=2$. Otherwise, we check if there exists a partition $s_{i}$ such that $\pi(L\setminus s_{i})=1$. If such a partition exists, we update the memory space to $L\setminus s_{i}$ and set $n=n-1$. If neither of the above conditions are met, we double the number of partitions $n$. We repeat this process until $n$ is greater than the number of partitions in the memory space. Finally, we return the memory space $\mathbf{S}^{*}$ corresponding to the updated memory space $L$.
The worst-case complexity of this algorithm is $\mathrm{O}(|S|^{2})$.

Figure: Figure 5: Example of LLama-2-7b Model Memory Space Search.
Refer to caption: x5.png

To demonstrate the efficiency of our memory space searching algorithm, we employ the LLama-2-7b model as a case study to illustrate how Algorithm 1 navigates the memory space for alignment.
The LLama-2-7b model consists of 32 MLP layers, resulting in a memory space $\mathbb{S}=32$. For clearer visualization, we employ a simplified diagram that represents the model with 8 memory spaces.
Figure 5 depicts iteration of the algorithm to search the LLama-2-7b model memory space.

Figure: Algorithm 1 Memory Search Algorithm in Dapa

### 4 Experimental Studies

We perform a series of experiments to evaluate Dapa in enhancing the alignment performance of unaligned models against unethical prompts, in § 4.1.
We also assess the impact of the Dapa aligner on the model’s performance in § 4.2, including linguistic capabilities and reasoning abilities.
Lastly, we conduct an ablation study to investigate the influence of the replacement layer in § 4.3, including the model’s safety and overall performance.

###### Models and Parameter Efficiency.

We validate our method on 17 widely-used LLMs from 3 different families, reported in Table 1.
These models include both foundational and fine-tuned models, with the fine-tuning approach including SFT, DPO, and RLHF.
Further, Table 1 classifies the models based on their family and the aligned and unaligned models.
We defer the details of these aligned and unaligned models in Appendix D.
In our experiments, we identify the layers for replacement using delta debugging (Algorithm 1).
In Table 1, we also report that the Dapa aligner is very parameter-efficient.
Dapa not only updates an average of 6.26% of parameters accross 3 model families,
it also updates as little as 3.25% parameters in the commonly used LLama-2-7b.

**Table 1: Model Families Employed in the Experiments. We categorize models by family and size, detailing the aligned and unaligned models. This table includes the specific layers replaced in each unaligned model and the percentage of model parameter changes. The Dapa aligner alters only an average of 6.26% of the model parameters, with as little as 3.25% change in parameters.**
| Family | Size | Aligned Model | Unaligned Model | Replace layers | Average Parameter change |
| --- | --- | --- | --- | --- | --- |
| llama-2 | 7b | llama-2-7b-chat | llama-2-7b, chinese-alpaca-2-7b | [3,7] | 3.25 % |
| 13b | llama-2-13b-chat | llama-2-13b, chinese-alpaca-2-13b,<br>redmond-Puffin-13B | [5,12] | 4.32 % |  |
| Mistral | 7b | mistral-7B-instruct | mistral-7B, openHermes-2-mistral-7b,<br>dolphin-2.2.1-mistral-7b, zephyr-7b-alpha | [9,18] | 8.11 % |
| mistral-7B-forest-dpo,<br>dolphin-2.6-mistral-7b-dpo, openchat-3.5 | [7,15] | 7.31 % |  |  |  |
| gemma | 2b | gemma-2b-it | gemma-2b, gemmalpaca-2B | [12,16] | 6.69 % |
| 7b | gemma-7b-it | gemma-7b, gemma-7b-ultrachat-sft,<br>gemma-orchid-7b-dpo | [7,13] | 6.19 % |  |

**Table 2: Comparing Dapa in 3 Common LLM Families. We demonstrate the enhancement in alignment abilities for unaligned models through our Dapa aligner across 17 models, measured by the Defense Success Rate (DSR). Furthermore, we exhibit the linguistic performance of our model post-application of the Dapa aligner. We report the average perplexity and Cosine Similarity score with variance omitted as they are all $\leq$ 2%. In every scenario, Dapa achieves a significant increase in DSR, approximately 14.41% on average and up to 51.39%. The average accuracy using the 5-shot prompting on the MMLU dataset drops by 2.06% and the average drop in perplexity is 1.69. In all settings, Dapa delivers a significant improvement in DSR, without substantially affecting the original capabilities of the unaligned model.**
| Family | Model Name | DSR | Perplexity | MMLU | Cosine Similarity |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Before | After | Before | After | Before | After |  |
| Llama-2 | chinese-alpaca-2-7b | 82.03 | 87.50 | 7.54 | 7.46 | 38.71 $\pm$ 0.41 | 37.43 $\pm$ 1.42 | 0.88 |
| Llama-2-7b | 37.16 | 42.19 | 4.77 | 4.78 | 36.37 $\pm$ 1.01 | 39.30 $\pm$ 0.00 | 0.79 |  |
| Llama-2-13b | 37.50 | 46.09 | 4.28 | 4.28 | 34.74 $\pm$ 2.46 | 37.08 $\pm$ 1.33 | 0.76 |  |
| chinese-alpaca-2-13b | 70.31 | 85.16 | 5.63 | 5.60 | 48.77 $\pm$ 0.70 | 47.60 $\pm$ 1.07 | 0.91 |  |
| Redmond-Puffin-13B | 22.66 | 47.66 | 4.30 | 4.30 | 30.06 $\pm$ 0.88 | 32.38 $\pm$ 1.22 | 0.89 |  |
| Mistral | Mistral-7B | 21.09 | 25.78 | 4.58 | 4.60 | 45.38 $\pm$ 1.66 | 47.72 $\pm$ 0.70 | 0.76 |
| OpenHermes-2-Mistral-7b | 33.59 | 46.88 | 5.00 | 5.02 | 41.29 $\pm$ 0.81 | 42.46 $\pm$ 1.22 | 0.88 |  |
| dolphin-2.2.1-mistral-7b | 24.22 | 41.41 | 5.18 | 5.19 | 60.12 $\pm$ 0.41 | 58.25 $\pm$ 1.05 | 0.90 |  |
| zephyr-7b-alpha | 24.22 | 32.81 | 5.11 | 5.11 | 54.04 $\pm$ 1.53 | 56.73 $\pm$ 0.41 | 0.88 |  |
| mistral-7B-forest-dpo | 19.38 | 15.62 | 5.13 | 5.10 | 54.62 $\pm$ 0.88 | 54.04 $\pm$ 0.61 | 0.72 |  |
| dolphin-2.6-mistral-7b-dpo | 24.22 | 55.47 | 5.41 | 5.42 | 60.47 $\pm$ 0.20 | 62.69 $\pm$ 0.54 | 0.91 |  |
| openchat-3.5 | 58.68 | 67.19 | 5.15 | 5.10 | 61.40 $\pm$ 0.35 | 58.71 $\pm$ 0.41 | 0.89 |  |
| Gemma | gemma-2b | 22.05 | 73.44 | 7.92 | 24.15 | 33.57 $\pm$ 0.41 | 24.80 $\pm$ 2.06 | 0.33 |
| Gemmalpaca-2B | 37.01 | 51.56 | 9.92 | 22.00 | 40.94 $\pm$ 0.81 | 21.17 $\pm$ 1.42 | 0.51 |  |
| gemma-7b | 26.56 | 34.38 | 6.09 | 6.27 | 39.65 $\pm$ 1.75 | 42.11 $\pm$ 0.93 | 0.66 |  |
| gemma-7b-ultrachat-sft | 34.15 | 41.41 | 7.17 | 7.48 | 42.11 $\pm$ 0.00 | 29.24 $\pm$ 0.54 | 0.76 |  |
| gemma-orchid-7b-dpo | 21.88 | 35.16 | 7.22 | 7.42 | 42.26 $\pm$ 0.61 | 38.01 $\pm$ 0.88 | 0.76 |  |
| Average Change |  | 34.39 | 48.81 | 5.91 | 7.60 | 44.98 $\pm$ 0.88 | 42.92 $\pm$ 1.00 | 0.87 |

#### 4.1 Alignment Performance

To evaluate Dapa’s effectiveness in aligning unaligned models, we substitute the relative memory in 17 models with Dapa’s configuration and assess their performance in defending against jailbreak attacks. Because we set the response generation with deterministic, the variance of the evaluation is 0. As a result, we only need to run one evaluation for each model.

###### Dataset.

In our experiment, we use the AdvBench (Zou et al., ) to validate the performance of Dapa. It is a benchmark dataset that contains various unethical prompts to evaluate the alignment of language models, encompassing different categories such as violence, hate speech, and misinformation. We sample 128 prompts from the AdvBench dataset to form our evaluation dataset.

###### Metrics.

We employ the DSR as the primary metric to evaluate the alignment performance. For each unethical prompt, if the model provides a refusal or an ethical response, it is considered aligned. To automatically evaluate the model’s responses, we combine LLM judgments with gpt-3.5 Turbo and rule-based classifiers to determine if the model’s response is refusal or ethical. For LLM judgment, we use the system prompt provided in (Yu et al., ) to guide the LLM to evaluate the response. For rule-based classifiers, we employ and expand the keyword list like “Sorry, as an" or “As a responsible” provided in (Zou et al., ) to classify the response. If the response contains any of the keywords, it is classified as a refusal. A response is deemed aligned if it is classified as refusal or ethical by both the LLM and rule-based classifiers.

###### Results.

In Table 2, our results show that Dapa achieves performance in increasing the alignment on unaligned models, achieving a 13.77% average increase in DSR across all 17 models. Notably, the gemma-2b model achieves a significant 51.39%
increase in DSR. These improvements in DSR underscore Dapa aligner’s effectiveness in enhancing model safety against jailbreak prompts.

#### 4.2 Model Performance

To assess the model’s performance before and after Dapa alignment, we evaluate the generative and reasoning capabilities in a deterministic setting. For each pre-alignment and post-alignment model, we measure the model’s generative ability using perplexity and assess the response variation caused by the Dapa alignment through cosine similarity score.
We also validate the model’s reasoning ability by employing real-life question-answering and STEM problem-solving tasks, using Chain-of-Thought (CoT) (Wei et al., ) and few shot prompting approach. We conduct each evaluation three times and present the average and standard deviation for each metric.

**Table 3: Comparing Dapa with CoT Abilities in 3 Common LLM Families. We present an experiment assessing the impact on Chain of Thought (CoT) capabilities using Dapa, evaluated through the Exact Match (EM) score. Our Dapa aligner reduces the average EM of the Chain of Alignment (CoA) methodology on the Big-Bench datasets by 2.77%. This reduction indicates that our Dapa aligner substantially impacts the original reasoning abilities of the unaligned model.**
| Family | Model Name | TruthQA | GK | SocialQA |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Before | After | Before | After | Before | After |
| Llama-2 | chinese-alpaca-2-7b | 20.67 $\pm$ 2.08 | 24.67 $\pm$ 2.08 | 38.10 $\pm$ 7.05 | 40.00 $\pm$ 1.43 | 21.67 $\pm$ 2.31 | 19.67 $\pm$ 3.21 |
| Llama-2-7b | 36.67 $\pm$ 3.51 | 27.00 $\pm$ 3.51 | 58.57 $\pm$ 7.14 | 46.67 $\pm$ 5.95 | 22.33 $\pm$ 2.52 | 24.00 $\pm$ 7.21 |  |
| Llama-2-13b | 39.33 $\pm$ 2.52 | 24.67 $\pm$<br>4.93 | 64.76 $\pm$ 2.97 | 45.24 $\pm$ 5.95 | 39.33 $\pm$ 2.52 | 22.67 $\pm$ 3.06 |  |
| chinese-alpaca-2-13b | 35.33 $\pm$ 5.13 | 36.33 $\pm$ 5.51 | 40.48 $\pm$ 9.72 | 49.05 $\pm$ 6.44 | 35.33 $\pm$ 5.13 | 19.00 $\pm$ 3.61 |  |
| Redmond-Puffin-13B | 33.67 $\pm$ 0.58 | 24.67 $\pm$ 4.04 | 55.71 $\pm$ 4.29 | 41.43 $\pm$ 1.43 | 33.67 $\pm$ 0.58 | 19.00 $\pm$ 3.61 |  |
| Mistral | Mistral-7B | 34.00 $\pm$ 1.73 | 33.67 $\pm$ 2.08 | 79.05 $\pm$ 2.97 | 77.14 $\pm$ 2.47 | 39.33 $\pm$ 3.51 | 37.67 $\pm$ 2.08 |
| OpenHermes-2-Mistral-7b | 39.67 $\pm$ 3.51 | 42.33 $\pm$ 5.51 | 67.14 $\pm$ 1.43 | 71.43 $\pm$ 4.29 | 30.00 $\pm$ 2.65 | 40.00 $\pm$ 1.73 |  |
| dolphin-2.2.1-mistral-7b | 51.00 $\pm$ 4.00 | 48.33 $\pm$ 3.21 | 85.24 $\pm$ 2.18 | 85.71 $\pm$ 2.47 | 53.00 $\pm$ 2.52 | 53.00 $\pm$ 1.00 |  |
| zephyr-7b-alpha | 35.00 $\pm$ 1.00 | 42.67 $\pm$ 3.06 | 64.76 $\pm$ 7.87 | 71.90 $\pm$ 2.97 | 44.00 $\pm$ 3.21 | 46.00 $\pm$ 7.51 |  |
| mistral-7B-forest-dpo | 41.00 $\pm$ 3.00 | 47.33 $\pm$ 6.33 | 71.43 $\pm$ 3.78 | 75.71 $\pm$ 4.29 | 38.33 $\pm$ 6.03 | 40.00 $\pm$ 4.58 |  |
| dolphin-2.6-mistral-7b-dpo | 48.67 $\pm$ 2.08 | 46.33 $\pm$ 2.89 | 87.14 $\pm$ 2.47 | 90.00 $\pm$ 0.00 | 39.33 $\pm$ 3.51 | 30.00 $\pm$ 1.01 |  |
| openchat-3.5 | 49.67 $\pm$ 4.93 | 55.67 $\pm$ 1.53 | 84.76 $\pm$ 2.18 | 83.81$\pm$ 2.97 | 61.00 $\pm$ 6.56 | 56.00 $\pm$ 2.65 |  |
| Gemma | gemma-2b | 29.33 $\pm$ 5.77 | 29.00 $\pm$ 3.61 | 51.43 $\pm$ 3.78 | 43.81 $\pm$ 2.18 | 29.00 $\pm$ 3.61 | 15.67 $\pm$ 2.52 |
| Gemmalpaca-2B | 33.67 $\pm$ 3.21 | 31.67 $\pm$ 2.52 | 61.43 $\pm$ 1.43 | 52.38 $\pm$ 6.75 | 41.00 $\pm$ 4.58 | 16.33 $\pm$ 2.08 |  |
| gemma-7b | 49.33 $\pm$ 4.16 | 50.00 $\pm$ 3.00 | 88.10 $\pm$ 1.65 | 89.52 $\pm$ 4.12 | 42.00 $\pm$ 2.89 | 35.33 $\pm$ 2.52 |  |
| gemma-7b-ultrachat-sft | 27.67 $\pm$ 4.04 | 29.33 $\pm$ 3.51 | 68.10 $\pm$ 9.51 | 60.00 $\pm$ 9.90 | 13.33 $\pm$ 2.52 | 15.33 $\pm$ 3.21 |  |
| gemma-orchid-7b-dpo | 41.33 $\pm$ 2.08 | 39.33 $\pm$ 1.53 | 80.48 $\pm$ 2.18 | 79.52 $\pm$ 0.82 | 29.00 $\pm$ 3.61 | 38.33 $\pm$ 3.51 |  |
| Average Change |  | 38.00 $\pm$ 3.14 | 37.24 $\pm$ 3.45 | 67.45 $\pm$ 4.27 | 64.90 $\pm$ 3.95 | 36.04 $\pm$ 3.43 | 31.04 $\pm$ 3.24 |

###### Dataset.

We employ four real-world datasets: ShareGPT (Chiang et al., ), WikiText-2 (Merity et al., ), Big-Bench (et al., ) (TruthQA, General QA, SocialQA), and MMLU datasets (Hendrycks et al., ). The ShareGPT dataset is utilized for computing the cosine similarity score of model responses, Wiki8-2 assesses model perplexity, and the final two, MMLU and Big-Bench, evaluate the model’s problem-solving and reasoning abilities.

###### Metrics.

In our experiment, we evaluate the responses generated by both pre-alignment model and post-alignment model. We use cosine similarity to measure the impact of the aligner on model response generation. Additionally, we use perplexity for comparative analysis of the models’ generative capabilities.
A high cosine similarity score or comparable perplexity indicates using our aligner improves the defense success rate while maintaining the original performance. Additionally, to evaluate the model’s reasoning abilities, we administer real-life question-answering and STEM problem-solving tasks, measuring performance with the Exact Match (EM) metric.

###### Setup.

We assess post-alignment performance by examining reasoning capacity, response similarity, and perplexity.
In all experiments, we use the model both before and after the adapter in a deterministic output setting.
In the response similarity test, we compare the average similarity of responses on the same generated question. For comparing model responses, we embed responses from both models using the text-embedding-3-small model(^8^88[https://openai.com/blog/new-embedding-models-and-api-updates](https://openai.com/blog/new-embedding-models-and-api-updates)) and analyze 128 questions sampled from ShareGPT. In the perplexity test, we compute the perplexity score with Huggingface Evaluate(^9^99[https://huggingface.co/docs/evaluate/index](https://huggingface.co/docs/evaluate/index)) on Wiki8-2 dataset (Merity et al., ).
In assessing model reasoning capacity, we conduct tests using 5-shot prompting on the MMLU dataset (Brown et al., ) and Chain-of-Action (CoA) (Pan et al., , ) methodology on the Big-Bench dataset, excluding memory retrieval. We conduct each evaluation three times and present the average and standard deviation for each
metric.

**Table 4: Influence of Different Sets of MLP modules. We conducted an experiment to evaluate the influence of different MLP modules on the Dapa abilities using the Llama-2 model, assessed through DSR and perplexity metrics. The best results are highlighted in bold, and the second-best results are underlined. Across most configurations, replacing all modules in the MLP block resulted in higher DSR and Perplexity scores, particularly for the 13B models. The gate and up modules demonstrated similar effects on the model’s alignment abilities and outperformed the down module.**
| Model Name | DSR | Perplexity |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | gate (ours) | all | up | down | gate (ours) | all | up | down |
| chinese-alpaca-2-7b | 87.50 | 92.97 | 87.28 | 86.72 | 7.46 | 7.18 | 7.42 | 7.41 |
| Llama-2-7b | 42.19 | 31.25 | 42.19 | 37.50 | 4.78 | 4.86 | 4.77 | 4.78 |
| Llama-2-13b | 46.09 | 55.47 | 39.06 | 36.72 | 4.28 | 4.41 | 4.28 | 4.28 |
| chinese-alpaca-2-13b | 85.16 | 88.28 | 85.12 | 82.81 | 5.60 | 5.61 | 5.60 | 5.58 |
| Redmond-Puffin-13B | 47.66 | 100.00 | 50.78 | 46.09 | 4.30 | 4.42 | 4.30 | 4.30 |

**Table 5: Influence of Different Positions Memory. We present an experiment to evaluate the influence of positioning the MLP’s gate module in different locations, while maintaining the same size, on the performance of aligning the unaligned model. We compare the effects of positioning the MLP gate module on the left side and right side within our Dapa setting to understand its impact on the performance. The best results are highlighted in bold, and the second-best results are underlined. Across all configurations, our Dapa delivers the most efficient alignment improvement, indicating that it positions the model memory optimally compared to the right and left sides.**
| Model Name | Dapa (ours) | Left-most | Right-most |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | DSR | Perplexity | DSR | Perplexity | DSR | Perplexity |
| chinese-alpaca-2-7b | 87.50 | 7.46 | 85.16 | 7.46 | 82.81 | 8.05 |
| Llama-2-7b | 42.19 | 4.78 | 35.16 | 4.78 | 35.16 | 4.79 |
| Llama-2-13b | 46.09 | 4.28 | 38.28 | 4.28 | 36.72 | 4.30 |
| chinese-alpaca-2-13b | 85.16 | 5.60 | 75.78 | 5.64 | 74.22 | 5.65 |
| Redmond-Puffin-13B | 47.66 | 4.30 | 21.14 | 4.30 | 23.44 | 4.34 |

Figure: Figure 6: The Influence of Different Memory Space Size on LLama-2 Model We present an experiment to assess how varying memory space influence the alignment ability of the LLama-2 model. We evaluate the LLama-2-7b and chinese-alpaca-2-13b model using DSR and perplexity metrics across different memory spaces. As the memory space increases, the model’s alignment capability also improves. The only exception is when more than 11 layers of the llama-2-7b model are altered, leading to a significant deterioration in performance.
Refer to caption: x6.png

###### Results.

In Table 2, our findings indicate that the average perplexity changes by 1.69, with the LLama-2-13b model showing no change in perplexity. In one specail case, the Gemme 2b family’s models display the most significant increase in perplexity, at 16.23. Additionally, the average cosine similarity is 0.82, with Dolphin-2.6-mistral-7b-dpo achieving the highest similarity of 0.91. Those indicate that the system does not adversely affect the original capabilities of the language model. Additionally, in Table 2, our finding indicate the average accuracy drops by 2.06% using 5-shot prompting on the MMLU dataset. Most models exhibit only slight changes in accuracy. The
only exception is gemma-2b and gemma-7b-ultrachat-sft experience significant drops of 19.77% and 12.87%, respectively.

In Table 3, our results our results show a 2.77% average accuracy decrease using the CoA methodology on the Big-Bench datasets. In one exception, OpenHermes-2-Mistral-7B shows the most significant improvement, achieving a 10% increase in accuracy on SocialQA dataset, while Gemma-alpaca-2B shows the largest decrease, with a 24% decrease on the SocialQA dataset.

Overall, these findings regarding models’ perplexity, responses’ cosine similarity, and performance on the real-life question-answering and problem-solving tests indicate that the Dapa aligner does not significantly impair the models’ performance after using Dapa aligner.

#### 4.3 Ablation: Influence of Different MLP Module Setting

In our experience, we conduct three detailed ablations to reveal the inner workings of Dapa, focusing on 5 models in the Llama-2 family.

###### Dataset.

Building on the methodologies described in § 4.1 and § 4.2, our ablation study utilizes the AdvBench and WikiText-2 datasets.

###### Metrics.

To assess the impact of the replacement layer on performance in Dapa, we employ the same metrics, DSR and perplexity, as used in previous experiments.

###### Impact of Various MLP Parts in Dapa .

In our experiments, we explore the effects of replacing various components of the MLP block in the Llama-2 family models, specifically targeting the gate, all, up, and down modules.
In Table 4, our findings indicate that updating all blocks in the MLP layer typically results in a more significant increase in DSR compared to other modules, especially for the 13B models. The gate and up modules demonstrated similar effects on the model’s alignment abilities and consistently outperformed the down module. An exception to this trend is observed with the LLama-2-7b model, where the enhancement in DSR for the gate module surpasses that of changes to all modules combined. Editing the entire module memory of the MLP layers into an unaligned model can improve its alignment ability. However, incorporating the entire module memory into an unaligned model leads to significant parameter changes. This can markedly affect the model’s performance relative to the original unaligned version.

###### Impact of Various Memory Module in Dapa .

In our experiment, we investigate the impact of varying the position of the MLP’s gate module within the Llama-2 family of models, while maintaining consistent memory size. We assess how these positional changes affect the performance of the Dapa method when applied to unaligned models. We compare the effects of positioning the MLP gate module on the left side and right side within our Dapa setting to understand its impact on the system’s performance. As indicated in Table 5, the alignment capability of Dapa diminishes when the memory positions are shifted to the extreme left or right.

###### Impact of Various Memory Length in Dapa .

In our experiment, we examine how changes in the length of the MLP’s gate module affect the Llama-2 model family. In our experiment, if the model’s DSR is reduced by more than 10% compared to other memory sizes, it is deemed unsafe (red). Similarly, if the perplexity increases by more than 5% relative to other memory sizes, we consider that the editing may let the model become a hallucination (yellow). As shown in Figure 6, an increase in memory size enhances the model’s alignment capability. Additional visualization and experiment results are provided in Section E.2. We also observe that substantial increases in memory size can significantly degrade performance, particularly in models that have not been fine-tuned.

### 5 Discussion and Conclusion

We introduce the Decoupled Alignment for Robust Plug-and-Play Adaptation, Dapa, which edits the unaligned model memory to enhance the model’s defenses against jailbreak attacks. This method improves model alignment without the substantial computational expense typically associated with fine-tuning. It also efficiently identifies the optimal memory space for alignment. Visualizations confirm that the ethical boundary of model alignment is predominantly situated within the middle MLP’s gate layers. Empirically, Dapa achieves a 14.41% improvement in model alignment, reaching up to 51.39% in one of the Gemma family models, with an average parameter change of only 6.26%. Moreover, Dapa minimally impacts the model’s performance in generation and reasoning tasks.

###### Limitation and Future Work.

One limitation of our approach is the extent of memory space editing required. Although the average memory modification across three family models is 6.26%, popular model adapters like Lora (Hu et al., ) and QLora (Dettmers et al., ) typically require only about 1% of parameter changes. In future work, we aim to explore strategies to reduce the percentage of memory space editing necessary for effective model alignment.

###### Broader Impact.

Our proposal improves LLMs’ defenses against jailbreak attacks.
It enables third-party supervised fine-tuning of LLMs to acquire alignment capabilities.
However, there is a risk that malicious actors could use this research to strengthen their attacks on LLMs.
Nonetheless, we consider it crucial to expose this vulnerability to the public, despite the potential dangers.

### Acknowledgments

HL is partially supported by NIH R01LM1372201.
This research was supported in part through the computational resources and staff contributions provided for the Quest high performance computing facility at Northwestern University which is jointly supported by the Office of the Provost, the Office for Research, and Northwestern University Information Technology.
The content is solely the responsibility of the authors and does not necessarily represent the official
views of the funding agencies.
The content is solely the responsibility of the authors and does not necessarily represent the official
views of the funding agencies.

## Appendix

[sections]
\printcontents[sections] 1

### Appendix A Broader Impact

Our proposal improves LLMs’ defenses against jailbreak attacks.
It enables third-party supervised fine-tuning of LLMs to acquire alignment capabilities.
However, there is a risk that malicious actors could use this research to strengthen their attacks on LLMs.
Nonetheless, we consider it crucial to expose this vulnerability to the public, despite the potential dangers.

### Appendix B Ethical Considerations

Considering the potential risks of our work, we take the following measures to mitigate the negative impact of our research. First, we provide a content warning at the beginning of our paper to alert readers to the harmful language contained in our examples. Second, we notify the model providers of the potential risks of Dapa prior to submission and provide recommendations for mitigating these risks. Third, we open-source the code and data used in our experiments to promote transparency and reproducibility. Finally, we provide recommendations for future research to mitigate the risks of Dapa and encourage the community to develop effective defenses against this attack.

### Appendix C Experiment System and Implement Settings

We perform all experiments using a single NVIDIA A100 GPU with 80GB of memory and a 12-core Intel(R) Xeon(R) Gold 6338 CPU operating at 2.00GHz. Our code is developed in PyTorch and utilizes the Hugging Face Transformer Library for experimental execution. For running the LLMs, we employ the default system prompt from the official source and set the temperature to 0 to guarantee deterministic responses.

### Appendix D Unaligned Models Details

In our experiments, we categorize all unaligned models based on the fine-tuned techniques they employ, as outlined in Table 6.

**Table 6: Links to Hugging Face Pages of Unaligned LLMs Used in The Experiments.**
| Fine-tuned | Model | Hugging Face page |
| --- | --- | --- |
| RLHF | Openchat-3.5 | [openchat/openchat_3.5](https://huggingface.co/openchat/openchat_3.5) |
| Foundation Model | Llama-2-7b | [meta-llama/Llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b) |
| Llama-2-13b | [meta-llama/Llama-2-13b](https://huggingface.co/meta-llama/Llama-2-13b) |  |
| Gemma-2B | [google/gemma-2b](https://huggingface.co/google/gemma-2b) |  |
| Gemma-7B | [google/gemma-7b](https://huggingface.co/google/gemma-7b) |  |
| Mistral-7B | [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) |  |
| DPO | Mistral-7B-forest-dpo | [abhishekchohan/mistral-7B-forest-dpo](https://huggingface.co/abhishekchohan/mistral-7B-forest-dpo) |
| dolphin-2.6-mistral-7b-dpo | [cognitivecomputations/dolphin-2.6-mistral-7b-dpo](https://huggingface.co/cognitivecomputations/dolphin-2.6-mistral-7b-dpo) |  |
| gemma-orchid-7b-dpo | [macadeliccc/gemma-orchid-7b-dpo](https://huggingface.co/macadeliccc/gemma-orchid-7b-dpo) |  |
| SFT | chinese-alpaca-2-13b | [hfl/chinese-alpaca-2-13b](https://huggingface.co/hfl/chinese-alpaca-2-13b) |
| chinese-alpaca-2-7b | [hfl/chinese-alpaca-2-13b](https://huggingface.co/hfl/chinese-alpaca-2-7b) |  |
| Redmond-Puffin-13B | [NousResearch/Redmond-Puffin-13B](https://huggingface.co/NousResearch/Redmond-Puffin-13B) |  |
| dolphin-2.2.1-mistral-7b | [cognitivecomputations/dolphin-2.2.1-mistral-7b](https://huggingface.co/cognitivecomputations/dolphin-2.2.1-mistral-7b) |  |
| OpenHermes-2-Mistral-7b | [teknium/OpenHermes-2-Mistral-7B](https://huggingface.co/teknium/OpenHermes-2-Mistral-7B) |  |
| zephyr-7b-alpha | [HuggingFaceH4/zephyr-7b-alpha](https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha) |  |
| Gemmalpaca-2B | [mlabonne/Gemmalpaca-2B](https://huggingface.co/mlabonne/Gemmalpaca-2B) |  |
| Gemma-7b-ultrachat-sft | [CorticalStack/gemma-7b-ultrachat-sft](https://huggingface.co/CorticalStack/gemma-7b-ultrachat-sft) |  |

### Appendix E Supplementary Material for Experiments

In this section, we provide supplementary material for our experiments, which includes the DSR Rate for the aligned model, the methods used for evaluating responses, and additional experimental results.

#### E.1 Aligned Model DSR Rate

We present the DSR rate of the aligned model in AdvBench (Zou et al., ) to demonstrate the original performance of the aligned model in protecting LLMs against jailbreak attacks. We list the model name and their Defense Success Rate (DSR) in Table 7.

**Table 7: The DSR Rate of Aligned Model We detail the DSR performance across three model families and five aligned models, focusing on the effects of our memory editing techniques.**
| Family | Aligned Model Name | DSR |
| --- | --- | --- |
| Llama-2 | Llama-2-7b-chat(^10^1010https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) | 99.21 |
| Llama-2-13b-chat(^11^1111https://huggingface.co/meta-llama/Llama-2-13b-chat-hf) | 100.00 |  |
| Mistral | Mistral-7B-Instruct(^12^1212https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) | 75.59 |
| Gemma | gemma-2b-it(^13^1313https://huggingface.co/google/gemma-2b-it) | 97.64 |
| gemma-7b-it(^14^1414https://huggingface.co/google/gemma-7b-it) | 96.06 |  |

#### E.2 Additional Experimental Results

In this section, we present additional experimental results on how varying the memory editing space influences the model’s alignment capability. As shown in Tables 8 and 9, increasing the memory space generally enhances alignment abilities in the Llama2 7b model. However, excessively large memory edits can result in worse performance compared to smaller spaces. Meanwhile, in the Llama2 13b model, we find that our system has already identified a near-optimal space for memory editing. Also, we present additional experiments on the effects of varying memory space sizes on the LLama-2 model in Figure 7.

**Table 8: Additional Results: The Influence of Different Memory Space in LLama2 7b Models. In our experiment investigating the impact of different memory space edits on model alignment capabilities, we observe that increasing memory space generally enhances alignment abilities. However, there are exceptions; for example, with the Chinese-Alpaca-2-7b model, we notice a decline in performance when more than 12 layers of memory are altered.**
| Model Name | Memory Space Size |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | 13 | 12 | 11 | 9 | 7 | 5 (ours) | 3 | 1 |
| chinese-alpaca-2-7b | 89.84 | 91.41 | 90.62 | 86.72 | 88.28 | 87.5 | 87.5 | 83.59 |
| Llama-2-7b | 40.62 | 39.84 | 39.06 | 40.62 | 39.84 | 42.19 | 38.28 | 28.91 |

**Table 9: Additional Results: The Influence of Different Memory Space in LLama2 13b Models. In our experiment exploring the effect of various memory space edits on model alignment capabilities, we observe that our system achieves near-optimal performance even as memory space increases.**
| Model Name | Memory Space Size |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | 18 | 16 | 14 | 12 | 10 | 8 (ours) | 6 | 4 | 2 |
| Llama-2-13b | 37.50 | 41.41 | 39.06 | 37.50 | 43.75 | 46.09 | 41.41 | 45.31 | 42.97 |
| chinese-alpaca-2-13b | 87.50 | 86.72 | 86.72 | 86.72 | 83.59 | 85.16 | 80.47 | 80.47 | 78.12 |
| Redmond-Puffin-13B | 57.81 | 55.47 | 56.25 | 49.22 | 46.77 | 47.66 | 36.22 | 32.81 | 25.78 |

Figure: Figure 7: Additional Experiments on The Influence of Different Memory Space Size on LLama-2 Model We conduct an experiment to evaluate the impact of different memory space capacities on the alignment capabilities of the LLama-2 model. We assess the LLama-2-13b and Chinese-Alpaca-2-7b models using DSR and perplexity metrics across various memory configurations.
Refer to caption: x7.png

#### E.3 Response Evaluation

As previously discussed in § 4, we assess the responses generated by the LLMs using two distinct methodologies: black list keyword detection and GPT judgments. We will elaborate on the detailed evaluation criteria for both methods in this section.

###### Black List Keyword Detection .

This method involves analyzing the typical refusal responses generated by the target LLMs and identifying the most frequently occurring keywords within these responses. These keywords, which are listed in Table 10, serve as indicators for refusal responses. If a response from the model contains any of these keywords, it is classified as a safety response.

**Table 10: Description of Keywords by Family.**
| Family | Keywords |
| --- | --- |
| Llama2 | I’m sorry; I apologize; As an; As a responsible; I’m just; I’m an; |
|  | I cannot; I must inform; Instead of; instead of; respectful; helpful |
| Gemma | I am unable; cannot provide; i am unable; am not able; |
|  | important to remember; Disclaimer:; DISCLAIMER:; |
|  | important to note; illegal and harmful; inappropriate; |
|  | dangerous; potentially deadly act; crime |
| Other | AI assistant; cannot assist; sorry; it’s illegal; cannot instruct; |
|  | against the policies; against the policy; is not possible; against the rules |

###### GPT Judgment .

For this method, we utilize GPT-4-turbo due to its efficient performance. We outline the specific prompt used in Prompt E.2, where we instruct GPT to evaluate whether a response is harmful. If the response is deemed harmful, GPT should output 1. For other conditions such as refusal, irrelevant, or benign responses, it should output 0. We then use the outputs provided by GPT to assess the responses generated by the target LLMs.

#### E.4 Experimental Details of LLMs Reasoning Performance

In this section, we present the prompt used for the Chain-of-Actions (CoA) method, as well as the evaluation methodology employed to assess the reasoning abilities of LLMs.

##### E.4.1 Prompt of CoA.

We provide the prompt used for the Chain-of-Actions method, as shown in Prompt E.3

##### E.4.2 Performance Evaluation of LLMs Reasoning Abilities.

We employ GPT-3.5 Turbo (Bevilacqua et al., ) to evaluate the performance of LLMs in reasoning tasks. The specific prompt used for this evaluation is detailed in Prompt E.4. This allows us to assess the reasoning abilities of the LLMs.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: Please see my comments in the weakness.
soundness: 2
strengths: 1. The paper presents an interesting idea of identifying the important weights from the teacher models for safety alignment. By simply replacing student's weights with these weights from the teacher, we can achieve significant DSR improvements with minimum effort.
2. The paper introduces a simple algorithm based on previous works.
3. The experiments conducted are extensive including comparison with lots of models from different model family.
summary: This paper proposes an interesting idea of isolating the weights of LLMs that impact the safety alignment the most. It then applies these weights to unaligned models to improve their safety alignment. Various experiments demonstrate the effectiveness of the proposed method.
weaknesses: 1. The paper **doesn't follow the ICLR formatting guideline** by significantly changing the margins. It affects the readability of this paper.
2. The presentation of the paper is poor, e.g. some table and figure titles are too lengthy such as table 2 and 3 and figure 6. Lots of the content should be put into the main text instead of title. I don't know if this is caused by changing the margin or the authors intend to put them in the title.
3. Unnecessary highlight of the figure and table titles. This paper uses lot of bold and highlight which are not necessary.
4. While the authors tried to get more space by changing the margins, they split the tables which should be merged into one into multiple tables such as table 6 and 7, which occupies more spaces. 
5. The contribution is not enough. The main contribution of the paper is a naive search algorithm based on a calibration set. The results are likely highly dataset dependent. There is an experiment in the ablation study to test it on a different dataset, but it's not enough to justify the generability of the proposed method.
6. The evaluation dataset is only 128 prompts. It's very likely the results are overfit to the dataset or calibration set.
7. There is no comparison with the teacher model. E.g in table 1 and 2, after replacing the layers, have we achieved similar or same performances with the teacher model? If not, what's the gap there?
8. Although I understand the motivation of the authors, I don't the palpability of this method from the data provided by the authors. E.g. how is the proposed method compared to just simply fine-tuning using PEFT. Given the number of samples are small, it's possible that fine-tuning with PEFT is more efficient and effective. It's hard to tell from the provided data.
9. The paper claims to that the gate projection has the most significant impact, followed by up projection. But judging from figure 4, it looks like the down projection has a bigger impact than up projection, is this a typo?

Overall, I think this paper presents some interesting and promising findings, but it requires significant more data points to support the generability of the proposed idea.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: Please check the aforementioned issues in "Weaknesses" section. Some additional questions:

I wonder if the proposed method can be easily extended to large multimodal models.

Does the proposed method have potential interpretability?
soundness: 2
strengths: In general, the paper is well-written and easy to follow.

Unlike previous methods which require costly SFT or RLHF, the proposed method can be an additional module as a "plug-and-play approach" to various large language models with extra low-cost resources required. 

The proposed method is tested on models from different families, which shows the flexibility of DAPA that can potentially adapt to other models.

The use of knowledge distillation for alignment is very interesting.

In this paper, the authors present extensive testing with well-documented evaluation metrics across various models and achieve very competitive results. The ablation studies are also well set.
summary: In this paper, the authors are trying to address a challenging task which is to enhance the safety alignment of large language models without SFT and RLHF. The proposed method(called DAPA), unlike previous methods, only requires limited extra resources by employing a knowledge distillation approach. Two detection/alignment modules, i.e. MLP alignment and Gate alignment, are applied to store the alignment knowledge and check the model's output. The proposed method achieves competitive results with several large base models on the tested datasets.
weaknesses: The paper has tested the proposed DAPA mostly on small/mid-sized models (up to 13B models), there is uncertainty around its performance and resource efficiency when scaled to the large LLM models.

The proposed method introduces layer-specific edits without necessarily clarifying the deeper alignment process which may lead to alignment outcomes that are difficult to interpret or predict. This may make it hard to understand how the proposed module changes specific model behaviors.

As the ethical guidelines and alignment standards often evolve, I wonder if the continuous updates or recalibration of the alignment modules will become difficult, especially with large models.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. In the absence of a safer teacher model, how can effective model alignment be achieved? Are there alternative sources of knowledge or alignment strategies?
soundness: 2
strengths: 1. Clear writing and easy to follow.

2. Comprehensive experimental results demonstrate the effectiveness of the proposed method.

3. In my opinion, it is interesting and somewhat reasonable to enhance the safety capabilities of LLMs by identifying and editing unethical knowledge, with the help of another safer LLM.
summary: This paper proposes a novel low-resource safety enhancement method for LLM by firstly locating and then editing the specific knowledge, without the need of SFT or RLHF. Experiments on several datasets prove the effectiveness.
weaknesses: 1. In my opinion, the identification and editing of unethical knowledge within LLMs need to be conducted with great caution. The internal behavior of LLMs may be difficult to measure and analyze accurately, and different knowledge measurement tools may yield varying conclusions about the areas where knowledge is stored. Maybe this could affect the foundational research of this paper, although the conclusions drawn in terms of methodological design and experimentation are somewhat reasonable.  Have you considered using multiple knowledge measurement tools to cross-validate the findings?


2. Modifying knowledge in LLMs raises another issue: it may impact the LLM's ability to respond to general questions. The experimental section of the paper also supports this problem that editing unsafe knowledge can affect its normal response capabilities. Although the decreasing in general capabilities is not significant in experiments, in my opinion, the relationship between the two may not be a trade-off: enhancing safety capabilities does not necessarily require a decrease in general capabilities.

3. The safety knowledge alignment used in this paper depends on a safer teacher LLM from the same family. So, which model will teach these teacher LLMs to be safer? Furthermore, I believe that teaching LLMs to recognize harmful prompts and learn when to refuse to respond may be a more effective way to enhance their safety capabilities.

4. In my view, the ideas and motivations of this paper have some similarities with unlearning, such as in [1-3], both focusing on how to erase specific knowledge. It is necessary to include some discussion on the connection and difference between knowledge unlearning and editing.

[1] Safe Unlearning: A Surprisingly Effective and Generalizable Solution to Defend Against Jailbreak Attacks

[2] Towards Safer Large Language Models through Machine Unlearning

[3] Large Language Model Unlearning

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: My questions to authors are as follows.
- Among the unaligned models, there are several base candidates that have not been tuned to follow instructions. Have you launched additional instruction following on non-harmful datasets in addition to the relative memory substitution described in section 4.1? If not, the result with base models will be hard to compare against instruction-tuned ones.
- When extending Figure 3 to incorporate multiple harmful prompts, is it still true that the MLP's influence is the strongest?
- Can DAPA maintain its benefit when encountering more advanced jailbreak adversaries?
soundness: 2
strengths: The tackled problem, LLM safety and jailbreak, is timely. The proposed delta debugging algorithm for finding alignment knowledge within the MLP module is novel, and the ablation study as well as utility test is comprehensive.
summary: This paper proposes DAPA, an LLM safety enhancement method without SFT/RLHF. This is performed by first identifying the "alignment knowledge" stored in the MLP module of an aligned model, and transfer the knowledge to unaligned models. Empirically, the defense success rate was improved, and the utility was not sacrificed.
weaknesses: The main weaknesses I consider are as follows.
- DAPA requires an aligned LLM. Although claimed as "a low-resource safety enhancement method for aligning LLMs", the preliminary of DAPA algorithm for aligning an unaligned LLM is **the very existence** of an aligned LLM. This makes the stated contribution vague since it cannot serve as a substitution to SFT/RLHF, and thus cannot reduce the resource used for alignment or be an "economic solution to LLM safety enhancement".
- The safety test is weak. Throughout the paper, the defense success rate is measured using AdvBench **directly**. However, given the rapid development of LLM safety (or specifically, jailbreak), there are a suite of jailbreak adversaries proposed and been collected by some benchmark projects (e.g., [1] [2]). Thus, it is insufficient to claim safety improvement without testing the model performance against **any** of these representative adversaries.
- The claim of "the alignment knowledge is stored in the Feed-Forward Network (FFN) layers" lacks supportive evidence. This claim is deduced in the paper via Figure 3, where only **a single harmful prompt** is utilized to compare the influence of the MLP module and the attention module across layers. More quantitative results are needed to support this result.

[1] HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
[2] JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models