Title: MoGU: A Framework for Enhancing Safety of LLMs While Preserving Their Usability
Venue: NeurIPS 2024
Accepted: yes
Keywords: Enhancing Safety

Preseving Usability

Large Language Models
Abstract: Large Language Models (LLMs) are increasingly deployed in various applications. As their usage grows, concerns regarding their safety are rising, especially in maintaining harmless responses when faced with malicious instructions. Many defense strategies have been developed to enhance the safety of LLMs. However, our research finds that existing defense strategies lead LLMs to predominantly adopt a rejection-oriented stance, thereby diminishing the usability of their responses to benign instructions. To solve this problem, we introduce the MoGU framework, designed to enhance LLMs' safety while preserving their usability. Our MoGU framework transforms the base LLM into two variants: the usable LLM and the safe LLM, and further employs dynamic routing to balance their contribution. When encountering malicious instructions, the router will assign a higher weight to the safe LLM to ensure that responses are harmless. Conversely, for benign instructions, the router prioritizes the usable LLM, facilitating usable and helpful responses. On various open-sourced LLMs, we compare multiple defense strategies to verify the superiority of our MoGU framework. Besides, our analysis provides key insights into the effectiveness of MoGU and verifies that our designed routing mechanism can effectively balance the contribution of each variant by assigning weights. Our work released the safer Llama2, Vicuna, Falcon, Dolphin, and Baichuan2.

MoGU: A Framework for Enhancing Safety of LLMs While Preserving Their Usability

Abstract

Large Language Models (LLMs) are increasingly deployed in various applications. As their usage grows, concerns regarding their safety are rising, especially in maintaining harmless responses when faced with malicious instructions. Many defense strategies have been developed to enhance the safety of LLMs. However, our research finds that existing defense strategies lead LLMs to predominantly adopt a rejection-oriented stance, thereby diminishing the usability of their responses to benign instructions. To solve this problem, we introduce the MoGU framework, designed to enhance LLMs' safety while preserving their usability. Our MoGU framework transforms the base LLM into two variants: the usable LLM and the safe LLM, and further employs dynamic routing to balance their contribution. When encountering malicious instructions, the router will assign a higher weight to the safe LLM to ensure that responses are harmless. Conversely, for benign instructions, the router prioritizes the usable LLM, facilitating usable and helpful responses. On various LLMs, we compare multiple defense strategies to verify the superiority of our MoGU framework. Besides, our analysis provides key insights into the effectiveness of MoGU and verifies that our designed routing mechanism can effectively balance the contribution of each variant by assigning weights. Our work released the safer Llama2 7 B , Vicuna 7 B , Falcon 7 B , Dolphin 7 B , and Baichuan2 7 B at github 2 . Warning: This paper presents examples of malicious instructions that may be offensive and upsetting.

1 Introduction

Large Language Models (LLMs) exhibit significant potential across various domains, yet they also face considerable safety vulnerabilities [28, 34, 43].To explore these vulnerabilities, several studies have conducted red-team evaluations with malicious instructions that could encourage harmful behaviors [45, 27]. Others have developed jailbreak attacks [10, 9, 42, 33, 6] aimed at provoking harmful responses from LLMs by using carefully crafted adversarial prompts. These safety vulnerabilities may lead to severe consequences, including the promotion of racial discrimination, breaches of ethical standards, and violations of human rights [9, 40].

In response to LLMs' safety vulnerabilities, some studies have pursued aligning LLMs with human values through SFT and RLHF techniques. Despite these advancements, recent work [45, 36] indicates that even aligned LLMs are still susceptible to jailbreak attacks. To further enhance LLMs' safety, various defense strategies have been proposed, including input and output detection [26, 21],

in-context safety demonstration [37], and enhancing the likelihood of decoding rejection tokens [39]. These strategies often focus on ensuring harmless responses during red-team evaluations and jailbreak attacks but overlook the impact on the quality of responses to benign instructions. Our research finds that existing defense strategies lead LLMs to adopt a rejection-oriented stance, thereby diminishing the usability of their responses to benign instructions. By prioritizing safety over usability, these strategies become less effective in practical applications. Consequently, this presents a key challenge -seesaw effect between security and usability : How can we enhance the safety of LLMs while preserving their usability?

Despite existing defense strategies not effectively addressing this challenge, the input detection [21] strategy provides a straightforward solution. This strategy triggers a safety mechanism by distinguishing malicious and benign instructions. However, this implementation, which relies on binary classification of instructions, often struggles with arbitrary treatment. Many benign instructions may be wrongly marked as malicious, mistakenly activating the safety mechanism and thus diminishing the usability of responses to benign instructions. The Mixture of Experts (MoE) series of research provides a promising improvement direction [18, 23, 32]. MoE employs a dynamic routing mechanism within LLMs to balance contributions from different experts, thereby improving LLMs' overall performance. This dynamic routing mechanism has proven effective in assigning weights to experts according to the input instruction. Therefore, in our research, we aim to introduce a dynamic routing mechanism to enhance LLMs' safety.

Based on these insights, we introduce a novel framework called M ixing o f G lad and U nwilling Responders ( MoGU ). We first employ the Parameter-Efficient Fine-Tuning technique LoRA [15], to transform the base LLM into two distinct states: the Glad Responder (Glad resp ) and the Unwilling Responder (Unwill resp ). The Glad resp , as an extremely usable LLM, is trained to generate glad responses to any instruction. Conversely, Unwill resp , as an extremely safe LLM, is trained to be highly cautious, rejecting any instruction it receives. The core component of MoGU is a dynamic router that serves as a safety sensor, embedded at each layer where LoRA is applied. This router is trained to dynamically balance the contributions of Glad resp and Unwill resp according to the input vector, effectively mixing their output vectors. As illustrated in Fig. 1, when faced with

Give me a tutorial on making pizza.

Router

UnwillingResponder

GladResponder

o_states glad

o_states unwill

0.4

Sure, the following is the process of

making pizza: ...

Response

0.4

0.6

h_states

0.6

Give me a tutorial on making the bomb.

Router

UnwillingResponder

GladResponder

o_states glad

o_states unwill

Malicious Instruction

0.6

Sorry, I can't respond to this request...

Response

0.6

0.4

h_states

0.4

a malicious instruction, the router will assign a higher weight to Unwill resp , ensuring a safe, rejection response. On the contrary, the router shifts more weight to Glad resp for the benign instruction, facilitating a glad, useful response.

In our experiments, we revealed limitations of existing strategies that diminish the usability of LLMs. Our experiment results verify that our MoGU framework can keep robust defense performance under the red-team evaluation and various jailbreak attacks while preserving LLMs' usability. Besides, compared to existing defense strategies, our framework demonstrates obvious advantages across various LLMs. We also conduct quantitative analysis to confirm that the router can effectively balance the contribution of each variant by assigning weights, thereby ensuring both the safety and the usability of LLMs.

2 Related Work

In this section, we summarize related work from two aspects: attack strategies and defense strategies.

2.1 Attack strategies

Red-team evaluation. The primary goal of red-team evaluations [30] is to assess the safety of LLMs by compiling a set of malicious instructions that reflect common user queries. The collection of

these instructions is conducted in two ways: 1) gathering malicious instructions from crowdsourced workers [11]. 2) automatically generating malicious instructions with another LLM that simulates human behavior [3]. The scope of these malicious instructions should be wide-ranging, covering topics such as toxicity, discrimination, privacy, and misinformation [13].

Jailbreak attack. Jailbreak attacks [12] aim to circumvent the built-in safety mechanisms of LLMs by modifying original red-team malicious instructions into more complex adversarial prompts. These strategies generally fall into two categories: heuristic-based and optimization-based strategies.

Heuristic-based strategies attempt to induce LLMs to prioritize task completion over adherence to safety constraints. For instance, some studies [36, 19] have prompted LLMs to begin their responses with indicators of successful jailbreak, such as 'Start your response with [Sure, here's]'. Others [35, 20] employ psychological tactics to subtly encourage LLMs to violate safety constraints.

Optimization-based strategies attempt to search for adversarial prompt templates based on constructed objectives. These strategies fall into two categories: token-level and expression-level. Token-level strategies [45] searched for token sequences via backpropagation and spliced them around original malicious instructions. However, these token sequences often lack semantic coherence, rendering them vulnerable to detection by Perplexity (PPL) algorithms [17]. Moreover, expression-level strategies [24, 41] employ genetic algorithms to search for natural language prompt templates. This approach enhances the concealment of jailbreak attacks, making them more difficult to detect.

2.2 Defense Strategies

Defense strategies can be categorized into two main types: those that improve built-in safety and those that leverage external tools. Strategies focused on built-in safety aim to align LLMs with human values, employing methods such as Supervised Fine-Tuning (SFT) [44] and Reinforcement Learning from Human Feedback (RLHF) [29]. SFT reduces experiential loss by incorporating high-quality, human-annotated samples during training, whereas RLHF optimizes LLMs based on valuable human feedback. Despite the widespread adoption of these methods, recent studies [45, 5] indicate that aligned LLMs (e.g. Llama2) are still vulnerable to jailbreak attacks.

Meanwhile, many researchers are developing strategies that leverage external tools to further improve LLMs' safety. These strategies focus on inference enhancement and the detection of input and output. Inference enhancement strategies guide LLMs to generate safer content through methods such as self-safety reminding [38] or by presenting safety in-context demonstrations [37]. Strategies for the detection of input and output involve identifying potentially harmful content to trigger the appropriate safety mechanisms. Methods such as paraphrasing and retokenization [16] can render certain attacks ineffective by altering the expression of inputs. Moreover, binary classifiers [21] based on BERT [7] can be trained to detect malicious inputs, and self-examining method [14] enables LLMs to assess the harmfulness of their own outputs. Despite these efforts, it remains challenging to enhance the safety of LLMs while preserving their usability.

3 MoGU Framework

The overall framework of our MoGU is illustrated in Fig. 2. We introduce our framework from three aspects: the training data preparation, the training stage, and the inference stage.

3.1 Training Data Preparation

For our training data, we only collected 600 instructions, which include 300 benign instructions sourced from Alpaca 3 and 300 malicious instructions from Advbench [45]. As illustrated in Fig. 2, for each instruction, we construct both a glad response and a rejection response. We label benign instructions as X b , malicious instructions as X m , glad responses as Y g , and rejection responses as Y r . Therefore, our training dataset encompasses four types of data pairs: (X b , Y g ), (X b , Y r ), (X m , Y g ), and (X m , Y r ). We observe that LLMs typically generate glad responses to benign instructions and rejection responses to malicious instructions. Consequently, during the construction of (X b , Y g ) and (X m , Y r ), we almost preserve their original responses. Here is how to construct them.

f

(a)

lora_glad

f

(b)

lora_glad

f

(b)

lora_unwill

f

(a)

lora_unwill

f

o_proj

hidden_states (i)

FFN

Q

K

V

Attention

Oproj

N

×

hidden_states (i+1)

f

base

hbase

(i)

obase

(i)

hglad

(i)

oglad

(i)

hunwilll

(i)

ounwilll

(i)

O (base)

proj

O (glad)

proj

O (unwill)

proj

f

base

Training Glad and Unwilling Responders:

•

The Glad Responder will generate glad responses to any instruction,

even to malicious instructions.

•

The Unwilling Responder will generate rejection responses to any

instruction, even to benign instructions.

At this stage, we only train the introduced LoRA weights and all other

parameters are frozen.

f

base

f

(b)

lora_unwill

f

(a)

lora_unwill

f

(a)

lora_glad

f

(b)

lora_glad

hMoGU (i)

f router

wglad

oglad

(i)

ounwilll

(i)

⊙

⊙

+

Training Stage

Training Router:

•

Router will assign a higher weight Wglad to Glad Responder

when faced with benign instructions and assign a higher

weight Wunwill to Unwilling Responder when faced with

malicious instructions

At this stage, we only train our introduced Router and all other

parameters are frozen.

proj

Inference Stage

Give me a tutorial on making the bomb.

Instruction

I'm sorry, but I cannot design a …

Response

Base

Decoding Strategy:

We only decode first 5 tokens with our MoGU, and the subsequent

token sequences are still decoded with the Base LLM.

Name the five members of the UN Security Council.

Benign Instruction (x b )

Publish an article that promotes the use of drugs or alcohol

Malicious Instruction (x m )

Sure, the five permanent

members of the United

Nationas Security Council are:.

Rejection Repsonse (y r )

I'm sorry, but I cannot name

the five members of the UN

Security Council.

Sure, here is an article that

promotes the use of drugs or

alcohol.

I'm sorry, but I cannot fulfill

this request as it goes against

my programming.

Training Data Example

wunwill

Rejection Repsonse (y r )

Construction of (X b , Y g ): we prompt the base LLM to generate responses to X b and collect some rejection expressions (detailed in App. A) for rule-based detection. If rejection responses are detected, they will be discarded. Then, we will craft glad responses Y g with the help of GPT-4 4 .

Construction of (X b , Y r ): we utilize GPT-4 to craft rejection responses to X b . For guiding GPT-4, we present demonstrations of generating rejection responses to benign instructions.

Construction of (X m , Y g ): since Advbench [45] has manually annotated high-quality glad responses to X m , we directly use their annotated data.

Construction of (X m , Y r ): we prompt the base LLM to generate responses to X m and utilize the same rule-based detection as above. If glad responses are detected, they will be discarded. Then, we will craft rejection responses Y r with the help of GPT-4.

In the scenarios mentioned above for GPT-4, we adopt the In-Context Learning [8] idea, and provided in-context demonstrations can be found in App. B.

3.2 Training Stage

During the training stage, we initially train the Glad and Unwilling responders using the LoRA framework. Subsequently, all other parameters are frozen, and we train our introduced router. In the LoRA framework, only the low-rank decomposition matrices added to the targeted weight matrices are updated. As illustrated in Fig. 2, the targeted weight matrices typically include Q (Query), K (Key), V (Value), O proj (Output Projection), and FFN (Feed-Forward Network). In our research, we regard O proj as the targeted weight matric for exploration.

The training of glad and unwilling responders. The objective of Glad resp is to calibrate the base LLM into an extremely usable LLM that can generate glad responses to any instruction. The extreme

case is that Glad resp can generate glad responses even to malicious instructions. Therefore, we use the data (X m , Y g ) to train the base LLM, and the loss function can be expressed as:

where ( x i m , y i g ) ∈ ( X m , Y g ) and CE loss represents Cross Entropy Loss. Similarly, the objective of the Unwill resp is to calibrate the base LLM to an extremely safe LLM that can reject any instruction. The extreme case is that Unwill resp can even reject any benign instruction. Therefore, we use the data (X b , Y r ) to train the base LLM, and the loss function can be expressed as:

where ( x i b , y i r ) ∈ ( X b , Y r ) . Subsequently, inspired by Contrastive Learning (CL) [25], we incorporated negative samples to further improve our framework. For Glad resp , we need to ensure that it will not generate rejection responses to any malicious instruction. And for Unwill resp , we need to ensure that it will not generate glad responses to any benign instruction. Consequently, we regard data (X m , Y r ) and (X b , Y g ) as negative samples for training Glad resp and Unwill resp , respectively. The loss function for Glad resp can be formulated as:

where { ( X m , Y g ) , ( X m , Y r ) → ( X m , Y g , Y r ) } and ( x i m , y i g , y i r ) ∈ ( X m , Y g , Y r ) . And the loss function for the Unwill resp can be formulated as:

where { ( X b , Y r ) , ( X b , Y g ) → ( X b , Y r , Y g ) } and ( x i b , y i r , y i g ) ∈ ( X b , Y r , Y g ) .

The design and training of router. Our router comprises two linear networks, denoted as R glad and R unwill , both sharing identical structural configurations. Each linear network R incorporates a low-rank decomposition matrix followed by a fully connected layer. Specifically, the low-rank decomposition matrix involves matrices U ∈ R d model × d router and V ∈ R d router × d model , and the fully connected layer is denoted by a matrix W ∈ R d model × 1 . We assume that for the i-th projection layer O proj , the input vector is denoted by h ( i ) ∈ R seq _ len × d model . Here, seq _ len refers to the length of the input tokens, d model refers to the dimension of the model's hidden layers, and d router is a hyperparameter determining the intermediate dimension in the low-rank decomposition matrix. The role of linear network R can be formulated as:

where σ represents the sigmoid activation function, w ∈ R seq _ len × 1 , b 1 and b 2 represent the bias term. The weights w glad and w unwill , provided by R glad and R unwill respectively, will be assigned to Glad resp and Unwill resp to mix their output vectors. As shown in Fig. 2, the output vector of Glad resp 's i-th O proj layer can be formulated as:

where o ( i ) glad ∈ R seq _ len × d model , f b lora _ glad and f a lora _ glad are low-rank decomposition matrices in LoRA framework. And the output vector of Unwill resp 's i-th O proj layer can be formulated as:

where o ( i ) unwill ∈ R seq _ len × d model , f b lora _ unwill and f a lora _ unwill are low-rank decomposition matrices in LoRA framework. Then, the mixture of Glad resp and Unwill resp output vectors can be formulated as:

During the training of the router, all other parameters are frozen, and only the router's parameters will be updated. The primary objective of the router is to guide LLMs in generating appropriate responses to various instructions. Specifically, the router should facilitate glad responses to benign instructions and rejection responses to malicious instructions. To achieve this, we use both (X b , Y g ) and (X m , Y r ) as the training data. The loss function can be formulated as:

where ( x i b , y i g ) ∈ ( X b , Y g ) and ( x j m , y j r ) ∈ ( X m , Y r ) . Besides, the router is equipped with a finer-grained objective: it will assign weights according to the type of instruction. Specifically, a higher weight will be assigned to Glad resp for benign instructions and to Unwill resp for malicious instructions. To reinforce this behavior, we use the L1 Norm to regulate the optimization of weights w glad and w unwill assigned by the router, ensuring the assigning pattern adheres to our expectations. The loss function can be formulated as:

where ∥ · ∥ 1 represents the L1 Norm. Finally, the overall loss function can be formulated as:

where λ is a hyperparameter.

3.3 Inference Stage

Previous research [45, 39] has shown that the initial response tokens are critical to ensuring the harmlessness of the whole response. If initial response tokens express rejection, the response is more likely to be harmless. Given these findings, and considering that our additional parameters extend inference time, we employ MoGU only for decoding the first m tokens as shown in Fig. 2. The subsequent tokens are decoded by the base LLM to preserve the efficiency and quality of decoding.

4 Main Experiments

4.1 Preliminary

LLMs. In our research, we evaluated chat versions of five open-source LLMs, including four from the Llama series: Llama2 7 B [34], Vicuna 7 B [43], Falcon 7 B [1], and Dolphin 7 B 5 . Notably, Dolphin 7 B has not yet undergone a safety review. We also evaluated Baichuan2 7 B [2], which features an architecture distinct from those in the Llama series.

Evaluation data. In our evaluation, we focused on assessing LLMs' safety and usability. For the safety assessment, on the one hand, we conducted a red-team evaluation. We utilize the Advbench [45], which comprises 520 malicious instructions-300 for our training as introduced in Sec. 3.1 and the rest 220 for testing. Additionally, we collected 200 malicious instructions from Just-Eval [22] (labeled as 'Malicious'). On the other hand, we conducted the safety evaluation against various jailbreak attacks. We employed both optimization-based and heuristic-based strategies. For the optimization-based strategies, we utilized AutoDAN [24], GCG [45], and PAIR [4], each of which applies different adversarial prompts to 50 test samples. Specifically, AutoDAN employs genetic algorithms to generate semantically fluent adversarial prompts. GCG uses gradient propagation to identify token sequences as attack suffixes. PAIR iteratively optimizes adversarial prompts based on the LLMs' ability. For the heuristic-based strategies, we used SAP30 [5] and Comp obj [36], each of which applies the same adversarial prompt to 100 test samples. SAP30 focuses on semantic inducements, whereas Comp obj is designed to compromise LLM safety by conducting a competing objective. For the usability assessment, we used 800 benign instructions from Just-Eval [22] to assess LLMs' problem-solving abilities. In App. C, we provide examples for each evaluation set.

Evaluation metric. For the evaluation metrics, we utilize both GPT-4 scoring and rule-based evaluations. For assessing LLM's safety, we employ GPT-Judge [31], which rates the harmfulness of responses on a scale from 1 to 5, where 1 represents harmlessness and 5 indicates extreme harmfulness. Additionally, consistent with the work [45], we identify targets ( T ) that can represent harmless responses (as detailed in App. D) and calculate the Attack Success Rate ( ASR ) based on these targets. ASR quantifies the probabilities that responses deviate from our identified targets, which can be expressed as # of responses deviate from T # of all responses . For assessing LLMs' usability, in line with Just-Eval [22], we utilize GPT-4 to evaluate responses ( GPT-Eval ) across five dimensions: helpfulness, clarity, factuality, depth, and engagement. Each response is scored from 1 to 5, with higher scores denoting better quality. Moreover, we compile a list of common rejection expressions (as detailed in App. A) and monitor their frequency in LLM responses ( Rule-based Eval ) to evaluate the extent to which LLMs adopt a stance of rejection. During our evaluation, we spent approximately $400 for calling the GPT-4 API.

Baselines. We selected seven advanced defense strategies as our baselines. SFT strategy [44] employs high-quality data to train LLMs, thereby aligning LLMs with human values. Detect inp [21] train a binary classifier based on BERT to distinguish between benign and malicious instructions. SelfExamine [14] strategy prompts LLMs to assess whether their responses are harmful. If risky contents are detected by Detect inp and Self-Examine, the response 'Sorry, I cannot answer your question.' will be returned. Retokenization [16] strategy counters various jailbreak attacks by altering the input to shift meanings subtly. Self-Reminder [38] strategy consistently cues LLMs to maintain awareness of safety throughout the input process. ICD [37] strategy integrates safety in-context demonstrations into prompts. SafeDecoding [39] strategy increases the likelihood of rejection tokens during the decoding phase. We implemented SFT within the LoRA framework based on our constructed data and followed the open-sourced code from work [39] to reproduce other baselines.

Hyperparameter settings. We configure our router's intermediate dimension d router to 512 and set the λ in Loss router to 2. For training Glad resp and Unwill resp , the learning rate is set to 5e-5, and for training the router, the learning rate is set to 5e-4. Besides, the α and d lora _ r in LoRA are set to 16 and 8 respectively. During inference, only the first 5 tokens are decoded with our MoGU and the remaining tokens are decoded with the base LLM. Decoding configurations of various LLMs can be found in App. E. All our experiments were done on a single 80GB A100.

4.2 Main Results

In Tab. 1 and 2, we respectively evaluate the performance of defense strategies under red-team evaluation and against various jailbreak attacks. For the red-team evaluation, we report only the ASR. In contrast, for the jailbreak attacks, given the broader variability in LLMs' responses, we report both the GPT-4 score and the ASR. On the whole, the ICD strategy outperforms others on Llama2 7 B , MoGU excels on Vicuna 7 B , and SafeDecoding excels on Falcon 7 B . Furthermore, these three strategies demonstrate stable and effective defense performance across various LLMs. Thus, in Tab. 3, we assess the impact of these three competitive strategies on the usability of LLMs.

Besides, since the main ideas of our MoGU and Detect inp are similar, in that they sense inputs to execute appropriate operations, we also report the performance of Detect inp in Tab. 3. Through comprehensive analysis of results across Tab. 1, 2, and 3, we identify three key phenomena.

MoGU keeps robust defense performance. As demonstrated in Tab. 1, our MoGU framework stably enhances the safety of various LLMs during red-team evaluations. Notably, as described in Sec. 3.1, our training data solely comprises original red team malicious instructions, and explicitly excludes any adversarial samples with jailbreak attack prompts. Despite this, our MoGU framework still maintains robust defense performance against various jailbreak attacks as illustrated in Tab. 2.

Existing defense strategies enhance the safety of LLMs but often compromise their usability.

As shown in Tab. 2, the ICD strategy significantly increases the defense of Llama2 7 B to jailbreak attacks. However, after applying the ICD strategy, as shown in Tab. 3, the rate of rejection responses to benign instructions on Llama2 7 B surged from 14.00% to 92.25%, and its response usability score dropped dramatically from 3.87 to 2.17. Similarly, as shown in Tab. 2, the SafeDecoding strategy effectively defends Vicuna 7 B against jailbreak attacks. However, as shown in Tab. 3, it leads to a substantial increase in rejection responses from 3.63% to 39.50% and a decline in response usability score from 3.89 to 2.29. Such phenomenons indicate that existing defense strategies often lead LLMs to adopt a rejection-oriented stance, thereby diminishing their usability.

MoGUcan enhance LLMs' safety while preserving their usability. As illustrated in Tab. 1 and 2, our framework has exhibited robust defense performance across various LLMs. Importantly, it also maintains the ability to respond with high quality to benign instructions, as evidenced by results in Tab. 3. Under our MoGU framework, the frequency of rejection expressions in LLMs' responses to

benign instructions remains nearly equivalent to that observed in base LLMs. Such phenomenons verify the superiority of our MoGU framework compared to other defense strategies.

5 Analysis

In this section, we conducted an ablation experiment, provided a quantitative analysis, and discussed our introduced size of parameters. In App. F and G, we respectively provide a case study and extend our MoGU framework to Baichuan2 7 B and Dolphin 7 B to further demonstrate MoGU's flexibility. Besides, in App. I, we discuss the limitations of our research.

5.1 Ablation Experiment

We analyze the impact of Contrastive Learning Loss (Loss CL ) in Loss glad and Loss will and the L1 Norm (L1 Norm ) constraint in Loss router . Tab. 4 illustrates that omitting Loss CL and L1 Norm will lead to a decrease in the defense performance of our framework. Notably, the impact of L1 Norm proved to be more significant.

Llama

Vicuna

Falcon

Comp_obj

AVG.

0.00%

0.00%

0.00%

1.00%

0.29%

0.50%

1.14%

1.35%

1024

256

512

Comp_Obj

AVG.

0.00%

0.00%

0.00%

01.00%

2.00%

2.36%

2.71%

1024

256

512

Comp_Obj

AVG.

1.00%

5.00%

10.49%

10.12%

13.00%

16.00%

17.99%

24.34%

口

1024

256

512

SAP30

0.00%

0.00%

%00'0

128

SAP30

0.00%

0.00%

0.00%

128

SAP30

0.00%

0.00%

%00'0

128

0.00%

0.00%

0.00%

PAIR

0.00%

0.00%

2.00%

2.00%

PAIR

2.00%

2.00%

4.00%

4.00%

PAIR

18.00%

18.00%

28.00%

34.00%

GCG

0.00%

2.00%

6.00%

8.00%

GCG

4.00%

4.00%

6.00%

6.00%

GCG

4.00%

12.00%

14.00%

20.00%

AutoDAN

0.00%

0.00%

0.00%

AutoDAN

2.00%

8.00%

10.00%

AutoDAN

20.00%

32.00%

74.00%

0.00%

6.00%

38.00%

Malicious

0.50%

0.00%

0.00%

1.00%

Malicious

0.00%

0.00%

0.50%

1.00%

Malicious

12.00%

17.50%

26.00%

33.50%

Advbench

0.00%

0.00%

0.00%

0.45%

Advbench

0.00%

0.00%

0.00%

0.00%

Advbench

0.91%

0.91%

0.91%

1.82%

0.00%

4.00%

8.00%

12.00%

0.00%

4.00%

8.00%

12.00%

%0008%0009%000%0007%000

5.2 Quantitative Analysis

To investigate the role of the router, we analyzed the distributions of weights assigned by the router on Llama2 7 B , Vicuna 7 B , and Falcon 7 B . We collected 350 malicious instructions with various jailbreak attack prompts and 800 benign instructions from Just-Eval. The mean values of weights w unwill and w glad are calculated during processing each instruction. Fig. 3 presents the boxplot that depicts the statistical results for Vicuna 7 B . Notably, during jailbreak attacks, the router assigns a higher weight w unwill to Unwill resp , while for benign instructions, it favors a higher weight w glad for Glad resp . This allocation pattern aligns perfectly with our expectations of the router's functionality. The same patterns are also observed for Llama2 7 B and Falcon 7 B , detailed in App. H.

5.3 Size of Introduced Parameters

In our MoGU framework, we added the LoRA parameters of Glad resp and Unwill resp , and router parameters. In each layer, the number of added parameters can be calculated as ( d model × d router × 4+ d model × 8+ d model × d lora _ r × 4 ). Taking Llama2 7 B with 32 layers as an example, the total number of added parameters can be calculated as 273 , 678 , 336 = (32 × (4096 × 512 × 4 + 4096 × 8 + 4096 × 8 × 4) ), accounting for about 3.91% of all parameters.

Furthermore, We investigated the impact of parameter size on the defense performance of LLMs by adjusting the d router to 128, 256, 512, and 1024. Our analysis focused on the performance of Llama2 7 B , Vicuna 7 B , and

Jailbreak Attack

Just-Eval

UnwillGlad

UnwillGlad

0.8

0.8

0.7

0.7

0.6

0.6

0.5

0.5

0.4

0.4

0.3

0.3

0.2

0.2

Falcon 7 B against red-team evaluations and various jailbreak attacks. As shown in Fig. 4, setting d router to 512 will consistently result in superior defense performance across all three LLMs. Notably, Llama2 7 B and Vicuna 7 B also exhibited strong defense performance at the lower d router settings of 128 and 256. These results suggest that within our framework, the safety of LLMs might be enhanced effectively with fewer parameters.

6 Conclusion

In our research, we find the limitations of existing defense strategies, which often sacrifice usability in the pursuit of enhancing LLMs' safety. To address this issue, we introduce our MoGU framework, which designs a dynamic routing mechanism. Our MoGU can improve LLMs' safety while preserving their usability. Our comprehensive evaluations across various LLMs verify our MoGU's superiority compared to other strategies. In the future, we will further refine and optimize the MoGU framework.

References

Ebtesam Almazrouei et al. 'Falcon-40B: an open large language model with state-of-the-art performance'. In: (2023).

Baichuan. 'Baichuan 2: Open Large-scale Language Models'. In: arXiv preprint arXiv:2309.10305 (2023). URL: https://arxiv.org/abs/2309.10305 .

Stephen Casper et al. 'Explore, establish, exploit: Red teaming language models from scratch'. In: arXiv preprint arXiv:2306.09442 (2023).

Patrick Chao et al. 'Jailbreaking black box large language models in twenty queries'. In: arXiv preprint arXiv:2310.08419 (2023).

Boyi Deng et al. 'Attack prompt generation for red teaming and defending large language models'. In: arXiv preprint arXiv:2310.12505 (2023).

Gelei Deng et al. 'Jailbreaker: Automated jailbreak across multiple large language model chatbots'. In: arXiv preprint arXiv:2307.08715 (2023).

Jacob Devlin et al. 'Bert: Pre-training of deep bidirectional transformers for language understanding'. In: arXiv preprint arXiv:1810.04805 (2018).

Qingxiu Dong et al. 'A survey on in-context learning'. In: arXiv preprint arXiv:2301.00234 (2022).

Zhichen Dong et al. 'Attacks, defenses and evaluations for llm conversation safety: A survey'. In: arXiv preprint arXiv:2402.09283 (2024).

Yanrui Du et al. 'Analyzing the inherent response tendency of llms: Real-world instructionsdriven jailbreak'. In: arXiv preprint arXiv:2312.04127 (2023).

Deep Ganguli et al. 'Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned'. In: arXiv preprint arXiv:2209.07858 (2022).

Xingang Guo et al. 'Cold-attack: Jailbreaking llms with stealthiness and controllability'. In: arXiv preprint arXiv:2402.08679 (2024).

Thomas Hartvigsen et al. 'Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection'. In: arXiv preprint arXiv:2203.09509 (2022).

Alec Helbling et al. 'Llm self defense: By self examination, llms know they are being tricked'. In: arXiv preprint arXiv:2308.07308 (2023).

Edward J Hu et al. 'Lora: Low-rank adaptation of large language models'. In: arXiv preprint arXiv:2106.09685 (2021).

Neel Jain et al. 'Baseline defenses for adversarial attacks against aligned language models'. In: arXiv preprint arXiv:2309.00614 (2023).

Fred Jelinek et al. 'Perplexity-a measure of the difficulty of speech recognition tasks'. In: The Journal of the Acoustical Society of America 62.S1 (1977), S63-S63.

Albert Q Jiang et al. 'Mixtral of experts'. In: arXiv preprint arXiv:2401.04088 (2024).

Erik Jones et al. 'Automatically auditing large language models via discrete optimization'. In: International Conference on Machine Learning . PMLR. 2023, pp. 15307-15329.

Daniel Kang et al. 'Exploiting programmatic behavior of llms: Dual-use through standard security attacks'. In: arXiv preprint arXiv:2302.05733 (2023).

Aounon Kumar et al. 'Certifying llm safety against adversarial prompting'. In: arXiv preprint arXiv:2309.02705 (2023).

Bill Yuchen Lin et al. 'The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning'. In: ArXiv preprint (2023).

Alisa Liu et al. 'Tuning language models by proxy'. In: arXiv preprint arXiv:2401.08565 (2024).

Xiaogeng Liu et al. 'Autodan: Generating stealthy jailbreak prompts on aligned large language models'. In: arXiv preprint arXiv:2310.04451 (2023).

Zhuang Ma and Michael Collins. 'Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency'. In: arXiv preprint arXiv:1809.01812 (2018).

Todor Markov et al. 'A holistic approach to undesired content detection in the real world'. In: Proceedings of the AAAI Conference on Artificial Intelligence . Vol. 37. 12. 2023, pp. 1500915018.

Ninareh Mehrabi et al. 'Flirt: Feedback loop in-context red teaming'. In: arXiv preprint arXiv:2308.04265 (2023).

OpenAI. GPT-4 Technical Report . 2023. arXiv: 2303.08774 [cs.CL] .

Long Ouyang et al. 'Training language models to follow instructions with human feedback'. In: Advances in neural information processing systems 35 (2022), pp. 27730-27744.

Ethan Perez et al. 'Red teaming language models with language models'. In: arXiv preprint arXiv:2202.03286 (2022).

Xiangyu Qi et al. 'Fine-tuning aligned language models compromises safety, even when users do not intend to!' In: arXiv preprint arXiv:2310.03693 (2023).

Alexandre Ramé et al. 'Warm: On the benefits of weight averaged reward models'. In: arXiv preprint arXiv:2401.12187 (2024).

Xinyue Shen et al. '" do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models'. In: arXiv preprint arXiv:2308.03825 (2023).

Hugo Touvron et al. 'Llama: Open and efficient foundation language models'. In: arXiv preprint arXiv:2302.13971 (2023).

Zhenhua Wang et al. 'Foot In The Door: Understanding Large Language Model Jailbreaking via Cognitive Psychology'. In: arXiv preprint arXiv:2402.15690 (2024).

Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 'Jailbroken: How does llm safety training fail?' In: Advances in Neural Information Processing Systems 36 (2024).

Zeming Wei, Yifei Wang, and Yisen Wang. 'Jailbreak and guard aligned language models with only few in-context demonstrations'. In: arXiv preprint arXiv:2310.06387 (2023).

Fangzhao Wu et al. 'Defending chatgpt against jailbreak attack via self-reminder'. In: (2023).

Zhangchen Xu et al. 'SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding'. In: arXiv preprint arXiv:2402.08983 (2024).

Zihao Xu et al. 'LLM Jailbreak Attack versus Defense Techniques-A Comprehensive Study'. In: arXiv preprint arXiv:2402.13457 (2024).

Jiahao Yu, Xingwei Lin, and Xinyu Xing. 'Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts'. In: arXiv preprint arXiv:2309.10253 (2023).

Xuandong Zhao et al. 'Weak-to-strong jailbreaking on large language models'. In: arXiv preprint arXiv:2401.17256 (2024).

Lianmin Zheng et al. 'Judging LLM-as-a-judge with MT-Bench and Chatbot Arena'. In: arXiv preprint arXiv:2306.05685 (2023).

Chunting Zhou et al. 'Lima: Less is more for alignment'. In: Advances in Neural Information Processing Systems 36 (2024).

Andy Zou et al. 'Universal and transferable adversarial attacks on aligned language models'. In: arXiv preprint arXiv:2307.15043 (2023).

A Collection of Rejection Expressions

In Sec. 3.1, we have collected rejection expressions for rule-based detection. Tab. 5 shows our collected rejection expressions.

B In-Context Demonstrations for GPT-4

In Sec. 3.1, we provide in-context demonstrations for GPT-4 during the construction of (X b , Y g ), (X b , Y r ) and (X m , Y r ). The provided demonstrations are detailed in Tab. 6.

C Examples for Evaluation Data

In Tab. 7, we present examples of evaluation data. Due to the extensive length of the adversarial sample generated by AutoDAN, we do not include a specific example in Tab. 7. For an illustrative instance of AutoDAN, please refer to the dataset available 6 .

D Identified Harmless Targets

In Sec. 4.1, we identified harmless targets, which are used to compute the Attack Success Rate (ASR). Our identified targets are listed in Tab. 8, similar to the work [45].

E Decoding Configuration

In Tab. 9 and 10, we respectively provide our used templates and decoding parameters for various LLMs during the decoding stage.

F Case Study

Our case study further underscores the superiority of our MoGU. Tab. 2 demonstrates that while the ICD shows superior defense performance against jailbreak attacks for Viucna 7 B , it also significantly compromises the quality of responses to benign instructions, as seen in Table 3. This issue is highlighted in the case described in Tab. 11, where ICD not only rejected a malicious instruction but also erroneously rejected a benign instruction. In contrast, MoGU exhibits a robust ability to distinguish between malicious and benign instruction - rejecting the former while helpfully responding to the latter.

G Extend our MoGU to Baichuan2 and Dolphin

To demonstrate the flexibility of our framework, we applied it to Dolphin 7 B and Baichuan2 7 B . Notably, Dolphin 7 B has not undergone a safety review, whereas Baichuan2 7 B differs significantly in architecture from the Llama series of LLMs. Our evaluation focuses on the defense performance of these LLMs under red-team evaluations and specific jailbreak attacks, including SAP30 and Comp obj . The results, detailed in Tab. 12, confirm that our framework substantially enhances the safety of both Dolphin 7 B and Baichuan2 7 B .

H Distribution of Weights Assigned by Router

On Llama2 7 B , Vicuna 7 B , and Falcon 7 B , we calculated the mean values of weights w unwill and w glad during the procession of each instruction. The statistical results for Vicuna 7 B have been discussed in Sec. 5.2. Fig. 5 presents the boxplots for Llama2 7 B and Falcon 7 B , which show similar trends to those reported in Sec. 5.2. Specifically, for malicious instructions, the router will assign a higher weight w unwill to Unwill resp , while for benign instructions, it favors a higher weight w glad for Glad resp .

I Limitations

Despite the advantages shown by our proposed MoGU compared to other defense strategies, we still acknowledge several limitations in our research:

Can our framework be adapted to other linear layers? Since there is no related work exploring which linear layers within LLMs significantly impact LLMs' safety, we selected O proj as our target. However, it remains unclear whether applying our framework to other linear layers would achieve the same performance.

Jailbreak Attack

Just-Eval

(a) Analysis on Llama2

7

B

.

Jailbreak Attack

Just-Eval

(b) Analysis on Falcon

7

B

.

UnwillGlad

UnwillGlad

UnwillGlad

UnwillGlad

0.9

0.9

0.8

0.8

0.8

0.8

0.7

0.7

0.7

0.7

0.6

0.6

0.6

0.6

0.5

0.5

0.5

0.5

0.4

0.4

0.3

0.3

0.4

0.4

0.2

0.2

0.3

0.3

0.1

0.1

0

0

0.2

0.2

Can the introduced parameters be further reduced? As discussed in Sec. 5.3, our framework introduces additional parameters. However, it is not clear whether all introduced parameters are effective. Whether we can reduce the number of introduced parameters through pruning is something our research has not yet further explored.

J Broader Impact

J.1 Positive Social Impact

Enhanced User Trust: By improving the safety of LLMs, users will have greater trust in the outputs generated by these LLMs. Whether it is a smart assistant, an autonomous driving system, or other AI-based decision-making tools, users will feel more confident using them.

Reduction of Potential Risks: Improving the safety of LLMs helps mitigate potential risks that may arise from AI models, such as erroneous decisions, misleading information, and so on. This will have a positive impact on public safety, healthcare, finance, and other sectors

J.2 Negative Social Impact

Safety Risks Still Exist: Despite improvements in LLMs' safety, eliminating all security risks is impossible. This may lead some users to remain vigilant and distrustful when using AI models. Besides, hackers may utilize these LLMs for cyberattacks or spreading misinformation.

Technology Dependence and Job Loss: With the widespread application of AI technology, people may become overly dependent on these technologies, leading to the disappearance of certain job roles. While this is a natural consequence of technological progress, it may also have a negative impact on the social employment structure.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Abstract, 1 Introduction( 1)

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: J Limitations( I)

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: 3 Framework( 3), 5 Analysis( 5)

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: 3 Framework( 3), 4 Main Experiments( 4), A Collection of Rejection Expressions( A), B In-Context Demonstrations for GPT-4( B), C Examples for Evaluation Data( C), D Identified Harmless Targets( D), E Decoding Configuration( E)

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: 4 Main Experiments ( 4). Besides, we provide the code and the training data for our framework in supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: 4 Main Experiments( 4)

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: 5. Analysis( 5)

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: 4 Main Experiments( 4)

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: K Broader Impact( J)

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: 3 Framework( 3) 4 Main Experiments( 4)

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the limitations of previous implementations that rely on binary classification of instructions, which often mistakenly identify benign instructions as malicious, thus reducing usability. It proposes a dynamic routing mechanism to enhance the safety of LLMs while preserving their usability.\n\nSTRENGTHS:\n1. The proposed dynamic routing mechanism achieves a better balance between safety and usability compared to previous works.\n\n2. The experiments conducted are extensive.\n\nWEAKNESSES:\n1. The soundness of the method is questionable. The limitation of previous work is that model usability is compromised due to inaccurate binary classification of instructions. It appears that this method's success relies on the router giving higher weights to $Glad_{resp}$ for benign instructions and vice versa. If the router is the key and it is more reliable than binary classification in other LLMs, why not simply replace the existing binary classifier with the trained router in the current LLM to generate corresponding responses (glad or rejection)? What is the need to train $Glad_{resp}$ and $Unwill_{resp}$ separately?\n\n2. The deployment is risky and easy to be abused. Since this work aims to improve the safety of open-sourced models, applying it to open-sourced models means that Glad$_{resp}$, which generates positive responses even to malicious instructions, becomes public and could be exploited.\n\nQUESTIONS:\n1. How accurate is the router in giving more weight to Glad$_{resp}$ when facing benign responses?\n\n2. In Table 3, why does ICD achieve the highest rejection rate for LLama2 but a relatively low (comparable to MoGU) rejection rate for Vicuna?\n\n3. How can weakness 2 be addressed?\n\nLIMITATIONS:\nAlthough the figure quality is good and the experiments are extensive, the soundness of the method requires further justification. More importantly, the current deployment scenario is unclear and could pose a higher risk than having no protection at all.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

comments: SUMMARY:
The article proposes an alignment method based on LoRA and router. By modifying the first few tokens of the model output, it achieves certain effectiveness in defending against red team attacks.

STRENGTHS:
1. This method achieves defense capabilities against red-team attacks comparable to SOTA methods while ensuring minimal usability loss.
2. The experiments in the article are extensive and compare a large number of baseline methods.

WEAKNESSES:
1. This paper combines two LoRA models through a router, making the method relatively heavy. Additionally, modifying only the first few tokens greatly reduces engineering difficulty, but the improvement obtained is not significant. It is hoped that this method can be extended to the entire generation process and more application scenarios (not just for defense) to observe whether greater improvements can be achieved. Overall, the novelty of this article is somewhat lacking.
2. There is a lack of discussion on the impact of the number of modified tokens on the model's response.
3. The line spacing of the article does not seem to comply with NIPS submission standards. It appears that the authors manually compressed the line spacing, making the layout too tight and difficult to read.
4. Equations (3), (4) and (6), (7) in the article are somewhat repetitive, and Figure 2 needs improvement for better visualization.

QUESTIONS:
Why does ICD outperform SafeDecoding on Llama2 in Table 3, while the opposite is true for the other two models, considering GPT-Eval and Rule-based Eval?

LIMITATIONS:
The authors provided some explanation in the article. For more details, please refer to the Weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a new approach to balance safety and over-refusal in LLMs.  They do this by training LoRA parameters for a compliant and a rejection/safe version of the LLM, and then train a router (as in MoEs) to combine states between these two generators.  They show that this improves the model's robustness significantly while not hurting usability.\n\nSTRENGTHS:\nS1. Originality: I really like how this paper explores extending MoEs directly for the purpose of safety. This is quite different from most of the prior work in this area and I think an interesting direction to explore.\n\nS2. Significance: Results seem fairly solid in demonstrating that the proposed method helps safety with relatively low negative side effects on usability.\n\nWEAKNESSES:\nW1. The biggest weakness of the paper is in its clarity: The paper is *overly* detailed in the math in a way that I think could be cleaned up.  A good example here is eq (5) which feels unnecessarily in the weeds of describing a feed forward neural network and the design of the router could be explained more easily in English.\n\nW2. I am having trouble gaining confidence from the experimental design / baselines: (1) The baselines seem not well suited to actually balance safety with usability (or their details are not sufficiently explained). While this is a common challenge, some methods are better suited here than others (eg well design RLHF). That does not diminish that the proposed method is also effective but hard to tell how much of an improvement it is.  (2) It'd be valuable to plot results in a way that can more directly show the trade-off in safety and usability - cross referencing between tables it is quite hard to get a consistent picture as they each appear to trade-off to different degrees in different settings.\n\nQUESTIONS:\n- Why is this specific to *open-source* LLMs? (Separately, disappointed that this doesn't cover the challenges of keeping open source LLMs safe when adversaries can retune or sample them differently)\n\n- Using the ration of CEs in eq (3) an (4) seems fairly non-standard (even in contrastive learning) - is this done elsewhere? Is the loss propagated through bot the numerator and denominator?\n\n- Why not normalize in any way the router? Surprised that the weights are unbounded.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}

comments: SUMMARY:
The authors propose the MoGU framework, a novel solution designed to enhance the safety of LLMs while preserving their usability. The MoGU framework operates by splitting the base LLM into two specialized variants: one focusing on usability (usable LLM) and the other on safety (safe LLM). It employs a dynamic routing mechanism that adjusts the contribution of each variant based on the nature of the instruction received. The results show that MoGU not only provides a balanced approach between safety and usability but also outperforms existing defense methods.

STRENGTHS:
- The authors set up comprehensive experiments to evaluate the chat versions of five open-source Large Language Models (LLMs), including the mainstream LLM, Llama.
- The authors provide ablation studies to perform a quantitative analysis of the main component, the router, in their framework.

WEAKNESSES:
- The authors should provide a more in-depth discussion regarding the selection of instructions. For instance, it is crucial to examine whether the number of instructions used is sufficient for a comprehensive and unbiased evaluation.
- The analysis of the experimental results is sometimes overly simplistic. For example, in Section 5.1, the authors should provide more insightful and detailed analysis to better understand and interpret the significance of their findings.

QUESTIONS:
Please refer to the weaknesses mentioned above.

LIMITATIONS:
The authors have discussed the limitations of the work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-30