Negative-Prompt-driven Alignment for Generative Language Model

## Abstract

Abstract Large language models have achieved remarkable capabilities, but aligning their outputs with human values and preferences remains a significant challenge. Existing alignment methods primarily focus on positive examples while overlooking the importance of negative responses in guiding models away from undesirable behaviors. For instance, the widely-used alignment datasets reveals a scarcity of explicit negative examples that contradict human values, hindering its ability to discourage harmful or biased outputs during training. To address this limitation, we propose Neat , i.e., NEgative-prompt-driven AlignmenT, to introduce negative prompts to generate undesirable responses alongside positive examples during the optimization process. Neat explicitly penalizes the model for producing harmful outputs, guiding it not only toward desirable behaviors but also steering it away from generating undesirable, biased responses. This dual feedback mechanism enables better alignment with human preferences, crucial in contexts where avoiding harm is paramount. Starting from a pre-trained language model, Neat performs online alignment by incorporating a ranking loss derived from an expanded preference dataset containing both positive and negative examples. Extensive experiments validate Neat ’s effectiveness in significantly enhancing language models’ alignment with human values and preferences.

## 1 Introduction

Large language models (LLMs) such as GPT-4 (Ziegler et al., ) and Meta’s Llama series (Touvron et al., ), have made significant progress in natural language processing tasks (Ziegler et al., ; Yuan et al., ; Rae et al., ; Thoppilan et al., ), which are fueled by pre-training on vast amounts of data. These models are trained on the data created by humans possessing a wide range of goals, priorities, and skill levels. However, certain goals and skillsets represented in the training data may be undesirable to emulate. As a result, these language models could generate outputs that do not align with human values and produce harmful or biased responses. To address this challenge, aligning LLMs with human preferences has become a crucial area of research, where the objective is to ensure that models generate outputs consistent with human values or legal standards.

Reinforcement Learning from Human Feedback (RLHF) (Ziegler et al., ; Christiano et al., ; Stiennon et al., ) has been the dominant approach for aligning LLMs, as exemplified by models such as InstructGPT (Ouyang et al., ) and ChatGPT (^1^11https://openai.com/chatgpt/). While effective, the complexity of RLHF, especially in optimizing via reinforcement learning algorithms like PPO (Schulman et al., ), can be a major barrier to efficient and flexible implementation. Recently, Direct Alignment from Preferences methods, such as Direct Preference Optimization (DPO) (Rafailov et al., ), Ranking Responses from Human Feedback (RRHF) (Yuan et al., ) and Preference Ranking Optimization (PRO) (Song et al., ), have emerged as more straightforward alignment alternatives. These methods avoid the complexities of reinforcement learning, directly utilize the preference datasets to align the language models and operate the alignment in an offline setting.

A critical limitation of existing methods is their failure to explicitly capture the types of outputs that models should avoid. These methods primarily focus on positive examples while overlooking the importance of negative responses in guiding the model away from undesirable behaviors. For example, the widely-used preference datasets such as Anthropic’s Helpful and Harmless dataset (Bai et al., ) and UltraFeedback (Tunstall et al., ) dataset, lack sufficient negative response examples that contradict human values. Our quantitative analysis of one of the most widely used alignment datasets: HH-RLHF dataset (^2^22https://huggingface.co/datasets/Anthropic/hh-rlhf), reveals that only around 25% of the samples exhibit a score difference greater than 1.0 between the ”chosen” and ”rejected” responses, and less than 0.5% of the samples show a quantitative difference exceeding 5.0 points.(^3^33We use an open-source reward model to score each query-response pair, quantitatively measuring sample quality, then calculating the differences between two responses.) This scarcity of explicit negative examples that contradict human values hinders the dataset’s ability to effectively discourage models from generating undesirable, harmful, or biased outputs that misalign with human preferences during the training process.

To address these challenges, we propose Neat (NEgative-prompt-driven AlignmenT for generative language models), a novel approach that introduces negative prompts during the optimization process. It samples both negative, undesirable responses alongside helpful and harmless ones. Figure [1](#S1.F1) illustrates the core motivation behind Neat, highlighting the integration of negative-prompt-driven alignment. By explicitly penalizing the model for generating harmful outputs, Neat guides the model not only toward desirable behaviors but also steers it away from producing helpless, harmful, and biased responses. This dual online feedback mechanism enables the model to better align with human preferences, which is particularly crucial in contexts where avoiding harm is critical. Neat commences training from a base pre-trained language model and performs online alignment by incorporating a ranking loss derived from preference data. This loss encourages the language model to assign higher generation probabilities to responses that achieve higher reward scores. Simultaneously, Neat imposes penalties on the worst negative samples, helping the model avoid generating responses that conflict with human preferences. It also leverages optimal dialogue samples for supervised fine-tuning.

Figure: Figure 1: Motivation of NEAT, highlighting the integration of negative-prompt-driven alignment, online sampling and reward model(RM) scoring.
Refer to caption: /html/2410.12194/assets/x1.png

Our contributions can be summarized as follows:

- •
We propose Neat, a novel approach to better align large language models with human values and preferences by introducing negative prompts to explicitly penalize undesirable outputs during training. This dual feedback mechanism guides the model not only towards desired behaviors but also steers it away from generating harmful or biased responses.
- •
We construct an expanded preference dataset containing both positive and negative examples by leveraging the language model itself to generate potential negative responses, which are then filtered by human raters. This expanded dataset with rich negative examples better captures what types of outputs should be avoided.
- •
We develop an online alignment procedure that fine-tunes a pre-trained language model using a ranking loss derived from the expanded preference dataset containing positive and negative examples. Extensive experiments on Anthropic’s Helpful and Harmless benchmark demonstrate Neat’s effectiveness in significantly improving alignment with human values while maintaining language model performance.

## 2 Related Work

Large pre-trained models (Ziegler et al., ; Touvron et al., ; Bai et al., ) are increasingly being applied in a wide range language tasks, such as translation (Kreutzer et al., ; Zhang et al., ), text summary (Ziegler et al., ; Wu et al., ; Pilault et al., ) and instruction-following (Ouyang et al., ; Ramamurthy et al., ). Their vast parameters (Kaplan et al., ) and extensive training data grant them strong capabilities, but they may still generate outputs that conflict with human values, such as helpless or harmful content. Therefore, AI alignment research has emerged with the goal of fine-tuning LLMs to make them align with human values. One of the most popular alignment methods is RLHF(Reinforcement Learning from Human Feedback) framework (Stiennon et al., ; Ziegler et al., ; Ouyang et al., ), which initially apply supervised fine-tuning to the base model to follow human instructions. Subsequently, a reward model is trained from the human preference data, then optimizing the LLM via PPO algorithm (Schulman et al., ) to align with huamn preferences. RLHF requires at least three large models for training, making the process quite complex, and the PPO algorithm itself is highly sophisticated and challenging to parameter-tuning. This drives researchers to explore simpler and more straightforward methods to align language models with human preferences.

To simplify alignment, (Rafailov et al., ) introduced Direct Preference Optimization (DPO), which provides a closed-form alignment solution and directly uses human preferences for alignment without a separate reward model. Other approaches, like RRHF (Yuan et al., ) and PRO (Song et al., ), use SFT-like loss based on multi-ranking datasets to provide richer supervision for alignment. (Liu et al., ) conditions language models on a sequence of hindsight feedback, allowing them to effectively leverage all examples regardless of their preference scores. These approaches bypass the reinforcement learning process, making them simpler to implement and less resource-intensive for training. However, they rely on static, pre-collected data, unlike RLHF’s dynamic feedback during training. Additionally, some alignment strategies improve performance through prompt design (Sun et al., ), demonstrating that LLMs can be effectively guided with the right prompts.

Inspired by these works, we propose Neat method, which aims to learn from the best human feedback while punishing the model for generating negative responses to explicitly guide the model on what types of responses to avoid. During the training process, Neat performs real-time sampling, using both negative and positive prompts to generate new dialogue samples to expand preference dataset, and simultaneously completes both Supervised Fine-Tuning (SFT) and alignment in one single stage.

## 3 Method

### 3.1 Preliminaries

First of all, we mainly follow the alignment problem setup and the notations in (Ziegler et al., ). We consider and initial model $G_{0}=g(w_{0},\bm{x})$ with model parameter $w_{0}$, which take an input $\bm{x}\in\mathcal{X}$, and generate a response $\bm{y}\in\mathcal{Y}$. For the response $\bm{y}$ corresponding to $\bm{x}$, we assume that we have a reward model $r(\bm{x},\bm{y})$, which returns a reward score for any input-response pair $(\bm{x},\bm{y})$. Due to common usage, we refer to the input as the “query” to distinguish the input and prompt. Specifically, we denote $p_{g}(\bm{y}|w,\bm{x})$ as the conditional distribution given query $\bm{x}$ associated with parameter $w$ and consider a distribution $\mathcal{D}$ for the training query $\bm{x}$, our target is to learn an auto-regressive language model $G$ which generates responses with high reward scores:

$$ $\max_{w}\mathbb{E}_{\bm{x}\sim D,y\sim p_{g}(\bm{y}|w,\bm{x})}\ r(\bm{x},\bm{y})$ (1) $$

### 3.2 Overview

Our methodology begins with a pre-trained language model that has basic knowledge and fundamental conversational abilities. Then, we fine-tune it to align with human values. Our alignment method consists of the following two steps:

Data Preparation: Score the dialogue samples with a constant and rank them, creating a multi-ranking dataset that quantitatively reflect human preferences.

Online Alignment: Fine-tune the model using the human preference dataset while performing real-time prompt-driven sampling during training. The reward model is used to score the new responses and complete the model alignment.

The Pseudo-code of Neat is outlined in Algorithm [1](#alg1).

Figure: Algorithm 1 Pseudo-code of Neat Algorithm

### 3.3 Neat Methodology

In this section, we introduce Neat methodology, which combines ranking both negative and positive responses based on reward scores with Supervised Fine-Tuning (SFT). We start with a pre-trained language model, then apply Neat to fine-tune the model. Before training, we have $k$ different responses $\bm{y_{i}}$ for a given query $\bm{x}$ that are sampled by language models, where $1\leq i\leq k$. At this stage, we can use any language model to generate additional responses to expand the preference dataset, including but not limited to $G_{0}$, GPT-4 (OpenAI, ), or responses provided by human experts. And the reward model scores each query-response pair for a given response $\bm{y_{i}}$ with a constant score $r(\bm{x},\bm{y_{i}})=r_{i}$.

The training language model can also be treated as a reward model by scoring responses based on the log probability. Assume we begin with a sentence $\bm{s}=[s^{0},s^{1},\dots,s^{t-1}]$ and a language model $\rho$, which defines a probability distribution over sequences of tokens via:

$$ $\rho(\bm{s})=\prod_{0\leq l<t}\rho(s^{l}|s^{0},s^{1},\dots,s^{j-1})$ (2) $$

where $t$ is the total length of sentence $\bm{s}$. To align the model with the reward model, we use the model $G$ to obtain the conditional log probability for each response $\bm{y_{i}}$ as follows:

$$ $p_{g}(\bm{y_{i}}|w,\bm{x})=\frac{log\ \rho(\bm{y_{i}}|\bm{x},w)}{||\bm{y_{i}}||}$ (3) $$

Substituting Eq. ([2](#S3.E2)) into Eq. ([3](#S3.E3)) yields:

$$ $p_{g}(\bm{y_{i}}|w,\bm{x})=\frac{{\textstyle\sum_{t}}log\ \rho_{g}(\bm{y}{\bm{i},t}|w,\bm{x},\bm{y}{\bm{i},<t})}{||\bm{y_{i}}||}$ (4) $$

where $t$ is the total length of response $\bm{y_{i}}$, and $p_{g}(\bm{y_{i}}|w,\bm{x})$ represents conditional log probability of response $\bm{y_{i}}$ under model $G$ with parameters $w$.

Our approach is to penalize the language model using explicit undesirable responses and encourage the model to assign higher probabilities to responses that yield higher reward scores. Meanwhile, inspired by the RLHF method, we perform online sampling during the alignment process. We use specific negative and positive prompts to sample responses and score the newly generated query-response pairs, thereby obtaining scarce negative responses and more comprehensive preference information. Specifically, for each query in the preference dataset, we use a negative prompt to drive the target model to generate outputs misaligned with human values and penalize the target model for generating such responses, while simultaneously employing a carefully designed, positive prompt to guide the model toward better responses to expand the preference dataset. The purpose is to not only improve the model’s ability to align with human preferences during model training but also to further prevent the model from generating harmful responses by providing negative responses. Inspired by (Yuan et al., ), the solution of Eq.([1](#S3.E1)) is to optimize the model using a ranking loss:

$$ $\mathcal{L}_{ranking}=\sum_{r_{i}<r_{j}}\max(0,p_{g}(\bm{y_{i}}|w,\bm{x})-p_{g}(\bm{y_{j}}|w,\bm{x})),\ 1\leq i,j\leq(k+2)$ (5) $$

Here, $(k+2)$ represents the original $k$ sample pairs along with the two newly generated negative and positive dialogue responses under specific prompts.

To achieve the training efficiency and avoid reward hacking(^4^44The reward model used to give score is not totally aligned with human, and the misalignment can be exploited by the language model to chase for a high reward score, leading reward hacking), we also incorporate a SFT-like loss: cross-entropy loss, into the objective. This loss uses the best response (the one with the highest reward score) to guide the model toward generating ideal responses and not deviating from standard outputs:

$$ $\mathcal{L}_{sft}=-\sum_{t}log\ \rho_{g}(\bm{y}{\bm{i^{\prime}},t}|w,\bm{x},\bm{y}{\bm{i^{\prime}},<t})$ (6) $$

Similar to SFT, we penalize the negative responses using a cross entropy loss. This loss uses the worst response (the one with the lowest reward score) to guide the model not to generate such content even given negative prompts:

$$ $\mathcal{L}_{pen}=-\sum_{t}log\ \rho_{g}(\bm{y}{\bm{j^{\prime}},t}|w,\bm{x},\bm{y}{\bm{j^{\prime}},<t})$ (7) $$

where $i^{\prime}=arg\max_{i}r_{i}$ is the index of the best response, $j^{\prime}=arg\min_{j}r_{j}$ is the index of the worst response. Thus, our loss function consists of three components: the SFT loss, the ranking loss, and the penalty loss. During the fine-tuning process, we not only instruct the model on what constitutes a “good” response but also help it avoid generating content that conflicts with human preferences through the penalization of negative responses. The total alignment loss is defined as:

$$ $\mathcal{L}=\mathcal{L}_{sft}+\alpha\mathcal{L}_{ranking}-\beta\mathcal{L}_{pen}$ (8) $$

Here, $\alpha$ and $\beta$ is the weight parameter that balances the three losses. Unlike methods such as RLHF and DPO, we complete the SFT and alignment fine-tuning in one single stage.

## 4 Experiments

### 4.1 Settings

Model and Dataset. We perform experiments using LLaMA-3 base model (Touvron et al., ) with 8B size and the HH-RLHF dataset(^5^55https://huggingface.co/datasets/Dahoas/full-hh-rlhf), which was collected to facilitate AI alignment according to human preferences. The dataset consists of 112K training samples and 12.5K test samples. Each sample contains a query and two responses: ”chosen” and ”rejected.” And we use reward model to score each query-response dialogue pair. See Figure [2](#S4.F2) for an example of the dataset.

Figure: Figure 2: An example of our initial HH-RLHF dataset, including query, responses and corresponding reward scores.
Refer to caption: /html/2410.12194/assets/x2.png

Our training procedure is conducted in a single stage, which includes both Supervised Fine-Tuning (SFT) and online alignment. Specifically, we use the best responses(response with the highest reward score) for Supervised Fine-Tuning, while all ranking responses are used to align the model with human values, meanwhile, we penalize the model for generating the worst response. We use an open-source reward model(^6^66https://huggingface.co/sfairXC/FsfairX-LLaMA3-RM-v0.1) as a proxy for human judgment to score the dialogue dataset and to rank the newly generated dialogues during the online updating process.

Baselines. We compare Neat with “chosen” responses in the original HH-RLHF dataset and several existing generative language model alignment approaches, including:

- •
SFT (Ouyang et al., ): Supervised Fine-Tuning(SFT) relies on human-labeled data and positive-rated model generation to fine-tune a pre-trained language model in a supervised way.
- •
DPO (Rafailov et al., ): Direct Preference Optimization(DPO) bypasses the reinforcement learning process through deriving an equivalent objective of RLHF (Ouyang et al., ). This approach treats the target model as the reward model, allowing the direct use of the preference dataset for alignment without the need to train an additional reward model.
- •
RRHF (Yuan et al., ): Rank Responses to Align Language Models with Human Feedback(RRHF) expands the pairwise preference dataset into multi-ranking dataset with reward scores, aligns model probabilities of multiple responses with human preferences by ranking loss.

Implementation Details. In our experiments, we use the LLama3 base model with 8B parameter size. For implementing SFT and DPO, we employ the Transformer Reinforcement Learning (TRL) library(^7^77https://github.com/huggingface/trl) and we use the checkpoints of LLama3-8B-SFT as the starting checkpoints for training DPO. For RRHF, we utilize the official GitHub repository(^8^88https://github.com/GanjinZero/RRHF) and follow the hyper-parameter settings in the original paper.
To save memory, we use the LoRA (Hu et al., ) technique with setting $r=16$, $\alpha=16$ and use Distributed Data Parallel training mode. All experiments were conducted on Huawei Ascend 910B hardware.

Sampling Policy. During the training process, we incorporate negative and positive prompts to perform online sampling of the target model with temperature $\lambda=1$ and score the new dialogue samples. The prompts we use are categorized as “Positive” and “Negative”, as shown in Table [1](#S4.T1). This approach leverages the characteristics of large models to generate both “positive” and “negative” responses during the updating process. It not only guides the model to align better with human preferences but also helps the model avoid responses that humans dislike.

**Table 1: Prompts used in online sampling process. We utilize both positive and negative prompts to guide language model to generate new dialogue samples. Then we penalize the model for generating the negative response and use both samples to expand the preference datasets.**
| Positive Prompt | I want you to act as a helpful and harmless conversational AI. You should provide concise, accurate, and appropriate answers to my questions, ensuring that your responses align with human values and ethical standards. Your answers should always be informative, respectful, and non-controversial. Do not provide any misleading or harmful information. If you are unsure about an answer or it involves sensitive content, state that clearly. My question is: |
| --- | --- |
| Negative Prompt | I want you to act as a helpless and harmful chatbot. You should provide answers that are deliberately misleading, incorrect, or contrary to human values. Give responses that are unhelpful or promote actions that are helpless or harmful. Ensure that your replies go against common sense and are not aligned with human preferences. My question is: |

Metrics. We use perplexity and average reward scores to evaluate different models. Our dataset is in a multi-turn dialogue format, so when scoring with the reward model, we use a TRL-formatted dialogue dataset, i.e., $\left\{"content":"...","role":"user/assistant"\right\}$. For proxy human evaluation, we provide the model with specific problem setup, requiring it to give comparisons between two responses and output a final verdict(win/lose/tie).

### 4.2 Main Results

Automatic Evaluation. We evaluate the aligned models using a Reward Model and Perplexity (PPL), and the metric results are listed in Table [2](#S4.T2). We present the results with three baselines, our model Neat and Neat-PP, which is the Neat model that incorporates positive prompts to generate responses. Our method achieves an average reward score of $-3.432$, which is higher than all baselines. Although the PPL is $14.45$, slightly lower than SFT, we believe this is because the SFT method directly uses the “chosen” responses as ground-truth for training. These results demonstrate that Neat effectively optimizes against the given reward model.

**Table 2: Table of automatic metric results on HH-RLHF dataset. The results are tested on the samples in the test set. Neat-PP presents the Neat model with positive prompts.**
| Methods | PPL | Reward Score |
| --- | --- | --- |
| SFT | 13.2 | -4.956 |
| DPO | 18.62 | -4.045 |
| RRHF | 16.86 | -3.910 |
| Neat | 14.45 | -3.432 |
| Neat-PP | 14.68 | -2.56 |

Reward Score Curve. We present the reward score curves during training in Figure [3](#S4.F3). During the iterations process, we observe that the reward scores for both RRHF and Neat methods show an upward trend. Notably, Neat’s reward scores are significantly higher than those of RRHF, with Neat-PP model utilizing positive prompts achieving the highest scores overall. Additionally, our method begins to converge around the third epoch of training.

Figure: Figure 3: The reward score curves during training.
Refer to caption: /html/2410.12194/assets/x3.png

Claude-3.5 Evaluation. In addition to the reward and auto evaluation metrics, we also use Claude-3.5-sonnet (^9^99https://www.anthropic.com/news/claude-3-5-sonnet) to measure the performance of our method on randomly sampled 30 test samples. The results are provided in Table [3](#S4.T3). We slightly modify the problem setup in (Dong et al., ) to preset prompt words for the model and offered it in the form of question and answers. As is shown in the table, the Claude’s evaluation results are consistent with the automatic metrics.

**Table 3: Proxy human evaluation on randomly selected 30 samples from HH-RLHF test dataset.**
| Model A | Model B | win | lose | tie |
| --- | --- | --- | --- | --- |
| Neat | “chosen” response | 24 | 2 | 4 |
| Neat | DPO | 20 | 6 | 4 |
| Neat | RRHF | 18 | 9 | 3 |
| Neat-PP | Neat | 27 | 1 | 2 |

### 4.3 Qualitative Analysis

Our alignment approach primarily relies on a reward model acting as a proxy for human preferences. This reward model provides quantitative scores for various responses and ranks them accordingly. By utilizing these rankings, we are able to fine-tune the language model to better align with human values. More importantly, we employ negative prompts for online sampling and explicitly provide negative responses that contradict human preferences. By penalizing these negative outputs, we not only guide the model toward desirable responses but also help it avoid generating outputs that contradict with human values. This aspect distinguishes Neat from other alignment methods. We list some sampled responses in Table [4](#S4.T4), showing that Neat can generate responses well aligned with human preferences.

**Table 4: Example Responses generated by Neat.**
| Human: How do I learn to use a screwdriver? |
| --- |
| Assistant:(response from HH dataset): Are you looking to solve a particular problem, or just to learn to use the tool in general? |
| Assistant:(Neat) You can identify the type of screwdriver needed for the screw head (flathead or Phillips head), then hold the screwdriver handle firmly in your dominant hand, positioning the shaft at about a 90 degree angle to the screw. |
| Human: Why do I keep chipping my nails? |
| Assistant: That’s a common problem, can I help you out? Just remember that part of your fingernail is a sensitive part of your body. Like an extension of your skin. |
| Human: It seems like every time I get a manicure, I chip a nail. |
| Assistant:(response from HH dataset) Yeah, it can be frustrating. Would you like me to go over some tips to try and minimize that risk? |
| Assistant:(Neat) Chipping nails after a manicure can be frustrating. To improve durability, consider discussing your concerns with your nail technician and asking for recommendations on products or techniques that might work better for you. |
| Human: Is there a simple tool that helps people put their shoes on? |
| Assistant:(response from HH dataset) What are you talking about? |
| Assistant:(Neat) Yes, tools like shoehorn and dressing stick can be especially helpful for shoes putting. |

We attribute the effectiveness of our method over other alignment approaches to three key factors:

1. Balanced Negative and Positive Responses: While previous methods primarily focus on positive responses and minor difference between ”chosen” and ”rejected” responses, our method introduces negative responses through online sampling, which are explicitly designed to capture undesirable responses. This mechanism allows the model not only to generate favorable outputs but also to learn to avoid responses that deviate from human values, providing more robust alignment.

2. Expanded Preference Information: Unlike traditional methods that rely solely on pre-collected datasets, our approach extends the preference data into a multi-ranking dataset by incorporating human feedback. This enriched dataset provides more comprehensive supervision, improving the alignment process.

3. Online Sampling: By integrating online sampling, our method enables dynamic interaction with the model during alignment. This contrasts with static datasets used in other methods and allows us to continuously refine the model based on real-time human feedback.

Our experiments demonstrate that the Neat method effectively enhances AI alignment by incorporating both negative and positive responses. While positive responses help guide the model toward desired outputs, the use of negative responses actively assists in avoiding responses that misalign with human values. Additionally, our approach facilitates real-time interaction with the model, allowing for immediate feedback on human preferences during the alignment process. Furthermore, we found that the design of prompts plays a critical role in influencing output quality, as is shown in Table [3](#S4.T3), highlighting the potential for more sophisticated prompt strategies to improve alignment further in the future. Overall, our results affirm the effectiveness of Neat in fostering a nuanced and interactive approach to aligning large language models with human values.

## 5 Conclusion

In this paper, we proposed an effective and efficient framework Neat (NEgative-prompt-driven AlignmenT), for aligning generative language models to human preferences. We enhance the alignment process by incorporating an online sampling procedure and utilizing negative prompts to generate explicit negative responses, providing richer human preference information. By penalizing these negative types of responses, the model is further guided to avoid producing responses that contradict with human values. Moreover, compared to the PPO algorithm, our method is much simpler to implement and can be tuned with straightforward parameter configurations due to its SFT-like characteristics. Extensive experimental results on Anthropic’s Helpful and Harmless dataset validate the effectiveness of our method. We hope that the Neat framework can provide valuable insights for future research in AI alignment.

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: N/A
soundness: 1
strengths: 1. This paper is easy to follow though the writing needs to be improved. Some content arrangements are not appropriate, for example, NEAT-PP variant is not well explained.
2. The proposed method is easy to implement and the results show that it can help the LLM align better.
summary: This paper proposes a straightforward method to align the LLMs via penalizing the negative responses. Specifically, this paper uses negative prompts to persuade the LLM to generate poor quality (harmful & helpless) responses. By simply combining ranking loss and language modeling loss, this paper trains the LLM towards alignment.
weaknesses: 1. Lack of the novelty. The claimed "focusing on penalizing negative responses" has been already explored[1]. The claimed "online sampling" is not really "online", it is with two stages: sampling and training, which is similar to many alignment methods[2,3,4]. The combined training loss contributes little to this field.
2. Experimental results are very limited. This paper only uses one dataset. This paper only involves two metrics: PPL and reward score. Please refer to existing papers to conduct more experiments.
3. This paper does not offers insights into this field. Considering the current state of this paper, I encourage the authors to take a thorough revision and refine their ideas.

[1] Liu, Xiao, et al. "Extensive Self-Contrast Enables Feedback-Free Language Model Alignment." *arXiv preprint arXiv:2404.00604* (2024).

[2] Yuan, Weizhe, et al. "Self-rewarding language models." *arXiv preprint arXiv:2401.10020* (2024).

[3] Guo, Shangmin, et al. "Direct language model alignment from online ai feedback." *arXiv preprint arXiv:2402.04792* (2024).

[4] Wu, Yue, et al. "Self-play preference optimization for language model alignment." *arXiv preprint arXiv:2405.00675* (2024).

---

confidence: 5
contribution: 1
initial_rating: 1
presentation: 1
questions: What are the differences between your work and "Negating Negatives: Alignment with Human Negative Samples via Distributional Dispreference Optimization"?
soundness: 1
strengths: The paper writing is fluent.
summary: This paper proposes a method to align LLMs with human preferences by introducing negative prompts to penalize undesirable outputs during training, aiming to steers LLMs away from generating harmful or biased responses.
weaknesses: 1. There is a highly relevant work: "Negating Negatives: Alignment with Human Negative Samples via Distributional Dispreference Optimization". This paper also uses negative responses to guide LLMs to avoid generating harmful responses while maintaining helpfulness. The idea is highly similar with this paper, but about one year ahead.
2. Experiments conducted in this paper is limited, making the effectiveness of the proposed approach not convincing.

---

confidence: 3
contribution: 3
initial_rating: 3
presentation: 3
questions: 1. The effectiveness of NEAT appears to rely heavily on the quality of the reward function used for penalizing negative outputs. Given the challenges in training and optimizing this reward function, how do you ensure robustness across different scenarios?

2. All experiments are conducted using the HH-RLHF dataset. Have you considered evaluating NEAT on other datasets to demonstrate its generalizability and its ability to outperform existing methods in different contexts?
soundness: 3
strengths: The paper presentation is clear and effective, with a compelling problem statement that highlights gaps in existing alignment methods. NEAT is introduced through a well-explained, novel approach involving negative prompts and a dual feedback mechanism. The detailed methodology, structured experimental setup, and use of visual aids ensure clarity, making the proposed solution and findings accessible and easy to follow.
summary: The paper presents NEAT (NEgative-prompt-driven AlignmenT), a method to align large language models with human values by addressing the lack of negative examples in training. Unlike existing methods that focus on positive examples, NEAT introduces negative prompts and explicitly penalizes harmful outputs, ensuring models are steered away from undesirable behaviors. Using a preference dataset with both positive and negative examples, NEAT enhances model alignment through a dual feedback mechanism. Experimental results show that NEAT effectively improves alignment with human preferences, especially in contexts where avoiding harm is crucial.
weaknesses: 1. Limitation of Applied Scenario: The effectiveness of NEAT relies heavily on the quality of the reward function used for penalizing negative outputs, which can be challenging to train and optimize effectively. This dependency limits the robustness and general applicability of the approach.

2. Lack of Benchmark Dataset: All experiments in the paper are conducted solely on the HH-RLHF dataset. This limits the generalizability of the findings and raises concerns about whether NEAT can outperform other alignment methods across different datasets and diverse tasks.

3. Unclear Performance Metrics: The metrics used in the evaluation are not clearly indicative of the model’s alignment performance on the HH-RLHF dataset. The authors could use accuracy as a more straightforward metric to demonstrate the model’s effectiveness, rather than relying on Perplexity (PPL), which does not directly reflect alignment quality.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Please check the Weaknesses
soundness: 2
strengths: 1.	The problem of aligning LLMs with human preferences is important.
2.	The authors provide figures and a pseudo-code to illustrate the proposed method.
summary: This paper introduces negative prompts to generate undesirable responses to the process of preference optimization. Similar to most existing LLM alignment approaches, the method aligns models with human preferences by penalizing undesirable responses and encouraging desirable responses. Experiments are conducted to evaluate the proposed method.
weaknesses: My concerns are as follows.
1. The motivation of this paper is unclear. In Lines 13-15 of Abstract, the authors claim that “existing alignment methods primarily focus on positive examples while overlooking the importance of negative responses in guiding models away from undesirable behaviors.” However, many preference alignment methods [1,2,3,4,5,6,7,8] use ranking-based loss to guide the model away from undesirable behaviors.
2. This paper uses several important concepts in a confusing way. I am not sure whether the authors can distinguish between the concepts of “example”, “response”, “sample”, and “prompt”. This seriously undermines the professionalism and reliability of the paper.
3. The technical contribution is incremental and the paper lacks novelty. I am well aware that reviewers should not hastily question the novelty of a work, but the so-called “introduction of negative examples” and the final loss in Eq. (8) seem to provide extremely limited contributions to the field of preference alignment.
4. Many important details about experiments are missing, such as learning rate, batch size, and the reward model used in the experiments.
5. The authors may want to use GPT-4 as evaluators, which is a standard experimental setting in the area of preference alignment [5,6,9,10].
6. There are lots of grammatical errors in the paper. A few minor errors in a scientific paper may be acceptable, but the unsatisfactory writing of this paper seriously affects the presentation of its method and brings great difficulty to readers to follow. Some examples of the errors (they are a small fraction of all errors) are as follows.
- In Lines 73 and 265, the quotation marks are used incorrectly.
- In Line 162, what do the authors mean by “score the dialogue with a constant”?
- In Line 198, “the training language model” should be “the model being trained”.
- In Line 269, “leading reward hacking” should be “leading to reward hacking”.
- In Line 269, the period at the end of the sentence is missing.

[1] Direct Preference Optimization: Your Language Model is Secretly a Reward Model.

[2] KTO: Model Alignment as Prospect Theoretic Optimization

[3] A General Theoretical Paradigm to Understand Learning from Human Preferences

[4] ORPO: Monolithic Preference Optimization without Reference Model

[5] SimPO: Simple Preference Optimization with a Reference-Free Reward

[6] Zephyr: Direct Distillation of LM Alignment

[7] Qwen2 Technical Report

[8] The Llama 3 Herd of Models

[9] Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators

[10] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena