Reasons to Reject? Aligning Language Models with Judgments
Authors: name: Weiwen Xu, name: Deng Cai, name: Zhisong Zhang, name: Wai Lam, name: Shuming Shi

As humans, we consistently engage in interactions with our peers and receive
feedback in the form of natural language. This language feedback allows us to
reflect on our actions, maintain appropriate behavior, and rectify our errors.
The question arises naturally: can we use language feedback to align large
language models (LLMs)? In contrast to previous research that aligns LLMs with
reward or preference data, we present the first systematic exploration of
alignment through the lens of language feedback (i.e., judgment). We commence
with an in-depth investigation of potential methods that can be adapted for
aligning LLMs with judgments, revealing that these methods are unable to fully
capitalize on the judgments. To facilitate more effective utilization of
judgments, we propose a novel framework, Contrastive Unlikelihood Training
(CUT), that allows for fine-grained inappropriate content detection and
correction based on judgments. Our offline alignment results show that, with
merely 1317 off-the-shelf judgment data, CUT (LLaMA2-13b) can beat the 175B
DaVinci003 and surpass the best baseline by 52.34 points on AlpacaEval. The
online alignment results demonstrate that CUT can align LLMs (LLaMA2-chat-13b)
in an iterative fashion using model-specific judgment data, with a steady
performance improvement from 81.09 to 91.36 points on AlpacaEval. Our analysis
further suggests that judgments exhibit greater potential than rewards for LLM
alignment and warrant future research.