does this paper: ""KTO: Model Alignment as Prospect Theoretic Optimization
Kawin Ethayarajh
Winnie Xu
Niklas Muennighoff
Dan Jurafsky
Douwe Kiela
Abstract
Kahneman & Tversky’s prospect theory tells us that humans perceive random variables in a biased but well-defined manner (1992); for example, humans are famously loss-averse. We show that objectives for aligning LLMs with human feedback implicitly incorporate many of these biases—the success of these objectives (e.g., DPO) over cross-entropy minimization can partly be ascribed to them being human-aware loss functions (HALOs). However, the utility functions these methods attribute to humans still differ from those in the prospect theory literature. Using a Kahneman-Tversky model of human utility, we propose a HALO that directly maximizes the utility of generations instead of maximizing the log-likelihood of preferences, as current methods do. We call this approach Kahneman-Tversky Optimization (KTO), and it matches or exceeds the performance of preference-based methods at scales from 1B to 30B. Crucially, KTO does not need preferences—only a binary signal of whether an output is desirable or undesirable for a given input. This makes it far easier to use in the real world, where preference data is scarce and expensive.

Machine Learning, ICML
1Introduction
Aligning generative models with human feedback has been successfully used to make generations more helpful, factual, and ethical, among other desiderata (Ouyang et al., 2022; Tian et al., 2023). For LLMs, alignment methods such as RLHF and DPO have consistently proven to be more beneficial than doing supervised finetuning (SFT) alone. However, human feedback is often discussed only in the context of preferences (e.g., output 
�
≻
�
 for input 
�
), despite preferences being a kind of data that is relatively scarce and expensive to collect in the real world (Casper et al., 2023). This is largely because the alignment methods shown to work best—RLHF (Christiano et al., 2017) and the mathematically equivalent DPO (Rafailov et al., 2023)—take preference data as input.

To understand why these alignment methods work so well, and whether feedback needs to be in the form of preferences, we frame them through the lens of prospect theory (Kahneman & Tversky, 1979; Tversky & Kahneman, 1992). Prospect theory explains why humans make decisions about uncertain events that do not maximize expected value. It formalizes how humans perceive random variables in a biased but well-defined manner; for example, relative to some reference point, humans are more sensitive to losses than gains, a property called loss aversion. We show that popular alignment methods such as PPO (Schulman et al., 2017), DPO (Rafailov et al., 2023), and SLiC (Zhao et al., 2023) implicitly model such biases, helping explain their success independently of the data used. For this reason, we call them human-aware loss functions (HALOs).

Refer to caption
Figure 1:The traditional pipeline for LLM alignment starts with supervised finetuning, followed by fitting the LLM to paired preference data using a method such as RLHF or DPO. However, the paired preferences that existing approaches need are hard-to-get. Kahneman-Tversky Optimization (KTO) only needs to know whether a given output is (un)desirable for the input, giving it access to a source of data that is much more abundant, cheaper, and faster to collect in the real world.
Although it is impossible to say that HALOs are categorically better than non-HALOs, we find that among existing methods, those that meet the definition of a HALO work better than those that do not. We find that DPO performance can even be matched at most scales by running an offline PPO variant on dummy +1/-1 rewards, suggesting that preference data might not be needed if the inductive bias in the loss function is good enough. However, despite the surprising success of this simple baseline, it significantly lags behind DPO at the 30B model scale and suffers from hyperparameter sensitivity, making it difficult to use.

Taking a more principled approach, we derive a HALO using the model of human utility that Kahneman & Tversky empirically derived to describe how humans make decisions about uncertain monetary outcomes (Tversky & Kahneman, 1992). This approach, which we call Kahneman-Tversky Optimization (KTO), directly maximizes the utility of generations instead of maximizing the log-likelihood of preferences, as most current methods do. KTO only requires a binary signal of whether an output is desirable or undesirable for a given input. This data is much more abundant, cheaper, and faster to collect in the real world than preferences, making it easier to scale alignment in production environments and rapidly iterate on models.

In our experiments, we find that:

• KTO matches or exceeds DPO performance at scales from 1B to 30B parameters.1 That is, taking a preference dataset of 
�
 DPO pairs and breaking it up into 
2
⁢
�
 examples for KTO can yield better generations, despite the model ostensibly learning from a weaker signal. We provide some theoretical explanations for this phenomenon (§4.3).
• KTO can handle extreme data imbalances, matching DPO performance while using up to 90% fewer desirable examples (i.e., examples of good generations). Its success thus cannot be ascribed to the alignment data being sourced from a preference dataset.
• When the pretrained model is sufficiently good, one can skip supervised finetuning and go straight to KTO without a loss in generation quality. In contrast, we find that without doing SFT first, DPO-aligned models are significantly worse at all scales.
The fact that KTO can match and sometimes even outperform DPO is surprising, given that it learns from a weaker signal. We conclude by discussing some theoretical explanations for this phenomenon.

2Background
Feedback-aligned LLMs are traditionally trained in three stages (Ouyang et al., 2022):

Pretraining
Given a large corpus, train the model to predict the next token conditioned on the preceding text using the cross-entropy loss. Let 
�
 denote the pretrained model.

Supervised Finetuning
Finetune the model to predict the next token on data that is more relevant to the downstream task. Often, such data will comprise instructions and an appropriate response (i.e., instruction finetuning). Let 
�
ref
 denote the finetuned model.

RLHF
Given a dataset 
𝒟
 of preferences 
(
�
,
�
�
,
�
�
)
—where 
�
 is an input, 
�
�
,
�
�
 are the preferred and dispreferred outputs (i.e., 
�
�
≻
�
�
 for 
�
), and 
�
*
 is the “true” reward function underlying the preferences—it is first assumed that the probability that 
�
�
 is preferred to 
�
�
 can be captured with a specific function class, typically a Bradley-Terry model (Bradley & Terry, 1952). Where 
�
 is the logistic function:

�
*
⁢
(
�
�
≻
�
�
|
�
)
=
�
⁢
(
�
*
⁢
(
�
,
�
�
)
−
�
*
⁢
(
�
,
�
�
)
)
(1)
Since getting the true reward from a human would be intractably expensive, a reward model 
�
�
 learns to serve as a proxy, done by minimizing the negative log-likelihood of the human preference data:

ℒ
�
⁢
(
�
�
)
=
𝔼
�
,
�
�
,
�
�
∼
�
⁢
[
−
log
⁡
�
⁢
(
�
�
⁢
(
�
,
�
�
)
−
�
�
⁢
(
�
,
�
�
)
)
]
But solely maximizing the reward might come at the expense of desiderata such as generating grammatical text. To avoid this, a KL divergence penalty is introduced to restrict how far the language model can drift from 
�
ref
. Where 
�
�
 is the model we are optimizing, the optimal model 
�
*
 is that which maximizes

𝔼
�
∈
�
,
�
∈
�
�
⁢
[
�
�
⁢
(
�
,
�
)
]
−
�
�
KL
(
�
�
(
�
|
�
)
∥
�
ref
(
�
|
�
)
)
(2)
where 
�
>
0
 is a hyperparameter. Since this objective is not differentiable, we need to use an RL algorithm like PPO (Schulman et al., 2017).

However, RLHF is often slow (largely because of having to sample generations) and quite unstable in practice (especially in a distributed setting). For this reason, recent work has focused on designing closed-form losses that maximize the margin between the preferred and dispreferred generations, such as Sequence-Likelihood Calibration (SLiC) (Zhao et al., 2023) and Direct Preference Optimization (DPO) (Rafailov et al., 2023). The latter has become popular due to its mathematical equivalence with RLHF:

ℒ
DPO
⁢
(
�
�
,
�
ref
)
=
𝔼
⁢
[
−
log
⁡
�
⁢
(
�
⁢
log
⁡
�
�
⁢
(
�
�
|
�
)
�
ref
⁢
(
�
�
|
�
)
−
�
⁢
log
⁡
�
�
⁢
(
�
�
|
�
)
�
ref
⁢
(
�
�
|
�
)
)
]
(3)
3A Prospect Theoretic View of Alignment
Kahneman & Tversky’s prospect theory explains why, faced with an uncertain event, humans make decisions that do not maximize the expected value (1992). For example, because humans are loss-averse, given a gamble that returns $100 with 80% probability and $0 with 20% probability, a person might accept $60 to avoid the gamble, despite their certainty equivalent of $60 being less than the expected value of $80.

3.1Prospect Theory
In prospect theory, human utility depends on a value function and a weighting function:2

Definition 3.1.
A value function 
�
:
�
→
ℝ
 maps an outcome 
�
, relative to some reference point 
�
ref
, to its perceived (or subjective) value. For example, these functions capture the fact that humans tend to be more sensitive to relative losses than relative gains of the same magnitude.

Definition 3.2.
A weighting function 
�
 is the derivative of a capacity function that maps cumulative probabilities to perceived cumulative probabilities. These functions capture, for example, the fact that humans tend to overestimate the chance of rare events. Let 
�
�
 denote the weight placed on outcome 
�
.

Definition 3.3.
The utility of a random variable 
�
 is a function of its outcomes: 
�
⁢
(
�
)
≜
∑
�
∈
�
�
�
⁢
�
⁢
(
�
−
�
ref
)
.

However, because humans do not see the full probability distribution of an LLM, weighting functions are not salient to this discussion; we will focus only on value functions. Using experiments that presented real humans with monetary gambles and asked for their certainty equivalent, Tversky & Kahneman (1992) proposed the following functional form for human value:

�
⁢
(
�
,
�
ref
;
�
;
�
)
=
{
(
�
−
�
ref
)
�
if 
⁢
�
>
�
ref
−
�
⁢
(
�
ref
−
�
)
�
if 
⁢
�
<
�
ref
(4)
where the median value of hyperparameter 
�
=
0.88
 and 
�
=
2.25
 across individuals. 
�
 controls how quickly utility changes and 
�
 controls the degree of loss aversion. While the shape of the median Kahneman-Tversky value function is illustrated in Figure 2, it should be noted that it varies across individuals (Tversky & Kahneman, 1992). There are also other functional forms for the value function that have been proposed in later work (Gurevich et al., 2009). The salient qualities of a value function are: the existence of a reference point that is added or subtracted to get the relative gain or loss respectively; concavity in relative gains (i.e. diminishing sensitivity away from 
�
ref
); loss aversion (i.e., greater sensitivity to losses).

3.2HALOs
Informally, HALOs are loss functions that model the human biases in Tversky & Kahneman (1992). Formally,

Refer to caption
Figure 2:The utility that a human gets from the outcome of a random variable, as imputed by the value function implicit in HALOs. Notice that the imputed functions share properties such as loss aversion with the human value functions that Kahneman & Tversky empirically derived (1992).
Definition 3.4 (HALOs).
Let 
�
∈
𝒳
 denote an input and 
�
∈
𝒴
 an output. Then 
�
:
(
�
,
�
)
→
ℝ
 is a human-aware loss function if there exists the following: a parameterized reward function 
�
�
 such that 
∀
(
�
1
,
�
1
)
,
(
�
2
,
�
2
)
∈
𝒳
×
𝒴
,

�
�
⁢
(
�
1
,
�
1
)
>
�
�
⁢
(
�
2
,
�
2
)
⇔
(
�
1
,
�
1
)
≻
�
�
(
�
2
,
�
2
)
reference point distributions 
�
�
⁢
(
�
′
)
,
�
�
⁢
(
�
′
|
�
′
)
, a value function 
�
�
:
ℝ
→
ℝ
 that is monotonic non-decreasing and concave in 
(
0
,
∞
)
, and a negative affine function 
�
 such that

�
⁢
(
�
,
�
;
�
)
=
�
⁢
(
�
�
⁢
(
�
�
⁢
(
�
,
�
)
−
𝔼
�
′
,
�
′
⁢
[
�
�
⁢
(
�
′
,
�
′
)
]
)
)
(5)
where 
�
′
∼
�
�
⁢
(
�
′
)
 and 
�
′
∼
�
�
⁢
(
�
′
|
�
′
)
.

Put simply, the requirement for the reward function is that it assigns higher rewards to input-output pairs that are more preferred under it. The reference point is the expected reward with respect to input-output pairs sampled from the distributions 
�
�
,
�
�
. We require that the value function be concave in gains but not necessarily convex in losses—unlike the canonical Kahneman-Tversky value functions—because in the original work on prospect theory, a minority of individuals were found to be risk-averse in both the gain and loss regime (i.e., concave in both gains and losses) (Kahneman & Tversky, 1979). Note that risk-aversion is different from loss-aversion; they relate to the curvature and magnitude of the slope respectively.

Proposition 3.5.
DPO, SLiC (calibration loss only), and PPO-Clip are human-aware loss functions.

Refer to caption
Figure 3:Among existing alignment methods, the HALOs (DPO and our offline PPO variant) generally outperform non-HALOs (SLiC and CSFT), though the gap is only significant 
(
�
<
0.05
)
 at 13B+ model sizes. In fact, only the HALO-aligned Llama-{13B, 30B} models are able to match or exceed the generation quality of SFT target sequences, which are drawn directly from the alignment dataset. It is also worth noting that up to a scale of 7B parameters, virtually all of the gains from LLM alignment come from the SFT stage.
The proof is deferred to Appendix A. In Figure 2, we can see this more intuitively by plotting the value function for each loss (i.e., the implied human utility). We see that the value functions of all three losses incorporate a sense of loss aversion, although this is not needed to meet the definition of a HALO, since there are individuals and scenarios for which loss aversion does not necessarily apply. The value functions are also either concave or affine (depending on the interval), unlike the standard Kahneman-Tversky value function, which is concave in gains but convex in losses. The reference point distributions used also differs across the losses.

3.3Does being a HALO matter?
A natural question is whether the modeling of human biases in HALOs has practical benefits. This is difficult to answer, since both HALOs and non-HALOs are diverse function classes, but we attempt to do so by comparing popular non-HALO and HALO baselines on the exact same data:

1. CSFT: Conditional SFT is a simple alignment method where a control token is prepended to the output during training; then, at inference, the control token corresponding to desirable generations (e.g., 
) is appended to the input to induce good generations (Korbak et al., 2023). This is a non-HALO loss.
2. SLiC: SLiC with a regularization penalty (
�
reg
≠
0
) is a non-HALO loss:
ℒ
SLiC
⁢
(
�
�
,
�
ref
)
=
ℒ
cal
⁢
(
�
�
)
+
�
reg
⁢
�
reg
⁢
(
�
�
)
ℒ
cal
=
𝔼
�
,
�
�
,
�
�
∼
�
⁢
[
max
⁡
(
0
,
�
−
log
⁡
�
�
⁢
(
�
�
|
�
)
�
�
⁢
(
�
�
|
�
)
)
]
ℒ
reg
=
𝔼
�
∼
�
,
�
∼
�
ref
⁢
(
�
)
⁢
[
−
log
⁡
�
�
⁢
(
�
|
�
)
]
Although the max-margin loss 
ℒ
cal
 is a HALO on its own (Proposition 3.5), the complete loss is not, since the 
ℒ
reg
 term is the standard language modeling loss.

3. DPO: DPO, as defined in (3), is a HALO loss (Proposition 3.5).
4. PPO (offline): The standard RLHF objective in (2) is typically optimized with PPO-Clip, which works by “clipping” how far 
�
�
 can drift from the version 
�
old
 at the previous step:
ℒ
PPO (offline)
=
−
𝔼
�
,
�
,
�
∼
�
[
min
(
�
�
�
(
�
,
�
<
�
,
�
�
)
,
clip
(
�
�
,
1
−
�
,
1
+
�
)
�
(
�
,
�
<
�
,
�
�
)
)
]
where 
�
�
=
log
⁡
�
�
�
old
 and 
�
⁢
(
�
,
�
<
�
,
�
�
)
 is the per-token advantage (i.e., the surplus benefit from producing a given token in a given state).
PPO is an online algorithm—generations are sampled from the current model, judged by a reward model, and then used to update the current version. However, this process is slow (due to having to sample generations), so we choose to use offline data instead. Because RLHF is also quite unstable in a distributed setting, we never update 
�
old
 and keep it as 
�
ref
, instead clipping less conservatively than we traditionally would (see Appendix B for details). Baheti et al. (2023) found that these changes, along with treating the entire output sequence as a single action, greatly improves stability. However, since RLHF has historically calculated token-level advantages, we omit the third change and only preserve the first two. The PPO-Clip loss itself is left unchanged and is therefore a HALO (Proposition 3.5).

Calling this method PPO is somewhat imprecise, because it is offline and takes only one step, but to avoid introducing too many new terms, we will call this PPO (offline). Instead of using learned rewards, we simplify even further and use dummy +1/-1 rewards for 
�
�
 and 
�
�
 instead. Further details on the implementation of this method can be found in Appendix B.

We compare these baselines on a suite of 7 models spanning two model families, Pythia-{1.4B, 2.8B, 6.9B, 12B} (Biderman et al., 2023) and Llama-{7B, 13B, 30B} (Touvron et al., 2023). This permits us to see how LLM alignment scales within a model family (Llama-2 lacks a 30B model, hence our use of Llama). Later experiments (§4.2) are done on Mistral-7B and its derivatives (Jiang et al., 2023). The models were trained on a combination of Anthropic HH (Ganguli et al., 2022), OpenAssistant (Köpf et al., 2023), and SHP (Ethayarajh et al., 2022).

All models were aligned under identical settings on the same data (e.g., same effective batch size, same optimizer, etc.), save for hyperparameters unique to them. Similar to Rafailov et al. (2023), the target sequences for SFT are a subset of the generations used to subsequently align the model; however, for a more realistic SFT setup, we do not necessarily set the most preferred generation to be the target (with the exception of HH, since the dispreferred output in that dataset is often harmful). Then we used GPT-4-0613 to judge whether the aligned model’s response was better than the SFT target for the given input with respect to helpfulness, harmlessness, and conciseness, a now standard practice (Zheng et al., 2023; Li et al., 2023).3 Note that while the SFT target is considered a desirable output for 
�
, it is by no means the best output, meaning that it can be improved upon by an aligned model.

In Figure 3, we see the results of this analysis:

• The HALOs we tested (DPO and our PPO variant) either match or outperform the non-HALOs at all scales, though the gap is only significant 
(
�
<
0.05
)
 at 13B+ model sizes. In fact, only the HALO-aligned Llama-{13B, 30B} models match or exceed a win rate of 50% (i.e., are able to match or exceed the generation quality of the SFT targets in the test data).
• Up to a scale of 7B parameters, alignment provides virtually no gains over SFT alone. However, it is worth noting that if the SFT data distribution were less similar to the preference data, then the gains from the alignment stage would ostensibly be greater.
• Surprisingly, despite only using dummy +1/-1 rewards, our offline PPO variant performs as well as DPO for all models except Llama30B. This challenges conventional wisdom, which places heavy emphasis on reward learning (Casper et al., 2023), suggesting that even the simplest rewards can prove useful when used in a loss function that has a strong inductive bias. Despite its surprising success, our offline PPO baseline still suffers from hyperparameter sensitivity and training instability, albeit not to the same extent as traditional RLHF.
4Kahneman-Tversky Optimization
The surprising success of offline PPO with dummy +1/-1 rewards suggests that—with the right HALO—a binary signal of good/bad generations may be sufficient to reach DPO-level performance, even if the offline PPO approach itself was unable to do so past a certain scale (§3.3). Taking a more principled approach, we now derive a HALO using the Kahneman-Tversky model of human utility, which allows us to directly optimize for utility instead of maximizing the log-likelihood of preferences. This Kahneman-Tversky Optimization (KTO) loss only needs a binary signal of whether an output is (un)desirable for a given input, giving it access to a source of data is more abundant, cheaper, and faster to collect in the real world.

4.1Derivation
Refer to caption
Figure 4:Kahneman-Tversky Optimization (KTO) is as good or better than DPO at all scales, both when preceded and not preceded by supervised finetuning (SFT). In fact, for the Llama models, KTO alone matches the performance of SFT+DPO and is significantly better than DPO alone. Error bars denote a 90% binomial confidence interval.
From prior work (Go et al., 2023; Peng et al., 2019; Peters & Schaal, 2007), we know that the policy that maximizes the KL-constrained RLHF objective in (2) is

�
*
⁢
(
�
|
�
)
=
1
�
⁢
(
�
)
⁢
�
ref
⁢
(
�
|
�
)
⁢
exp
⁡
(
1
�
⁢
�
*
⁢
(
�
,
�
)
)
where 
�
⁢
(
�
)
 is a partition function. Rafailov et al. (2023) rewrite this in terms of the optimal reward for an input-output pair:

�
*
⁢
(
�
,
�
)
=
�
⁢
log
⁡
�
*
⁢
(
�
|
�
)
�
ref
⁢
(
�
|
�
)
+
�
⁢
log
⁡
�
⁢
(
�
)
(6)
They then plug this expression into the Bradley-Terry model of preferences and take the negative logarithm of that objective to get the DPO loss (3).

Instead, we plug this expression into the Kahneman-Tversky model of human utility, with some changes to make it more amenable to the LLM setting:

1. The exponent in the Kahneman-Tversky value function (4) makes it difficult to optimize, so we set 
�
KTO
 to be the logistic function 
�
, which is also concave in gains and convex in losses. We replace the loss-aversion coefficient with two hyperparameters 
�
�
,
�
�
 that weight the losses for desirable and undesirable outputs respectively.
2. The Kahneman-Tversky value function was derived based on experiments with humans and monetary gambles. Since LLM generations do not have a monetary reward associated with them, we set 
�
KTO
 to be the implicit reward under the RLHF objective (6).
3. Rather than having just one dispreferred generation 
�
�
|
�
 as the reference point, we assume that humans judge the quality of 
(
�
,
�
)
 in relation to all input-output pairs they have seen. Thus we write the reference point to be the expected reward under the optimal policy, not just for generations following 
�
 but following any input 
�
′
: 
𝔼
�
′
∼
�
,
�
′
∼
�
*
⁢
[
�
*
⁢
(
�
′
,
�
′
)
]
. Under the assumption that the expected value of the partition function across 
�
′
 is zero, this simplifies to the KL divergence between 
�
*
 and 
�
ref
 scaled by 
�
.
Combining all of these changes, we can optimize the following loss, where the notion of an output being “desirable” or “undesirable” corresponds to the Kahneman-Tversky notion of a relative gain or loss.

�
KTO
⁢
(
�
�
,
�
ref
)
=
𝔼
�
,
�
∼
�
⁢
[
�
⁢
(
�
)
⁢
(
1
−
�
KTO
⁢
(
�
,
�
;
�
)
)
]
(7)
where

�
KTO
⁢
(
�
,
�
)
=
�
⁢
log
⁡
�
�
⁢
(
�
|
�
)
�
ref
⁢
(
�
|
�
)
�
ref
=
𝔼
�
′
∼
�
[
�
KL
(
�
�
(
�
′
|
�
′
)
∥
�
ref
(
�
′
|
�
′
)
)
]
�
KTO
⁢
(
�
,
�
;
�
)
=
{
�
⁢
(
�
KTO
⁢
(
�
,
�
)
−
�
ref
)
⁢
if 
⁢
�
∼
�
desirable
|
�
�
⁢
(
�
ref
−
�
KTO
⁢
(
�
,
�
)
)
⁢
if 
⁢
�
∼
�
undesirable
|
�
�
⁢
(
�
)
=
{
�
�
if 
⁢
�
∼
�
desirable
|
�
�
�
if 
⁢
�
∼
�
undesirable
|
�
Intuitively, KTO works because if the model increases the reward of a desirable example in a generic way, then the KL penalty will also rise and no progress will be made on the loss. This forces the model to learn exactly what makes an output desirable, so that the reward can be increased while keeping the KL term flat (or even decreasing it). A similar argument works in the other direction as well, though the non-negativity of the KL term allows faster saturation.

Implementation
In practice, we estimate the KL term by matching inputs 
�
′
 with unrelated outputs 
�
�
′
 in a batch of size 
�
 and then calculating 
max
⁡
(
0
,
1
�
⁢
∑
log
⁡
�
�
⁢
(
�
�
′
|
�
′
)
�
ref
⁢
(
�
�
′
|
�
′
)
)
 over the entire batch. We do not back-propagate through the KL term, as it makes training much more stable. This means that the KL term purely serves to control how saturated the loss is.

�
 has the same meaning as in DPO; the lower it is, the less we penalize 
�
�
 from moving away from the SFT model 
�
ref
. We find that 
�
=
0.1
 is close-to-best on most datasets. Where 
�
�
 and 
�
�
 refer to the number of desirable and undesirable examples respectively, we set 
�
�
,
�
�
 such that

�
�
⁢
�
�
�
�
⁢
�
�
∈
[
1
,
4
3
]
(8)
where at least one of the two should be set to 1 and the ratio is controlled by changing the other. For example, if there is a 1:1 ratio of desirable:undesirable examples, we would set 
�
�
=
1
,
�
�
∈
[
1
,
1.33
]
. If we then discard 90% of the desirable examples and only keep 10%, then we would set 
�
�
=
1
,
�
�
∈
[
10
,
13.33
]
. The interval 
[
1
,
4
/
3
]
 was determined empirically and suggests a value function that is more gain-sensitive than loss-sensitive, in contrast to the original Kahneman-Tversky value function (4). However, the ideal interval is also task-dependent; for example, if avoiding negative outcomes were very important, then we might consider a setting of 
�
�
>
1
 instead.

Refer to caption
Figure 5:Without doing SFT first, DPO-aligned models tend to ramble and hallucinate entire conversations. KTO does not suffer from this issue.
Data
If the alignment data is naturally binary, every positive example can be assumed to be drawn from 
�
desirable
|
�
 and every negative example from 
�
undesirable
|
�
. However, the canonical feedback datasets in academic research (HH, SHP, OASST) are in preference format, since the methods that have worked best up until now are preference-based. In our experiments, we converted preference data 
�
�
≻
�
�
 by assuming that 
�
�
 is drawn from the desirable distribution and 
�
�
 from the undesirable one. To enable an apples-to-apples comparison with DPO, we apply KTO on the same data for most experiments. However, to ensure that KTO can be used with non-preference data, we also subsample one output 
�
 per 
�
 for some experiments (denoted one-
�
-per-
�
).

If the data is score-based, where a higher score denotes greater desirability, one has multiple options:

• Assume that any output with a score above some fixed threshold 
�
 is desirable.
• Assume that any output with a score above the mean or median (either across all inputs or just the input it was conditioned on) is desirable.
• Let desirability be a Bernoulli random variable where 
�
⁢
(
�
∼
�
desirable
|
�
)
 is some function of its score (e.g., logistic). Then randomly sample to determine whether 
�
 is desirable or not.
4.2Experiments
KTO 
≥
 DPO
As seen in Figure 4, SFT+KTO is competitive with SFT+DPO at model scales from 1B to 30B, despite learning from a weaker signal. KTO alone is better than DPO alone for the Llama-{7B, 13B, 30B} models, and this gap is significant (
�
<
0.01
) at 7B and 30B even after correcting for multiple comparisons (Holm, 1979). Perhaps most surprising is the fact that a KTO-aligned Llama-{13B, 30B} model is competitive with its SFT+KTO counterpart, despite not undergoing supervised finetuning first, and is the only alignment method of the ones we tested to show this behavior. This is perhaps due to the fact that KTO keeps the average response length roughly the same as it is for the SFT model. In contrast, doing DPO without SFT first causes the average response length to increase dramatically.

Refer to caption
Figure 6:Even after discarding 90% of the desirable examples while keeping all of the undesirable data (leading to a 1:10 ratio of desirable:undesirable data), a KTO-aligned Llama-7B model still outperforms its DPO counterpart. This implies that preference pairs do not have to be the source of KTO data.
KTO data need not come from preference datasets.
Might KTO be secretly benefiting from the fact that its 
2
⁢
�
 examples in the previous experiment came from 
�
 preference pairs instead of a naturally unpaired data distribution? To test this, we randomly discard increasingly large fractions of the desirable data before KTO-aligning a Llama-7B model. For example, if we discard 90% of the desirable data while leaving the undesirable data untouched, then the ratio of desirable:undesirable examples goes from 1:1 to 1:10 and the vast majority of examples no longer have a preferred output counterpart. We handle such imbalances by changing the loss weights 
�
�
,
�
�
 to satisfy the criteria in (8); when we drop 90% of the desirable data, we set 
�
�
=
1
,
�
�
=
13.33
. The full results are given in Figure 6. For Llama-7b, we find that up to 90% of the desirable data can in fact be discarded while still outperforming DPO. A similar trend holds when discarding undesirable data. For different models and datasets, the optimal settings of 
�
�
,
�
�
 differ.

We further verify this claim by aligning Mistral-7B on OpenAssistant using DPO (on 
�
 pairs), standard KTO (on all 
2
⁢
�
 outputs), and KTO where only one 
�
 per 
�
 is used. Since the output of one 
�
 in OpenAssistant is not conditioned on the other outputs for the same input, the latter effectively captures the setting where the data is from an inherently unpaired distribution. Despite the one-
�
-per-
�
 setup decreasing the amount of training data by 72%, the KTO-aligned model still outperforms both its DPO counterpart and the official instruction-tuned version of Mistral-7B (Jiang et al., 2023), as seen in Table 1.

On average, KTO improves performance across generative benchmarks.
Zephyr-
�
 is a variant of Mistral-7B that has been instruction-tuned and DPO-aligned on the UltraFeedback dataset (Tunstall et al., 2023; Cui et al., 2023). We find that substituting KTO for DPO (and changing nothing else) improves performance across MMLU (0-shot) (Hendrycks et al., 2021), GSM8K (8-shot, CoT) (Cobbe et al., 2021), HumanEval (0-shot) (Chen et al., 2021), and BigBench-Hard (3-shot CoT) (Srivastava et al., 2022). On GSM8K, just swapping DPO for KTO improves performance by 13.5 points. Even when we align with KTO using only one 
�
 per 
�
 (i.e., reducing the data volume by half), we still outperform DPO on all but one benchmark.

Table 1:In aligning Mistral-7B on the OpenAssistant dataset, we find that using KTO with only one output per input still outperforms DPO, despite this restriction reducing the amount of training data by 72%. A 90% confidence interval is given.
Method	Winrate vs. SFT Target
Mistral-7B (unaligned)	0.525 
±
 0.037
Mistral-7B + DPO	0.600 
±
 0.037
Mistral-7B + KTO (all 
�
 per 
�
)	0.652 
±
 0.036
Mistral-7B + KTO (one 
�
 per 
�
)	0.631 
±
 0.036
Mistral-7B-Instruct	 0.621 
±
 0.031
4.3Theoretical Analysis
KTO was designed with the motivation that even if it had to learn from a weaker signal, it would make up for this limitation with the fact that it has access to much more data in the real world, where thumbs-up/thumbs-down data is common but preferences are scarce and expensive to collect. So why does KTO perform as good or better than DPO in our experiments, when it sees the same amount of data? Data efficiency may not be the only answer. Our theoretical analysis suggests that preference likelihood can be maximized without necessarily maximizing underlying human utility and that KTO implicitly ignores noisy and intransitive data.

Proposition 4.1.
KTO does not learn from undesirable examples with sufficiently high rewards or desirable examples with sufficiently low rewards.

Informally, if an example is too difficult to learn from, then the KTO update will not change 
�
�
. This may be a blessing in disguise, since human preferences are often noisy and not every given preference can be recovered with the true reward 
�
*
 (Hoeffler & Ariely, 1999). This means that it may be useful to avoid unlearnable preferences. However, this is a double-edged sword: it also means that KTO could end up ignoring some data that is hard-to-learn but necessary to recover 
�
*
, resulting in under-fitting.

Theorem 4.2.
Assuming the value function is logistic, for any bounded reward function 
�
�
, there exists a reward function in its equivalence class (i.e., 
�
�
⁢
(
�
,
�
)
=
�
�
⁢
(
�
,
�
)
+
ℎ
⁢
(
�
)
 for some 
ℎ
⁢
(
�
)
) that induces the same optimal policy 
�
*
 and Bradley-Terry preference distribution but a different human value distribution.

A key insight from Rafailov et al. (2023) is that reward functions in the same equivalence class (i.e., differing only in an input-specific component) induce the same optimal policy under (2) and the same Bradley-Terry preference distribution. However, we show under mild assumptions that the value distribution—i.e., human utility—is affected by such input-specific changes, so maximizing preference likelihood does not mean one is maximizing human utility. Approaches that directly maximize utility, such as KTO, may thus perform better in open-ended evaluation.

Table 2:Aligning Zephyr (Tunstall et al., 2023), a derivative of Mistral-7B, on UltraFeedback with KTO instead of DPO improves results across a suite of benchmarks. This is true even when only one of the two outputs in each preference is seen by KTO, despite this reducing the volume of data by half (one-
�
-per-
�
).
Dataset (
→
)	MMLU	GSM8k	HumanEval	BBH
Metric (
→
)	EM	EM	pass@1	EM
Zephyr-
�
 SFT	57.2	39.0	30.1	46.3
+DPO	58.2	40.0	30.1	44.1
+KTO	58.6	53.5	30.9	52.6
+KTO (one-
�
-per-
�
)	58.0	50.0	30.7	49.9
Theorem 4.3.
Let two humans 
�
,
�
 have value functions 
�
�
,
�
�
 and contradicting preferences 
�
1
≻
�
�
2
 and 
�
2
≻
�
�
1
 for some input 
�
. Assume 
�
𝑟𝑒𝑓
⁢
(
�
|
�
)
=
0
⟹
�
�
⁢
(
�
|
�
)
=
0
 for all 
�
,
�
. In the worst-case, the optimal policy under DPO decreases the expected value of both humans. In contrast, if each preference is broken up into two examples, then KTO (with default settings) does not change the policy.

Informally, we assume that humans want the model to increase and decrease the probability of generations they like and dislike respectively. However, the preferences of two humans often contradict, leading to a dataset containing intransitive preferences. In the worst-case, DPO allows one of the two preferences to be recovered while decreasing the expected value of both humans. In contrast, KTO will change nothing at all in any case. Since existing datasets contain preferences from multiple annotators, the existence of intransitivity may help explain why KTO works better.

4.4KTO vs. DPO – when to use which?
When human feedback is in a binary format, and especially when there is an imbalance between the number of desirable and undesirable examples, KTO is the natural choice. When your data is in the form of preferences, the choice is less clear. Putting aside the greater data efficiency of KTO, our theoretical analysis suggests that if your preference data has sufficiently little noise and sufficiently little intransitivity, then DPO will work better, since there is some risk of KTO underfitting. But if there is enough noise and transitivity, then the better worst-case guarantees of KTO will win out. Most publicly available preference datasets (e.g., SHP, OpenAssistant) contain noisy feedback from many different humans whose preferences likely contradict, which explains why KTO was able to match or exceed DPO performance in our experiments. Even AI feedback can be noisy and intransitive, which helps explain why KTO outperforms DPO when aligning with the synthetic UltraFeedback data.

5Related Work
Human feedback has been used to improve LLM capabilities in translation (Kreutzer et al., 2018), summarization (Stiennon et al., 2020), sentiment-conditioned generation (Ziegler et al., 2019), and instruction-following (Ouyang et al., 2022). The RLHF framework (Christiano et al., 2017; Bai et al., 2022) traditionally used to accomplish this is detailed in §2.

Still, momentum has largely shifted in favor of closed-form losses that directly operate on offline preferences, such as DPO (Rafailov et al., 2023). This single stage of optimization distinguishes DPO from the conventional approach in preference-based RL, which learns a reward and then fits the policy to those rewards  (Jain et al., 2013; Busa-Fekete et al., 2014). A recent string of work has centered on the idea of “self-training” or “self-play”, during which new preference data is inferred from a model’s generations  (Chen et al., 2024; Yuan et al., 2024). Despite not being a human-aware loss, unlikelihood training was among to first to methods to align language models using a binary signal (Welleck et al., 2019). However, work by Korbak et al. (2023) found that it is worse than the CSFT baseline we tested in our work.

Prospect theory, despite being highly influential in behavioral economics, has had a fairly muted impact in machine learning, with work concentrated in human-robot interaction (Kwon et al., 2020; Sun et al., 2019; Chan et al., 2021). Learning from sparse binary feedback is a staple of information retrieval and recommender systems (He et al., 2017; Koren et al., 2009), although to our knowledge it has not been used to generate open-ended text.

6Future Work
The existence of HALOs raises many questions. For one, the KTO loss is based on the Kahneman-Tversky value function for monetary gains and losses, which is almost certainly different from how humans perceive the relative goodness of text. What value function—and corresponding HALO—best describes how humans perceive language?

Given that the data that KTO needs is much more abundant, cheaper, and faster to collect—both as human and AI feedback—how far can we push synthetic data? For example, if we wanted to create a toxicity dataset to align our models to be less toxic, creating a tuple 
(
�
,
�
�
,
�
�
)
 where 
�
�
 is more toxic than 
�
�
 is non-trivial. However, with KTO, we can easily create a dataset where desirability is determined by some black-box toxicity detection API. What other kinds of desiderata can we synthetically optimize for with KTO? Can we convert signals like “conversation lead to sale made” or “support ticket resolved” into KTO data?

Currently, KTO can learn from score-based data when the score is used to infer desirability. However, can we design a HALO where scores are directly incorporated into this loss?

7Conclusion
We proposed a class of functions called human-aware losses (HALOs) based on the idea of a Kahneman-Tversky value function, which models some of the key cognitive biases that inform how humans make decisions about uncertain outcomes. We showed that among existing alignment methods, those that met the definition of a HALO performed better than those that did not, suggesting a benefit to the modeling of human biases. We then designed a human-aware loss called KTO for directly maximizing the utility of generations instead of maximizing preference likelihood. Despite only learning from a binary signal of whether an output is (un)desirable, KTO is as good or better than DPO at scales from 1B to 30B. Still, we make no claims that KTO is the best HALO for all scenarios; there remains much work to be done in discovering the optimal human-aware for each setting."" solve this problem?: ""2.2Training Language Models to Reason
One direction that researchers have used to train language models to reason or improve their reasoning is training the language model on mined reasoning traces or reasoning-like data (Rajani et al., 2019; Wei et al., 2021a; Lewkowycz et al., 2022; Chung et al., 2022; Gunasekar et al., 2023). Although this approach has been demonstrated to be effective, it comes with drawbacks. It requires either manual annotation, which is sensitive to the capability of the annotators and is off-policy for the language model (i.e., the distribution of reasoning is not text that the language model would otherwise likely have generated). This approach is also expensive, difficult to scale, and provides no clear path to solving problems harder than those that the annotators are capable of solving.

Another direction for teaching reasoning relies on a language model’s own generated reasoning, which can be seen as building on a large body of literature on self-play (Silver et al., 2017; Anthony et al., 2017; Polu & Sutskever, 2020). These include methods such as the Self-Taught Reasoner (Zelikman et al., 2022), which demonstrated that a language model iteratively trained on its reasoning that led to correct answers could solve increasingly difficult problems. Later work aimed to leverage additional information or assumptions such as Huang et al. (2022) which demonstrated that the algorithm proposed in STaR could still work if one assumed that the majority-vote answer was correct (although this has a lower ultimate performance). Further work has generalized the results of Zelikman et al. (2022), such as Uesato et al. (2022) which demonstrated additional usefulness to “process-based” supervision where incorrect reasoning traces were filtered, recently V-STaR (Hosseini et al., 2024) that demonstrates that training a verifier to guide generation also improves performance, as well as TRICE (Hoffman et al., 2024) which maximizes the marginal likelihood of the correct answer given several reasoning traces per problem. Finally, related work has also explored learning intermediate reasoning in the constrained setting of making mathematical statements, where statements in the model’s intermediate reasoning could be constrained to only be valid mathematical statements (Poesia et al., 2023). We include further discussion of related reasoning works in Appendix F.""