Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data
Authors: Siqi Guo
Ilgee Hong
Vicente Balmaseda
Changlong Yu
Liang Qiu
Xin Liu
Haoming Jiang
Tuo Zhao
Tianbao Yang
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
Discriminative Likelihood
Supervised Finetuning
Primary area: deep_learning->large_language_models

Abstract: Supervised fine-tuning (SFT) has become a crucial step for aligning pretrained large language models (LLMs) using supervised datasets of input-output pairs. However, despite being supervised, SFT is inherently limited by its generative training objective. To address its limitations, the existing common strategy is to follow SFT with a separate phase of preference optimization (PO), which relies on either human-labeled preference data or a strong reward model to guide the learning process. In this paper, we address the limitations of SFT by exploring one of the most successful techniques in conventional supervised learning: discriminative learning. We introduce **Discriminative Fine-Tuning (DFT)**, an improved variant of SFT, which mitigates the burden of collecting human-labeled preference data or training strong reward models. Unlike SFT that employs a generative approach and overlooks negative data, DFT adopts a **discriminative paradigm** that increases the probability of positive answers while suppressing potentially negative ones, aiming for **data prediction** instead of token prediction. Our contributions include: (i) a discriminative probabilistic framework for fine-tuning LLMs by explicitly modeling the discriminative likelihood of an answer among all possible outputs given an input; (ii) efficient algorithms to optimize this discriminative likelihood; and (iii) extensive experiments demonstrating DFT's effectiveness, achieving performance better than SFT and comparable to if not better than SFT→PO. The code can be found at https://github.com/Optimization-AI/DFT.

Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data
Siqi Guo
llgee Hong? Vicente Balmaseda
Changlong Yu Liang Qiu Xin Liu Haoming Jiang
Tuo Zhao Tianbao Yang"
Abstract
https///gihhbooomm/ppipiiiiiaa/p
Supervised fine-tuning (SFT) has become cial step for aligning pretrained large language models (LLMs) using supervised datasets of input- output pairs. However, despite being supervised, SFT is inherently limited by its generative training objective. To address its limitations, the existing common strategy is to follow SFT with sepa- rate phase of preference optimization (PO), which relies either human-labes
1 Introduction
Fine-tuning large language models (LLMs) has become an essential step to adapt pretrained models to specific tasks, significantly improving their performance and practical util- ity (Wei et al., 2022; Ouyang et al., 2022; Touvron et al., 2023; OpenAl, 2024; Guo et al., 2025). While pretraining enables LLMs to acquire vast amounts of general knowl- edge, fine-tuning tailors the model to exhibit desirable be- haviors, and excel in specialized domains. As LLMs be- come integral to applications like conversational AI, content generation, and gecision-makinng developing effective and efficient fine-tuning methods remains critical challenge.
preference data strong reward model to guide the learning process. In this paper, we address the limitations of SFT by exploring one of the most successful techniques in conventional supervised learning: discriminative learning. We introduce Diserimi- native Fine-Tuning (DFT), an improved variant of SFT, which mitigates the burden of collecting human-labeled preference data or training strong reward models. Unlike SFT that employs gener- ative approach and overlooks negative data, DFT adopts a discriminative paradigm that increases the probability of positive answers while suppress- ing potentially negative ones, aiming for data prediction instead of token prediction. Our con- tributions include: (i) discriminative probabilis- tic framework for fine-tuning LLMs by explicitly modeling the discriminative likelihood of an an- swer among all possible outputs given an input; (ii) efficient algorithms to optimize this discrimi- native likelihood: and (iii) extensive experiments demonstrating DFT's effectiveness, achieving per- formance better than SFT and comparable to if not better than SFT-POO. The code can be found
The current standard for aligning LLMS typically involves supervised fine-tuning (SFT) followed by preference op- timization (PO) denoted by SFT-PO, including tech- niques such as reinforcement learning from human feed- back (RLHF) (Stiennon al., 2020; Ouyang et al., 2022; Bai et al., 2022; Rafailov et al., 2024). In this approach, SFT first aligns the model with supervised data, and PO further refines the model using preference data labeled by humans or a reward model that simulates human preferences. This two-stage process has achieved significant success, particu- larly in improving human alignment and response quality, However, PO methods often require extensive human anno- tations the construction of robust reward models, both of which resource-intensive and may limit scalability and applicability in highly specialized areas.
This raises an intriguing question: Can we align LLMs without human preference data or reward models while achieving competitive performance to SFT-PO?
'Department of Computer Science and Engincering, Texas A&M University, College Station, USA 'H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Tech- nology, Atlanta, USA AAmazon.com Inc, CA. USA. Correspon- dence to: Tianbao Yang <tianbao- yang tamu. 'I.H's 's work was done during the internship at Amazon.
To address this question, we propose Discriminative Fine- Tuning (DFT), novel alternative to SFT-PO that mit- igates the burden of collecting human-labeled preference data training strong reward models. Unlike SFT, which uses generative approach and overlooks negative examples, DFT adopts a discriminative notion, explicitly discriminat- ing good from "bad" outputs generated by the base model to be finetuned. We formalize this approach by introducing
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data
diseriminative probabilistic framework that models the discriminative likelihood of an answer among all possible outputs for given input. This stands in stark contrast to SFT, which uses a generative probabilistic framework to model only the generative likelihood of individual tokens in the answers. To implement this framework, we develop effi- cient algorithms to optimize the discriminative likelihood of good answers, ensuring both scalability and practical- ity. Due to its strong discriminative capability, DFT delivers competitive or superior performance compared to SFT- mitigating the requirement for human preference data and reward models.
2019; Stiennon al., 2020; Ouyang al., 2022; Bai et al.,
2022). These methods leverage separate reward model, trained on human-labeled preference data, and optimize the SFT model against it using policy gradient methods such as PPO (Schulman et al., 2017) and REINFORCE (Williams, 1992). Rafailov et al. (2024) proposed direct preference optimization (DPO), which removes the step of training reward model and directly optimizes a pairwise loss of the policy model the preference data. Following DPO, many PO methods have been proposed with different loss functions, including R-DPO (Park et al., 2024), CPO (Xu et al., 2024a), IPO (Azar et al., 2024), SimPO (Meng et al., 2024), KTO (Ethayarajh et al., 2024), ORPO (Hong et al., 2024), DPO-p (Pal et al., 2024), to name just few among others (Zhao et al., 2023; Jung et al., 2024).
Our main contributions are summarized as follows:
A novel discriminative framework: We introduce a probabilistic framework that explicitly models the discriminative likelihood of an answer among all pos- sible outputs, in contrast to the generative likelihood approach used in SFT.
Several works (Chen et al., 2024b; Yuan et al., 2023a; Song et al., 2024) have considered PO with list of ranked prefer- ence data that may be explicitly labeled with reward value. Rosset et al. (2024) assumed general preference function is given that can produce a probability telling one output is preferred over another output given an input. Different from these works, we do not assume any preference data or preference model other than annotated input-output pairs.
Efficient optimization algorithms: We propose seal- able and practical methods to maximize the discrimi- native likelihood of good answers, ensuring effective fine-tuning of LLMs.
Finetuning via Self-play. Training a model on its own self generated responses has been widely explored in the PO stage. For example, many variants of RI.FF (Chris- tiano et al., 2017; Ziegler et al., 2019; Stiennon et al., 2020; Ouyang et al., 2022; Bai et al., 2022; Li et al., 2023; Chan et al., 2024; Ji et al., 2024) use on-policy samples produced by the current policy under optimization. Some studies have exhibited benefits of using self-generated data for PO by reducing the distribution gap between the training data and the current model while fostering exploration of diverse response spaces (Xu et al., 2024b; Tajwar et al., 2024; Tang et al., 2024). Moreover, leveraging synthetic data has proven essential for iterative (online) algorithmic improvement of these methods (Xu al., 2023b; Guo et al., 2024; Yuan et al., 2024; Chen et al., 2024a; Dong et al., 2024). A more closely related work is SPIN (Chen et al., 2024c), which uses a similar preference optimization objective as DPO but with data generated by the model to be finetuned as the losing responses. Although we also use self- celf-generated data from the base model be finetuned as our negative data, our formulation is derived from discriminative learning framework, making our approach aided by advanced opti- mization better than the pairwise loss function used in SPIN and other pairwise preference optimization objectives (cf. Section 6).
Extensive empirical validation: We conduct exten- sive experiments to demonstrate that DFT consistently outperforms standard SFT and achieves comparable results than preference optimization methods that rely on explicit preference datasets.
These contributions establish DFT as a new paradigm for enhancing pretrained language models, offering both theo- retical and practical advancements in the field.
Related Work
Supervised Finetuning. The standard approach to SFT is mimicking pretraining, which maximizes the likelihood of tokens in the output response given input prompt (Ouyang et al., 2022: Wei et al., 2022; Xu et al., 2023a; Wang al., 2023; Zhang et al., 2023; Li et al., 2024b). Although sim- ple implement, it often captures only superficial patterns rather than fostering a deeper understanding of task seman- ties (Kung& & Peng, 2023; Zhang et al., 2023; Gudibande et al., 2024). Recent works have studied inverse reinforce- ment learning to address this limitation, but such methods often involve interdependent updates between different mod- els (c.g., the policy and a reward-related model), which com- plicates the training process (Li et al., 2024a; Wulfmeier et al., 2024). In contrast, our method neither requires on- line sampling from the current policy model nor involves a reward-related model, making it much more efficient.
3 Preliminaries: SFT
For SFT, we are given a set of data D = {(x,, 1,...,m}. where x; is an input prompt and y: is labeled output answer. Both the input x and output y are expressed
Preference Optimization. Pioneering works proposed RL- based PO methods (Christiano et al., 2017: Ziegler et al.,
2

---

Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data
as sequence of tokens from a vocabulary of size K denoted by V = {uI,---,, vK)- We let x [..]))-......) ((4)) and y (yp))..... Wim')), where Tij E V., E V, V.
data. If humans label the two answers such that y > y', the model might be able push the likelihood of y' given smaller than that of y given x. As result, the good answer y will be generated more likely than the bad answer y'.
SFT considers the next-token prediction in the output y given an input x. For an input-output pair (x,y). y), it mod- els the conditional probability of y given x by (y(x) = The token conditional proba- bility Py (Iy) (11.............. "1...,,,.. 1)11) is modeled by a Transformer:
However, traditional supervised learning methods never use human preference data. For example, in image classifica- tion, training data (x. y) denote an imput image and its true class label y E {1, [1,.... K). We do not need the preference optimization step on preference data saying that dog class is preferred to cat class for an image of a dog. So what is the difference between traditional supervised learning and supervised finetuning of LLMs that makes SFT not enough? The answer lies in the fact that traditional supervised learn- ing methods are usually discriminative approaches, while the standard SFT method is not discriminative.
I1 Po Xx MI......................... 1)-1).
Pa Pay (131,,;;)))
Ty T
E exp(h(w: xz-1) Wi:)
where Wi...., WK denotes the token embedding vectors of that in V. h(w; #1...., x)-1) denotes the representation of the input sequence of tokens produced by transformer network parameterized by w. We let 0 (w, W) to denote all parameters of the LLM.
Below, we introduce our discriminative finetuning (DFT) framework of L.Mss. A discriminative approach aims to push the "score" of the true output to be higher than that of other possibly wrong outputs. In this paper. we examine a classical approach through discriminative probabilistic model. To this end, we introduce a parameterized scoring function se sy x) IR which measures the fitness of y given x. This is similar to the prediction score in traditional supervised learning. We will discuss shortly how to set the scoring function for learning an LLM. In discriminative probabilistic model, we model the conditional probability Pa(y|x) of one output y out of the space of all possible texts denoted by V. In particular, we define
By minimizing the negative log- ligelikelihood of all y1,..., Yn. SFT solves the following problem from a pretrained model:
min
log (x(y)(x;).
(1)
n
In order to differentiate from approach, we refer to (y(x) as the generative likelihood, as it decomposes the likelihood of generating y given x into the product of likelihood of generating each token in y.
DFT: Discriminative Finetuning
In order to motivate our approach, let us first examine the limitation of SFT. Our goal of finetuning LLMs is to ensure that LLMs generate good answers more likely than bad answers. However, SFT only has one-sided optimization power by maximizing the likelihood of tokens in the good output y given x and their preceding tokens. It does not necessarily guarantee that the likelihood of tokens in the bad answer is low. Let us consider simple example:
exp(soly,
Palylx) = Eyeey exp(ss(y', x)/7) Vy E V. (2)
where + > 0 is a temperature hyperparameter. Then, given a set of training data D = (1), y1),.... (xn.yn)} we learn by maximizing the log-likelihood of observed data, i.e. mins
F(0)
(3)
1
n
where
F(0) =
t log Pa(y:[x;)= =
n
Motivation Example
T log
n
exp( s0 xx))
se so x4)
(x) What is the bigger number between 9.11 1 and 9.9?
n2
(y) The bigger number between 9.11 and 9.9 is 9.9.
-Y'y
(y') The bigger number between 9.11 and 9.9 is 9.11.
where scaling the negative log-likelihood by T is for in- creasing the numerical stability, which does not change the optimal solution.
The good answer y and the bad answer y' only differ in the last token. The likelihood of all preceding tokens are the same. Even though the likelihood of the last token "9" in y conditioned on preceding tokens is increased during the finetuning with this data, the likelihood of the token "I1" as the last one might still be high, making generating the bad answer y' likely.
DFT marks a paradigm shift from "token" prediction to "data" prediction. To differentiate from the generative likeli- hood (y()),,, we refer to Pa(y|x) ) in (2) as discriminative likelihood of y given x. By maximizing the discriminative log- likeeiklioood of the training data, we not only increase the score of the true output y; for each input x,, corresponding to the numerator of the discriminative likelihood, but also decrease the scores of other potentially bad answers in ),
To address this issue, the current mainstream approach is to finetune the model further using PO on human preference
3

---

Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data
which correspond the denominator of the discriminative likelihood.
se sy x) 1 log (g(y|x), where |y| denotes the number ly] of tokens in y. This will allow us to compare DFT with some PO approaches using length normalized reward (Meng et al., 2024). As result, the problem becomes:
Finally, we note that while both DFT and traditional discrim- inative classification methods (e.g., logistic regression) use supervised data D = {(x, y:), i = 1, .....,77 n. and model discriminative likelihood through softmax functions, the differences are: (1) DFT operates over an infinite space of possible text outputs y', whereas traditional classification works with a finite set of class labels; (2) the summation over all possible y' in DFT requires advanced optimization techniques, while summation over class labels in traditional approaches is straightforward This fundamental distinction necessitates our novel sampling and estimation approach while maintaining the core advantages of discriminative learning principles.
1 min 0
-
log Pg(xxlx.)
Ys
log Po xx))
)) (5)
Σ Y'EY'
log
exp
ly'lT
4.2
The Optimization Algorithm
Although our DFT formulations (2) and (3) are nearly iden- tical to that of the traditional discriminative probabilistic approach for classification (e.g., logistic regression), the key challenge lies in solving the optimization problem in (3), particularly in handling the second term of F(0), where y encompasses all possible texts. Indeed, the optimization problem in (3) is an instance of empirical X-risk mini- mization (Yang, 2022; Yuan et al., 2023b). We address the optimization challenge by employing advanced opti- mization techniques of finite-sum coupled compositional optimization framework (FCCO) (Wang & Yang, 2022). The idea is to write the second term of F(0) into the form of +/t Ei ((Eg) ()), where Ç is some random variable. To this end, we introduce a sampling distribution P.(.,, which specified later. Then we define
4.1
The Scoring Function
The above framework is similar to the discriminative prob- abilistic modeling for self-super representation learn- ing (Wang et al., 2025). However, we cannot directly borrow the same idea of discriminative representation learning to design the scoring function. In particular, discriminative representation learning uses encoder network e(x) to induce an embedding of any input text x, and computes the scoring function by using the cosine similarity between e(x) and e(y). However, this representation model e(-) is of no use for generative tasks of LLMss.
gi (i()) : Σ exp( soly', xxi) Ey-nP(() Po(y')
exp( ()0(((
To circumvent this issue, we define the scoring function based on the generative log-likelihood log Po (x)x,, as it measures the likeliness of generating y given x. For expect that a high value of the generative
T
y'Ey
The objective becomes:
good model, log- ongllikelhhood log Py(y) x) would indicate high fitness score of y to answer x. With such correspondence, the above discriminative learning framework would increase the chance of generating good output y given x and decrease the chance of generating possibly bad outputs given x. We will examine two simple settings of the scoring function.
n
min 0
80
x;),
n
1
exp(so(y',x)) :.i//)x))))))) (6)
Σ n
log Bymp.()
Py(y')
Next, we discuss three components of our algorithm for solving the above problem.
Setting 1: sp x) log (y))). Plugging this into (3)
results in the following objective:
Sampling Distributions. We need three properties of these sampling distributions: (1) it is easy to sample data from them; (2) is possible to compute the probability value of sample y'; (3) the sampled outputs y' ~ P.(-) are likely be bad outputs in answering x,, To this end, we let P.(.) (-18s),, where pe corresponds to the base LLM 6g to be finetuned, and x; is an augmented text of x; including some system prompts to facilitate the generation of bad outputs. We explore this in our experiments.
min n
log (y)(x))
1 11
log (Erey exp
log Py (x:)
T
Comparing the above objective of DFT to that of SFT in (1), we can see that the first term in (4) is exactly the same as the objective of SFT. The difference lies in the second term that penalizes the possibly bad outputs in y' for each x,, trying to decrease their generative log- liellikei.ood.
Key Updates. Computing a stochastic gradient estimator for the first term is the same as in SFT. The challenge is how to estimate the gradient of log(g:(())) in the sec- ond term using random samples. Its gradient is given by (( (l.) Although (0,(0,) can be simply estimated
Setting 2: For the second setting, we use length normal- ized generative logleiilioood as the scoring function, e.g.,

---

Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data
Algorithm 1 The DFT Algorithm
tion issues of Algorithm that are worth discussing.
1: Initialize Og as the base LLM, and Un 1
The first issue is the Step 5, which sample outputs from the sampling model (- This could increase the train- ing time if it is done online. However, since the sampling model is fixed, we can generate these data offiine for all X1, i = 1,.... n. This could dramatically reduce our train- ing time. In our experiments, we generate m ber of outputs for each data x; from the sampling model and sample B outputs from this pool in Step 5 without replacement, where E is the number of epochs.
2: for = 0, .......T'' 1 do
3: Sample mini-batch St C (x1,...,Xx! Xn
4: for each x; E St do
5: Sample a mini-batch B,, from (- (xx,) via an
offline pool
B num-
6: Update u1,4+1 according to (7)
7: end for
8: Compute gradient estimator G, according to (8)
9: Update 1.+1 using Adam-W
10: end for
Another issue is the numerical stability when calculat- ing the stochastic gradient estimator in Step 8 (c.f. (8)). Take soly, x) log (y(y/x) as an example. Then, (x(1x)) which could be
using unbiased stochastic gradient, estimating 9.(6.j can-
not simply use an unbiased stochastic estimator of gi (O.) since it is a non-linear function gi (0e)), which will yield biased estimator. Following Wang & Yang (2022), we main- tain and update moving average estimators {ui...., Un to track g: (0) for each x;. In particular, at the I-th iteration given solution Gr, we first sample a mini-batch of data S, C [x)..... 3. For each data x; E S,, we sample one or multiple outputs y' ~
exp(s0, (y', /// very small value we are trying decrease the gener- ative likelihood of generated outputs y'. As result, the value of estimators Mj, can be extremely small, e.g., 10-* where x is large number, causing some numerical is- sues. This issue is tackled by maintaining and updating {log uI,..., log un instead of (ui...., ). Specifically,
P (- e.g., by generating them through feeding x, as the input prompt to the base L.LM po. We denote these outputs as B {y'
we denote by Wi,yyy exp( "0, ty reformulated to:
Then (7) can be 18))
( "...,).....Y,,.... V.,.,..
Then update 1j,4+1 by:
exp(log ui,e-1) exp(log(1 - + log ui.))
exp( 86 (x))))
01,+1 = (1- 7)uit + "BBBBB
Σ
(7)
+ +xxp(log log - yEy, wa,ty").
( x2)
Y'EB
For simplicity, let bi, log(1 - 7) + log u4, and Wast log + log Eyerg, Wh,ty's, we have
where 7 (0, 1). With Wie the gradient of log(g:(O.)) (B.(),), where
can be estimated by 21,0+1
exp(logur,t1)) = exp((b,.) exp(us,t)
(0,) Y'EB
Σ
exp(
86, (y x.)
V see (y',
If wi,t < by,t, we let
T T (y')s)
exp(log u1,++1) = exp((b,,) exp(uur - bI,)));
denotes a mini-batch estimator of gz, (0.). We emphasize that the moving average estimator (7) is critical to calculat- ing an accurate gradient estimator of the objective. In our experiments, we show 7 = 1 (i.c., simply using a mini-batch estimator of gi (0.) will yield much worse performance.
otherwise, we let
exp(log u1,41) exp(mx,)) + exp(br: wi,)))
Combining these two cases, we have the following:
exp(log 11,
(9)
eexp(max(uo wt, ()( expp wa,l))
exp(-
Thus, compute estimator of the gradient F(,)) by:
exppmax((e Wi,r))o '(bx, -,,,),
G = [
se. x,)+
where (-)) denotes the sigmoid function. Taking the log on both sides gives the update for log u4,441- To summarize,
X;ES,
exp( 80, [y' Xi)
V (y', xx)
maintain and update Ui,t = log u4,t following:
(8)
S
X,ES, Ui Ui
Y'EB
(y (x4)
bi, log(1 - 7) + Uiit
Finally, we can update the model parameter 0r+1 following the momentum- based methods (c.g., Adam, Adam-W This method has a provable convergence guarantee for solving (3) following Wang & Yang (2022). Our optimization method summarized in Algorithm
W4e: log + log B
1
Ey E Whity"
(10)
ū,,41 max(bi,, w.r} log o wi.el).
Then G, is calculated using exp(ū;,441) in place of ug
Efficient Implementation. There several implementa-
5

---

Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data
Algorithm 2 The DFT2 Algorithm
technique, except for the change on W4,++1 and Ge:
1: Initialize 0g as the base LLM, and Un 1
Wi,+1 (1 7) +
?
Σ
exp(
80, (y', x:)
(12)
2: for t = 0, 1....... T 1 do
B
3: Sample mini-batch batch St C fxI,..., xn)
Y'E
4: for each x; E Sy do
1
5: Sample a mini-batch B3,, from Sp
G
V8g, (y1:
(13)
[
6: Update Wi,r+1 according to (12) and (10)
X.ES. 1
7: end for
50, (y', xi) Veo, (y', x:).
expl
8: Compute gradient estimator G, according (13)
S
11.++1
B
T
9: Update oc+1 using Adam-W
X; X:
Y'EB,,
10: end for
We refer to the algorithm for solving (11) similar to Algo- rithm with the above updates of 14,++1 and G. as DFT2.
5 DFT2: An Approximation Approach
Compared with SFT, DFT has an extra cost of computing pg (x))x)) at each forward step during training. We can further reduce this cost by using approximation. The idea is simple by just using the wongth-normalized gen- erative log-likelihooe as the scoring function so x) = by log Pa (y|x) and dropping P (y''1xx) (x.) in the update of 11,+1 and the gradient estimator G,. Below, we explain this approximation from two perspectives.
Computational Costs: As shown in Figure 4, DFT2 has dramatic reduction in computation costs compared with DFT, it does not need to load the sampling model p5 into the memory and compute Po (y)))) which DFT requires. Compared with SFT, DFT2 has additional costs for comput- ing V8,, (y', x,), y' E B.. Nevertheless, such costs appear in the preference optimization step of existing approaches.
6 Comparison with PO and Self-play
We first explain the approximation via approximating Eyer exp(so(y'; (o(yy,x)) x)) by using the data generated by the base LLM model. Let S = I'........''mm denote a set of outputs sampled for each data x; following the base model Pg (-1x.). Considering that the base LLM has already been trained significantly on a large corpus, hence exp(se(y', x)/r) for y' (-xx) would be much larger than random y' in y. This is verified by Fig- ure 1. Hence, we approximate Ev' exp(s>(y', x)/T) x Ey'sp (y'ssey' drawn from an observation made in Meng et al. (2024). They observed that the samples from a LLM have roughly the same values of so x) =
Let us compare DFT2 with preference optimization (PO) approaches. A standard setting of PO is to finetune a L.LM based on a set of preference data {xx, Yr, y'))) where y, denotes a winning response to x, and y' denotes losing response, labeled either by a human a reward model. Most PO approaches can be cast into the following pairwise loss minimization problem:
min 1 Σ 0 ni
(1)) x;)),
The second explanation is
==1
where re(y, x) denotes some reward function. This framework can be easily extended to incorporate multiple losing responses in the preference data {(x, Ys. V D by solving the following problem:
1 1 log P (x)) Hence,
we can approximate g: (0) by
exp( so(y x) PO xx
exp( sply',x xx)) T
(+)
min 1
m
m
x.)).
(14)
Y'ES!
Y'ES?
1=1
J=1
where the second step is justified by that exp(s>(y'; x)/r) are approximately the same, hence the weighting by
For example, DPO uses reward function P,(y())
x)(y, = Blog Poly) and a logistic loss function
(y' xx;) becomes insignificant.
-loga - - x)). Self-play finetuning (SPIN) uses the same objec- tive DPO except for that y' is generated by the base LLM. SimPO uses a reward function ro(y, = by log (y(x)) and adds a margin parameter 7 to the logistic loss function E(ro(y,, xx), ro(y),)))) log orgooro,,,,,. x,) ro(y!, 77).
x)) x))
With either approximation, we end up with the following optimization problem:
min 0
so x,))
7
1
E (p(so(y",x)/T) (11)
T log
n
m
Similarity: From the perspective of PO, we can regard the sampled data y' E S in DFT2 as potentially losing response to x; and the given output y; as the winning re- sponse. Hence, the objective of DFT2 integrates both the
We solve the above problem using the same optimization
6

---

Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data
are an unhelpful assistant," to the input x; in a chat template (cf. Appendix B.5) and use it input to the base L.M for generation. For both settings, we use Mistral-7B-vO. as our base model. More details of implementation and hyper-parameter tuning are described in Appendix A.
2500
Sampling Random
number of samples
2000
1500
1000
Evaluation Benchmarks. For the first training setting, we evaluate our methods on two widely adopted benchmarks: GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021b). We use zero-shot test accuracy our evaluation metric to assess the model's true reasoning capabilities. For the second training setting. we evaluate on seven diverse benchmarks from the Huggingface Open Leaderboard, in- cluding MMLU (Hendrycks et al., 2021a), TruthfulQA (Lin et al., 2022), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2019), GSM8K (Cobbe al., 2021), ARC (Clark al., 2018), and IFEval (Zhou al., 2023). We follow the few-shot evaluation protocol from Chen et al. (2024c). For IFEval, we report the prompt-level strict accu- racy. In addition, we also consider evaluation using GPT4- as-a-judge on AlpacaEval2 (Dubois et al., 2024).
500
12
-10
6 soly', x)
-4
-2
Figure 1: Distribution of so x) computed with the model checkpoint after one epoch of training on UltraFeedback data, comparing two groups of y'. Random y' samples are generated using a very high temperature, while the "Sam- pling" group represents y' generated with temperature less than used in our method.
power of SFT and PO.
Difference: To understand the difference between the objec- tive of DFT2 and the pairwise loss used in PO approaches, we can rewrite (11) as:
7.1 Results
Table 1 shows the performance of DFT(2) on improving mathematical reasoning capabilities. Table 2 and Table 3 compare DFT(2) with self-play methods and SFT-PO methods, respectively, for the second training setting, We describe our observations below.
sy x) so so so))(x
min-
log Σ y'ESP
expl
72
m
can be seen that the difference between DFT2 and PO is that they use different losses for each data x,. In particular, DFT2 uses the log-sum- exp loss function 7 log( m Ey'esp exp( e0(y' (o=tx'x)==oxxxxx)) for each data x,, while PO uses an averaged loss (oyy,.,.) x)) for each data. Like the cross-entropy loss for classification or the contrastive loss for self- supervuue representation learning, the sum-exxe loss has the property of giving higher weights to potentially bad output y' with larger score so soly,, x) in the gradient computation. In contrast, the averaged loss in PO does not enjoy this property.
DFT variants improve standard SFT. Both DFT and DFT2 surpass SetaMath-Mistrall trained by SFT, achieving state-of-the-an performance among 7B- parameter models GSM8K (79.15%) and MATH (28.62%) Similarly, for general language tasks (Table 2), DFT improves SFT across almost all benchmarks except for MMLU, on which both methods are competitive. In addition, both DFT and DFT2 outperform SFT on average.
Observation 1:
1
1 m -i== (yro(y;;x;) ro('',
DFT variants consistently outperform PO methods on self-play data. In Table 2, we compare DFT(2) with PO approaches using self-play data that is gen- erated by the base model as negative data, including SPIN, SimPO, KTO, ORPO, DPO-p and SimPO-SFT, where the last one just combines the SimPO loss and the SFT loss simi- lar to Xu et al. (2024a). Comparing with SimPO-SFT allows us to verify the advantage of our objective over SimPO loss. For these baselines, we use the same generated y' in DFT as their negative data and finetune the same base model. The results in Table 2 show that these PO approaches on self- play data can not improve SFT. This is different from the observation in Chen et al. (2024c); Xu et al. (2024a), as their experiments are for finetuning an SFT model. Comparing DFT(2) with SPIN and SimPO can justify the effectiveness of our objectives and the optimization algorithm.
Observation 2:
7 Experiments
Setup: We evaluate our proposed DFT framework under two distinct training settings. First, we focus on improving the mathematical reasoning capability of base LLM by using DFT on the MetaMathQA dataset (Yu et al., 2024), which contains 395K samples generated through bootstrapped mathematical questions with augmented reasoning paths. In this setting, we B- 4. Second, we fine-tune base LLM using DFT on the UltraFeedback (UF) dataset (Cui et al., 2023), comprising 61K samples. UF is originally used as preference dataset, where each data pairs a winning re- sponse yu and a losing response yı. For SFT and DFT, we regard the winning responses yw as the ground-truth and discard all losing responses. In this setting, we set B = and generate y' by adding an adversarial prompt like "You
Observation 3: DFT is competitive with SFT-PO ap-
7

---

Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data
Table 1: Testing accuracy on GSM8K and MATH
15
5mmD
62
SHN
260
5
METHOD
GSM8K
MATH
10
KTD
LETR
SFT
METAMATH-7B
66.5
19.8
DP12
METAMAATH-MISTRAL-TTT
77.7
28.2
OFT
36-
DFT (MISTRAI-7B-BASS)
79.15
28.34
52
0.95
DFT2 (MISTRAL-7B -B-SRE)
78.77
28.62
AlpacaEvaa2..
(b) Average accuracy.
proaches. We compare with glate-of-the-ar PO methods including DPO, SimPO, RRHF, R-DPO, CPO, IPO, and KTO after SFT in Table 3. It is notable that the existing training pipeline for PO-based methods first trains an SFT model on the UtraChat-200k (UC) dataset (Ding et al., 2023) and then applies PO on the UF preference dataset. However, this training pipeline gives worse performance than SFT on UF winning data. It is probably because the winning responses in the UF dataset give better ground-truth than the data in UC. To mitigate this issue, we implement three methods SFOO SFT- and SFT- using an improved pipeline (UF-+UF) that first performs SFT on UF winning data and then applies PO on the UF preference data.
Figure 2: (a) AlpacaEval2 LC win rate of DFT, SFT, and PO methods based on self-play data. (b) Average testing accuracy of DFT under different values.
SFT SPPO
ImpP//yki)
SFT
100%//100
-2000
OFT
-1000
4000-
150
150
53
150
Thaining
(a) Log- loeeleiksisoods of positives
(b) Log-likelihoods of negatives
Figure 3: (a) Log-likelihoods of positive examples during training for different methods. (b) Log-likelihoods of nega- tive examples during training for different methods. iteration.
Our results in Table 3 show that DFT variants signifi- cantly outperform all methods using the standard UC-UF pipeline. When compared to the improved UF-UF pipeline, DFT enjoys competitive performance with state-of- the-art PO methods including DPO, SimPO and KTO. This is particularly noteworthy as DFT achieves these results in single training stage, directly fine-tuning the pretrained Mistral-7B-vO.11 model without the preference data.
Training Curves for the Log-likelihood. Figure 3a and Figure 3b illustrate the learning dynamics of different meth- ods by tracking the log-likelihood of positive and generated negative examples during training. We compare DFT with SFT and SimPO as in the Table 2. For positive examples (Figure 3a), DFT maintains a trajectory similar to SFT, while SimPO shows a decreasing trend, which means the objective of SimPO is not effective if using the self-generated nega- tive examples for PO. For self-generated examples (Figure 3b), DFT successfully decreases the log- leelikelihood, demon- strating its effectiveness in distinguishing between positive and negative examples.
Ava22 Results. Finally, we briefly discuss the GPT4- as-a-judge evaluation on instruction following by reporting the congth-controlled winning rate (LC) on AlpacaEval2. The comparison between (2) with SFT and PO-based methods on self-play data shown in Figure 2a, which demonstrates DFT(2) outperforms these baselines. The comparison between DFT(2) and SFT-PPO approaches preference data is shown in Appendix 1. which shows that DFT(2) is competitive with some PO approaches using the preference data, such as KTO, but worse than SimPO and DPO. However, 22T(2) have much shorter lengths for the outputs with an average length of 1359. In contrast, KTO, DPO, SimPO have average lengths of 1449, 1477, 1868, respectively. The GPT4-as-a-judge evaluation tends to favor outputs that are longer. Nevertheless, DFT has competitive if not better performance on verifiable instruction following benchmark IFEval (cf. Table 3).
Comparison with SFT-PO without Preference data Table 4 presents results comparing DFT methods with SFT~>PO without human preference data, where all meth- ods use the same generated outputs from the base model as negative examples. We use the same UF- pipeline as in Table 3, while the only difference is that we replace the losing responses with the same generated responses as DFT in the PO stage of SFT->PO methods. These results demonstrate that the single stage training of DFT(2) is more effective than the two-stage training of SFT~PPO methods for using the self self-generated data.
7.2 Ablation Studies
The advantage of using moving-average estimators In the second training setting, we train DFT using differ- ent values of y ranging from 0.8 to 1.0. The value of 1.0 corresponds to using the mini-batch estimator of gx(() for estimating the gradient. The average testing accuracy on
We present more results to illustrate the effectiveness of DFT compared with SFT and PO-based objectives, the advantage of our optimization using moving-average estimators and the effect of the number of generated samples B in each
8

---

Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data
Table 2: Comparison between DFT, SFT, and PO methods on self-play data in the second training setting. All methods use the same UF winning responses as positive examples and the same generated outputs from the base model äs negative examples, ensuring fair comparison.
METHOD
MMLU
TRUTHFULQA
HELLASWAG
WINOGRANDE
GSM8K
ARC
IFEVAL
AVG.
SFT
62.18
50.04
83.59
78.06
45.26
63.65
49.72
61.79
SPIN
61.99
49.91
83.75
77.90
46.02
61.95
23.11
57.80
SIMPO
62.39
52.08
83.89
78.14
2.58
61.86
18.85
51.40
SIMPO-SFT
62.28
49.59
83.46
77.90
42.53
61.52
43.62
60.13
КTO
61.59
49.32
82.88
79.24
43.97
61.60
38.08
59.53
ORPO
62.26
48.26
83.07
79.16
45.41
62.20
53.41
61.97
DPO-P
62.01
48.66
84.03
78.61
40.48
62.20
25.32
57.33
DFT
61.69
52.23
83.95
78.37
48.22
64.25
51.20
62.84
DFT2
61.66
54.14
83.20
77.82
45.49
64.42
51.20
62.56
Table 3: Comparison between DFT and SFT-PO approaches on preference data in the second training setting. DFT use only the UF winning responses, while SFT-PO methods use explicit preference pairs.
METHOD
DATA
MMLU
TRUTHFULQA
HELLASWAG
WINOGRANDE
GSM8K
ARC
IFEVAL
AVG.
SFT-DPO
UC -FF
57.49
53.15
83.60
77.43
30.55
61,52
39.93
57.67
SPT-SIMPO
UC -UF
58.33
50.67
83.39
76.95
33.36
61.86
40.48
57.86
SFT-RRHF
UC -UF
56.40
43.70
80.37
77.51
0.45
52.99
37.52
49.85
SFT-R-DPP
UC -F
58.29
46.10
84.11
76.40
28.43
61.26
38.63
56.17
SFT-CPO
UC-UF 58.05
47.10
80.73
77.11
35.86
57.17
40.67
56.67
SFT-IPO
-C 59.10
45.45
83.14
77.43
34.12
60.24
42.88
57.48
SFT-KTO
UC-UF
59.72
56.65
84.92
78.14
40.49
63.57
43.62
61 .01
SFT-KTH
UF
62.15
54.53
84.77
77.98
45.41
64.51
51.94
63.04
SFT-
UF -FF
62.28
55.67
84.79
78.22
47.54
64.68
47.69
62.98
SFT-SIMPO
UF
61.20
59.36
85.18
77.03
44.12
63.74
43.81
62.06
DFT
(x,
61.69
52.23
83.95
78.37
48.22
64.25
51,20
62.84
DFT2
UF (F,
61.66
54.14
83.20
77.82
45.49
64.42
51.20
62.56
Table 4: Comparison of DFT with SFT-PO methods without human preference data, where all methods use the same generated outputs from the base model as negative examples.
METHOD MMLU
TRUTHFULQA
HELLASWAG
WINOGRANDE
GSM8K
ARC
IFEVAL
AVG.
DFT 61.69
52.23
83.95
78.37
48.22
64.25
51.20
62.84
DFT2
61.66
54.14
83,20
77.82
45.49
64.42
51.20
62.56
SFT-DDPO
61.11
62.22
85,31
78.69
30.71
65.53
26.43
58.57
SFT-
60.59
66.47
85.65
78.22
2.43
66.13
39.37
56.98
the seven benchmarks of Huggingface Open Leaderboard is shown in Figure 2b with more detailed results reported in Ta- ble 8 in Appendix B.2 B.2.1. It shows that using moving-avern estimator with ? E [0.80, 0.95] significantly improves the performance compared to not using the moving-average average estimators corresponding to 7 1.0. justifying the effec- tiveness of our optimization algorithm.
it to B decreases the performance. We suspect that this is probably due to overfitting, and expect more training data will accommodate larger B, e.g., DFT on the larger MetaMathQA dataset with = better than B 2.
8 Conclusion
In this paper, we have proposed a discriminative probabilis- tic framework for finetuning a pretrained large language model without using any preference data or reward model. Efficient and effective optimization algorithms are devel- oped. Extensive experiments have demonstrated the ef- fectiveness of the proposed methods compared with the standard supervised finetuning method and the existing pref- erence optimization methods.
The effect of B. We compare different values of B 1, 2, in the second training setting on UF data. The results of DFT(2) and other PO-based approaches using the self- play data are shown in Appendix B.3. The results show that increasing from 1 to 2 improves the performance, especially on AlpacalEval2. However, further increasing

---

Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data
Acknowledgments
Cui, G., Yuan, L.., Ding, N., Yao, G., Zhu, W., Ni, Y., Xie, G., Liu, Z., and Sun, M. Uttrafeedbcck: Boosting language models with high-quality feedback, 2023.
Guo, V. Balmaseda, and T. Yang were partially supported
by the NSF SCH grant 2306572.
Impact Statement
Ding, N., Chen, Y., Xu, B., Qin, Y., Zheng, Z., Hu, S., Liu, Z., Sun, M., and Zhou, B. Enhancing chat language mod- els by scaling high-quality instructional conversations. arXiv preprint fttb::335./4233.. 2023.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Dong, H., Xiong, W. Pang, B., Wang, H., Zhao, H., Zhou, Y.. Jiang, N., Sahoo, D., Xiong, C.,, and Zhang, T. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint urXiv:2405.0786. 2024.
References
Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm understand learning from human preferences. In International Conference Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
Dubois, Y., Liang, P., and Hashimoto, T. sength-controlles alpacaeval: A simple debiasing of automatic evaluators. In First Conference on Language Modeling. 2024.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.05862. 2022.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Model alignment as prospect theoretic optimiza- tion. In International Conference on Machine Learning, 2024.
Gudibande, A., Wallace, E., Snell, C. V., Geng, X., Liu, H., Abbeel, P., Levine, S., and Song, D. The false promise of imitating proprietary language models. In The Twelfth International Conference Learning Representations, 2024.
Chan, A. J., Sun, H., Holt, S., and van der Schaar, M. Dense reward for free in reinforcement learning from human feedback. arXiv preprint arXiv:2402.0078% 2024.
Chen, C., Liu, Z., Du, C., Pang, T., Liu, Q., Sinha, A., Varakantham, P., and Lin, M. Bootstrapping lan- guage models with dpo implicit rewards. arXiv preprint (rXiv:2406.09760 2024a.
Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi. X., et al. Deepseck-rl: In- centivizing reasoning capability in llms via reinforcement learning, arXiv preprint arXiv:2501.12948, 2025.
Chen, H., He, G., Yuan, L., Cui, G., Su, H., and Zhu, J. Noise contrastive alignment of language models with explicit rewards. Advances in Neural Information Pro- cessing Systems, 2024b.
Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares, F., Rame, A., Mesnard, T., Zhao, Y., Piot, B., et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792 04992. 2024.
Chen, Z., Deng, Y., Yuan, H. Ji, K., and Gu, Q. Self- play fine-tuning converts weak language models to strong language models. In Forty-first International Conference on Machine Learning, 2024c.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D.,, and Steinhardt, Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021a.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardr, J. Measuring math- ematical problem solving with the math dataset. NeurIPS, 2021b.
Clark, P., Cowhey, I., Etzioni, O.,, Khot, T.. Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457711 2018.
Hong, J., Lee, N., and Thorne, Orpo: Monolithic pref- erence optimization without reference model. arXiv preprint arXiv:2403.070 2024.
Cobbe, K. Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.1416. 14168, 2021.
Ji, X., Kulkarni, S., Wang, M., and Xie, T. Self-play with adversarial critic: Provable and scalable offline alignment for language models. arXiv preprint arXiv:2406.0 04274, 2024.
10

---

Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data
Jung. S., Han, G., Nam, D. W., and On, K.-W. Bi- nary classifier optimization for large language model alignment, 2024, URL http:::///// /estts::/arxiie /ttpxvxiivvoo////// org/abs/ 2404. 04656.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Kung, P.-N. and Peng, N. Do models really learn to follow instructions? an empirical study of instruction tuning. In Proceedings of the 61st Annual Meeting of the Associ- ation for Computational Linguisties (Volume 2: Short Papers), pp. 1317-1328, 2023.
Rosset, C.,, Cheng, C.-A., Mitra, A., Santacroce, M., Awadal- lah, A., and Xie, T. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint urXiv:2404.03715 2024.
Li, J., Zeng, S., Wai, H. T., Li, C.,, Garcia, A., and Hong, M. Getting more juice out of the SFT data: Reward learning from human demonstration improves SFT for LLM alignment. In The Thirty-ighth Annual Conference on Neural Information Processing Systems, 2024a.
Sakaguchi, K., Bras, L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint arXiv:1907.106.: 2019.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: 1a0iv:1107..663300 06347, 2017.
Li, X., Yu, P., Zhou, C., Schick, T., Levy, O.,, Zettlemoyer, L., Weston, J. E., and Lewis, M. Self-alignment with instruction backtranslation. In The Twelfth International Conference Learning Representations, 2024b.
Song, F., Yu, B., Li. M., Yu, H., Huang, F., Li, Y., and Wang, H. Preference ranking optimization for human alignment. Proceedings of the Ehirty-Eightl AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium Educational Advances in Arti- ficial Intelligence, AAAI'24AAAT22AAAATTTS AAAI Press, 2024. ISBN 978- 1778-1-57735-887-22 doi: 10. 10.1609/aaai, v38i17.2988 URL
Z., Xu, T., Zhang, Y., Lin, Z.,, Yu, Y., Sun, R., and Luo, Z.-Q. Remax: A simple, effective, and efficient rein- forcement learning method for aligning large language models. In Forty-firt International Conference on Ma- chine Learning, 2023.
Lin, S., Hilton, J., and Evans, O. TruthfulQA: Measur- ing how models mimic human falsehoods. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214 3252, 2022.
aaai. faai.v38i17. v38113111.00 9865..
Stiennon, N., Quyang, I.., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, F. Learning summarize from human feedback. Advances in Neural Information Processing Systems, 2020.
Meng, Y., Xia, M., and Chen, D. Simpo:
Simple preference optimization with a reference-fre reward. In Advances in Neural Information Processing Systems, 2024.
Tajwar, F., Singh, A., Sharma, A., Rafailov, R., Schneider, J., Xie, T., Ermon, S., Finn, C., and Kumar, A. Preference fine-tuning of Ilms should leverage suboptimal, on-policy data. arXiv preprint ar arXhv:2404.14367, 2024.
OpenAl.
Learning to reason with 11ms. http:////oonni..oo/////e/e/aaaee to-aons-----------ee llms/, 2024.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-2771 2022.
Tang, Y., Guo, D. Z., Zheng, Z., Calandriello, D., Cao, Y., Tarassov, E., Munos, R., Pires, B. Á., Valko, M., Cheng, Y., et al. Understanding the performance gap between online and offline alignment algorithms. arXiv preprint MmXiv:2405.08444 2024.
Pal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., and White, C. Smaug: Fixing failure modes of pref- erence optimisation with dpo-positive. arXiv preprint arXiv:2402, 13228, 2024.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint 20i0:2307.00928,, 2023.
Park, R., Rafailov, R., Ermon, S., and Finn, C. Dis- entangling length from quality in direct preference optimization. In Annual Meeting of the Association for Computational Linguistics, 2024. URL https: //api. 268733207.
von Werra, L., Belkada, Y., Tunstall, L., Beeching, E., Thrush, T., Lambert, N., and Huang, S. Trl: Trans- former reinforcement learning, 2020. URL https: //github, wtt.:/gg.o/o.caaeeeeccc htw.://gggggggggg.o/..go/a////ee

---

Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data
Wang, B. and Yang. T. Finite-sum coupled compositional stochastic optimization: Theory and applications. In In- ternational Conference Machine Learning, pp. 23292- 23317. PMLR, 2022.
Yang, T. Algorithmic foundation of deep x-risk optimization. arXiv preprint arXiv:2206. 00439, 2022.
Yu, L., Jiang, W., Shi, H.,, YU, J., Liu, Z., Zhang, Y., Kwok, J., Li, Z.,, Weller, A., and Liu, W. Metamath: Bootstrap your own mathematical questions for large language mod- els. In The Twelfth International Conference Learning Representations, 2024.
Wang, B., Lei, Y.. Ying, Y., and Yang. T. On discriminative probabilistic modeling for self-supervised representation learning. In The Thirteenth International Conference on Learning Representations, 2025. URL https:// openreview net/ www.:/orummiiii 5HHqCqbb..
Yuan, H., Yuan, Z.,, Tan, C., Wang, W., Huang, S., and Huang, RRHF: Rank responses align lan- guage models with human feedback. In Thirty-seventh Conference Neural Information Processing Systems, 2023a. URL https::tp//......... http:///pppeeeeeeeeeeeeeee net. forum? Ad=EdIGMCHK40
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions. In Pro- ceedings of the st Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 13484-13508, 2023.
Yuan, W., Pang, R. Y., Cho, K., Li, X., Sukhbaatar, S., Xu, J., and Weston, J. E. Self-rewarding language mod- els. In Forty-first International Conference on Machine Learning, 2024.
Wei, J., Bosma, M., Zhao, V., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned lan- guage models are zero-shot learners. In International Conference on Learning Representations, 2022.
Yuan, Z., Zhu, D., Qiu, Z.-H., Li, G., Wang, X., and Yang, T. Libaue: A deep learning library for x-risk optimization. In 29th SIGKDD Conference on Knowledge Discovery and Data Mining, 2023b.
Williams, J. Simple statistical gradient-following algo- rithms for connectionist reinforcement learning. Machine learning, 8:229 1992.
Zellers, R., Holtzman, A., Bisk. Y., Farhadi, A., and Choi, Y. HellaSwag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 4791-48000 2019.
Wulfmeier, M., Bloesch, M., Vieillard, N., Ahuja, A., Born- schein, J., Huang, S., Sokolov, A., Barnes, M., Desjardins, G., Bewley, A., Bechtle, S. M. E., Springenberg. J. T., Momchev, N., Bachem, O., Geist, M., and Riedmiller, M. Imitating language via scalable inverse reinforcement learning. In The Thirty-eightt Annual Conference on Neural Information Processing Systems, 2024.
Zhang, S., Dong, L.,, Li, X.,, Zhang, S., Sun, X., Wang, S., Li J., Hu, R., Zhang, T., Wu, F,, et al. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308. 10792, 2023.
Xu, C., Sun, Q.. Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., and Jiang, D. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprinl arXiv:2304. 12244, 2023a.
Zhao, Y., Khalman, M., Joshi, R., Narayan, S., Saleh, M., and Liu, P. J. Calibrating sequence likelihood improves conditional language generation. In The Eleventh Inter- national Conference on Learning Representations, 2023.
Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L., Durme, B. V., Murray, K., and Kim, Y. Contrastive preference optimization: Pushing the boundaries of LLM perfor- mance in machine translation. In Forty-first International Conference on Machine Learning, 2024a. URL https: // //ooerreview wwtp:/iwwww...eeeeeeemmmmee / d=51 i wwwokpn.
Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S.. Luan, Y., Zhou, D., and Hou, L. Instruction evaluation for large language models. arXiv preprint arXiv:2311.0991 2023.
Ziegler, D. M., Stiennon, N., Wu, Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv: 1909 08593, 2019.
Xu, J., Lee, A., Sukhbaatar, S., and Weston, J. Some things are more cringe than others: Preference opti- mization with the pairwise cringe loss. arXiv preprint arXiv:2312.16682 2023b.
Xu, S., Fu, W., Gao, J., Ye, W. Liu, W., Mei, Z., Wang, G., Yu, C., and Wu, Y. Is DPO superior to PPO for LLM alignment? a comprehensive study. In Forty-first International Conference on Machine Learning, 2024b.
12

---

Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data
A
Implementation Details
Our implementations are based on alignment- 'hanbentthandbook' and TRL (von Werra et al., 2020) training framework. Below provide comprehensive details about our experimental setup and implementation choices.
A.1
Similarity Score
In all experiments, we use the unnormalized generative score s0(y,)) log (y((x) for DFT, and use the normalized generative score so xy) ly] log Pg (y/x) for DFT2.
A.2 Haparetasameters Tuning
Setting 1. For both DFT and DFT2, we follow Yu et al. (2024), and set the batch size to 128, max sequence length to 512, and number of epochs to 3. We tune the learning rate in {5e 7, 8e 7, 2c 6). For DFT, we tune the 7 in {0.8, 0.9, 1.0] and for DFT2, we tune the r in {0.1, 0.2, 0.3]. Details of the chosen hyper-parameters are summarized in Table 5,
Table 5: Hyper- payeer-parameters for DFT methods under setting 1.
Hyperparameters
DFT
DFT2
1.0
0.1
0.95
Batch Size
128
max sequence length
512
Learning Rate
8e-7
LR Scheduler
Cosine
Warmup Ratio
0.1
Optimizer
AdamW
Epochs
Setting 2. We compare DFT variants against several baseline methods: SFT, SPIN, KTO, SimPO, and SimPO-SFT. For all methods, a batch size of 128, maximum sequence length of 1024, and training duration of 2 epochs. We perform learning rate tuming across {3e 7, 5e 7, Se 7, 2e 6), except for KTO where use larger learning rate of 5e-6. For DFT variants, we tune in {0.8, 0.9, 1.0] for DFT and in {0.1, 0.2, 0.3, 0.4] for DFT2, with ry in {0.80, 0.85, 0.9, 0.95] for both variants. The most effective values of T are found to be 1.0 for DFT and 0.3 for DFT2. For the baseline methods, we tune their respective hyperparameters: 33 in (0.01, 0.05, 0.1) } for SPIN; 33 in (6, 8, 10, 12) with a gamma-beta-ratio of 0.5 for SimPO; and 3 in {6, 8, 10, 12] with a -ataaaabeta-ratio of 0.5 and combining weight of for SimPO-SFT. For KTO, we set ÀØ 1, and tune {0.01, 0.05, 0.1), An E [B. 1.5B).
Details of the chosen hyperparameters summarized in Table 6.
Table 6: Hyper-parameters for DFT methods under setting 2.
Hyperparameters
DFT
DFT2
1.0
0.3
0.85
0.90
Batch Size
128
max sequence length
1024
Learning Rate
2e-6
LR Scheduler
Cosine
Warmup Ratio
0.1
Optimizer
AdamW
Epochs
2
13

---

Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data
A.3
Implementation of PF
For SFT-PPO methods under the UC-UF pipeline, we use the released checkpoints produced by SimPO (Meng et al., 2024), where models are first trained using SFT UltraChat-200k for epoch and then undergo preference optimization on UltraFeedback for 1 epoch. For SFT-PP methods under the UF-UF pipeline, we first train the SFT model on UltraFeedback with learning rate of 2e-6 for 2 epochs, then conduct preference optimization UhraFeedback for I epoch.
A.4 Training Costs
We compare the training efficiency of different methods, with results shown in Figure 4. For DFT methods, the time for training of 2 epochs. For SFT-SimPP and SFT-DDPO, training time is split into two phases: an initial SFT phase of 2 epochs followed by preference optimization phase of epoch. Despite the higher computational cost compared to SFT alone, DFT2 offers comparable efficiency to the two-stage SFT- pipeline. All experiments were conducted on 4xA100 80G GPUs. Generation time for negative samples (approximately 1.33 hours for both DFT methods) is not included in this comparison, as it can be performed offline preprocessing step.
DFT
9.21
DFT2
Total Training (hours)
SFT
SimPO
7.27
7.34
6.35
(B=2)
(B=2)
SFT-SimPO
SFT-DPO
Figure 4: Comparison of total training time (in hours) for DFT methods and SFT~>PO methods on the Ulraaeedback dataset.
1750
20
1500
1250
15
Langgh
1000
k70
OPT2
RDPD
DPT
PP3
SIMPO
KTD*
DPO
RRHP
DRO
cPO
SIMPO
0
CP0
ARM
RTO
DFT2
DFT
KTC
DRQ
DRo
RDPo
SWAPD
SIMPO
(a) Average generation length
(b) LC win rate
Figure 5: (a) Alaa2aEval2 average generation length of DFT and SFT-PPO approaches. (b) AlpacaEiva22 LC win rate of DFT and SFT-PO approaches. SIMPO*, KTO*, and DPO* denote training under the UF-UF pipeline.
A.5 Benchmark Details
Table 7 provides detailed information about the evaluation protocol used for each benchmark in the second training setting.
14

---

Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data
Table 7: Benchmark evaluation details including number of shots, metrics, and use of chat templates.
Benchmark
Shot(s)
Metric
Chat Template
GSM8k
strict-match
ARC
25
acc_nos moc_m
HellaSwag
10
acc_norm
TruthfulQA
0
X
MMLU
5
acc
X
Winogrande
5
acc
X
IFRval
0
prompt_level_strict
Table 8: Results DFT with different y values for finetuning on UF winning data.
METHOD
MMLU
TRUTHFULQA
HELLASWWA
GSM8K WINOGRANDE
ARC
IFEVAL
AVG.
DFT W/ y= 0.8
61.42
51.08
83.90
78.14
48.22
64.51
52.50
62.82
DFT W/1 7 0.85
61.69
52.23
83.95
78.37
48.22
64.25
51.20
62.84
DFT W/ 7 0.9
61.75
52.19
83.88
78.37
47.84
64.33
50.46
62.69
DFT W/ 7/>-00 0.95
61.82
50.78
83.93
78.30
46.47
64.42
51.94
62.52
DFT w/7=1
59.46
45.38
80.54
76.64
22.14
63.05
19.96
52.45
B
Additional Evaluation Results
B.1
AlpacaEval2 Results
Figure 5a and Figure 5b presents detailed comparison between DFT variants and SFT PPO approaches on AlpacaEval2. The results demonstrate that DFT variants perform competitively some PO methods such SFT-KTO. They also tend to generate shorter output than models finetuned by most PO-based approaches. It is interesting that SFT-SimPPO trained using UC-UF pipeline generates the longest outputs and it has the highest AlpacaEval2 score. However, its results on benchmark datasets in Table is much worse than our method.
B.2 DFT with Different y
As shown in Table 8, y 1.0 leads significant performance degradation across all tasks, with particularly severe drops in GSM8K and IFEval. But the performance across this range y (0.8, 0.9) is consistently good.
B.3
Detailed Analysis of B Effect
We conducted extensive experiments varying the number of negative samples (B) used during training. Table 9 presents comprehensive results comparing DFT, SFT, and PO methods across different values of B (1, 2, and 4). Figure 6 shows the AlpacaEval2 results. We can see that B has a dramatic improvement over 1, especially on AlpacaEval2 evaluation. However, increasing B to 4 will decrease the performance.
15
Figure 6: Comparison of different methods (DFT, SFT and PO methods) on AlpacaEval2 LC win rate across B E (1,2,4).
B.4 Experiments with Different Base Models
To validate that our approach is effective across different model architectures, we conducted additional experiments using Qwen-2.5-0.5B and Llama3-8 -B-instruttt models. For both models, we applied the same training protocol as Setting 2, using the Feebback dataset for fine-tuning.
15

---

Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data
Table 9: Comparison between DFT, SFT and PO methods for fine-tuning Mistral-7B base model on self-play data across
B E 41. The results show performance across 7 different benchmarks and their average.
METHOD
MMLU
TRUTHFULQA
HELLASWAG
WINOGRANDE
GSM8K
ARC
IFEVAL
AVG.
SET
62.18
50.04
83.59
78.06
45.26
63.65
49.72
61.79
B
SPIN
62.16
50.23
83.67
78.06
46.10
62.03
19.59
57.41
SIMPO
62.29
50.75
83.84
78.06
2.88
61.69
19.41
51.27
SIMPO-SIT
62.53
48.83
83.55
77.82
43.44
61.60
42.70
60.07
KTO
61.27
50.28
83.30
78.53
40.79
63.74
42.51
60.06
DFT
61.76
50.89
83.95
77.98
46.70
64.42
50.65
62.33
DFT2
61.92
52.87
83.02
78.14
44.81
64.68
51.57
62.43
B 2
SPIN
61.99
49.91
83.75
77.90
46.02
61.95
23.11
57.80
SIMPO
62.39
52.08
83.89
78.14
2.58
61.86
18.85
51.40
SIMPO-SFT
62.28
49.59
83.46
77.90
42.53
61.52
43.62
60.13
КTO
61.59
49.32
82.88
79.24
43.97
61.60
38.08
59.53
DFT
61.69
52.23
83.95
78.37
48.22
64.25
51.20
62.84
DFT2
61.66
54.14
83.20
77.82
45.49
64.42
51.20
62.56
B
SPIN
61.94
49.60
83.70
77.98
46.02
62.03
20.70
57.42
SIMPO
62.31
51.39
83.83
77.90
4.47
61.86
19.04
51.54
SIMPO-SFT
62.46
49.47
83.47
78.06
42.15
61.69
39.93
59.60
KTO
60.99
49.87
82.06
78.61
41.62
61.86
40.11
59.30
DFT
61.52
51.80
83.97
78.53
46.32
64.51
50.65
62.47
DFT2
61.78
53.67
83.35
77.90
45.64
65.02
48.80
62.31
Table 10 shows the performance DFT and DFT2 compared SFT when applied to the Qwen-2.5-0.5B model. Both DFT variants demonstrate improvements over standard SFT, with average gains of 0.44% and 0.54% respectively.
Table 11 presents the results when applying our methods to Llama3-8B-instruc.. Both DFT variants show substantial improvements, achieving average gains of 1.88% and 1.86% respectively over SFT.
B.5 Impact of Prompting Strategies
We investigated four different prompting strategies for generating the self- self-play data: (1) direct prompting without any special formatting: (2) structuring prompts using the same chat template during fine-tuning; (3) in addition to (2), structuring prompts with beneficial system messages. (4) in addition to (2), structuring prompts with deliberately adversarial system messages. For setting (3), we follow the generation scripts of UltraFeedback. For setting (4), we add "You are an unhelpful assistant." to the system prompt. Examples of these prompts are given in the box at the end of the appendix. For all settings, we sample the negatives with the following parameters: a temperature of 0.7, a top-p 1.0, a top-k of 50, and a max-tokens of 320.
Figure illustrates the comparative performance these strategies. Our analysis reveals that using structured prompts with adversarial system messages tends to generate more challenging negative samples, leading better performance. Below, provide example of how each prompting strategy is implemented:
16

---

Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data
63.0
62.8
62.6
Average
62.44
62.4
62.34
62.35
No CT
CT Good sys msg
No sys msg
sys msg
Bad sys msg
Figure 7: Average performance scores across different prompting strategies for generating negative samples. "CT" indicates whether chat template formatting was used, and "sys msg" refers to system messages. Results show that using chat templates with adversarial ("Bad") system messages achieves the best performance (62.84%). while beneficial ("Good") system messages yield lower score (62.35%).
Table 10: Performance comparison using Qwen-2.5-0.3 as the base model
METHOD
MMLU
TRUTHFULQA
HELLASWAG
WINOGRANDE
GSM8k
ARC
IFEVAL
AVG.
SFT
47.34
42.88
51.20
55,41
33,43
36.60
17.56
40.63
DFT
47.49
42.77
51.30
56.59
35.56
36.43
17.38
41.07
DFT2
47.15
44.86
51.57
56.67
32.83
37.37
17.74
41.17
Table 11: Performance comparison using Llama3-8B-instruct as the base model
METHOD
MMLU
TRUTHFULQA
HELLASWAG
WINOGRANDE
GSM8K
ARC
IFEVAL
AVG.
SFT
65.66
49.93
78. 90
76.40
73.76
58.95
69.31
67.56
DFT
65.72
54.43
79.66
75.84
75.74
63.73
70.97
69.44
DFT2
65.40
56.03
78.96
75.45
74.60
63.82
71.71
69.42
Examples of Different Prompting Strategies
Raw Prompt Message:
("content": ": "Which animal has two hands, hyrax or dog?", "role": "user")
Direct Prompting:
Which animal has two hands, hyrax or dog?
Chat Template:
</user|>
Which animal has two hands, hyrax or a dog?

<lassistant/>
Chat Template Good System Messages:
< </sytem/>>
The assistant should answer truthfully and be faithful to factual knowledge as well as given contexts, never making up any new facts that aren't true or cannot be grounded in the instruction.</st 

Which animal has two hands, hyrax dog?
 < <lassistant|>
Chat Template Bad System Messages:
<|system/>
You are unhelpful assistant.<// 
>


which animal has two hands, hyrax or dog?
 <lassistant|>
17

---

Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data
B.6 Impact of Sampling Temperature
This subsection examines how different sampling temperatures when generating negative examples from the base model affects DFT's performance. Table 12 presents the results of DFT trained with negative samples generated four different temperature values ranging from deterministic sampling (0) high-temperature sampling (1.0). The results show that moderate temperatures (particularly 0.7) yield the best overall performance across benchmarks.
Table 12: Results of DFT with different temperaure values during sampling.
SAMPLING
TEMPERATURE
MMLU
TRUTHFUL
HELLASWAG
WINOGRANDE
GSM8K
ARC
IFEVAL
AVG.
62.01
50.75
83.76
77.90
46.17
63.99
50.46
62.15
0.3
61.96
50.29
83.77
77.82
46.63
63.99
52.13
62.37
0.7
61.69
52.23
83.95
78.37
48.22
64.25
51.20
62.84
1.0
62.04
52.32
83.90
78.61
45.94
64.25
51.76
62.69
18


Reviews:
[{"review_id": "9Nts8AYCZ2", "reviewer": "Reviewer_DsZh", "summary": "The authors present a new loss function for LLMs called DFT (discriminative finetuning) which combines an SFT loss objective for positive examples with a score-decreasing term for sampled negative answers. They theoretically justify the DFT objective and also present a modified DFT2 loss objective which simplifies several terms in the DFT objective with near approximations that are easier to compute.\n\nThe authors argue that the DFT objective can effectively replace the entire standard SFT-->DPO pipeline of a trained LLM. To justify this point, they evaluate DFT and DFT2 in two different training settings on a variety of datasets (primarily math and QA), showing performance that is generally superior to SFT alone and comparable to SFT-->DPO. They relate the DFT objective to pair preference objectives such as DPO, but argue key theoretical differences that support better training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for detailed comments. We hope our rebuttal can address your concerns. \n\n**Q1**: About the claim \"Eliminates the need for preference data\".\n\n**A:** (1) We intended to mean the human labeled preference data. (2) Actually, our discriminative framework does not necessarily assume the sampled outputs are *bad*, just like like traditional logistic regression for multi-class classification. This is indeed the strength of the discriminative method. As long as the the likelihood of sampling a bad output is much larger than sampling a good output, our discriminative model would be able to leverage that.  We will change our claim to the following **\"..., a novel alternative to SFT→PO that mitigates the burden of collecting human-labeled preference data or training strong reward models\"**\n\n**Q2:** About the claim \"a discriminative ... by explicitly modeling ... among all possible outputs given an input\".\n\n**A:** This claim just states how the discriminative probabilistic framework is defined.  There are two separate things: (1) formulation in theory and (2) optimization in practice. Please refer to response to 8W3x's Q1 as well.   \n\n**Q3:** Using the same generated bad examples for DFT in the UF --> UF pipeline would result in the best comparison.\n\n**A:** Yes, your understanding is correct. Besides the data, we would like to emphasize the difference in the loss function. The benefit of our loss function is detailed in section 6.  We have done the experiment as suggested by using the same generated examples as losing responses in PO methods. The average performance for SFT --> DPO and SFT --> SimPO is 58.57 and 56.98, respectively, worse than DFT. Please refer to Table 1 in https://anonymous.4open.science/r/datas-C111/README.md for detail results.\n\n**Q4:** The comparisons for Table 2 is based on unequal numbers of epochs.\n\n**A:** No. All methods in Table 2 use 2 epochs for training (see details in A.1 Setting 2). And all methods in Table 2 uses the same data, including the number of \"bad examples\". The \"self-play data\" in the caption of Table 2 denotes the data is generated by the base model. \n\n**Q5**. For Table 3, DFT used twice as many as \"bad examples\" comparing UF $\\rightarrow$ UF pipline.\n\n**A**: It is not fair to just compare how many \"bad examples\" are used by our method and PO methods as these \"bad examples\" in our method are generated for free by the base model, while that in PO methods are created using strong reward models. That is also one limitation of existing methods. Our study shows it is helpful to leverage multiple generated samples in a principled manner. Response to your Q3 compares them with exactly the same data and the same training volume for  UF-->UF pipeline. \n\n\n**Q6:** Compare with RPO (Hong et al. 2024).  DPO-p (Pal et al. 2024)\n\n**A:**  We have added comparison with ORPO and DPO-p, where the ORPO has the average score of 61.97 and DPO-p has 57.33, worse than DFT. Please refer to Table 2 in https://anonymous.4open.science/r/datas-C111/README.md for detail results.\n\n**Q7:** Regarding Table 1, it is not clear whether the improvements over competing models in this table were earned by DFT or instead by the particular dataset or training choices made. \n\n**A:** We confirm that we finetuned the same base model on the same dataset as the baseline **METAMATH-MISTRAL-7B** that uses SFT. Hence, comparison with it directly verifies the effectiveness of our DFT method. Other results are included for simple reference. In the revision, we will focus on comparing with the baseline **METAMATH-MISTRAL-7B**. \n\n\n**Q8:** Empirical differences in the likelihoods produced for positives and negatives while training?\n\n**A:** We have plotted the curves for the log likelihood of positives and generated negatives while training, referring to Figure 1 & 2 in https://anonymous.4open.science/r/datas-C111/README.md.  DFT increases log-likelihood of positives while decreasing that of generated data. SFT increases that of positives but does not decrease that for generated data. SimPO decreases that for both. \n\n**Q9:** Figure 3 may not truly capture the full volume of training time.\n\n**A:** A more detailed analysis on training cost has been conducted, please refer to our response to Q5 of Reviewer UHUu. \n\n**Q10:** Is tuning DFT more expensive, regarding the larger grid of $\\tau \\times \\gamma$ ?\n\n**A:** Not really. Actually, the Table 6 and Figure 2 (b) show that $\\gamma$ in the range of 0.8~0.95 works almost equally well. We recommend a value of 0.9 like the momentum parameter in Adam optimizer.  For $\\tau$, we tune it among three values  in our experiments (cf A.1).\n\n**Q11:**  Why do DFT1 and DFT2 rely on such widely different values of tau for success?\n\n**A:** Because (1) different score functions are used by DFT2 (length normalized log generative likelihood) and DFT (unnormalized one);  (2) the elimination of division by sampling probabilities in the loss function in DFT2.", "claims_and_evidence": "\"We address this question by introducing Discriminative Fine-Tuning (DFT), a novel approach that eliminates the need for preference data.\"\n\n* \"novel\" -- in my experience, I agree. There are other loss objectives which combine an SFT \"term\" with a term designed to reduce the likelihood of other examples, but this particular derivation seems novel.\n* \"eliminates the need for preference data\" -- I disagree with this claim. It is clear that there are still preference sets in the data used for this approach. These sets consist of a preferred (original) example and a dispreferred pool of sampled \"bad\" completions. This is clearly still preference data, even if it is not derived from a human's preferences. In fact, this is the single clearest issue I have with this paper's messaging. The proposed loss function is simply a way to use preference data of a particular type.\n\n\"a discriminative probabilistic framework for fine-tuning LLMs by explicitly modeling the discriminative likelihood of an answer among all possible outputs given an input\" \n* This claim is slightly too strong given that both DFT methods actually heavily approximate the sum of \"all possible outputs given an input\" using a small set (often around 2 examples in size) of dispreferred examples.\n\n\"efficient algorithms to optimize this discriminative likelihood\"\n* I agree.\n\n\"extensive experiments demonstrating DFT’s effectiveness, achieving performance better than SFT and comparable to if not better than SFT→PO\"\n* \"comparable to if not better than\" is a mild claim, I think this claim is reasonable.", "methods_and_evaluation_criteria": "The datasets used are classic and standard, if a few are now mostly saturated (e.g. GSM8K). The finetune methods and metric (accuracy) are reasonable.", "theoretical_claims": "There are not proofs so much as derivations -- with occasional empirical justification for approximation. I did read all of the derivations, and did not find glaring issues, but as I am not an optimization expert I do not have direct feedback for the application of FCCO and for the application of various approximation techniques in the derivations.", "experimental_designs_or_analyses": "Yes, I checked the experimental designs for the main comparisons and ablations. I have a few thoughts:\n\n* If I understand correctly, for the UF --> UF pipeline, the original dispreferred examples are used for PO methods, while for DFT, the new \"bad\" examples are generated. Would using the same data not result in the best comparison? For example, using the generated bad examples for DFT in the other PO objectives?\n* Related to my Q1: I find it very important to get a matching \"volume of training\" when comparing different finetuning approaches. The comparisons for Table 2 seem based on unequal numbers of epochs. In addition, I believe that there may be twice as many \"bad examples\" being used for DFT in UF --> UF pipeline as compared to original dispreferred examples used by other PO methods in the UF --> UF pipeline, since we are sampling B=2 bad examples for each good example with DFT. I would appreciate some clarity on the fairness of training in terms of volume of training.", "relation_to_broader_scientific_literature": "The main thrust of this work relates to the DPO (direct preference optimization) (Rafailov et al., 2024) objective produced to simplify the process of conducting reinforcement learning from human feedback (RLHF) (Stiennon et al., 2020; Ouyang et al., 2022; Bai et al., 2022;\nRafailov et al., 2024). -- citations here given by the authors.\n\nOver the entirety of 2024, numerous shortcomings were identified in DPO, and numerous works proposed to address these shortcomings or simplify the process of using DPO and SFT together. The authors identify several of these DPO-derivative works: \"R-DPO (Park et al., 2024), CPO (Xu et al., 2024a), IPO (Azar et al., 2024), SimPO (Meng et al., 2024), KTO (Ethayarajh et al., 2024),\" although I will add a few missing references in the \"Essential references\" section.\n\nThe vast majority of the relevant works are theoretical pair preference optimization objective works, but the paper also relies heavily on the recent optimization work FCCO (Wang & Yang, 2022).\n\nAll references are available in the paper's bibliography.", "essential_references_not_discussed": "Here are a few critical works in the 2024 literature of \"improving upon DPO\" that I find very relevant for inclusion both in terms of context and also, possibly, as experimental baselines:\n\n* ORPO (Hong et al. 2024). This popular work presented a method for using single pair-preference data which simultaneously has an element of SFT and an element of pair-preference learning in its objective. It proved empirically useful.\n* DPO-p (Pal et al. 2024). This work was motivated differently, focusing on a failure mode of DPO, but the conclusion was basically the same: a pair-preference component and an SFT-like component, added together, result in strong performance.\n\nThere are likely others from mid-to-late 2024 that even I am not aware of. This is a hot space. However, I address these two in particular because they might represent strong baselines.\n\n\n\nHong, Jiwoo, Noah Lee, and James Thorne. \"Orpo: Monolithic preference optimization without reference model.\" arXiv preprint arXiv:2403.07691 (2024).\nPal, Arka, et al. \"Smaug: Fixing failure modes of preference optimisation with dpo-positive.\" arXiv preprint arXiv:2402.13228 (2024).", "other_strengths_and_weaknesses": "* S1: Quality argumentation, presentation, and writing.\n* S2: Table 1 shows impressive results for a single stage finetune.\n\n* W1: I do not consider Table 1 to be an effective comparison table, as it is not clear whether the improvements over competing models in this table were earned by DFT or instead by the particular dataset or training choices made. This is a possible downside of using \"benchmark-climbing\" as justification of a new training method in a paper such as this: we do not understand the full context of the comparison and so cannot derive conclusions. In addition, several of the 7b models in this comparison are now very out of date (such as Qwen-7B).\n\n* W2: I would often hope for a work like this to conduct analyses on the trajectories of the likelihoods of examples over training and how it compares to other approaches. For example, the claim from section 6: \" the log-sum-exp loss has the property of giving higher weights to a potentially bad output y′ with a larger score sθ(y′, x) in the gradient computation\" justifies how this approach differs theoretically from other standard PO approaches. Does this difference lead to any empirical differences in the likelihoods produced for positives and negatives while training?", "other_comments_or_suggestions": "* 376 right column: Alapaca --> Alpaca\n* 654: Implemenation --> Implementation\n* Please use bolding to indicate best results in large tables.\n\nNote: score moved from 2->3 after rebuttal.", "questions_for_authors": "* Q1: In Figure 3, the training time per iteration is given. The conclusion one would draw from this chart is that the 2 epochs of DFT are faster than the 2 epochs of SFT + 2 epochs of pair-preference optimization. However, it seems that in section A.2 the actual training done for the SFT+DPO and SFT+SIMPO (and other pair-preference methods) is usually 1-2 epochs of SFT and 1 epoch of pair preference optimization. I feel then that the chart in Figure 3 may not truly capture the full volume of training time, as it implies that both methods had the same number of iterations. Could you please elaborate on this?\n* Q2: Another question about the train time/results comparison: most of the baselines are tuned across a small range of hyperparameters, but the DFT methods are tuned across a larger grid of tau x gamma. Does this mean that in fact the training process, all in, of DFT is much more expensive?\n* Q3: Why do DFT1 and DFT2 rely on such widely different values of tau for success? Even the optimization windows for the two are given differently.\n* Q4: A note about line 206-210 (left column): Should the expectation in line 206 really be outside of g_i, considering that later on 210 we define the expectation over the random sample as part of g_i?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e6V7Q6lsqF", "reviewer": "Reviewer_8W3x", "summary": "This paper introduces Discriminative Fine-Tuning (DFT), a new approach for fine-tuning large language models (LLMs) without using preference data or reward models. Unlike Supervised Fine-Tuning (SFT), which employs a generative approach focusing only on positive examples, DFT adopts a discriminative paradigm that explicitly increases the likelihood of positive (good) answers while suppressing negative (bad) ones. The authors formulate this as a discriminative probabilistic framework that models the likelihood of an answer among all possible outputs given an input.\n\nThe contribution of this paper includes: (1) a discriminative probabilistic framework for fine-tuning LLMs; (2) efficient algorithms to optimize the discriminative likelihood; and (3) extensive experimental validation showing that DFT outperforms standard SFT and achieves performance comparable to or better than preference-based methods. The authors further propose DFT2, a more computationally efficient approximation of DFT.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer's comments. While we understand the reviewer's main concern of lack of theoretical analysis of DFT and DFT2, we would like to emphasize that this is a new yet challenging research area, especially considering the discriminative framework is defined over the infinite data space instead of finite label space. \n\n**Q1**: Negative sample batch size (B) from 2 to 4 causes unexpected performance degradation.\n\n**A**: The performance drop from $B=2$ to $B=4$ only happens on the UF dataset not on the MetaMath dataset. We have discussed the effect of $B$ in the paper at the end of section 7.2. Indeed, our experiments have suggested that more training examples could enjoy larger negative sample batch size $B$. On MetaMath with 395k samples, using  $𝐵=4$ generated outputs is better than $𝐵=2$ generated outputs. \n\n**Q2:** Lacks rigorous theoretical analysis regarding the convergence properties.\n\n**A:** Since we cast the objective of DFT and DFT2 into instances of FCCO and used the same optimization framework as in Wang & Yang, their convergence analysis would apply under appropriate conditions, which guarantees converging to a stationary solution.  We have attributed the convergence analysis to Wang & Yang in lines 233 -234 (left). \n\n**Q3:** The connection between the discriminative likelihood and traditional probabilistic classification models could be developed more thoroughly.\n\n**A**: We will elaborate more in the revision due to limit of space here. \n\n**Q4:** Insufficient theoretical justification for the approximation of DFT2 to DFT.\n\n**A:** We would like to refer the reviewer to a recent work of Wang et al. 2025. They have presented a generalization theory which can be useful for justifying DFT2 in our context. In particular, their Theorem 1 establishes the generalization error of the global contrastive loss in the discriminative probabilistic framework (similar to DFT2's objective) using the uniform approximation of $P_i(\\cdot)=P_g^0(\\cdot|\\mathbf x_i)$, compared with the original discriminative log-likelihood (expectation of DFT's objective) using the sampling probabilities $P_i(\\cdot)$ to re-weight each sampled negative data. The difference lies at the error between uniform approximations and the true ones. \n\nWang et al. On Discriminative Probabilistic Modeling for Self-Supervised Representation Learning. ICLR, 2025. \n\n**Q5:** Needs more rigorous analysis on log-sum-exp loss offers advantages over averaged loss functions.\n\n**A:** The advantage is log-sum-exp loss has the property of giving higher weights to  potentially bad outputs $\\mathbf{y}'$ with a larger score $s_\\theta(\\mathbf{y}', \\mathbf{x})$ in the gradient computation. Please refer to Eq. (13) in the paper, where the gradient has a weight depending on the score $s_{\\theta}(\\mathbf{y}', \\mathbf{x}_i)$  (higher scores will give higher weights). However, averaged loss does not has such property. \n\nWe would also like to point out the connection between log-sum-exp loss and distributionally robust optimization (DRO) (Qiu et al. 2023). The existing theory of DRO (Duchi et al. 2019) can help justify the benefit of DRO compared with the empirical averaged losses. \n\n\nQiu et al. Not All Semantics are Created Equal: Contrastive Self-supervised Learning with Automatic Temperature Individualization. ICML, 2023. \n\nDuchi et al. Variance-based Regularization with Convex Objectives. 2019. \n\n**Q6:**  Training costs doesn't fully account for the cost of generating negative examples.\n\n**A:** Please refer to our response to Reviewer UHUu's Q5.\n\n**Q7:** There's limited analysis of how the quality of generated negative examples affects performance\n\n**A**: We did include in Figure 6 in appendix about the how generation template affect the results. For rebuttal, we also examined how different temperatures of sampling affect the results. Please refer to response to reviewer UHUu's Q4.\n\n**Q8:** How does the quality of the base model affect DFT's performance?\n\n**A:** We have conducted additional experiments on weaker Qwen-2.5-0.5B and stronger Llama3-8B-instruct. Please refer to the response to Reviewer ZjDq's Q2. In all cases, DFT have improvements over SFT. \n\n**Q9:** The GPT-4-as-a-judge evaluation on AlpacaEval2 shows that DFT generally underperforms compared to leading PO methods. \n\n**A**: The AlpacaEval2 evaluation has two parts: (1) comparing with PO methods using the same self-play data, DFT is better as shown in Figure 2; (2) comparing with PO methods using the reward-model labeled preference data, DFT is worse. This is expected as the responses of UF data are ordered by GPT-4. Hence, PO methods for finetuning models on the UF data will be inherently biased to AlpacaEval.  However, we'd like to point out on the verifiable instruction following benchmark IFEval, DFT is competitive if not better than PO methods.", "claims_and_evidence": "- The discriminative approach is theoretically interesting, but several experimental findings warrant further exploration. Notably, increasing the negative sample batch size (B) from 2 to 4 causes unexpected performance degradation (Table 7), contradicting the intuition that more diverse negative samples should improve model robustness. This inconsistency suggests aspects of the method that would benefit from a more detailed explanation. \n\n- Furthermore, the paper lacks rigorous theoretical analysis regarding the convergence properties of the proposed DFT optimization algorithm. No formal guarantees are provided about what objective the model ultimately converges to, even for the ideal setting with infinite negative samples. Similarly, while DFT2 is presented as a computationally efficient approximation of DFT, the paper provides insufficient theoretical justification for this approximation and lacks characterization of the potential performance gap between the two methods. These theoretical aspects could be strengthened to better support the method's foundation and generalizability beyond the specific experimental settings described.", "methods_and_evaluation_criteria": "The methods proposed in the paper address an interesting direction for LLM fine-tuning, but there are some concerns about the evaluation methodology:\n\n- The GPT-4-as-a-judge evaluation on AlpacaEval2 shows that DFT generally underperforms compared to leading PO methods\n\n- The generation of negative examples relies on specific prompting strategies, which raises questions about the claim of eliminating the need for preference data", "theoretical_claims": "The theoretical formulation of the discriminative approach is mathematically sound, but several aspects deserve more scrutiny:\n\n- The connection between the discriminative likelihood and traditional probabilistic classification models could be developed more thoroughly\n\n- The approximations made in DFT2 need stronger theoretical justification, better with theorems\n\n- The claim that the log-sum-exp loss offers advantages over averaged loss functions in PO methods needs more rigorous analysis\n\nWhile the optimization algorithm addresses numerical stability concerns, the paper doesn't thoroughly analyze the convergence properties of this approach for the specific problem of LLM fine-tuning.", "experimental_designs_or_analyses": "- The performance sensitivity to B (number of negative samples) is noteworthy, with performance decreasing when B increases from 2 to 4\n\n- The comparison of training costs favors DFT2 but doesn't fully account for the cost of generating negative examples\n\n- Error bars and statistical significance tests are missing from most experimental results\n\n- There's limited analysis of how the quality of generated negative examples affects performance\n\n- Addressing these points would help readers better assess the reliability and generalizability of the proposed methods.", "relation_to_broader_scientific_literature": "The paper positions itself within the literature on LLM fine-tuning, the literature review is pretty good.", "essential_references_not_discussed": "The paper mentions SPIN (Wu et al., 2024) as a closely related work, which appears to be a citation error. SPIN should be properly attributed to Chen et al., 2024c (\"Self-Play Fine-tuning Converts Weak Language Models to Strong Language Models\"), which was published in the Forty-first International Conference on Machine Learning.", "other_strengths_and_weaknesses": "Additional Strength: \n- The paper introduces an interesting conceptual approach to LLM fine-tuning\n\n- The optimization algorithm addresses practical implementation challenges\n\nAdditional Weakness: \n\n- The observed sensitivity to hyperparameters suggests opportunities for improving robustness\n\n- The paper would be strengthened by expanding the theoretical analysis of why DFT outperforms PO methods in certain scenarios", "other_comments_or_suggestions": "A deeper analysis of when DFT fails would strengthen the paper\n\nThe computational costs should include a more complete accounting of the negative sample generation process", "questions_for_authors": "In your experiments, increasing B from 2 to 4 often decreases performance, which you attribute to potential overfitting. This seems counterintuitive, as more negative samples should theoretically provide more diverse learning signals and reduce overfitting. Could you provide a more detailed explanation of this phenomenon? Are there other factors beyond overfitting that might explain why larger B values lead to worse performance?\n\nHow does the quality of the base model affect DFT's performance? Would a stronger or weaker base model change the relative performance compared to SFT and PO methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dMJpnb4Y1P", "reviewer": "Reviewer_UHUu", "summary": "This paper introduces Discriminative FineTuning (DFT), a novel approach to fine-tuning large language models (LLMs) that diverges from the traditional Supervised FineTuning (SFT) followed by Preference Optimization (PO) paradigm. Unlike methods that rely on preference data or reward models, DFT frames fine-tuning as a discriminative problem, training the LLM to distinguish between good and bad outputs. The core idea is to model the likelihood of an answer being correct among all possible outputs, rather than directly optimizing for a specific target output. The authors propose a scoring function that measures the likelihood of a correct answer relative to other possible answers, and they develop efficient optimization algorithms to maximize this discriminative likelihood. Specifically, the method involves generating negative samples using the base LLM with adversarial prompts, and then training the model to assign higher scores to the correct outputs. The paper presents two variants of DFT: the original DFT, which computes the probability of each negative sample, and DFT2, which approximates this computation for efficiency. The empirical results demonstrate that DFT achieves competitive or superior performance compared to standard SFT and PO methods, without requiring explicit preference data or reward models. \n\nThe paper's findings suggest that DFT offers a promising alternative to existing fine-tuning techniques, particularly in scenarios where preference data is scarce or difficult to obtain. Overall, the paper presents a conceptually simple yet effective approach to fine-tuning LLMs, with strong empirical support for its claims. The method's ability to achieve strong performance without relying on preference data or reward models is a significant contribution to the field, potentially simplifying the fine-tuning process and making it more accessible.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** Weakness about the reliance on self-generation using the base model.\n\n**A:** We agree with what the reviewer said. The quality and diversity of these generated negative samples will matter to some degree. If the base model is super strong, it would not make sense to assume the generated data is bad. However, our framework can be extended if one has a way to verify if an output is good or not, e.g., as in the recent development of reasoning models. We can extend the discriminative framework to discriminate between verified positive outputs from verified negative outputs.\n\n**Q2:** Why choose Mistral as your sole pre-trained model in experiments?\n\n**A:** We conducted additional experiments on different pre-trained models Qwen-2.5-0.5B and Llama3-8B-instruct. Please refer to our response to Reviewer ZjDq's Q2.\n\n**Q3:** Which setting for the scoring function is used in the experiments?\n\n**A:**  For DFT, we use the unnormalized generative score: $s_{\\theta}(\\mathbf{y}, \\mathbf{x}) = \\log P_g(\\mathbf{y}|\\mathbf{x})$.\n\nFor DFT2 we use the normalized generative score: $s_{\\theta}(\\mathbf{y}, \\mathbf{x}) =\\frac{1}{|\\mathbf{y}|} \\log P_g(\\mathbf{y}|\\mathbf{x})$.\n\n**Q4:** How are the negative samples $y'$ generated? What temperature is used for sampling?\n\n**A:** We generated the samples using a temperature of $0.7$, a top-p of $1.0$, a top-k of $50$ (c.f. Appendix B.4). Four different prompting strategies are investigated for the UF data (cf Figure 6). The table below shows the additional experimental results of different temperatures while sampling $y'$:\n\n| Method | MMLU | TruthfulQA | HellaSwag | Winogrande | GSM8k | ARC | IFEval | Avg. |\n| :---- | ---: | ---------: | --------: | ---------: | ----: | ------------: | -----: | ---: |\n| DFT (sampling $\\tau=0$)   | 62.01 | 50.75 | 83.76  |  77.90  | 46.17  | 63.99 |  50.46  | 62.15    |\n| DFT (sampling $\\tau=0.3$) | 61.96 | 50.29 | 83.77 | 77.82 | 46.63 | 63.99 | 52.13 | 62.37 |\n| DFT (sampling $\\tau=0.7$) | 61.69 | 52.23 | 83.95 | 78.37 | 48.22 | 64.25 | 51.20 | 62.84 |\n| DFT (sampling $\\tau=1.0$) | 62.04 | 52.32 | 83.90 | 78.61 | 45.94 | 64.25 | 51.76  | 62.69\n\n**Q5:** Computational cost in terms of FLOPs, GPU hours, and memory usage?\n\n**A:** The table below compares DFT and DFT2 with other methods in total training time. DFT and DFT2 use 2 epochs, SFT→SIMPO and SFT→DPO use 1 epoch for SFT and 1 epoch for preference optimization. For the negative data, we report the generating time of sampling 4 different outputs for each input prompt (2 of 4 are used in the first epoch, the remaining 2 are used in the second epoch). We use 4xA100 80G for the experiments.\n\n| Method     | Generating Time (h) | Training Time (h) |\n| ---------- | ------------------- | ----------------- |\n| DFT (B=2)  | 1.33                | 9.21              |\n| DFT2 (B=2) | 1.33                | 7.27              |\n| SFT→SIMPO  | /                   | 4.49              |\n| SFT→DPO    | /                   | 5.48              |\n\nIt's important to emphasize that there are extra costs in generating negative samples and labeling the preference in the preference dataset used by SFT→PO, especially in scenarios where preference data is scarce or difficult to obtain (e.g., GPT-4-as-a-judge is used to form UF preference data). These hidden costs are not reported in the comparisons because we lack the data or access needed to measure them.", "claims_and_evidence": "The proposed method is based on the assumption that it is easy to sample bad responses. However, this may not always be the case. The quality of these generated negative samples could significantly impact the training process. If the generated samples are not sufficiently diverse or challenging, the model's performance may be limited. \n\nAs evidenced by the paper claimed:\n\"We need three properties of these sampling distributions: (1) it is easy to sample data from them; (2) it is possible to compute the probability value of a sample $\\mathbf{y}^{\\prime}$ ; (3) the sampled outputs $\\mathbf{y}^{\\prime}\\sim P_{i}(\\cdot)$ are likely to be bad outputs in answering $\\mathbf{x}_{i}$.\" (Section 4.2)\"\n\"To this end, we let $P_{i}(\\cdot)=P_{g}^{0}(\\cdot|\\bar{\\bf x}{i})$ ,where $P{g}^{0}$ corresponds to the base LLM $\\theta_{0}$ to be finetuned, and $\\bar{\\bf x}{i}$ is an augmented text of $\\mathbf{x}{i}$ including some system prompts to facilitate the generation of bad outputs. We explore this in our experiments.\" (Section 4.2)\"", "methods_and_evaluation_criteria": "The paper shows results on widely used benchmarks, including math reasoning and commonsense knowledge understanding, based on Mistral-7B-v0.1 model only without the preference data., implying the generated samples were sufficient for training. The ablation study on the number of generated samples (B) (Appendix B.3) suggests the quality and quantity of negative samples impact performance.", "theoretical_claims": "I have checked the correctness of this paper and no issues found.", "experimental_designs_or_analyses": "The experiments of commonsense reasoning part is relied on Mistral-7B-v0.1 and the math reasoning part is based on  MetaMath-Mistral-7B. Why choose Mistral as your solely pre-trained model in experiments? Is it because mistral is the easiest model to gain the performance benefits?", "relation_to_broader_scientific_literature": "The key contributions of this work include the novel discriminative framework, the efficient optimization algorithms, and the empirical validation of the method's effectiveness. The paper's findings suggest that DFT offers a promising alternative to existing fine-tuning techniques, particularly in scenarios where preference data is scarce or difficult to obtain.", "essential_references_not_discussed": "Yes.", "other_strengths_and_weaknesses": "Weaknesses:\n\nWhile the paper presents a compelling approach to fine-tuning LLMs, several weaknesses warrant careful consideration. First, the method's reliance on the assumption that it is easy to sample bad responses is a significant limitation. As the paper states, the sampling distribution for negative samples must be easy to sample from, and the sampled outputs should be likely to be bad outputs. The authors address this by using the base LLM with adversarial prompts to generate these negative samples. However, as I've verified, the quality and diversity of these generated negative samples can significantly impact the training process. If the generated samples are not sufficiently diverse or challenging, the model's performance may be limited. Furthermore, the method's reliance on a self-generated negative dataset introduces a potential for bias, where the model might overfit to the specific characteristics of its own generated negatives, rather than learning a more general notion of quality. This is a valid concern, as the model may learn to discriminate between good and bad outputs based on the specific style or patterns present in its own generated negatives, rather than on a more robust understanding of correctness. \n\nStrengths:\n\nThe primary strength of this paper lies in its introduction of a novel discriminative framework for fine-tuning LLMs. This approach, which contrasts with the traditional generative methods, explicitly models the likelihood of an answer being correct among all possible outputs. This is a significant departure from standard Supervised FineTuning (SFT) and Preference Optimization (PO) techniques, which often rely on preference data or reward models. The paper's conceptual simplicity is also a notable strength. The core idea of training the model to discriminate between good and bad outputs is easy to grasp, and the proposed method is relatively straightforward to implement. The authors' development of efficient optimization algorithms is another key strength. These algorithms ensure the scalability and practicality of the DFT method, making it a viable alternative to existing fine-tuning techniques. The paper also provides extensive empirical validation of the DFT method. The experiments demonstrate that DFT consistently outperforms standard SFT and achieves comparable results to preference optimization methods that rely on explicit preference datasets.", "other_comments_or_suggestions": "No.", "questions_for_authors": "Q1: Several questions arise. First, in Section 4.2, the authors propose two settings for the scoring function. I am curious which setting is used in the experiments. The paper does not explicitly state which setting is used, and it would be helpful to clarify this point.\n\nQ2: In the experiments, how are the negative samples $y'$ generated? What temperature is used for sampling? I wonder whether different temperatures will affect the experimental results. \n\nQ3: How does the computational cost of the DFT method compare to other fine-tuning techniques, such as SFT and RLHF, in terms of FLOPs, GPU hours, and memory usage? The paper provides some information on training time per iteration, but a more detailed comparison using standard computational metrics would be beneficial.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9mKozHa4aa", "reviewer": "Reviewer_ZjDq", "summary": "The paper proposes Discriminative Fine-Tuning (DFT) as an alternative to the standard two-stage fine-tuning pipeline (Supervised Fine-Tuning followed by Preference Optimization, denoted SFT→PO). DFT aims to fine-tune large language models without using any human preference data or reward models, yet still achieve performance on par with approaches like RLHF (Reinforcement Learning with Human Feedback). Instead of the conventional generative fine-tuning (which only maximizes likelihood of the given “good” outputs), DFT introduces a discriminative paradigm. The model is trained to explicitly prefer correct or high-quality answers over incorrect (“negative”) answers for a given input​. This is done by pushing up the likelihood of positive responses while suppressing the likelihood of negative responses​. Theory and experiments are provided to corroborate the algorithm.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** The General language tasks evaluation metric should be written explicitly.\n\n**A:** Thank you for this detailed comment. The information on the evaluation metric is shown in the table below.\n\n| Benchmark  | Shot(s) |  Metric |Apply chat template |\n| ---------- | ------- |  ------ |------------------- |\n| GSM8k      | 5       | strict-match        |x       |\n| ARC        | 25      | acc_norm            |x       |\n| Hellaswag  | 10      | acc_norm            |x       |\n| TruthfulQA | 0       | truthfulqa_mc2,acc  |x       |\n| MMLU       | 5       | acc                 |x       |\n| Winogrande | 5       | acc                 |x       |\n| IFEval     | 0       | prompt_level_strict |✓       |\n\n**Q2:** Additional Experiments on other based models\n\n**A:**  We have tried two more models, a weaker base model Qwen-2.5-0.5B and a stronger base model Llama3-8B-instruct. We did observe improvements as well in these cases.\n\nFor the weaker base model Qwen-2.5-0.5B, we have the following results:\n\n| Method | MMLU | TruthfulQA | HellaSwag | Winogrande | GSM8k | ARC | IFEval | Avg. |\n| :----------- | -------: | ---------: | --------: | ---------: | -------: | ------------: | -------: | -------: |\n| SFT          | 47.34 |   42.88 |   51.20 |   55.41 | 33.43 |      36.60 | 17.56 |  40.63 |\n| DFT          | 47.49 |   42.77 |  51.30 |   56.59 | 35.56 |      36.43 | 17.38 | **41.07** |\n| DFT2         | 47.15 |   44.86 |  51.57 |   56.67 | 32.83 |       37.37 | 17.74 | **41.17** |\n\nFor the stronger base model Llama3-8B-instruct, we have the following results:\n\n| Method | MMLU | TruthfulQA | HellaSwag | Winogrande | GSM8k | ARC | IFEval | Avg. |\n| :----- | ---: | ---------: | --------: | ---------: | ----: | ----: | -----: | ---: |\n| SFT    |   65.66   |      49.93 |     78.90 |      76.40 | 73.76 | 58.95 |  69.31 | 67.56     |\n| DFT    |   65.72   |      54.43 |79.66 |      75.84 | 75.74 | 63.73 |  70.97 |  **69.44**    |\n| DFT2   | 65.40     |  56.03          |  78.96        |  75.45         | 74.60      |   63.82    |   71.71     |   **69.42**   |", "claims_and_evidence": "Claim 1: DFT variants improve standard SFT. Evidence: Both DFT and DFT2 surpass MetaMath-Mistral-7B trained by SFT, achieving state-of-the-art performance among 7B-parameter models on GSM8K (79.15%) and MATH (28.62%).\n\nClaim 2: DFT variants consistently outperform PO methods on self-play data. Evidence: In Table 2, we compare DFT(2) with PO approaches using self-play data that is generated by the base model as negative data, including SPIN, SimPO, KTO, and SimPO-SFT, where the last one just combines the SimPO loss and the SFT loss similar to (Xu et al., 2024a)\n\nClaim 3: DFT is competitive with SFT→PO approaches\n\nThese are all copied from the paper. The claims are supported by evidence.", "methods_and_evaluation_criteria": "Mathematical Reasoning Evaluation: In the first setting, models fine-tuned on a math reasoning dataset (MetaMathQA) are evaluated on GSM8K and MATH, which are well-known math word problem benchmarks. Metric: zero-shot acc.\n\nGeneral Language Tasks Evaluation: In the second setting, the paper evaluates on seven diverse benchmarks from the HuggingFace Open LLM Leaderboard​. These include MMLU (a test of world knowledge across subjects), TruthfulQA (measuring the truthfulness of generated answers), HellaSwag (commonsense reasoning in story completions), WinoGrande (coreference/common-sense reasoning), GSM8K (math, again), ARC (question answering), and IFEval (instruction following evaluation)​. Metric: follows previous work. I think the authors should write it explicitly in the appendix (e.g., few shot or zero shot) (i think it's few shot?)\n\nAlpacaEval2 to assess instruction-following ability​. Metric: winrate and length-controlled WR", "theoretical_claims": "The paper makes a significant theoretical contribution by presenting a discriminative probabilistic framework for fine-tuning LLMs. The core theoretical claim is that one can define a discriminative likelihood over the space of all possible outputs, and optimize this to train the model. \n\nThey further claim a theoretical connection between DFT and preference-learning losses. In particular, the DFT2 algorithm (a variant introduced for efficiency) is shown to relate to the pairwise loss used in preference optimization. The paper points out that DFT’s loss can be seen as a log-sum-exp (softmax) approximation that integrates over multiple negatives at once, whereas typical preference optimization (PO) uses a pairwise logistic loss per comparison​", "experimental_designs_or_analyses": "I think the experiment design is sound", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "I think the experiment is relatively weak compared to the overall contribution.  maybe also try other based models? for example some small models like Qwen 2.5 0.5b or llama 3.2 1/3B, and some newer models rather than Mistral.", "questions_for_authors": "see comments", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a new approach, Discriminative Fine-Tuning (DFT), aimed at improving LLM performance without relying on reward models or explicit preference data. Most reviewers agree that the paper is well-motivated and the core idea is interesting. On the other hand, several concerns were raised, particularly regarding its central claim. While DFT claims to eliminate the need for preference data, reviewers noted that it essentially still utilizes a form of preference data (using labeled "good" examples versus self-generated data as negative examples). This raises questions about whether the paper fully delivers on its core premise. Concerns were also raised about the initial sufficiency of experiments, although the authors provided additional results during the rebuttal phase.

Overall, I would suggest that this paper is borderline, leaning slightly positive. It would be significantly stronger once the suggested revisions, particularly regarding the framing of its claims and potentially integrating the rebuttal's supplementary findings, are completed.
Decision: Accept (poster)