Preference Alignment with Flow Matching
Authors: Minu Kim
Yongsik Lee
Sehyeok Kang
Jihwan Oh
Song Chong
Se-Young Yun
Venue: NeurIPS 2024 poster (2024)
Keywords: Preference Alignment
Flow Matching
Primary area: reinforcement_learning

Abstract: We present Preference Flow Matching (PFM), a new framework for preference alignment that streamlines the integration of preferences into an arbitrary class of pre-trained models. Existing alignment methods require fine-tuning pre-trained models, which presents challenges such as scalability, inefficiency, and the need for model modifications, especially with black-box APIs like GPT-4. In contrast, PFM utilizes flow matching techniques to directly learn from preference data, thereby reducing the dependency on extensive fine-tuning of pre-trained models. By leveraging flow-based models, PFM transforms less preferred data into preferred outcomes, and effectively aligns model outputs with human preferences without relying on explicit or implicit reward function estimation, thus avoiding common issues like overfitting in reward models. We provide theoretical insights that support our method’s alignment with standard preference alignment objectives. Experimental results indicate the practical effectiveness of our method, offering a new direction in aligning a pre-trained model to preference. Our code is available at https://github.com/jadehaus/preference-flow-matching.

Preference Alignment with Flow Matching
Minu Kim
Yongsik Lee'"
Sehyeok Kang'
Jihwan Oh
Song Chong
Se-Young Yun 2f
KAIST AI
minnukki dldydt1r93, kangsehyeok0329, ericoh929, songchong, yunseyoung @kaist.
Abstract
We present Preference Flow Matching (PFM). a new framework for preference alignment that streamlines the integration of preferences into an arbitrary class of pre-trained models. Existing alignment methods require fine-tuning pre-trained models, which presents challenges such as scalability, inefficiency, and the need for model modifications, especially with black-box APIs like GPT-4, In contrast, PFM utilizes flow matching techniques to directly learn from preference data, thereby reducing the dependency on extensive fine-tuning of pre-trained models. By leveraging flow-based models, PFM transforms less preferred data into preferred outcomes, and effectively aligns model outputs with human preferences without relying explicit or implicit reward function estimation, thus avoiding common issues like overfitting in reward models. We provide theoretical insights that support method's alignment with standard preference alignment objectives. Experimental results indicate the practical effectiveness of our method, offering a new direction in aligning pre-trained model to preference. Our code is available at attp:///itttooooo/dddaa///ctcete
Introduction
Preference-ben based reinforcement learning (PbRL) has emerged groundbreaking approach with significant contributions to performance improvement [Akrour et al.) 2011 Wilson et al. 2012]. particularly in the realm of artificial intelligence where understanding and incorporating human preferences are crucial. Unlike traditional reinforcement learning, which struggles due to the absence of explicit reward functions or the infeasibility of defining comprehensive environmental rewards. PbRL leverages a variety of feedback forms from humans to guide the learning process. This class of PbRL method is often referred to reinforcement learning from human feedback (RLHF) [Ziegler
et al.
2019
Levine et al. 2018 Ouyang et al. 2022).
Despite their effectiveness, these methods necessitate fine-tuning pre-trained models to align with user preferences, introducing several challenges such as scalability, accessibility, inefficiency, and the need for model modifications. For instance, with black-box APIs like GPT-4 [OpenAl al. [2024] customization based on user preferences is constrained due to restricted access to the underlying model. Moreover, even if fine-tuning were feasible, the large model size results in inefficient training and high resource consumption. Aligning black-box models with user preferences remains an under-explored area in research, despite its critical importance and growing demand.
In this line of research, we propose Preference Flow Matching (PFM), which redefines the integration of human preferences by directly learning preference flow from the less preferred data to the more preferred ones. This direct modeling of preference flows allows our system to better characterize
Equull contribution. Cooreepponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

RIH Medhods
Flow Matching [Traningt
Flow Matching Inferenee)
SFT Model
Final Model
SFT Model
Model
Fies
Fina-luna
Sumpling
Sampling
Samplng
For training
Train
Solve
Reward Model 7g.0.91
Flow Matching Model
Flow Matching Model
Figure 1: Illustration of our PFM framework. In the typical RLHF scenarios (left), we first sample preference data from the supervised fine-tuned (SFT) reference model. reward model is learned from the collected dataset, either implicitly (as DPO) or explicitly. The reward model is then used to fine-tune the reference policy to obtain the final model. Our method directly learns the preference flow from the collected preference data, where the flow is represented as a vector field to (middle). For inference, we again sample a point from the reference policy, and improve the quality of alignment by using the trained flow matching model, without the need of fine-tuning the existing reference model (right).
and replicate the marginal distribution of the favored outcomes. We adopt a novel flow matching framework (Lipman et al. 2022]. which is simple, intuitive, yet relatively under-explored method for preference alignment. By simply adding a preference flow matching module to black-box models, PFM eliminates the need for fine-tuning the black-box model itself, providing a significant advantage.
Additionally, our method offers a highly robust approach for preference alignment, by circumventing the need for explicit or implicit reward function estimation. In typical RLHF scenarios, model is initially trained approximate reward function based on human preferences. This reward model is then used to guide the policy learning process, aiming to align agent behaviors more closely with human preferences. However, this approach can introduce complexities and potential biases in translating human preferences into numerical rewards. In particular, learned reward models can often overfit the ground truth preference model, especially in the finite data regime [Azar et al.) 2023). Recent ad adveemeents such Direct Preference Optimization (DPO) [Rafallov al. 2024b] address the complexities of RLHF by climinating the need for reward learning. However, these methods still inherently optimize for the reward model, and hence they are also susceptible to reward overfitting. In contrast, PFM directly learns preference flow, thereby removing the need for any reward model assumptions and resolving the challenges associated with reward model learning.
We prove both theoretically and empirically that our method is able to learn an object that similar to the standard RLHF objectives, while being robust to the preference overfitting observed in traditional RLHF pipelines. We also demonstrate how we can further improve the quality of alignment via iterative flow matching, with theoretical guarantees. Experimentally, we find that while typical RLHF methods and DPO suffer from preference overfitting, our method can robustly align with preference and still achieve comparable performances.
2 Preliminaries
2.1 Reinforcement Learning from Human Feedback (RLHF)
Reinforcement learning from human feedback generally begins with obtaining a pre-trained reference policy Tref that can generate samples ~ (-ee(-lz) given a context z. For example, a context could be a text prompt given by user, and the sample y could represent an appropriate text response generated by the reference policy Tre.. We then collect a dataset of N preference pairs D = {( vf y/ }) where each I; denotes the context, and each y, 3/4 ~ (-13() are generated responses to and marked as good or bad samples, respectively. Here, we assume that the preference y,th > y, is generated from a ground-truth reward r²: X Y + R22, where and Y are context space and response space, respectively. The goal of general RLHF is recover

---

optimal policy n such that
argmax Pi (Fa (r u))
3DDL (<(1)))) Tret (
(I)
RLHF pipelines generally require reward learning. One of the most popular choices of the reward model is the Bradley-Terry model [Bradley and Terry 1952]. which assumes that the preference yt y is generated from the probability Ply* > y x:) a(r (1r"(x, yx) r (ar, y )), where
is the logistic function. Under this model, the general RLHF framework learns the reward model
rs r" by minimizing the negative log-likelhhoo::
L D) == (a wor (ro((r) rrr(""")""))))")
Once the reward model ri is trained, we then use it to optimize for ① to obtain To 2 n using standard reinforcement learning algorithms.
There is also class of reward-free methods that eliminates the need of reward learning phase Ratailoy et al. 2024b Azar al., 2023]. Direct Policy Optimization (DPO) [Rfailov et al.) 2024b is
representative reward-frre free method that optimizes for (I) directly without learning reward model. Although being a reward-free free method, DPO implicitly optimizes for the reward function in by replacing ig (z. y) = 8 log( (v x)/ Tref (v)z)) as the implicit reward estimate.
2.2 Flow Matching
Flow matching is a class of generative model, where given a prior distribution Po. we aim to model a target distribution P1 from po- A key difference of the flow matching to the other generative models is that the prior po can be an arbitrary distribution, (diffusion, for example, starts from a Gaussian prior Po) and that the flow matching algorithm learns to modify the prior distribution po the target distribution p1 with neural network.
Throughout, consider pair of data distributions over Rdd with densities y Po and y+ Pi. possibly unknown (but able to sample). The flow matching considers the task of fitting mapping Rdd + Rdd that transforms po pl. that is, if y ~ Po. then f(y pt- Inspired as
in the motivation for the diffusion models, one can define a smooth time-varying vector field w: [0, 1] Rd Rd that defines an ordinary differential equation (ODE),
dy ut, d)
(3)
where we use the notation utt, y) interchanably with (y). Denote the solution of the above ODE by étt, y) (or dr (y)) with initial condition Φo(3) = y. In other words, (y)) is the point y transported along the vector field u from time 0 to t. In order to obtain samples from the target distribution pl.
we simply compute &i (y)) where y Po. The integration map induces pushforward measure
Pt A (Po), which is the density of points y ~ Po transported via u from 0 to t.
To train the vector field vo with a neural network that mimies the vector field u of our interest, we can solve for the conditional flow matching objective, as proposed by [ipman et al. 2022 :
(OF A Te (o, ()..sr)(() (Ilva(t, v) (=())=)
where q(2) =(y ,
) is some coupled distribution of samples y y and un (vl22) y y is
straight path from source sample to target sample. The conditional distribution g(2) can be chosen
to be an independent coupling of source and target distribution q(2) Pu(y )P (y* Lipman et al.
2022. or the 2- Wasserstein optimal transport plan as proposed by Tong et al. 2003
3 Preference Flow Matching
In this section, we describe how we can use flow matching to learn (human) preferences. In the first subsection, we illustrate our flow matching framework for learning preferences, and compare it with typical RI.F pipelines. Then in Section 3.2 we demonstrate our method in simple 2-dimensional
toy experiment. Finally in Section (.33 we provide an extension of our framework that can iteratively
improve the performance, with theoretical guarantees.
3

---

3.1 Flow Matching for Preference Learning
Instead of trying to optimize for the unknown reward r" or the preference probability model Ply+ > y [x), we simply learn a flow from the marginal distribution of less preferred data po (x)) to the marginal distribution of more preferred data pı (yr (x) by leveraging what is explicitly characterized in the collected preference data:
> y lxydy Po (x:) x Tree TV (z) / Teee
(5)
Pi (x) x mrer(y" (x) Tred (v)))())) y/a)dy
(6)
=Teertl [x) (y
Ply* 2/22)
In other words, view that our collected data D is in fact generated from each of the marginal distributions y ~ (o(z)) and yt ~ pi(1(z) obtained from P(yt > y [r)), respectively. Hence, following the conventions in the literature, [Tong et al. 2023
we define the flow matching objective
for preference dataset D as follows:
(8) where we define the condition = (xr, y conditional flow ue (e(y)z) = y ----"" yy, and the probability path pe [v]z) = -N + (1- fyy o²). Once we obtain the vector field do. we can improve upon the generated negative samples y p-(-))) by solving 3 using
()) = P...10.1,.=-..uppp(( (IIv((4, ur (v(2)111)
off-the-shelf numerical ODE solver (Runge 1895 Kutta 1901] to obtain samples fly ~ P1. Specifically, we start from a sample y with t 0. and "flow" along the ODE trajectory using Ug until f 1, to arrive Notably, generating improved
at the target y². Detailed algorithm can be found in Algorithm | samples can be done without the need of fine-tuning the existing model, since we learn a separate vector field that transports negative samples from po to Pl. Furthermore, we did not require any assumption for the probability model P(y > y [ar), so our method can extend to general scenarios that do not adopt the Bradley- Terry model. Our method is outlined in Figure
careful reader might notice that for inference, we require negative samples y from the marginal distribution po. to obtain aligned samples y* However, this pa is inaccessible during inference step, as we must first acquire preference label y > y for samples generated from Tef- Instead, simply start from y ~ Trel as the starting point, and apply flow matching to obtain f(y) = yt p1- We emphasize that PFM can still robustly generate positive samples, we assume non-deterministic preferences i.e., supp(pr) 2 supp(Po). We also empirically find that using Reef instead of Po as the source distribution can produce comparable results in practical scenarios, Further details can be found in Appendix B
3.2 Illustrative Example: Preference Generated from 8- 8-Gaussians Density
Here, we demonstrate how our method learns to improve generated samples to better align with the preference, in simple 2-dimensional toy experiment. We consider ground truth reward function generated from an 8- 8-Gaussians density as illustrated in Figure 2a We then pre-train a Gaussian mixture model to obtain samples as in Figure 2c The pairwise preference labels are then generated using the ground truth 8-Gaussians reward function, as done in many existing meference-based reinforcement learning (PbRL) settings [Christiano et al. 2017 Ibarz al. 2018 Shin et al.! 2023)
As can be seen in Figure 2b the learned reward model overfits in the unseen region, which causes the RLHF method to diverge (Figure 2e DPO also fails to learn the correct preference, as can be seen in Figure We note here that DPO is also subjective to the reward overfitting since DPO also implicitly learns to optimize for the reward using the Bradley- model 2 [Xu et al. 2024
Once preference data are collected, we first learn a reward model Fg via
Azar et al. 2023
However, PFM is free of such reward overfitting issues, we do not optimize for the reward function using the Bradley-Terry model. Unlike other RLHF methods, our model robustly learns to align with
the preference from the provided dataset (Figure 2g). Notably, our method does not try to overfit beyond the unseen region, since the learned target distribution from the flow matching model tries to mimic the distribution pı (y+) of collected preferred samples. (Compare Figure 2dland and Figure 2g

---

10
40
400
(a) Ground truth reward
(b) Learned reward model
(c) Pre-trained model
(d) Preferred samples
Eneoor
pio 1elr
500
(c) RLHE
(f) DPO
gg) PFM
(h) PFM (5 iter.)
Figure 2: Comparison RL.FE DPO, and PFM 2-dimensional toy experiment. We generate preference lahels from a ground truth reward in (a) and a pre-trained Gaussian reference policy (c). Both the RLHF (e) and DPO (f) methods struggle to align with the preferences, due to the overfitted reward model (b), even with the presence of KL regularizer (9 1). PFM is able to mimic the distribution of the positively-labeled samples (d), and therefore achieves the highest performance (g). Repeating PFM iteratively the marginal samples further improve the alignment with the preference (h).
3.3 Improving Alignment with Iterative Flow Matching
As done in iterative variants of DPO [Xiong et al. 2023 Yuan et al. 2024], we can also further improve the quality of aligmment with iterative flow matching. Specifically. upon obtaining marginal distribution P1 by applying flow matching, we again collect a new preference data y y from the obtained marginal distribution p1 in (6). We repeat this process iteratively, by replacing the source distribution (which Tref in the first step) with the marginal distribution P1 obtained in the latest iteration. This iterative process be summarized follows.
(n) (y (x:)
Pi (n 1
(1) /
(n 1) (ylx))() y lx)dy
Po
Pi
(9)
(d) (g+lx) x pi (y+l)(y (ut|x)) (""" (ylx) ylz)dy,
Pi Tref:
(10)
where we denote po" and p1") to be the source and target distribution of the flow matching model at the 7ith iteration, respectively. By repeatedly marginalizing the distribution with respect to the preference Py > y [x). we can effectively "narrow" the sampling distribution towards the outputs with higher preference probability. See Figure 2h for the results of the iterative method in our toy experiment. Note that even during this iterative approach, we leave the parameters of the pre-trained model Aroe untouched, and only require sampling from this model throughout the whole process. Later in Section 4 we formally prove that the iterative method allows us to obtain distribution class that maximizes the ground truth expected preference, and hence yields an optimal policy n² in (T) with 3 = O. See Theorem 4.2
Theoretical Analysis of Preference Flow
In this section, we theoretically analyze why PFM framework can effectively learn to align with
the preference. Interestingly, learning to generate samples from the marginal distribution p1 in (6) optimizes objective that is similar the goal of general RLHF in (T). Following the formulation
provided by Azar et al. 2023 one can observe that the objective ( is equivalent to the below form:
n = argmax E Eyys Eyy Eyg/ (o- > v')) - 3LL.
Trof
(11)
π
where o -1((() = log(E/(1 - {) is the logit function, and we drop the conditional dependence for simplicity. Note DPO also optimizes for the same objective in (II)
5

---

Let us take a step back, and characterize the failure modes of the RLHF and DPO frameworks, by figuring out when these methods overfit the reward. Consider a case where the preferences are deterministic, L.e. Ply > v²) = 1, so that is always preferred to v/. If we plug it into (II), we see that a (P(y > 2/)) - +00. Therefore, the solution π of (11 ends up overfitting for the preference likelihood, resulting in weak or even null KL regularization, regardless of the size of B.
Although in the case where the preference is not deterministic, this phenomenon can still be pro- nounced in the finite data regime [Azar et al., 2023 Even if the true preference is strictly less than This means that overfitting can be a critical issue in general, especially if the action space Y or the context space X is extremely large, as in the case of aligning large language models to human preferences.
we may have access to only a few data samples to estimate P(y y²)
In contrast, the PFM framework learns to generate a marginal distribution pt. One can show that this marginal is solution for the optimization problem that is similar the objective (11).
Theorem 4.1 (Characterization of the Marginal.. Let pi denote the marginal distribution of the soifiveey-labbled samples y². Then the marginal distribution pı oblained from the preference model Fly > y/ (x) is an optimizer of the optimization problem
log Eng' (P(g > v')) DKL (T])
P1 argmax Eyswx
T (Tee(
(12)
We defer the proof to the Appendix C Similar to the RLHF and DPO objective (II.. the solution
P1 of (12) drives the original distribution Tref towards the points where the preference probability Ply > y' is increasing. However, unlike the RLHF or DPO objectives, the objective (12) is bounded
even in the deterministic case Ply > y²) =1, making it robust to reward overfitting.
Interestingly, maximizing the objective [22 is equivalent to minimizing the KL. distance between the policy ग and the normalized preference score with a cross-entropy constraint:
P1 argmin DKL [P]
By x (log (y) where
P(y) By'NTT (P(y v/))- (13)
Hence, the objective pushes the policy x to match the preference P. while encouraging и to align with the high-probability samples in Tref- Since the constraint restricts the policy to the high-probabilityy regions of Tee where the preference labels are collected from, our method is less prone to reward overfitting in the
samples due to the distribution shift in from Tee [Gao et al. 2023, Rafailov et al. [024a].. Though we find this information-theorede formulation interesting, we leave further analysis to future work.
Despite its robustness, one may notice that the objective (12) less flexible compared to the original objective, due to the fixed regularization constant with 3 1. Below, we show that if we apply the iterative algorithm provided in Section 3.3 one can further reduce the KI. regularization strength and obtain an optimal policy x² in (11) with / + O.
Theorem 4.2 (Convergence of Iterative Method). Assume Tref E L and Py > v/) E L Consider an iterative update of the marginal distribution pı in (7). Then, the iteration converges to the uniform distribution of points y where the value Eny- (IP w-)) )) is the largest, i.e.,
Pi (20) - ((U argmax By-naeed (P > })
(14)
where U stands for uniform distribution, and we drop the conditional dependence of x for simplicity.
We defer the proof to the Appendix C Intuitively, the proof follows from the fact that the marginal- ization iteratively "narrows" down the distribution towards the outputs with higher preference. We note here that the 1 assumptions are generally valid in practical domains. See Appendix C
5 Experimental Results
In this section, we conduct experiments to address the following questions: QI. Can PFM align generated samples from the black-box model with preference and achieve comparable results in pructical lasks? Q2. PFM more beneficial than methods optimizing for explicis/implici reward model? and Q3. Is PFM more beneficial than naive add-on methods, e.g., separutely training generative models to imitate preferred samples? To answer these questions, we validate our method in three domains: Conditional text and image generation, and offline reinforeement learning tasks.

---

8
4
0
3
4
7
3 7
3
4
2
9
3
+
9
5
2
3
4
4
4
6
V
3
4
6 7
A
0
5
4
3
8
4
A7 7
O
d
3
4
5 7
2 3
4 5
0
/
J
4
J
7
6
O
M
2
3
4
5 6
4
6
0
5
5 47
2
4444504 6
3
4
/
1
4
5
9
0
/
2/22
3
4
5 6
A 4 9
/
3
4
4
6
+
0
3
5 6
2
3
4
5 6
7
8
2M
7
94
D6
C4
64
(a) Pretrained (0.8389)
(b) Rejected y (0.3951)
Preferred y+ (0.9976)
(d) PFM (0.9841)
2
3
4
f
6
8
a 4
5
6
7
3.3
4 4
5
6
8
3
4
5
7
8
1 4
5
6
8
5
7
2
2
1
5
7
0 3.340
1344500 6
5
3.5 14435
b 2
P
+
0 / 9/134
1.445000 6
3.4
4
A. 52445677
3
4
7.8
/
0/
/ J
0
2 3 5 (
2
4
6
38
0 / ) 3 4 4.5550
6
20
0
2 AAS.0
4 x >
JK
2
3
4
6
4
5
#
F
7
5
7
0
3
4
6
?
(c) RLHF (0.9171)
(f) DPO (0.8936)
DPO 3 << 1 (0.5397) (h) Iterative PFM (0.9996)
Figure 3: Comparison of RLHF, DPO, and PFM on a conditional MNIST image generation task. Numbers represent the preference score. PFM (d) demonstrates superior sample quality and preference alignment compared to RLHF (e) and DPO (f), where DPO collapses with small size of B (g). The iterative PFM with only two iterations (h) results in almost perfectly aligning with the preferences.
5.1 Conditional Image Generation
We first evaluate PFM on a conditional image generation task using the MNIST dataset [LeCun et al. 1998]. Specifically, we utilize a pre-trained DCGAN [Radford et al., 2015 generator as Tref and collect sample pairs from (-(z)) conditioned on the digit labels x E (0.- 9). To construct preference datasets, we assign preferences to sample pairs according to the softmax probabilities of the labels from a LeNet [LeCun et al. 1998). Then, we learn PFM flow vo that transports y to y² given a condition x. More experimental details are provided in the Appendix
Figure 3a illustrates the generated samples from Tref, and the rejected and preferred images depicted in Figure 3bland and Figure Bel respectively, where the values in parenthesis are the measured preference score. As shown in Figure 3d PFM achieves higher preference alignment and better sample quality than RLHF (Figure Be) and DPO (Figure 3f) without fine-tuning Tref- Moreover, PFM achieves nearly perfect alignment with the preferences after only two iterations (Figure 3hh) demonstrating the effectiveness of iterative PFM.
5.2 Conditional Text Generation
Next, we adopt a controlled (positive) sentiment review generation task. As done in Rafailov et al. (2024b), to perform a controlled evaluation, we adopt the pre-trained sentiment classifier as the preference annotator. We train a preference flow on randomly selected pairs of movie reviews y+, y from the IMDB dataset [Mas et al. [001]. For our PFM framework to be applied to variable-length inputs, we employ a T5-based autoencoder to work with fixed-sized embeddings. We adopt GPT-2 SFT model on the IMDB dataset as reference model Tref. We also compare our method with RLHF (PPO) fine-tuned policy ТPPO- See Appendix D for detailed experimental settings.
Below, we report the average preference score (from the classifier annotator) of 100 randomly generated review samples for each method. As shown Table [ PFM is able to improve the preference score of any baseline model to which it is attached. We emphasize here that our method requires relatively smaller training cost compared to the standard RLHF frameworks, even in the iterative settings. See Table Appendix D for the number of parameters that require training for each framework tackling this task. PFM requires training a much smaller number of parameters (around 1. 1.2%) while still achieving better performance. We also note here that instead of iteratively training the PFM module as described in Section 3.3 simply applying the same learned preference flow iteratively to the improved samples achieves the best performance. (See Appendix D
7

---

Dbiribtion hroforance Scors Sampas
Obiribtion Proferance Scors Sampls
PurPPM
Inderna Paler
SFT vs PFM
PPO PFM
Figure 4: Distribution of preference scores for each method. Left visualizes the distribution of scores for the pre-trained reference policy and PFM-attached policy. Without fine-tuning the reference policy, PFM can obtain substantially better results by only adding a small flow-matching module. Right visualizes the preference distribution the RLHF (PPO) fine-tuned policy, and PFM added policy to the PPO fine-tuned policy. Note that PFM is trained with the original dataset, not by the dataset generated from the PPO fine-tuned policy.
Table I: Average preference scores of 100 test instances.
Table 2: GPT-4 win rate over 100 test samples.
Tea
Rref
PFM
Tref + PFMx5
Method
vs. Nze
ТPPO
00.3677
0.6399
2.7469
Tref PFM
100%
2%
Tref + PFMx5
85%
ТPPO
#PPO + PFM
ПРPо
5
100%
2.0156
2.178
2.7894
TPPO + PFM
99%
ЯPPО + PFMx5
100 %
Interestingly, observe that the distribution of the scores tends to shift more toward that of the preferred samples with an increasing number of PFM iterations. (See Figure(4) This result aligns with theoretical insights: the PFM framework learns to shift the source distribution (i.e., the distribution of the reference policy) toward the marginal distribution of the more preferred samples.
We also compute the win rate with GPT-4 evaluation. The results are summarized in Table Both PFM and PPO fine-tuned policy excel the reference policy with win rates 100%. (First column observe that the iterative PFM with 5 iterations
of Table 2 Furthermore, outperforms the PPO fine-tuned policy. If PFM is added on top the PPO fine-tuned policy, we observe near 100% win rates for both PPO + PFM and PPO + Iterative PFM.
the reference model
5.3 Offfine Reinforcement Learning
Finally, we employ the D4RI. (Fu et al., 2020 benchmark to assess the performance of PFM in reinforcement learning tasks. Following prior works on the PbRL literature, we adopt trajectory- based preference alignment [Hejna et al. 2023 Kim et al. 2023). We first randomly choose an starting state so S. and sample two trajectories T and with fixed length é from Fref
-+ (ag; (1, ag) Tee
(15)
F = a'. .,) Tee(
(16)
Then, we obtain the preference + > T- given the starting state 89 using a scripted teacher approach that has also been widely adopted in the PbRL settings (Lee al. 2021 Kim et al. 2023 which prioritizes trajectories with higher rewards based on the ground truth reward. For inference at a given state 8y, we again sample an action trajectory T (a.. (++4))) from (-)s,) and apply flow matching to obtain a better action sequence.
The baseline methods for comparing the performance of PFM include behavior cloning (BC), which we adopt as our pre-trained reference model Tyef, and a DPO fine-tuned model from the BC model. Additionally, train separate behavior cloning model to the collected preferred samples y* P1. aiming to replicate the marginal distribution of the "good" trajectories. Further experimental details deferred to Appendix D
Table presents the outcomes from evaluations conducted on 12 offline datasets. Our findings indicate that PFM consistently demonstrates comparable even superior performance with lower

---

In this remaining section, we focus on answering the remaining questions, Q2 and Q3. We first investigate why PFM can be advantageous over previous methods with explicit/implict reward modeling. As can be seen in Figure DPO, like typical RLHF approaches, is also prone reward overfitting, and may cause the agent to fail. This is because if the preference estimate is close to 0 or 1, these methods may end up overoptimizing to the exploding reward model [Ziegler et al. 2019 adopt Table Normalized results MuloCo datasets, Mean and standard deviation from seeds reported.
Dataset
BC
DPO Fine-tuned
PFM (Ours)
Marginal BC
anl-random-v2
31.59 0.05
31.52 = 0.08
31.62 0.13
25.01 4.64
ant-medium-v2
90.16 = 21
95.04 = 13.93
96.73 = 2.47
99.67 = 1.57
ant-expert-v2
125.83 24.07
134.96 3.76
132.20 = 2.69
99.29
34.74
hopper-random-v2
3.17 0.25
3.23 = 53,47
7.69 =
5,48 4.46
hopper-medium-v2
52.83 = 5.03
= 3.92
58.76 2.62
40.44
1.69
hopper-expert-v2
111.27 0.48
111.51 = 0.92
111.70
+ 0.77
32.39 +
halfchhcttah-random-yy
2.25 = 0.01
2.26 = 0.01
2.26 = 0.0
2.21 0.02
hallcheetta--mediu--22
40.97 0.89
41.94 .9.94 0.68
43.49 0.88
38.79 1.27
hallechectah--exert--2
91.02 = 1.24
92.15 = 0.76
90.05 = 0.83
4.77 = 2.5
walker244randdom-22
1.47 = 0.1
1.38 = 0.08
1.77 = 0.13
2.45 0.38
walker2d-meeiium--2
60.35 18.16
74.05 12.05
72.59 =
65,29
12.58
walker2d-expert-v2
108.62 0.39
108.38
+ 0.28
108.36
+ 0.21
15.8 + 0.54
Random Average
9.62
9.60
10.84
8.79
Medium Average
61.08
66.13
67.89
61.05
Expert Average
109.19
111.75
110.58
38.06
D4RL Average
59.97
62.49
63.10
35.97
ireied thile
eetivation stable
ovneeottmated
ovet-arlimated maned,
Figure 5: Analysis of sample episode of a DPO fine-tuned model on the MuJoCo ant environment. DPO finc- tuned model often overestimates the reward due to reward overfitting (e.g.. / = 196). This can cause the policy choose problematic actions. Here, the implicit reward estimation is fa(s, a) 3 log (=)(als)/TTre(()
variance across all baseline methods. Notably, our method excels particularly in datasets generated from suboptimal behavioral policies, achieving better performance. Furthermore, PFM manages to match the performance on expert datasets even in the absence of reward model, underscoring its robustness and effectiveness. This demonstrates that PFM can effectively align black-box models with preference through flow matching, without the need fine-tune the pre-trained model.
5.4 Is Learning Flow Truly Beneficial?
Gao et al. 2023]. PFM, on the other hand, is inherently robust such over-estimation. completely different optimization framework that does not require a reward proxy. (See Theorem
On the other hand, we observe less performance gain in the expert datasets. This is possible failure mode PFM, where the generated samples are already near-optimal. In such regime, an arbitrary source y ~ Tref has near zero probability of being sampled from the true marginal Po. suggesting that PFM with prior as गeef might suffer from shifted source distribution. We verify this experimentally on walker2d, where PFM struggles the most. By adopting true marginal po as the source, PFM with prior Po can achieve the highest performance among all baselines. This performance is evident even on the expert dataset, matching theoretical analysis. See Appendix

---

Next, we compare PFM to an alternative approach that simply tries to approximate the marginal distribution directly from the positive samples. Intuitively, training a generative model from scratch that replicates the marginal p1 is as computationally costly training the original reference model. Experimentally, we observe that PFM achieves better performance than training a behavior cloning model (Marginal BC) to replicate the distribution of the preferred samples (Table However, it is also worth mentioning that the Marginal BC model does occasionally yield the best results, suggesting the potential of using marginal distribution for preference alignment.
Related Works
Contrastive Preference Learning (CPL) [Hejna et al. 2023] is a class of reward-fres methods that utilizes contrastive learning techniques to align model outputs with the preferences observed in the dataset. By leveraging contrastive loss, CPI. encourages the model to distinguish between preferred and less preferred outcomes effectively. Flow-to-Better (FTB) [Zhang et al. (023) innovatively uses diffusion model to transition from less preferred data to more preferred data, similar to the flow-based approach in our work. However, FTB mainly focuses data augmentation, where they used the trained diffusion model to generate more data samples for behavior cloning. Despite their strengths, both works rely on the Bradley-Teri Terry model to implicitly learn the reward function.
Identity Preference Optimization (IPO) [Azrr al., 2023] builds upon the foundation laid by DPO, extending the framework to accommodate broader range of preference models beyond the Bradley- Terry paradigm. particular, they focus on finding an objective that is bounded even in deterministic preference regime, by replacing the function a in (11 with an identity function T(x:) x. This effectively mitigates the reward overfitting problem observed in DPO and standard RLHF methods.
Our method distinguishes itself from these approaches by not requiring the Bradley-Terr assumption nor the fine-tuning of pre-trained models. This eliminates the risk of reward overfitting associated with the Bradley-Terry model and reduces the computational cost significantly. By avoiding the need for fine-tume fine-tuning, our method offers a more efficient and scalable solution for integrating human preferences into reinforcement learning systems. This makes our approach particularly suitable for scenarios where computational resources are limited or where quick adaptation human feedback is essential. The comparison of these related works is summarized in Table [
Table 4: Comparison of method to other works.
RLHF
DPO
IPO
CPL
FTB
PFM (Ours)
Reward Model Free
V
/
No Reward Assumptions (e.g. BT)
X
Applicable to Black-Bol Models
X
X
X
X
/
/
7 Conclusion and Limitations
In conclusion, this research introduces Preference Flow Matching (PFM), novel add-on approach that offers a practical, efficient, and scalable solution for integrating human preferences. This research highlights the potential of flow matching powerful tool for preference alignment and opens new avenues for further exploration and development in the field of RLHE The ability to align black-box models with human preferences without extensive model modifications marks critical step forward, with broad implications for the deployment and usability of Al systems in real-world applications.
Our theoretical and empirical analyses demonstrate that PFM achieves alignment performance comparable to standard RLHF methods while being more resilient to preference overfitting. The iterative flow matching technique further enhances alignment quality, by continually refining the preference alignment without modifying the underlying pre-trained model parameters.
Despite these promising results, the current design of the PFM framework entails several challenges the natural language processing (NLP) domain. The PFM framework, as currently designed, relies on the autoencoder to work with fixed-sized embeddings to handle variable-length texts. To scale our method to more complex NLP tasks, future research should explore ways to adapt the PFM framework to form texts.
10

---

Acknowledgements
This work was supported by Institute for Information communications Technology Promotion([ITP grant funded by the Korea goveriment(MSIT)NN... --A.TTNN..OS-211111111100 Artificial Intelligence Graduate School Program(KAIn) ( the Institute of Information & communications Technology Planning Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2022-0-00871, Development of AI Autonomy and Knowledge Enhancement for AI Agent Collaboration), the National Research Foundation of Korea(NRF) grant funded by the Korea (Nvermmen(MMST] (No. RS- RS-2019- N0--0000000 Stochastic Analysis and Application Research Center (SAARC)), Institute of Information & com- munications Technology Planning & Evaluation (ITTP) under Open RAN Education and Training Program
grant funded by the Korea government(MISIT) and con- ducted by Center for Applied Research in Artificial Intelligence (CARAI) grant funded by DAPA and ADD (UD230017TD).
We thank Taehyeon Kim at KAIST for pointing out the strengths of our work and providing motiva- tions of this work. We would also like to appreciate Sihyeon Kim and Yongjin Yang at KAIST for providing experimental details and suggesting relevant settings for PbRL tasks. Finally, we thank Junghyun Lee KAIST for revising the details of the proofs of theorems.
References
Riad Akrour, Mare Schoenauer, and Michele Sebag. Preference-based policy learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, PKDD 2011, Athens, Greece, September 5-9, 2011. Proceedings, Part 11, pages 12-27. Springer, 2011.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:2310. 12.0..220...... 2023.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 394344) 1952.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 00835-10866 PMLR, 2023.
Joey Hejna, Rafacl Rafailov, Harshit Sikchi, Chelsea Finn, Scott Nickum, W Bradley Knox, and Dorsa Sadigh. Contrastive prefence learning: Learning from human feedback without rl. arXiv preprint arXiv:2310. 13639, 2023.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Pref- erence transformer: Modeling human preferences using transformers for rl. arXiv preprint arXiv:2303.00957, 2023.
Wilhelm Kutta. Beitrag zur näherungsweisen Integration totaler Differentialgleechumeen Teubner, 1901.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-232 1998.
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv preprint (rX,5;2106.00001, 2021.
11

---

Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, and Deirdre Quillen. Learning hand- eye coordination for robotic grasping with deep learning and large-scale data collection. The International journal of robotics research, 37(4-5):421 4436, 2018.
Yaron Lipman, Ricky TQ Chen, Heli Ben- Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142-150, 2011.
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine. Gabriel Bernadett-Shapirg Christopher Berner, Lenny Bogdonoff,, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage. Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang. Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang. Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, tukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger. Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning. Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil. David Medina, Aalok Mehta, Jacob Menick, Luke Metz. Andrey Mishchenko. Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichir Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder. Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shich, Sarah Shoker, Pranav Shyam, Szymon Sidor, Erie Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong. Lauren Workman. Sherwin Wu, Jeff WU, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan. Wojciech Zaremba, Rowan Zellers, Chong Zhang. Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow
12

---

instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06433 2015.
Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization direct alignment algorithms. arXiv preprint arXiv:2406.02900 2024a.
Rafacl Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024b.
Carl Runge. Über die numerische auflösung von differentialgllichunnen.. Mathematische Annalen, 46(2):167-178, 1895.
Daniel Shin, Anca D Dragan, and Daniel S Brown. Benchmarks and algorithms for offline preference- based reward learning. arXiv preprint arXiv:2301.01392, 2023.
Alexander Tong, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector- Brooks, Kilian Fatras, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport. arXiv preprint arXiv:2302.000 00482, 2023.
Aaron Wilson, Alan Fern, and Prasad Tadepalli. A bayesian approach for policy learning from trajectory preference queries. Advances in neural information processing systems, 25, 2012.
Wei Xiong,
Hanze Dong. Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl- constraint. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for IIm alignment? a comprehensive study. arXiv preprint arXiv:2404. 10719, 2024.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint 11XXi::2401.10020, 2024.
Zhilong Zhang, Yihao Sun, Junyin Ye, Tian-Shur Shuo Liu, Jiaji Zhang, and Yang Yu. Flow to better:
Offline preference- based reinforcement learning via preferred trajectory generation. The Twelfth International Conference Learning Representations, 2023.
Tianchen Zhu, Yue Qiu, Haoyi Zhou, and Jianxin Li. Decoding global preferences: Temporal and cooperative dependency modeling in multi-agent preference-base reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 17202-17210, 2024.
Daniel M Ziegler. Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv 2019.
preprint
13

---

A Flow Matching Algorithm for Preference Alignment
Below, we outline our method in Algorithm [ Although the vector field vg is trained to transport the source distribution of less preferred samples po to the target distribution, we use Tref for inference instead of Po due to inaccessibility to the preference model.
Note that with the non-deterministic preference assumption, PFM is still theoretically guaranteed obtained from Tee instead of po- In particular, we
obtain Φ1 (1) P1. even the sample assume supp(p1) 2 supp(Po), i.e., for any sample yt obtained from the target distribution Pt. the probability of y ~ Po is non-7ero. Then, for any sample y Tyef sampled from the reference model, we can guarantee that this sample has non-zero probability of being sampled as less preferred, indicating that there a learned flow from this point to a better sample, with non-zero probability, In the next section, we provide empirical evidence that further justifies the use of Reef instead of po for inference.
Algorithm 1: PFM: Preference Flow Matching
Training
2 repeat
Sample (I, gyttty1 D
ue(y]z) + yt 11
pe(y]z) +- NVyltyt + +11 t)y
Sample ~ 10. 1| and y ~ Pa (-12)
0 0 - v)((() # from
until Ug converges;
Inference
10 Sample TTe((-12)
Solve ODE 3 with initial condition (o(y) y using ve
12 return (1))
B Empirical Evidence for Using Reference Policy Instead of the True Marginal During Inference
We conduct experiments on the D4RL walker2d environment to compare the performance of the flow matching method using ref and po as source distribution, respectively.
Table 5: Normalized results on MuloCo datasets. Mean and standard deviation from seeds are reported.
Pretrained (BC)
PFM from Tye
PFM from Po
Planning
walker2d-random-y v2
1.47 0.1
1.77 0.13
2.03
1.01
1.93 1: 0.13
walker2d-medium-v2
60.35 1: 18.16
72.59
15.8
80.58
1.80
76.27
1: 7.97
walker2d-expertt-2
108.62 + 0.39
108.38 = 0.28
109.19
= 0.37
109.08
+ 0.11
Average
56.81
60.91
63.93
62.43
To replicate the actual marginal distribution of less preferred samples Po, we sample two trajectories from given state s,, and use the ground truth reward function select the less preferred data (with lower reward). Note that this use of the environment reward is restricted in practice, as it assumes access to the preference model during inference. We also compare another baseline model, namely planning model, that searches among the two generated trajectories r+,-" T ~ Tree (= and chooses action sequence with higher environmental reward.
As can be seen in Table 5 the flow matching model with the source distribution matched to actual marginal distribution po (as in the training process) achieves the highest performance among all baselines, including flow matching model that uses Tref instead of Po- However, we observe that the flow matching model with Teef as the source distribution yiclds comparable results to the
14

---

model using Po as source. It is also worth noting that the flow matching model (with true po) achieves better performance even compared to the planning method, that explicitly uses a trajectory with higher reward. This suggests that using flow matching for preference alignment can provide better alignment than simply conducting an exhaustive search.
C Proof of Theorems
In this section, provide proof to the theorems in the paper. Throughout this section, we write μe as the reference policy instead of The for the case of writing.
First, we prove Theorem where we rewrite the theorem statement for completeness.
Theorem C.1 (Characterization of the Marginal). Suppose that a preference dataset D = V(z,y". . )) is collected from pre-trained reference policy μ., i.e., y²,y= u(-lzr,, where the preference yt > y is labeled with probability P(y+ > >y y= Let p1 denote the marginal distribution of the pasitively-labelee samples y² Then the marginal distribution p1 obtained from the preference model P(y > f'(z) is an optimizer of the optimization problem
P1 = argmax Bynn (log (Pl
(Ply ((w>>>)) DKK (T/lI)
(17)
Proof. For simplicity, drop the conditional dependence of x. Notice that the general RLHF optimization objective IT attains a unique optimal solution
(y) plv) exp Bu (0
(P(y (
(18)
As it is a widely known result, we defer the proof to prior works, e.g.. Azar et al. 2023
Intuitively,
comparing the above solution with the target distribution p1 as rewritten 7 allows us to formulate
a similar objective to (11) which p1 optimizes. Formally, we begin with considering the optimization objective in (17)
argmax Eyynr log Byg-n (P(U > v')) - (x(
(19)
log Bygn mp [P(y v') log (ty)
argmaxx Eywx
(20)
(1))
T(V)
argmin E Byss
log p(y)
log Eayirna (Ply v3))
(21)
π T
argmin
log
(22)
(yy) [P > y'))
(v)
argmin
log
log Z
(23)
u1( (P y v')
v\) dy the normalization constant. Notice that the distribution in the denominator is precisely the marginal distribution P1 of our interest, which is indeed valid distribution. Hence, the last optimization objective can be reorganized as follows.
where Z (()) (P
argmin DKL (p1) log Z.
(24)
Since the Z constant, the KL-divergence objective is minimized if and only if the two distributions π and p1 are identical, by the Gibbs' inequality. It follows that p1 is the optimal solution of the optimization problemm (17). which completes the proof.
Before stating and proving the theorem, we note here that the L assumptions the pre-trained model mref and the probability model Pyy > s/) is generally valid in practical domains. For instance, assuming गref being trained on La loss as in many practical domains, Tref E T,2 generally holds. Also, the preference model usually provided in a finite support, or modeled as Bradley-Teere Terry model, which generally behaves well.
Next, we aim to prove Theorem 4.2
15

---

Theorem C.2 (Convergence of Iterative Method), Assume that μ Consider an iterative update of the marginal distribution P1 as follows.
L2, and P(y > 5) E L??
Pi (m) (1) x pi (g)
(n- (r')"y' ( d'y, Pi (0)
=
(25)
L.., we iteratively compute the marginal distribution of more preferred samples from the update marginal distribution. Then, the limiting marginal distribution (oo) converges to the uniform distribution U of points where the value Eyg-nn (IP( > v'') is the largest, i.e.
Pi (00) +U
argmmax Ey (P =1))
(26)
Proof. Let denote by T the transformation applied each step:
1 2,)( / p() > d'y' T(l(v) Z
(27)
where 2p = f ((y))((y > 3')dy' dy is the normalization constant. Then, the update rule (25) can written simply as pi - TTel-->) We aim to show that this iterative procedure converges to distribution that places uniform weight on the set of points where Eyr=p (P(y > y")) is maximized. Given a probability distribution p. Let us define the function fo as follows.
(v) Eng'rp [PV (P()>)) - ff p(y))P(() -
d')dy'.
(28)
Observe that (v) increases ply) proportionally to fy()). Consequently, the regions where fy( is higher will see an amplification in probability mass relative to regions where fp is lower. This amplification occurs iteratively, with higher fp(y) regions increasingly dominating the distribution.
Formally, we claim that the fixed-point iterator T is compact. Notice that the kernel Ky, y') p(y))((y y') is bounded, provided that (P > y/) is bounded by 1, and that p is probability density, which integrates to 1. By Schur's 's test and the properties of Hilbert-Schhie operators, i.e.,
// (K(y,' 5' < oo.
(29)
T can be shown to be a compact operator, from the square integrability assumptions.
Next, consider the behavior of T any sequence of probability densities ( ===-- By the
properties of compact operators in the space of continuous functions, any sequence {p(m) has convergent subsequence in the weak topology. Letp" be the limit of any convergent subsequence. The uniform boundedness of K (( 3y') and the compactness of T suggest that 7 Tp" - p", establishing that p² is fixed point of T. To determine the nature of p", observe that
f f (' (
y d'ydy' dy =
(30)
Since Ply > y²) is maximized uniformly over when Ey > y²)) is maximized, p" must concentrate its mass on the set where this expectation maximized. Therefore, p" converges to uniform distribution over the set
arg max Egg-z (P(y v))).
(31)
Formally, recall that from Theorem 4.1 the updated distribution p| is solution to the following optimization problem:
(n) Pi = argmax (TT) == Eyyur log E y
(1-1) (P(y > v1)) DEL 7 (m|()(--))
(32)
Hence, we note that if any point y not in this set were to have a positive probability under p", then T[p"] would not be equal to p". due the strict maximization condition, contradicting the fixed-point property. Thus, (x)) converges to the uniform distribution over the optimal set, as stated. We note here that because the space of probability densities is closed under the topology induced by the
function space norm, p" should also probability density, and ultimately the unique minima. O
16

---

D Experimental Details
In this section, we describe our experimental details.
Resource All experiments were conducted on a single Nvidia Titan RTX GPU and a single 19-10850K CPU core for each run. The time required varies by task, but it only involves collecting preference datasets and learning the flow, it completes within a few hours.
D.1
Conditional Image Generation
Task We employ the MNIST [LeCun et al. 1998 dataset to evaluate PFM on conditional image generation task.
Preference Dataset We utilize pre-trained DCGAN RRdford et al. 2015 generator as Tref and collect sample pairs from ree conditioned the digit labels contexts. To construct preference datasets, we assign preferences to sample pairs according to the softmax probabilities of the labels from LeNet [LeCun al. [998].
Baseline We consider the pre-trained DCGAN generator (Tref), a RLHF fine-tuned model and DPO fine-tuned model of Tee as baselines. All methods trained until convergence, and we report the normalized episodes returns with the standard deviation from 5 different random seeds.
D.2 Conditional Text Generation
Task To evaluate PFM on the NLP domain, we adopt controlled (positive) sentiment review generation task with the IMDB dataset MMas et al. 2011].
Preference dataset As done in Rafailov et al. [022bb]. to perform a controlled evaluation, we adopt the pre-trained sentiment classifie as the preference annotator. The preference dataset is constructed from randomly selected pairs of moview reviews yt,y" from the IMDB dataset, where the preference obtained from the classifier logit probability p(positively*) > >>((p p(positively"" We then train our PFM model on this preference dataset, to obtain the marginal distribution of the preferred (positive sentiment) review pi ()
Baseline We consider the GPT-2 SFT model on the IMDB dataset (Tref) and a RLHF fine-tuned model of rel using PPO (TPPO) as baselines. We apply PFM baseline models add-on module without fine-tuning, For the iterative PFM, we iteratively apply the initially obtained Icarned preference flow without collecting new data from the improved PFM model.
Embedding for variable-length input For our PFM framework to be applied to variable-length inputs, we employ T5-based autoencoder" to obtain fixed-sized di02--iimenio vector) embed- ding the input texts, allowing us to work within the fixed-size latent space. Once the fixed-size embeddings = and 2 obtained for each text sample y* and y learn the conditional flow using PFM from =- to = During inference, we apply PFM to the latent embedding of the given input y, and decode the improved latent embedding using the T5 decoder. We adopt the same U-Net architecture used in our MNIST experiment, where we reshape the 1024-dimensional vector into a wwo-dimesioonal (32, 32) image tensor. As shown in Table 6 PFM requires a much smaller number of parameters (around 1.2%) than RLHF or DPO which involve fine-tuning LLMs.
Table 6: Parameters required for training for each method. PFM only requires 1.2% parameters to be trained compared naive approaches (RLHF, DPO, etc.), and still achieves better performance in preference alignment.
GPT-2 (RLHF) U-Net (PFM)
124M
1.5M
Iterative PFM For our iterative PFM, we do not iteratively collect data and re-train the module. Instead, we simply iteratively apply the learned preference flow to the output samples. particular, apply the learned flow to the embedding iteratively with the same flow module.
http:/ / /yann, htt.:/uunn.cmm////////o/// -
ht
http://waagiggss..../eeeeeeeeeee bottleneck
bazge-wikipedis
17

---

Prompt for GPT-4 Evaluation Below we include the prompts used to generate win rates,
Which of the two movie reviews has positive sent ment??
Response A: <Response A>
Response B: <Response B>
D.3 Offline Reinforcement Learning
Task To assess the performance of PFM in reinforcement learning tasks, employ the D4RL Fu ct al. 2020] benchmark from https::/github.. wtt.//aaaaaaaPooraaaaaiiaaaa DARRL where the datasets and code are licensed under the Creative Commons Attribution 4.0 License (CC BY) and the Apache 2.0 License, respectively. We consider four different tasks (ant, hopper, hal fcheetah, wa ke 22)) from Gym-Mujoco domain with three different levels of offline dataset quality (random, medium um, expert) for each task.
Preference dataset We first pre-train Tref using behavior cloning (BC) for each offline dataset. We then collect segment pairs of the rollout trajectories from Teef. with each pair starting from the same state context. To construct preference datasets, utilize scripted teacher which prioritizes trajectories with higher rewards based on the ground truth rewards provided by the environment. This approach has been widely adopted in the PbRL settings [Kim et al. 2023 Lee al. 2021 Zhu et al.)
2024). The preference datasets consist of 1. 1,000 pairs of preferred and rejected segments and their context for each offline dataset, with the segment length 10.
Baseline The baseline methods for comparing the performance of PFM includes behavior cloning (BC), which we adopt
pretrained reference model Tref, and DPO fine-tuned model from the BC model. For DPO fine-tuned models, we search KL regularization coefficient 3 from 0.01 to 100 and adopt the best one. Additionally, we train separate behavior cloning model to the collected preferred samples y* ~ P1. aiming to replicate the marginal distribution of the "good" trajectories. All methods are trained until convergence, and we report the normalized episodes returns with the standard deviation from different random seeds.
E Broader Impacts
As add-on module, PFM be seamlessly integrated into various real-world Al applications, such as generative models and continuous control systems, without the need to modify the underlying application models. This integration enables the applications deliver personalized results that align with individual user preferences. However, since PFM utilizes preference data, it raises potential privacy concerns similar to those found in typical PbRL methods. These concerns can be mitigated by ensuring that access to user preferences is granted only with explicit user consent.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claims made in abstract and introduction reflect the paper's contributions and scope. Theoretical and empirical results in Section 4 and Section 5 demonstrate our contributions. Furthermore, address limitations in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer Section 7
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiaatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer: [Yes]
Justification:
Please refer to Section and Appendix C
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide descriptions of our method in Section 3.1 and in Appendix Furthermore, we provide experimental details in Section 5 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
20

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We attach the code and related information in the supplementary mate- rial. Furthermore, we release the revised code at
com/
preference-ELe mam-mmatciin
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (bttps://nipe) cc/
public/ public/quidss/ /ee//CodeSSuaa cwrmimsssoooooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https?
//nips httppscccppbbccccgocc.cec/cs giweeeiiieeeeee des Sedeeeemmm as onPol ooooyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide experimental settings and details in Section |5] Appendix D We also release our code in supplementary materials and at https: higggttthhuuhgggggggggggg com/
jacchaas//prefrrencee-- fl www-matching
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Please refer to Section 5 and Appendix D For RL tasks, we report mean episode returns with standard deviations for 5 different random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification: All authors read and followed the NeruIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This paper poses such risks since we do not release datasets or pre-trained models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cited the original papers of the assets. URLs and licenses are mentioned in Appendix D
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paapersw haperswithcoos com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: We provide instructions to use our codes in README.md file which is included in the supplementary materials and at our repository httpp::/ggttuu.... com/
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mooell as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "5nh1qf8nCn", "reviewer": "Reviewer_vT4A", "summary": "This paper introduces Preference Flow Matching (PFM), a novel framework for preference-based reinforcement learning that addresses key limitations of existing methods like Reinforcement Learning from Human Feedback (RLHF). PFM leverages flow matching techniques to directly learn from preference data. The core innovation of PFM lies in its formulation of preference alignment as a problem of learning a flow between marginal distributions of less preferred and more preferred samples, moving away from the implicit discriminative classification formulation of DPO into a discriminative/generative formulation. Methodologically, PFM employs a conditional flow matching objective to learn the vector field representing the preference flow. It also does not require the Bradley-Terry assumption. PFM is also compatible with 'reflow' like the original flow matching formulation. The authors derive the marginal distribution and prove it converges to a maximizing uniform. This all represents a clear, if straightforward, combination of (reward-free) preference learning and flow-matching providing a ready-made comparison with existing *-PO methods.", "questions": "- What stopped the authors (other than lack of time, infrastructure, compute) from trying for small-scale language modelling experiments, since this is the domain that originally stemmed the derivation of DPO and IPO ? In reference to 'Another notable limitation is its applicability to the natural language processing (NLP) domain. The PFM framework, as currently designed, cannot be directly applied to NLP tasks because the source and target texts may have different lengths', it would technically be possible to model distributions over fixed length sequences, along with padding and/or an EOS token. One great advantage of working in the text domain is the ability to display sample completions from finetuned models in a qualitative study. Also, testing with actual human preferences would actually be better.\n\n- Even if additional experiments are out of scope of this current review period, which other empirical designs can the authors think of that would better showcase the benefits of PFM compared to *-PO ?", "limitations": "Limitations are noted in the 'weaknesses' section below and mostly acknowledged in the paper.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper has multiple strengths. The combination of preference learning with flow matching is clear, if straightforward ; and its exposition is sound and easy to follow. The authors provide solid theoretical analysis, including proofs of optimality and convergence. This gives the method a strong mathematical grounding, which is often lacking in more heuristic approaches. In theory, PFM tackles important issues in existing methods, particularly the problem of reward overfitting and the challenges of aligning black-box models.", "weaknesses": "- Paper writing could be improved as several typos remain ('A careful reader might noticed' l.126, 'ordinary diffusional (sic) equation' l.89...). Some proofreading is in order !\n\n- Most importantly, the empirical contribution is not overly compelling. It is surprising that on MNIST the DPO results are that poor, both in absolute terms and relatively (worse than RLHF). IPO is not considered, DPO is probably taken offline rather than online (which generally tends to improve results), and most importantly, the beta parameter is not iterated upon/optimized (if it is, then this needs to be made clear, if possible in the main text, but it is not even in Appendix D.1). It is well known that the beta parameter is crucial for the performance of those methods and one advantage of very small scale experiments such as MNIST is that iterating is inexpensive. Tuning the baseline to the maximum would lend credence to these experiments; besides, it is done in the other suite of experiments.\n\n- Similarly, the offline RL experiments are small scale (Gym-MuJoCo. This time the beta parameter is optimized, and on all expert domains, the PFM performance is not massively different from DPO - in fact, the expert average for DPO fine-tuned is stronger than that of PFM (even though PFM provides variance reduction). While we know that DPO has tendencies to overfit and reward-hack, this is noted by the authors themselves who ask in section 5.3 'is learning a flow truly beneficial' ? Finding this detailed and balanced discussion in the paper was a nice display of intellectual fortitude and very welcome. Nonetheless, the empirical benefits of the proposed approach so far are quite ambiguous. The paper would truly benefit from showcasing one indisputable and conclusive empirical setting in favour of PFM.\n\n- Finally, the paper doesn't deeply explore the computational costs of PFM compared to other methods, which could be an important practical consideration. In particular, is reflow worth it ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we appreciate your thorough review and valuable feedback. Below are our responses to your comments and questions.\n\n- **W1: Weak empirical contribution.**\nWe have obtained promising results in a new domain, please refer to the above common response section for our results in the selected NLP task. We believe this new result is strong enough to prove the empirical usefulness of our framework.\n- **W2: Regarding experimental results for D4RL tasks.**\nAs you pointed out, PFM may not show significant improvement over DPO in D4RL benchmarks. However, it is important to note that we achieved similar performance increases from the reference policy without any fine-tuning of the original policy. Hence, the greatest advantage of our method is its ability to be attached to any black-box model for improved preference alignment. Combined with the new empirical results obtained in an NLP task, and experiments in the vision domain with MNIST examples, we believe it provides a comprehensive advantage for our framework in general preference alignment.\n- **W3: Regarding MNIST results.**\nThe RLHF and DPO results of MNIST experiments are obtained from the best working $\\beta$ found from the same search space as in the D4RL tasks. We thank you for pointing out this missing detail; we will add this to our paper.\n- **W4: Regarding computational cost.**\nFor the computational cost incurred during the training phase, PFM is significantly more efficient than existing frameworks, as it only requires training a small add-on module. For example, please refer to the above common response section to see the difference in required parameter sizes for training in even a simple NLP task. Due to the limited time, we were not able to directly compare the training cost of RLHF fine-tuning and PFM, since we did not fine-tune the reference language models on our own, but adopted an open-source fine-tuned model that is already publically available. During inference, there is negligible computational complexity due to the small size of the attached PFM module. We believe this additional computational cost during inference will be even more negligible as the size of the base reference model increases.\n\nFor your questions regarding the NLP domains and additional experiments, we believe that our new results provided in the common response will address your concerns and demonstrate the applicability of our proposed methods across various scenarios. Thank you once again for your valuable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CNB8uuMfWJ", "reviewer": "Reviewer_yRMJ", "summary": "The paper presents a new framework for preference-based reinforcement learning, relying on Flow Matching. This framework utilizes flow-based models with optimal transport that interpolate between less preferred data and more preferred data, eliminating the need to estimate a reward function implicitly or explicitly. The authors provide a rigorous framework supported both by formal proofs and clear experimental evidence of the method's performance and demonstrate the effectiveness of iterative PFM, which iteratively narrows down the distribution towards preferred outputs.", "questions": "- Consider an expansion of failure mode explanations for DPO from the paper https://arxiv.org/pdf/2404.10719, namely susceptibility to distribution shift, which can cause issues with generalizability in the regions without data support. They tie back to the author's critique of the failure modes of DPO, like overfitting.\n\n- line 210: \"Is PFM beneficial than methods optimizing for explicit/implicit reward model\" is this missing \"more beneficial\"? (same for Q3)\n\n- lines 126-129 highlight that the starting distribution $p_0$ is inaccessible during the inference step, and instead, they simply start from $y \\sim \\pi_{ref}$ as the starting point. This is an important implication for sampling/inference. Thus, a comment explaining why this works and whether it can be applied in any scenario under consideration is crucial. Consier is adding this to limitations if there are issues with applicability.\n\n- In the conclusion and limitations, the authors state limitations of applicability to NLP tasks: \" Future research should explore ways to adapt the PFM 314 framework for variable-length data, potentially through innovative alignment techniques or alternative 315 frameworks suited for text generation tasks.\" Diffusion Forcing: https://arxiv.org/abs/2407.01392 has been introduced recently to combine both autoregressive next token predictions with full-sequence diffusion. Consider this as a minor suggestion given the time constraints and timelines of the initial submission, the could be a useful source for future work.", "limitations": "The authors adequately addressed the limitations of their work throughout the paper and in the conclusions and limitations section.\n\nA minor suggestion:\n- The authors state the challenges of existing preference-based RL methods such as scalability, inefficiency, and the need for model modifications, especially with black-box, and later address them in the paper, except for scalability, which calls for stating more clear limitations or suggesting lines of future work to address this (extending the limitations in lines 309-310 would suffice).", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper notably excels along each of the dimensions: \n- originality: the paper presents a novel approach to preference alignment that hasn't been explored in the literature and shows a very promising potential for further exploration.\n- quality: the problem is well positioned, claims in the paper are clearly stated and the flow of argument circles back to support the claims throughout the paper. The authors do a good job of illustrating the nature of RLHF overfitting by starting with deterministic preferences, which are easy to illustrate and extend to non-deterministic cases. Then they prove that the proposed objective (12) for the marginal distribution $p1$ obtained from the preference model $\\mathbf{P}(y>y')$ is robust to overfitting even in the deterministic case.\n- clarity: the flow of argument is delivered with exceptional clarity. Notably, the authors provide both formal proofs of theorems and intuitive explanations, e.g. in lines 204-205 \" marginalization iteratively \"narrows\" down the distribution towards the outputs with higher preference\", which are then supported by the results in Table1, demonstrating performance improvements with lower variance across all baseline methods and supporting the claim in lines 36-37 \"By simply adding a preference flow matching module to black-box models, 37 PFM eliminates the need for fine-tuning the black-box model itself, providing a significant advantage.\"\n- significance: the contributions of the paper are significant yet the impact on more complex tasks is yet to be explored. In the current form, the method is not applicable to NLP tasks as LLMs, but flow matching per se is capable of working on multiple modalities and textual data. \n\n\n\n\nDemonstrate", "weaknesses": "The paper is very strong, no notable weaknesses observed, suggestions for improvements are provided in the next section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer yRMJ, we appreciate your invaluable comments and feedback. Below, we provide our response to your comments and questions.\n\n- **Q1. Shifted source distribution during inference.**\nAs mentioned in our paper, since the source distribution $p_{0}$ is inaccessible during inference, we instead start from a shifted source distribution $\\pi_{\\mathrm{ref}}$. This can indeed be a factor that may deteriorate the performance of our method. However, as discussed thoroughly in Appendix A and B of our paper, it is still theoretically guaranteed to work if the support of the distribution $p_{0}$ (of less preferred samples) is included in the support of the distribution $p_{1}$ (of more preferred samples). In other words, for any sample $y \\sim p_{1}$, as long as it has a non-zero probability of being labeled as less preferred (i.e. $p_{0}(y) > 0$), we can guarantee that for any samples $y \\sim \\pi_{\\mathrm{ref}}$, there is a non-zero probability of being included in the negatively labeled dataset. Hence, we can guarantee a non-zero probability of existing learned flow from that sample $y \\sim \\pi_{\\mathrm{ref}}$ to a more preferred sample. That being said, a failure regime of our method may include a scenario where the distribution of the ground-truth measure of the preference scores is sparse, that is, only the extremes (either very preferred or nearly non-preferred) are present in the dataset. We will include content in the limitation section.\n- **Q2. Future works for NLP domains.**\nThank you for pointing out a useful source (Diffusion Forcing) for future work. We also believe that these lines of work can be adapted to our framework and applied to NLP tasks. Aligned with this direction, we have shown a promising result of our method on an NLP task; please refer to the common response above.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5bjYFFdfpT", "reviewer": "Reviewer_XhGj", "summary": "This paper proposes an add-on method named Preference Flow Matching (PFM) to achieve preference alignment without learning a reward model. PFM can transfer $y$ generated from the original model to the preferred $y^+$ through a few flow iterations. Experiments demonstrate its superior performance compared with RLHF and DPO.", "questions": "- During inference on the D4RL tasks, how do you choose actions to interact with the environments? As the paper claims, PFM samples an action trajectory $\\tau$ from $π_{\\rm ref} (·|s_t)$, and then applies flow matching to obtain a better action sequence $\\tau_{\\rm new}$. How can you sample a trajectory $\\tau$ without training a dynamic model? Do you apply all actions in $\\tau_{\\rm new}$ to interact with the environment, or select some actions from $\\tau_{\\rm new}$ to take?", "limitations": "Limitations have been discussed in this paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-This paper is well-written. I really enjoy reading it.\n- While flow matching is not novel, this paper proposes to adapt it to the preference alignment area and yields many advantages (e.g., simplicity and efficiency).\n- Experiments on both image generation and offline RL demonstrate its comparable performance.", "weaknesses": "- During inference, PFM with prior as $π_{\\rm ref}$ suffers from a shifted source distribution, which can affect the performance.\n- Limited testing benchmarks. RLHF is getting a lot of attention these days, largely due to its success in the language model area. However, this paper does not include the language-based task because PFM is limited in its ability to generate variable text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer XhGj, we thank you for your valuable comments and review. We are glad to hear that you have enjoyed reading our paper. Below, we provide our response to your comments and questions.\n\n- **W1: Shifted source distribution during inference.**\nAs you have pointed out, the shifted source distribution might affect the performance of our method. However, as studied in depth in Appendix A and B of our paper, our method is still guaranteed to work under a mild assumption, and under general non-deterministic preferences. In particular, our method is theoretically guaranteed to work if the support of the distribution $p_{0}$ (of less preferred samples) is included in the support of the distribution $p_{1}$ (of more preferred samples). This means that as long as the preference is continuous, we can guarantee that for any samples $y \\sim \\pi_{\\mathrm{ref}}$, there is a non-zero probability of being included in the negatively labeled dataset. We have also provided empirical evidence for our choice of source distribution $\\pi_{\\mathrm{ref}}$ instead of $p_{0}$ in Appendix B of our paper. We will clarify this assumption in the main paper, as well as include further details in the limitation section of our paper.\n- **W2: Experiments for the NLP domain.**\nWe have obtained a promising result in one of the popular NLP tasks, one that is also adopted in the DPO paper. Please refer to the above common response section. We believe that the results above will serve as strong evidence and support that our method can be applied to general language-based tasks.\n- **Q1: Inference on D4RL tasks.**\nFor the inference on the D4RL tasks, we sample a reference trajectory $\\tau$ using a copy of the current environment, with the same initial state. In scenarios where using a copy of such an environment is available (e.g., simulations), this is no problem, except for increased computation costs. However, as you pointed out, in some real-world applications, we may need an additional dynamics model in order to compute a reference trajectory. We will include this as one of our limitations. We would like to note here that this limitation is only for the case of RL tasks, and not for the image and language domains.\n- **Q1-1: Inference on D4RL tasks.**\nOnce the reference trajectory $\\tau$ is obtained, we then apply PFM to obtain an improved trajectory $\\tau^{+}$. In our implementation, we choose only the first action from the improved trajectory during inference. Of course, one may choose all actions from the improved trajectory to reduce the computational cost incurred during flow-matching, at the expense of performance loss. Hence, there is a trade-off between the computation (for flow-matching) and the performance gain, and one may find an appropriate intermediate as a design choice, suitable for their task.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8ub6pooy95", "reviewer": "Reviewer_2tji", "summary": "This paper presents Preference Flow Matching (PFM), which learns an ODE to transform less-preferred data into more-preferred ones. Due to PFM not explicitly or implicitly relying on reward functions, it avoids reward overfitting. Empirical evidence demonstrates that PFM outperforms DPO in image conditional generation and Offline RL tasks.", "questions": "Please see Weaknesses.", "limitations": "Please see Weaknesses.\n\nThe main areas for improvement in the paper are concentrated in the experimental section, as the current experimental results lack persuasiveness.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written, with illustrative figures and toy examples that are intuitive and easy to understand.\n- The approach of using a flow model to transform less-preferred data into more preferred ones is straightforward and effective.", "weaknesses": "- The novelty of the paper appears somewhat limited, as PFM seems to use Conditional Flow Matching (CFM) to transform between less and more preferred data distributions. However, the paper does not thoroughly discuss the benefits of introducing CFM. Similar transformations can be achieved with Rectified Flow or Diffusion models, and work using Diffusion models (such as FTB) already exists.\n- The experimental validation in the paper is not sufficiently comprehensive, leading to less convincing results. For instance, in the context of Offline RL tasks, the following issues may exist:\n    - PbRL is commonly used to address tasks with difficult-to-define or sparse rewards. In D4RL, tasks like antmaze are more commonly used for better algorithm performance comparison.\n    - In MuJoCo tasks, PFM does not exhibit a significant improvement over DPO. Additionally, the standard deviation of the results appears to be quite large, making comparisons based solely on average means less reliable. Could this be due to limited random seeds? Typically, MuJoCo tasks have low standard deviations for having dense rewards.\n    - While the authors summarize many RLHF methods in the related works section, only DPO and vanilla RLHF are compared across the experiments, indicating a potential lack of baseline algorithms. For example, FTB uses a diffusion model to do similar tasks as PFM. Given the similarity between these two algorithms, why are they not compared in the study?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 2tji, thank you for your careful review and valuable comments. Our responses are presented below.\n\n- **W1: Main novelty of our method.**\nThe main novelty of our method comes from its capability of being added on top of any existing black-box models, without the need of fine-tuning this reference policy class. FTB for instance, requires an additional policy extraction phase, as FTB only focuses on the data augmentation for the behavior cloning policy. This means that FTB also requires a fine-tuning of the given reference policy in order to obtain the final policy.\n- **W1-1: Choice of conditional flow matching (CFM)**\nRegarding the choice of conditional flow matching (CFM), our method can indeed be extended to frameworks that use rectified flow and diffusion models. As mentioned in the CFM paper (Tong, Alexander, et al. \"Improving and generalizing flow-based generative models with minibatch optimal transport.\" *Transactions on Machine Learning Research*), rectified flow and conditional flow matching can be unified under a single flow-matching framework. The key difference of the rectified flow and the CFM only comes from the design choice of the flow transport model; CFM chooses a Gaussian probability path, whereas the rectified flow chooses a Direc probability path for density transportation. We would like to emphasize that the novelty of our approach comes from the probability transport module to be added on top of the existing black-box reference policy. To the best of our knowledge, we are the first to adopt such novel fine-tuning-free methods to improve alignment with the preference. This approach will be more favorable in cases where the fine-tuning of the original reference policy is unavailable, or computationally expensive.\n- **W2: Weak empirical results.**\nAlthough PFM does not gain significant improvement over DPO in D4RL benchmarks, it is important to note that we have gained a similar performance increase from the reference policy, without any fine-tuning of the original policy. That being said, the greatest advantage of our method is its ability to be attached to any black-box model for improved preference alignment. This is verified across variable domains: vision, RL, and even language domains. Please refer to the common response above for our newly added experimental results for the NLP task. We believe these results will serve as strong enough evidence that our method can be used to improve preference alignment.\n- **W2-1: Regarding D4RL results**\nAs you pointed out, we observe high variance across all the results in the MuJoCo experiments. However, this is not due to limited random seeds or insufficient experiments. In offline RL tasks, compared to online RL settings, the variance may be large due to its limited access to the environment interaction. Especially in PbRL settings, the variance may be larger due to the lack of available preference-labeled data. This large variance can be observed in some other PbRL papers too; for instance, see Preference Transformer (Kim, Changyeon, et al. \"Preference transformer: Modeling human preferences using transformers for rl.\") for a similar scale of variance.\n- **W2-2: comparing our method to existing frameworks.**\nAs mentioned previously, we are the first to apply a fine-tuning-free method to improve preference alignment. Hence, our experiments mainly focused on comparing our new framework to the existing standard frameworks like RLHF and DPO. In the case of FTB, this method also requires an additional policy extraction phase that fine-tunes the reference policy. Furthermore, FTB suffers from heavy computational costs both for the policy extraction and the data augmentation using diffusion models. Therefore, we believe including FTB as a baseline is unfair, as it trains on a larger augmented dataset. Of course, our method can be applied concurrently with the FTB; FTB can be used to augment more quality datasets, where PFM can then be applied on top of the augmented preference dataset for additional performance gain.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZnWSgKuzSN", "reviewer": "Reviewer_XK5w", "summary": "The paper introduces a novel framework called Preference Flow Matching (PFM) for preference-based reinforcement learning (PbRL). The PFM approach aims to integrate human preferences into pre-trained models without the need for extensive fine-tuning, which is a common requirement in existing PbRL methods. The PFM framework utilizes flow matching techniques to learn directly from preference data, reducing the dependency on pre-trained model fine-tuning. It transforms less preferred data into more preferred outcomes by leveraging flow-based models, aligning model outputs with human preferences without explicit reward function estimation. The paper provides theoretical insights and empirical results that demonstrate the effectiveness of PFM in aligning pre-trained models with preferences and offers a new direction in PbRL. PFM is proposed as a solution to challenges such as scalability, inefficiency, and the need for model modifications, especially with black-box APIs like GPT-4.", "questions": "1. I believe the assumptions in the paper are overly idealistic, as the method appears impractical for application to black-box models like GPT-4, and may even struggle with open-source models. If the authors could provide a specific plan on how to apply the method to open-source models (such as llama) and black-box models (such as GPT-4), including details of experiments and experimental results, I would consider raising the score. Otherwise, we suspect the research motivation in the paper has been exaggerated.\n\n2. How to apply the simple assumptions from lines 116-117 to LLM?", "limitations": "The authors have made an effort to address the limitations and potential negative societal impacts of their work.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "Innovative Approach: PFM offers a fresh perspective on integrating human preferences into AI systems by using flow matching, which is a relatively under-explored method in preference alignment.\n\nTheoretical Foundation: The paper provides a solid theoretical basis for PFM, proving its alignment with standard PbRL objectives and robustness against overfitting in reward models.\n\nEmpirical Validation: Experimental results support the practical effectiveness of PFM, showing its ability to robustly align with preferences and achieve comparable performance to traditional RLHF methods and DPO.\n\nScalability and Efficiency: By eliminating the need for fine-tuning large pre-trained models, PFM addresses significant challenges related to scalability and computational efficiency.\n\nApplicability to Black-Box Models: PFM's ability to work as an add-on module for black-box models, like GPT-4, extends its applicability to scenarios where model access is limited.", "weaknesses": "Exaggerated Motivation:  I believe the research motivation stated by the author in the abstract and introduction may be exaggerated. For example, the author claims that the method can be applied to black-box models like GPT-4, but there is no experimental validation. In fact, even for general language models, the author provides no experimental evidence to show that the proposed method is superior to DPO.\n\nLimited Domain Testing: The paper's experiments may not fully explore the potential of PFM across diverse domains, particularly in more complex tasks like general contextual generation. \n\nAssumption is too strong : The assumption is not general for RLHF scene. For example,  conditional flow is $y^+ - y^-$ (line 116), and the  probability path is subject to gaussian distribution,  which is not pratical for general AI model, such as GPT-4.\n\nPotential for Overfitting: Although PFM is designed to be robust against overfitting, the method might still be susceptible to overfitting in certain scenarios, particularly if the preference data is limited or biased.\n\nApplicability to NLP Tasks: The current design of PFM may not be directly applicable to natural language processing tasks due to challenges with variable-length data, which could be a significant limitation given the prevalence of NLP applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer XK5w, thank you for your careful review and constructive feedback. Our responses to each of your comments are presented below.\n\n- **W1: Exaggerated motivation, limited domain testing and applicability to NLP tasks.**\nThank you for pointing out this concern. We also acknowledge the importance of further evaluation of PFM, especially on NLP domain. To address this issue, we conducted additional experiments on the NLP task. Please refer to our common response.\n    - We applied PFM to the open-source LLM (GPT-2) for text generation task.\n    - We designed a pipeline to use PFM for variable-length text data.\n    - We compared PFM with RLHF fine-tuned model.\n- **W2: Too strong assumption on conditional flow.**\nWe follow the unified framework for conditional flow matching (Tong, Alexander, et al. \"Improving and generalizing flow-based generative models with minibatch optimal transport.\" *Transactions on Machine Learning Research*). We highlight that the Gaussian probability path is not a required assumption, rather it is a design choice. Following the original work of the CFM, we define the probability path between $y^+$ and $y^-$ as gaussian distribution for ease of usage and theoretical guarantee. Hence this does not hinder the applicability of PFM. All assumptions required for our framework to be theoretically guaranteed are thoroughly discussed in the Appendix A and C of our paper. We note here that these assumptions are generally valid across various ML domains.\n- **W3: Potential for Overfitting.**\nIf the preference data is limited or biased, preference-based methods may generally suffer from overfitting issue. This limitation is not restricted to our method; we believe that overfitting is generally inevitable in case of limited data for any existing ML frameworks.\n- **Q1: Overly idealistic motivation.**\nPlease refer to our common response. We applied PFM to the open-source LLM (GPT-2) for text generation task, included details of experiments, and reported the results compared with SFT and RLHF models.\n- **Q2: How to apply the simple assumptions from lines 116-117 to LLM?**\nIn our additional experiments included in our global response, we embedded text data to fixed-size latent vectors and learned the vector field for the probability path in the latent space.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# (Common Response) New Domain: Added NLP Task

We applied our method (PFM) to the new NLP domain. We adopt a controlled (positive) sentiment review generation task, which is one of the main tasks previously tackled by the DPO paper. As done in the DPO paper, to perform a controlled evaluation, we adopt a pre-trained sentiment classifier as the preference annotator from huggingface library. The preference dataset is constructed from randomly selected pairs of moview reviews $y^{+}, y^{-}$ from the well-known IMDB dataset, where the preference is obtained from the classifier logit probability $p(\mathrm{positive} | y^{+}) > p(\mathrm{positive} | y^{-})$. We then train our PFM model on this preference dataset, to obtain the marginal distribution of the preferred (positive sentiment) review $p_{1}(y^{+})$. 

For our PFM framework to be applied to variable-length inputs, we employ a T5-based autoencoder from a huggingface library to obtain fixed-sized (1024-dimensional vector) embedding of the input texts, allowing us to work within the fixed-size latent space. Once the embeddings $z^{+}$ and $z^{-}$ are obtained for each text sample $y^{+}$ and $y^{-}$, we learn the conditional flow using PFM from $z^{-}$ to $z^{+}$. During inference, we apply PFM to the latent embedding $z$ of the given input text $y$, and decode the improved latent embedding using the T5 decoder. We adopt the same U-Net architecture used in our MNIST experiment, where we reshape the 1024-dimensional vector into a two-dimensional (32, 32) image tensor.

Note that the traditional approach, including standard RLHF frameworks and DPO, involves fine-tuning the LLMs on this preference dataset. Due to the LLMs' large number of parameters, fine-tuning these models is generally a computationally expensive task. However, we emphasize here that our PFM method requires a relatively smaller network than the original reference policy (LLM). We summarize below the number of parameters that require training for each framework in tackling this task. As we will discuss later on, PFM requires training a much smaller number of parameters (around 1.2%) while still achieving better performance. Furthermore, PFM does not require fine-tuning the original reference model, (and keeps the original model’s representation) as PFM can be attached to any existing black-box model as an add-on module.

| GPT-2 (RLHF/DPO fine-tuning) | U-Net (PFM) |
| --- | --- |
| 124M | 1.5M |

For our pre-trained reference policy for inference, we adopt the GPT-2 SFT model on the IMDB dataset from the huggingface library. We also compare our method with the RLHF fine-tuned model from the same huggingface source. Below, we report the average preference score (from the classifier annotator) of 100 randomly generated test samples (reviews) for each method. Please refer to the attached PDF files for detailed visualization of the score distributions. As shown in the table below, PFM is able to improve the preference score of any baseline model to which it is attached. Notably, iterative PFM with only five iterations achieves the best performance over all the baselines. We also note here that the PFM is trained with the original dataset, not by the dataset generated from the RLHF fine-tuned policy. In other words, PFM can be trained on the base dataset generated from the SFT model, and still be attached to arbitrary fine-tuned policies to further improve its performance. Due to the limited time constraints, we were not able to obtain results for DPO fine-tuned policy, as we could not find a DPO fine-tuned policy that is publically available. However, we believe that our results will also be comparable with DPO.

| Reference | PFM | Iterative PFM (5 iter) |
| --- | --- | --- |
| -0.3607 | 0.6399 | **2.7469** |
| **RLHF fine-tuned** | **RLHF + PFM** | **RLHF + Iterative PFM (5 iter)** |
| 2.0156 | 2.178 | **2.7894** |

We also compute the win rate with GPT-4 evaluation. Please refer to the attached PDF for the prompts we used to generate win rate. Both PFM and RLHF (PPO) fine-tuned policy excels the reference (SFT) policy with win rates 100%. We also compare the win rates of our various PFM models with the RLHF fine-tuned policy. Notably, we observe that the iterative PFM with 5 iterations on SFT model outperforms the RLHF fine-tuned policy. If PFM is added on top of the RLHF fine-tuned policy, we observe near 100% win rates for both RLHF + PFM and RLHF + Iterative PFM. See the below table for the summarized results. Note that this win rate is also correlated with the preference scores provided by the pre-trained classifier.

|  | SFT + PFM | SFT + Iterative PFM (5 iter) | RLHF + PFM | RLHF + Iterative PFM (5 iter) |
| --- | --- | --- | --- | --- |
| RLHF (PPO) | 2% | 85% | 99% | **100%** |

Interestingly, we observe that the distribution of the scores tends to shift more toward that of the preferred samples with an increasing number of PFM iterations. (Please refer to the histogram provided in the attached PDF.) This result aligns with our theoretical insights: the PFM framework learns to shift the source distribution (i.e., the sampled distribution of the reference policy) toward the marginal distribution of the more preferred samples. 

If time permits, we plan to further experiment on additional NLP tasks before the camera-ready deadline. Currently, being in the initial stages, we have only demonstrated the add-on capability of PFM with GPT-2. As future work, we aim to apply our method to larger models like GPT-4 and tackle more complex tasks. Recent studies have been actively exploring language generation using diffusion models in continuous latent spaces, such as latent diffusion for language generation. Additionally, as reviewer yRMJ suggested, there are promising studies that combine diffusion and autoregressive generation. By integrating our method with these approaches, we anticipate broader applicability and increased usage of PFM in the language domain.

Meta-review: This papers proposes Preference Flow Matching (PFM), a new approach for preference-based RL that doesn't require an explicit (or implicit) reward model, leverating optimal transport to shift the original sampling distribution towards that of preferred samples.

Reviewers' ratings exhibited a high variance (3 to 9), with a common main concern regarding the lack of application to NLP tasks (which are the main reason behind the current high interest in preference-based learning). However, the authors provided new experimental results on an NLP task that at least partially alleviate this concern.

One reviewer (2tji / rating 4) still remained not entirely convinced after extensive discussions with the authors, in particular regarding the relevance of the full set of experimental results. I do agree with the reviewer that there is room for improvement in this area, however, I also want to show appreciation for the originality of this work that offers a fresh approach on preference-based learning, which I believe may serve as source of inspiration to future developments in similar directions. Considering that all reviewers were appreciative of this novel contribution, which is clearly presented and motivated, I consider this paper worth accepting at NeurIPS.

(I also note that the lowest-scoring reviewer -- with a 3 -- did not acknolwedge the authors' response, in spite of my reminder to do so)
Decision: Accept (poster)