Preference Alignment with Flow Matching

Authors: minu  kim
yongsik  lee
sehyeok  kang
jihwan  oh
song  chong
se-young  yun
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 21:13:52

Abstract: We present Preference Flow Matching (PFM), a new framework for preference alignment that streamlines the integration of preferences into an arbitrary class of pre-trained models. Existing alignment methods require fine-tuning pre-trained models, which presents challenges such as scalability, inefficiency, and the need for model modifications, especially with black-box APIs like GPT-4. In contrast, PFM utilizes flow matching techniques to directly learn from preference data, thereby reducing the dependency on extensive fine-tuning of pre-trained models. By leveraging flow-based models, PFM transforms less preferred data into preferred outcomes, and effectively aligns model outputs with human preferences without relying on explicit or implicit reward function estimation, thus avoiding common issues like overfitting in reward models. We provide theoretical insights that support our method’s alignment with standard preference alignment objectives. Experimental results indicate the practical effectiveness of our method, offering a new direction in aligning a pre-trained model to preference. Our code is available at https://github.com/jadehaus/preference-flow-matching.

Main paper: 
 introduction: 
 Preference-based reinforcement learning (PbRL) has emerged as a groundbreaking approach with significant contributions to performance improvement [Akrour et al., 2011 , Wilson et al., 2012] , particularly in the realm of artificial intelligence where understanding and incorporating human preferences are crucial. Unlike traditional reinforcement learning, which struggles due to the absence of explicit reward functions or the infeasibility of defining comprehensive environmental rewards, PbRL leverages a variety of feedback forms from humans to guide the learning process. This class of PbRL method is often referred to as reinforcement learning from human feedback (RLHF) [Ziegler et al., 2019 , Levine et al., 2018 , Ouyang et al., 2022] .Despite their effectiveness, these methods necessitate fine-tuning pre-trained models to align with user preferences, introducing several challenges such as scalability, accessibility, inefficiency, and the need for model modifications. For instance, with black-box APIs like GPT-4 [OpenAI et al., 2024] , customization based on user preferences is constrained due to restricted access to the underlying model. Moreover, even if fine-tuning were feasible, the large model size results in inefficient training and high resource consumption. Aligning black-box models with user preferences remains an under-explored area in research, despite its critical importance and growing demand.In this line of research, we propose Preference Flow Matching (PFM), which redefines the integration of human preferences by directly learning a preference flow from the less preferred data to the more preferred ones. This direct modeling of preference flows allows our system to better characterize Figure 1 : Illustration of our PFM framework. In the typical RLHF scenarios (left), we first sample preference data from the supervised fine-tuned (SFT) reference model. A reward model is learned from the collected dataset, either implicitly (as in DPO) or explicitly. The reward model is then used to fine-tune the reference policy to obtain the final model. Our method directly learns the preference flow from the collected preference data, where the flow is represented as a vector field v θ (middle). For inference, we again sample a point from the reference policy, and improve the quality of alignment by using the trained flow matching model, without the need of fine-tuning the existing reference model (right). and replicate the marginal distribution of the favored outcomes. We adopt a novel flow matching framework [Lipman et al., 2022] , which is a simple, intuitive, yet relatively under-explored method for preference alignment. By simply adding a preference flow matching module to black-box models, PFM eliminates the need for fine-tuning the black-box model itself, providing a significant advantage.Additionally, our method offers a highly robust approach for preference alignment, by circumventing the need for explicit or implicit reward function estimation. In typical RLHF scenarios, a model is initially trained to approximate a reward function based on human preferences. This reward model is then used to guide the policy learning process, aiming to align agent behaviors more closely with human preferences. However, this approach can introduce complexities and potential biases in translating human preferences into numerical rewards. In particular, learned reward models can often overfit the ground truth preference model, especially in the finite data regime [Azar et al., 2023] . Recent advancements such as Direct Preference Optimization (DPO) [Rafailov et al., 2024b] address the complexities of RLHF by eliminating the need for reward learning. However, these methods still inherently optimize for the reward model, and hence they are also susceptible to reward overfitting. In contrast, PFM directly learns preference flow, thereby removing the need for any reward model assumptions and resolving the challenges associated with reward model learning.We prove both theoretically and empirically that our method is able to learn an object that is similar to the standard RLHF objectives, while being robust to the preference overfitting observed in traditional RLHF pipelines. We also demonstrate how we can further improve the quality of alignment via iterative flow matching, with theoretical guarantees. Experimentally, we find that while typical RLHF methods and DPO suffer from preference overfitting, our method can robustly align with preference and still achieve comparable performances.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Our claims made in abstract and introduction reflect the paper's contributions and scope. Theoretical and empirical results in Section 4 and Section 5 demonstrate our contributions. Furthermore, we address limitations in Section 7.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
reinforcement learning from human feedback (rlhf): 
 Reinforcement learning from human feedback generally begins with obtaining a pre-trained reference policy π ref that can generate samples y ∼ π ref (•|x) given a context x. For example, a context x could be a text prompt given by a user, and the sample y could represent an appropriate text response generated by the reference policy π ref . We then collect a dataset of N preference pairs D = {(x i , y + i , y - i )} N i=1 , where each x i denotes the context, and each y + i , y - i ∼ π ref (•|x i ) are generated responses to x i and marked as good or bad samples, respectively. Here, we assume that the preference y + i > y - i is generated from a ground-truth reward r * : X × Y → R ≥0 , where X and Y are context space and response space, respectively. The goal of general RLHF is to recover an optimal policy π * such that (•|x) .π * = argmax π E x E y∼π(•|x) r * (x, y) -βD KL π(•|x) π ref(1)RLHF pipelines generally require reward learning. One of the most popular choices of the reward model is the Bradley-Terry model [Bradley and Terry, 1952] , which assumes that the preference y + > y -is generated from the probability P(y + > y -|x) = σ(r * (x, y + ) -r * (x, y -)), where σ is the logistic function. Under this model, the general RLHF framework learns the reward model r ϕ ≈ r * by minimizing the negative log-likelihood:EQUATIONOnce the reward model r ϕ is trained, we then use it to optimize for (1) to obtain π θ ≈ π * using standard reinforcement learning algorithms.There is also a class of reward-free methods that eliminates the need of reward learning phase [Rafailov et al., 2024b , Azar et al., 2023] . Direct Policy Optimization (DPO) [Rafailov et al., 2024b ] is a representative reward-free method that optimizes for (1) directly without learning a reward model. Although being a reward-free method, DPO implicitly optimizes for the reward function as in (2), by replacing rθ (x, y) = β log(π θ (y|x)/π ref (y|x)) as the implicit reward estimate.Flow matching is a class of generative model, where given a prior distribution p 0 , we aim to model a target distribution p 1 from p 0 . A key difference of the flow matching to the other generative models is that the prior p 0 can be an arbitrary distribution, (diffusion, for example, starts from a Gaussian prior p 0 ) and that the flow matching algorithm learns to modify the prior distribution p 0 to the target distribution p 1 with a neural network.Throughout, we consider a pair of data distributions over R d with densities y -∼ p 0 and y + ∼ p 1 , possibly unknown (but able to sample). The flow matching considers the task of fitting a mapping f : R d → R d that transforms p 0 to p 1 , that is, if y -∼ p 0 , then f (y -) ∼ p 1 . Inspired as in the motivation for the diffusion models, one can define a smooth time-varying vector field u : [0, 1] × R d → R d that defines an ordinary differential equation (ODE),dy = u(t, y)dt (3)where we use the notation u(t, y) interchanably with u t (y). Denote the solution of the above ODE by ϕ(t, y) (or ϕ t (y)) with initial condition ϕ 0 (y) = y. In other words, ϕ t (y) is the point y transported along the vector field u from time 0 to t. In order to obtain samples from the target distribution p 1 , we simply compute ϕ 1 (y) where y ∼ p 0 . The integration map ϕ t induces a pushforward measure p t ≜ [ϕ t ] ♯ (p 0 ), which is the density of points y ∼ p 0 transported via u from 0 to t.To train the vector field v θ with a neural network that mimics the vector field u of our interest, we can solve for the conditional flow matching objective, as proposed by Lipman et al. [2022] :L CFM (θ) ≜ E t∼[0,1],z∼q(•),y∼pt(•|z) ∥v θ (t, y) -u t (y|z)∥ 2 (4)where q(z) = π(y -, y + ) is some coupled distribution of samples y -, y + and u t (y|z) = y + -y -is a straight path from a source sample to a target sample. The conditional distribution q(z) can be chosen to be an independent coupling of source and target distribution q(z) = p 0 (y -)p 1 (y + ) [Lipman et al., 2022] , or the 2-Wasserstein optimal transport plan as proposed by Tong et al. [2023] .Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Please refer to Section 7.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preference flow matching: 
 In this section, we describe how we can use flow matching to learn (human) preferences. In the first subsection, we illustrate our flow matching framework for learning preferences, and compare it with typical RLHF pipelines. Then in Section 3.2, we demonstrate our method in a simple 2-dimensional toy experiment. Finally in Section 3.3, we provide an extension of our framework that can iteratively improve the performance, with theoretical guarantees.Instead of trying to optimize for the unknown reward r * or the preference probability model P(y + > y -|x), we simply learn a flow from the marginal distribution of less preferred data p 0 (y -|x) to the marginal distribution of more preferred data p 1 (y + |x) by leveraging what is explicitly characterized in the collected preference data:EQUATIONIn other words, we view that our collected data D is in fact generated from each of the marginal distributions y -∼ p 0 (•|x) and y + ∼ p 1 (•|x) obtained from P(y + > y -|x), respectively. Hence, following the conventions in the literature, [Tong et al., 2023] we define the flow matching objective for preference dataset D as follows:L(θ) = E t∼[0,1],z∼D,y∼pt(•|z) ∥v θ (t, y|x) -u t (y|z)∥ 2 (8)where we define the condition z = (x, y + , y -), conditional flow u t (y|z) = y + -y -, and the probability path p t (y|z) = N (y|ty + + (1 -t)y -, σ 2 ). Once we obtain the vector field v θ , we can improve upon the generated negative samples y -∼ p 0 (•|x) by solving (3) using an off-the-shelf numerical ODE solver [Runge, 1895 , Kutta, 1901] to obtain samples f (y -) ∼ p 1 . Specifically, we start from a sample y -with t = 0, and "flow" along the ODE trajectory using v θ until t = 1, to arrive at the target y + . Detailed algorithm can be found in Algorithm 1. Notably, generating improved samples can be done without the need of fine-tuning the existing model, since we learn a separate vector field that transports negative samples from p 0 to p 1 . Furthermore, we did not require any assumption for the probability model P(y + > y -|x), so our method can extend to general scenarios that do not adopt the Bradley-Terry model. Our method is outlined in Figure 1 .A careful reader might notice that for inference, we require negative samples y -from the marginal distribution p 0 , to obtain aligned samples y + . However, this p 0 is inaccessible during inference step, as we must first acquire preference label y + > y -for samples generated from π ref . Instead, we simply start from y ∼ π ref as the starting point, and apply flow matching to obtain f (y) ≈ y + ∼ p 1 . We emphasize that PFM can still robustly generate positive samples, if we assume non-deterministic preferences i.e., supp(p 1 ) ⊇ supp(p 0 ). We also empirically find that using π ref instead of p 0 as the source distribution can produce comparable results in practical scenarios. Further details can be found in Appendix B.Here, we demonstrate how our method learns to improve generated samples to better align with the preference, in a simple 2-dimensional toy experiment. We consider a ground truth reward function generated from an 8-Gaussians density as illustrated in Figure 2a . We then pre-train a Gaussian mixture model to obtain samples as in Figure 2c . The pairwise preference labels are then generated using the ground truth 8-Gaussians reward function, as done in many existing preference-based reinforcement learning (PbRL) settings [Christiano et al., 2017 , Ibarz et al., 2018 , Shin et al., 2023] .Once preference data are collected, we first learn a reward model r ϕ via (2). As can be seen in Figure 2b , the learned reward model overfits in the unseen region, which causes the RLHF method to diverge (Figure 2e ). DPO also fails to learn the correct preference, as can be seen in Figure 2f . We note here that DPO is also subjective to the reward overfitting since DPO also implicitly learns to optimize for the reward using the Bradley-Terry model (2) [Xu et al., 2024 , Azar et al., 2023] .However, PFM is free of such reward overfitting issues, as we do not optimize for the reward function using the Bradley-Terry model. Unlike other RLHF methods, our model robustly learns to align with the preference from the provided dataset (Figure 2g ). Notably, our method does not try to overfit beyond the unseen region, since the learned target distribution from the flow matching model tries to mimic the distribution p 1 (y + ) of collected preferred samples. (Compare Figure 2d and Figure 2g .) Figure 2 : Comparison of RLHF, DPO, and PFM on a 2-dimensional toy experiment. We generate preference labels from a ground truth reward in (a) and a pre-trained Gaussian reference policy (c). Both the RLHF (e) and DPO (f) methods struggle to align with the preferences, due to the overfitted reward model (b), even with the presence of KL regularizer (β = 1). PFM is able to mimic the distribution of the positively-labeled samples (d), and therefore achieves the highest performance (g). Repeating PFM iteratively to the marginal samples can further improve the alignment with the preference (h).As done in iterative variants of DPO [Xiong et al., 2023 , Yuan et al., 2024] , we can also further improve the quality of alignment with iterative flow matching. Specifically, upon obtaining a marginal distribution p 1 by applying flow matching, we again collect a new preference data y -, y + from the obtained marginal distribution p 1 in (6). We repeat this process iteratively, by replacing the source distribution (which is π ref in the first step) with the marginal distribution p 1 obtained in the latest iteration. This iterative process can be summarized as follows.p (n) 0 (y -|x) ∝ p (n-1) 1 (y -|x) p(n-1) 1(y|x)P(y > y -|x)dy (9)p (n) 1 (y + |x) ∝ p (n-1) 1 (y + |x) p(n-1) 1(y|x)P(y + > y|x)dy, pEQUATIONwhere we denote p (n) 0 and p (n)1 to be the source and target distribution of the flow matching model at the n-th iteration, respectively. By repeatedly marginalizing the distribution with respect to the preference P(y + > y -|x), we can effectively "narrow" the sampling distribution towards the outputs with higher preference probability. See Figure 2h for the results of the iterative method in our toy experiment. Note that even during this iterative approach, we leave the parameters of the pre-trained model π ref untouched, and only require sampling from this model throughout the whole process. Later in Section 4, we formally prove that the iterative method allows us to obtain a distribution class that maximizes the ground truth expected preference, and hence yields an optimal policy π * in (1) with β = 0. See Theorem 4.2.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We attach the code and related information in the supplementary material. Furthermore, we release the revised code at https://github.com/jadehaus/ preference-flow-matching.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
theoretical analysis of preference flow: 
 In this section, we theoretically analyze why PFM framework can effectively learn to align with the preference. Interestingly, learning to generate samples from the marginal distribution p 1 in (6) optimizes an objective that is similar to the goal of general RLHF in (1). Following the formulation provided by Azar et al. [2023] , one can observe that the objective (1) is equivalent to the below form:EQUATIONwhere σ -1 (ξ) = log(ξ/(1 -ξ)) is the logit function, and we drop the conditional dependence of x for simplicity. Note DPO also optimizes for the same objective as in (11).Let us take a step back, and characterize the failure modes of the RLHF and DPO frameworks, by figuring out when these methods overfit the reward. Consider a case where the preferences are deterministic, i.e., P(y > y ′ ) = 1, so that y is always preferred to y ′ . If we plug it into (11), we see that σ -1 (P(y > y ′ )) → +∞. Therefore, the solution π * of (11) ends up overfitting for the preference likelihood, resulting in a weak or even null KL regularization, regardless of the size of β.Although in the case where the preference is not deterministic, this phenomenon can still be pronounced in the finite data regime [Azar et al., 2023] . Even if the true preference is strictly less than 1, we may have access to only a few data samples to estimate P(y > y ′ ) ≈ 1. This means that overfitting can be a critical issue in general, especially if the action space Y or the context space X is extremely large, as in the case of aligning large language models to human preferences.In contrast, the PFM framework learns to generate a marginal distribution p 1 . One can show that this marginal is a solution for the optimization problem that is similar to the objective (11). Theorem 4.1 (Characterization of the Marginal). Let p 1 denote the marginal distribution of the positively-labeled samples y + . Then the marginal distribution p 1 obtained from the preference model P(y > y ′ |x) is an optimizer of the optimization problemEQUATION)We defer the proof to the Appendix C. Similar to the RLHF and DPO objective (11), the solution p 1 of ( 12) drives the original distribution π ref towards the points where the preference probability P(y > y ′ ) is increasing. However, unlike the RLHF or DPO objectives, the objective ( 12) is bounded even in the deterministic case P(y > y ′ ) = 1, making it robust to reward overfitting.Interestingly, maximizing the objective ( 12) is equivalent to minimizing the KL distance between the policy π and the normalized preference score with a cross-entropy constraint:EQUATIONHence, the objective pushes the policy π to match the preference P, while encouraging π to align with the high-probability samples in π ref .Since the constraint restricts the policy to the high-probability regions of π ref where the preference labels are collected from, our method is less prone to reward overfitting in the out-of-distribution samples due to the distribution shift in π from π ref [Gao et al., 2023 , Rafailov et al., 2024a] . Though we find this information-theoretic formulation interesting, we leave further analysis to future work.Despite its robustness, one may notice that the objective ( 12) is less flexible compared to the original objective, due to the fixed regularization constant with β = 1. Below, we show that if we apply the iterative algorithm provided in Section 3.3, one can further reduce the KL regularization strength and obtain an optimal policy π * in (11) withβ → 0. Theorem 4.2 (Convergence of Iterative Method). Assume π ref ∈ L 2 and P(y > y ′ ) ∈ L 2 .Consider an iterative update of the marginal distribution p 1 in (10). Then, the iteration converges to the uniform distribution of points y where the value E y -∼π ref (P(y > y -)) is the largest, i.e.,EQUATION)where U stands for uniform distribution, and we drop the conditional dependence of x for simplicity.We defer the proof to the Appendix C. Intuitively, the proof follows from the fact that the marginalization iteratively "narrows" down the distribution towards the outputs with higher preference. We note here that the L 2 assumptions are generally valid in practical domains. See Appendix C. 
experimental results: 
 In this section, we conduct experiments to address the following questions: Q1. Can PFM align generated samples from the black-box model with preference and achieve comparable results in practical tasks? Q2. Is PFM more beneficial than methods optimizing for explicit/implicit reward model? and Q3. Is PFM more beneficial than naïve add-on methods, e.g., separately training generative models to imitate preferred samples? To answer these questions, we validate our method in three domains: Conditional text and image generation, and offline reinforcement learning tasks. We first evaluate PFM on a conditional image generation task using the MNIST dataset [LeCun et al., 1998 ]. Specifically, we utilize a pre-trained DCGAN [Radford et al., 2015] Figure 3a illustrates the generated samples from π ref , and the rejected and preferred images are depicted in Figure 3b and Figure 3c , respectively, where the values in parenthesis are the measured preference score. As shown in Figure 3d , PFM achieves higher preference alignment and better sample quality than RLHF (Figure 3e ) and DPO (Figure 3f ) without fine-tuning π ref . Moreover, PFM achieves nearly perfect alignment with the preferences after only two iterations (Figure 3h ), demonstrating the effectiveness of iterative PFM.Next, we adopt a controlled (positive) sentiment review generation task. As done in Rafailov et al. [2024b] , to perform a controlled evaluation, we adopt the pre-trained sentiment classifier as the preference annotator. We train a preference flow on randomly selected pairs of movie reviews y + , y - from the IMDB dataset [Maas et al., 2011] . For our PFM framework to be applied to variable-length inputs, we employ a T5-based autoencoder to work with fixed-sized embeddings. We adopt GPT-2 SFT model on the IMDB dataset as a reference model π ref .We also compare our method with a RLHF (PPO) fine-tuned policy π PPO . See Appendix D for detailed experimental settings.Below, we report the average preference score (from the classifier annotator) of 100 randomly generated review samples for each method. As shown in Table 1 , PFM is able to improve the preference score of any baseline model to which it is attached. We emphasize here that our method requires relatively smaller training cost compared to the standard RLHF frameworks, even in the iterative settings. See Table 6 in Appendix D for the number of parameters that require training for each framework in tackling this task. PFM requires training a much smaller number of parameters (around 1.2%) while still achieving better performance. We also note here that instead of iteratively training the PFM module as described in Section 3.3, simply applying the same learned preference flow iteratively to the improved samples achieves the best performance. (See Appendix D.) π ref + PFM 100% 2% π ref + PFM×5 - 85% π PPO 100% - π PPO + PFM - 99% π PPO + PFM×5 - 100%Interestingly, we observe that the distribution of the scores tends to shift more toward that of the preferred samples with an increasing number of PFM iterations. (See Figure 4 .) This result aligns with our theoretical insights: the PFM framework learns to shift the source distribution (i.e., the distribution of the reference policy) toward the marginal distribution of the more preferred samples.We also compute the win rate with GPT-4 evaluation. The results are summarized in Table 2 . Both PFM and PPO fine-tuned policy excel the reference policy with win rates 100%. (First column of Table 2 .) Furthermore, we observe that the iterative PFM with 5 iterations on the reference model outperforms the PPO fine-tuned policy. If PFM is added on top of the PPO fine-tuned policy, we observe near 100% win rates for both PPO + PFM and PPO + Iterative PFM.Finally, we employ the D4RL [Fu et al., 2020] benchmark to assess the performance of PFM in reinforcement learning tasks. Following the prior works on the PbRL literature, we adopt trajectorybased preference alignment [Hejna et al., 2023 , Kim et al., 2023] . We first randomly choose an starting state s 0 ∼ S, and sample two trajectories τ + and τ -with fixed length ℓ ≥ 2 from π ref :EQUATIONThen, we obtain the preference τ + > τ -given the starting state s 0 using a scripted teacher approach that has also been widely adopted in the PbRL settings [Lee et al., 2021 , Kim et al., 2023] , which prioritizes trajectories with higher rewards based on the ground truth reward. For inference at a given state s t , we again sample an action trajectory τ = (a t , variance across all baseline methods. Notably, our method excels particularly in datasets generated from suboptimal behavioral policies, achieving better performance. Furthermore, PFM manages to match the performance on expert datasets even in the absence of a reward model, underscoring its robustness and effectiveness. This demonstrates that PFM can effectively align black-box models with preference through flow matching, without the need to fine-tune the pre-trained model.In this remaining section, we focus on answering the remaining questions, Q2 and Q3. We first investigate why PFM can be advantageous over previous methods with explicit/implicit reward modeling. As can be seen in Figure 5 , DPO, like typical RLHF approaches, is also prone to reward overfitting, and may cause the agent to fail. This is because if the preference estimate is close to 0 or 1, these methods may end up overoptimizing to the exploding reward model [Ziegler et al., 2019 , Gao et al., 2023] . PFM, on the other hand, is inherently robust to such over-estimation, as we adopt a completely different optimization framework that does not require a reward proxy. (See Theorem 4.1.)On the other hand, we observe less performance gain in the expert datasets. This is a possible failure mode of PFM, where the generated samples are already near-optimal. In such regime, an arbitrary source y ∼ π ref has a near zero probability of being sampled from the true marginal p 0 , suggesting that PFM with prior as π ref might suffer from a shifted source distribution. We verify this experimentally on walker2d, where PFM struggles the most. By adopting a true marginal p 0 as the source, PFM with prior p 0 can achieve the highest performance among all baselines. This performance is evident even on the expert dataset, matching our theoretical analysis. See Appendix B.Next, we compare PFM to an alternative approach that simply tries to approximate the marginal distribution directly from the positive samples. Intuitively, training a generative model from scratch that replicates the marginal p 1 is as computationally costly as training the original reference model. Experimentally, we observe that PFM achieves better performance than training a behavior cloning model (Marginal BC) to replicate the distribution of the preferred samples (Table 3 ). However, it is also worth mentioning that the Marginal BC model does occasionally yield the best results, suggesting the potential of using a marginal distribution for preference alignment. 
related works: 
 Contrastive Preference Learning (CPL) [Hejna et al., 2023] is a class of reward-free methods that utilizes contrastive learning techniques to align model outputs with the preferences observed in the dataset. By leveraging contrastive loss, CPL encourages the model to distinguish between preferred and less preferred outcomes effectively. Flow-to-Better (FTB) [Zhang et al., 2023] innovatively uses a diffusion model to transition from less preferred data to more preferred data, similar to the flow-based approach in our work. However, FTB mainly focuses on data augmentation, where they used the trained diffusion model to generate more data samples for behavior cloning. Despite their strengths, both works rely on the Bradley-Terry model to implicitly learn the reward function.Identity Preference Optimization (IPO) [Azar et al., 2023] builds upon the foundation laid by DPO, extending the framework to accommodate a broader range of preference models beyond the Bradley-Terry paradigm. In particular, they focus on finding an objective that is bounded even in a deterministic preference regime, by replacing the function σ -1 in ( 11) with an identity function I(x) = x. This effectively mitigates the reward overfitting problem observed in DPO and standard RLHF methods.Our method distinguishes itself from these approaches by not requiring the Bradley-Terry assumption nor the fine-tuning of pre-trained models. This eliminates the risk of reward overfitting associated with the Bradley-Terry model and reduces the computational cost significantly. By avoiding the need for fine-tuning, our method offers a more efficient and scalable solution for integrating human preferences into reinforcement learning systems. This makes our approach particularly suitable for scenarios where computational resources are limited or where quick adaptation to human feedback is essential. The comparison of these related works is summarized in Table 4 .  
conclusion and limitations: 
 In conclusion, this research introduces Preference Flow Matching (PFM), a novel add-on approach that offers a practical, efficient, and scalable solution for integrating human preferences. This research highlights the potential of flow matching as a powerful tool for preference alignment and opens new avenues for further exploration and development in the field of RLHF. The ability to align black-box models with human preferences without extensive model modifications marks a critical step forward, with broad implications for the deployment and usability of AI systems in real-world applications.Our theoretical and empirical analyses demonstrate that PFM achieves alignment performance comparable to standard RLHF methods while being more resilient to preference overfitting. The iterative flow matching technique further enhances alignment quality, by continually refining the preference alignment without modifying the underlying pre-trained model parameters.Despite these promising results, the current design of the PFM framework entails several challenges in the natural language processing (NLP) domain. The PFM framework, as currently designed, relies on the autoencoder to work with fixed-sized embeddings to handle variable-length texts. To scale our method to more complex NLP tasks, future research should explore ways to adapt the PFM framework to long-form texts. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks since we do not release datasets or pre-trained models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cited the original papers of the assets. URLs and licenses are mentioned in Appendix D.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: We provide instructions to use our codes in README.md file which is included in the supplementary materials and at our repository https://github.com/ jadehaus/preference-flow-matching.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Below, we outline our method in Algorithm 1. Although the vector field v θ is trained to transport the source distribution of less preferred samples p 0 to the target distribution, we use π ref for inference instead of p 0 due to inaccessibility to the preference model.Note that with the non-deterministic preference assumption, PFM is still theoretically guaranteed to obtain ϕ 1 (y) ∼ p 1 , even if the sample y is obtained from π ref instead of p 0 . In particular, we assume supp(p 1 ) ⊇ supp(p 0 ), i.e., for any sample y + obtained from the target distribution p 1 , the probability of y ∼ p 0 is non-zero. Then, for any sample y ∼ π ref sampled from the reference model, we can guarantee that this sample has non-zero probability of being sampled as less preferred, indicating that there is a learned flow from this point to a better sample, with non-zero probability. In the next section, we provide empirical evidence that further justifies the use of π ref instead of p 0 for inference.Algorithm 1: PFM: Preference Flow Matching1 # Training 2 repeat 3 Sample z = (x, y + .y -) ∼ D 4 u t (y|z) ← y + -y - 5 p t (y|z) ← N (y|ty + + (1 -t)y -, σ 2 ) 6 Sample t ∼ [0, 1] and y ∼ p t (•|z) 7 θ ← θ -η∇L(θ) # from (8) 8 until v θ converges; 9 # Inference 10 Sample y ∼ π ref (•|x) 11 Solve ODE (3) with initial condition ϕ 0 (y) = y using v θ 12 return ϕ 1 (y)We conduct experiments on the D4RL walker2d environment to compare the performance of the flow matching method using π ref and p 0 as source distribution, respectively. To replicate the actual marginal distribution of less preferred samples p 0 , we sample two trajectories from a given state s t , and use the ground truth reward function to select the less preferred data (with lower reward). Note that this use of the environment reward is restricted in practice, as it assumes access to the preference model during inference. We also compare another baseline model, namely a planning model, that searches among the two generated trajectories τ + , τ -∼ π ref (•|s t ) and chooses an action sequence with higher environmental reward.As can be seen in Table 5 , the flow matching model with the source distribution matched to the actual marginal distribution p 0 (as in the training process) achieves the highest performance among all baselines, including a flow matching model that uses π ref instead of p 0 . However, we observe that the flow matching model with π ref as the source distribution yields comparable results to the model using p 0 as source. It is also worth noting that the flow matching model (with true p 0 ) achieves better performance even compared to the planning method, that explicitly uses a trajectory with higher reward. This suggests that using a flow matching for preference alignment can provide better alignment than simply conducting an exhaustive search.In this section, we provide proof to the theorems in the paper. Throughout this section, we write µ as the reference policy instead of π ref for the ease of writing.First, we prove Theorem 4.1, where we rewrite the theorem statement for completeness.Theorem C.1 (Characterization of the Marginal). Suppose that a preference dataset D = {(x, y + , y -)} is collected from a pre-trained reference policy µ, i.e., y + , y -∼ µ(•|x), where the preference y + > y -is labeled with probability P(y + > y -|x). Let p 1 denote the marginal distribution of the positively-labeled samples y + . Then the marginal distribution p 1 obtained from the preference model P(y > y ′ |x) is an optimizer of the optimization problemEQUATION)Proof. For simplicity, we drop the conditional dependence of x. Notice that the general RLHF optimization objective (11) attains a unique optimal solutionEQUATIONAs it is a widely known result, we defer the proof to prior works, e.g., Azar et al. [2023] . Intuitively, comparing the above solution with the target distribution p 1 as rewritten in ( 7) allows us to formulate a similar objective to (11) which p 1 optimizes. Formally, we begin with considering the optimization objective in ( 17):EQUATION)where Z = µ(y)E y ′ ∼µ P(y > y ′ ) dy is the normalization constant. Notice that the distribution in the denominator is precisely the marginal distribution p 1 of our interest, which is indeed a valid distribution. Hence, the last optimization objective can be reorganized as follows.EQUATION)Since the Z is a constant, the KL-divergence objective is minimized at 0 if and only if the two distributions π and p 1 are identical, by the Gibbs' inequality. It follows that p 1 is the optimal solution of the optimization problemm (17), which completes the proof.Next, we aim to prove Theorem 4.2. Before stating and proving the theorem, we note here that the L 2 assumptions of the pre-trained model π ref and the probability model P(y > y ′ ) is generally valid in practical domains. For instance, assuming π ref being trained on L 2 loss as in many practical domains, π ref ∈ L 2 generally holds. Also, the preference model P is usually provided in a finite support, or modeled as Bradley-Terry model, which generally behaves well.Theorem C.2 (Convergence of Iterative Method). Assume that µ ∈ L 2 , and P(y > y ′ ) ∈ L 2 . Consider an iterative update of the marginal distribution p 1 as follows.EQUATIONi.e., we iteratively compute the marginal distribution of more preferred samples from the update marginal distribution. Then, the limiting marginal distribution p (∞) 1converges to the uniform distribution U of points y where the value E y ′ ∼µ (P(y > y ′ )) is the largest, i.e.,EQUATION)Proof. Let us denote by T the transformation applied at each step:EQUATION)where Z p = p(y) p(y ′ )P(y > y ′ )dy ′ dy is the normalization constant. Then, the update rule in ( 25) can be written simply as p(n) 1 = T [p (n-1) 1]. We aim to show that this iterative procedure converges to a distribution that places uniform weight on the set of points y where E y ′ ∼µ (P(y > y -)) is maximized. Given a probability distribution p, Let us define the function f p as follows.EQUATION)Observe that T [p](y) increases p(y) proportionally to f p (y). Consequently, the regions where f p (y) is higher will see an amplification in probability mass relative to regions where f p (y) is lower. This amplification occurs iteratively, with higher f p (y) regions increasingly dominating the distribution.Formally, we claim that the fixed-point iterator T is compact. Notice that the kernel K(y, y ′ ) = p(y ′ )P(y > y ′ ) is bounded, provided that P(y > y ′ ) is bounded by 1, and that p is a probability density, which integrates to 1. By Schur's test and the properties of Hilbert-Schmidt operators, i.e.,EQUATIONT can be shown to be a compact operator, from the square integrability assumptions.Next, consider the behavior of T on any sequence of probability densities {p (n) } ∞ n=0 . By the properties of compact operators in the space of continuous functions, any sequence {p (n) } has a convergent subsequence in the weak topology. Let p * be the limit of any convergent subsequence. The uniform boundedness of K(y, y ′ ) and the compactness of T suggest that T [p * ] = p * , establishing that p * is a fixed point of T . To determine the nature of p * , observe that p * (y) p * (y ′ )P(y > y ′ )dy ′ dy = 1.(30)Since P(y > y ′ ) is maximized uniformly over y when E y ′ ∼µ (P(y > y ′ )) is maximized, p * must concentrate its mass on the set where this expectation is maximized. Therefore, p * converges to a uniform distribution over the setEQUATION)Formally, recall that from Theorem 4.1, the updated distribution p(n)1 is a solution to the following optimization problem:EQUATION)Hence, we note that if any point y not in this set were to have a positive probability under p * , then T [p * ] would not be equal to p * , due to the strict maximization condition, contradicting the fixed-point property. Thus, p (∞) 1converges to the uniform distribution over the optimal set, as stated. We note here that because the space of probability densities is closed under the topology induced by the function space norm, p * should also a probability density, and ultimately the unique minima.In this section, we describe our experimental details.Resource All experiments were conducted on a single Nvidia Titan RTX GPU and a single i9-10850K CPU core for each run. The time required varies by task, but as it only involves collecting preference datasets and learning the flow, it completes within a few hours.Task We employ the MNIST [LeCun et al., 1998] 2 dataset to evaluate PFM on a conditional image generation task.Preference Dataset We utilize a pre-trained DCGAN [Radford et al., 2015] generator as π ref and collect sample pairs from π ref conditioned on the digit labels as contexts. To construct preference datasets, we assign preferences to sample pairs according to the softmax probabilities of the labels from a LeNet [LeCun et al., 1998 ].Baseline We consider the pre-trained DCGAN generator (π ref ), a RLHF fine-tuned model and a DPO fine-tuned model of π ref as baselines. All methods are trained until convergence, and we report the normalized episodes returns with the standard deviation from 5 different random seeds.Task To evaluate PFM on the NLP domain, we adopt a controlled (positive) sentiment review generation task with the IMDB dataset [Maas et al., 2011] .Preference dataset As done in Rafailov et al. [2024b] , to perform a controlled evaluation, we adopt the pre-trained sentiment classifier3 as the preference annotator. The preference dataset is constructed from randomly selected pairs of moview reviews y + , y -from the IMDB dataset, where the preference is obtained from the classifier logit probability p(positive|y + ) > p(positive|y -). We then train our PFM model on this preference dataset, to obtain the marginal distribution of the preferred (positive sentiment) review p 1 (y + ).Baseline We consider the GPT-2 SFT model on the IMDB dataset (π ref ) and a RLHF fine-tuned model of π ref using PPO (π PPO ) as baselines. We apply PFM to baseline models as an add-on module without fine-tuning. For the iterative PFM, we iteratively apply the initially obtained learned preference flow without collecting new data from the improved PFM model.Embedding for variable-length input For our PFM framework to be applied to variable-length inputs, we employ a T5-based autoencoder4 to obtain fixed-sized (1024-dimensional vector) embedding of the input texts, allowing us to work within the fixed-size latent space. Once the fixed-size embeddings z + and z -are obtained for each text sample y + and y -, we learn the conditional flow using PFM from z -to z + . During inference, we apply PFM to the latent embedding z of the given input text y, and decode the improved latent embedding using the T5 decoder. We adopt the same U-Net architecture used in our MNIST experiment, where we reshape the 1024-dimensional vector into a two-dimensional (32, 32) image tensor. As shown in Table 6 , PFM requires a much smaller number of parameters (around 1.2%) than RLHF or DPO which involve fine-tuning LLMs.Table 6 : Parameters required for training for each method. PFM only requires 1.2% parameters to be trained compared to naive approaches (RLHF, DPO, etc.), and still achieves better performance in preference alignment.1.5MIterative PFM For our iterative PFM, we do not iteratively collect data and re-train the module. Instead, we simply iteratively apply the learned preference flow to the output samples. In particular, we apply the learned flow to the embedding z iteratively with the same flow module.Prompt for GPT-4 Evaluation Below we include the prompts used to generate win rates.Which of the two movie reviews has a more positive sentiment?Response A: <Response A> Response B: <Response B>Task To assess the performance of PFM in reinforcement learning tasks, we employ the D4RL [Fu et al., 2020] benchmark from https://github.com/Farama-Foundation/D4RL where the datasets and code are licensed under the Creative Commons Attribution 4.0 License (CC BY) and the Apache 2.0 License, respectively. We consider four different tasks (ant, hopper, halfcheetah, walker2d) from Gym-Mujoco domain with three different levels of offline dataset quality (random, medium, expert) for each task.Preference dataset We first pre-train π ref using behavior cloning (BC) for each offline dataset. We then collect segment pairs of the rollout trajectories from π ref , with each pair starting from the same state as a context. To construct preference datasets, we utilize a scripted teacher which prioritizes trajectories with higher rewards based on the ground truth rewards provided by the environment. This approach has been widely adopted in the PbRL settings [Kim et al., 2023 , Lee et al., 2021 , Zhu et al., 2024] . The preference datasets consist of 1,000 pairs of preferred and rejected segments and their context for each offline dataset, with the segment length 10.Baseline The baseline methods for comparing the performance of PFM includes behavior cloning (BC), which we adopt as our pretrained reference model π ref , and a DPO fine-tuned model from the BC model. For DPO fine-tuned models, we search KL regularization coefficient β from 0.01 to 100 and adopt the best one. Additionally, we train a separate behavior cloning model to the collected preferred samples y + ∼ p 1 , aiming to replicate the marginal distribution of the "good" trajectories. All methods are trained until convergence, and we report the normalized episodes returns with the standard deviation from 5 different random seeds.As an add-on module, PFM can be seamlessly integrated into various real-world AI applications, such as generative models and continuous control systems, without the need to modify the underlying application models. This integration enables the applications to deliver personalized results that align with individual user preferences. However, since PFM utilizes preference data, it raises potential privacy concerns similar to those found in typical PbRL methods. These concerns can be mitigated by ensuring that access to user preferences is granted only with explicit user consent.NeurIPS Paper Checklisthttp://yann.lecun.com/exdb/mnisthttps://huggingface.co/lvwerra/distilbert-imdbhttps://huggingface.co/thesephist/contra-bottleneck-t5-large-wikipedia 


Decision: This papers proposes Preference Flow Matching (PFM), a new approach for preference-based RL that doesn't require an explicit (or implicit) reward model, leverating optimal transport to shift the original sampling distribution towards that of preferred samples.

Reviewers' ratings exhibited a high variance (3 to 9), with a common main concern regarding the lack of application to NLP tasks (which are the main reason behind the current high interest in preference-based learning). However, the authors provided new experimental results on an NLP task that at least partially alleviate this concern.

One reviewer (2tji / rating 4) still remained not entirely convinced after extensive discussions with the authors, in particular regarding the relevance of the full set of experimental results. I do agree with the reviewer that there is room for improvement in this area, however, I also want to show appreciation for the originality of this work that offers a fresh approach on preference-based learning, which I believe may serve as source of inspiration to future developments in similar directions. Considering that all reviewers were appreciative of this novel contribution, which is clearly presented and motivated, I consider this paper worth accepting at NeurIPS.

(I also note that the lowest-scoring reviewer -- with a 3 -- did not acknolwedge the authors' response, in spite of my reminder to do so)

Reviews: review: paper_summary: paper_summary: This paper introduces Preference Flow Matching (PFM), a novel framework for preference-based reinforcement learning that addresses key limitations of existing methods like Reinforcement Learning from Human Feedback (RLHF). PFM leverages flow matching techniques to directly learn from preference data. The core innovation of PFM lies in its formulation of preference alignment as a problem of learning a flow between marginal distributions of less preferred and more preferred samples, moving away from the implicit discriminative classification formulation of DPO into a discriminative/generative formulation. Methodologically, PFM employs a conditional flow matching objective to learn the vector field representing the preference flow. It also does not require the Bradley-Terry assumption. PFM is also compatible with 'reflow' like the original flow matching formulation. The authors derive the marginal distribution and prove it converges to a maximizing uniform. This all represents a clear, if straightforward, combination of (reward-free) preference learning and flow-matching providing a ready-made comparison with existing *-PO methods.
strength_weakness: strength_weakness: - Paper writing could be improved as several typos remain ('A careful reader might noticed' l.126, 'ordinary diffusional (sic) equation' l.89...). Some proofreading is in order !

- Most importantly, the empirical contribution is not overly compelling. It is surprising that on MNIST the DPO results are that poor, both in absolute terms and relatively (worse than RLHF). IPO is not considered, DPO is probably taken offline rather than online (which generally tends to improve results), and most importantly, the beta parameter is not iterated upon/optimized (if it is, then this needs to be made clear, if possible in the main text, but it is not even in Appendix D.1). It is well known that the beta parameter is crucial for the performance of those methods and one advantage of very small scale experiments such as MNIST is that iterating is inexpensive. Tuning the baseline to the maximum would lend credence to these experiments; besides, it is done in the other suite of experiments.

- Similarly, the offline RL experiments are small scale (Gym-MuJoCo. This time the beta parameter is optimized, and on all expert domains, the PFM performance is not massively different from DPO - in fact, the expert average for DPO fine-tuned is stronger than that of PFM (even though PFM provides variance reduction). While we know that DPO has tendencies to overfit and reward-hack, this is noted by the authors themselves who ask in section 5.3 'is learning a flow truly beneficial' ? Finding this detailed and balanced discussion in the paper was a nice display of intellectual fortitude and very welcome. Nonetheless, the empirical benefits of the proposed approach so far are quite ambiguous. The paper would truly benefit from showcasing one indisputable and conclusive empirical setting in favour of PFM.

- Finally, the paper doesn't deeply explore the computational costs of PFM compared to other methods, which could be an important practical consideration. In particular, is reflow worth it ?
questions: questions: - What stopped the authors (other than lack of time, infrastructure, compute) from trying for small-scale language modelling experiments, since this is the domain that originally stemmed the derivation of DPO and IPO ? In reference to 'Another notable limitation is its applicability to the natural language processing (NLP) domain. The PFM framework, as currently designed, cannot be directly applied to NLP tasks because the source and target texts may have different lengths', it would technically be possible to model distributions over fixed length sequences, along with padding and/or an EOS token. One great advantage of working in the text domain is the ability to display sample completions from finetuned models in a qualitative study. Also, testing with actual human preferences would actually be better.

- Even if additional experiments are out of scope of this current review period, which other empirical designs can the authors think of that would better showcase the benefits of PFM compared to *-PO ?
limitations: limitations: Limitations are noted in the 'weaknesses' section below and mostly acknowledged in the paper.
paper_summary: paper_summary: The paper presents a new framework for preference-based reinforcement learning, relying on Flow Matching. This framework utilizes flow-based models with optimal transport that interpolate between less preferred data and more preferred data, eliminating the need to estimate a reward function implicitly or explicitly. The authors provide a rigorous framework supported both by formal proofs and clear experimental evidence of the method's performance and demonstrate the effectiveness of iterative PFM, which iteratively narrows down the distribution towards preferred outputs.
strength_weakness: strength_weakness: The paper is very strong, no notable weaknesses observed, suggestions for improvements are provided in the next section.
questions: questions: - Consider an expansion of failure mode explanations for DPO from the paper https://arxiv.org/pdf/2404.10719, namely susceptibility to distribution shift, which can cause issues with generalizability in the regions without data support. They tie back to the author's critique of the failure modes of DPO, like overfitting.

- line 210: "Is PFM beneficial than methods optimizing for explicit/implicit reward model" is this missing "more beneficial"? (same for Q3)

- lines 126-129 highlight that the starting distribution $p_0$ is inaccessible during the inference step, and instead, they simply start from $y \sim \pi_{ref}$ as the starting point. This is an important implication for sampling/inference. Thus, a comment explaining why this works and whether it can be applied in any scenario under consideration is crucial. Consier is adding this to limitations if there are issues with applicability.

- In the conclusion and limitations, the authors state limitations of applicability to NLP tasks: " Future research should explore ways to adapt the PFM 314 framework for variable-length data, potentially through innovative alignment techniques or alternative 315 frameworks suited for text generation tasks." Diffusion Forcing: https://arxiv.org/abs/2407.01392 has been introduced recently to combine both autoregressive next token predictions with full-sequence diffusion. Consider this as a minor suggestion given the time constraints and timelines of the initial submission, the could be a useful source for future work.
limitations: limitations: The authors adequately addressed the limitations of their work throughout the paper and in the conclusions and limitations section.

A minor suggestion:
- The authors state the challenges of existing preference-based RL methods such as scalability, inefficiency, and the need for model modifications, especially with black-box, and later address them in the paper, except for scalability, which calls for stating more clear limitations or suggesting lines of future work to address this (extending the limitations in lines 309-310 would suffice).
paper_summary: paper_summary: This paper proposes an add-on method named Preference Flow Matching (PFM) to achieve preference alignment without learning a reward model. PFM can transfer $y$ generated from the original model to the preferred $y^+$ through a few flow iterations. Experiments demonstrate its superior performance compared with RLHF and DPO.
strength_weakness: strength_weakness: - During inference, PFM with prior as $π_{\rm ref}$ suffers from a shifted source distribution, which can affect the performance.
- Limited testing benchmarks. RLHF is getting a lot of attention these days, largely due to its success in the language model area. However, this paper does not include the language-based task because PFM is limited in its ability to generate variable text.
questions: questions: - During inference on the D4RL tasks, how do you choose actions to interact with the environments? As the paper claims, PFM samples an action trajectory $\tau$ from $π_{\rm ref} (·|s_t)$, and then applies flow matching to obtain a better action sequence $\tau_{\rm new}$. How can you sample a trajectory $\tau$ without training a dynamic model? Do you apply all actions in $\tau_{\rm new}$ to interact with the environment, or select some actions from $\tau_{\rm new}$ to take?
limitations: limitations: Limitations have been discussed in this paper.
paper_summary: paper_summary: This paper presents Preference Flow Matching (PFM), which learns an ODE to transform less-preferred data into more-preferred ones. Due to PFM not explicitly or implicitly relying on reward functions, it avoids reward overfitting. Empirical evidence demonstrates that PFM outperforms DPO in image conditional generation and Offline RL tasks.
strength_weakness: strength_weakness: - The novelty of the paper appears somewhat limited, as PFM seems to use Conditional Flow Matching (CFM) to transform between less and more preferred data distributions. However, the paper does not thoroughly discuss the benefits of introducing CFM. Similar transformations can be achieved with Rectified Flow or Diffusion models, and work using Diffusion models (such as FTB) already exists.
- The experimental validation in the paper is not sufficiently comprehensive, leading to less convincing results. For instance, in the context of Offline RL tasks, the following issues may exist:
    - PbRL is commonly used to address tasks with difficult-to-define or sparse rewards. In D4RL, tasks like antmaze are more commonly used for better algorithm performance comparison.
    - In MuJoCo tasks, PFM does not exhibit a significant improvement over DPO. Additionally, the standard deviation of the results appears to be quite large, making comparisons based solely on average means less reliable. Could this be due to limited random seeds? Typically, MuJoCo tasks have low standard deviations for having dense rewards.
    - While the authors summarize many RLHF methods in the related works section, only DPO and vanilla RLHF are compared across the experiments, indicating a potential lack of baseline algorithms. For example, FTB uses a diffusion model to do similar tasks as PFM. Given the similarity between these two algorithms, why are they not compared in the study?
questions: questions: Please see Weaknesses.
limitations: limitations: Please see Weaknesses.

The main areas for improvement in the paper are concentrated in the experimental section, as the current experimental results lack persuasiveness.
paper_summary: paper_summary: The paper introduces a novel framework called Preference Flow Matching (PFM) for preference-based reinforcement learning (PbRL). The PFM approach aims to integrate human preferences into pre-trained models without the need for extensive fine-tuning, which is a common requirement in existing PbRL methods. The PFM framework utilizes flow matching techniques to learn directly from preference data, reducing the dependency on pre-trained model fine-tuning. It transforms less preferred data into more preferred outcomes by leveraging flow-based models, aligning model outputs with human preferences without explicit reward function estimation. The paper provides theoretical insights and empirical results that demonstrate the effectiveness of PFM in aligning pre-trained models with preferences and offers a new direction in PbRL. PFM is proposed as a solution to challenges such as scalability, inefficiency, and the need for model modifications, especially with black-box APIs like GPT-4.
strength_weakness: strength_weakness: Exaggerated Motivation:  I believe the research motivation stated by the author in the abstract and introduction may be exaggerated. For example, the author claims that the method can be applied to black-box models like GPT-4, but there is no experimental validation. In fact, even for general language models, the author provides no experimental evidence to show that the proposed method is superior to DPO.

Limited Domain Testing: The paper's experiments may not fully explore the potential of PFM across diverse domains, particularly in more complex tasks like general contextual generation. 

Assumption is too strong : The assumption is not general for RLHF scene. For example,  conditional flow is $y^+ - y^-$ (line 116), and the  probability path is subject to gaussian distribution,  which is not pratical for general AI model, such as GPT-4.

Potential for Overfitting: Although PFM is designed to be robust against overfitting, the method might still be susceptible to overfitting in certain scenarios, particularly if the preference data is limited or biased.

Applicability to NLP Tasks: The current design of PFM may not be directly applicable to natural language processing tasks due to challenges with variable-length data, which could be a significant limitation given the prevalence of NLP applications.
questions: questions: 1. I believe the assumptions in the paper are overly idealistic, as the method appears impractical for application to black-box models like GPT-4, and may even struggle with open-source models. If the authors could provide a specific plan on how to apply the method to open-source models (such as llama) and black-box models (such as GPT-4), including details of experiments and experimental results, I would consider raising the score. Otherwise, we suspect the research motivation in the paper has been exaggerated.

2. How to apply the simple assumptions from lines 116-117 to LLM?
limitations: limitations: The authors have made an effort to address the limitations and potential negative societal impacts of their work.

Comments: title: Additional Response to Reviewer 2tji
Key Contributions and Differences of PFM Compared to FTB
Clarification of the D4RL Results.
Thanks for your response!
comment: Thank you for your rebuttal and additional empirical results. Cognizant of the fact these take time to run, we have to stress the scale-dependent nature of those results and would reserve judgment for now as to whether the billion parameters+ LLM scale will improve as much as GPT-2. This leaves the empirical section fairly ambiguous in our view, and as such, we maintain our score.
We thank you again for your time and effort. We will incorporate your comments and feedback in the revision.
Thank you so much for your valuable feedback and the productive discussions. We appreciate your detailed comments that helped us improve our research.

As you suggested, we are working on further improving our experimental setup. Specifically, we are planning to conduct additional experiments in datasets such as medium-replay and medium-expert, with additional baselines. We also intend to obtain more results for NLP tasks.

One of the key advantages of PFM is its ability to consistently enhance preference alignment as an add-on method, regardless of the given black-box reference policy. We have demonstrated this theoretically. To further emphasize this strength, we will include a wider range of reference policies in our additional experiments. Particularly for the D4RL benchmarks, we aim to employ various baselines and incorporate more reference policies to showcase that PFM is a versatile framework, capable of being applied across diverse settings and black-box models. We thank you again for your time and effort.
Thank you for your response. I greatly appreciate the authors' efforts. Here's my reply:

**Regarding the D4RL experimental results**
The new table's score report is much more readable. However, I still have some doubts about the DPO results. From the table, it seems that DPO fine-tuning has almost no effect in many environments. In my implementation, though, I saw noticeable improvements with low beta values and few-step updates. I used a very simple MLP neural network structure, so the performance difference might be due to different beta values, or perhaps because I used a squashed Gaussian distribution, which may be more robust. Anyway, I believe using standard D4RL scores improves readability, and I suggest the authors adopt this approach in their revisions.

**Data collection for PFM**
I still maintain that offline PbRL settings shouldn't allow additional online interaction data collection. However, since the authors used the same training method for PFM and all baselines in their algorithm comparison, I think this fairly reflects the algorithms' performance in preference matching. So this may not be an issue after all.

**Empirical results for PFM on various tasks / Regarding additional baselines / Regarding NLP results**
I still believe the current testing tasks are insufficient. I appreciate that the authors conducted experiments across RL, CV, and NLP domains, demonstrating PFM's versatility, but the tasks used in these three domains are too simple (MuJoCo, MNIST, IMDB). Regarding the experimental setup, I have the following suggestions:

* Add IPO and FTB as baselines. For RL tasks, I believe the MPC baseline is unnecessary, as mentioned in the "Data collection for PFM" part.
* For RL tasks, try medium-expert and medium-replay datasets. For PbRL (or RLHF), clear contrasts are important. The currently chosen random/medium/expert datasets are collected from policies with uniform performance, making comparisons difficult and potentially obscuring the algorithm's full capabilities.
* For RL tasks, try antmaze datasets. These provide diverse datasets and use extremely sparse reward functions, which is where PbRL can show its capability.
* For all tasks, add real human feedback experiments. Almost PbRL or RLHF papers include experiments with real human feedback on typical tasks to better compare algorithm performance in practical applications. The current script teacher experiments may have some reward hacking issues - for example, in Figure 3, I think (d) PFM is clearly more recognizable than (h) Iterative PFM.
* For NLP tasks, add experiments with black-box models like GPT-3.5 or GPT-4. Sorry if I was unclear before - I understand that black-box models can't be fine-tuned using DPO/RLHF. What I meant was that GPT-4 could provide good real human feedback experiments. For instance, train a PFM on a small text dataset to make sentences less friendly, then have humans evaluate whether PFM can make GPT-4's output less friendly. It's known that GPT-4 is fine-tuned to reduce unfriendly content generation, so if PFM could change this, it would be very interesting.

All in all, I still think the paper's experimental content is insufficient, so I'll maintain a negative rating. However, I appreciate the authors' efforts in the rebuttal and our productive discussion. I look forward to the additional baselines being added, so I'll increase my rating by 1.

Also, this paper's ratings currently show a high variance. As the discussion period is ending, I've noticed that I seem to be the only reviewer discussing with the authors. I strongly recommend that the authors remind (or ask the AC to remind) other reviewers who haven't participated in the discussion to share their opinions. Thank you again for your response.
Thank you again for your invaluable comments. Below, we address your additional concerns:

**Regarding the D4RL experimental results.**

Thank you for addressing the appropriate normalization scheme for D4RL tasks. Below, we refine our results based on the call `env.get_normalized_score` on our trained models.

Environment|BC|DPO Fine-tuned|PFM (Ours)|Marginal BC
-|-|-|-|-
ant-random|31.59±0.05|31.52±0.08|**31.62±0.13**|25.01±4.64
ant-medium|90.16±21.48|95.04±13.93|96.73±2.47|**99.67±1.57**
ant-expert|125.83±24.07|**134.96±3.76**|132.2±2.69|99.29±34.74
hopper-random|3.17±0.25|3.23±0.25|**7.69±0.08**|5.48±4.46
hopper-medium|52.83±5.03|53.47±3.92|**58.76±2.62**|40.44±1.69
hopper-expert|111.27±0.48|111.51±0.92|**111.7±0.77**|32.39±0.1
halfcheetah-random|2.25±0.01|2.26±0.01|**2.26±0.0**|2.21±0.02
halfcheetah-medium|40.97±0.89|41.94±0.68|**43.49±0.88**|38.79±1.27
halfcheetah-expert|91.02±1.24|**92.15±0.76**|90.05±0.83|4.77±2.5
walker2d-random|1.47±0.1|1.38±0.08|1.77±0.13|**2.45±0.38**
walker2d-medium|60.35±18.16|**74.05±12.05**|72.59±15.8|65.29±12.58
walker2d-expert|**108.62±0.39**|108.38±0.28|108.36±0.21|15.8±0.54
    
Please notice that the scale is now close to the values that you reported. We note that our comparison with DPO is fair since we trained on the same dataset, using the same policy network. The reason your reported values might be slightly different from our results include:
- Different datasets used to train your DPO and ours.
- Policy network used for both DPO and PFM for our experiments is slightly different from yours; we adopted the same architecture used in the DDPG implementation of D4RL repository.

Nonetheless, we believe that our D4RL experiments were conducted fairly, and our current experimental results are adequate. Additionally, our results are comparable to DPO, if not slightly better in some cases, indicating that our method could be viable for black-box settings where DPO may not be applicable. We've also provided comprehensive results across various tasks and domains, including the NLP results mentioned in the common response section. We believe that these results sufficiently demonstrate the general applicability and effectiveness of our framework.
    
**Data collection for PFM.**

In the case of PbRL tasks, we conducted all the experiments on a pre-collected dataset from the behavior policy and trained policies on this data in an offline setting. Therefore, contrary to your concern, we did not violate the offline setting. Of course, it is true that we collected data separately, rather than using the existing offline dataset, in order to obtain trajectories starting from the same state. However, this is a common issue to be faced by all methods that make inferences from the same context (in the case of RL tasks, the initial state), not just by PFM. The assumption of PFM is to align preferences for a given context, so we collected a dataset to best demonstrate this setting. Please note that in other domains (e.g., vision, language), data is generally provided in this form, so this issue only arises in RL tasks.

**Empirical results for PFM on various tasks.**

We believe that our experimental results are sufficient to demonstrate the value of PFM. We conducted additional experiments not only in the RL and vision domains, but also on NLP tasks. In all of these tasks, our method performed comparably to baselines or, at times, even better. As mentioned above, we believe this is sufficient to show that our method has the potential to function as a new framework. Furthermore, our method is based on a theoretical guarantee, which is also one of our main contributions.

**Regarding additional baselines.**

For RL tasks, we plan to include FTB and IPO as additional baselines. We are currently working with the code, but due to time constraints and the challenge of ensuring a fair comparison between PFM and FTB under the same conditions, we were unable to include these results during the discussion period. However, we will make an effort to include these baseline results by the camera-ready deadline. Furthermore, we will also try to implement the preference-based reward function planning that you suggested, as an additional baseline.

**Regarding NLP results.**

We have provided results on an NLP task in the common response to support our claim that PFM can be applied to black-box models. Specifically, we used GPT-2 as a baseline in our experiments, where we treated the GPT-2 as a black-box model in our experiments. The reason we chose GPT-2 instead of GPT-4 is that the publicly available SFT model trained on the IMDB task, as well as the RLHF fine-tuned model, are both based on GPT-2. While our method could be applied to GPT-4, a fair comparison with the RLHF method isn't possible because no GPT-4-based RLHF fine-tuned model is available. Additionally, it's worth noting that the DPO paper also used the same GPT-2 base model for performance comparisons.
Thank you for your response. However, I have to say that it does not fully address my concerns. I still have the following questions:

- Why does the author still not use the exact D4RL score in the new Table 1? The author only needs to call `env.get_normalized_score` without training new models or re-testing existing ones. What's the rationale behind training a new BC-random and a new BC-expert for normalization? This undoubtedly makes fair comparisons difficult. Moreover, the reported scores in new table don't seem competitive compared to other Offline PbRL algorithms with script teachers. I suspect the author may not have thoroughly tuned the implementations. To verify this, I implemented a DPO myself. The policy is a squashed Gaussian distribution, using a 3-layer MLP with 256 hidden units. Following the same procedure as in Appendix D.2, I obtained scores on Hopper tasks that are significantly better than those reported by the author (see table below). This might be due to my use of very small beta values: 1e-10 for expert, 1e-8 for medium, and 5e-9 for random, fine-tuned over 500 gradient steps with a batch size of 32. Regardless, I believe PFM's current performance on D4RL-MuJoCo is not satisfactory.

|env|my DPO| author's DPO|
|---|---|---|
|hopper-expert|110.7+-0.7|100.2+-0.9|
|hopper-medium|79.7+-1.2|46.5+-3.6|
|hopper-random|7.6+-0.1|0.1+-0.2|

- While implementing DPO, I noticed that PFM requires online interaction with the environment when collecting the preference dataset. This seems entirely inappropriate for an offline setting. One of the main challenges in offline settings is the limited dataset, and introducing online interaction data would significantly alleviate the OOD issue. I believe this approach may introduce unfair comparisons. Additionally, PFM requires interaction with the environment during inference to obtain a reference trajectory and then optimizes it, which assumes access to correct dynamics. If this is the case, why not directly use a preference-based reward function for planning? At the very least, I think this would be a reasonable baseline to include.

- I believe the experimental section is currently insufficient, a point also raised by other reviewers. Firstly, the chosen benchmarks are relatively simple: MNIST for image generation and only MuJoCo for PbRL. The author's analysis of the experimental results lacks depth, seemingly offering not enough insightful takeaways beyond "PFM has better performance." Also, I suggest the author include more baselines, such as IPO and FTB. These methods are compared in the related works section but not in the experiments, especially FTB, which is most similar to PFM.

- I strongly agree with reviewer XK5w's point that while the author claims the method can be applied to black-box models like GPT-4, there's no experimental validation of this. More experiments on black-box models would better showcase PFM's advantages and support the author's claims. Why didn't the author choose to include experiments on GPT-4 in the rebuttal? If cost is a concern, even experiments with GPT-3.5 would help demonstrate PFM's strengths.

Given these unresolved issues, I will maintain my current rating.
To address your second question and provide a clearer understanding of the differences and contributions of PFM compared to FTB, we offer the following explanation:

Firstly, let us revisit FTB and its approach. FTB creates $K$ blocks of preference levels. Since the labeled or unlabeled datasets are not initially clustered into these $K$ levels, FTB requires score function estimation to assign trajectories to each block (or cluster). Once the dataset is classified into these $K$ performance-based blocks, FTB learns a diffusion model that transports a trajectory to match the score level of the neighboring better block. According to FTB, this transition between neighboring blocks ensures that the performance gap between each pair of trajectories is relatively consistent. Notably, FTB chose $K=20$ as the number of blocks and experimented with a minimum of 10 blocks even in their ablation study. The key motivation of FTB is to *unconditionally* generate good-quality trajectories within the fine-grained blocks. Note that this approach does not condition on the current state, but on the entire trajectory, making direct inference application challenging without additional steps. They also rely on the relatively close distance between these trajectory samples, as they do not condition on the same initial state, but simply optimizes for the complete trajectories, unconditionally. In short, FTB focuses on obtaining better trajectories, but not necessarily better trajectory *for the current starting state*.

In contrast, PFM aims to improve the trajectories generated by a reference policy in a manner that is immediately usable during inference. In other words, we aim to directly find the trajectory (or improve upon the current trajectory) that will lead to better results given the current state and situations. This is achieved by learning a *vector field* that directly enhances the trajectory according to current conditions (e.g. current initial state condition). Unlike FTB, which improves trajectories unconditionally within predefined score blocks, PFM conditions on the current state, making direct inference application more straightforward without additional steps. Notably, once we obtain a vector field that characterizes the probability transport of trajectories, we can iteratively apply this learned vector field to achieve the same improvement effect as FTB’s iterative upgrade to the neighboring higher-scored block. Our flow model improves the trajectory while preserving the conditional information of the current situation, making our method more interpretable and adaptable.

If one reduces FTB to having a block number $K=2$ and removes the policy extraction phase, FTB's approach becomes approximately similar to PFM, as you suggested. However, as mentioned above, the core motivations differ significantly. The authors of FTB did not reduce the block number to 2 (as in standard RLHF frameworks) and added a policy extraction phase due to the inability of FTB to be directly applied during inference, as its flow does not conditionally improve the trajectory. 

Another primary benefit of PFM is its ability to enhance action trajectories without the need for score functions. PFM avoids overfitting in reward models by directly aligning model outputs with human preferences through flow-based models. This approach contrasts with FTB’s reliance on score functions for block assignment, which can lead to overfitting and suboptimal performance, as mentioned in our paper. PFM’s method of learning from preference data directly addresses these concerns, providing a more robust solution.

While FTB’s unconditional data augmentation approach can be beneficial in certain reinforcement learning (RL) scenarios, it is less adaptable to other tasks, such as language modeling. In language tasks, generating contextually appropriate sentences is crucial. PFM’s conditioning on the current state allows for more relevant and context-specific trajectory improvements, making it more versatile across various applications. FTB, on the other hand, focuses on generating high-return sentence sets, which is less practical for tasks requiring specific contextual appropriateness. Furthermore, directly applying FTB to language tasks requires an additional SFT phase to the obtained augmented dataset.

In summary, PFM provides a more efficient and versatile method that is particularly well-suited to black-box models and diverse applications beyond RL. By learning a vector field conditioned on the current state, PFM can directly enhance trajectories in real-time, making it a valuable contribution to general preference alignment in diverse domains. We hope this explanation clarifies the unique aspects and contributions of PFM. We appreciate your insightful questions and are happy to engage in further discussion to deepen the understanding of our work.
Thank you for your detailed feedback and for raising these important questions. We appreciate the opportunity to further elucidate the contributions and distinctions of our framework, and the opportunity to clarify the metrics used in our paper, particularly in Table 1.

First, let us clarify the metrics reported in Table 1. The standard D4RL normalized score is calculated using the formula:

$\text{normalized score} = 100 \times \frac{\text{score} - \text{random score}}{\text{expert score} - \text{random score}} $

In other words, they report the score for the random policy as 0, and the score for the optimal (or expert) policy as 100. While the FTB paper uses this D4RL normalized score, in our work, we opted to report the cumulative rewards of each method (referred to as un-normalized scores in the D4RL paper, Table 3) normalized to a scale of 1000 for better readability. This choice was made to provide a clear and direct comparison of the methods' performance. Accordingly, in our normalization process, we divided the standard deviation by $\sqrt{1000}$, resulting in the mean and standard deviation appearing similar in scale. This method was chosen for consistency and to maintain a straightforward interpretation of the results. We would like to note that this choice also allows us to better compare the performance levels with respect to the reference policy that each method started with, not with the ground-truth worst and best policy as standards. Below, we provide the results of Table 1 obtained via the D4RL normalization scheme as reference:

| Environment            | DPO Fine-tuned    | PFM (Ours)       |
|------------------------|-------------------|------------------|
| ant-random-v2          | -0.1 ± 0.1        | **0.0** ± 0.1    |
| ant-medium-v2          | 67.3 ± 14.8       | **69.1** ± 2.6       |
| ant-expert-v2          | **109.7** ± 4.0   | 106.8 ± 2.9      |
| hopper-random-v2       | 0.1 ± 0.2         | **4.2** ± 0.1    |
| hopper-medium-v2       | 46.5 ± 3.6        | **51.4** ± 2.4   |
| hopper-expert-v2       | 100.2 ± 0.9       | **100.4** ± 0.7  |
| halfcheetah-random-v2  | **0.0** ± 0.0     | **0.0** ± 0.0    |
| halfcheetah-medium-v2  | 44.7 ± 0.8        | **46.5** ± 1.0   |
| halfcheetah-expert-v2  | **101.3** ± 0.9   | 98.9 ± 0.9       |
| walker2d-random-v2     | -0.1 ± 0.1        | **0.3** ± 0.1        |
| walker2d-medium-v2     | **67.7** ± 11.3   | 66.4 ± 14.7      |
| walker2d-expert-v2     | **99.8** ± 0.3    | **99.8** ± 0.2   |

Here, we used the scores of BC-random and BC-expert in place of the scores of actual random and expert policies for normalization. (That is, the pretrained-BC models are set to 0 and 100, for random and expert, respectively.) Although this is not the exact same normalization scheme as the standard D4RL benchmarks, since both methods transform the scores to the same scale, the results remain consistent.
Thanks for your response and hard work. After reading your rebuttal and other reviews, I've decided to maintain my score.
Thank you very much for the thorough response. I still have some unanswered questions. I provide more details below:
- What metric is used in Table 1? To my knowledge, the standard D4RL normalized score differs significantly in magnitude from the results presented in Table 1. I excerpt the D4RL scores for PFM, FTB, PT, and BC in the table below. The results for FTB and PT are sourced from the FTB paper's report [1], while the BC results are from the D4RL report [2]. Except for BC, all three methods utilized script teachers. PFM's results differ significantly in magnitude. So I wonder if PFM used a different metric. However, the appendix only mentions "we report the normalized episodes returns." My mention in the review that "the standard deviation of the results appears to be quite large" also refers to the discrepancy in magnitude between the standard deviation and mean values. It is understood that offline PbRL would naturally exhibit a notably larger standard deviation, but comparing it to the mean score in terms of magnitude is also reasonable (see results of FTB and PT).

|                         | PFM (Yours) | FTB         | PT (Preference Transformer) | BC   |
| ----------------------- | ----------- | ----------- | --------------------------- | ---- |
| hopper-medium-expert-v2 | -           | 111.1 ± 2.0 | 77.8                        | 52.5 |
| hopper-medium-replay-v2 | -           | 89.6 ± 4.9  | 31.4                        | 18.1 |
| hopper-expert-v2        | 3.6 ± 0.8   | -           | -                           | -    |
| hopper-medium-v2        | 1.9 ± 2.7   | -           | -                           | 52.9 |

- It is indeed a default setting in FTB to employ an additional policy extraction phase. However, FTB can also, like PFM, directly enhance the quality of action trajectories during inference. This means that FTB can also operate on black-box models. In this context, if we use FTB in this manner, what are the main contributions of PFM compared to FTB? (I am not challenging the authors' work; I simply seek a better understanding.) Additionally, I do not consider the policy extraction phase in FTB to be very costly, nor do I view it as a drawback. The BC policy used in FTB's policy extraction phase can be a very small MLP that does not require significant computational resources. Moreover, during inference, it avoids calling the generative model for trajectory optimization at each step, making it faster and more lightweight.

---
References

[1] Zhilong Zhang, et al, "Flow to Better: Offline Preference-based Reinforcement Learning via Preferred Trajectory Generation," in The Twelfth International Conference on Learning Representations, 2024.

[2] Justin Fu, et al, "D4RL: Datasets for Deep Data-Driven Reinforcement Learning," in arXiv, 2004.07219, 2021.