Learning Parametric Distributions from Samples and Preferences
Authors: Marc Jourdan
Gizem Yüce
Nicolas Flammarion
Venue: ICML 2025 spotlightposter (2025)
Keywords: Statistical learning
Continuous parametric distributions
Preference feedback
Estimation error rate
Primary area: theory->learning_theory
TL;DR: Preference feedback can significantly enhance parameter estimation in continuous parametric distributions, achieving faster convergence rates and lower variance compared to sample-only methods.

Abstract: Recent advances in language modeling have underscored the role of preference feedback in enhancing model performance. This paper investigates the conditions under which preference feedback improves parameter estimation in classes of continuous parametric distributions. In our framework, the learner observes pairs of samples from an unknown distribution along with their relative preferences depending on the same unknown parameter. We show that preferences-based M-estimators achieve a better asymptotic variance than sample-only M-estimators, further improved by deterministic preferences. Leveraging the hard constraints revealed by deterministic preferences, we propose an estimator achieving an estimation error scaling of $\mathcal{O}(1/n)$---a significant improvement over the $\Theta(1/\sqrt{n})$ rate attainable with samples alone. Next, we establish a lower bound that matches this accelerated rate; up to problem-dependent constants. While the assumptions underpinning our analysis are restrictive, they are satisfied by notable cases such as Gaussian or Laplace distributions for preferences based on the log-probability reward.

Learning Parametric Distributions from Samples and Preferences
Mare Jourdan Gizem Yüce!
Nicolas Flammarion
Abstract
data-generating process (Ge et al., 2024).
Recent advances in language modeling have un- derscored the role of preference feedback in en- hancing model performance. This paper investi- gates the conditions under which preference feed- back improves parameter estimation in classes of continuous parametric distributions. In framework, the learner observes pairs of samples from an unknown distribution along with their relative preferences depending on the same un- known parameter. We show that preference- hasse M-estimators achieve a better asymptotic variance than sample-only M-estimators, further improved by deterministic preferences. Leveraging the hard constraints revealed by deterministic preferences, we propose an estimator achieving an estimation error scaling of O(1/n)=a significant improve- ment over the (1/ rate attainable with sam- ples alone. Next, we establish a lower bound that matches this accelerated rate; up to dimen- sion and problem-dependen constants. While the assumptions underpinning our analysis are restric- tive, they are satisfied by notable cases such as Gaussian or Laplace distributions for preferences based on the log-probability reward.
To understand the role of preference feedback, we focus on simpler yet illustrative problem: parameter estimation for parametric distributions and preferences. Specifically, the learner observes pairs of samples from unknown distribu- tion, along with preferences informed by the same parameter. For instance, preferences based on log-probabilities natu- rally link the preference and probability models, though other formulations are possible (Huang et al., 2024).
For continuous distributions, we uncover a significant statis- tical learning gap between preference-based and sample- only estimators. This paper primarily investigates this gap, taking the sample-only maximum likelihood estimator (MLE))optimal among unbiased estimators- as a base- line. The well-established theory of M-estimators (Van der Vaart, 2000) suggests that preference-basee M-estimators improve asymptotic variance under certain conditions. Yet, this improvement is modest: when samples are of similar quality, preference feedback approaches fair coin toss, providing minimal additional information. While reducing asymptotic variance is encouraging, it does not fully explain the substantial performance gains observed empirically in large-scale language models.
For deterministic preferences, we prove a more striking re- sult: preference- beaeeennee-beeddddddee estimators achieve statistically sig- nificant acceleration in parameter estimation. Specifically, we show that the estimation error scales as (l/nn) instead of the O(1 rate achieved by sample-only estimators. This acceleration is supported by a matching lower bound, up to dimension and
1. Introduction
Recent progress in language modeling has showcased the effectiveness of preference feedback for fine-tuning (Ziegler et al., 2019; Ouyang et al., 2022; Bai et al., 2022; Tou- vron et al., 2023; Dubey et al., 2024). Preference data- indicating relative quality between outperforms approaches using positive examples only like supervised fine-tuning (Ivison et al., 2024). This empiri- cal success suggests that preference feedback introduces new, complementary information beyond the observed data. Understanding how and why preferences provide this ad- vantage requires connecting the preference model to the
constants.
While this acceleration might sound surprising, the ((1/n) rate can already be observed in special case of sample- only parameter estimation. For instance, consider estimat- ing the location parameter of a uniform distribution on [0, 0 + 1] based solely on samples (Wainwright, 2019). The minimax rate for estimation error ((1/n). The optimal estimator achieving the accelerated rate is the minimum of uniform observations whose density is positive at 8. This improved rate arises from the accumulation of random vari- ables having positive density at a specific point through a minimum (or maximum) operator, in contrast to the slower aggregation inherent to averaging, Similarly, for determin- istic preferences with liglliieldoood rewards, we observe
School of Computer and Communication Sciences, EPFL, Lausanne, Vaud, Switzerland. Correspondence to: Mare Jourdan smare.jourdan @ellaan@effff ept..chh>
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Learning Parametric Distributions from Samples and Preferences
the true ordering between likelihoods. As it enforces hard cettraiintt-hrruuge minimum operator-on on the admis- sible parameters, our preference-based estimator achieves accelerated convergence.
method (LeCam, 1973), Fano's method (Fano, 1952), and Assouad's method (Assouad, 1983), and provide fundamen- tal limits on estimation accuracy (Tsybakov, 2009).
Learning parametric value/preference functions. In the tabular setting, learning from pairwise comparisons aligns with the ranking problem. The performance of MLE under the Bradley-Terry model (Bradley & Terry, 1952) and its extensions has been extensively studied (Hunter, 2004; Ne- gahhan et al., 2012; Hajek et al., 2014; Rajkumar & Agarwal, 2014; Shah et al., 2016; Shah & Wainwright, 2018; Mao et al., 2018). The continuous setting, where generalization beyond observed preferences required, has received less attention, except for linear utility functions (Zhu et al., 2023; Ge et al., 2024; Yao et al., 2025). Beyond analyzing the sample complexity of reward learning with MLE under the Bradley-Tene noise model, Zhu et al. (2023) study the per- formance of policies trained on the learned reward model. They show that while MLE may fail, a pessimistic variant can yield a policy with improved performance. Relaxing the noise assumption, Ge al. (2024) show that utility pa- rameters remain unidentifiable without strong modeling as- sumptions, even with noise-free query responses. However, they demonstrate that, in the active learning setting, utility can still be learned, even in the absence of noise. Their results highlight that the sampling distribution of observa- tions must be aligned with the utility function to achieve improved sample complexity. Yao al. (2025) leverages sparsity in the preference model and establish sharp estima- tion rates depending on the sparsity level. Finally, related estimation problems have also been studied in the contexts of dueling bandits and reinforcement learning (Faury et al., 2020; Saha et al., 2023).
To illustrate this acceleration, consider the standard nor- mal distribution with preferences based on log- pog--abbbilities. Let n. E N and [n] : (1. (1.....)) .n]. For each E [n], observe samples (X,; Xi) ~ (02, 12) along with their bog-likelihodd deterministic preference Z; := sYg X,). where S: := (X; + Y2)/2 is their average. The triplet (X, Y, Z,) imposes a hard constraint based on S; on the location of candidate estimators 0 that are consistent with this log- likelihoodd deterministic preference. Specifi- cally, they satisfy 0 < S, if S. > 0, and 0 2 S, otherwise. The set of feasible parameters satisfying all constraints is thus [maxxe[n) 9,<0 St, min, 9,20 Si]. Since the den- sity of (0, 1/2) is positive near zero, the length of this interval decreases ((1/n) with high probability.
1.1. Contributions
For continuous parametric probability distributions, we study the statistical learning gap between preference-based estimators and sample-only estimators.
First, we show that preference-basse M-estimators achieve a better asymptotic variance than sample-only M-estimators. The variance is further improved for deterministic preference.
Second, we introduce an estimator satisfying the con- straints revealed by the deterministic preferences, and prove an accelerated estimation error rate of (1//n). This constitutes a significant improvement over the ((1v rate achieved by M-cstimators.
Third, we provide a lower bound of ((1/n), matching our upper bound up to problem-specific constants.
Fine-tuning with preference data. Large language mod- els often go through post-training phase focusing mainly on learning from preference feedback (Lambert, 2024), to improve capabilities such as summarization, instruction fol- lowing, and reasoning. The standard approach, reinforce- ment learning from human feedback (RLHF) (Ziegler et al., 2019), trains reward model align with human prefer- ences and then optimizes the policy using reinforcement learning, typically with PPO (Schulman et al., 2017). RLHF follows three main steps: supervised fine-tuning, reward model training, and policy optimization. Another line of work has explored alternatives to PPO to simplify training. One such method, direct preference optimization (DPO), reformulates the reward function to learn a policy directly from preference data, avoiding explicit reward model. Other preference optimization objectives have also been proposed (Meng et al., 2024). Finally, while preference data has traditionally been gathered through human annotators, the learning paradigm has recently expanded to include self- play where the model critiques its own generations (Dubey
Our results are derived under general assumptions on the distributions and the preferences. While restrictive, they are satisfied by notable cases such as Gaussian or Laplace dis- tributions for preferences based on log-probabilities.
1.2. Related Work
Learning parametric distributions. Parametric estima- tion is a central approach in statistics, reducing inference about a distribution to the estimation of finite-dimensional parameter (Lehmann& & Casella, 2006; Wasserman, 2013). The maximum likelihood estimator (MLE) the most fun- damental method in this setting. Its asymptotic properties are well studied (Cramér, 1946; Ibragimov & Has" Minskii, 2013; Van der Vaart, 2000), while non-asymptotic guar- antees have been established in Birgé & Massart (1993) and Spokoiny (2012). Lower bounds in parametric es- timation rely on techniques such as Le Cam's two-point
2

---

Learning Parametric Distributions from Samples and Preferences
al., 2024; Huang et al., 2024).
they do provide gradient information to distinguish 0 from neighboring alternative parameter. Only the prefer- ences of samples in Gi (0*) can provide information on 0*, hence preference learning is meaningful if these samples are observed, i.e., P,, oa (91 (f*)) > 0 for all 0* E e
2. Problem Statement
Parameter estimation. Let @ S R be set of param- eters for a class of continuous probability distributions F over X C R4. Let Be == maxøge ||0]| be the bound on for the norm || || specific to F. Let Sk-1 be the unit sphere for this norm. Let pa 2 be the distribution of two independent observations of po-
P6*
Negative examples. The above condition is restrictive both on to and po, even when considering re log Po- For example, taking Py as the uniform distribution over [0, 0}. we have TP 02 (9) =0.
pa
Let 0* be an unknown parameter to estimate. Our sam- 82 We use two archetypal examples satisfying our assumptions. First, the class Fv of multivariate Gaussian distributions with known covariance E, where are the natural parameters with norm || || where ||>]|| =- Vx'Ex. Second, the class FLap. of Laplace distributions with known scale b, where ④ are the mean parameters with norm --
ples are drawn from $2 i.e., (X,Y) ~ Pg-
2.1. Sample-only MLE
In the absence of preference observations, a natural baseline is estimate 0* directly from the observations. Given (X,, Yi)
~ Pg- -0?", the sample-only (SO) MLE
Øoo E Earg min (00 n
with
s0 (0) == Σ KE(M)
log 82 (X,,
(SO MLE)
Preference feedback. Let lo X²2 - R be para- metric preference function. Given a parametric reward function rø, a reward- based preference function defined as (1 .) = re(x) - (v)). As a concrete example for our derivations, we consider preference based on the log-probability reward re log pg. Given observations (z,, v), the true preference of x over y is governed by (ign(Co(1,,))) v)) E (=1.0). In many settings, however, the observed preference Z can be stochastic due to noise or randomness in human feedback.
Asymptotic normality. Under enough regularity (Van der Vaart, 2000), SO MLE asymptotically normal, i.e.
vn((0% 0*) aa-too I(0g )())
where I(po) Epp V3 log is the Fisher information matrix of py and denote the convergence in distribution. Let > denote the Loewner order on p.s.d. matrices. By the Cramér-Rao bound (Rao, 1992), SO MLE has optimal asymptotic covariance among the class of unbiased sample- only estimators, i.c., all sample-only unbiased estimator with asymptotic variance V satisfy V > I(p
Conditioned on (X,Y) 22 we denote the p.d.f. of the law of the preference Z by (X,,(X, -), On X²2 x (+,, the p.d.f. of the law of (X,Y, is denoted as qe (z, y, 2) := 2 ())((o(zz, 2). Under determinis- tic feedback, the true preferences are observed:
While asymptotic guarantees provide insight into estima- tor behavior as - co, they do not capture performance in the relevant regime of moderate sample sizes. Modern statistics gives meaningful non-asymptotic concentration results on empirical estimators, e.g., for high- nigh-dimensional statistics (Vershynin, 2018; Wainwright, 2019).
haae (1 = sign(-))
(1)
Under stochastic feedback, noisy preferences E {+11 are observed based on the sigmoid link:
h 2) == : (z))) with (x)) :=)==(( :c)==(1 (2)
Regularity conditions. The asymptotic statistics litera- ture has devised weak regularity conditions under which asymptotic normality holds. "Classical conditions" assume stronger conditions, e.g., 0 + log Pe(:) is three times con- tinuously differentiable for every x E & and the integral of its third derivative converges uniformly for all (Van der Vaart, 2000, Chapter 5.6). Those "weak" or "classical" conditions ensure that integrals and derivatives can be ex- changed, and Taylor approximations around 6* are well controlled. Throughout this paper, we use "under enough regularity" to refer to these regularity conditions on both po and lg.
Informative preferences. A natural question is to see when preference Z ~ h(Ea- (X, -)) helps to estimate 6* compared to using samples YX ~ .22 only. Intuitively, given observations with null preference gradient, parameters close to 8* could have similar preferences. Therefore, those samples are not sufficient to discriminate between them. For that, let (6*) = {(x, y) E X²2 |Lo- (I, 0} (resp. G, (0*) (zx y) E Go( | En- (æ. v) > 0}) be the set of pairs with non-zero preference (resp. gradient) func- tion. For observations in ( 0, the preference is zero, hence uninformative. For observations in G1 (8+),, the pref- erence is locally independent of the parameter. Therefore,
For preferences based on the reward rg = log Pe. the regu- larity of po implies the one of the preference Eo due to the
3

---

Learning Parametric Distributions from Samples and Preferences
properties of the logarithm. Moreover, those regularity con- ditions are satisfied for numerous well-known distributions such as FNyE and FLap.d. When studying deterministic preferences, we introduce general geometric assumptions on po and Eg. Since these conditions are inherently more re- strictive, our goal is not to identify the weakest possible regu- larity assumptions under which our derivations hold.
For preferences based on the reward rg = log Pg. this con- dition holds for both Laplace and Gaussian distributions: A D A Lap(U.1) for Fiap, (Appendix G), and A
2 2 La)
SP 2//² for FN,E (Appendix F).
Thus, stochastic preferences can improve parameter estima- tion compared to sample-only estimators. However, non- asymptotic performance can differ, and in practice, the re- duction in asymptotic variance may be small, as we inves- tigate empirically in Section 6. Next, we examine whether M-estimators based on deterministic preferences can further improve upon their stochastic counterparts.
3. Preferencee-asee M-estimator
In this section, we investigate when preference- based esti- mators can improve upon sample-only only estimators. Given we de-
preference-labeled observations {(X,, Y., fine the stochastic preferences MLE (SP MLE) as
3.2. Deterministic Preferences
P E arg min L (0)
with
We now consider the setting where true preferences are ob- served, meaning that the preference labels Z, are determinis- tic. We observe (X,, Y,, Z))xe(m) ~ q8- haass where haet is defined in Equation (1). We use the same M-estimator as in the stochastic setting. @STP (0) E arg ming L (0). but now with deterministic preferences. To distinguish this setting, we introduce the notation det for the greference-based estimator under deterministic feedback.
0
T.SP(0) = 1. (0)
(0)=
log a (Zieo Vi))- MLE)
< [n]]
This objective extends SO MLE by adding a preference- based term: a binary classification loss using the logistic function log a (z)). When preferences are stochastic, this estimator corresponds to the MLE under a probabilis- tic preference model, justifying its name. Under sufficient regularity, M-estimators achieve asymptotic normality, so our goal is to obtain lower asymptotic covariance for SP MLE than for SO MLE. In addition, want show that SP MLE reaches lower asymptotic covariance for deter- ministic preferences than for stochastic preferences.
Consistency of det- Define the population-level objec- tive: (0) = a= [gog No (X, sign(Ee (X, Y)))].
Under enough regularity, PSP converges to a maximizer of M (0) (Van der Vaart, 2000, Chapter 5.2). However, un- like in the stochastic setting, 0* may not a maximizer of M since standard regularity conditions on po and Eg are insufficient. A sufficient condition for consistency is
3.1. Stochastic Preferences
Under stochastic feedback, we given noisy preference @n observations (X,, Y,, Zt)sejn] ~ A where hato is de- fined in Equation (2). SP MLE is a specific instance of M-estimator. Under enough regularity (Van der Vaart, 2000, Chapter 5.5), SP MIE is asymptotically normal, i.e.,
E ez (sign (Co. (-((- V Cao] = Ok:
(3)
which holds for FNYE (Appendix F) and Fran, (Ap- pendix G) when using reward re = log Po-
Vn(0sp - 0²) ma- NO2,, I(go- I(go=(haa)))) !)),
Asymptotic variance of SP If Equation (3) holds, then under additional regularity conditions (Van der Vaart, 2000, Chapter 5.3) SPdet det is asymptotically normal with covariance V given by the following lemma.
where I(go,hau)) == Egg |- V log qu.b.a.] denotes the Fisher information matrix of qo, Awn. By the Cramér- Rao bound (Rao, 1992), this variance is optimal among unbiased estimators that rely on stochastic preferences. Lemma 3.1 compares its efficiency to the sample-only MLE.
Lemma 3.2. Ler Hol : E E,00 (ug- [.
A aP : ==, (200((22(( - 1)o((116-1)) o- La- V [-_] and RT == E,09 luo- (Mp- + MM-)1 where ug- == 18g 002 (Vor
Lemma 3.1. Let - == Ex
Then, I(go- [haa)
I( I(p @2) + ASP. The p.s.d. matrix A
and Mg. == -Va log
is definite if P 22 (1Ku, Vg. La-) > 0) > 0 for all u E Sa-1-
Then, we have Vol == VI V2.0- V1,00- where
Lemma 3.1 shows that I(ga= ) > I(pg²) and exhibits a condition under which 8SP is asymptotically better than 0 meaning that incorporating preference data improves asymptotic efficiency. The condition in Lemma 3.1 ensures that Vg- lg. spans all directions with some probability, mak- ing the preference-be based estimator asymptotically superior to the sample-only MIEE.
V1.9 = I(go- Ana) - HSPaa and V2.8 = II(go Nnn)) A &P R
SP amt If 1P,,2(((0 o (u, Vo- > 0) > 0 for
all u E Sk-1, the p.s.d. matrix Ag. SPae is definite.
Equation (3) and ySSPass X I(ga=rhe..)- depend on the g=- geometry of pg- and Eg.. We verify that these conditions hold for FN,, (Appendix F) and Frap. b (Appendix G) when
4

---

Learning Parametric Distributions from Samples and Preferences
using reward rg. logpe- Po. For Laplace distribution, we have Hg- del
Lemma 4.1. For all n E N and almost surely, we have,
Rgg. O and Ag- des
4 A SPa For
La)(
for FNEE, 10 0 1|
S
10 0 01
Gaussian distributions, we have H.grues 0dxd= 0-
SPaa
// A SPas E1/2 and RgPPa det with SS Fa) = Odxd-
1/ C (Oa.la))
des 1//
For stochastic preferences, minimizing the 0 - 1 loss is generally NP-hard, requiring a convex surrogate like the logistic function. However, for deterministic preferences, computing C₁ is more tractable. If + Eg is affine, then Cn is convex polytope, defined by most n. half-space constraints. For Gaussian-based preferences, i.c., те log pe and FN we have Z (X,, Yi) 2 0 if and only if Z, - Yi, 0 - --1 (X, + Y.)/2) 2 0.
((aadla))
Thus, deterministic preferences improve parameter estima- tion compared to stochastic preferences.
In conclusion, preference-based M-estimators provide asymptotic improvements in estimation efficiency. Next, we explore whether estimators beyond the M-estimation framework can achieve further gains, potentially exceeding the asymptotic normality limitations.
Consistency of 0 1 loss minimization. Define the disagreement probability between 0 and 0 as m(0) == {(x,)) X²
TP P,ga (D(0",0) where D fg (e, y) < 0} is the set of observations where 0 and H* assign informative yet opposite preferences.
4. Beyond M-estimators
While computationally efficient, the Set estimator does not fully leverage the constraints imposed by deterministic preferences. Unlike in the stochastic setting, determinis- tie preferences provide separability: there exist parameters that classify training examples perfectly, including 0* it- self. A key limitation of SP det is that, like standard logistic regression, it minimizes convex surrogate loss (negative log- sigliikeiioodd. This approach can lead to misclassification of training examples.' This limitation suggests an opportu- nity to directly minimize the 0-1 loss", potentially achieving faster rates of convergence.
Under enough regularity (Van der Vaart, 2000, Chapter 5.2).
@ and @DP converge in C(0*) = {0 E θ m (0)
R 0}. which is the non-empty set of minimizers of (0) as m(0) 2 0 m(0*). We note the set C(8*) contains 0*, but possibly others. To ensure consistency (AAE, ,0PP + 0*), we impose the following identifiability assumption that guarantees C(0*) = {0*).
Assumption 4.2 (Identifiabilityy. For all 0 00", m(0) > O.
When re log pe. it holds for both Gaussian (Fv,z) (Ap-
pendix F) and Laplace (Fi.ap.) (Appendix G) cases.
0-1 loss minimization. Given (X,, Y1, Z()ee(n)) x][n] go ' we consider the set Cn of parameters that mini- mize the empirical 0 - loss, i.e.,
Fast estimation rate. Once consistency is established, the next goal is to analyze the convergence rate of the estimation errors || - 0* and ||0Dp - 0* Since these are not M- estimators, they are not necessarily limited to the typical parametric rate 22(1/
Cn ;= arg min #EA
Σ 11 (Z,8a(X,, <) EE|n]
(4)
= e ViE [n], 2,6o(x, Yi) 2 20}
Theorem 4.3 states our main result for Laplace and Gaussian distributions when using log probability rewards, i.c., a high- probability accelerated rate in (11/n).
which is non-empty as 8* E C.. Parameters 0 E C,, per- fectly classify all training examples. Any estimator BAE E C, is referred to as an arbitrary estimator (AE).
Theorem 4.3. Let 5 E 10, For FLap,1 and FN we have, for all n 2 OOlog( /6)). with probability 1 - 8,
Alternatively, we constrain MLE to this feasible set, defining the deterministic preferences MLE (DP MLE), ie..
v0, E Cn, na@a, 00 = O (log(1/6))
@PP arg min (L, (0) 0 E Cn
(DPMLE)
For FNI with d > 1, there exists positive As =d0+00 O(vi) such that, for all n 2 Ō(log(1/8)), with probabiliy 1-6. ö.
if 05 & Cn. and @rr_ := T otherwise. This estimator minimizes the negative log- likeliieehood of the samples while ensuring perfect preference classification. For Gaussian with rg= log pg. 0.r estimates 0* better than @ for all n, i.e., DP MLE dominates SO MLE statistically.
VO, E Cn, nl10, - 0- S O log(1/)) logn).
Theorem 4.3 is a direct comollary of our main result, showing that maxøee, |10-0- 0+ = ((1/n) (see Theorem 4.8 below). It directly guarantees faster convergence rates for both A and 0.P Theorem 4.8 holds under general geometric con- ditions on pe and le that we introduce with intuitions, while sketching the proof in Section 4.1.
'Frr binary classification with separable data, logistic regres- sion converges in direction toward separating hyperplane. FFor non-separable data, the minimization of the 0-1 classi- fication loss can he NP-hard even for the simple class of linear classifiers, e.g., Feldman et al. (2012).
5

---

Learning Parametric Distributions from Samples and Preferences
Negative examples. Assumption 4.2 is restrictive both to and po, even when considering rg log po- For example, when all the distributions in F agree on their preferences, Sign(f((z v)) is independent of 6. Therefore, we have mn() = 0 for all * 0", since lo (x. (o. y) 2 O. Such cases include scenarios where po (2) is a monotonic function, e.g., the exponential distribution and the Pareto distribution with a known location, well as the Laplace distribution with known location. This motivates later assumptions on the directionality of Va- Ca. for observed samples.
Directional analysis and informative constraints. To quantify the geometry of C. relative to 0', we analyze deviations along directions E Sa-1. Define the set of informative samples along direction u:
G1 (0", ,u) == ((x,)) v) lo (1,, (u, V (I, < <0}.
This set contains observations whose preferences give infor- mation along the direction u. Assuming that preferences are informative along all directions, we prevent degenerate cases where some directions lack preference information.
Assumption 4.5 (Informative Preferences). For all u E
Sk-1, IP oz (91 (0','))) u)) > 0.
Link to iterative human preference alignment. Many human preference alignment methods build on the Bradley- Terry model for preference, based on rewards. Direct align- ment algorithms use variants of the log- log-likelihood to define the implicit reward of policy (Rafailov et al., 2023). Choos- ing fo(2;, v) = log po (2:) log Po (v) coincides with the op- timal policy for maximum entropy RL (Swamy et al., 2025). When leveraging offline preference data, the assumption (X,Y) ~ pg. unrealistic, as Eg. collected from a fixed data set of pairs of observations. However, "online" pref- erence data has become popular paradigm in the training of recent LLMs. Those iterative alignment procedures rely on the preference data from an earlier model (Dubey et al., 2024). At stage N, the model Рay is trained based on the preference data for generations by the previous model, i.e., (X ~ Pow-1- Under the realizability assumption and without mode collapse, this self-refinement paradigm should converge towards the true model pa.. Our setting charac- terizes the limiting behavior of this iterative process, i.e., preference based on Eg. for observations from pas. Nonethe- less, we do not claim the direct applicability of DP MLE for realistic L.M training.
Deviation bound via minimum informative sample. De- u as the maximal deviation from 0* within C, along
fine Ru, the direction u, i.e.,
RA max 2010 0+ +eu
eu E C..)
We define the scaling factor
Eg (I, y) V(z, E CO", V (o,, == {u,, V En- (xr, y))
The value Vo-,a(X;; Yi) quantifies the amount of infor- mation in the preference between X; and Y; to discrim- inate O from other parameters on the half-line directed (X,,Y) by u. The lower (X,,, Yi) is, the more discrimina- tive is the preference between X, and Y. Since (æ.y) Gi (0", u) \ D(O*, 0* + +uu) if and only if Vo-,a(z, (2,,)) 2 s, we obtain
R S min (Vo v (X,, VD) I (X,,)) E (0",0" KE(p]
Therefore, the maximal deviation Rn,u is upper bounded by the minimum of positive random variables. It remains to upper bound the resulting value of this minimum with high probability and conclude provided some regularities hold, e.g., positive density at zero. By analyzing the distribution of Vor we derive the following probabilistic bound.
4.1. Upper Bound on the Estimation Error We establish a high-probabiliy, upper bound on the esti- mation error maxøec. |10 - 0*Il in the general case. This requires grasping the geometry of C, relative to l*
Linearized feasibility set. Since C, is defined by nonlin- ear preference constraint, analyzing its geometry is chal- lenging, and we thus consider linearized approximation of it. We define the linearized constraint set as
Lemma 4.6. Suppose Assumption 4.5 hold. For all u E Sk-1, with probability 1 88
Rn S F (min log(1/))/())
č, 10 E θ Vi E [n], (X,,Y) Y2) $ 00),
with For (e) == TP Pot
(Vor,u E (0, c.d.f. of Vor.
where D(00,0) 0) = ((x,8) E X² | la- (a, 2²
Eg (x,y) - 0r, (x, < 0). This set replaces to with its first-order Taylor expansion around 8*, neglecting higher-order terms. A key assumption is that the true con- straints are at least as strong the linearized ones. This ensures C, C C,,, allowing us to control C, via C,.. Assumption 4.4 (Linearization validity). For all 0 + 0*, D(0".0) C D(0", 8).
Since maxsec,, 110 - 0- < max Rn,.. Lemma 4.6 shows that the estimation error can be controlled by the behavior of Fg.l, around zero, where Fg.1, (0) 0.
Regularity assumption. To control F near zero, the density Fy should be positive near zero, and we assume control on (Fo-1)""
6

---

Learning Parametric Distributions from Samples and Preferences
Assumption 4.7 (Positive density at zero and regularity of inverse c.d.f.), For all Sk-1, Fb-,. (0) E (0, +000 and there exists (zar.u au Mar,a .a) E (0, 1) x R+ such that sup ... [ (x)) I MM
Extended discussions. In Appendix B, we discuss how to verify weaken our assumptions (Appendix B.1), the sources of misspecification (Appendix B.2) and reward models than log- ligeeiid (Appendix B.3).
Using this assumption and (F))) = 1/F2- (0), the first-order Taylor expansion with remainder yields
5. Lower Bound for Deterministic Feedback
Vo E [0, . |Fo (x) x/F6- .. (0) < Mor,u>²// /2.
In this section, we show that the rate (1//n) is minimax optimal (up to a logarithmic factor) by deriving a match- ing lower bound. The standard approach to minimax lower bounds in estimation relies on Fano-type inequalities and hypothesis testing reductions. However, due to Assump- tion 4.2, the Kullback-Leible divergence and x² distance be- tween qa- and go are infinite for 0 # 8", mmaking these tools ineffective. Instead, we use Assouad's ' Lemma (Tsybakov, 2009), which provides lower bounds via the total variation distance (defined as TVPP, Q) == |PP Qll, for distribu- tions IP and @). Since TV is not bel-bbe for product distributions, we use the squared Hellinger distance, defined as (P, Q) = I - VQl2 which satisfies
This argument leads to our main theorem, directly for l = and using a covering argument for k >>> 1. Theorem 4.8. Suppose Assumptions 4.2, 4.4, 4.5 and 4.7 hold. Let ô E (0,1). Let ? > 0 and N(r) be the n-covering number of θ for the norm || -1.- Ler Ap. milless (0), Bg milneSs zu
and Co- max.es.-- w/2. When k = 1, for all 2 BB- log(2/6),
Agr log(2/8) + Co. n²
max 110 -0-11 $ REC,.
1og(2/8)2
n
with probability - 5. When k > 1, for all n N
Bg. ())()())/)), with probability 1 8,
TV(PS", Q (@n) S V 2H (Pon, Q S V (P, Q).
Ag. log
N(7)
Co. ((7)
max ||0 10 S BEC,
log
For further analytical convenience, we also employ the Bhat- tacharyya coefficient, BC(P,, Q) == V 1. which re- lated to the Hellinger distance by (P, = 1 BC((P Q). Since qo-Qo Qo is zero for disagreeing preferences, we define the restricted BC as
8
n²
When 1, the choice of the optimal parameter y depends on the norm || ||. Since θ is bounded by Be, N(7) is upper bounded by the covering of the ball having diameter Be. As an example, let N2(() be the e-covering number of the unit ball in R for the Euclidean norm. Then, it is known that log N2(e) as log(e"))/E2 if & Z 1/VK and log (e) se k log से if S 1/Vk. Therefore, optimiz- ing over yields an upper bound on maxsec, 112 seal- ing as (AA k/n) when n 2 Ag. Ag.k3/2, and ((Ag- (((o)//n)//) otherwise, where Ō(-) hides logarithmic terms. For large sample size compared to the dimension, i.e., n Z Ag. k3/2,
(C 0) = E pg
[1 (D(0,))UDo(0,0))
@2 02 v Po /p
where (o,0, Go (0) is the set where the preferences zero for exactly one parameter.
Lemma 5.1 decomposes the Hellinger distance between two distributions over the preference triplets into the Hellinger distance between sample-only distributions and the disagree- ment restricted Bhattacharyya coefficient.
we recover a rate of ((1/n).
In conclusion, we have derived generic assumptions under which the rate of decay of the estimation error of ΦAE and @DP is in ((1/n). This is significant improvement com- pared to the asymptotic normality of the SPadet estimator that implies a rate of (n).
Lemma 5.1. (gg. qo) = BC(Ø,0) 0) + (1 PP
82 82 for
all f, 0 E O.
As H (2 02 .P 20 < 2H po), deriving a lower bound Pa) and BC(Ø, 0), hence, we impose the following assumption.
requires controlling H² Po)
Positive examples. While Assumptions 4.4, 4.5 and 4.7 are restrictive, they hold for FN (Appendix F) and FLap.b (Appendix G) when using reward ro- = log po- This yields Theorem 4.3. We have Ag- 2b. Bg- = 8 and Co- ()(a3)
Assumption 5.2. There exists positive constants cı 02 in- dependent of & and a dimension and problem-dependent scaling function (x)(k) such that for all 0, E O,
16b for FLap.b, and Ag- "I 1)/2) +o0 O(vd) for FAYD, hence a rate in ((d/22/)) /n) when n >> d.
22) S BC (C(0,0) H) 2 a5(k)
C1
0 0 + c2 10 0
(k)
'E.g., using Varthhmmyy for the lower bound (Gilbert, 1952) and Maurey's 's empirical method for the upper bound.
Theorem 5.3 bounds the minimax estimation error.
7

---

Learning Parametric Distributions from Samples and Preferences
10 0.5
10
10 1.0
10
SO
0
50
10
WE
RU
AE
DP
10
SP (sto)
SP (sto) (det)
SP (det)
10
10²
10
10
10
10
10 4.0
n
n
Figure 1. Estimation emrors for NO La) where 0 ~ ( for (a) d with Nans 10a and (b) 20 with Na 10
Theorem 5.3. Let Rmax == inf supa ce Egg- |1|0 Suppose Assumption 5.2 holds. Then,
I1]-
for Gaussian distributions, and defer to Appendix H.1 for experiments on Laplace and Rayleigh distributions. In par- ticular, consider uniformly drawn mean parameter 0+ [[1, 2) and the isotropic covariance Σ La. For sample size n E [[V with Maxx 10*, we compute the estimation errors 1|0, 0- We repeat this process for Nuns different instances and for various choices of d.
(k(kk
k n2 )
Rmax
Ω
min
This result confirms that the O(1/n) rate is minimax optimal up to logarithmic factors. The scaling (k) comes from BC( 0) (Assumption 5.2), yet it is challenging to link a (k) with Ag. without further assumptions.
For FN.Ja (Appendix F), the M-estimators can be imple- mented 030
1 Eie(n
(Xi+ (.),,.,
2n
SSP arg min 110 8
log (Z,e((())
Positive examples. While Assumption 5.2 is restrictive, even when using rewards re = log it holds for FLap.b (Appendix G), i.e.
n iE[mi
where (X,, V.) = (X; Y, 0- (X; Y2) Then, the estimators based on C. = 10 vi E [n]. Z,eo(X, Y,) > 0} are @DP arg mingec, |00 8 1| and arbitrary estimator و.B E C.. As C., is an interval for = 1, we use the randomized uniform (RU) estimator, i.e., ŌRU U (C,). We also consider the worst-case estimator (WE). defined as @WE == arg max ||0 0 ||I While it is not valid estimator due to its 0* dependency. it serves as a proxy for the worst estimation error in C.,-
BC = -00 -01((2)) + 0(10- - -13),
as well as for FN (Appendix F), i.e.,
BC( 0) 2 -11 Fe. (110* 0||x),
with Fe. (e) < A/A and (x) Agr.
Dimensionality gap. While the lower bound in Theo- rem 5.3 scales ((ar(k)vk/n)) for 2 (x()),, the upper bound in Theorem 4.8 scales O(Ag. k/n) for > hg-k3/2 Even for the simple case of Gaussian distri- butions where Ag. = (d), there is a dimensionality gap. Closing this gap is an important direction for future work. Improvements might come from tighter analysis, e.g.. both for the upper and lower bounds, or the derivation of better estimators based deterministic preferences.
Dependency sample size. Figure ((a) confirms pirically the difference in estimation rate between the M- estimators (SO MLE and SP MLE) obtaining ((1/ and our estimators based on C, accieviigg (11nn).
However, Figure ((b) also reveals that the performance of A and WE deteriorates quickly at small sample sizes when the dimension increases. In contrast, DP MLE consistently outperforms all the other estimators, including SO MLE as theoretically shown in Lemma 4.1.
6. Experiments
While Sed outperforms SO MLE, Figure also reveals that SP performs worse than SO MLE for finite sample size. Therefore, only an M-estimator based on deterministic pref- erences improves on sample-only M-estimators empirically.
In this section, we compare the empirical performance of the different estimators introduced this paper. For prefer- ences based on rg = log Po. we conduct a set of experiments
8

---

Learning Parametric Distributions from Samples and Preferences
7. Perspectives
0.25
WE
This work investigates the role preference feedback in pa- rameter estimation for continuous parametric distributions. We establish conditions under which preference-basee es- timators outperform sample-only methods. For stochastic preferences, the preference-basee MLE achieves lower asymptotic variance than its sample-only counterpart. For deterministic preferences, we demonstrate that preference- based estimators can significantly accelerate parameter es- timation, achieving an improved ((1/n) convergence rate compared the ((1/v)) rate of M-estimators. Our lower bound analysis further confirms that this acceleration is min- imax optimal up to cemeensionddeenndee constants.
AE
DP
00.0
SP (sto)
SP (det)
0.15
0.10
0.05
0.00
0
20
40
60
80
d
While our results provide solid theoretical foundation, several open questions remain. A finer analysis of beyond- M-estimators and their constraint set geometry would allow to better quantify the properties of DP MLE, and provide in- sights for designing improved estimators that better leverage deterministic preferences. Additionally, exploring alterna- tive preference functions beyond the pog-prbbability gap could extend the applicability of results.
Figure 2. Estimation errors function of d with (00 Iu)
where 0 ~ ((((11)))) for n = 10 and Nems = 10a.
This further highlights the weakness of asymptotic results compared to non-asymptotic guarantees.
While Figure ((a) suggests that RU and WE perform par with DP MLE, Figures ((b) and 2 highlight that DP MLE outperforms WE and AE for larger dimensions, where the gap increases when d nonnegligible compared to n. We conjecture that RU suffers from the same limitation as AE for larger d. As empirical evidence, we study other estimators that disentangle the effect of RU's randomness versus its mean behavior, see Appendix H.2.
Finally, key challenge for future work is quantify the benefits of preference-based estimation for discrete distri- butions. For distributions with small support, preference feedback may only localize the unknown parameter within subset of the simplex, leading to diminishing information gains as the sample size increases. However, understanding how preference- based estimators perform in finite-sample settings, particularly high-dimensional problems, remains interesting open problem. Addressing these questions could provide further insights into the role of preferences in machine learning and statistical estimation.
Dependency on dimension. Figure strengthens the aforementioned empirical observations. For fixed sample size and increasing dimension, DP MLE is the only estima- tor obtaining the best-of-both world estimation error rate, i.e. (min (d82 /n, v (d/n)).
A Acknowledgements
We thank Jaouad Mourtada for insightful early discussions on the univariate Gaussian case. This work was supported by the Swiss National Science Foundation (grant number 212111) and by an unrestricted gift from Google.
Covariance gap. We show that the covariance gap between SP MLE and SO MLE is relative ~ (0.16, 0.08) and (0.17, 0.08, 0.10). More- d4) and SPan 1a) are close to 0 is decreasing in (see Figure in
mild: (ASP Ap
A
Lap(o.1)
(ASP (o. A1(0,)) N(0..)
SPatt
Impact Statement
over, SP (,) A SPau
This paper presents work whose goal is to advance the field of Machine Learning. There many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
azla where ag
Appendix F). In addition having small empirical gap for moderate value of n, the asymptotic gaps between SO MLE and SP MLE are mild.
References
Supplementary experiments. Following the approach of Tang et al. (2024a), we compare estimators using other convex surrogates of the 0-1 loss (Appendix H.3): they all perform similarly. For the logistic loss, we showcase the "mild" impact of normalization and regularization (Ap- pendix H.4).
Assouad, P. Deux remarques sur l' I'estimation. R. Acad. Sei. Paris Sér. Math., 296(2));1021-0224 1983. ISSN 0249-6291.
Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M.,

---

Learning Parametric Distributions from Samples and Preferences
Valko, M., and Calandriello, D. A general theoretical paradigm understand learning from human preferences. In International Conference Artificial Intelligence and PMLR, 2024.
Gorbatovski, A., Shaposhnikov, B., Sinii, V., Malakhov, A., and Gavrilov, D. The differences between di- rect alignment algorithms are blur. arXiv preprint arXiv:2502.01237, 2025.
Statistics, pp.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J.,
Hajek, B.. Oh, S., and Xu, J. Minimax-optimal inference from partial rankings. Advances in Neural Information Processing Systems, 27, 2014.
Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C.,, al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.0801 2022.
Hong, J., Lee, N., and Thorne, J. Orpo: Monolithic prefer- ence optimization without reference model. In Proceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 11170-11189, 2024.
Bezanson, J., Edelman, A., Karpinski, S., and Shah, V. B. Julia: fresh approach to numerical comput- ing. SIAM Review, 59(1))65-98, 2017. doi: 10.1137/ 141000671. URL https:/ //pbb///pppbs../ siam. org/
Huang, A., Block, A., Foster, D. J., Rohatgi, D., Zhang, C., Simchowitz, M., Ash, J. T., and Krishnamurthy, A. Self-improvement in language models: The sharpening mechanism. In NeurIPS 2024 Workshop on Mathematics of Modern Machine Learning, 2024.
Birgé, L. and Massart, Rates of convergence for mini- mum contrast estimators. Probability Theory and Related Fields, 97:113-155 1993.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-344 1952.
Huangfu, Q. and Hall, J. Parallelizing the dual revised simplex method. Mathematical Programming Computa- tion, 10(1):1 142, 2018.
Cramér, H. Mathematical Methods of Statistics, volume vol. 9 of Princeton Mathematical Series. Princeton University Press, Princeton, NJ, 1946.
Hunter, D. Mm algorithms for generalized bradley-tery models. The annals of statistics, 32(1 ((11)))34440)))) 2004.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21788, 2024.
Ibragimov, A. and Has' Minskii, Z. Statistical estima- tion: asymptotic theory, volume 16. Springer Science & Business Media, 2013.
Ivison, H., Wang, Y., Liu, J., Wu, Z., Pyatkin, V., Lambert, N., Smith, N. A., Choi, Y., and Hajishirzi, H. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Dvoretzky, A., Kiefer, J., and Wolfowitz, J. Asymptotic minimax character of the sample distribution function and of the classical multinomial estimator. The Annals of Mathematical Statistics, 27(3):642-669, 1956.
Fano, R. Class notes for transmission of information. In Course 6.574. MIT Cambridge, MA, 1952.
Lambert, N. Reinforcement from Human Feedback. Online, 2024. URL /ttpttts::///tltn fbook. com.
Faury, L.,, Abeille, M., Calauzènes, C., and Fercoq, O. Im- proved optimistic algorithms for logistic bandits. In In- ternational Conference on Machine Learning, pp. 3052- 3060. PMLR, 2020.
LeCam, L. Convergence of estimates under dimensionality restrictions. The Annals of Statistics, pp. 38-53, 1973.
Feldman, V., Guruswami, V., Raghavendra, P., and Wu, Y.
Lehmann, E. L. and Casella, G. Theory of point estimation. Springer Science & Business Media, 2006.
Agnostic learning of monomials by halfspaces is hard. SIAM Journal on Computing, 41(6):1558-1500 2012.
Lubin, M., Dowson, O., Garcia, J. D.. Huchette. J., Legat, B., and Vielma, J. P. Jump 1.0: Recent improvements to a modeling language for mathematical optimization. Mathematical Programming Computation, 15(3):581-
doi: 10. 10.1137/120865094.
Ge, I.., Juba, B., and Vorobeychik, Y. Learning linear util- ity functions from pairwise comparison queries. arXiv preprint ar XrXiv:2405.02612. 2024.
589, 2023.
Gilbert, E. N. A comparison of signalling alphabets. The Bell System Technical Journal, 31(3):504 522, 1952. doi: 10. 1001/0 1538- 7305. 1952.661399..x.
Mao, C., Weed, J., and Rigollet, Minimax rates and efficient algorithms for noisy sorting. In Algorithmic Learning Theory, pp. 821-847. PMLR, 2018.
10

---

Learning Parametric Distributions from Samples and Preferences
mathlib Community, T. The lean mathematical library. In Proceedings of the 9th ACM SIGPLAN International Con- ference on Certified Programs and Proofs, POPL '20. ACM, January 2020. doi: 10. 10.1144//37728853337733
Shah, N. B., Balakrishnan, S., Bradley, J., Parekh, A., Ramch, K., Wainwright, M. J., al. Estimation from pairwise comparisons: Sharp minimax bounds with topol- ogy dependence. Journal of Machine Learning Research, 17(58):1-47, 2016.
URL http://ddd //x. doi. www.doo..ooooooooooooooooooo..o. 1144 00011111533322 3373824.
Spokoiny, V. Parametric estimation. Finite sample theory. The Annals of Statistics, 40(6):2877 - 2909, 2012.
Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with a reference- reward. Advances in Neural Information Processing Systems, 37:124198- 124235, 2024.
Swamy, G., Dann, C., Kidambi, R., Wu, S., and Agarwal, A. A minimaximalist approach to reinforcement learn- ing from human feedback. In Forty-first International Conference on Machine Learning, 2024.
Munos, R., Valko, M., Calandriello, D., Azar, M. G., Row- land, M., Guo, Z. D., Tang, Y., Geist, M., Mesnard, T., Fiegel, C., et al. Nash learning from human feedback. In International Conference Machine Learning, pp. 36743-36768, PMLR, 2024.
Swamy, G., Choudhury, S., Sun, W., Wu, Z. S., and Bag- nell, J. A. All roads lead to likelihood: The value of reinforcement learning in fine-tuning. arXiv preprint arXiv:2503.01067,200 2025.
Negahban, S., Oh, S., and Shah, D. Iterative ranking from pair-wise comparisons. Advances in neural information processing systems, 25, 2012.
Tang, Y., Guo, Z. D., Zheng, Z.., Calandriello, D., Munos, R., Rowland, M., Richemond, P. H., Valko, M., Avila Pires, B., and Piot, B. Generalized preference optimization: A unified approach to offline alignment. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett,
Ouyang, L.,, Wu, J., Jiang, X., Almeida, D., Wainwright, C.,, Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-2777 277444 2022.
J., and Berkenkamp, (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 47725 477742. PMLR, 21-27 Jul 2024a.
Rafailov, R., Sharma, A., Mitchell, E., Manning, D., Er-
Tang, Y., Guo, 7. D., Zheng, Z.,, Calandriello, D., Munos, R., Rowland, M., Richemond, P. H., Valko, M., Pires, B. Á., and Piot, B. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05770 2024b.
mon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. Ad- vances in Neural Information Processing Systems, 36: 53728-53741, 2023.
The Sage Developers. SageMath, the Sage Math- ematics Software System (Version 9.7), 2022.
Rajkumar, A. and Agarwal, S. A statistical convergence per- spective of algorithms for rank aggregation from pairwise data. In International conference on machine learning, pp. 126. PMLR, 2014.
https: /www://wwwwwwww sagemath. aagemath.orgg.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama Open foundation and fine- tuned chat models. arXiv preprint arXi::2307009288 2023.
Rao, C. R. Information and the accuracy attainable in the estimation of statistical parameters. In Breakthroughs in Statisties: Foundations and basic theory, pp. 235-247.
Springer, 1992.
Tsybakov, A. B. Nonparametric estimators. Introduction to Nonparametrie Estimation, pp. 1-76, 2009.
Saha, A., Pacchiano, A., and Lee, Dueling rl: Reinforce- ment learning with trajectory preferences. In Interna- tional Conference on Artificial Intelligence and Statistics, pp. 6263-6289. PMLR, 2023.
Van der Vaart, A. W. Asymptotic statistics. volume 3. Cam-
bridge university press, 2000.
Vershynin, R. High-dimensional probability: An introduc- tion with applications in data science, volume 47. Cam- bridge university press, 2018
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint 17X1v:1707.063477 2017.
Wächter, A. and Biegler, L. T. On the implementation of an interior-pon point filter line-search algorithm for large-scale nonlinear programming. Mathematical programming. 106:25-57, 2006.
Shah, N. B. and Wainwright, M. J. Simple, robust and optimal ranking from pairwise comparisons. Journal of machine learning research, 181199):138, 2018

---

Learning Parametric Distributions from Samples and Preferences
Wainwright, M. J. High-Dimensional Statisties: A Non- Asymptotic Viewpoint, volume 48. Cambridge University Press, 2019.
Wang, R., Sun, J., Hua, S., and Fang, Q. Asft: Aligned supervised fine-tuning through absolute likelihood. arXiv preprint arXiv:2409. 10571, 2024.
Wasserman, L. All of statistics: a concise course in sta- tistical inference. Springer Science Business Media, 2013.
Yao, Y., He, L.. and Gastpar, M. Leveraging sparsity for sample-efficient preference learning: theoretical per- spective. arXiv preprint wrXiv:2501.1822, 2025.
Zhu, B., Jordan, M., and Jiao, J. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In International Conference on Machine Learning, pp. 43037-43067. PMLR, 2023.
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv: 1909.00999, 2019.
12

---

Learning Parametric Distributions from Samples and Preferences
A. Outline
The appendices are organized as follows:
In Appendix B, we provide detailed discussions on our assumptions, the sources of misspecification and other reward models.
In Appendix C, we prove the general results presented in Section 3 such as Lemma 3.1.
In Appendix D, we focus on Section and detail the proof' of Lemma 4.6 and Theorem 4.8
In Appendix E, we prove the results presented in Section 5.
In Appendix F, for Fxyx and preferences based on ro = log po. we prove the assumptions introduced in this paper.
In Appendix G, for Fiap, and preferences based F3 log pe, we prove all the assumptions introduced in this paper.
In Appendix H, provide supplementary experiments support our theoretical findings.
B. Extended Discussions
We provide detailed discussions on how to verify or weaken our assumptions (Appendix B.1), the sources of misspecification (Appendix B.2) and other reward models than log- ligllikel (Appendix B.3).
B.1. Verifying or Weakening our Assumptions Since our assumptions are restrictive, it is natural to wonder how they can be verified or weakened.
Verifying our assumptions. Even a closed-form definition of pg and lg is given, our assumptions are challenging to verify, hence we suggest using a formal verifier (e.g., Lean (mathlib Community, 2020)) or software (e.g.. SageMath (The Sage Developers, 2022)). Empirically, they can be confirmed or rejected by sampling from p Assumption 4.4 is rejected by exhibiting (X,, Yi) E D(O", D(0", Assumptions 4.2 and 4.5 are confirmed by finding (X;, Yi) E D(0", 0) and (X,YY,) E ( u). Those tests' sampling complexity scales as the inverse event's probability. Using Dvoretzky- Kiiferr-Woofowiit inequality (Dvoretzky al., 1956), Far,a can be estimated to verify that Assumption 4.7 holds.
Restrictive assumptions. When studying DP MLE only, we conjecture that the "global" Assumptions 4.2 and 4.4 be weakened local versions. Using time-uniform concentration results, regions (Rn), around SO MLE that contains l* for all time with high probability. Then, we modify DP MLE to be constrained on R,, n,, For n large enough and with high probability, R, nC, will be included in a local neighborhood of 0" under which the "local" Assumptions 4.2 and 4.4 are satisfied. Given that Assumption 4.4 is based on "ignoring" the reminder term in a first-order Taylor expansion, assuming a local version is a significantly weaker requirement.
build a sequence of shrinking confidence
B.2. Sources of Misspecification
There are several possible sources of misspecification not taken into account by our current analysis.
Preference model. The Bradley-Terry model that uses reward-based preferences has limited expressivity as it doesn't t allow for intransitive preferences. Even when individuals exhibit transitive preferences, their averaged preferences can be intransitive due disagreements, see Munos al. (2024) or Swamy al. (2024).
Parameter space. When 0* O, the deterministic preferences might not provide separability within The definition of DP MLE should be modified combine the cross-entropy loss and the classification 0-1 loss, i.e.,
@DP E arg min BEB
(0) + Σ 1 (2,6o Y)) < 0)
(5)
{E(x]
where À > 0 is a regularization between those two losses. Equation (5) is reminiscent of single-stage alignment procedures such ORPO (Hong et al., 2024) and ASFT (Wang al., 2024), see, e.g., Gorbatovski et al. (2025). Without separability, solving Eq. (5) can NP-hard. Under sufficient regularity, @DP converges to lg E arg mingee (KT((0*, (0*,000 O) + Xm(()) where
13

---

Learning Parametric Distributions from Samples and Preferences
m0)) P (D(O-,0) 0) and On + 0*. As 0 m0)) can be non-convex, computing Og might be challenging. Deriving pue
p
a tractable ELBO method for this optimization is an interesting direction to obtain tractable and robust estimators. As On lies in the boundary of θ, we should control the maximal deviation with respect to 8g for directions that point towards the interior of θ to prove an accelerated rate. While parts of our analysis could be used, we believe that finer technical arguments are required.
Parametric model. The true distribution p" of the observations might not even be member of our class of distributions F. i.e., p" $ F. These situations occur when F doesn't It contain the true structure, e.g., other parametric or non- con-parametric class of distributions. Then, ( can be interpreted as quasi-log- logeliikl term. Let us denote by SO Mus the estimator based on SO MLE for this quasi-loglikelihood. Under sufficient regularity, SO quasi-MI.E converges towards 80 E arg mingge (I.(p".. po) where p" + po, E F. Without the separability from well-specified deterministic preference, we define DP quasi-MLE as in Eq. (5). Under sufficient regularity, DP quasi-MLE converges towards the minimizer of a similar optimization problem combining the KL term and a misspecified equivalent of m(0).
B.3. Reward models
Except for Theorem 4.3, all the derivations in Section hold for general (hence seward-basede preference models provided Assumptions 4.2, 4.4, 4.5 and 4.7 hold. Characterizing the expressivity parametric rewards satisfying those assumptions interesting, yet challenging. We provide two positive and one negative examples.
Positive: monotonic reward. Suppose that (z(2,)) = f(po(=)) - f(p(()) where f is increasing [0, 1]. Since sign((n) = s(g)((n), hence the parameters with zero classification loss and our estimators are the same. Therefore, our results hold for this class of rewards when our assumptions hold for the log- reward. When f is decreasing, the preferences are "reversed", and similar arguments can be made. This example includes (1) normalization by multiplicative constant (e.g., temperature 3) and (2) the odds-ratio reward- bassed preference based on f(x) = loooozz 2/((1 - x)) used by ORPO in Hong et al. (2024).
Positive: margin with Gaussian. Suppose that Eo la + e where c is a constant and Eg is the Gaussian -ikeliiel preference. By extending our computations from Appendix F, Assumptions 4.2, 4.4, 4.5 and 4.7 hold with c-dependent positive constants. Margins are used by SimPO from Meng et al. (2024) and IPO from Azar et al. (2024).
reference model with Gaussian. Suppose that Eo = lo - lo where O is known and to is the Gaussian log-likelihood preference. Since (0 v) (ær y, 0 0o) and Volo(I, x Assumption 4.5 is violated for u = 8* Po. i.e., IP 92 (0", 0r - 0o)) = O. Not all direct alignment algorithms rely on a reference model, see, e.g.,
Negative:
SimPO and ORPO.
C. Proofs of Section 3
C.1. Stochastic Preferences
Under enough regularity, by swapping the integration and the differentiation operators, we can show that
Eyg, Auto V log q0,
Vel 0g and Ego,buto Vi log 90.hrau
Ego
[V log 90 Vo log houh
Below, we detail the proof of Lemma 3.1.
Proof. Direct computation yields that
+ V log qo- (z.V,2) = V log (z4- h (zo (x, lo- v)
Vo- log qo- v) = V log 22
(z.v))) +20 (z. (x.9).
where we used that g'(x) = o-=x) and = - -x)) with g(x) = log (x). By definition of hott
E E
(X, (X)
- (X, (X) (X))) allo (X, Y))o (X, V3.
Vz. lo- (X, Y) 0axd
14

---

Learning Parametric Distributions from Samples and Preferences
Therefore, have I(qo ) I(pg.) + A with A _ Eegg For all R*, we have
x' 3 x = ||0|| 22 (o)) 2
= It is direct to see that this inequality is strict except if P. p0a (1z/ | Vola²= = 0) = 1. Therefore, A is positive definite
matrix if 2e2 (l,, (o- > 0) > for all u E Sk-1- Note that this condition is implied by Assumption 4.5.
C.2. Deterministic Preferences
Consistency of SP. Let (0) == E s2 [log he (x, Y,, sign(Eg. (X, Y) Under enough regularity, we obtain
E Pg*
22 a- log Pg
O and Vo-)((0) = E (sign Co (X, (-) (
Therefore, l* is the unique maximizer of M(0) if o- M(0)) = 0ç. i.e., if (3) holds true.
Asymptotic normality of SP. Provided (3), under enough regularity, the theory of M-estimator yields that
Vn(OPPPa - 0²)) ~00 N(On, Vi V2 Vi,ūo)
where
V2,0- B (x 02
V log go- haro (X, Y, sign(Lo- (X,
V2.0- pxx
Va- log qa- a (X, Y, sign(Lo- (X, V)) log go An (X, sign((a- (X, Y]
Below we detail the proof of Lemma 3.2.
Proof. Combining sign(Ls- (I, y)) with the same manipulation above yields
a. log qu- Ann (1, 2) V logpo" 2 (I, y) + sign(Co- v))o (2, (x,
V g- log y0 Jnn (x, 1/, 2) v- log yo- Ann (x, 2)² a. log 002 Vg. log (x.1))
+ (-11f (I, 2) f-- (x, Vo (x.
(-))))(((((6_)))))) + sign(Lo (I, ()) ( 160- (z, y) Vor log p0 (z, V (2, v)¹ V Lg- Vo log pg. (x, v)))
Vi- log gø Amo (z, U, V log p (I, 9) o (z. 4) o la- (z. Ve-
y)
+ sign(Le lg- (x, ( l (z.
ly- (z, v)
where we used that g' (x) d--x) and g² (x)
(-x) -o(x)((-1) with g(x) = loog (x)) Using that
a- (x, v)l2 = 01 (2. (2. v)l) - aa- (z, ) (lo- (z, y)). we have
V1,0- = I(p00) + A. H Sr
det
and V22,0 I(p + M2 A RRS des
where A
E g3 (0 pl (v in Lemma 3.1, and we define
am Am g.
E E, ssign(fe
)0 (ann(e-))((1160
Vg- Lor1
M2 = E. s2 Pge
(-g Vo- lg- (V and
1 SPan = E 02 siign Cg. )o Cg- (Va- log @2 (Vo-la))" + V Eg- (V log p02)
Using that I(go- ((a.) = I(pp") A (Lemma 3.1), det is asymptotically better than SP if and only if Vi-1 V2,0 Vi,a- I(go- Now) This condition can be rewritten as
I(pa.) 92 + M2 - A
S - (I(Dz3) A H.SPan (I(D0- Aono))
"H SPane
I(ga> )
(6)
The condition (6) heavily depends on the geometry of pg- and lor, hence it unreasonable assume in all generality.
15

---

Learning Parametric Distributions from Samples and Preferences
In the following, we consider the special case where H&Fd Odxd- This occurs when 0 Eg linear, e.g., for FN and & and preferences based ro log po- Then, the condition (6) rewrites
R det + A SP des
> Oaxd
with
A de == 24 M2.0
Using that minzeR ([æ:)) 1/2 achieved only
have directly that, for all RR.
8 am = | s2 ((6-1)) - ()o( ////////1, Va- lo-) 0,
It is direct that this inequality is strict except if no (Lo- (2//||=/1, V fo.) 0) = 1. Therefore, g. is a positive definite matrix if PP pga (1 (u, Ve- > 0) > 0 for all Sk-1. Then, sufficient condition for the condition (6) to hold is that Hggd is p.s.d. matrix, i.c., ast det - 0axd.
In summary, have derived sufficient conditions for SP be asymptotically better than SP, namely HTpPam Oaxds RSP det - Oaxd and P. o0 (lea (u, Vo-lo.). 0) > for all u Sx-1. Note that this last condition is implied by Assumption 4.5.
D. Proofs of Section 4
D.1. Proof of Lemma 4.1
Proof. For Gaussian distributions, this is direct consequence of the following facts: 0" Cu. 0.P E arg mingec, |10
1 and C, is convex.
D.2. Proof of Lemma 4.6
Proof. Let u E Sk-1. Let For,u be the of Vor,a(X, (X,Y) Y) when (X,Y) ~ (p (0-, .a), i.e., Pg. truncated to 91 (0", u). Then, For,u(E) Fer, Fer,a(e)/ag- (0))/a Let arr = P p (91 (8*, u)) and No Eien) ((X,, V,) E ( u)) Bin(n, au). Let Rn,u (x, (0+, Vo (X,, Yi). Using the derivation in Section 4.1, we have that Rn,a I Rn, Let > Conditioned on Nø-,as Na. direct to see that
Fgg (e))) -(1 (1- (F)) No.. P( > 8 No. = (1 (1 (1- F
Using that No-, Bin(n, ag., w.). E Bin(.p)) 1 (1 11-ptt p ps)" and 1-21-2 exp((=z) we obtain that
ag (1-a0raa For FFr sexp expp-=Fo: (E))) P(Rn,u > E) < P S > E) (1-
Taking & = Fo.. (min (1, log(1/6)/n) concludes the proof.
D.3. Proof of Theorem 4.8
Proof. For all u E Sk-1, let (Ag., Bg. Cg.) defined as Theorem 4.8. Since Cn C C₂ under Assumption 4.4, we obtain that maxgec. |0 - 0" S max ||0 0*|
Case l: 1. Since Sal 2, using Lemma 4.6 with union bound yield that, with probability least 5,
max BEC.
||0 - 0* < max R I max F WeSo
(mgnn),
weSa
Under Assumption 4.7, for > Bg- log(2/6),
conclude the proof since
max
||0 0*1| S max || - 0- S Ag- log + Co. log(2/8)3/n. BEC,
BEC.
Case k > 1. Let (n) be the y-covering number of θ for the norm || ||. Let {o))je(N(>) be such y-covering, For all j E [N())), let E ||0, 8*1 and u, (0, - 0*) )/E, Using triangular inequality, we obtain
max 10-60 8rll s max PEC.
|10, 8* + max
(0r + E E,),, S JE((GG)
max Rm,y,
(E[N(T)) EČ,
JE[N(t)
16

---

Learning Parametric Distributions from Samples and Preferences
Using Lemma 4.6 with union bound yield that, with probability least 1 8, n. max 0 1* S my + #EC,
max nFo (log(N - ny + Ao- log(N(r)/0) Co. log( (2)))))/)
jE(N(r))
where the last inequality relies on Assumption 4.7 for n 2 Bg- log (2)
E. Proofs of Section
E.1. Proof of Lemma 5.1 Proof. It is direct to see that
0
if (x, v) E D(0",0) (20(0))840000)))
V qa 2)00
\VPa- (I) (1))o()))))))
otherwise
Therefore, we have
BC(g)-,90 "| V po- (x) (t)po)))pp))po))dddaa) BC pp BC(0",0)
Using that (P,,, 1001-BBCL Q), conclude the proof.
E.2. Proof of Theorem 5.3
Consider the hypercube O' {0, 8b : b E [0,1)*) C. Note that 10, Ov ll
(b, where du (b, b') denotes
vk
the Hamming distance between and V/. Then using Assouad's lemma we have
8v
(1-
Riax
max (b,b)))
TV TV on on
40g
4
Upper bounding TV with H2, Lemma 5.1 yields
(gb,
@n @ny Mow
S V (BC((00 0v) H2 (p0 poy )))).
Then, Assumption 5.2 implies
Rmax
8v
11 (1-mm
c (z(k))
+ 20202
4
Picking 8
[(2)+20)) m (k)
ensures that the term in parenthesis is always greater than 1/2, hence
Vk
(oy(k)
R Rmaxx
min
8(c + 202)
n
Vn
F. Multivariate Gaussian with Known Covariance In the following, 0 = -ー11, 1 denote the natural parameter of multivariate Gaussian with known covariance matrix Σ. We have X = Rd and k = d. Let E O and E Sa-1 for the norm ||x, i.e., [|/]|x = 1. Let S2,d = for E Rd
||u||2 = 1
is direct to see that
Po (z 8. Σ 2x v)/2) and Vg-
lo(z.y) log
(p())
Therefore, we have
(o() ((a,y) (Rd) 1/ - y, 0r (x + v)/23/ 0
(8") {(x.y) Go( || v/l > 0}
D(O", 0) = ((x,y) E (Rd)2 (xr y, 0* 1-1 (x v) 2)/2 (x y, 0" -E- v)/2)(0 0- --,--1) v) < 0)
(0",", u) = [(r,u) E (Rd)2 (x - 0r - -- - (s + v) 2) (u. x - u) < <0}
V(z,y) (0", u), (z, (r- E. N y)/2 8r)
{u, x 1)
17

---

Learning Parametric Distributions from Samples and Preferences
10
(sto) 0.20
(sto) A SP (det) SP (det)
A SP (det)
(det)
10 10- 10
||A agd
0.15
a.
0.10
10 10-20
0.05
0.00
25
50
75
100
25
50
75
100
d
Figure 3. Approximations of A Ta) da
and R aas
by (a) aala and (b) associated eroor for varying d. News 10
Proof that P. pgr (C > is direct see that dim(Gu(0*))) < 2d and dim(Go((")) \ (i(00)) < 2d. Given that Pe- (91 s2 (90())) 1.
202 a continuous distribution on (I,)), we obtain that
The condition of Lemma 3.1 is implied by Assumption 4.5, hence refer to the proof of this result below. Therefore, we have I(ge- hara) I(pg.)..
Condition in Lemma 3.1.
Consistency of SPdtt. To study SP for FNYE, we use the change of variable D = E-11112 - Y)/V2 and S 1/2 1/2 (E8" - (X + +)/2). Then, we have (D,S) NO2a,, I2a) and
la. (X,Y) D)
Vo- (X,Y) /2D!/2D
Vg- log 82 (X,Y) 20'" 1/21/22.
for all (D, E 1R2d By integration of an odd function with respect to Og with a symmetric distribution around O2a, we obtain ((Da)))N((0a.11)) [M = Og. Therefore, the condition (3) is satisfied and the SPdet det is consistent estimator.
Let M(D,S) = sign( (D,S)) ()) KD, D)DD Then, -S-
5)
PPSaact det and Rg defined in Lemma 3.2. Since la y) = (x y, y.0 (x 22/l2 is linear in 0 we have Eg. Oaxd and HS Odxd: The condition p, (16o- (u, Te-Lo-) > 0) > 0 for all E Sa-1
Asymptotic variance of SPdet. Let Ht
is implied by Assumption 4.5, hence refer the proof of this result below. Then, the condition R.SPar des Z Oaxd is equivalent to M3 Odxa where
M3 -E(D.)) (-ID,))) (DS + SDD)]
When 1, we have M3 = (.E Jo (-ID,S)) DS > When > 1, for all u E St-1, we have
[sign( (D, S) (-I(D,))) (u, DYu, S)] By rotational symmetry of N0(2s, Tea) and the function to be integrated, showing that min u M3u 2 0 is equivalent to showing that ef M3e1 2 0, i.e.
u Mzu
RD..S~~ (sign((D, (-1(D,5)) D.S:
By symmetry, we conjecture that ASP (a) A SPat la)
and SPee (a(aa.aa
are of the form aaTa where oa > 0 is decreasing in
d. Figure 3 validates this conjecture numerically.
Using the sufficient condition derived in Appendix C.2, we have shown that SPa. det is asymptotically better than SP.
Proof of Assumption 4.4. Since (x v) = (xr y. 0 (x + v)/2) is linear we have D(0*,0) D((0'
18

---

Learning Parametric Distributions from Samples and Preferences
Proof of Assumption 4.5 For Y) Pø- let D E- - - and S V2E (Y)2) E0")
Then, we have (D (O2dd: /2a)- Defining U DII have U U(S2,a-1) is independent of S. Since
= D/1|D1I ~ a() and E -ー 1)2 0" - we obtain
P P( (X,,) (I(0,, u)) -PU..s)-U((S....))....... (U,S) (U,S)),,\'/1)) >0)
=P ( U) 0) /2 + PU (2 U, U) < <0) 12 1/2.
where we used that, conditioned on U. (U, ~ (0, 1) and Px Px~N1on1 (X <0) = =P -Px-Nio.)(XX > 0) = 1/2. The last
equality uses that P ((E)/2 u, U) > 0) Pu- S2. (( 2u, U) < 0) 1/2 by symmetry of the uniform distribution. Therefore, we have shown that 22 ( (0", 4) = 1/2 for all u E S2,d-1-
Proof of Assumption 4.7. Let us define v = E2/2u, hence v E a(S2.a-1). 1). Let o denote the c.d.f. of /V 1) and erf(x) = 22 (xv2) 1 be the error function. Let > 0. Similarly as above, we obtain that
(E) P ((x,))00 < V (X, E)
(0<
(v, S v2e)
(U,S)
Berulss 20 (V2el(o, U3l) -1] _ -B lerf (elK, U)D] (Kk,
where we use conditioning by U above. By change of variable, we obtain that [
Fer (o)
1 Eu-ur(sa
e dt
[KO.UD|| /' e-
VT
V
Using that - x² see x²
< 1, obtain that
5² (Buee(Szan)) [Ku U3
0 ε
Fe. (e)) -Eu [Ku; U\ 33135
Using that
- 1// x (1-2xx²2( (v, u} 2e 20,02 dr - - (/' c- dx.
obtain
28 1 Bu-u((Saa 1) Uk, and (e) For,((s) V
E EU- [10,0))08==0000000
V
Therefore, using Lemma F.I, we have
2 T(d/2)
(0) = Eu BU- [Kt,, d - <I( 1)/2)
=d++o O(1VV))
V
Let us define
Bu- Kk, U'
Eu-u(S),a (1,
60-
2E [R, 3
and Mo 4th / (U- (v, U)
Then, for all E E (0. .ul. we obtain that
(E) πE BU-u(Saa-))) (Ke Be tmuyy
Fg..a(e)
2 Ev-((S2a-)))
BUre((S2a-)) U113]
II1, - e2 E( (1, I Ma
2 (Bu-el(ssa-1)
19

---

Learning Parametric Distributions from Samples and Preferences
För (F (x)) Fig P (o (x))) we obtain
Since we have (F)
By-u((22 11/21/21, Uy|
( ((Fora))())) S M where zg For (E $ (
sup
2T [1/21//2,, U\1
Proof of Assumption 4.2, Let 1|0" -8 0|| and (f* f)/e. Then, we have
P pga (D(0", 0)) = P pe (D(0", fr + Eu)) 2 P pa*
(D(0", 0r + Eu) n (0", = ((x, (0 < Vo (X,,, <<)
Using the above computation, we obtain that TP (x, 02 (0 Vo wu (x, Y) < <e) > 0, hence P pe (D(0', 0)) 0.
Proof of Assumption 5.2. Using that 1 - e < x, we obtain
(po po) = - cxp (- -1100-0 - 01
S 10 0
First, we notice that dim
< 2d, hence we can show that
f [ v pa- (z)pe- (1)p((2)po(y)ddddg 0.
Second, we see that
|| 1-- + l1y Σ0 [2- + || E8| + -
|| - v + 110 0- + 11x E(00 + -)
D(O", 0) (xx, v) E ( (xr y, 0r - -- v)) 2) (xr y, 0" --- (x 000,))00000xxx1)) < <0}
Then, we consider the change of variable E-1/2 y) and --1/2 y), whose Jacobian has det(E)2- as absolute value of its determinant. Therefore, obtain
e 100 11 BC(O",0) 0) (4r)d S 10 (u, 11/20= v/2)
< 1 e- - --l-~>y' 55 (0+00)) dude
( - -0*), u)
(ū,
< <
e |||111 100 dūdō
(2 (ū,))
(1)/2(0
0+), ū)
(X,Y)
-(xe.))
< <v2
(5)/2(0 - -A*), x)
=Bune(szaan) Cerf
(1E1/22000 0*), U?1) (s)
+ where the second equality uses the change of variable ū = u/ 22 and ū = (u 00/2(0+ (*))// whose Jacobian has determinant 2d. The third and the fourth re-uses computation done previously with E = =1[0 0011> and (0 0*) 0')/E. Using Lemma F.1 and the above upper bound For,, For,u(E), obtain
T(d/2)
BC( ,0) = -/ Fo. (E) < d - mT((d - 1)/2) |10 - -111
Lemma F.1. Let I' be the I function. Then.
2
r(d/2)
Yu E S2 EU-u(Ssa-)) [K U\|
d V 1l)-) =d++ OLL/Vd).
20

---

Learning Parametric Distributions from Samples and Preferences
Proof. Due rotational symmetry of the distribution, for any unit vector u.
Bu-u((aaa-)) (u, U\ = EV-ur(s U)[] -U-((() [11]].
The density of U1 is given by
1(
4-3
(x)
(1 - 22)
x x 1-11]]
V
and the expectation can be computed as
B| = [" |a| fur (x) dx = =2 =2//" I
d-3
(T(
d-3
(1 x
2 dr
x - x²)
2 dx
0 V
V I
= (+ / (1 (1-u)
I(
3
1
2
1
du = v
d
=
d
-
2
Therefore, for large E[ -oo+oo O(1/V)).
G. Laplace with Known Scale In the following, denote the mean parameter of Laplace distribution with known scale b. We have IR and Let 0 E @ and {+1}. is direct to see that
y/
if min(2, y}
()(x,)) = log Po (x(x)
by 0 lr 01/b=
if 0 > max max v)
b
(20 (x + v)) - y)
if 0 E [min(2, y) max y11
f < min (x,8) or 8" > max(2, y)
and
Vo (x, = [sggn(== (x--)) 9) if 0
[min [I, y} max(x,
Therefore, have
9o(0') = ((I, v) E R²| [yy 1- 1 0011 > 0}
Gi(0") = ((I,)) E x² 0" = [min( y), max [z, wH}
Gi(0", u) -{(r,) E R²| 0* E [minn v), (ax y}1 A u((x + w)/2 0*) 0},
D 0) = {(2,,) E R2 | 8 e [min(x, 3) maxfz, vll < sign(0 0')((z + +v\/2 - 0*) < - -0-001}
V(r,y) E Gi (0", Vør (x, u((x + --/2-00) 0*)
When 8* > 0, we have
D(O" 0) {(x,)) v) ()",0) c [min(a, max A 0 < (a /)/2
((,,y) 0 min(z,y) A 0" ((x + /)/2, max(r,y))
U {Zz,)) 1 min(x.y] A 0" max(x,
U {(x,)) 6* max(x,y) y} A A [min(x, v). (r u)
When 0* < 0, we have
D(0*000 0) = ((z, v) (0 C [min v), max(æ, vHl A 0* < (x v)/2 <0}
U ((x,)) 10> max(x, y} A 0* E [min(r y) (x /2)}
{(x,)) 0² < <inffx,)] y) > max(x,y)}
((,,)) 0* <<in(a,y] 0 ((x + v)/2, max [,,
21

---

Learning Parametric Distributions from Samples and Preferences
Proof that P. p0 (G (0*))))) > 0. It is direct see that dim(Gu(0+))) < 2. Given that pg. is continuous distribution on (R)?, obtain that P pg: (Gu(())))) 1. Using the symmetry of the Laplace distribution around its mean, have that
P 02 (0") o ((-o0, 0*) +00) + + pga (00) (-o0, 1/2.
Condition in Lemma 3.1. The condition of Lemma 3.1 is implied by Assumption 4.5, hence we refer to the proof of this result below. Therefore, we have I(go- hhra)) (pp)e)
Consistency of SPate To study Sett for Fiap,d we use the change of variable D = 0" - X and 5 = 0" - Y. For all (D,S) E (0), we have
lo (X,Y) 1 1 + S)sign(S D), Co,lo-(X,))
2 - sign(S - DD), Vo- log (X, =.
For all (D,S) ((D,S) E ( S//)s//s + 5). Then, M(-D, -S) = - MMD, for all (D. E TR? By integration of an odd function with respect to 0 with a symmetric distribution around 02, we obtain E(D,S)~M(0aall (02addad) [M, S)] 0. Therefore, the condition (3) is satisfied and SP det is a consistent estimator.
(0),, we have V lo-
0 and V log pg- (X,Y) * 0. Let M(D, S)
Asymptotic variance of det. Let Pp and ltg- aes defined in Lemma 3.2. By definition of Lg. we obtain Vg, l.. = and Hg 4PSPact = 0. Moreover, using the above formula, we have &g- (X, Y) V- log 22 (X, Y) = 0 for all () e Sa-1 is implied by
Pø-
(0), hence obtain RSPam O. The condition P p.- (16o- (u, V Vo- 0) 0 for all Assumption 4.5, hence we refer to the proof of this result below. Using the sufficient condition derived in Appendix C.2, we have shown that SP det asymptotically better than SP.
Proof of Assumption 4.4. Using that D(0 0) S (0*), we simply need to show that D(00, 0) S (0 n D(O".)) 0). Let us consider the case 6" 0. Then, we have
D 0) = {(xx v) E R² 0* E [min 2}, max vHl A 0 < (x+
= -(x, | (0", 0} C [min(a, y), max(2, y}l A < v/)2
U ((x,))
min(x.y) y) 0 E ((x + 2), max(2, v|l] = (0') (D(("
The same result follows when 0* < 0 by using the same argument. In summary, we have shown that D000,0) (0*) 0) S D(0",0). 0).
Proof of Assumption 4.5. Using the symmetry of the Laplace distribution around its mean, we have P 29 (0',,0)) pp (9' 1)) for all {+1. Then, by integrating for y, obtain
(Veron uiiddy dx
P oz (9 (0", 1)) 212 I
eith
2h /
2a-20 /b 4
UxE(-00,01
JZE(-20,01)
Proof of Assumption 4.7. Let E O. Using the symmetry of the Laplace distribution around its mean, we have Fg. (E)= = For for all u E {+1. Similarly as above, by integrating for x < y. we obtain that
(e) = P (xx))) 02 < (X,Y) 56)
I
e//b
woddy dr
26
JaE(-00,01))
12,228-2.26+-00 20 mx)
1
(20 (20--20)10
1 (1 (1-e) (1-e-2e/) 20- 2r)
2h
220(-20,00)
22

---

Learning Parametric Distributions from Samples and Preferences
Therefore, have
8b
Fg. (x))(())
2 2z/b
Fo (x)= log(1 4x) and (Fo""a"( = (1 4x)
2h
Then, we obtain (.)),(()
and we can take zg = 1/8 and Max,a 32b.
2h
Proof of Assumption 4.2. Let 10 - 01 and sign(O* 0). Using the above computation, we have
IP Pg- 1(00,0) IP Po* (DiOr 0r + +u)) P po+
DiO0, 0r + n (0",,,,) .u)) IP (x.Y))pp3 (0 < Vor Vo,,((X, <))
Using the above computation, we obtain that IP
(x, ~xp o2 (0 < Vo+ (X, Y) < E) > 0, hence F paa (D(0* 8)) > 0.
Proof of Assumption 5.2.
Using that [(x) x² 1
1 +
+ x)6 - is positive on R we obtain
(po =1 (1+ 10 0
10
(0 0)²
exp
S
2b
2b
402
First, we notice that dim (
2, hence we can show that
[ ( v pe- (x)pp- (p)oyddd 0.
We consider the case l 0 since H" is done similarly BC( 8) BC(0, Let = 0 0", By integrating for y, have
BC 0)
262 / ea/b
(110)50<<at)))2
(e dr
+ e (e 1/b
[000 (1 2)/2) dx
26
e s/b c/b
(f1(00 (/100<<<0000
<<<v<<> - dr
212
e 0" /b
[ e u-b ( 1 (0 <x ^ (a + dy
+
2h2
Direct computation yields
J IESTO--co ez/b (1 (/ecomn-2001) ldy dr I Imsco-co0 5/ (x + -0") dx = -0)-
[E((0,))
SIcS /e (0/10(6 -b)+ +b)
[/11110 <<< + ty) dr 1EE( (0 + x)dx = 1
g²
2
(1 1dr dy des.
+0(00-+,00420) JSclo-1t0012200 = =
wu/b
v/b (0* + 28 y) dy
uE((,e)
23

---

Learning Parametric Distributions from Samples and Preferences
Moreover, have
[ e0 (/1(050 <(r <0 + e) e wdddy dr
e /y dæ + J0E(0r-5 - e²/b zE(-0000--o) VE(20* -.. 2.20'+22 +22
e²/b
e eddMdy dx
-0 22. 422--)
+2e dr + 1xE(0-- (0- - 0-220 +20 +2e-20)/0 201100 dr 120 1200~22)/1 -20)/6 120 +2e +2e-20)/1 20)/6)
JxE(-x0,
b 2z/b se/b + b (6 - 22//6) + 2
b ( (c-e//6 -e - = 122 (e-e1b - - - -2201)
Therefore, we have
(e (2-2/ + +e -2(0 -0000) (E-)(E-b) -b) +))+1) +b) + + ee/6 6²
BC( 0" +
e e/b c -22/10)
+ 2b
2
462
1
5/6 e 25/0)
(c-2//b = -210 -2100)))10) (6 //0000 -cs1b +1) + t --/6 82
2
2
452
1
(e-2(0'
-(00-=)/66 -b/6 6/_ c
- -(200+))/b
+ (e (c-0/b - -20000 E ee 6²
1
2
412
2 c/b (e (2-00// c/b e/b) +
E²
2/²
Then, we can conclude that
1 -/6b BC(A", 0 -e) = =BC( -,,0')
2(1000 (6-50) 1 e/b)
6²
2
2b2
Using that f(x) - 2²/2 - -e e is positive on R4, we obtain
0+ 0
(1
0 0
(1
+ E min/o0 mmin00000/// .0/10)
BC(0", 0)
2b
2h
H. Supplementary Experiments
Using the same empirical setup as Section 6, we conduct additional experiments to support our theoretical claims for other distributions (Appendix H)1), other estimators for Gaussian distributions based on C, (Appendix H.2), other convex surrogates of the 0-1 loss (Appendix H.3) or normalized/reegulrrized versions of the logistic loss (Appendix H.4).
Reproducibility. Code for reproducing our empirical results is available https: Learning-patee www.:aaraaaaaaaaiaaaaaaeeeeeeeee wtw.rammttii-diissiitattaaa
com www/toloee/tteeeeeeceee
from- samples lep and- www.ppreeeeeee....em Our code is implemented in Julia (Bezanson et al., 2017), version 1 1.115. The plots are generated with StatsPlots. The optimization problems defining some of our estimators are solved numerically with JuMP (Lubin er al., 2023), by using the Iрopt (Wächter & Biegler, 2006) and HiGHS (Huangfu & Hall, 2018) solvers. Other dependencies are listed in the Readme má that provides detailed julia instructions to reproduce our experiments, as well as a script. sh to run them all once. Our experiments are conducted on 12 Intel(R) Core(TM) Ultra 165U 4.9GHz CPU.
Gaussian distribution with known variance. For Fx.1 the SPaet des and SP estimators computed with the Ipopt solver. For FN the Sedt SP, DP and WE estimators are computed with the Ipopt solver, and the AE estimator uses the HiGHS solver.
H.1. Accelerated Rates for Other Distributions H. LAPLACE DISTRIBUTION WITH KNOWN SCALE Estimators. For Frap,1 (Appendix G), we have
6so median ((X))etx) U (Yi)e())) and Cn = (0 Vi E [n). Z(Y; - 01 - -|X, - 01) 2 20)
24

---

Learning Parametric Distributions from Samples and Preferences
10
10-
10
10
SO
10
SO
AE
WE
10
DP
AE
10²
10
10
102
10
10
n
n
Figure 4. Estimation emrors for (a) Lap(O", where 0 22((1,2] with Nome 10 and (b) Rayleigh (veleiggh((+))) where 0" ~2((11)] with
Nmms 10a.
The estimators based on Cn are F E Cn., O.E arg maxpec. |0 10' and
arg max DEC.
(12 0 +|x 01)
iE|n]
Those three estimators are computed with the Ipopt solver.
Experiments. Figure 4(a) confirms empirically the difference in estimation rate between the M-estimators (SO MLE) obtaining O(I/vn)-and and our estimators based on achieving (11/n). Moreover, AE and WE perform on par with DP MLE.
H.1.2. RAYLEIGH DISTRIBUTION
Let > 0 be the scale parameter characterizing a Rayleigh distribution. In the following, let 0 = 1 natural parameter of Rayleigh distribution. We have θ R X' E2 and 1. The probability density function defined as
20a <
denote the
Vr I po(x) exp (x + log(x) log(20))
Let f E θ and u {+1}. It direct to see that, for all (z, y) 123
(o(
dlg. d0 (z, x² --² (x vx + v)
Eo(x,z) log py(y)
(x²-)220 log(x/v) and
Therefore, we have
Gu(#") {(x.)) E | - 0²) log(#/))) > 0}.
Gi(#") = {(x,)) E R |2 > 0}
Gi(6", u)) {(x,v) E R |[((²² - 2²) + -22 y²) log(z/))) < 0}.
D(0",0) ((x,)) R ((x2 -20 + (gglz/v)) 2 + (x² (² 0*) -222- +log(x/v)) (v(x
log(z) log(y)
v(z,y) (i u). Vo, -u 0
x x
y
Proof that P PS (0*) 0. is direct see that dim (0 < 2 and (im(G)(0*)) ((+)) Given that @2 (G*())) 1.
continuous distribution (R4)², we obtain that IP pgr
(Go (8*))
P pg-
25

---

Learning Parametric Distributions from Samples and Preferences
10-
10
SO
10
WE
RU
DP
SO
CE
TGG
DP
TAMLE
CCE
10
10
10²
10
10
103.0
10
10
n
n
Figure 5. Estimation emors for N0 La) where 0 ~ with Nmm. 10 for (a) 1 and (b) d 20.
Proof of Assumption 4.4 and 4.5. Since (o(x,)) (x2--220 0 log(x//) ) is linear in A we have D(0*.0) ()
Let (X,Y) 2 Then, we have
1 log(Y/X)
P a (9) (0", 1)) P (X.Y)~
1r
>0,
X2 1 (V/X)3
1 log(Y
P pg.
(Go -1) P ((xY)~~p
0
00
X2 X21-
Estimators.
We have
Σ (E(m)
(X?+
and C, {0 Vi E [n]. ((Xx? Y, log(X, 2 0}. Y,,
4n
The estimators based on Cn are QAE E Cn. OWE == arg maX 00 - | Those two estimators are computed with the Ipopt solver.
Experiments. Figure 4(b) confirms empirically the difference in estimation rate between the M-estimators (SO MLE) obtaining O(L/Vn)-an and our estimators based on C, achieving O(1/n). Moreover, AE and WE perform simi- larly.
H.2. Other Estimators for Gaussian Distributions
To better understand the surprising performance of the RU estimator, we consider other estimators that disentangle the effect of RU's randomness versus its mean behavior.
Univariate Gaussian. The center estimator (CE) returns the center of the interval C.,. The truncated Gaussian estimator (TrG) returns realization from a Gaussian distribution with mean CE and variance 4/n, which is truncated to C,, The truncated MLE (TrMLE) returns the average of the observations ({ ((Xi)sem) U (Yi)vet))) O C.-
Figure 5(a) reveals that TrG performs on par with RU, yet CE and TrMLE outperform both TrG and RU. This suggests that being far away from the boundary of C. improves performance compared to DP that lies the boundary of C, (as observed empirically). Moreover, randomization on C, worsens performance compared to CE.
Using the derivation in the introduction on univariate Gaussian, it is coherent that CE improves on DP by a multiplicative constant: the average of those two (non-indes)) -on-independdnt) random variables decreases faster. Formally, this could be proven by refining the proof of Lemma 4.6 to account for the property that No.,-1 + No
26

---

Learning Parametric Distributions from Samples and Preferences
0.20
SO
AE
DP
CCE
0..5
0.10
0.05
0.00
20
40
60
80
d
Figure 6. Estimation errors as a function of with with No0", Ia) where 0" ~ 2((112))). for 10 and Nruss 102.
Multivariate Gaussian. For d > 1, multiple centers exist. We use the Chebyshev center estimator (CCE) of C.-
Figures 5(b) and 6 shows that CCE outperforms AE by constant margin. It only outperforms DP in the regime of large compared to d and performs worse than SO MLE for small n. Geometrically, for small n and large d, we conjecture that the random polytope C, is more likely to be "spiky" along some directions. Due to those distant vertices, the center would become worse estimator than DP, since the "average" is intuitively less robust to outliers. contrast, DP MLE dominates SO MLE statistically (Lemma 4. hence it achieves rate O(vd/n) when is small compared d.
H.3. Estimators Based on Convex Surrogate of the 0-1 Loss
While DP MI.E minimizes an objective that minimizes the 0-1 loss, SP MLE minimizes an objective involving the logistic can generalize this approach any convex surrogate of the 0-1 loss, Figure 7(a). For example, consider the Hinge loss (Hin), i.e., funn (æ) max - 2 the square loss (Squ), i..., fsqu (x) (1 2²) the truncated square loss (TrS), i.c., fns( max(0, x}? the Savage loss (Sav), i.c.
loss fLog (z) log(1 + exp(-x)). As Tang et al. (2024b),
fsw (x) = (1 + cxp(x))-2 and the exponential loss (Exp), i.e., fisp(2) = exp(-
Given (X,, Y, Z4) Ze)
2/n] and loss f. we consider the estimator _
0% E arg min L. (5) + Σ 1(2,6((x,))0)
lE
kE(m)
All those estimators computed with the Tpopt solver.
Figure 7(b) shows that all estimators perform on par with SP MLE, i.e., the one based on the logistic loss.
H.4. Impact of Normalization and Regularization
The estimator defined in Appendix H.3 can be further generalized by introducing regularization parameter À 2 0 and @(a)
a normalization parameter / 0, see, e.g., Gorbatovski et al. (2025). Given (X,, Y,, Zi)ie(n) ~ 4g- haat a loss f and regularization/nomaalaaaaoo (A, 3), we consider the estimator
A arg min
(. + Σ ((3Z,6((x,))))
PEE
vE(n)
While similar modifications could be made for other losses, we focus on the logistic loss fLog(z) = logg1 - exp(-x)). In particular, we recover SP MLE by taking À = 1.
Figures 8(a) and (b) showcase the "mild" impact of normalization and regularization.
27

---

Learning Parametric Distributions from Samples and Preferences
2.00
Logistic
1.75
Squared
Hinge
10-2
M/m
1.50
Exponential
Truncated squared
1.25
Savage
SO
flo)
0-1
DP
1.00
Log
0.75
10
10
3
Hin
Squ
TrS
Sav
Exp
0.25
0.00
102
10
10
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
2.5
3.0
P
n
Figure 7. (a) Figure 2 in Tang al. (2024b): notable examples of binary classification loss functions. (b) Estimation when minimizing the empirical losses for N00", 1) where 8* ~ ~u((11,) 2]) with Naus 10.
10-
10 2
10-3
SO
10-
SO
DP
0.1
A= 0.1
10.0
10.0
2
10 n
10
102
10
10
n
Figure 8, Estimation errors when minimizing the empirical losses for N0 1) where 0" ~ u((1,2 2)) with Num = 10a when (a)
normalizing by with regularization À and (b) regularizing by with normalization
28


Reviews:
[{"review_id": "55ODfMrplu", "reviewer": "Reviewer_vgrP", "summary": "The paper provides a set of estimators and conditions to improve the estimation error in learning the parameters of continuous parametric distributions when additional preference feedback is available. More concretely, the question is the following: For a continuous parametric distribution $p\\_\\theta$ with i.i.d. samples $\\{(x\\_i, y\\_i)\\}_i$ and a known reward function $r\\_\\theta$, how/when does including noisy/deterministic preferences $z\\_i \\propto r\\_\\theta(x\\_i) - r\\_\\theta(y\\_i)$ improve the estimation error of the parameter $\\theta$?\n\nTo answer the above question, the authors first leverage the asymptotic theory of M-estimators, showing that a maximum-likelihood estimator (MLE) that takes preference data into account has the same standard error rate as sample-only estimators, i.e., $\\Theta(\\frac{1}{\\sqrt{n}})$, while achieving a potentially improved asymptotic variance for noisy preferences and a further improved variance for deterministic preferences.\n\nFor deterministic preferences, they take a further step and provide another estimator: an MLE with the hard constraints given by preferences. They then make several assumptions on $p_\\theta$ and $r_\\theta$ to show that this new estimator can achieve an accelerated error rate of $\\mathcal{O}(\\frac{1}{n})$ compared to the standard $\\Theta(\\frac{1}{\\sqrt{n}})$. In particular, they show that Normal and Laplace distributions with log-probability reward functions satisfy these assumptions.\n\nFinally, they prove that the rate of $\\mathcal{O}(\\frac{1}{n})$ is minimax optimal up to problem-specific dimensions and logarithmic factors. Toy experiments on a multivariate Normal distribution are provided to support the theoretical findings.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer vgrP for the time spent and the detailed comments. Due to the limited space, we only address some of the reviewer’s concerns.\n\n**Restrictive assumptions**\nWhile our research question is inspired by iterative human preference alignment (see the answer to Reviewer Gth4), we do not claim the direct applicability of DP MLE for realistic LLM training. When studying DP MLE only, we conjecture that the “global” assumptions 4.2 and 4.4 can be weakened to local versions. Using time-uniform concentration results, we can build a sequence of shrinking confidence regions $(R_n)_n$ around SO MLE that contains $\\theta^\\star$ for all time $n$ with high probability (whp). Then, we modify DP MLE to be constrained on $R_n \\cap C_n$ that contains $\\theta^\\star$ whp. For $n$ large enough, $R_n \\cap C_n$ will be included in a local neighborhood of $\\theta^\\star$ under which the “local” assumptions 4.2 and 4.4 are satisfied. Given that Assumption 4.4 is based on “ignoring” the reminder term in a first-order Taylor expansion, assuming a local version is a significantly weaker requirement.\n\n**1. Misspecification**\nThere are two possible sources of misspecification not taken into account by our current analysis.\n- Misspecified observations. See answer to Reviewer yquN when $\\theta^\\star \\notin \\Theta$. When $p^\\star \\notin F$, $L_n(\\theta)$ is a quasi-log-likelihood term, as $F$ doesn’t contain the true structure. Under sufficient regularity, SO quasi-MLE converges towards $\\theta_{0} \\in argmin_{\\theta \\in \\Theta} KL(p^{\\star}, p_{\\theta})$ where $ p^\\star \\ne p_{\\theta_0} \\in F$. Without the separability from well-specified deterministic preference, we define DP quasi-MLE as in the answer to Reviewer yquN. Under sufficient regularity, this estimator converges towards the minimizer of a similar optimization problem based on the above KL and a misspecified equivalent of $m(\\theta)$.\n- Misspecified preferences. The Bradley-Terry (BT) model that uses reward-based preferences has limited expressivity as it doesn’t allow for intransitive preferences. Even when individuals exhibit transitive preferences, their averaged preferences might be intransitive due to disagreements. See Munos et al. (2024, Nash Learning from Human Feedback) or Swamy et al. (2024, A Minimaximalist Approach to Reinforcement Learning from Human Feedback).\n\n**2. Verifying our assumptions**\nIn all generality, it is challenging to give a general recipe to formally verify those assumptions. A formal verifier (Lean) or software (SageMath) might be useful given a closed-form definition. Numerically, those assumptions can be confirmed or rejected by sampling from $p_{\\theta^\\star}^{\\otimes 2}$. Assumption 4.4 is rejected by exhibiting $(X_i,Y_i) \\in \\tilde D(\\theta^\\star,\\theta) \\setminus D(\\theta^\\star,\\theta)$. Assumptions 4.2 and 4.5 are confirmed by finding $(X_i,Y_i) \\in D(\\theta^\\star,\\theta)$ and $(X_i,Y_i) \\in G_{1}(\\theta^\\star,u)$. The sampling complexity of such tests scales as the inverse event’s probability. Using Dvoretzky–Kiefer–Wolfowitz inequality, $F_{\\theta^\\star,u}$ can be estimated to verify Assumption 4.7 hold.\nOur additional experiments with accelerated rate include Laplace and Rayleigh distributions, see the anonymous plots at https://anonymous.4open.science/r/ICML25SuppExp .\n\n**RU versus DP**\nSee answer to Reviewer Gth4.\n\n**Covariance Gap**\nOur simulations suggest that the asymptotic gaps between SO and SP are mild. The empirical gap is also small for moderate $n$.\n\n**0-1 loss**\nSee answer to Reviewer yquN. For non-separable data, the minimization of the 0-1 classification loss can be NP-hard even for the simple class of linear classifiers, e.g., Feldman et al. (2018, Agnostic Learning of Monomials by Halfspaces is Hard).\nInspired by (Tang et al., 2024, Generalized Preference Optimization: A Unified Approach to Offline Alignment), we implement estimators based on other convex surrogates: Hinge, Square, Truncated square, Savage, and Exponential. All estimators perform on par with the logistic loss, see the plot at https://anonymous.4open.science/r/ICML25SuppExp .\n\n**Intuition on $V_{\\theta^\\star,u}$**\nIt quantifies the amount of information in $(X_i,Y_i)$ to discriminate $\\theta^\\star$ from other parameters on the half-line directed by $u$. The lower $V_{\\theta^\\star,u}(X_i,Y_i)$ is, the more discriminative $(X_i,Y_i)$ is. \n\n**Informative preferences**\nFor observations with null preference gradient, parameters close to $\\theta^\\star$ could have similar preferences. Therefore, those samples are not sufficient to discriminate between them.  \n\n**Negative examples**\nThose claims are a direct consequence of the definitions and will be proved in Appendix for completeness. \n\n**Typo**\nIt will be fixed.", "claims_and_evidence": "The paper is a theory paper, where all its theoretical claims have been rigorously proved under the stated assumptions. The authors do not overstate their contributions and clearly acknowledge the limitations of the work (e.g., the restrictiveness of the assumptions). Moreover, the toy experiments in Section 6 are consistent with and provide empirical support for the theoretical claims established in the previous sections.", "methods_and_evaluation_criteria": "The paper primarily contributes to the theoretical aspects of preference learning. The main methodology described is the deterministic preferences MLE (DP-MLE) presented in Section 4, which uses a 0-1 loss to constrain the set of feasible parameters based on the implicit assumption that reward models are well-specified. This approach makes sense if there are good reasons to believe the reward model is indeed well-specified. However, I do have some concerns about the assumptions used in the analysis, which I will outline in the following sections.", "theoretical_claims": "The paper has several theoretical claims: \n1. ${\\color{green}\\text{Lemma 3.1}}$ and Lemma 3.2 on the asymptotic variance of the preference-based M-estimators. \n2. ${\\color{green}\\text{Lemma 4.1}}$ on the benefits of using their proposed constraint-based estimator compared to M-estimators for Normal distributions.\n3. ${\\color{green}\\text{Theorem 4.8}}$ (including ${\\color{green}\\text{Lemma 4.6}}$) and its corollary ${\\color{green}\\text{Theorem 4.3}}$ on proving the accelerated rate of $\\mathcal{O}(\\frac{1}{n})$ for their proposed estimator.\n4. Theorem 5.3 (including Lemma 5.1) on the estimation lower bound for the deterministic feedback case.\n5. Proving that the Normal and Laplace distributions satisfy Assumptions 4.2, 4.4, 4.5, 4.7, and 5.2, which are necessary for the theoretical claims in the paper (Appendices E, F).\n\nI have only checked the correctness of the results shown in ${\\color{green}\\text{green}}$ and did not find any issues.", "experimental_designs_or_analyses": "As far as I can tell, the experiment section presents a toy multivariate Gaussian setting with the sole purpose of supporting the theoretical findings. The code is provided, but I did not verify it directly. However, the experimental results appear sound and provide appropriate empirical support for the theoretical claims.", "relation_to_broader_scientific_literature": "The contributions are directly related to the empirical success of preference-based fine-tuning of large language models through methods like RLHF, compared to methods that only rely on positive examples such as supervised fine-tuning. In this context, the work attempts to develop a deeper theoretical understanding of when such preferences can help improve learning, using a simplified parametric setting.\n\nThe ideas presented in Section 3, regarding the effect of adding preferences to standard M-estimators, are primarily based on the well-established asymptotic normality theory of parametric MLE. The authors apply similar techniques and tools to calculate the Fisher information matrix in the preference-based setup and investigate the conditions under which it can be strictly more informative than the standard M-estimator.\n\nThe results in Section 4, however, appear more novel and rely on the hard constraints imposed by deterministic preferences. The authors have appropriately discussed previous related results that use similar hard constraints to achieve better estimation error: for example, the parameter estimation of a uniform distribution on $[\\theta, \\theta + 1]$ by taking the minimum of samples (Wainwright, 2019), which has a known minimax rate of $\\Theta(\\frac{1}{n})$.", "essential_references_not_discussed": "As far as I know, the essential references have been discussed.", "other_strengths_and_weaknesses": "**+ Soundness, Novelty, and Technical Contributions**\n\nAll assumptions are concretely specified, and I find the technical contribution around achieving the accelerated rate of $\\mathcal{O}(\\frac{1}{n})$ both interesting and novel. While I have concerns about the restrictiveness of the assumptions, the authors demonstrate that both Normal and Laplace distributions satisfy these assumptions, which can be seen as a meaningful degree of practicality. All results are well-supported by rigorous proofs and tested by the toy experiments.\n\n**- Quality of Presentation**\n\nThe presentation of the work could be significantly improved. The current writing, especially in Sections 1 to 3, reads more like a collection of independent chunks of information without a cohesive story connecting them. I believe the authors have developed several ideas and attempted to articulate them, but in doing so, they relied on implicit contextual understanding that isn't provided in the text. I would suggest approaching the writing from the perspective of a reader encountering the paper with no prior knowledge of the work and providing sufficient context throughout. Additionally, in Section 2, definitions and motivations are sometimes intermingled, making it unclear what constitutes a formal definition versus what serves as intuitive examples. The following concrete instances illustrate these issues, though they are more cases:\n\n1. I needed to read the entire paper first to understand the paragraph in lines 57-63 about how hard constraints can help achieve better estimation error for a standard normal distribution. The presentation would benefit from more context regarding what is known and not known by the estimator about the estimation task, what the parameter of interest actually represents, and why $S_i$ is defined in this way.\n\n2. In Section 2, the paragraph about *informative preferences* (lines 137-150) defines the two sets $\\mathcal{G}_0$ and $\\mathcal{G}_1$ based on a vague notion of \"informativeness\" without providing context for why one might be interested in samples with non-zero preference gradients. What does it mean to say \"only preferences of samples in $\\mathcal{G}_1(\\theta^\\star)$ can provide information on $\\theta^\\star$\"? Why is $\\mathcal{G}_0$ defined if informativeness is only based on $\\mathcal{G}_1$?\n\n3. Also in Section 2, the paragraph on *negative examples* (lines 152-164) is extremely unclear. This paragraph could be placed anywhere in the paper without affecting the overall narrative. The claims lack concreteness, and no proof or proof sketch is provided.\n\n**- Significance of the Results for the Community**\n\nThe main limitation of this work is perhaps its overly restrictive set of assumptions, which may limit its significance and applicability in the broader community, particularly in preference learning. Reading the first lines of the abstract, It seems that the authors motivate the applicability of their theory based on advances in preference learning for language models. While the authors acknowledge the restrictiveness of their assumptions, they rely on their results showing that Normal and Laplace distributions satisfy these assumptions to claim broader applicability. However, I do not believe that merely demonstrating compliance with these assumptions for Normal/Laplace distributions is sufficient to establish applicability in more complex scenarios like preference learning in language models. My concerns are twofold:\n\n1. The deterministic method (DP-MLE) with 0-1 loss in Section 4 relies on the well-specification of the reward (preference) model. Although standard asymptotic theory for M-estimators also assumes well-specification of the parametric model class, there seems to be a big difference. In standard MLE, if the model is misspecified, one can employ quasi-MLE to obtain robust estimation (see [1]). However, in the deterministic case, if the reward model is misspecified, the constraint set $\\mathcal{C}_n$ may not necessarily converge to a set containing the true parameter $\\theta^\\star$ as $n \\to \\infty$. Since DP-MLE is constrained to $\\mathcal{C}_n$, I suspect it could yield arbitrary estimates under misspecified models and lack robustness. This is particularly concerning given that model misspecification is almost always a possibility, especially when dealing with human annotators, where reward models are known to be misspecified [2].\n\n2. Even assuming correctly specified models, the paper provides no recipe to verify whether Assumptions 4.4, 4.5, and 4.7 hold for a given parametric model. These assumptions appear extremely difficult to check for arbitrary parametric models. The authors devote six pages of mathematical derivations just to prove them for the relatively simple cases of Normal and Laplace distributions.\n\n\n**References**\n\n[1] White, Halbert. \"Maximum likelihood estimation of misspecified models.\" Econometrica: Journal of the econometric society (1982).\n\n[2] Casper, Stephen, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman et al. \"Open problems and fundamental limitations of reinforcement learning from human feedback.\" arXiv preprint arXiv:2307.15217 (2023)", "other_comments_or_suggestions": "1. The definition of 0-1 loss for stochastic preferences in lines 230-231 is unclear. I suggest the authors clarify what this means exactly and explicitly state why minimizing such a loss is NP-hard.\n\n2. Theorem 4.3 is presented as a corollary of the main Theorem 4.8, yet it appears in an earlier section. This may cause confusion for readers. I suggest the authors first state Theorem 4.8 and then present the corollary specifically for Normal/Laplace distributions to improve logical flow.\n\n3. The proof of Theorem 4.8 could be easier to grasp if the authors provided some intuition behind the definition of $V_{\\theta^\\star, u}$ on line 302.\n\n4. There appears to be a typo in lines 431-432.", "questions_for_authors": "Despite the limitations I mentioned, I still think the paper has the potential to be accepted at the conference. The deciding factor for me is the authors' response to the limitations I highlighted under \"Significance of the Results for the Community\" in the weaknesses section. Could the authors elaborate on the implications of model misspecification in DP-MLE and also explain how one can verify assumptions 4.4, 4.5, and 4.7 for realistic models? I don't necessarily expect proof that DP-MLE is robust to misspecification, but I would expect at minimum an acknowledgment of this as a major limitation of the work.\n\n*Minor Questions:*\n\n1. In Figure 1.a, how does the RU method outperform DP? This seems somewhat counter-intuitive. Could you elaborate on this observation?\n\n2. In Section 6, what is the goal and implication of including the paragraph about covariance gap starting in line 434? It seems disconnected from the other points in the experiment section. Could you provide more context for its relevance?\n\n3. How do you envision applying the DP-MLE method for preference learning in realistic language model training? Can you provide at least an outline of when/how this approach might be feasible in practice?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "76Vw29PO3P", "reviewer": "Reviewer_Gth4", "summary": "This paper studies when adding preference feedback can boost the parameter estimation in the cases of Gaussian and Laplace distributions. The results are mainly theoretical, containing three parts: (1) For M-estimators, adding an additional ``preference'' term related to the logarithm of probability helps reduce the asymptotic covariance; (2) For estimators based on hard preference constraints, the error converges at a rate of $\\mathcal{O}(1/n)$ with high probability; (3) This rate is mini-max optimal up to dimension and problem-dependent constants, using Assaud's Lemma.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer Gth4 for the time spent and the encouraging feedback. We address the reviewer’s questions below.\n\n**Iterative human preference alignment**\nWe investigate the case where pairs of observations and their preferences are tied together, which includes the log-likelihood ratio as preference. We detail the connections with iterative human preference alignment. \n- Many human preference alignment methods build on the Bradley-Terry (BT) model for preference, based on rewards. Direct alignment algorithms use variants of the log-likelihood to define the implicit reward of a policy. Choosing $\\ell_{\\theta}(x,y) = \\log p_{\\theta}(x) - \\log p_{\\theta}(y)$ coincides with the optimal policy for maximum entropy RL (see, e.g., Swamy et al, 2025, All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning). \n- For offline preference data, the assumption $(X,Y) \\sim p_{\\theta^\\star}^{\\otimes 2}$ is unrealistic as $\\ell_{\\theta^\\star}$ is collected from a fixed data set of pairs of observations. Recent LLMs are built on iterative alignment procedures. At stage $N$, the model $p_{\\theta_N}$ is trained based on the preference data for generations by the previous model, i.e., $(X,Y) \\sim p_{\\theta_{N-1}}^{\\otimes 2}$. Under the realizability assumption and without mode collapse, this self-refinement paradigm converges towards the true model $p_{\\theta^\\star}$. Our setting characterizes the limiting behavior of this iterative process, i.e., preference based on $\\ell_{\\theta^\\star}$ for observations from $p_{\\theta^\\star}$.\n\n**1. RU versus DP**\nFigure 1(a) provides evidence suggesting that the randomized estimator (RU) and the worst-case estimator (WE) perform on par with DP MLE: RU is slightly better than DP, itself slightly better than WE. Figures 1(b) and (2) highlight that DP outperforms WE and AE for larger dimensions, where the gap increases when $d$ is nonnegligible compared to $n$. Therefore, only DP obtains the best-of-both world estimation error rate. For large $d$, implementing RU is challenging. We conjecture it suffers from the same limitation as AE. This is supported by additional experiments on new estimators using the setting of Section 6, see anonymous plots at https://anonymous.4open.science/r/ICML25SuppExp .\n- Figure 1(a) extended. The center estimator (CE) returns the center of the interval $\\mathcal C_n$. The truncated Gaussian estimator (TrG) returns a realization from a Gaussian distribution with mean CE and variance $4/n$, which is truncated to $\\mathcal C_n$. TrG performs on par with RU. CE outperforms both TrG and RU. This suggests that being far away from the boundary of $\\mathcal C_n$ improves performance compared to DP that lies on the boundary of $\\mathcal C_n$ as observed empirically. Moreover, randomization on $\\mathcal C_n$ worsens performance compared to CE.\nUsing the derivation in lines 55-63, it is coherent that CE improves on DP by a multiplicative constant: the average of those two (non-independent) random variables decreases faster. This can be proved by refining the proof of Lemma 4.6 to account that $n = N_{\\theta^\\star,-1} + N_{\\theta^\\star,1}$ (defined in Line 658).\n- Figures 1(b) and 2 extended. For $d>1$, multiple centers exist and we use the Chebyshev center estimator (CCE) of $\\mathcal C_n$. While CCE outperforms AE by a constant margin, CCE only outperforms DP in the regime of large $n$ compared to $d$. It performs worse than SO for small $n$. Geometrically, for small $n$ and large $d$, the random polytope $\\mathcal C_n$ is more likely to be “spiky” along some directions. Due to those distant vertices, the center becomes a worse estimator than DP, since the “average” is intuitively less robust to outliers. In contrast, DP dominates SO statistically (Lemma 4.1), hence it achieves rate $O(\\sqrt{d/n})$ when $n$ is small compared to $d$.\n\n**2. Line style**\nThe dashed line is shorter than the solid line, see SP (sto) and SO, yet others are not distinguishable. We will correct this.", "claims_and_evidence": "Most of the results are theoretical and supported by proofs. The assumptions are satisfied by the Gaussian or the Laplace distributions.", "methods_and_evaluation_criteria": "The estimators in this paper are mostly theoretical. SO and SP are typical M-estimators. AE and DP require (1) solving a feasibility problem, which can be NP-hard for general cases; and (2) the hard preference assumption (analogous to the feasibility condition). I don't think this could be the real case.", "theoretical_claims": "I checked the proof sketch. It makes sense to me.", "experimental_designs_or_analyses": "I checked the experiment results. As I mentioned before, this paper is theoretical. Yet there are still some minor issues.\n1. It seems a little weird that any randomized estimator in $\\mathcal{C}_n$ (RU) outperforms the one that maximizes the log-likelihood in $\\mathcal{C}_n$ (DP) in Figure 1(a).\n2. The legend doesn't match the figure's line style.", "relation_to_broader_scientific_literature": "This paper provides a new perspective in analyzing the role of preference. This paper's case is not the same as the human preference alignment (e.g. RLHF): the paper is studying an estimator ``plus'' some preference data as an additional source of information, while RLHF or DPO is trying to learn something from only the preference data, implying that the paper may be of limited value to the LLM literature. However, this paper still provides some interesting observations, which might be of interest to the community of statistics.", "essential_references_not_discussed": "No (as far as I know).", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SdLQ1ihSXX", "reviewer": "Reviewer_yquN", "summary": "This paper studies the conditions under which preference feedback improves parameter estimation. The authors show that preference-based estimator can achieve a better assympototic variance than sample-only estimators. When incorporated with hard constraints with deterministic preference, the authors prove an estimation error of $\\mathcal{O}(1/n)$, improving upon traditional rate $\\mathcal{O}(1/\\sqrt{n})$, under some restrictive assumptions. They also develop a matching lower bound.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer yquN for the time spent and the positive feedback. We address the reviewer’s questions below.\n\n**1. Reward models**\nExcept for Theorem 4.3, all the derivations in Section 4 hold for general (hence reward-based) preference models provided Assumptions 4.2,4.4,4.5 and 4.7 hold. Characterizing the expressivity of parametric rewards satisfying those assumptions is interesting, yet challenging. We provide two positive and one negative examples.\n- Positive: monotonic reward. Suppose that $\\tilde \\ell_{\\theta}(x,y) = f(p_{\\theta}(x)) - f(p_{\\theta}(x))$ where $f$ is increasing on $[0,1]$. Since $sign(\\tilde \\ell_{\\theta}) = sign(\\ell_{\\theta})$, hence the parameters with zero classification loss and our estimators are the same. Therefore, our results hold for this class of rewards when our assumptions hold for the log-likelihood reward. When $f$ is decreasing, the preferences are “reversed”, and similar arguments can be made. \nThis example includes (1) normalization by a multiplicative constant (e.g., temperature $\\beta$) and (2) the odds-ratio reward-based preference based on $f(x) = \\log(x/(1-x))$ and defined by Hong et al. (2024, ORPO: Monolithic Preference Optimization without Reference Model).\n- Positive: margin with Gaussian. Suppose that $\\tilde \\ell_{\\theta} = \\ell_{\\theta} + c$ where $c$ is a constant and $\\ell_{\\theta}$ is the Gaussian log-likelihood preference. By extending our computations from Appendix E, Assumptions 4.2, 4.4, 4.5, and 4.7 hold with $c$-dependent positive constants. \nMargins are used by Meng et al. (2024, SimPO: Simple Preference Optimization with a Reference-Free Reward) and IPO from Azar et al. (2023, A General Theoretical Paradigm to Understand Learning from Human Preferences).\n- Negative: reference model with Gaussian. Suppose that $\\tilde \\ell_{\\theta} = \\ell_{\\theta} - \\ell_{\\theta_0}$ where $\\theta_0$ is known and $\\ell_{\\theta}$ is the Gaussian log-likelihood preference. Since $\\tilde \\ell_{\\theta}(x,y) = \\langle x-y, \\theta - \\theta_0 \\rangle$ and $\\nabla_{\\theta} \\tilde \\ell_{\\theta}(x,y) = x-y$, Assumption 4.5 is violated for $u=\\theta^\\star - \\theta_0$. Not all direct alignment algorithms rely on a reference model (see SimPO or ORPO).\n\n**2. Accelerated rate**\nAccelerated rates arise when accumulating random variables having a positive density at a specific point through a minimum (or maximum) operator. \n- When estimating the location parameter $\\theta$ of a uniform distribution over $[\\theta,\\theta+1]$, the optimal estimator achieving the accelerated rate is the minimum of uniform observations whose density is positive at $\\theta$.\n- For deterministic preferences with log-likelihood rewards, we observe the true ordering between likelihoods. This enforces a hard constraint on the admissible parameters, which can be expressed with a minimum operator. More precisely, the maximal deviation $R_{n,u}$ along direction $u$ is upper bounded by the minimum of positive random variables whose density is positive at zero under Assumption 4.7. With high probability, this min operator is upper bounded by $O(1/n)$. The proof combines Lemma 4.6 and an upper bound on the inverse of the cdf based on a Taylor expansion around $0$.\n\n**3. Misspecification**\nUnder misspecification, the deterministic preferences might not provide separability within $\\Theta$ since $\\theta^\\star \\notin \\Theta$. Then, DP MLE should be defined as SP MLE by using the 0-1 loss $1(u < 0)$ instead of the logistic loss $-\\log \\sigma(u)$. This combines a cross-entropy loss and a classification 0-1 loss, reweighted by a regularization $\\lambda > 0$. This objective is reminiscent of single-stage alignment procedures such as ORPO and ASFT, see Gorbatovski et al. (2025, The Differences Between Direct Alignment Algorithms are a Blur).\nWithout separability, computing DP MLE can be NP-hard. Under sufficient regularity, DP MLE converges to $\\theta_{0} \\in argmin_{\\theta \\in \\Theta} KL(\\theta^\\star,\\theta) + \\lambda m(\\theta)$ where $m$ as in line 270, where $\\theta_0 \\ne \\theta^\\star$. This minimization is challenging, as $\\theta \\to m(\\theta)$ might not be convex. Deriving a tractable ELBO method for this optimization is an interesting direction to obtain tractable and robust estimators. \nAs $\\theta_0$ lies in the boundary of $\\Theta$, we should control the maximal deviation wrt to $\\theta_0$ for directions that point towards the interior of $\\Theta$ to prove an accelerated rate. While some elements of our analysis might be salvaged, we believe that finer technical arguments should be derived to capture this interesting setting. \n\n**Restrictive assumptions**\nSee the answer to Reviewer vgrP for a detailed discussion on how to weaken them to local conditions.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No", "experimental_designs_or_analyses": "Yes, in Section 6", "relation_to_broader_scientific_literature": "It is generally related to machine learning community", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "## Strengths\n\nThe paper studies a very interesting problem, i.e. how does preference labels improve statistical estimation. The theoretical results are solid and also surprising overall, providing important insights on the benefits of preference.\n\n## Weaknesses\n\nAs mentioned by authors, the assumptions are quite restrictive and are only verified under simple setup. This is not a big issue, though.", "other_comments_or_suggestions": "see Strengths And Weaknesses part above", "questions_for_authors": "### 1.\nDo the results in Section 4 rely on the specific reward model $r_\\theta$? Does it have to be the log likelihood $\\log p_\\theta$? If not, can the authors provide some examples satisfying all the assumptions while supporting general reward functions? \n\n### 2.\nCan the authors provide more high-level intuitions behind the accelerated rate of $\\mathcal{O}(1/n)$? For examples, when the reward model is just log likelihood, the deterministic preference label only provides additional information on the magnitude relationship between $\\log p(x),\\log p(y)$. Why is this sufficient to improve the estimation error?\n\n### 3.\nIs it possible to consider misspecification case, i.e., $\\theta_*\\not\\in \\Theta$? In this case, $\\hat{\\theta}$ should converge to an optimal estimator in $\\Theta$. Would similar acceleration effects hold?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper studies shws how learning with preferences can improve parameter estimation rates for Gaussian and Laplace distributions. They show that for estimators based on hard preferences the error converges at a rate of O(1/n) and that this rate is mini-max optimal up to dimension and problem-dependent constants. The reviewers found the paper measured in its statements, the contributions novel, and well supported by theory and experiments. All reviewers agreed the work should be accepted. Given the widespread use of supervised fine-tuning and preference based learning to align LLMs, I do think some of the insights here can be translated into practical learning algorithms to improve LLMs -- this may be worth commenting on within the discussion in terms of the practical ramifications of this work. I recommend the authors use the extra page to incorporate vgrP's suggestions to reframe the writing for sections 1-3 with an emphasis on clarity since doing so should mitigate some of the questions raised by other reviewers as well.
Decision: Accept (spotlight poster)