Preferential Normalizing Flows
Authors: Petrus Mikkola
Luigi Acerbi
Arto Klami
Venue: NeurIPS 2024 poster (2024)
Keywords: normalizing flow
elicitation
random utility models
prior distribution
Primary area: probabilistic_methods
TL;DR: We show how normalising flows can be fitted for preferential data representing expert's choices between a set of alternatives, as a function-space maximum a posteriori estimate with a novel functional prior.

Abstract: Eliciting a high-dimensional probability distribution from an expert via noisy judgments is notoriously challenging, yet useful for many applications, such as prior elicitation and reward modeling. We introduce a method for eliciting the expert's belief density as a normalizing flow based solely on preferential questions such as comparing or ranking alternatives. This allows eliciting in principle arbitrarily flexible densities, but flow estimation is susceptible to the challenge of collapsing or diverging probability mass that makes it difficult in practice. We tackle this problem by introducing a novel functional prior for the flow, motivated by a decision-theoretic argument, and show empirically that the belief density can be inferred as the function-space maximum a posteriori estimate. We demonstrate our method by eliciting multivariate belief densities of simulated experts, including the prior belief of a general-purpose large language model over a real-world dataset.

Preferential Normalizing Flows
Petrus Mikkola, Luigi Acerbi, Arto Klami" Department of Computer Science, University of Helsinki first. last@helsinkki.....
Abstract
Eliciting nigh-dimensional probability distribution from expert via noisy judgments is notoriously challenging. yet useful for many applications, such as prior elicitation and reward modeling. We introduce method for eliciting the expert's s belief density as a normalizing flow based solely on preferential questions such comparing ranking alternatives. This allows eliciting in principle arbitrarily flexible densities, but flow estimation is susceptible to the challenge of collapsing diverging probability mass that makes it difficult in practice. We tackle this problem by introducing a novel functional prior for the flow, motivated by a -heooei argument, and show empirically that the belief density can be inferred as the function-space maximum a posteriori estimate. We demonstrate our method by eliciting multivariate belief densities of simulated experts, including the prior belief of general-purposs purpose large language model over a real-world dataset.
Introduction
Representing beliefs as probability distributions can be useful, particularly as prior probability distributions Bayesian inference - especially in high-dimensional, non-asymptotic settings where the prior strongly influences the posterior [Gelman et al. 2017] or as probabilistic alternatives to reward models [Leike al., 2018, Ouyang al. [022]. Our goal elicit a complex multivariate probability density from an expert, as a representation of their beliefs. By expert, we mean an information source with belief over a problem of interest, termed belief density. which does not permit direct evaluation or sampling. The problem is an instance of expert knowledge elicitation, where the belief is elicited by asking elicitation queries such as quantiles of the distribution O''Hagan 2019). The current elicitation literature (see Mikkola et al. 2023 for recent overview) focuses almost exclusively on extremely simple distributions, mostly products of univariate distributions known parametric form. Some isolated works have considered more flexible distributions, for instance quantil--arammttriiee distributions [Perepolkin et al. 2024] for univariate cases, or Gaussian processes [Oakley and O' Hagan 2007] and copulas for modelling low- www-dimensional dependencies [Clemen et al. 2000). but we want to move considerably beyond that and elicit flexible beliefs using modern neural network representations [Bishop and Bishop 2023 The main challenges identifying elicitation queries that are sufficiently informative to infer the belief density while being feasible for the expert to answer reliably, and selecting model class for the belief density that represent flexible beliefs without simplifying assumptions but that can still be efficiently estimated.
Normalizing flows are natural family for representing flexible distributions (Papamakarios et al. 2021]. When using flows for modelling density p(x), learning usually based on either a set of samples p(x) drawn from the distribution (density estimation; Dinh et al. 2014) or on the log density log p(x) evaluated at flow samples, x ~ q(x), in the variational inference formulation [Rezende and Mohamed, 2015]. Neither strategy applies to our setup, since we do not have the luxury of sampling from the belief density and obviously cannot evaluate it either. In addition to
"Equal contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) 10, w/o prior
(b) n 10, w/o prior
(c) n 10, w/ prior
(d) 100, w/ prior
Figure 1: Illustration of belief densities elicited from preferential ranking data by normalizing flow (contour: true density; heatmap: estimated flow; red: preferred points; blue: non-preferred points). (a)-(b): Typical failure modes of collapsing and diverging mass, when training flow with just n = 10 rankings. (c)-(d): The proposed functional prior resolves the issues, and already with 10 rankings we can learn the correct belief density, matching the result of the flow trained on larger data.
the well-known challenges of training normalizing flows, the setup introduces new difficulties; in particular, flexible flow easily collapses or finds way allocating probability mass in undesirable ways. Significant literature on resolving these issues exists [Behrmann et al. 2021 Salmona et al. 2022 Cornish et al., 2020], but conclusive solutions that guarantee stable learning are still missing. Our solution offers new tools for controlling the flow in low-density areas, and hence we contribute for the general flow literature despite focusing on the specific new task.
We build on established literature on knowledge elicitation for the interaction with the expert. Distributions are primarily characterized by their location and covariance structure, yet humans are notoriously bad at assessing covariances between variables [Jennings et al., 1982 Wilson 1994). However, human preferences, with potentially strong interconnections between variables, can be recovered by asking individuals to compare or rank alternatives, a topic studied under discrete choice theory [Triin] 2009]. The most studied random utility models (RUMs) interpret human choice as utility maximization with an additive noise component [Marschak] 1959]. To infer the correlation structure in human beliefs indirectly from elicitation data, we study a setup where the expert compares or ranks alternatives (events) based on their probability so that their decisions can be modeled by a RUM. In practice, this means that the data for learning the flow will take the form of choice sets C = fxi,... .... X2 } of candidates presented to the expert, combined with their choices indicating the preference over the alternatives based on their probability. We stress that candidates x are here not samples from the belief density but are instead provided by some other unknown process, such as an active learning method [Houlsby et al. 2011]. The only information about the belief density comes from the choice.
We are not aware of any previous works that learn flows from preferential comparisons. We first discuss some additional challenges caused by preferential data, and then show how we can leverage preferential structure to improve learning. Specifically, our learning objective corresponds a function-spant space maximum a posteriori (FS-MAP), where Bayesian inference is conducted on the function (flow) itself, not its parameters [Wolpert] 1993 Qiu al. 2024]. The learning objective is exact, in contrast to flow-based a algorithms that model phenomena involving discontinuities [Nielsen] al. 2020 Hoogeboom et al. [202]], such as the argmax operator the RUM model. By construction, the choice sets explicitly include candidates that were not preferred by the expert, carrying information about relative densities of preferred vs. not preferred points. This allows us to introduce functional prior that encourages allocating more mass to regions with high probability under RUM with exponential noise, solving the collapsing and diverging probability mass problem that poses a challenge for flow inference in small data scenarios.
In summary, we introduce the novel problem of inferring probability density from preferential data using normalizing flows and provide a practical solution. We model the expert's 's choice RUM with exponentially distributed noise, and query the expert for comparison or ranking of l: alternatives. We derive the likelihoods for k-wise comparisons and rankings and study the distribution of the most preferred point among i alternatives, which we term the k-wise winner. Based on the interpretation of the w-wis winner distribution tempered and tilted belief density, we introduce empirical

---

function prior and the FS-MAP objective for learning the flow. Finally, we validate our method using both synthetic and real data sets.
2 Why learning the density from preferential data is challenging?
Learning flows from small samples is challenging. especially in higher dimensions even when learning from direct data, such as samples from the density. Figure D illustratts two common challenges of collapsing and diverging probability mass; the illustration is based on our setup to showcase the proposed solution, but the same problems occur in the classical setup. The "collapsing mass" scenario is a form of overfitting, similar to mode collapse in mixture models [Li et al. 2007]. but more extreme for flexible models.
In the "diverging mass" problem, the model places probability mass in the regions of low probability.
The problem has connections to difficulties in training (Behrmann et al. 2021 Dhaka al., 2021
Vaitl et al., 2022, Liang et al.) [2022] and issues with coupling flows with increasing depth. which tend to produce exponentially large sample values [Behrmann al., 2021 Andrade 2024 One intuitive explanation is that we simply have no information on how the flow should behave far from the training samples, and an arbitrarily flexible model will least in some cases behave unexpectedly,
If already learning a flow from samples drawn from the density itself is difficult, is it even possible to infer the belief density from preferential data? For instance, for the most popular RUM model (Plackett-Luce: Luce| [959 Plackett 1975) we cannot in the noiseless case differentiate between the true density and any normalised positive monotonic transformation it:
Proposition 2.1 (Unidentifiability of a noiseless RUM). Let p+ be the expert's 's belief density. For k 2
2, let Drmnk == {x1= > X2 > > Xx be a k-wise ranking (see Definition 3.3). IfW Gumbel(O, 3
then for any positive monotonic transformation g holds lim P(D (Drami, rank g Pxy 3) = 1. Proof in B
In other words, the noiseless solution is not even unique and resolving this requires a way of quantifying the relative utility. Noisy RUM induces such a metric due to the noise magnitude providing a natural scale but even then the belief is identifiable only up to a noise scale; see A for a concrete example for the Mourstone-Mostelle model (Thurstone) 1927 Mosteller 1951]
Another new challenge is that the candidates x presented to the expert are given by some external process. In the simplest case, they are drawn independently from some unknown distribution A(x), which does not need relate to the belief density p.. We need a formulation that affords estimating P. directly. ideally under minimal assumptions on the distribution besides X(x) > 0 for (x( > 0.
Despite these challenges, explained next, in part by leveraging standard machinery in discrete choice theory to model the expert's choices and in part by introducing a new functional prior for the normalizing flow. The choice process separates the candidate samples x into preferred and non-preferred ones, and we can use this split to construct prior that helps learning the flow. That is, the preferential setup also opens new opportunities to address problems in learning flows.
can indeed learn flows as estimates of belief densities will be
3 Random utility model with exponentially distributed noises
The random utility model represents the decision maker's stochastic utility U the sum of deterministic utility and a stochastic perturbation [Train] 2009].
U(x)
fxx Wx).
where f X + R is a deterministic function called representative utility, and W is a stochastic noise process, often independent across x. The relationship between these concepts and the task will be made specific in Assumptions/ to We assume that the domain X' is a compact subset of Rd. Given set C C X' of possible alternatives, the expert selects specific opinion x C ttroough the noisy utility maximization,
X arg max Uxx).. X'EC
(2)
Definition 3.1 (choice set). Let A > 2. The choice set is set of k alternatives, denoted by Cs = (xi,..., } We assume that Cs a set of i.i.d. samples from probability density À over X. but note that the formulation can be generalized to other processes.
3

---

For example, if we ask for pairwise comparison C2 = (x, x'), the expert's answer would be x > x' f(x) + w(x) > f (xx) + w(x') for given a realization of W. We denote the random utility model with representative utility S, stochastic noise process W, and choice set C₃, by RUM(C J,W).
We make the common assumption of representing the probabilistic beliefs of (human) expert in logarithmic form [Dehaene 2003].
Assumption 1. f(x) log p. (x); noise is additive for log-density.
Assumption 2. bounded and continuous.
Inspired by Malmberg and Hössjer 2012 2.14]. we assume that the noise is exponentially distributed and thus belongs to the exponential family Azzri et al.) 2012). Assumption 3. (x)) Exp(s) independently for any x E X
With Assumption [ this corresponds to a model where the limit infinitely many alternatives, the expert chooses point by sampling their belief density (Corollary A.2). The parameter is here
precision parameter, the reciprocal of the standard deviation of Exp(s). There are two popular types of preferential queries [Fümkraaz and Hüllermeier 2011].
Definition 3.2 (k-wise comparison). A preferential query that asks the expert to choose the most preferred alternative from Ca is called k-wise comparison. The choice is denoted by x > Ca. Definition 3.3 (k-wise ranking). preferential query that asks the expert to rank the alternatives in C2 from the most preferred to the least preferred is called k-wise ranking. The expert's 's feedback is the ordering Xx(C.), ... Xx(C)), for some permutation
Note that the top-ranked sample of k-wise ranking is the same as the k-wise comparison choice, and the k-wise ranking can be formed as a sequence of k-wise comparisons by repeatedly removing the selected candidate from the choice set, as assumed in the Plackett-Luce model [Plackett 1975 Hence, common theoretical tools cover both cases.
3.1 The k-wise winner
The chosen point of k-wise comparison is central for reasons. First, its distribution provides the likelihood for inference when data come in the format of k-wise rankings or comparisons. Second, its distribution in the limit k - do offers insights for designing prior that mitigates the challenge of collapsing and diverging probability mass.
Definition 3.4 (k-wise winner). A random vector X3 given by the following generative process is called as k-wise winner.
1. Sample k-samples from A(x), and denote C2 = (Xi, X
2. Sample x from a Categorical distribution with support Cy and with probabilities given by RUM(Cs; log p. (x). Exp(s)).
The density of the k-wise winner is proportional to the k-wise comparison likelihood p(x C2 Cx). namely to p(x > Cs (CC)( The likelihood of the k-wise comparisons takes the following form. Proposition 3.5. Let Ca be choice set of k > 2 alternatives. Denote C = Cs \ {x} and fö.
ma fx2)). The winning probability of a point x E C₁ equals to
k-1
exp ( - s(1 + 1) max(fe - f(x),
P(x > Cs Cc)
Σ
-xxp(ee(f(x)-
1+1
xymx
(3)
where E Ex denotes the thh elementary symmetric sum of the set C.
Proof. See B
The w--e ranking likelihood is a product the k--isse comparison likelihoods where the winners sequentially removed from the choice and provided in Appendix as Equation (A.4).
In the limit of infinitely many comparisons, the w-wise distribution reduces to a tempered belief density tilted by the sampling distribution À [Malmberg and Hössjer 2012 2014).

---

0.4
p.
0.4
k=2
k-1
p
8-8
k-6
p
k=4
k3
p
k-2
0.0
0.0
-4
-2
4
-2
4
(a)
(b)
Figure 2: (a) The k-wise winner distribution converges to the belief density as - cc. (b) The k-wise winner distribution can be approximated by tempered belief density. For example. the tempered belief density with an exponent 1/5 approximates well the pairwise winner distribution.
Theorem 3.6. If is bounded and continuous, then the limit distribution of X as K - oo exists and its density is given by,
exp (sf((x) X(x)
p(x) = sexp (sf(x') X(x')
(4)
Proof. Apply Theorem 18.4 in |Malmberg and Hössjer 2014] to setting and note that the first
sentence in the proof of Theorem 18.4 incorrect. For random variable Y = -//s with X Exp(s) it holds that Y Ex.( However, for random variable Y = sX with X Exp(s) it holds that Y ~ Exp(1). Thus, the correct standardization is Y + +y.
3.2 The k-wise winner distribution as tilted and tempered belief density
Building on the definitions and theorems above, we now introduce the central idea of how model the belief density based on the k-wise winner distribution. The RUM precision parameter acts as a temperature parameter for the belief density, as p(x) x A(x)p, by Eq. (4). In general, there is no connection between A(x) and p. (x). but intuition can be gained by considering some extreme cases. For À = P. we have p(x) xx Pa (x)*+1, whereas for uniform A(x) and
limit distribution is the belief. This is also apparent from Corollary A.2 However, interest is in cases where & is finite.
For h < oc, forming the k-wise winner distribution requires marginalising over the choice set (Proposition 3.5). The formulas can be found the Appendix (Corollary [A.3), and do not have elegant analytic simplifications. However, they empirically resemble tempered versions of the actual belief illustrated in Figure
other words, finite i plays similar role as the RUM noise precision When resorting to k < the choice distribution (Eq. A.2) does not match the belief density for the true noise precision 8, but we can improve fit by selecting some alternative noise precision such that the choice distribution better approximates the belief. We will later use this connection to build a prior over the flow, and note that for this purpose we do not need an exact theoretical characterization: It is sufficient to know that for some choice of and the choice distribution resemble the target density, at least to the degree that can be used as a basis for prior information. Given that & is typically fixed, can be varied in the prior, implying that the further is from the 'optimal' value, the greater the prior misspecification.
The idea is empirically illustrated in Figure 2 The k-wise winner distribution for varying & is shown in Figure 2(a) where the belief density is a truncated standard normal on the interval [-5, 5), comparisons are sampled uniformly over the interval, and 1. As & increases, the k-wise winner distribution approaches the belief density (here 10 already very close), but we can equivalently approach the same density by changing the noise level (Figure 2(b)).
5

---

4 Belief density as normalizing flow
We model the belief density p. with a normalizing flow [Rezende and Mohamed 2015 Papamakarios
et al. 2021]. We introduce a new learning principle and objective for the flow which is compatible with any standard flow architecture, as long as affords easy computation of the flow density. A normalizing flow is an invertible mapping T from a latent space Z C R into a target space X' C R T consists of sequence of invertible transformations, so that the forward (generative) direction z + [z)) is fast to compute and the backward (normalizing) direction x + T (x) is either known in closed form or can be approximated efficiently.
The base distribution on 2 is a simple distribution such as a multivariate normal, whose density is denoted by Pz- If we denote the parametrized T by To given the flow network parameters Ø, the parameterized model of the log belief density, denoted by far can be written as,
fx(x) = log (x)) + log det Jr, (x)]
(5)
where - is the Jacobian of T What complicates the learning of fø in our case is the absence of a direct method to sample from p. (x), ruling out the conventional algorithms [e.g.. Papamakarios et al. 2021 Instead, we devise new learning objective explained next.
4.1 Function-space Bayesian inference
Our starting point to perform Bayesian inference for the flow network parameters given pref-
erential dataset D = ((x(0, cka) x(0) ck Ji=1 (k-wise comparisons) or DD = {(0(0) c( |
(1) is a permutation on cl Fimi (k-wise rankings),
x p(D o)r()):
where p(D ②) is the likelihood and p(0) is the prior for the flow network parameters. It is difficult to devise a good prior p(ő), and we instead perform inference over the function [Wolpert 1993
pffo DD & p(D fo)p(fo)
where P(D fo) is the preferential likelihood for comparisons, Eq. (3) or rankings, Eq. (A.4)
The function-space prior is easier to specify we can focus on the characteristics of the log belief density itself, not its parametrization. The function-span -asccion-space space prior is often evaluated at a finite set of representer points X (x1, ..,, x where m should chosen to be large to capture the behavior of the function at high resolution p (X)) [Wolpert 1993 Qiu et al. 20244. For example, when is a Gaussian process, the prior representer points in the posterior corresponds to the datapoints (e.g.. Equation 3.12 in Rasmussen and Williams, 2006). Following the considerations above, we construct prior knowledge of fo subset of datapoints.
4.2 Empirical functional prior
To address the issue of collapsing or diverging probability mass, we introduce an empirical functional prior whose finite marginals at winner points (x1...., x,, are independently distributed as
p(f) Pfna(() (f) II exp((.),
(6)
where f : (fx( (x,)) and Punit is an uninformative bounded (hyper) prior that guarantees that the functional prior is proper.
The functional prior Eq. (6) is special case of a class of priors, p(f) x II. A exp(sf,), derived from the following thession- argument under the exponential RUM model. Let us decompose the preference dataset into winners and losers Dj = Dj UD by defining Dj= == fx 3C, s.t. x > Ck and D : D Dj- The functional prior is the probability of observing only the k-wise winners.
p(f) p(D) f, s, A))
where Punir (f) x (when f is bounded, Assumption 2p. The idea to consider higher i or s (less noise) than the true ones, both choices make the density more peaked around the modes of p. (see

---

Figures 2(a) and 2(b).. This choice encourages the flow to place more mass on the winner points in a way that is consistent with the underlying decision model. We consider where s should be an increasing function of the true k. While setting k = reduces the functional prior to closed form Eq. ④ by Theorem 3.6] the normalizing constant remains difficult. However, for the special case of À < and 1, the normalizing constant equals one. We make this choice to retain computational tractability, reminding that the construct is only used as a prior intended for regularizing the solution and does not need to match the true density as such. This comes at the cost of increased prior misspecification, which can, in turn, degrade the quality of the fit, especially when the true value of lo is small (compare Figure 4(a) (k=2) versus Figure ((d) (k=5)).
oo and E (0, oc).
4.2.1 Function-space maximum posteriori
We train the flow To by maximizing the unnormalized function-spant posterior density of fø con- ditioned on the preferential data D = DD UDD using stochastic gradient ascent [Kingma and Ba 2014). Denoting all points in D by X and all winner points in D by Xx, the training objective is
Σ log L(D ((DIs((X)) (x)), 1o(s))
where L is the k-wise comparison or ranking likelihood as per Eqs. and (A.4). In the case of ranking data, the winner point x E X is the first-ranked alternative in each individual w-wise ranking, meaning that x is k-wise winner. A global optimum of Eq. (7) is the function-space maximum posteriori estimate of the belief density. Pseudo-codes for the overall algorithm (Algorithms I and the forward pass for the unnormalized log- posteriorr (Algorithms 22 are provided in the Appendix. The computational cost of training is similar to standard flow learning from equally many samples.
5 Experiments
We first evaluate our method on synthetic data with choices made by simulating the RUM model, to validate the algorithm in cases where the ground truth is known while covering both cases where the responses follow the assumed RUM model and where they do not. We then demonstrate how the method could be used in realistic elicitation scenario, using large language model (LLM) proxy for a human expert [Choi et al.) [0022]. As with a real human, an LLM is unlikely to follow the exact RUM model, but compared to a real user, the LLM expert tirelessly answer unlimited questions and possible ethical issues and risks relating to human subjects are avoided. LLMs carry their own biases and risks (Tiuatja et al.) 2024]. but the focus here on evaluating our algorithm. Code for reproducing all experiments is available at httpps:/ggthubb.co. wwwppperrs-mmkkkollaaa
Setup In the main experiments we use k-wise ranking with A = 5, using relatively few queries to remain relevant for the intended use-cases where the expert's capacity in providing the information is clearly limited. Since learning a preference of higher dimensions is more difficult, we scale the number of queries linearly with d but still stay substantially below the large-sample scenarios typically considered in flow learning. The details, together with the choice of the flow and the candidate distribution X, are provided below for each experiment. As flow model, we use RealNVP [Dinh] et al., 2017 when d = 2 and Neural Spline Flow (Durkan et al. [209]] when d > 2, implemented on top of (Simmer et al. 2023). For more details, see Appendix C.4
Evaluation We assess performance qualiatively via visual comparison of 2d and 1d marginal distributions between the target belief density and the flow estimate of the belief density, and quantitatively by numerically computing three metrics: the log-likelihood of the preferential data, the Wasserstein distance, and the mean marginal total variation distance [MMTV:[Acebi] 2020) between the target and the estimate. The numerical results are reported as the means and standard deviations of the metrics over replicate runs. As a baseline, we report the results of method that uses the same preferential comparisons and optimizes the same training objective, but instead of using a flow to represent exp(/) we directly assume the density is factorized normal distribution parameterized by means and (log-trans log-transformed) standard deviations of all dimensions. This exact method has not been presented in the previous literature, but was designed to validate the merit of the flow representation.
7

---

Table 1: Accuracy of the density represented as a flow (flo) compared to a factorized normal distribution (normal), both learned from preferential data, in three metries: log-likelihood, Wasserstein distance, and the mean marginal total variation (MMTV), Averages over 20 repetitions (but excluding a few crashed runs), with standard deviations.
log-likelihodd (1)
wasserstein
(4)
MMTV (L)
normal
flow
normal
flow
normal
Blo
Onemoon2D
-1.98 (#0.112)
-L.9 (40.12)
0.45 (4:0.04)
0.25 (=0.04)
0.30 (#0.00)
0.21 (+0.02)
Gaussian6D
-1.40 (#0.07)
-0.12 (40.02)
1.74 (4:0.06)
1.29 (=0.05)
0.20 (#0.01)
0.09 (400.01)
Twogaussians 10D
-3.99 (#0.06)
-0.09 (40.011)
7.31 (+-0.12)
2,60 (=0.06)
0.47 (#0.01)
0.08 (+00000
Twogaussians20D
-6,35 (=0.12)
-0.08 (20001)
11.07 (#0.15)
4.55 (=0.07)
0.47 (=0.00)
0.08 (40000)
Funnel10D
-2.21 (20.06)
-0.09 (20.01)
5.13 (4:0.04)
3.92 (=0.04)
0.27 (20. 000
0.18 (1:0001)
Abalone7D
-5.53 (=0.03)
-2.16 (20.12)
0.53 (4:0.00)
0.34 (=0.01)
0.26 (:0 (=0.00) 00)
0.29 (4:0.01)
1L
Denwaa
05
Figure 3: Cross-plot of selected variables of the estimated flow in the Abalone (left) and LLM knowledge elicitation experiment (middle), and the marginal density of the same variables for the ground truth density in the LLM experiment (right). See Figures C.6 and and C.7 for other variables.
5.1
Synthetic tasks
First, we study the method on synthetic scenarios. For the first set of experiments, we assume known density P. and simulate the preferential responses from the assumed RUM log p., Exp(1)). We consider five different target distributions: Onemoon2D, Gaussian6D, Twogaussians IOD, Twogaus- sians20D, and Funnel1OD. The densities of the target distributions can be found in Appendix C. For all cases we used 100d queries and À as a mixture of uniform and Gaussian distribution centered on the mean of the target, with the mixture probability 1/3 for the Gaussian; this technical simplification ensures a sufficient ratio of the samples to align with the target density even when d is high. Table [ shows that for all scenarios we can learn useful flow; all metrics are substantially improved compared to the method based on the normal model and visual inspection (Appendix C.5 confirms the solutions match the true densities well.
Abalone regression data. Having validated the method when the queries follow the assumed RUM model with a synthetic belief density, we consider a more realistic target density. We first fit a flow model to the continuous covariates of the regression data abalone [Nash et al., [995]. and then use the fitted flow as ground-truth belief density in the elicitation experiment. The elicitation queries correspond all k-combinations of the dataset size n 4177. The numerical results are again provided in Table [ Figure shows that the leamned flow captures the correlations between variables almost perfectly, which can be hard to see as the flow (heatmap) overlaps the true density (contour). There is some mismatch in the marginals, which is also indicated by the MMTV metric. In terms of the Wasserstein distance and visual comparison (Figure C.8), the flow based method clearly outperforms the baseline.
8

---

(a) Onemoon2D, k = 2, n 100 (b) Gaussian6D, k: = 2, n 1000
Figure 4: Illustration of belief densities elicited from pairwise comparisons by normalizing flow.
5.2 Expert elicitation with LLM as the expert
In this experiment, we prompt LLM to provide its belief on how the features of the California housing dataset [Pace and Barry 1997 are distributed. This resembles a hypothetical expert elicitation scenario, but the human expert is replaced with LLM (Claude 3 Haiku by Anthropic in March 2024, see Appendix C.2 for the prompt and detailed setup) for easier experimentation. From the perspective of the flow learning algorithm the setup is similar to the intended use-cases.
We query in total 220 k-wise rankings through prompting, where the alternatives C2 are uniformly sampled over the domain specified by 1st and 99th percentiles of each variable in the California housing dataset. The range was chosen to ensure ((z) covers approximately the support of the density, but avoiding outliers. While we lack access to the ground-truth belief density, we can compare the learned LLM's belief density the empirical data distribution of the California housing dataset, not known for the LLM. 3 such as the marginals of the variables AveRooms, AveBedrms, Population, and AveOccup are all correctly distributed on the lower ends of their ranges (which are very broad due to the uniform A(z)). Figure [ shows that the flow trained without the functional prior of (6) is considerably worse, confirming the FS-MAP estimate is superior maximum likelihood. While there might be multiple mechanisms for how the LLM forms its knowledge about this specific dataset [Brown et al. 2020), many of the features have clear intuitive meaning. For instance, houses are all but guaranteed to have only a few bedrooms, instead of tens.
that there is remarkable similarity between the distributions
5.3 Ablation study
We validate the sensitivity of the results with respect to the cardinality of the choice set k, the number of comparisons/rankings.
the noise level 1/8, and the choice of distribution À from which the candidates are sampled. In this section, we report a subset of the analysis for varying k while the synthetic scenarios (Section 5.1 by varying & E [2, 3. 5, 10} while keeping n fixed. We observe that the accuracy naturally improves through pairwise comparisons (k = 2) is shown Figure 4 for the Onemoon2D experiment with already
rest can be found in Appendix D Table 2 presents the results of experiments
function of k. The common special case in which the expert is queried
100 and the Gaussian6D experiment with roughly learn the target with & = 2 that is most convenient for a user, but naturally with somewhat lower accuracy. For further analysis and more details, see Appendix [ The main takeaway is that low values of or k; especially when n is large, can cause the flow estimate to become overly dispersed due to higher prior misspecification.
1000. The results indicate that
6 Discussion
Theoretical and empirical analysis validate our main claim: It is possible learn flexible distributions from preferential data, and the proposed algorithm solves the problem for some non-trivial but

---

Table Wasserstein distances for varying across different experiments
k: == 2
k: == 3
k 5
k: 10
Onemoon2D (n 100)
0.70 (#0000)
0.39 (#0.06)
0.17 (#0.03)
0.11 (+0.03)
Gaussian6D (n -100)
2.69 (#0030)
2.01 (#0.25)
1.46 (#0.11)
1.04 (40.04)
Funnel10D (n 500)
4.82 (40.12)
4.36 (#0.12)
3,96 (#0.05)
3.83 (40.04
Twogaussians 1OD (n = 500)
5.47 (#0.24)
3,81 (#0.26)
2.57 (#0.08)
2.20 (+0.02)
synthetic scenarios, with otherwise arbitrary but largely unimodal true beliefs. However, open questions worthy of further investigation remain the path towards practical elicitation tools.
The method is efficient only for exponential noise with # = that gives an analytic prior. Other choices would require explicit normalization energy-based modeling techniques (Chao al. 2024 For a given RUM precision we can, in principle, solve for k such that 8 becomes approximately correct due to the tempering interpretation (Figure 2) but there are guarantees that enough for any & sufficiently small for practical use, and this requires an explicit estimate of the noise precision. The ablation studies show that for fixed k increasing generally improves the accuracy and already fairly small n is sufficient for learning a good estimate (Table D.2. For very large n, the accuracy slightly deteriorate. We believe that this is due prior misspecification that encourages overestimation of the variation due to the fact that is finite but in the prior is assumed to be infinite. Figure D.4 confirms that for large the shape of the estimate is extremely close to the target density and the slightly worse Wasserstein distance is due to overestimating the width.
is good
We primarily experimented with k-wise ranging with k 5 and relatively few comparisons. However, we demonstrated that we can learn the beliefs with somewhat limited accuracy already from the most convenient case of pairwise comparisons (A: 2), which is important for practical applications. Finally, we focused on the special case of sampling the candidates independently from A(x). In many elicitation scenarios they could be result of some active choice instead, for example acquisition function maximizing information gain [MacKay] 1992]. The basic learning principle generalizes for this setup, but additional work would be needed for theoretical and empirical analysis.
Conclusions
The current tools for representing and eliciting multivariate human beliefs are fundamentally limited. This limits the value of knowledge elicitation in general and introduces biases that are difficult to analyze and communicate when the true beliefs do not match the simplified assumed families. Modern flexible distribution models offer a natural solution for representing also complex human beliefs, but until now we have lacked the means of inferring them from ecologically valid human judgements. We provided the first such method by showing how normalizing flows can be estimated from preferential data using functional prior induced by the setup. Our focus was in specifying the problem setup and validating the computational algorithm, paving way for future applications involving real human judgements. Despite focusing on the scenario where the elicitation judgements are made by a human expert, the algorithm can be used for learning flows from all kinds of comparison and ranking data.
Broader Impact Our goal is to eventually provide methods for accurately characterising human expert belief',, complementing the existing toolbox with techniques that make less restrictive assump- tions and hence support adaptation of better computational tools in broad range of applications. Knowledge elicitation tools are frequently used e.g. in becision-making and policy recommendations as assistive tools. For such applications, it is critically important to ensure that the mathematical tools are reliable and transparent, and further validation of the methodology is needed before the specific method proposed here could be used in critical applications.
Acknowledgments and Disclosure of Funding
The work was supported by the Research Council of Finland Flagship programme: Finnish Center for Artificial Intelligence FCAI, and by the grants 345811, 358980, 356498, and 363317. The authors acknowledge support from CSC - IT Center for Science, Finland, for computational resources.
10

---

References
Luigi Acerbi. Variational Bayesian Monte Carlo with noisy likelihoods. Advances in Neural Information Processing Systems, 33:821 -8222, 2020.
Daniel Andrade. Stabilizing training of affine coupling layers for high-dimensional variational inference. Machine Learning: Science and Technology, 2024.
Hossein Azari, David Parks, and Lirong Xia. Random utility theory for social choice. In Advances in Neural Information Processing Systems, volume 25, 2012.
Jens Behrmann, Paul Vicol, Kuan-Chich Wang, Roger Grosse, and Joern-Henrik Jacobsen. Un- derstanding and mitigating exploding inverses invertible neural networks. International Conference Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learn- ing Research, pages 1792-1800) PMLR, 2021.
Christopher M Bishop and Hugh Bishop. Deep learning: Foundations and concepts. Springer Nature, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin. Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33: 13:1877-1901 2020.
Chen-Hao Chao, Wei-Fang Sun, Yen-Chang Hsu, Zsolt Kira, and Chun-Yi Lee. Training energy-based normalizing flow with score-matching objectives. Advances in Neural Information Processing Systems, 36, 2024.
Kristy Choi, Chris Cundy, Sanjari Srivastava, and Stefano Ermon. LMPriors: Pre-trained language models as task-specifts priors. arXiv preprint arXiv:2210.125300 2022.
Robert T. Clemen, Gregory W. Fischer, and Robert L. Winkler. Assessing dependence: Some experimental results. Management Science, 46(8): 100-1115, 2000. ISSN 00251909, 15265501.
Rob Cornish, Anthony Caterini, George Deligiannidis, and Arnaud Doucet. Relaxing bijectivity constraints with continuously indexed normalising flows. International Conference Machine Learning, pages 2133-2143, 2020.
Stanislas Dehaene. The neural basis of the Weber-Fechned law: logarithmic mental number line. Trends in cognitive sciences, 7(4):445-147, 2003.
Akash Kumar Dhaka, Alejandro Catalina, Manushi Welandawe, Michael R Andersen, Jonathan Huggins, and Aki Vehtari. Challenges and opportunities in high dimensional variational inference. Advances in Neural Information Processing Systems, 34:7787-7798, 2021.
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using Real NVP. In International Conference on Learning Representations, 2017.
Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. Neural spline flows. In Advances in Neural Information Processing Systems, volume 32, 2019.
Johannes Fürnkranz and Eyke Hüllermeier. Preference Learning. Springer- Verlag New York, Inc.. 2011.
Andrew Gelman, Daniel Simpson, and Michael Betancourt. The prior can often only be understood in the context of the likelihood. Entropy, 19(10):555, 2017.

---

Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 34:12454- 12465, 2021.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745 2011.
D. Jennings, M. Amabile, and L.. Ross. Informal covariation assessment: Data-based vs. theory- based judgments. In Daniel Kahneman, Paul Slovic, and Amos Tversky, editors, Judgment Under Uncertainty: Heuristics and Biases, pages 211-230. Cambridge University Press, 1982.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint wrXiv:/4126698% 2014.
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: research direction. arXiv preprinl urXiv:181 181000777, 2018
Jia Li, Surajit Ray, and Bruce Lindsay. nonparametric statistical approach clustering via mode identification. Journal of Machine Learning Research, 8(8), 2007.
Feynman I. Liang, Michael Mahoney, and Liam Hodgkinson. Fat-tailed variational inference with anisotropic tail adaptive flows. In International Conference on Machine Learning, pages 13257- 13270. PMLR, 2022.
R Duncan Luce. Individual choice behavior, volume 4. Wiley New York, 1959.
David IC MacKay. Information-based objective functions for active data selection. Neural computa- tion, 4(4):590 1992.
Hannes Malmberg and Ola Hössjer. Argmax over continuous indices of random variables-an approach using random fields. Technical report, Technical report, Division of Mathematical Statistics, Department of Mathematics, Stockholm University, 2012.
Hannes Malmberg and Ola Hössjer. Probabilistic choice with an infinite of options: an approach based on random sup measures. Modern Problems in Insurance Mathematics, pages 291-312, 2014.
Jacob Marschak. Binary choice constraints on random utility indicators. Technical report, Cowles Foundation for Research in Economics, Yale University, 1959.
Petrus Mikkola, Osvaldo A. Martin, Suyog Chandramouli, Marcelo Hartmann, Oriol Abril Pla, Owen Thomas, Henri Pesonen, Jukka Corander, Aki Vehtari, Samuel Kaski, Paul-Christian Bürkner, and Arto Klami. Prior Knowledge Elicitation: The Past, Present, and Future. Bayesian Analysis, pages 1333, 2023.
Frederick Mosteller. Remarks on the method of paired comparisons: The least squares solution assuming equal standard deviations and equal correlations. Psychometrika, 16(1):3-9, 1951.
Warwick Nash, Tracy Sellers, Simon Talbot, Andrew Cawthorn, and Wes Ford. Abalone. UCI Machine Learning Repository, 1995.
Didrik Nielsen, Priyank Jaini, Emiel Hoogeboom, Ole Winther, and Max Welling. SurVAE flows: Surjections to bridge the gap between VAEs and flows. Advances in Neural Information Processing Systems, 13:55-6112 2020.
Jeremy Oakley and Anthony H'Hagan. Uncertainty in prior elicitations: A nonparametric approach. Biometrika, 94, 2007.
Anthony O' O'Hagan, Expert knowledge elicitation: Subjective but scientific. The American Statistician, 13(8 2019.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-277 277444, 2022.
12

---

R Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics & Probability Letters, 33 (3):291-297, 1997.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. The Journal of Machine Learning Research, 22(1):2617-2680, 2021.
Dmytro Perepolkin, Benjamin Goodrich, and Ullrika Sahlin. Hybrid elicitation and quantile- parametrized likelihood. Statistics and Computing, 34(1):11, 2024.
Robin L Plackett. The analysis of permutations. Journal of the Royal Statistical Society Series C: Applied Statistics, 24(2): 1932)2 1975.
Shikai Qiu, Tim GJ Rudner, Sanyam Kapoor, and Andrew G Wilson. Should we learn most likely functions or parameters? Advances in Neural Information Processing Systems, 36, 2024.
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian processes for machine learning, volume 2. MIT press, 2006.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning, pages 1530-1538. PMLR, 2015.
Antoine Salmona, Valentin De Bortoli, Julie Delon, and Agnes Desolneux. Can push-forward generative models fit multimodal distributions? Advances in Neural Information Processing Systems, 15:10766-1777, 2022.
Vincent Stimper, David Liu, Andrew Camphell, Vincent Berenz, Lukas Ryll, Bernhard Schölkopf, and José Miguel Herández--Lobato normflows: PyTorch package for normalizing flows. Journal of Open Source Software, 8(86):5361, 2023.
L. L. Thurstone. A law of comparative judgment. Psychological Review, 34(4):273- 286, 1927.
Lindia Tjuatja, Valerie Chen, Tongshuang Wu, Ameet Talwalkwar, and Graham Neubig. Do LLMs exhibit human-like response biases? A case study in survey design. Transactions of the Association for Computational Linguistics, 12:101 1-26,, 2024.
Kenneth E Train. Discrete choice methods with simulation. Cambridge university press, 2009.
Lorenz Vaitl, Kim A Nicoli, Shinichi Nakajima, and Pan Kessel. Gradients should stay on path: better estimators of the reverse-and forward KI. divergence for normalizing flows. Machine Learning: Science and Technology. 34400045 2022.
Alyson Wilson. Cognitive factors affecting subjective probability assessment, volume 4. Citeseer, 1994.
David H Wolpert. Bayesian backpropagation over io functions rather than weights. Advances in Neural Information Processing Systems, 6, 1993.
John I Yellott Jr. The relationship between Luce's choice axiom, Thurstone's theory of comparative judgment, and the double exponential distribution. Journal of Mathematical Psychology. 15(2): 109-144, 1977.
13

---

A Theoretical results
In Section 2 we mentioned that noisy RUM models can be identified for known noise levels. This can be illustrated by the following example:
Example A.1. Consider probit model (Thurstone-Mostellee model (hurstoone) 1927 Mosteller 1951]. Denote the probability mass function by p. its values at two points p(x) and p(x') and their difference by Ap - p(x) p(x'). For sufficiently large data set of pairwise comparisons, estimate the winning probability of x: P(x > x')
can Since noise follows [(0, a²), we can deduce that P(x x') Φ (Ap), where $ is the cumulative distribution function of N(, o2) So, P(x x') = g (A) and Ap
Tb 0² (q). Since p is the probability mass function, from pair of equations we obtain p(x) (1 $b (g))/2 and p(x') = (1 + 6 2 (2))/2 For the known noise level 2 p is identified.
Section refers the following corollary that relates the RUM model with sampling.
Corollary A.2. Consider that presenting an co-wise comparison with the choice set = X the expert is equivalent to presenting k-wise comparison with large k and points sampled uniformly over X. If the expert choice model follows RUM (X; p. (x), Exp(1)), then asking the expert to pick the most likely alternative out of all possible alternatives is equivalent to sampling from their belief density.
Proof. Since X(x)= 1/'ol( the terms X(x) and A(x') in Eq. cancel out. The denominator equals J p. (x)dx = 1, because f(x) = log P. (x) and 1. Thus, p(x > X) exp (sf(x)) (x).
Corollary A.3. For k- 2, the probability density of X equals to
px; (x) = 2X(x) / Flaploce(0.1/5 (xogp.(x) (x A(x')
(A.1)
For 2 < k < oc, the probability density of X is proportional to
Pxę (x) x(x)
P(x > Cn CaCc (x)),
(A.2)
X2-1
where P (x Cs C&) is given by Proposition 3.5
For 0o, the probability density of Xg equals to
Px; (x) (x)),
(A.3)
where > 0. If 1/vol(X) and 1. then C2(()) =
Proof. Case hs
(xç(x) x / p(x> x' Pr, (o, s(x'))
The normalizing constant can be computed by using Fubini's theorem. Since P(x x x' | p.) p.)A(x(A\()) is x X' integrable, it holds that
Je P(x x' I pe, ())(()))
f J P(x x' pe, ()x(())(x))xx(x)
0.5,
by the symmetry and the fact that P(x x' p., + P(x' > constant is 2. By Proposition 3.5 and after straightforward algebraic manipulations, ( > C. C.) simplifies Fraplace(0, (log p. (x) log p. (x')), where Fraphace is the cumulative distri- bution function of the Laplace distribution with a location parameter of 0 and a scale parameter of 1/8.
p.. 1. So. the normalizing
Case k It follows from Thcorem 3.6
14

---

k-wise ranking likelihood
The k-wise ranking likelihood P ((xx(C.), > > Xx(Ca)e Ccs)
can be computed as prod-
of k-wise comparison likelihoods,
k-1
II ==1
P xx(Ck)s > x(i) cli)
(A.4)
where (()) Ck: ..... ((k-1)= Cs (xT(Ch)) Xz(Cula-2)
B Proofs
This section provides the proof' for the propositions made in the main paper.
Proposition. Let p" be the expert's belief density. Denote N k - so that DN fx- > X2 > > > XN+1). IfW ~ Gumbel(O, 3), then for any positive monotonic transformation g. and for f = g p" holds,
P(DNlf) 8-+ 1.
Proof. Let f(x) (x)). Then, by Yellott Jr [177]]
N+1
(xx,)
N+1
(x,)) (xx.)
((DN(f.B)= = II
(B.1)
N+1 Zizn (xx))
iin eza(x
N+1
n=1
By the product law for limits,
N+1 II
e tolo (x..))
lim p(DN1f.8)= 8-0+
lim
0-
N ex,(
N+1
(xx)) = max g(p (x:))
uSiSN+1
N+1
II p"(xx) = max
p((xi)
a=1
N+1
II II [pppppppp)) (x,,) pp (x(xx)
==1
1.
The second equation holds because the softmax converges pointwise to the argmax in the limit of the temperature approaches zero. The third equation holds because g preserves the order. The fourth equation holds because p (x1) > p" (x2)
p² (xxN+1).
Proposition. Let Cs be a choice set of k 2 2 alternatives. Denote C = Cs \ {x} and fg ma f(x;). The winning probability of a point x E C₁ equals
exp ( s([ + 1) max - f(x),0))
P(x: > Ca Cx)
Σ
exp(=s([[(x -(xx)))
1+1
symxx EC
where Etoxx denotes the tth elementary symmetric sum of the set C.
15

---

Proof. Fix x E Ck2 and for any 2 0 denote 1w = #w2x))
P(x> Cs (x) P
n
( f(x)) + W ((2)))
P
n
[f(x)+++++)))))(
w f(x)) + (xx))) ((x) P(dW(x))
P
n (W(x)) < + +w-(x)))+++x)))) w se sww
0
EGEC
Se 8800
II (((W(()) < f(x) + - --xx)) w) dw
ZyEC
II (1---3((0)t0-1001))) / (2
Se 8W
m - ( w
3yEC
st -----((x {(xz)) dw ayEC
-((()))(((3))))
SE
Denote o₁ ;= - exp(-s((((( - f(x)))) Let by be the h elementary symmetric sum of the oj js. The 1th elementary symmetric sum is the sum of all products of distinct ej over js. We can write,
ksw
e - -==((((x)-((x))) dw
se
EGEC
k-
ksw
bie (k- 1 1) dw
se
1=0
bie 11- ksw dw
1=0
k
bi
s(1 1)e +(1+1) dw
s(1 + 1) J
s(1 + 1)c-<((+1) dw
1max(f2-f(x)p)))
bı(1 - G. (max (fc f(x),
by exp ( s(1 1) max ff f()),
1+1
with convention that 6o 1 and Gr, denotes the cumulative distribution function of Exp(n).
16

---

Algorithm Full algorithm
Algorithm 1SSPosterrio(()))
require: preferential data Ptull while not converged do sample mini-batch D Diull
require:
precision
input:
flow parameters é, mini-batch 'D
== matrix of
Aá x V (.PS-Posteeior())1 end while
X,- winner points of X
loglik - > log L(D [x(X),
logprior = (x())
return: loglik + logprior
C Experimental details
C.1 Target distributions
The logarithmic unnormalized densities of the target distributions used in the synthetic experiments are listed below.
| -
To +
Onemoon2D
2
0.2
2
0.3
0.4
0.4
Gaussian6D
2 (x
1) (x 1t), μ = 2
(-1)
Σ
0.4
T0
6
...
0.4
(-1)6)
0.44
0.4
10
log { exp (log.\(x 14, E1)) + exp (logN(x 14, (2)))
Twogaussians:
o²
2
pdz
2a
2
po²
po
0² = 0.9, d E {10,20) / = 31d, EΙ
Pa
po?
o²
pa
pa?
2
pa²
a²
2
po?
1)d-1
po²
-po²
σ²
po7
d pa²
2
2
o²
d 2
Σg
(-1)d-1 po
(-1)d-2 p0²
(-1)d-3 pa?
(ro
1)2 a²
(x; - 1)2 exp( (xpp(2hon)
Funnel10D
log(2n exp(2bxo))
+
3, 0.25
C.2 LLM expert elicitation experiment
The version of the Claude 3 model used in the LLM expert elicitation experiment was claude-3- haiku-20240307. In the experiment, used the following prompt template to query the LLM. The configurations specified within the XML tags <configuratio>> were sampled from X, a uniform distribution. The prompt template is as follows:
Data definition: <data>
California Housing
17

---

We collected information on the variables using all the block groups in California from the 1990 Census. In this sample block group on average includes 1425.5 individuals living in a geographically compact area. Naturally, the geographical area included varies inversely with the population density. We computed distances among the centroids of each block group measured in latitude and longitude. This dataset was derived from the 1990 U.S. census, using one row per census block group. A block group is the smallest geographical unit for which the U.S. Census Bureau publishes sample data (a block group typically has population of 600 to 3,000 people). A household is group of people residing within a home.
Number of Variables: 8 continuous
Variable Information:
MedInc median income (expressed in hundreds of thousands of dollars)
in block group
HouseAge median house age in block group
AveRooms average number of rooms per household AveBedrms average number of bedrooms per household
Population block group population
AveOccup average number of household members
Latitude block group latitude Longitude block group longitude </data>
The variables are: <variables>
(MedInc, HouseAge, AveRooms, AveBedrms, Population, AveOccup, Latitude, Longitude)
</variables>
always reported in this order.
Given these combinations of variables below, please list them from most likely to least likely in your opinion. Consider what each variable represents and its realistic value in light of the properties of the dataset.
<configurations>
A=0. 0.1.::000..0.01.0110000000000000 B=0. |=0.000..0.2222220..000000001110 C=0. 01200000000000000000000000010000
D=0. 39, oee.s908..s222200000000...111110 
E-0. 01t.a0/..0....100000000000000000
</configurations
<task>
1. First, think your answer step by step, considering the model and data definition in detail.
2. Then discuss each combination separately in light of your thoughts about data. Do not assign ordering yet.
3. Finally, summarize all your considerations.
4. At the end, write your final ordering comma-separated
list of letters within an XML, tag <order></order>. </task>
18

---

Table C.1: Accuracy of the density represented as flow (flw) compared factorized normal distribution (normal), both learned from preferential data, in three metries: log-likelihood, Wasserstein distance, and the mean marginal total variation (MMTV). Averages over 20 repetitions (but excluding a few crashed runs), with standard deviations.
log-likelihood (1)
wasserstein
(4)
MMTV (L)
normal
flow
normal
flow
normal
flow
Abalone7D
-5.53 (4-0003)
-2.16 (=0.12)
0.53 (#0.00)
0.34 (#0.01)
0.26 (+0.00)
0.29 (+0.01)
-5.25 (4:0077)
-3.52 (=0.09)
1.05 (#0.00)
0.65 (+0.01)
C,3 Modified Abalone7D experiment
By modifying Abalone7D experiment, we can consider a synthetic technical validation constructed so that the data distribution is more realistic. We do this by mis-using regression data set so that the response variable is interpreted as indication of preference and the queries are formed by presenting the expert choice between different samples. If we denote by g(x;) the regression function for the ith covariate set, then X; is chosen over x, if (xx) > g(x;). We remark that the task itself is not particularly interesting as the response variable does not correspond to any real belief (instead, we learn a distribution over the covariates for which the response variable is high), but it is still useful for validating the algorithm we now need to cope with choice sets that do not match any simple distribution X(x). Instead, the choice sets are now formed by uniform sampling over the sample indices, which means they are drawn from the marginal distribution of the covariates. Note that this is different from the target density, which is the density of covariates for samples with high response variables.
In the experiment, we do not assume any noise on the expert response. Hence, the expert follows a noiseless RUM with the representative utility gxx;) equals to the response variable of äth covariate x;- This means that the choice distribution resembles Dirac delta function at the points with the highest response variables. For this reason, cannot compute the MMTV metric it involves integrating over the absolute differences of the marginals, which leads to numerical issues. The numerical results provided in Table C. and the visual results in Figure C.I
C.4 Other experimental details
Hyperparameters. In all the experiments, we use the value 1 in the preferential likelihood regardless of how misspecified is with respect to the ground-truth model. Neural Spline Models have 2 hidden layers and 128 hidden units. The number of flows is 6, 8, or 10 depending on the problem complexity. RealNVP models have 4 hidden layers and 2 hidden units. The number of flows is 36 when the number of rankings is more than 50, and 8 otherwise. Other architettue-sspecfii details correspond to the default values implemented in the normflows package, a Py TyToch-based implementation of normalizing flows (Stimper et al. 2023].
Optimization details. Models are trained for a varying number of iterations from 10 to 5 x 10 with the Adamax optimizer (Kingma and Ba. 2014] and varying batch size from 2 to 8. The learning rate varies from 10 5 to 10 depending on the problem dimensionality, with higher learning rates for higher-dimensional problems. A small weight decay of 10-6 was applied.
server with nodes of two Intel Xeon processors, code name Cascade Lake, with 20 cores each running at 2.1 GHz. Double precision numbers were used improve the stability of the training process. We did not explicitly record the training times or memory consumption, but note that the considered data sets and flow architectures are all relatively small.
Computational environment. Models are trained and evaluated
Experiment replications. Every experiment was replicated with 20 different seeds, ranging from 1 to 20, but a few replications failed due to not well-known reasons, sometimes due to memory issues and sometimes due to numerical instabilities that led the replication to crash. The results are reported over the completed runs. In the main experiment table (Table []. there was one failed replication in the Twogaussians20D experiment and two in the Onemoon2D experiment.
Dataset licence: Abalones: (CC BY 4.0) license, original source Nash et al. 1995
19

---

Ro2
035
0.54
6.8
Kww vie!
60
62-
02-
13
Figure C.I: Full result plot for the modified Abalone7D experiment, where the target (unnormalized) belief density corresponds to the abalone age.
C.5 Plots of learned belief densities
Figure 3 presented subset of the cross-plots for the multivariate densities for the Abalone regression data and the L.M experiment. Here, provide the complete visual illustrations over the full density for both, as well as corresponding visualisations for all of the other experiments in Figures C.2 to C.8
20

---

Figure C.2: Gaussian6D experiment. The target distribution is depicted by light blue contour points and marginal by a pink curve. The learned flow depicted by dark blue contour sample points and its marginal by black curve.
Figure C.3: Estimated belief density for the Onemoon21 data. See Figure 1 on the same density.
for other visualisations
21

---

Figure C.4: Twogaussians 10D experiment. The target distribution is depicted by light blue contour points and its marginal by a pink curve. The learned flow is depicted by dark blue contour sample points and its marginal by black curve.
22

---

Figure C.5: Estimated belief density for the Funnel10D data. The narrow funnel dimension (x1) is extremely difficult to capture accurately, but the flow still extends more in that dimension, seen as clear skew in all marginal histograms.
23

---

Jg
h
Figure C.6: Full result plot for the LLM expert elicitation experiment, complementing the partial plot presented in Figure 3
62
Vheee siight
65
1eigh
Hagg
View or
Figure C.7: Full result plot the Abalone7D experiment, complementing the partial plot presented in Figure The target distribution is depicted by light blue contour points and its marginal by pink curve. The learned flow is depicted by dark blue contour sample points and its marginal by a black curve.
24

---

4.45
Khick vitet
64-
62-
62
13
13
10
10
Figure C.8: Full result plot for the Abalone7D experiment for the baseline method.
25

---

price (ALE)
Ah
Learmned
Mm
Figure D.I: The effect of the functional prior on the learned belief density in Experiment 5.2 The left plot corresponds to learning the LLM's belief density using maximum likelihood in the traming, and the right plot to using function-space maximum a posteriori with the proposed functional prior. We hypothesize that the extreme marginals (e.g. median income) obtained from maximum likelihood estimation are due to problems with collapsing or diverging probability mass.
Table D.I: The means of the variables based on (first row) the distribution California housing dataset and the sample from the preferential flow fitted to the LLM's 's feedback trained (second row) with the likelihood only and (third row) with the both likelihood and prior.
MedIne HouseAge AveRooms AveBedrms Population AveOccup
Lat
Long
True data Flow w/ prior
3.87
28.64
5.43
1.1
1425.48
3.07
35.63 -119.57
9.83
43.01
125.18
8.07
22983.76
1290.0
28.81
-117.94
Flow w/o prior
5.91
27.19
6.28
1.58
2868.52
3.37
36.43 -119.75
D Ablation studies
This section reports additional experimentation to complement the results presented the main paper. Unless otherwise stated, the rest of the details in the experiments are as discussed in Sections aaad The only exception the number of flows, which are scaled by the number of rankings n to increase flexibility in line with the available data. However, when is as in the main paper, the number flows remains unchanged.
C.4
D.1 Effect of the functional prior
Figure D.I shows the effect of the functional prior for the LLM experiment, showcasing how the maximum likelihood estimate learning the flow without the functional prior exhibits the diverging mass property. Table summarizes the densities in a quantitative manner by reporting the means for all variables. The table shows how the solution without the prior be massively off already in terms of the mean estimate, for instance having the mean number of rooms at 125.
D.2 Effect of the noise level 1/s
Figure D.2 investigates the interplay of the true RUM noise and the assumed noise in the preferential likelihood on the Onemoon2D data.
26

---

Strue 0.01
Strue 1.0
Strue 5.0
Stsk 0.01
Slik 1.0
Stil 5.0
Figure D.2: Preferential flow fitted via FS-MAP with varying precision levels in the data generation process (in RUM) Strue, and precision levels in the preferential likelihood Stck- The first column shows that a lower precision level in RUM leads to more spread fitted flow, as expected. The middle plot the only scenario where both the likelihood and the functional prior are correctly specified, resulting in the best result. Since the prior is misspecified in the bottom-right plot, the best results are achieved, contrary to expectations. However, this misspecification does lead catastrophic performance deterioration but rather to a more spread-out fitted flow.
D.3 Effect of the candidate sampling distribution
We validate the sensitivity of the results in terms the choice of the distribution that the candidates sampled from. Figure D.3 studies the effect of A, the unknown distribution from which the candidates to be compared are sampled from, complementing the experiment reported in Section 5. and confirming the method robust for the choice. In the original experiment the candidates were sampled from a mixture distribution of uniform and Gaussian distribution centered on the mean of 1/3 for the Gaussian. Figure D.3 reports the accuracy as
the target, with the mixture probability a function of the choice of w for one of the data sets (Onemoon2D), so that goes from uniform to Gaussian, and includes also an additional reference point where İ equals the target. For all wr > 0.5 we reach effectively the same accuracy when sampling from the target itself, and even for the the distance significantly smaller than the reference scale
hardest case uniform À (with comparing the base distribution with the target.
D.4 Effeet of the number of rankings
We validate the sensitivity of the results in terms of the number of comparisons/rankings Table D.2 as well as Figures D.4 and D.5] report the results of an experiment that studies the effect n.
Increasing generally improves the accuracy and already fairly small n is sufficient for learning good estimate (Table For very large n, the accuracy can slightly deteriorate, We believe that this is due to prior misspecification that encourages overestimation of the variation due to the fact that & is finite but in the prior it is assumed be infinite. the Onemoon2D experiment, Figure 1000 the shape of the estimate is extremely close and the slightly worse Wasserstein distance is due to overestimating the width. The same holds for other experiments such Twogaussians LOD illustrated in Figure D.5
D.4 confirms that for
27

---

Figure D.3: The Onemoon2D experiment replicated for varying sampling distributions À from where the candidates are sampled. In original Onemoon2D, a mixture of uniform and Gaussian distribution centered the mean of the target, with the mixture probability Here, À E {Uniform, Gaussian- Uniform Mixture, Target) with letting the mixture probability w vary in {0.1, 1/4, 1/3, 1/ 2/3, 3/4, 1.0]. The of the details can be found in Section specifically n. = 200 and k = 5. The distance between the base density and the target density (1.85) provides scale reference. The method is robust for the sampling distribution and for broad range of w we reach essentially the same accuracy as when sampling from the target itself.
1/3 for the Gaussian.
Distance between flow and target density
1.75
1.50
Wasserstein distance
target density
1.25
1.00
0.75
Experiment
0.50
0.00
Uniform
0.1
0.250.33
0.5
0.66 0.75
1.0 Target
probability Caussian-uniform mixture
Table D.2: Wasserstein distances for varying n (fixed & = 5) across different experiments
7
25
50
100
1000
Onemoon2D
0.67 (#1.34)
0.18
(#0.04)
0.17 (#0.08)
0.23 (+0.02)
Gaussian6D
1.70 (+0.22)
1.50 (#0.19)
1.46 (+0.11)
1.26 (+0.04)
50
500
2000
10000
Funnel1OD
4.33 (0).10)
3.96 (#0.05)
3.89 (#0.04)
3.92 (4:0.04)
Twogaussians 10D
2.69 (+0.31)
2.57 (#0.08)
2.61 (#0.05)
2.66 (#0000)
(a) n 25, 1: = 5
(b) 100, h: = 5
(c) 1000, A:
Figure D.4: The estimated belief densities in the Onemoon2D experiment of Table D.2 (contour: true density: heatmap: estimated flow). While the coverage of the estimated density with 1000 is better than with n = 100, there is more spread with n 1000 than with = 100, which explains the slight performance deterioration Table
28

---

(a) n 50, A: 5
(h) n = 500, A: =
(c) n= 10000, = 5
Figure D.5: Cross-plots of selected variables of the estimated flow in the Twogaussians 10D exper- iment of Table D.2 (contour: true density; heatmap: estimated flow). While the coverage of the estimated density with n = 10000 is better than with n = 500, there more spread with n 10000 than with 500, which explains the slight performance deterioration in Table D.2
D.5 Effect of the cardinality of the choice set k
we rerun the LLM expert elicitation experiment with k = 2. Figure D.6 shows that the LLM expert also works with & = We replicated the original experiment conducted with k and report the estimates side-by-side, visually confirming we learn the essential characteristics of the distribution in both cases. The results are not identical and the case of k 5 is likely more accurate (see e.g. the marginal distribution of the last feature), but there are major qualitative differences between the two estimates.
Finally, complement the ablation studies for k synthetic settings in Section 5.3
Leamned LUM prior Dow5 2222
2025
LEEl
Figure D.6: The LLM expert elicitation experiment replicated for the setting of pairwise comparisons (left) and compared the original setting of 5-wise rankings (right). The estimated flow remains qualitatively the same for the variables shown here (other variables omitted due to lack of space), and this holds true for the rest of the variables as well.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract lists the main claims and they are summarized at the end of the Introduction, and the paper includes both theoretical analysis and empirical experiments verifying the claims.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The limitations are discussed in Section 6, titled Discussion. We explain technical limitations and limitations of the empirical validation, and explain in detail what kind of additional validation would be useful in addressing the limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
30

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We clearly and explicitly state the full set of assumptions, the formal definitions the key concepts, and provide three propositions and theorems that characterize the method's 's properties. The full proofs are provided in Appendix, with key intuitions explained in the main paper.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide sufficient details for reproducing the experiments. The newly proposed algorithm is described via concrete mathematical expressions and pseudocode. Majority of the empirical experiments are conducted on either simulated or publicly available data, and we provide description of the simulation process (in Appendix) and reference for the public data sets, and the code release allows replicating the experiments in whole. For the LLM experiment we provide the exact prompts and make available the LLM responses in the code release, but we note that from the perspective of these experiments the specific LLM or data is not particularly critical; the data is used for verification of the algorithm.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
31

---

(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide code that includes both implementation of the algorithm and scripts for re-running the experiments. Code is available at hatps://gihub..om/ppruug mikkola/prefflow
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wttpcc/ppppppp.....eeeee Seciiccgguiss///e /Coeee//emmomdmmiiicaa ssiinni asionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the information in writing and the same information can addition- ally be found in the code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
32

---

Answer: [Yes]
Justification: We do not make direct comparisons for which testing statistical significance would be applicable, but do report all numerical values with uncertainty intervals over multiple runs and disclose the details on how this was done.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We explain the used computational environment used to run the experiments in Section C "Experimental details". We state that the computational cost of the new algorithm is similar to broad range of related work, but we did not store exact exccution times or memory requirements during experimentation and hence do not report them. The core research questions in the paper are about learning models from very limited data and the question of computational resources is not particularly important for the work.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Fthics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:://haaaarppp.cpp/c/ccecccca ittppubiiiicciiicccciiiieeeeee
Answer: [Yes]
We follow the Code of Ethics as instructed and have not identified any risk of violations.
Justification:
Guidelines:
33

---

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: These aspects are briefly discussed at the end of Conclusions, in a clearly marked paragraph. As methodological contribution our work does not raise new concerns of negative societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our experimentation and data sets technical in nature and bear risk of misuse. They are only used for technical validation of the algorithms and we do not release data pretrained models relating real-world scenarios.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
34

---

12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We adequately cite all sources and follow the licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papprswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out the asset's ' creators.
New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We do not provide assets, other than releasing technical data for repro- ducibility purposes.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Even though our method designed eventually be used also in applications that involve human subjects (expert knowledge elicitation), we do not use any data relating humans in this paper but instead evaluate the method in simulated scenarios and by using a large language model as a proxy for a human expert.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
35

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Not required due using any data related human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "cDi7dkJDaI", "reviewer": "Reviewer_sR2g", "summary": "The paper presents an approach for expressing an expert's belief density using a normalizing flow. Interestingly, the flow is trained solely on preferential questions (e.g., comparing and ranking) which follow a random utility model (RUM). The approach avoids several optimization issues that could occur when trying to model the more complex underlying data. The authors method relies on an approach that defines a functional prior over the preference data. The authors demonstrate their method on both simulated and real-world data, showing a very promising method for representing the belief density of expert opinions.", "questions": "The Appendix explores various settings of the parameter $s$ when the true value of s varies, which shows that their choice to fix $s=1$ is reasonable. However, the authors almost mention using using $\\lambda \\propto 1$ as well to maintain tractability. The implications of this choice are unclear, did the authors run any validity checks to test the implications of their choice?", "limitations": "In practice collecting an training dataset of expert feedback is laborious, and often many experts are needed in order to collect enough data. The paper only considers the case of a single expert, with a flow model to that experts belief density. The authors may wish to comment on future work in this area — specifically, if their approach could be adapted to a multi-expert setting, if a single flow model is still sufficient or if a flow mixture model is preferred.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The authors overcome a unique challenge in modeling expert's beliefs with flows, namely that the true data distribution of experts beliefs $p^*$ cannot be directly sampled from. Instead, they sample from preference data $D$ and choose a specific function-space prior for the flow in order to model the likelihoods of the k-wise comparisons. This unique solution results in a flow that properly places mass on the winner points following the underlying decision model of the data. The experimental results confirm their hypothesis, showing much better results as a result of their prior.\n\nAdditionally, The use of LLMs as experts in their experiments is clever. The approach is reasonable and replicable, while also demonstrating and more end-to-end application that would be of interest to the broader community.", "weaknesses": "One primary advantage of normalizing flows is that they offer exact likelihood estimation. Because of the underlying data model for preference data (eq. 3), I suspect that the method here is actually optimizing a lower bound on the log likelihood. This may explain the optimization challenges and need for fixing the prior's precision $s=1$ described on lines 228-230. Specifically, their objective may be similar to the Max surjective flow described in Nielsen et. al. [2020, https://arxiv.org/abs/2007.02731]. In either case, the authors may want to clarify if their objective function is a lower bound or not, since adapting flows for non-bijective mappings is also a topic of interest to many.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">...Because of the underlying data model for preference data (eq. 3), I suspect that the method here is actually optimizing a lower bound on the log likelihood...\nSpecifically, their objective may be similar to the Max surjective flow described in Nielsen et. al. [2020, https://arxiv.org/abs/2007.02731]...\n\nThere are some similarities between the Max surjection transformation and the preferential likelihood given by (noiseless) RUM, which is essentially an argmax operator. However, the main difference is that the Max surjection transformation is a transformation that can be applied to any level of a composable transformation (denoted by \\\\(T\\\\) in the manuscript, by \\\\(f\\\\) in (Nielsen et. al., 2020)) with available likelihood contribution (Table 2, Nielsen et. al., 2020), while the RUM model conditions on the whole \\\\(T\\\\) and computes a (non-additive) likelihood contribution for the whole preferential flow.\n\nIn SurVAE Flows, the likelihood contribution of a deterministic bijective flow is the likelihood of a datapoint \\\\(\\mathbf{x}\\\\) given by the flow density \\\\(p\\\\) at that point, that is \\\\(p(\\mathbf{x})\\\\) computed using the change of variable formula, which results in the Jacobian of \\\\(T^{-1}\\\\). In contrast, a preferential flow uses a probabilistic modelling approach by having an additional likelihood function \\\\(\\mathcal{L}\\\\) which computes the likelihood contribution of (preference) data \\\\(\\mathbf{x}\\\\) conditioned on the flow density \\\\(p\\\\), that is \\\\(\\mathcal{L}(\\mathbf{x} \\mid p)\\\\). In a preferential flow, although the mapping \\\\(T\\\\) from a latent point \\\\(\\mathbf{z}\\\\) to a datapoint \\\\(\\mathbf{x}\\\\) is a deterministic bijective function, the likelihood contribution of preference data does not directly involve the Jacobian of \\\\(T^{-1}\\\\), unlike in SurVAE Flows (Algorithm 1, Nielsen et. al., 2020). \n\nThus, the likelihood contribution is exact and the objective is not a lower bound. We extended the discussion section of the manuscript to discuss the relationship between SurVAE Flows and preferential flows.\n\n> The Appendix explores various settings of the parameter \\\\(s\\\\) when the true value of \\\\(s\\\\) varies, which shows that their choice to fix \\\\(s=1\\\\) is reasonable. However, the authors almost mention using using \\\\(\\lambda \\propto 1\\\\) as well to maintain tractability. The implications of this choice are unclear, did the authors run any validity checks to test the implications of their choice?\n\nWe indeed use \\\\(\\lambda \\propto 1\\\\) for construction of the functional prior, but note that in all of the experiments we used as \\\\(\\lambda(x)\\\\) a mixture of a bounded uniform and a Gaussian. That is, we already conducted the experiments under conditions where the choice does not hold.\n\nTo further investigate the effect of the true sampling distribution, we now conducted an additional ablation study, where the true \\\\(\\lambda(x)\\\\) is varied. See the global response for description of the experiment and summary of the results confirming that the method works for a range of choices.\n\n>...The paper only considers the case of a single expert, with a flow model to that experts belief density. The authors may wish to comment on future work in this area — specifically, if their approach could be adapted to a multi-expert setting, if a single flow model is still sufficient or if a flow mixture model is preferred.\n\nThis is a good idea and we will extend the Discussion to cover also multi-expert settings.\n\nThe most common methods for aggregating expertise of multiple experts are *behavioral aggregation* and *mathematical aggregation*. The former is based on experts discussing their opinions and making consensus judgments for which an aggregate distribution is fitted (O'Hagan, 2019). This is agnostic to the elicitation algorithm and our method with a single flow could readily be used to capture the consensus. For *mathematical aggregation* we need a *pooling rule* that combines the elicited densities into a single one (EFSA, 2014). We think that a mixture of flows, which is equivalent to a linear pooling over multiple elicited flow densities, with equal mixture weights, would be a good default option. Further, Bayesian pooling such as having a hierarchical model on the joint preferential data from multiple experts could be investigated. \n\n**References**\n\nEFSA, E. F. S. A. (2014). “Guidance on Expert Knowledge Elicitation in Food and Feed Safety Risk Assessment.” EFSA Journal, 12(6): 3734. 2, 16\\\\\nO’Hagan, A. (2019). Expert knowledge elicitation: subjective but scientific. The American Statistician, 73(sup1), 69-81.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7D8MZ1OrF3", "reviewer": "Reviewer_FQE4", "summary": "The paper proposes a method to learn so-called belief densities based on k-wise rankings or comparisons of alternatives. This belief density is learned by combining function-space Bayesian inference and normalizing flows, which allows for the learning of complex (multivariate) probability densities. The authors evaluate their method empirically by studying synthetic scenarios, a regression task and a “real” use-case by querying an LLM and evaluating how close the resulting flow is to the ground truth density by three metrics.", "questions": "Is assumption 3 satisfied in the LLM experiment? If not, how does this affect the results?", "limitations": "The paper mentions fair limitations in the discussion section.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper addresses an interesting and very relevant problem, especially as human feedback has recently been used a lot in the training of LLMs. The paper has a strong theoretical contribution proposing a method how to learn the belief density using normalizing flows from rankings while preventing failure modes by using a functional prior.  The paper is well written and the authors clearly disentangles their contribution from the work in related works.", "weaknesses": "The experiments could have been more varied. At the moment all experiments use k = 5 and it would be interesting to see how the results vary with different values of k. Specifically, k = 2 is a very relevant real-life use case as this type of feedback is given a lot in e.g. chatbots. Similarly, it would be good to get some idea of how the flows converge to the ground truth density as a function of the number of preferential samples.\n\nMinor points and typos:\nFig. 1 Can you make the numbers larger? They are hard to read even when zooming in and without zooming in it’s not clear that they are numbers\nl. 78 “Diverging” -> “diverging”\nl. 94 “Mollester” -> “Mosteller”\nl. 201 “to Bayesian inference” -> “to use/do, etc. Bayesian inference”\nl. 201 “given a preferential” -> “given preferential”\nl. 405 citation ends abruptly \nl. 458 “probality” -> “probability”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> ...At the moment all experiments use k = 5 and it would be interesting to see how the results vary with different values of k. Specifically, k = 2 is a very relevant real-life use case as this type of feedback is given a lot in e.g. chatbots. Similarly, it would be good to get some idea of how the flows converge to the ground truth density as a function of the number of preferential samples.\n\nThank you for the excellent suggestion. We re-ran both Onemoon2D with \\\\(k=2,3,5,10\\\\) and LLM experiment with \\\\(k=2\\\\) to \\\\(k=5\\\\), as explained in the global response. The key result is that the method indeed works already with \\\\(k=2\\\\), though naturally not as well. We also studied the convergence as a function of \\\\(n\\\\), again describing the experiment and the results in the global response.\n\n\n> Minor points and typos:\n\nThank you for pointing out these errors. We corrected the errors for the revised manuscript and increased the font size of Fig. 1. \n\n> Is assumption 3 satisfied in the LLM experiment? If not, how does this affect the results?\n\nThe RUM model with this specific noise distribution serves as a theoretical model for the expert's choice, so it is highly unlikely that assumption 3 is satisfied in the LLM experiment. In fact, it most likely does not hold exactly for humans either. Hence this experiment shows that even under model misspecification that can be expected also in the real use-cases we can get sensible results. We hypothesize that a potential violation of assumption 3 can have similar effect as misspecification of the noise level, which can result in too high (or low) spread in the estimated flows (see e.g. 2nd col, 3rd row in Figure A.7).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ECWKt0sPH1", "reviewer": "Reviewer_xMnU", "summary": "This paper focuses on the problem of eliciting a complex multivariate probability density from an expert. Existing works mainly use simple distributions. This paper proposes to model the belief density with a flow based model. To apply normalizing flows to this problem, we will need to address a few challenges. (1) We want to train flows with a small set of samples, which may result in collapsing and diverging probability mass. (2) The samples presented for the expert might be drawn from an independent and known distribution.\n\nThis paper proposes to use Bayesian inference to address the challenges of collapsing and diverging probability mass. That is, we are interested in representing the belief density $p_{}(x)$ with a flow model. We then optimize the parameters of $p_{*}(x)$ with MAP estimation. The likelihood of samples, i.e., $p(D|f)$ can be computed with Proposition 3.5 of the paper. The prior is approximated by the probability of the k-wise winners i.e., $p(D_{k}^{\\succ})$. \n\nThe experiments on synthetic and real datasets show the effectiveness of the proposed method.", "questions": "1. I am not an expert of expert knowledge elicitation, so I may misunderstand something. Based on Eq. 1, the $f$ is a continuous function. Based on Eq. 6, $p(f) \\propto \\prod \\exp{f_i}$. Therefore, why do we need normalizing flows to represent $f$. That is, it seems that we can use an arbitrary neural network to represent $f$.\n\n2. What’s the meaning of Theorem 3.6. I may have missed something, but I did not see any discussion or analysis on this theorem.\n\n3. The paper mentions that one challenge of this task is samples presented for the expert might be drawn from an independent and known distribution. How this challenge is addressed by the proposed method?\n\n4. The experiments actually not very convincing, to improve the experiments,\n    \n    (1) Are there any existing methods that can be used as baselines?\n    \n    (2) Do we have any quantitative metrics to evaluate the proposed method?", "limitations": "In my opinion, the proposed method does not have potential negative societal impact.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. This paper applies normalizing flows to the problem of expert knowledge elicitation. I think this is a promising application of flow-based models. The idea is novel. \n\n2. The proposed method is effective, and has theoretical guarantees.", "weaknesses": "Some definitions and formulas are not very clear to me.\n\n1. In Proposition 2.1, it is not very clear what is the relationship between W and the limit $\\lim_{\\beta \\rightarrow 0} p(\\mathcal{D}) = 1$.\n\n2. The symbol $\\mathcal{D}$ is used in two places, i.e., Proposition 2.1 and Section 4.1\n\n3. In Assumption 3, what is $Exp(s)$.\n\n4. In Proposition 3.5, is $s()$ a function or a constant?\n\n5. In Eq.4, Is the numerator $\\exp{(sf(x)\\lambda(x))}$?\n\n6. The symbol $\\mathbb{x}$ is used in several places. Sometimes, it represents a random variable, sometimes, it represents a sample.\n\n7. In Eq. 6, is $f_i = f(x_i)$？", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> In Proposition 2.1, it is not very clear what is the relationship between W and the limit \\\\(\\lim_{\\beta \\rightarrow 0} p(\\mathcal{D}) = 1\\\\).\n\nAs the “noise level” \\\\(\\beta\\\\) goes zero, there is no noise in RUM (i.e. \\\\(W=0\\\\) with high probability), which implies that the expert always chooses \\\\(\\mathbf{x}\\\\) which maximizes their utility / belief density, that is \\\\(argmax_{\\mathbf{x}} p_{\\star}(\\mathbf{x})\\\\). Only observations consistent with argmax have non-zero probability and hence the probability of the only possible data collection is one. Since argmax is invariant to monotonic transformations, one cannot identify the target belief density, or it is unidentifiable at least up to a monotonic transformation.\n\n> The symbol \\\\(\\mathcal{D}\\\\) is used in two places, i.e., Proposition 2.1 and Section 4.1 \n\nWe modified the notation in Proposition 2.1 from \\\\(\\mathcal{D}\\\\) to \\\\(\\mathcal{D}_{\\textrm{pref}}\\\\).\n\n> In Assumption 3, what is \\\\(Exp(s)\\\\) \n\nExponential distribution with a rate parameter s>0. We will clarify this in the final version.\n\n> In Proposition 3.5, is \\\\(s()\\\\) a function or a constant? \n\nIt is a constant, the rate parameter of the exponential distribution. The equation reads as \\\\(s\\\\) times \\\\(f(\\mathbf{x}) - f(\\mathbf{x}_j)\\\\), the parenthesis being for the difference and not indicating a function.\n\n> In Eq.4, Is the numerator \\\\(\\exp{(sf(x)\\lambda(x))}\\\\)?\n\nNo, the equation in the paper is correct, i.e. the integrand in the numerator is \\\\(\\exp{(sf(\\mathbf{x}))\\lambda(\\mathbf{x})}\\\\). Note that \\\\(f(\\mathbf{x}) := \\log p_{\\star}(\\mathbf{x})\\\\), whereas $\\lambda(x)$ is directly a density.\n\n> The symbol \\\\(\\mathbb{x}\\\\) is used in several places. Sometimes, it represents a random variable, sometimes, it represents a sample.\n\nWe used \\\\(X\\\\) for a random variable and \\\\(\\mathbf{X}\\\\) for a set of samples (design matrix). To avoid confusion, we changed our notation so that the random variable is now denoted by \\\\(\\mathbb{X}\\\\).\n\n> In Eq. 6, is \\\\(f_i = f(x_i)\\\\)?\n\nYes. We will clarify this in the final version.\n\n> ...Based on Eq. 1, the \\\\(f\\\\) is a continuous function. Based on Eq. 6, \\\\(p(f) \\propto \\prod \\exp{f_i}\\\\). Therefore, why do we need normalizing flows to represent \\\\(f\\\\)... \n\nThe critical requirement is that the result has to be a density. More precisely, \\\\(\\exp(f)\\\\) needs to normalised over the argument x, i.e. \\\\(\\int \\exp(f(\\mathbf{x}))d\\mathbf{x} = 1\\\\). Normalising flows provide a natural tool for representing densities, whereas for an arbitrary network we would need explicit normalization that is not computationally feasible.\n\nFurther, Eq. (6) relates to the functional prior \\\\(p(f)\\\\), which defines a probability density over the *function values*.\n\n> What’s the meaning of Theorem 3.6. I may have missed something, but I did not see any discussion or analysis on this theorem.\n\nTheorem 3.6. provides the formula for the k-wise winner distribution in the limit of \\\\(k\\\\) (number of alternatives) approaches infinity. We use the limit result to construct the functional prior, which helps to solve the collapsing and diverging mass problem. Even though we eventually use only small \\\\(k\\\\), we can use the limit distribution as a prior, because we can temper the finite-k distribution to resemble the limit (Figure 2).\n\n> The paper mentions that one challenge of this task is samples presented for the expert might be drawn from an independent and known distribution. How this challenge is addressed by the proposed method? \n\nThe whole method is designed specifically to address this challenge.\nIf the samples were drawn from the target itself then standard flow learning would be sufficient as a solution, whereas learning the flow from preferential responses for samples drawn from any other distribution requires the full machinery we introduce. In other words, the challenge is addressed by the combination of the RUM model for preferential data and interpretation of the k-wise distribution as tilted version of the belief as described in Section 3, as well as the functional prior presented in Section 4. Further, the new ablation study (Figure R1) provides some insights on how the problem becomes more challenging when sampling distribution for the candidates is very different from the target density. Yet, the proposed method is able infer the target density, although the estimate is not accurate as it would be when the sampling distribution is closer to the target. \n\n> Are there any existing methods that can be used as baselines?\n\nMajority of prior elicitation works assume fixed prior family (e.g. a Gaussian) and would not be fair baselines; they can be made arbitrarily bad by making a poor choice of the distribution. Furthermore, there are no specific methods that learn from the preferential comparisons and hence conducting such a comparison would require deriving details for a new method anyway. While there are some methods that can estimate flexible densities (the GP-based methods we cited in Introduction), they all require completely different kind of input information and hence cannot be compared against in our setting. We learn from preferential comparisons, whereas e.g. Oakley \\& O'Hagan (2007) learn from percentiles.\n\nEven though there are no natural baselines, we hope that the additional experiments relating to sensitivity of the method in terms of the key parameters (\\\\(k\\\\), \\\\(n\\\\), \\\\(\\lambda(x)\\\\)) address in part the same request.\n\n> Do we have any quantitative metrics to evaluate the proposed method? \n\nSince the problem is a density estimation problem (from a preference data), a natural metric is the distance between the estimated density and the target density. The proposed method produces relatively low (Wasserstein and the MMTV) distances between the estimated flow density and the target density in the experiments.\n\n**References**\n\nOakley, J. E., \\& O'Hagan, A. (2007). Uncertainty in prior elicitations: a nonparametric approach. Biometrika", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are happy to see the reviewers both understood the paper well and perceived it positively. We thank the reviewers for the detailed constructive comments, and provide responses to specific comments and questions for each reviewer separately.

The rebuttal is accompanied by a pdf that reports results of new empirical experiments addressing comments from reviewers sR2g and FQE4, validating the sensitivity of the results in terms of the cardinality of the choice set \\(k\\), the number of comparisons \\(n\\), and the choice of the distribution \\(\lambda(x)\\) that the candidates are sampled from.

We report results for three new experiments:

1. Figure R1 studies the effect of \\(\lambda(x)\\), the unknown distribution from which the candidates to be compared are sampled from, complementing the experiment reported in Section 5.1 and confirming the method is robust for the choice. In the original experiment the candidates were sampled from a mixture distribution of uniform and Gaussian distribution centered on the mean of the target, with the mixture probability \\(w=1/3\\) for the Gaussian. Figure R1 reports the accuracy as a function of the choice of \\(w\\) for one of the data sets (Onemoon2D), so that \\(\lambda(x)\\) goes from uniform to a Gaussian,
and includes also an additional reference point where \\(\lambda(x)\\) equals the target. For all \\(w>0.5\\) we reach effectively the same accuracy as when sampling from the target itself, and even for the hardest case of uniform \\(\lambda(x)\\) (with \\(w=0\\)) the distance is significantly smaller than the reference scale comparing the base distribution with the target.

2. Tables R1 and R2, as well as Figure R2, report results of an experiment studying the effect of \\(k\\) and \\(n\\). Again the results are reported for only one of the data sets (Onemoon2D), but we will add similar results for all of the data sets in Appendix.
For fixed \\(n\\), we see that the accuracy naturally improves as a function of \\(k\\) (Table R1). The original manuscript used \\(k=5\\), but the new result reveals that we can learn the target already with \\(k=2\\) that is most convenient for a user, but naturally with somewhat lower accuracy. For fixed \\(k\\), increasing \\(n\\) generally improves the accuracy and already fairly small \\(n\\) is sufficient for learning a good estimate (Table R2). For very large \\(n\\) the accuracy can slightly deteriorate. We believe this is because of prior misspecification that encourages overestimation of the variation due to the fact that \\(k\\) is finite but in the prior it is assumed to be infinite. Figure R2 confirms that for \\(n=1000\\) the shape of the estimate is extremely close and the slightly worse Wasserstein distance is due to overestimating the width. For the final version, we extend the second paragraph of the Discussion to elaborate this aspect more.

3. Finally, Figure R3 shows that the LLM expert also works with \\(k=2\\). We replicated the original experiment conducted with \\(k=5\\) and report the estimates side-by-side, visually confirming we learn the essential characteristics of the distribution in both cases. The results are not identical and the case of \\(k=5\\) is likely more accurate (see e.g. the marginal distribution of the last feature), but there are no major qualitative differences between the two estimates.

Meta-review: This work proposes a method of learning an expert’s belief density in which the density is parameterized as a normalizing flow.  The main interesting elements are that the expert information is provided solely through preferential data (e.g., k-wise winner of a set), and that the authors devise a functional prior for the normalizing flow based on the choice density, or the distribution of elements selected to be present in the preferential sets.  They argue the reasonableness of this choice and show that empirically it helps them avoid density collapse during learning.  Reviewers felt that the experiments were somewhat limited, with a lack of alternatives or baselines to give a sense of the quality of the method.  The application of eliciting expert belief from LLMs is neat.

There are also a number of typos in the draft that should be fixed if accepted.
Decision: Accept (poster)