Preference Learning of Latent Decision Utilities with a Human-like Model of Preferential Choice
Authors: Sebastiaan De Peuter
Shibei Zhu
Yujia Guo
Andrew Howes
Samuel Kaski
Venue: NeurIPS 2024 poster (2024)
Keywords: preference learning
human-in-the-loop
AI-assistance for decision making
user modeling
cogntitive science
retrosynthesis planning
Primary area: human-AI_interaction
TL;DR: We improve learning of latent utilities from preferences for decision tasks, by using a cognitive model of preferential choice which models various context effects.

Abstract: Preference learning methods make use of models of human choice in order to infer the latent utilities that underlie human behavior. However, accurate modeling of human choice behavior is challenging due to a range of context effects that arise from how humans contrast and evaluate options. Cognitive science has proposed several models that capture these intricacies but, due to their intractable nature, work on preference learning has, in practice, had to rely on tractable but simplified variants of the well-known Bradley-Terry model. In this paper, we take one state-of-the-art intractable cognitive model and propose a tractable surrogate that is suitable for deployment in preference learning.  We then introduce a mechanism for fitting the surrogate to human data and extend it to account for data that cannot be explained by the original cognitive model. We demonstrate on large-scale human data that this model produces significantly better inferences on static and actively elicited data than existing Bradley-Terry variants. We further show in simulation that when using this model for preference learning, we can significantly improve utility in a range of real-world tasks.

Preference Learning of Latent Decision Utilities with a Human-like Model of Preferential Choice
Sebastiaan De Peuter
Shibei Zhu Aalto University shibei.. zhu@aalto. fi
Yujia Guo Aalto University yujia. grsia.guoaaatt.... fi
Aalto University
sebastiaan. www.:aan.dddeeeeeaetoaooooo
Andrew Howes
Samuel Kaski
University of Exeter
Aalto University
andrev.hovessexetrr.cc uk
University of Manchester
samuel .aslikkkkii taaiiaallooo.... .flato.ff
Abstract
Preference learning methods make use of models of human choice in order to infer the latent utilities that underlie human behavior. However, accurate modeling of human choice behavior is challenging due to a range of context effects that arise from how humans contrast and evaluate options. Cognitive science has proposed several models that capture these intricacies but, due to their intractable nature, work on preference learning has, practice, had to rely on tractable but simplified variants of the well-known Bradley-Terry model. In this paper, we take one state- of-the-art intractable cognitive model and propose a tractable surrogate that is suitable for deployment in preference learning. We then introduce mechanism for account for data that cannot be
fitting the surrogate to human data and extend explained by the original cognitive model. We demonstrate on large-scale human data that this model produces significantly better inferences on static and actively elicited data than existing Bradley-TTer variants. We further show in simulation that when using this model for preference learning, we can significantly improve utility in range of real-world tasks.
Introduction
AI systems need exact descriptions of tasks to be performed. However, humans find more complex tasks hard to describe. In response, preference learning has emerged as way to learn from human feedback. has been used to teach AI systems a variety of tasks from how to hand objects to humans to how to play Atari games [1-4]. More recently, human feedback has been used train large language models to summarize text 3. answer questions in natural language 66, and to train deep generative models to generate realistic medical images 77).
When learning from human feedback, it is generally assumed that some latent utility function guides an individual's 's behavior, but that the individual cannot describe this function to the machine. Thus, preference queries are used to elicit information about f from the user. A preference query gives user set of options Ti, ....... Ir and asks the user to select their preferred option, i.c., the one with the highest utility. Given a model how people make such choices, the machine can then infer the underlying function f from the user's chosen item y. For example, Stiennon et al. [5] learned utility function for text summaries by showing users a text with several summaries and asking them to choose the best summary.
There several models of choice that have been used for learning preferences from human choices. Some recent work on Reinforcement Learning from Human Feedback (RLHF), for example, has used
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

simple binary choice model 5 6 p(y = 21 (z) = (() - f((z)), over choices and Z2.
though generally, most preference learning upproaches have used the Tradley-Terry model [
exp(3f(x.))
ply x.( In) 2==1 exp(3
Although these models have proven be practical, they are not realistic models of human choice behavior. Specifically, both models make choices between two options without taking into account the rank orderings of option attributes; widely observed property of human decision-making [9] As a result, these models fail to predict a number of apparent biases in human choice behavior. These include contextual choice effects [12 [3] which occur in situations where decision maker's choice between two options is influenced by adding more options to the choice set [2] Say, for example, we have two options A and B and user exhibits probability of choosing between these. When a third decoy option C' is introduced which is strictly dominated by B, there is a shift in the probability of choices from A to B.
Though context effects are not certain to appear in preference queries posed to users, they are known to appear in wide range of human tasks including risky choice tasks [12], multi-attribute choice tasks 13 and perceptual judgement tasks [ and in many other species including jays and honeybees [16] These effects point to a potential gap in the accuracy of the models currently used, during preference learning, to interpret the choices made by users. Moreover, this gap has the potential to lead to incorrect inferences about the latent preference utilities of observed human decision-maken
The contribution of this paper is threefold. First, we show that we can improve preference learning by leveraging computational rationality theory, general oognitive-scientffe theory which posits that human behavior is rational under cognitive bounds [7 [8]. We learn preferences from human choice behaviors using a state-of-the-ard cognitive model that is based on computational rational analysis of context-dependent choice under uncertainty and is backed by substantial empirical support in the psychology literature [19]. Like all computationally rational models, behavior under this model emerges from the latent utility function and a latent set of cognitive bounds. This provides strong inductive biases when inferring these latent factors from human behavior which show experimentally - significantly improves learning from preferences. Our second contribution lies in making this cognitive model amenable to preference learning. To this end, we generalize it, and make inference practical by approximating intractable calculations with surrogate we call the Computationally Rational Choice Surrogate (CRCS) model. Finally, we find experimentally that CRCS can sometimes perform worse than the Linear Context Logit (LCL) model (20). We hypothesize that human context effects are partially consequence of cross-feature effects. These are not modeled in CRCS, but can be learnt by LCL. We therefore propose LC-CRCS, which takes advantage of these effects by combining CRCS with LCL.
we will
We report three sets of experiments. In the first, we show that CRCS matches the original model's prediction of human choice behavior. In the second, we compare preference learning with CRCS to preference learning with recently proposed variants of the Bradley- -erry choice model. Using existing human data sets, we show that CRCS outperforms these in choice prediction and utility function inference, but performs worse than LCL on some tasks. We then show that LC-CRCS can additionally outperform LCL in these tasks. In the third set of experiments, we show the applicability of CRCS in three real-world use cases and verify its parameter recovery capability.
2 Background
2.1 Learning from Preferences
Preference learning methods aim to infer latent utility functions from human choices. Depending on the type of queries presented, there are two main streams of research: (1) learning from pairwise comparisons or (2) learning from ranking, where humans rank a set of n options. Popular methods include Gaussian Process regression that captures the preference relationships of pairwise queries [2] 22]. Other work, such as as [3]25] 255, uses Deep Neural Networks trained on ranked demonstrations to approximate the underlying reward functions. To reduce the computational burden created by the necessity for numerous queries, active learning techniques 26 have been proposed for efficient query proposal with maximum information gain. However, these methods typically require consistent preference order within the ranking and do not consider any contextual effects within the query
2

---

dataset. Reinforcement Learning approaches include Preference-Best BaseddBBBeddd Reinforcement Learning reward function inferred from the preference feedback. Work using ranking queries and human feedback can reach or even exceed human-level performance in several RI. benchmarks 2 24
(PBRL) and Reinforcement Learning with Human Feedback (RLHF), where the
2.2 Modeling contextual choice
To date, preference learning research has yet to make use of plausible models of human decision- making such as (30 31. These models are inspired by extensive studies of human behavior and give rise to contextual choice effects. Consider a hypothetical choice between two sightseeing trips, one to Paris and the other to London. Both trips come with free coffee. Let's say that 70% of people prefer Paris and 30% London. Now imagine that we add a third option which identical to the London trip but without free coffee. If, for this three-trip choice problem, we observe that 40% prefer London with free coffee, then we will have observed a contextual choice effect known as a "preference reversal" [32]. The choice frequency for London with coffee is increased by context that includes dominated choice. This effect has been observed both in sample averages and, more interestingly, within individuals. It has been taken evidence that people are irrational [33] and have no stable preferences [3]. Needless to say, both instability and irrationality pose severe challenges to the viability of preference learning.
More recent theories, however, demonstrate that contextual choice effects can consequences of computationally rational processes that assume stable preferences. These theories explain contextual choice effects by modeling the fact that people compare attributes and/or utilities under uncertainty. These include Bayesian theories [35], rational analyses [19], and neurobiological relative encoding theories [13 [66. These theories use comparisons between option attributes to compute expected utilities, such that these expectations are sensitive to the reliability of the comparison as indicator of expected utility.
Other work has proposed variations on the Bradley-Terry model to include these contextual effects, with the same commitment to stable preferences. Bower and Balzano 37 posit that context effects are the result of humans comparing options only on the & most salient features within context. They propose a Bradley-Terry model where utilities are calculated only on the & most salient features, where saliency is measured by the sample variance of each feature within the current set of options. Tomlinson and Benson (20) do not propose a specific theory context effects, but rather propose to learn them from data. They introduce the Linear Context Logit (LCL) model, Bradley-Terry model with a linear utility function, in which context effects are modeled as a context- derreeex-eeeddeent change in the (globally stable) weights of the utility function. This change is modeled as linear function of the average attribute values of the set of options presented to a user (the context), and is inferred from human choice data. They further introduce the Decomposed LCL model, in which each feature induces its own context effect whereas in LCL. the features jointly induce a single context effect- - and where the final context effect results from mixture of these individual effects.
In the current paper, we commit to distinguishing behavioral choices, which are observable, from latent preferences, which are not. When we refer to "preferences" we are referring to the latent utility function (xx and not to the observable choice behavior.
3 Modeling computationally rational choice
To learn latent preferences from human choice behavior, we build on computationally rational model of choice behaviors by Howes et al. [19] which is sensitive to the aforementioned context effects. This model assumes that humans make uulity-maximizing choices, but that the option utilities are estimated from noisy observations of the true utilities and noisy comparisons between the option attributes. Here we will first describe the original model in a general form. We then extend it to general space of utility functions and introduce our Computationally Rational Choice Surrogate (CRCS) model, model which replaces intractable computations in the original model with learned surrogates to allow tractable inference of the latent utility function. Finally, we introduce the LCCCRCS model, an extension of the CRCS model which is able to learn additional context effects not captured by the CRCS model.
3

---

3.1 A computationally rational model of choice
Let 21,.... .i....... ......... In Rd be set of n options, each with d attributes. Let f R R be latent utility function that maps each option to its associated utility. As shorthand, we will denote the utilities of collection of options æ (.1...., Zn) as (U1..... un)) where u, = f(x.).
The cognitive model introduced by Howes et al. 19 assumes that when making choices, humans do observe the options 21, 2:,....:: 2n their utilities directly. Instead, humans assumed make utility-maximin choices based on two sets of noisy observations of the options. The first set are noisy observations (uI,.... ūn) of the true utility of each option. These modeled as samples from a Gaussian centered around the true utilities
E
UT ~ ((f(x))
with noise o oalle which will call the calculation noise. The second set are noisy observations of the ordinal relation between the values of each attribute for each pair of options. Given an attribute k and a pair of options (24, Zy), this ordinal relationship defined by the following observation function:
iff KiAh Jj.k
0 Tgkk)
iff Tisk Ij.k + T
else
with тı an attribute-speeific tolerance parameter. Intuitively, larger T& creates greater margin within which attribute values will be considered equal. For binary attributes, we set 7 to zero. Each noisy ordinal observation o( JJ,k) is sampled as follows: with probability 1 sample Ō(I;,k Fz,kk) o( k, otherwise sample uniformly at random from { - } of ordinal error e is parameter, and the sole source of noise within the ordinal observations. We will denote the set of noisy ordinal observations as δ {ō(21,t, T. lm... I...,,j.l............... j.....jii.l.... I...
The probability
Wnx and the choice model parameters = (o Coatee S Tl, Ta), the above model implies a posterior distribution over the options' true util- u,,ue o, () and associated expected values E[u it o, 0). As they do not observe true utilities of the options, humans are assumed to choose the option y with the highest expected utility:
Given these observations
and ő for options
ities p(u/u,
y
argmax Z/E (11....))
EE [u L1 0].
Preference learning requires that we are able to reason about how various utilities lead to different choice behaviors. Therefore, make the original cognitive model amenable to preference learning. we replace the fixed utility function f by a space of utility functions {fw / parameterized by utility parameter w. We assume that the user being modeled makes choices based on some chosen parameter value which is known only to them, and which we represent as an additional observed random variable in the model. Necessarily, any calculation of utility therefore depends on w. Under these assumptions the user's posterior over utilities, and thus their choice y. is:
argmax
B(u ū w. 0).
x: ((E.,.
where this expectation is caleulated under the posterior
p(u o, w. O) p(ui u) 0)
p(æ, u. 0)dx
n
I p(u.lu 8) p(o 0) II pp w))n
3.2 Learning from choice behaviors
In our description of the model above, we have taken the point of view of the user making the choices. However, we now return to preference learning perspective, i.e. that of an outside observer such as Al system trying to infer the utility function that underlies these choices. We assume that the Al system observes the presented options .1,..., In, as well as the option the user chooses. The goal is then to infer the unknown utility parameter and choice model parameters 0 from observed

---

choices (z.y). However, the noisy observations ū and o on which the user bases their choice are part of their internal perception of the options, and are therefore not observable an Al system. This means that in evaluating the likelihood of choice y under the above choice model we must treat the observations as latent. This yields the following choice policy for the user:
ply ar, w. 0) /, plulo u, w, 0)p(olar ar 0)p(ū w, 0)
Here, O u, w) is a point mass on y following equation ① Given m pairs (xx0), y ) prior p(w) over the space of utility parameters and prior p(0) over the space of choice model parameters, we can use the likelihood in equation to infer a posterior over the parameters w and 0:
m
p(u, (()(((())))))))))) x (())
II 1=1
(0 w)
3.3 Tractable inference through surrogates
The issue we face in calculating p(w, ()(((() ((((() is that the likelihood p(y]æ, w, 8) is intractable. First, the calculation of the expected values in equation (T) requires the evaluation of an intractable integral over in equation (2). The expected values can be approximated using Monte Carlo estimate [19]. but many samples are needed to achieve a good approximation. Second, the calculation of the likelihood itself requires the evaluation of an intractable integral over all possible observations in equation (3). As before, one could approximate this integral using a Monte Carlo estimate, but this would again require many samples.
Instead, we propose to train surrogate neural networks to approximate both these quantities. We introduce first neural network ū (, o, w, 8) trained predict a vector of the expected values Elu ū, o, w, 0] from given observations U, O and parameters w and 8. Then it(-) is trained by minimizing
Luil p(u,0,u.u.
E
[ o, w, 0) - u u2l2.
(4)
Samples (w, 0, u. σ) obtained by (1) sampling ~ p(w), 0 ~ p(0) and a p(xx from their respective priors, (2) calculating i, = f.. (2;) for each option, and (3) sampling the observations
0; ~ p(u.p((u 0) and ő p(o|2, 0). Note that the minimum of Lui (ū) is exactly the function that assigns to each tuple (u, o, w. 0) the vector of expectations E[(u ū O, w, 0).
Next, train second neural network @(y|æ, w, 8), which we will refer to approximate the user's policy x w, 0) over choice behaviors. By using the fact that ū( ū, w, 0) s ( O, w. 0] minimize the cross-entropy loss between q and choices based on utilities predicted by ū. The loss function is thus:
CRCS model,
l argmax 80 ū, o, w. 0) x, w, )] {
(
p(w.8,x.ü.))
Samples (w, 0, u, ü, σ) are obtained as above.
3.4 Modeling cross-feature influence in CRCS
Although our proposed model can predict range of context effects, it does not yet capture all. Although CRCS can model how each individual feature influences the expected utility of the options, cannot model how features can impact each other. This is something that LCL does do: its utility weight updating mechanism changes the weight of each feature based on the mean value of all other features. Thus, features can influence how other features are valued. the most general sense, LCL's fundamental mechanism corresponds to function g(w, x) which maps the utility weights and the set of options æ (which make up the context) to new set of weights w'. We therefore propose to integrate this same mechanism into CRCS, resulting in new model q(yl (w,, x)) 0). As ğ is differentiable, we can infer from data using gradient descent. In the experiments that follow, we will use this approach in settings where fa, is linear function. Thus, like LCL, we will define y as linear function of ag: the mean attribute values of the options æ. We will refer to the resulting model (Axc)*
q (y) x. (aac) () as LC-CRCS.
5

---

Table 1: Choice model NLLs human choice data sets. Bolded digits indicate a significant (p 0.01) improvement over baselines (BT, BB, LCL).
Dataset
Tradley-Terry Bower & Balzano
LCL
CRCS (ours)
LC-CRCS (ours)
Hotels
573
573
553
536
536
District-Smart
3432
3432
3305
3371
3276
Car-Alt
7414
7416
7290
7322
7345
Dumbalska
103669
103711
100683
100450
99147
4 Experiments
We first validate the proposed CRCS model by comparing its results with the original computationally rational choice model by Howes et al. [9]. Then we compare the proposed CRCS model and our LC-CRCS variant with three baselines on human choice data, and finally study the performance of the model on three case studies: car crash structure design, water drainage network design, and retrosynthesis planning |
We evaluate our proposed CRCS model on four datasets of human choices. These datasets are large sets of choices (xx"), () collected from human participants. The District-Smart dataset [8 contains pairwise preferences over voting districts, where participants were asked to choose the district they felt was most compact. The features extracted for each district are six geometric measures identified by the original authors as good measures of compactness. The Car-Alt dataset [39] contains choices between six hypothetical alternative fuel cars. Each car has 21 features, including size, range, operating cost, etc. We also use a dataset collected in 40). which we will call the Hotels dataset, where in a user study participants were asked which of three hotels they preferred. The hotels were collected from booking site and had as features the price per night and average review rating, For each participant, a choice was collected on one of six sets of options constructed to target three known context effects: attraction, compromise and similarity. Lastly, Dumbalska et al. (36] on property task, which we will refer to as Dumbalska. Here, participants ranked three properties in order of best to worst value. For purposes, item as the choice. Value was defined as the given rental cost minus the value participants thought the house was worth (which had been elicited in earlier stage). For each participant, responses were collected on a large collection choices, specifically engineered to span the entire range of potential context effects. Thus, unlike the other datasets, we have multiple recorded choices per participant. This allows us to make inferences per individual, rather than at the population level, and evaluate how well our choice models fit the preferences and context effects exhibited by individuals.
the data collected by
will treat the top-ranked
4.1 Validation of the CRCS model: risky choice tasks with preference reversals
In this experiment, we validate our CRCS model against the original implementation of Howes et al. 19 on risky choice task. In this task, user is presented with of three options, each of which is a pair (pr, v) consisting of a probability p; and payoff e;. Upon selecting option i, the user receives payoff by with probability Pt, meaning that each option has expected payoff f(p,, u.) = Pt 0/-
Comparing expected option values predicted by á with the Monte Carlo estimates used in [9]. we find that on sets of three options, both generally agreed the relative magnitude of the utilities, and agreed on the ranking of the utilities in 92.277% = 0.165% (Agresti- of cases. Next, we verified q ability to predict contextual preference reversals. This tested on Range-Frequency decoy conditions [19] where two optrito-optimal" options with equal utility are presented along with decoy option with slightly lower utility which is dominated by one of the other two options. Preference reversals specifically, increased likelihood of choosing the Pareto-optimal option that dominates the decoy have been observed in humans and predicted by the original model. Figure 3 in the appendix shows that ( reproduces the range reversal rates of the original model.
Implementation available at human mf
Learning- wittnng-vttttttth

---

Table 2: Consistency of inferred utility function with separately collected rankings District-Smart. Bolded digits indicate a significant improvement over baselines (BT, BB, LCL).
Dataset
Terry
Bower & Balzano
LCL
CRCS (ours)
LC-CRCS (ours)
District-Smant 0.162
0.217
0.286
0.622
0.525
4.2 Evaluation on static human choice data
In this set of experiments, we evaluate each models' ability to generalize to unseen data. We compare our proposed CRCS model and the LC-CRCS variant against three baselines: vanilla Bradley-Terry Terry. the variant proposed by(37] (referred to as Bower & Balzano) and LCL [20). On four different datasets, we infer the parameters for each model on a training set of observed choices (xx"", ( )}m and calculate the negative log-likelihood (NLL) of held-out test set under the inferred parameters. Inference was done using gradient descent on the NLL of the training set. We performed voss-vaaiiation, and report the sum of the test sets' NLLs across the folds. For Hotels, Car-Alt and District-Smart we split the choice data across 50, 20 and 10 folds respectively. By evaluating each choice model on each test fold, we obtained paired observations (one per condition) for each test fold, allowing us to perform Wilcoxon rank test across the folds to test significance. On Dumbalska. we look at how well the choice models can fit to individuals, and thus perform cross-validation for each participant individually. We then treated the sum of the NLLs of the test sets per participants as individual measures, and tested significance using a Wilcoxon test across the participants. Following prior work, we used a linear utility function in all choice models on all datasets.
Table shows the total NLL achieved by each model on each dataset. We observe that our proposed L LC-CRCS model achieves the highest NLL on Hotels, District-Smart and Dumbalska. This difference is significant (p < 0.01) in all three cases. On Car-Alt, we see that LCL performs better than all other models, with the difference being significant (p 0.01) for all except the CRCS model (p 0.2). We theorize that the poor performance of the CRCS model on Car-Alt is due to insufficient option data to train ū on (see Appendix AII)) leading to poor estimates of expected utility and therefore poor choice predictions.
4.2.1 Evaluating the inferred utility function
As part of the District-Smart human subject study, Kaufman et al. [38 collected rankings on six sets of districts from small groups of participants. Ranking such large sets is quite difficult, and we should expect these rankings be quite noisy. However, like the binary choices that were collected, these rankings are indicative of people's 's true preferences, and thus should be consistent with any ranking of the same districts implied by the utility function we infer from the binary choices. To test this, we use our choice models to infer utility parameters on the entire set of binary choices. For each of the six sets, we then measure using Kendall's т 4 how consistent the ranking implied by the inferred utility parameters is with the ranking collected in the study. We report the average consistency across all six sets. Because the log- likelihood of CRCS and LC-CRCS not convex, we repeat this procedure 25 times, starting from different points, to control for the effect local optima may have on the inferences. We test significance using a Wilcoxon test across the six sets of rankings.
Unlike the previous experiment, during inference we regularized the choice model parameters of LCL, CRCS and LC-CRCS. This was essential infer utility parameters that were consistent with the collected rankings. For LCL we used the L1 matrix norm of the weight adaptation mechanism's 's parameter matrix as a regularization term. The LI norm enforces sparsity and thus encourages LCL to fit only to the most significant context effects 20). For CRCS and LC-CRCS we used the probability of the choice model parameters under a chosen prior as the regularization term. Using our understanding of the model this allowed us to encode specific prior knowledge into the regularization. More details can be found in Appendix A.3 From the results Table 2 we observe that both CRCS and LCCCRCS infer utility parameters that are significantly (p < 0.001) more consistent with the collected rankings than the baselines. I.C-RCS performs slightly worse than CRCS, though the difference was not yet significant.
7

---

4.3 Elicitation on human choice data
We now evaluate how well the choice models perform in a preference learning setting, where we actively select the queries we put to user. Whereas in the previous set of experiments we evaluated how well the choice models perform on large amounts of data, here we are interested in how well they perform when minimal data is available. We use the wealth of data available per participant (between 530 and 1060 responses) in the Dumbalska data set to run a user experiment in silico. For each choice model, we use active learning to infer utility function and choice model parameters for each participant individually. At each time step of the experiment, we select from the set of queries recorded for the participant the most informative query using the expected information gain. The participant's s response to this query is then revealed, and the posterior over the utility and choice model parameters is updated. We use a particle filter to maintain the posterior belicfs. This elicitation process is performed for 25 time steps on 75 participants. To evaluate the inferences made by the choice models, we calculate the expected likelihood of the remaining queries where the choice has not been revealed yet. As the training data is actively selected for each choice model independently, the data on which they are evaluated - the remaining queries - will differ, meaning that we cannot use paired test we have done so far. Instead, we test significance using an independent t-test.
Figure JajureeTajshooo showsows the mean expected utility calculated over the participants as a function of time for each choice model. We observe that the two variants of our CRCS model make significantly (p < 0.01) better predictions of the participants' choices at all time steps (except 0). Where in the previous experiment (Table 1 saw that LCL was close in predictive power to our proposed models, we that in this low data setting the difference is much more pronounced. This is because a number of the context effects observed in the Dumbalska data set are built into the CRCS model. While LCL has shown it can learn some of these effects, it needs much more data to do so. Interestingly, we also observe that the LC-CRCS model, which can learn some new context effects on top of the ones built into q. shows significant (p < 0.05) improvement on the CRCS model itself, even when very little data is available. This shows that it provides us with the best of both worlds, showing quick adaptability in low data settings and good performance when more data is available.
4.4 Simulated case studies
We now test the feasibility of using our model to learn a utility function from simulated choice behaviors in real-world tasks, and to use the inferred utility function to help a designer solve a task by recommending design solutions to them. We consider preference learning setting where we learn utility and choice model parameters by iteratively eliciting a simulated designer's 's preferences over sets of candidate designs, chosen to maximize the expected information gain. Using T. we infer posterior over the unknown parameters from the observed choices. To simulate a variety of users, we run this experiment in silico, using 7 with utility parameters sampled from a non-informative prior and choice model parameters sampled from a prior designed to capture a wide range of behaviors exhibited by the CRCS model. At each time step measure two things: our ability to recover the unknown parameters from the observed choices, and the utility of the design recommendations make. The inference error is measured by the distance to the true parameters under our current posterior beliefs. The second is measured using the recommendation regret: the difference in utility between the designer's optimal design and the recommended design, which is chosen to be the design with the highest expected utility under the posterior.
4.4.1 Case study 1: learning from preferences in structural design
The first use case involves the design of the frontal crash structure of a car to optimize three separate objectives 91 (t), g2(),, (t) 42). The design parameterized by five parameters t = (41, 1s) which determine the thicknesses of various metal elements. We define the utility function as the Chebyshev scalarization of the original objectives: fu (t) = max (1,23) wy (t())) - = where 22 denotes the ideal value of 9:() and the weights w sum to one. Different choices of the utility weights w correspond to different trade-offs between the objectives, and therefore to different solutions on the Pareto frontier. Figure shows the average recommendation regret as function of the number of queries across 300 runs of this experiment. We observe that the recommendation regret reduces quickly, yielding good recommendations after as few as 10 queries. We attribute this to the utility inference error, shown in Figure 6a in Appendix B.I which reduces equally quickly.
8

---

0.05
0.8
LC-CRCS
Recommendatton regret
expected likelhood
Bradley-Terry
0.04
LCL
Bower Balzano
0.6
0.03
0.02
0.4
0.01
10
20
25
20
Number preference queries
Number of preference queries.
(a) Dumbalska
(b) Crash structure
0.78
Recomimendation regret
0.025
Maximum utility score
0.76
0.74
0.020
0.015
0.010
0.68
0.005
0.66
Aizynthfinder (Baseline)
Ours
0.64
20
40
60
Number of preference queries.
Number of top k.
(c) Water drainage
(d) Retrosynthesis
Figure 1: (a) Mean expected likelihood of unseen choice data function of the number of queries observed for various choice models on the Dumbalska elicitation task. (b-c) Mean recommendation regret function of the number of queries observed for the crash structure design and water drainage network design respectively. (d) Maximum utility within the top k of routes ranked by inferred utility function of k. All plots show the mean + twice the standard error around the mean.
4.4.2 Case study 2: learning from preferences in water drainage network design
Our second use case is a water drainage network design problem 43 This use case is another multi- objective problem involving six objectives 91 (z),..., G6 (z).
Here, scalarize the problem using weighted sum (z) = Ei=1 w=g)(=) where the weights w sum to one. As before, different choices of w correspond to different solutions on the Pareto frontier. We ran 300 runs of this experiment. Here too we see that recommendation regret (Figure Tc) and utility inference error (Figure 6a the appendix) drop quickly
put more queries to the designer, with the largest reduction within the first 10 queries. We achieve high-quality recommendations after less than 10 queries.
4.4.3 Case study 3: improving retrosynthesis planning with preference learning
Retrosynthesis planning, the problem of finding feasible reaction pathways to synthesize target molecules, is a central task of synthetic chemistry. Significant progress has been made in solving it through end-to-end automatic synthesis planning 44. Existing work has focused on expanding the search space of feasible reaction plans. Each route may satisfy an additional subset of proper- ties, and different individuals or organizations may have different preferences over the properties. Chemists' preferences over these plans are often highly complex. representing trade-offs between multiple objectives informed by personal experience and company policy. However, learning their preferences in way that can then inform Al-driven synthesis planning has not yet been done. We designed a tieemist-in-th-lore retrosynthesis planning framework to generate routes with an inferred user model.

---

To build a personalized retrosynthesis planner, we modified one of the state-of-the-ar automatic retrosynthesis platforms, Aizynthfinder [45], built on Monte Carlo Tree Search (MCTS) with fixed utility function. Details can be found in Appendix B.2. First, we proposed new utility function as the weighted combination of five feature properties gi (r)..... 95 (r), that correspond to reactants cost, intermediates stability, reaction feasibility, total reaction success rate, poor reaction success rate, and a route score computed by a data-driven scoring model gs. Given the input routes, this model predicts the distance between the current route and the (latent) optimal route. We trained this model on 47,055 synthetic routes extracted from the Journal of Medicinal Chemistry.
We report the inference error during preference learning in Appendix B.2.2 We integrated the inferred utility weights into our planning system and assessed the consistency of the generated route with the ground truth user utility preferences. We used inferred weights to synthesize 100 target molecules for each weight. In order to measure the recommendation quality, we evaluated the top-ranked routes from both Aizynthfinder and our model under the true utilities. Specifically, we measured the maximum true utility score among the list of top recommendations. This is to show how far down from the recommended options list the user needs to go to find their optimal choice. Figure 7 shows that within the top & options, the reaction pathways recommended from our model reaches higher maximum utility score compared to the ones generated by Aizynthfinder. As significance test, we use the Wilcoxon rank test across every molecule and every user utility with p < (1.61 x 10 10-53) for all k.
5 Conclusion
In this paper we have proposed a tractable surrogate model of choice, called CRCS, inspired by theories of human decision-making. This model was shown to be better basis for preference learning than some, but not all, existing models. In response, we modified the model so that could make cross-feature observations of feature values extending the opportunity for contextual decision- maasinn. We verified against human data from a range of tasks that the new model, called LC-CRCS, outperforms the tested models both in terms of its ability to predict choices and in its inferences of the utility function that underlies the observed choices. Moreover, we find that it corresponds well to previously reported experimental data demonstrating human susceptibility to contextual choice effects. Feasibility of using the new model for preference learning and its ability to recover parameters was also demonstrated in three case studies. Together, the results demonstrate the viability of CRCS and LC-CRCS in high performance preference learning systems.
Limitations and future work We identify two primary limitations. First, training CRCS requires sufficiently many choice sets, or a sufficient well-specified task so that new sets can be generated. As we saw with Car-Alt, when insufficient choice sets are available for training, performance can suffer. Second, CRCS and CRCS-LC only work on choice sets of fixed size. Extending these surrogates to variable size choice sets is promising direction for future work. Another promising direction for future work is the application of the current choice model to large language model (LLM) fine-tuning. Currently, given some featurization of LLM responses to a prompt. CRCS could be directly applied. However, this would ignore the reading and interpreting of these responses that human evaluators have do. As such, we see an extension of the current choice model that integrates these cognitive processes, based on the same computational rationality theory, as potentially transformational future work.
Societal impact This paper presents work motivated by the goal to advance the field of Machine Learning. The potential societal impact is in line with the broad body of prior work on learning from preferences and modeling humans, none of which we feel must be specifically highlighted here.
This work was supported by the Research Council of Finland (flagship pro- gramme: Finnish Center for Artificial Intelligence, FCAI; grants 345604, 341763 and 359207), and the UKRI Turing AI World-Leading Researcher Fellowship, EP/W002973/1. Computational resources were provided by the Aalto Science-IT IT Project
Acknowledgements
10

---

References
[1] Christian Wirth, Riad Akrour, Gerhard Neumann, Johannes Fürnkranz, et al. survey of Preference-Based Reinforcement Learning Methods. Journal of Machine Learning Research, 18(136):1-46, 2017.
[2] Paul F Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforcement Learning from Human Preferences. In Proceedings of the 31 Interna- tional Conference on Neural Information Processing Systems, pages 4302-4310, 2017.
[3] Andras Kupesik, David Hsu, and Wee Sun Lee. Learning dynamic robot-to-human object handover from human feedback. Robotics Research: Volume 1, pages 161-176, 2018.
[4] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
[5] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3022 2020.
[6] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 15:27730-27774- 2022.
[7] Shenghuan Sun, Gregory M Goldgof, Atul Butte, and Ahmed M Alaa. Aligning synthetic medi- cal images with clinical knowledge using human feedback. arXiv preprint 1244,,223006.24488, 2023.
[8] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4):324-344 345, 1952.
(9] John W Payne, James Bettman, and Eric Johnson. Adaptive strategy selection decision making. Journal of experimental psychology: Learning, Memory, and Cognition, 14(3):534, 1988.
[10] Takao Noguchi and Neil Stewart. Multialternative decision by sampling: A model of decision making constrained by process data. Psychological review, 125(4):512, 2018.
[11] Andrea M Cataldo and Andrew L Cohen. The comparison process an account of variation in the attraction, compromise, and similarity effects. Psychonomic Bulletin & Review, 26(3): 934-942 2019.
[12] Douglas H Wedell. Distinguishing among models of contextually induced preference reversals. Journal of Experimental Psychology: Learning, Memory, and Cognition, 17(4))767, 1991.
[13] Konstantinos Tsetsos, Rani Moran, James Moreland, Nick Chater, Marius Usher, and Christo- pher Summerfield. Economic irrationality is optimal during noisy decision making. Proceedings of the National Academy of Sciences, 113(11): (3(11)33022-107 2016.
[14] Joel Huber, John W Payne, and Christopher Puto. Adding asymmetrically dominated alterna- tives: Violations of regularity and the similarity hypothesis. Journal of consumer research, 9(1) 90-98. 1982.
[15] Konstantinos Tsetsos, Marius Usher, and Nick Chater. Preference reversal in multiattribute choice. Psychological review, (17(4):1275 2010.
[16) Sharoni Shafir, Tom A Waite, and Brian H Smith. Context-dependent violations of rational choice in honeybees (apis mellifera) and gray jays (perisoreus canadensis). Behavioral Ecology and Sociobiology, 5 51:180-188 2002.
[17] Richard L. Lewis, Andrew Howes, and Satinder Singh. Computational rationality: Linking mechanism and behavior through bounded utility maximization. Topics in cognitive science, (2):279-311. 2014.
11

---

[18] Falk Lieder and Thomas L Griffiths. Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources. Behavioral and brain sciences, 43:e1, 2020.
[19] Andrew Howes, Paul A Warren, George Farmer, Wael El-Deredy, and Richard L Lewis. Why contextual preference reversals maximize expected value. Psychological review, 123(4))368, 2016.
[20] Kiran Tomlinson and Austin R Benson. Learning interpretable feature context effects discrete choice. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1582-1592, 2021.
[21] Wei Chu and Zoubin Ghahramani. Preference learning with gaussian processes. In Proceedings of the 22nd international conference on Muchine learning, pages 137-144, 2005.
[22] Neil Houlsby, Ferenc Huszar, Zoubin Ghahramani, and Jose Hemáández-lbbao. Collaborative gaussian processes for preference learning. Advances in neural information processing systems, 25, 2012.
[23] Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond sub- optimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning, pages 783-792. PMLR, 2019.
[24] Daniel Brown, Wonjoon Goo, and Scott Niekum. Better-than-demmossrta imitation learning via automatically-raakk demonstrations, In Conference robot leamiing pages 330-359. PMLR, 2020.
[25] Vivek Myers, Erdem Biyik, Nima Anari, and Dorsa Sadigh. Learning multimodal rewards from rankings. In Conference on robot learning, pages 342-352. PMLR, 2022.
[26] Ashish Kapoor, Kristen Grauman, Raquel Urtasun, and Trevor Darrell. Active learning with gaussian processes for object categorization. In 2007 IEEE 11th international conference on computer vision, pages 1 8. IEEE, 2007.
[27] Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv: 2011.
[28] Kevin G Jamieson and Robert Nowak. Active ranking using pairwise comparisons. Advances in neural information processing systems, 24, 2011.
[29] Erdem Bryık, Nicolas Huynh, Mykel Kochenderfer, and Dorsa Sadigh. Active preference- based gaussian process regression for reward learning. arXiv preprint arXiv:2005.022 02575, 2020.
[30] Sudeep Bhatia, Graham Loomes, and Daniel Read. Establishing the laws of preferential choice behavior. Judgment and Decision Making, 1344)3333, 2021.
[31] Jerome R Busemeyer, Sebastian Gluth, Jörg Rieskamp, and Brandon M Turner. Cognitive and neural bases of multi-attribut, multi-alternative, value-based decisions. Trends in cognitive sciences, 23(3):251 2019.
[32] Amos Tversky, Paul Slovic, and Daniel Kahneman. The causes of preference reversal. The American Economic Review, pages 204-217, 1990.
[33] Paul W Glimcher. Efficiently irrational: deciphering the riddle of human choice. Trends in cognitive sciences, 26(8):669-687, 2022.
[34] Petter Johansson, Lars Hall, and Nick Chater. Preference change through choice. In Neuro- science of preference and choice, pages 121-141. Elsevier, 2012.
[35] Francesco Rigoli, Christoph Mathys, Karl Friston, and Raymond J Dolan. A unifying bayesian account of contextual effects in value-based choice. PL.S computational biology, 13(10): c1005769, 2017.
12

---

[36] Tsvetomira Dumbalska, Vickie Li, Konstantinos Tsetsos, and Christopher Summerfield. A map of decoy influence in human multialterative choice. Proceedings of the National Academy of Sciences, 117(40):5119-25177 2020.
[37] Amanda Bower and Laura Balzano. Preference modeling with context-dependent salient features. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1067-1077. PMLR, 13-18 Jul 2020. URLLattps:/pproceeding mlr press/v119/ bower20a.hta html
[38] Aaron Kaufman, Gary King, and Mayya Komisarchik. How measure legislative district compactness if you only know it when you see it. American Journal of Political Science, 65(3):
533-550, 2021.
[39] David Brownstone, David Bunch, Thomas F Golob, and Weiping Ren. transactions choice model for forecasting demand for alternative-fur vehicles. Research in Transportation Economicx, 4:87-129, 1996.
[40] David Ronayne and Gordon DA Brown. Multi-attribute decision by sampling: An account of the attraction, compromise and similarity effects. Journal of Mathematical Psychology, 81: 11 11-27, 2017.
[41] Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81-93 1938.
[42] Xingtao Liao, Qing Li, Xujing Yang, Weigang Zhang, and Wei Li. Multiobjective optimization for crash safety design of vehicles using stepwise regression model. Structural and multidisci- plinary optimization, 35:561-569, 2008.
[43] Ryoji Tanabe and Hisao Ishibuchi. An easy-to-use real-world multi-objective optimization problem suite. Applied Soft Computing, 89:106078, 2020.
[44] Ola Engkvist, Per-Ola Norrby, Nidhal Selmi, Yu-hong Lam, Zhengwei Peng, Edward C Sherer, Willi Amberg, Thomas Erhard, and Lynette A Smyth. Computational prediction of chemical reactions:
current status and outlook. Drug discovery today, 23(6):1203-120 2018.
[45] Samuel Genheden, Amol Thakkar, Veronika Chadimová, Jean-Louis Reymond, Ola Engkvist, and Esben Bjerrum. Aizynthfinder: fast, robust and flexible open-source software for retrosyn- thetic planning. Journal of cheminformatics, 12(1):70, 2020.
[46] Weihe Zhong, Ziduo Yang, and Calvin Yu-Chian Chen. Retrosynthesis prediction using an end-to-end graph generative architecture for molecular graph editing. Nature Communications, 14(1):3002 2023.
13

---

Us
Us
(a) Our choice model, originally introduced in [1]. posits that humans make utility- maximiinng choices (for some utility function parameters and choice model parameters 0) based only on observations (ū, o). The options Ti.-..,77 Th and their true utili-
(b) outside observer observes a set choices yd made associated options zi ..... =h From this data set, the objective is to infer the parameters
and 0 The noisy observations (z _0)) that are central to each of the user's choices internal the user and are therefore unobserved.
ties ui...., Un are not observed.
Figure 2: Graphical models of (a) our cognitive choice model and (b) the corresponding preference learning problem.
0.90
0.29
0.35
0.30
0.25
0.30
-0.55
Figure Reversal rate minus inverse reversal rate as a function da on Range-Frequency conditions for q ("surrogate") and for the original implementation of Howes al. [19 ("surrogate"). For q. we show the mean = std. dev. for 10 models trained with different seeds. The "reversal rate" is measured by calculating the rate which the Pareto-optimal To control for random variation we subtract from this the "inverse reversal rate", the rate at which the other Pareto-optimal option is chosen. For non-zero values of a datter see that though q is less sensitive to o² cate it reproduces the range of reversal rates of the original model.
eecoy-dominating option is chosen.
A Human data experiments
A.1 Priors
This section provides details how the CRCS model was trained for the choice tasks corresponding to the Hotels, District-Smart, Car-Alt and Dumbalska datasets. In order to train our CRCS model on a new choice task, we need define three priors: prior over sets of options p(z). a prior over utility function weights p(w), and a prior over choice model parameters p(0).
The prior sets of options p(x) is by far the most important prior for successfully training the CRCS model. is clearly important that this prior matches the distribution of choice sets we expect to see for the choice task target. However, it is proper support across the entire space of option sets. From equation 2 we see that in order to predict the utilities of the options x, essentially has infer the option (which it does not observe) from the observations ğ and ō. It can only learn to do this well if during training we can expect it to encounter all x that could have resulted in ū and ō.
more important ensure that that prior has
The priors p(x) for these tasks were defined as follows:
For Hotels we had access the set of 200 hotels the original authors had used to build their study. Thus, we generated option triplets from the prior by uniformly sampling (without replacement) three hotels from this set.
14

---

Table Mean = std. dev. around the mean for averaged NLL per choice pair on human choice data sets. This table shows the same results as Table but reports averaged as opposed to summed NLLs.
Hotels
District-Smart
Car-Alt
Dumbalska
Bradley-Terry
0.944 = 0.109
0.638 + 0.015
1.593 + 0.022
0.629 + 0.259
Bower Balzano
0,944 = 0.109
0.637 = 0.015
1.593 + 0.022
0.629 = 0.259
LCL
0.910 0.147
0.614 = 0.021
1.563 + 0.041
0.613 + 0.266
CRCS (ours)
0.882 0.104
0.627 0.016
1.573 + 0.023
0.612 0.266
L.CCRCS (ours)
0.882 0.115
0.610 + 0.020
1.591 + 0.029
0.605 + 0.273
2500
2500
participant valuation
participant valuation
1500
1000
1000
500
500
1000
1500
2000
2500
0
500
1000
1500
2000
2500
(a) Empirical option distribution
(b) Prior p(x)
Figure 4: (a) The distribution of individual options in the choice data Dumbalska. (b) The prior p(x,) over individual options we use to generate new option sets.
For District-Smart we had access 21778 electoral districts collected by the original authors. We generated pairs of options by sampling them uniformly from this set without replacement.
Options in the Dumbalska task have two features, property's 's rental cost and the partici- pant's valuation of it, both of which were bounded between and 2500. The human choice data suggested that both are strongly correlated. We created a prior over individual choices that reproduces this correlation by using multivariate normal distribution, mixed with uniform distribution over the entire option space, to ensure sufficient support even on less frequently encountered options. Figure f shows the empirical option distribution within the choice data, and our engineered prior over individual options. To generate option sets, we then sampled three options from this prior.
Car-Alt considers options sets consisting of hypothetical cars. Although know that the options were created from a set of 20 cars, the features of each option are determined both by the Car that corresponds example, the cost of each car is expressed as multiple of the participant's log income. The inclusion of participant-dependenn features creates correlation between the options. Unfortunately, we did not have enough information to engineer a new prior p(x) that would faithfully reproduce this correlation, and would faithfully match the empirical distributions over sets of options encountered in the original user study. Thus, we were forced to train the CRCS model on the limited number of choice sets that appear in the human choice data.
that option and the participant who makes the choice. For
As all tasks used linear utilities, and as the CRCS model is invariant to scaling of the utility function, we define p(w) in all cases as a uniform distribution over all vectors of length
The prior over the choice model parameters was set based on choice model parameters reported for risky choice in [19] after accounting for scale differences of utilities and feature values for each task. They are listed in table For I.C-CCS,, we additionally placed independent standard normal priors on all entries of A.
N denotes normal distribution truncated to the range (a, 6). Note that in order to be consistent with other baselines, within the CRCS model utilities are calculated on z- z-oomaalized features, while ordinal observations calculated -onnoommllzed features.
15

---

Table 4: CRCS model parameter priors for various choice tasks.
p(ocate)
p(c)
Attribute
p(Tk)
Price per night:
(0, 100)
Hotels
U00
Beta(1,3)
Review rating:
U(0,1)
hull
N (0.06, 0.12)
bbox:
(0.08, 0.16)
reock:
(0.05,
District-Sman
(0.0, 2.0)
Beta(1,3)
polsby:
No0 (0.08,, 0.16)
sym_x:
Af (0.11,
sym_y:
No (0.1,0.2)
Price divided by In(income):
(0.7,
Range:
(45, 90)
Acceleration:
(1,2)
Top speed:
(8.,
Car-Alt (0.0, 2.0)
Bela(1,3)
Pollution: (0.10.15
Luggage space:
(0.0077.0.15)
Operating cost:
N1.8,
Station availability:
No N (0.2,
Rental cost
N200 400) N (180, 360)
Dumbalska (0.0, 2.0) Beta(1,
Participant's 's valuation
Table 5: Mean = std. dev. of averaged NLLs on randomly selected held-out test set over 20 independent parameter inference runs.
Hotels
District-Smart
Car-Alt
Dumbalska
Bradley-Terry
0.890 8x10-6
0.632 + 1x10-5
1.575 + 3x10-3
0.567 3x10-6
Bower & Balzano
0.890 + 2x10-5
0.631 + 4x10-6
1.575 + 5x10-5
0.567 2x10-6
LCL
0.831 + 2x10-5
0.616 = 4x10-6
1.560 + 5x10-5
0.562 + 2x10-6
CRCS (ours)
0.855 + 0.014
0.628 0.009
1.620 0.032
0.568 = 0.007
LC-CRCS (ours)
0.902 + 0.063
0.616 + 0.006
1.590 - 0.013
0.572 = 0.007
A.2
Additional details on static data experiments
The experiments on static data were run using a cross-validation strategy. For each fold, we inferred utility parameters and choice model parameters jointly for each choice model by performing vanilla gradient descent on the train set log-likelihood. For the CRCS and LC-CRCS model the starting points were sampled from the priors defined in section A. For the Bradley-Tene variants the utility parameters were sampled independently from standard Gaussian, and for LCL. the learnable parameter matrix A was populated using sampled from (0, 0.1).
CRCS and LC-CRCS have non-convex likelihood functions, meaning that gradient descent is liable get stuck in local optima. To mitigate this, we performed inference multiple times (up to 50), starting from multiple starting points, and chose the parameters that achieved the best log-likelihood held-out part of the training data. Table 5 shows the variance in average NLL achieved by individual inference runs on the various choice problems. We can see that there is significantly higher standard deviation when doing inference with CRCS and LC-CRCS, pointing to the existence of local minima, and confirming the necessity repeated inference address this.
A.3 Additional details on rank consistency experiments
For District-Smart, we used gradient descent to infer the utility and choice model parameters for each choice model on the entire
binary choices. For LCL, CRCS, and LC-CRCS we noticed that inferred utility functions were highly inconsistent with the rankings that had been collected in the same user study. explained the main paper, we used regularization address this. We will go into some more detail why we think regularization was needed and how tuned the regularizers
16

---

expected eilllellhoddd
-1.0
1.5
-2.0
CRCS
LC-CRCS
Bradley-Terry
-2.5
LCL
Bower Balzano
-3.0
20
25
Number of preference queries
Figure 5: Mean Expected log- likelihood over unseen queries as function of the number of queries seen. Error bars respond to twice the standard error around the mean.
we used. Table beloow shows the consistency of the ranking implied by the inferred weights for each of these three choice models with the collected rankings. We can see immediately that compared
Table 6: Consistency between collected rankings and rankings implied by inferred weights with and without regularization of choice model parameter for LCL, CRCS and LC-CRCS.
LCL
CRCS (ours)
LC-CRCS (ours)
Without regularization
-0.36
-0.17
-0.04
With regularization
0.287
0.622
0.525
to Bradley-Tery and the Bower & Balzano models, the consistency is quite poor, especially for LCL. We hypothesize that this is caused by heterogeneity in the task utilities different participants used in making their choices. The intention of Kaufman et al. [38) was to capture humans' intuitive understanding of what it means for an electoral district be compact. Therefore, in the user study, people were encouraged to make choices "according to your own best judgement" (38]. As it is likely that the various participants in the study had slightly different intuitions about compactness, we can therefore expect that the recorded choices have been made with slightly different utility weights w. To fit choice model using single utility these choices could thus prove problematic. For future work, it would be interesting to consider a hicrarchical approach, where we would model the fact that any choice has been made according to an unobserved utility function drawn from some unknown distribution.
For the current work, resorted using regularization to ensure the choice models did overfit to the noise in the utility function. These regularization strategies were tuned using one of the six rankings collected, while for evaluating the consistency of the inferred weights with the collected rankings we used all six. For LCI., we regularized the weight update mechanism we + Axg by using the norm A. multiplied by 75 to get the desired regularization strength, as regularization term. For the CRCS model, to ensure that we make sensible inferences, we had to ensure that the noise in the utility function would be explained by the right source of noise in the model. The prior we placed on the choice model parameters was designed to do just this. We used placed Beta 1, 1000) ensure that e, which determines the level of noise on the attribute comparisons, would stay close to O. Attribute comparisons are the primary source of context effects and are necessary to fit to any such effects in the data. Additionally, we note that in the experiment conducted in [19] was fixed to We then placed a N (25.0,, 0.1) prior on Gentes the noise on the utility observations, to help explain the heterogeneity of utility functions itself. We also placed a weak prior on T....., Tá, namely the prior also used when training the CRCS model. The same priors used for LC-CRCS.
17

---

A.4 Additional details on elicitation experiment
The elicitation experiment on the Dumbalska dataset was performed for each participant in the original experiment individually. We excluded participants according to the same rule the original paper had used. For each run, we would select participant from the dataset and treat the queries to which responses had been collected for this participant as the queries we could put to the user. each time step, we used the posterior at that point to estimate the expected information gain of each query that had not been used yet, and selected the query with the highest information gain. The recorded response to this query would then be revealed, and the posterior would be updated with this new observation using the choice model. maintain the posterior, we used a particle filter containing up to five million particles representing combinations of utility and choice model parameters. The particle filter was not refreshed during experiment. At each time step we measured the expected likelihood, where the expectation was taken with regard to the posterior and uniform distribution over the remaining set recorded queries (those which had not yet been selected as part of the elicitation process). For completeness, we also measured the expected ogllikelihood and the entropy in the marginal utility parameter posterior. Those are shown in Figure
A.5 Additional information on the datasets used
We list below here the sources for the data we use in the human data experiments for Dumbalska [3)..
Car-Alt [39], Hotels [40] and District-Smart [38]. We obtained the human choice data for District-
Smart and Car-Alt from the excellent collection choice data collected by Tomlinson and Benson
20 for their implementation
Dataset
Location
Filename
Description
License
Dumbalska
OSF
decoy_233 Bee___paraptttsnnnneee
human choice
CC-By 4.0 Attr.
Data
Car-Alt
GDrive
car-alt.zip
human choice
data
Hotels
OSF
data.xls
human choice data and list of hotels used to train CRCS model.
GDrive
district-smart.pickle
human choice data
MIT
District-Smart
Github
training data. Data Preds.RDat
human rankings
MIT
Github
district features used to train CRCS model
MIT
B
Use cases
B.1
Structural design and water drainage network design
Here, we will describe additional details the crash structure design and water drainage network design use cases. Both use cases were run with the same priors for the utility parameters and choice model parameters. The utility parameter prior parameters for the CRCS model were chosen to capture the widest possible range of choice behaviors.
uniform Dirichlet distribution. The choice model
In each step, candidate choice queries were generated by sampling 1000 queries of three design options from a uniform prior over the domain of either use case. The candidate with the highest expected information gain was chosen. For crash structure design we elicited responses to 50 preference queries in each run the experiment and for water drainage network design we elicited responses to 100 queries, though for space reasons we only show the first 75 on the graphs in this paper. The recommendations on which we measured the recommendation regret at each step were selected by maximizing the expected utility under the current posterior over pre-caleulated Pareto
18

---

0.24
inference
0.23
Mean: inferencce
0.22
0.3
0.21
0.19
Number of preference queniss
Number of preference queries.
(a) Crash structure
(b) Crash structure
0.24
0.45
0.22
0.40
Mean model parameter
0.35
0.18
Number of preference queries.
Number of preference querles,
(c) Water drainage
(d) Water drainage
Figure 6: Additional Results for the experiments on car crash structure design and water drainage network design. (a) Utility parameter inference eтror on car crash structure design function the number of queries put to the user. (b) Choice model parameter inference error on car crash structure design as function of the number of queries put to the user. (c) Utility parameter inference error on water drainage network design as a function of the number of queries put to the user. (d) Choice model parameter inference error on water drainage network design as function of the number of queries put to the user. All plots show the mean = twice the standard error around the mean.
front for the user case. The user's optimal design was found by optimizing the true utility over this same front.
Figure shows the utility and choice model parameter inference errors for both use cases.
B.2
Retrosynthesis planning
B.2.1
Aizynthfinder
Aizynthfinde is open-source retrosyn- thesis planner that uses Monte Carlo Tree Search (MCTS) and a template-based expansion policy to search for possible reactions and an additional filter policy that discard the infeasible reactions. The expansion policy is a multi-class classification model that predicts the most probable reaction templates. In practice, this model produces the top 50 possible templates as the possible action during the tree expansion process. Then, the infeasible reactions are filtered out with the filter policy. During the expansion and selection phase of MCTS, it uses the upper confidence bound (UCB) to select and score routes as defined:
Here we provide further details about Aizynthfinder 45
Inn- Q + C
UCB
(5)
n
n
where is the visitation times of node, C is the bias hyperparameter set to 1.4, and Q is the accumulated reward that is defined as:
Nmolecutes stock
= 0.95
0.05 x depth
(6)
Nmotocutes
3 "odde available: 'download available:
19

---

0.45
utilty parameters
model parameters
0.40
Mean inference
0.35
0.30
0.25
0.20
0.15
10
20
30
40
50
Number of preference queries
Figure 7: Results for the experiments retrosynthesis planning. We show the mean inference error for the utility and choice model parameters as a function of the number of queries given to the user. Results shown with the mean + twice the standard error around the mean.
Table 7: Overview of layer sizes and training hyperparameters for ū and q for the choice tasks considered the experiments.
embedding (3 layers) emb. dim
main layers) output dims
oulput dims
batch
epochs
start/end
128,64
122,2
1024
35000
le-3/le-5
Risky Choice
g
128,64
1024,256, 128,3
1024
50000
1e-3/1e-6
Hotels
11
256,256.3
128
512.512,256
8192
10000
le-3/le-3
256,256
128
512,512,256.3
2048
50000
le-3/le-4
256,256
128
1024.512,128.2
8192
20000
le-2/le-4
District-Smart
512,256
1024,1024,256.2
4096
60000
1e-3/le-4
256,256
256
512,256,128.6
8192
30000
le-3/1e-3
Car-Alt
256,256
256
1024,1024,256,6
4096
100000
le-3/le-3
256,256
128
512.512.256.3
4096
40000
le-3/le-4
Dumbalska
128,128
128
512,256, 128,3
8192
25000
Ie-3/le-3
128,64
512,256, 128,3
1024
50000
1e-3/1e-6
Crash Structure
128,64
64
1024,256, 128,3
1024
35000
le-3/le-5
128,64
64
512,256, 128.3
1024
35000
le-3/le-5
Water Drainage
128,64
1024.256,128.3
1024
50000
le-3/le-6
128,64
512,256, 128.3
1024
35000
le-3/le-5
Retrosynthesis
128.64
1024,256,128,3
1024
50000
1e-3/le-6
where Nmotecates in stock is the current number of molecules in stock according to given database, N Nmotecates is the total number of the molecules in the current search tree and depth is the depth of the search tree.
B.2.2 Inference results
Now, report the inference results over the preference weighs. First, simulate the synthetic user weights by sampling 100 different weight combinations from uniform Dirichlet distribution Dir(a) with = (1, 1, 1, Figure 7 shows that both inference error and recommendation regret of the utility function and choice model parameters reduce during the inference using 50 preference queries.
In addition, we report the average number of solved routes from both Aizynthfinder and our model. For Aizynthfinder, we synthesize just 100 molecules, while for ours, we collect the statistics over 100 the target molecules for 100 inferred user utilities. The average number of solved routes is 45.62 28.99 and 42.75 28.89 for Aizynthfinder and ours respectively. These results are justified, the synthetic user utilities randomly sampled from non non-informative prior.
20

---

Embwoding Module
Hilden
Figure 8: Overview of the network architecture of ς. 77 has the same architecture but takes observations (u, as input. The outputs of are additionally transformed by log-softmax function (not shown).
C Surrogate architecture and training
We provide details here the architecture of the two neural networks ū and q. Both networks are multi-task networks; they make their predictions conditioned on the utility parameters w and the choice model parameters A E T) ū takes input vector of observations the expected utility of each option. 7 takes as input a set of options æ and predicts the likelihood that each option will be chosen.
and predicts
The architecture of both networks is virtually identical, differing only in their inputs, and in the fact that the output of is transformed with log-softmax function, while i's is not. Figure X visualizes the architecture of both networks. The networks consist two modules. The first in an embedding module consisting of three layers that takes the utility and choice model parameters and embeds them into latent embedding space. The second module, the main module, consists of four layers and takes input the set of options (or the observations in the case of i) and transforms these into log- ligliikl predictions (expected utilities for û). To condition the main module on the utility and choice model parameters,
concatenated the embedding from the embedding module with the input to layers 2 and 3 of the main module (see Figure We trained 7 and ② using the AdamW optimizer implemented in pytorch with exponentially decaying learning rate.
Both networks have number of hyperparameters such as layer output sizes. Other hyperparameters include training details such as learning rates and batch sizes. These hyperparameters were selected using a grid search over predefined set of possible values for each hyperparameter. We selected the hyperparameters that minimized the loss of each surrogate neural network each individual choice task. Table lists the selected network hyperparameters: the layer output sizes for both the embedding module and main module, and the size of the embedding produced by the embedding module. The last three columns list training byperparameters: the batch size, the number of training epochs, and the learning rate at the start and end of training.
D Computational resources
In table g we report the computational cost of the experiments reported in this paper. We only report the resources used to obtain the results presented. We estimate that if were include all testing and preliminary runs, the total compute time used would double or triple. We do not report the cost of validating the CRCS model on risky choice as the runtime was negligible.
All the experiments were run on either CPUs on cluster or on a MacBook Pro. The cluster CPU jobs were single core unless otherwise mentioned. MacBook CPU jobs used multiple cores (typically less than 2.5 cores) with 16GB of memory. We estimate the maximum power consumption of single core our cluster to be around 7.5W, and the power usage of single core on MacBook to be significantly less. Assuming the worst case scenario where all computation was run on the cluster, the total energy used to obtain the reported experiments would be 146KWh. Given an average carbon
21

---

Table Runtime for the various experiments reported in this paper. The column "runtime" lists the average computational resources used by each run of the experiment. The total time column lists the resources used by all runs combined.
Use case / Dataset
Experiment
Infrastructure
Hardware
runtime
Total time
T training
MacBook
CPU
lh
1h
Hotel
9 training
MacBook
CPU
1h
1h
NL.L evaluation
MacBook
CPU
7h
7 training
cluster
CPU
15h
15h
District-Smart
@ training
cluster
CPU
29h
29h
NLL. evaluation
MacBook
CPU
7h
7h
i training
MacBook
CPU
3h
3h
Car-Alt
9 training
MacBook
CPU
5h
5h
NLL evaluation
MacBook
CPU
20h
20h
i training
MacBook
CPU
1h
1h
q training
MacBook
CPU
1h
1h
elicitation BT
MacBook
CPU
85
10m
elicitation BB
MacBook
CPU
10s
13m
Dumbalska
elicitation LCL
cluster
CPU
28h
2100h
elicitation CRCS
cluster
CPU x2
20h
1500h
elicitation LC-CRCS
cluster
CPU x2
40h
3000h
NLL evaluation
cluster
CPU
1h
189h
7 training
MacBook
CPU
1h
1h
Crash Structure
i training elicitation
MacBook
CPU
2h
cluster
CPU
9h
4500h
7 training
MacBook
CPU
2h
2h
Water Drainage
ç training
MacBook
CPU
5h
5h
elicitation
cluster
CPU
10h
3000h
û training
MacBook
CPU
1h
1h
q training
MacBook
CPU
1h
lh
Retrosynthesis
generation AIZ generation w/ CRCS evaluation
cluster
CPU
6h
cluster
CPU
6h
589h
cluster
CPU
30m
30m
intensity of 94g CO2eq/KWh for the electricity supplied to cluster in 2023, this means that estimate the carbon footprint of these experimental results to be around 14kg CO2eq.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We test our proposed models on a varied set of data sets and use cases.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
The paper includes no theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide additional details the experiments in the appendices, such as sources for the data used, neural network architectures and priors used. The code for our experiments is additionally available online.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Our implementation of the experiments is available online. We provided direct links to the data needed run the experiments, and where processing the data needed the required code is provided.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
ctaips.cc/ubbicc/guidde
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All hyperparameters such as the number of experimental runs, number of data splits and priors are listed in the paper. These hyperparameters are also contained in the code which available online.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
The paper reports error bars suitably and correctly defined, providing appro- priate information about the statistical significance of the experiments. We ensured that all reported results include measures of variability, such as standard deviation or confidence intervals, to accurately represent the reliability and significance of our findings.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
25

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: All information about compute resources can be found in Appendix D includ- ing the type of compute workers, memory, execution time and infrastructure.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htw.sstttts.......
Answer: [Yes]
Justification; The research conducted in this paper adheres to the NeurIPS Code of Ethics in all respects. We adhered to privacy standards, obtaining explicit consent for data usage where applicable. Deprecated datasets were avoided, and data licensing terms were respected. We conducted thorough assessments to prevent biases and considered the societal and environmental impacts of our research, ensuring it does not facilitate harm, discrimination, privacy violations. Detailed documentation and secure data practices were followed to support transparency, reproducibility, and legal compliance.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
26

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impact of the paper in the conclusion.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: For the data we use we cite the papers which originally introduced those datasets. We also provide direct links to the data and license information in the appendices.
Guidelines:
27

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paapssttssppersssaaes paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification:
This paper introduces two new models, implementations for which are available
online.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Though the paper uses number of datasets collected from human subject experiments, it did not collect any new data itself.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
28

---

Answer:
[NA]
Justification:
This paper performed no new human subject experiments.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
29


Reviews:
[{"review_id": "kQyNYeA6RP", "reviewer": "Reviewer_CwtL", "summary": "The submission proposes an approach to preference learning using a model inspired by findings in cognitive science. Specifically, it uses an amortized inference variant of a previously proposed model to enable tractable inference of preference values, and applies it to a number of case studies, where it is shown to outperform both the classical Bradley-Terry model, and additional more recent baselines.", "questions": "* Reporting summed rather than averaged NLLs (with standard errors) seems like an odd choice -- why was this done? \n* Can the authors comment about the choice of Wilcoxon signed-rank test for some hypothesis tests and t-tests for others? \n* Why are baselines not present in the regret panels of figure 1?\n\nAdditional Notes:\n* Figure 1 tick labels should be larger. \n* L53 maybe \"computational rationality analysis\"? \n* L66 it's -> its\n* The notation in section 3.3. is a bit confusing, in the following way: u is a function of x, but the notation sometimes uses u and sometimes x (e.g. in the expectations in the utility and policy loss, or how \\tilde{u} is conditioned on u but \\tilde{o} is conditioned on x. I think this might be because of a distinction between the observer's and user's likelihoods, but this isn't quite spelled out here. \n* I believe that the overall setup here is a bilevel optimization problem (because L_util is optimized w.r.t. w and L_pol w.r.t. x), and I'm not sure how incomplete optimization or other issues in the utility network impact the policy network (especially considering that the accuracy of the utility network seems not ideal). \n* Line 278 I believe \\citet should be \\citep there?", "limitations": "Discussion is adequate, though as noted above I'm puzzled about the limitation on requiring sufficiently many choice sets for estimation.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper sets up its problem well and provides a suitable review of past work. I particularly appreciated the very crisp distinction between user's and outside observer's model in section 3.2. Performance is competitive relative to the baselines chosen, which include relatively recent methods.", "weaknesses": "My primary concerns with the paper are related to the actual technical implementation details, which are not articulated clearly enough, and where they are articulated give me some pause. Specifically: \n* If I understand things correctly, the policy network takes in $x, w, \\tilde{u} and \\tilde{o}$, and a sufficiently flexible surrogate should be able to learn arbitrary functions of x and w already. Is this correct? If so, why is the explicit feature engineering needed? The paper also discusses later \"built in\" context effects in CRCS and \"learned\" ones in LR-CRCS -- what is meant by this? \n* This also makes me wonder: what's the network architecture used and other training hyperparameters (optimizer, learning rates, etc)? In general I would expect a sufficiently flexible surrogate to also achieve better than ~92% agreement with the original model. I'm also puzzled about the claim of insufficient option data to claim the utility network -- shouldn't it be possible to simulate arbitrary amounts of observations in this case such that the network fully spans the range of possible outcomes and enabling amortized inference? I can see why sampling only from the training data would create this restriction, but I don't understand why this choice was made.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review of our paper and your feedback. We have addressed the main questions and concerns you have raised below.\n>the policy network takes in $x, w, \\tilde{u}$ and $\\tilde{o}$, and a sufficiently flexible surrogate should be able to learn arbitrary functions of x and w already. Why is the explicit feature engineering needed? [what is meant by] \"built in\" context effects in CRCS and \"learned\" ones in [LC]-CRCS.\n\nThank you for pointing out that this was unclear. To clarify, the observations $\\tilde{u}$ and $\\tilde{o}$ are not engineered features, but rather a core part of our cognitive model. As explained in section 3.1, the cognitive model theorizes that context effects come from the fact that humans make choices that maximize option utilities estimated from noisy observations [17], specifically noisy observations of each option’s utility ($\\tilde{u}$) and noisy feature value comparisons across options ($\\tilde{o}$). It has been empirically validated that maximizing these utility estimates given these observations leads to the context effects exhibited by humans. Our tractable surrogate for this model, CRCS, inherits these same context effects, which is why we describe them as “built in”. However, we found that this model did not reproduce all context effects present in the considered choice data. LC-CRCS addresses this by introducing a learnable component that is able to learn context effects not yet captured by CRCS itself. The advantage of having a model where these context effects are “built in” is that this creates inductive biases that lead to better utility function inference and choice prediction, as we show in the experiments.\n>This also makes me wonder: what's the network architecture used and other training hyperparameters (optimizer, learning rates, etc)?\n\nAlthough these details are available in the code we submitted in the supplement, we do agree that it would be easier for readers if they were available as part of the appendices as well. We will add details on the network architecture and sizes to the Appendix. We have provided a summary in the general response.\n>In general I would expect a sufficiently flexible surrogate to also achieve better than ~92% agreement with the original model.\n\nWe used the most stringent possible definition of agreement, namely that both models had to agree on the full ordering of the three option utilities. This is clearly a high bar. Moreover, the original model we compared to uses a Monte-Carlo estimator that estimates the expected values of the option utilities from a large but finite number of samples. Thus, the quantity we compare to is not some ground truth reference but rather another (noisy) estimate. As any error could be due to either the proposed surrogate or the original model being wrong, even a perfect surrogate would likely not reach 100% agreement.\n>I'm also puzzled about the claim of insufficient option data to [train] the utility network -- shouldn't it be possible to simulate arbitrary amounts of observations[...]? I can see why sampling only from the training data would create this restriction, but I don't understand why this choice was made.\n\nUnfortunately, in this specific instance, there was insufficient detail available regarding how the original user study, reported in [37], was done; no details were available on car and participant features that were used to construct the original option sets. We were therefore unable to generate option sets from a distribution that aligned well with the original experimental condition, forcing us to sample them from the training data only. This was not an issue for the other choice tasks, where the details of the user studies were more fully documented and we were able to simulate an arbitrary number of choices. Appendix A.1 discusses these issues in more depth.\n> Reporting summed rather than averaged NLLs (with standard errors) seems like an odd choice -- why was this done?\n\nWe reported summed NLLs in the same way as was done in prior choice modeling work [18]. However, we agree that additionally reporting standard deviations or errors would add value to the paper, especially - as reviewer hXMa explained - to help readers understand the robustness of the models to variations in the training data. We will add to the appendix a table listing the mean and standard deviations of the averaged NLLs obtained over the test folds in the choice data experiments (section 4.2). You can find this information in table 1 in the PDF attached to the global response.\n> Can the authors comment about the choice of Wilcoxon signed-rank test for some hypothesis tests and t-tests for others?\n\nIn most of our experiments we compared different model scores (e.g. NLL) on the same static data, and thus we used paired two-sample tests to compare the scores between models. As normality generally did not hold, we used Wilcoxon signed-rank tests for this. For experiment 4.3, where we used active learning, the evaluation data differed between the models as each would have selected different points to train on, thus we used an unpaired two-sample test, in this case an independent t-test.\nWe have noticed now that we misreported the hypothesis tests used in sections 4.2.1 and 4.3 – these should be switched. We used an independent t-test in 4.3 and a Wilcoxon signed rank test in 4.2.1. We will fix this in the paper and will explain briefly why each hypothesis test was chosen.\n> Why are baselines not present in the regret panels of figure 1?\n\nComparing to baselines was not the focus of these experiments. These are synthetic tests where we measured parameter recovery and practicality of CRCS in real-world experiments. No human data was used, and instead choices were generated from our own CRCS model. Clearly, our own model will fit best to choices generated by it, so any comparison to the baselines seemed to us as though it would be disingenuous, in this instance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KmkNQ6pzdN", "reviewer": "Reviewer_rsF2", "summary": "In this paper, the authors present a new model which approximates an existing intractable Bayesian model for preference learning. The paper describes a generative model for choice, and shows how inference can be approximated using two different neural networks. The proposed model is then augmented using a cross-feature correction inspired by another piece of existing work. Results show that the proposed model outperforms the baselines on sensible metrics like negative log likelihood of the test set.\n\n**Conclusion:** Overall, the paper makes a neat proposal and is fairly meticulous in developing the central idea. However, the contribution doesn’t appear substantial and might be of interest to a smaller subset of the community.", "questions": "NA", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper provides ample background and motivating examples, clearly elucidating the relevance of the problems.\n2. The development of the model is systematic, and key steps are formally defined making it easy to follow the progression of the ideas.\n3. The model appears more expressive in general since it takes into account different sources of noise, and more independent variables (such as the cross-feature influence or the error tolerance of comparing features).\n4. The results seem to be generally promising on both real and synthetic data.", "weaknesses": "1. While it was easy to follow the theory, I found it a little difficult to deeply understand the application section. A little more focus on methodology on at least one experiment might have been more helpful. Specifically, it was a little tricky to cleanly connect the theory to the application for me.\n2. The contribution is somewhat incremental, and might be better suited for area-specific workshop. Since the Bayesian model already exists, the main contribution is estimating the two intractable quantities using NNs, if I understand correctly. It’s useful to have such models, but I am not sure if they generate any foundational insights.\n3. For synthetic data, if the generative model is closer to the inference model, it’s not very surprising that the latter does better than another model unaware of the generative process. This weakens the evidence on synthetic data a little.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review and feedback on our paper. We address specific questions below:\n>While it was easy to follow the theory, I found it a little difficult to deeply understand the application section. A little more focus on methodology on at least one experiment might have been more helpful. Specifically, it was a little tricky to cleanly connect the theory to the application for me.\n\nThank you for this feedback. We will update the paper to more explicitly connect the data available in sections 4.2 and 4.3 to the observed and unobserved variables discussed in section 3: In 4.2 we are given a large dataset of tuples $(x^{l},y^{l})$ where each $x^{l}$ is a set of options presented to a user and each corresponding $y^{l}$ is the observed choice. Given this data we now look to infer the unobserved utility parameters $w$ and choice model parameters (e.g. $\\varepsilon, \\boldsymbol{\\tau}, \\sigma^2_{calc}$ in the case of CRCS) that best explain this data. The difference in 4.3 is that the dataset is not given all at once, rather we have access to the option sets $x^{l}$ and can actively select which $y^{l})$ we want to observe.\n>The contribution is somewhat incremental [...] Since the Bayesian model already exists, the main contribution is estimating the two intractable quantities using NNs, [...] It’s useful to have such models, but I am not sure if they generate any foundational insights.\n\nWe disagree that the contribution is incremental. The foundational insight is that a general theory of human cognition - namely computational rationality - offers a powerful and general inductive bias for machine learning. We study how to mitigate the effect context effects - a human bias widely studied in cognitive science - have on utility inference from preferences. Whereas prior work has attempted to account for these effects by learning them from data without any such inductive biases [18,35], we follow a model-based approach which leverages an existing cognitive model introduced by Howes et al. [17]. This model, which is an instantiation of computational rationality for decision making, is empirically validated to reproduce known context effects. This provides us with strong inductive biases that significantly improve utility function inference and choice prediction compared to prior work, showing that our model-based approach works for the important problem of learning from preferences. But the cognitive science literature suggests that deriving inductive biases from computational rationality can generalize to any setting in which people make decisions [A,B,C]. This is in our view a foundational contribution to both cognitive science and machine learning. We will discuss this contribution more explicitly in the introduction and discussion of the paper.\n\nWe also count a number of other tangible contributions. We have introduced a new choice model, CRCS, which improves utility function inference in preference learning compared to prior work. This required making the model introduced by Howes et al. [17] tractable using surrogates. We have also provided further empirical evidence for this specific model. This is significant because SOTA cognitive models such as [17, 38] are seldom tested on large choice, mainly because inference is generally intractable. Lastly, we have shown that there are context effects that the original Howes et al. [17] model does not capture, and have introduced LC-CRCS which introduced a learnable mechanism that can fit to these additional context effects.\n\n[A] Richard L. L., Howes A., and Singh S. \"Computational rationality: Linking mechanism and behavior through bounded utility maximization.\" Topics in cognitive science 6.2 (2014): 279-311.\n\n[B] Lieder F., and Griffiths TL. \"Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources.\" Behavioral and brain sciences 43 (2020): e1.\n\n[C] Oulasvirta A., Jokinen JPP., and Howes A. \"Computational rationality as a theory of interaction.\" Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 2022.\n>[The paper] might be of interest to a smaller subset of the community [...] might be better suited for area-specific workshop.\n\nWe disagree. Learning from preferences is an increasingly important approach in human-in-the-loop machine learning for inferring unknown utility functions. The modeling of those preferences (choices) is therefore relevant to a wide part of the ML community. We see this reflected in the fact that earlier work on choice modeling, which we improve on, has been published at top-level ML conferences such as ICML [35] and KDD [18]. Furthermore, our work fits within a wider body of recent work published at NeurIPS on human behavior modeling [D,E,F,G].\n\n[D] Belousov B., et al. \"Catching heuristics are optimal control policies.\" Advances in neural information processing systems 29 (2016).\n\n[E] Binz M., and Schulz E. \"Modeling human exploration through resource-rational reinforcement learning.\" Advances in neural information processing systems 35 (2022).\n\n[F] Chandra, K., et al. \"Inferring the future by imagining the past.\" Advances in Neural Information Processing Systems 36 (2023).\n\n[G] Teng T., Kevin L., and Hang Z. \"Bounded rationality in structured density estimation.\" Advances in Neural Information Processing Systems 36 (2024).\n>For synthetic data, if the generative model is closer to the inference model, it’s not very surprising that the latter does better than another model unaware of the generative process. This weakens the evidence on synthetic data a little.\n\nWe do not agree that this weakens the evidence. It is clear that aligning a system to a user’s utility will make the system more useful. However, how to achieve such alignment in practice is not trivial. The simulated use cases we present point to the fact that preference learning with our model is a practical solution for this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "paEaXFp68f", "reviewer": "Reviewer_hXMa", "summary": "The paper introduces two models for learning preferences from human choice behaviors: the Computationally Rational Choice Surrogate (CRCS) and a variant (LC-CRCS) that considers the context effect of the choice set. Both models are based on a state-of-the-art cognitive model of human decision-making. The paper replaces intractable computations in the original model with surrogates to enable feasible estimations and demonstrates the numerical performance of these models through several experiments.", "questions": "1. How can the true utility or its generation process be observed or estimated in real-world scenarios?\n2. How should the prior distributions in Table 3 be selected or estimated? Are there general methods for estimating these prior distributions from a given dataset?\n3. How do the models perform on other metrics (e.g., accuracy)?\n4. What are the architectures of neural networks $\\hat{q}$ and $\\hat{u}$?", "limitations": "See weaknesses.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and presents a new model, which is based on Howes et al. (2016), for understanding human choice behaviors. Compared to the original model, the new model is more computationally tractable. Also it uses several real-world datasets to test the models.\n\nAndrew Howes, Paul A Warren, George Farmer, Wael El-Deredy, and Richard L Lewis. Why contextual preference reversals maximize expected value. Psychological review, 123(4):368, 2016.", "weaknesses": "1. Limited Applicability: The models require data that includes the orderings of each feature (or their generation processes) and the true utilities (or their generation processes). Such data is not commonly available. Additionally, prior knowledge of several parameters must be estimated before training the models. For instance, each feature needs a distribution for the corresponding $\\tau$, which becomes challenging when the number of features is large.\n   \n2.Experimental Focus: The experimental results primarily focus on negative log-likelihoods (NLLs). Although the proposed methods outperform benchmarks in terms of NLLs, it is unclear if this improvement is practically significant. Comparing other metrics (e.g., accuracy) would better demonstrate the contribution of this work. Moreover, the paper does not provide standard deviations for the results to indicate robustness, even though the results are statistically significant.\n\n3. Training Robustness: Line 561 mentions that \"As CRCS and LC-CRCS have non-convex likelihoods, we performed gradient descent multiple times, starting from multiple starting points, and chose the parameters that achieved the best train set log-likelihood.\" It would be helpful to show the mean and standard deviations over several random seeds instead of just the best result to demonstrate training robustness. Training on large datasets can be time-consuming, making multiple training runs impractical.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We address specific questions below:\n>Limited Applicability: The models require data that includes the orderings of each feature […] and the true utilities\n\nThis is incorrect.The data we used was straightforward choice data consisting of the options presented to the user and the choices they made. While our cognitive model does assume that choices are informed by the ordering and utility observations you mention, these observations happen “inside the choice maker’s head” and are thus latent to an outside observer (see eq. (3)). We will update the paper to be more explicit about this. We have also included graphical models (Figure 1b and 1c in the rebuttal PDF document attached to the general comment) which clarify which variables are observed and which are not to both the choice maker and an outside observer.\n>The experimental results primarily focus on negative log-likelihoods (NLLs). […] Comparing other metrics (e.g., accuracy) would better demonstrate the contribution of this work.\n\nThe evaluation of our model and the baselines on choice data sets follows prior work [18]. All models we consider predict a probability for each choice (they are probabilistic classifiers), rather than a single deterministic choice. This is important as human choices do not appear to be deterministic. Thus, our main concern is that these models should be well-calibrated. This is not something we can check with a measure like accuracy. Rather, it requires a proper scoring rule, like the NLL we used. In our other experiments - which were more representative of a typical preference learning setting - we did use other metrics to show significant practical improvement. For example, we measured how consistent a ranking implied by the inferred utility is with ranking collected from users in section 4.2.1.\n>the paper does not provide standard deviations for the results to indicate robustness, even though the results are statistically significant.\n\nWe agree that this would be a good supplement to the statistical significance tests already carried out in the paper. We will add to the appendix a table listing the mean and standard deviations of the averaged negative log-likelihoods obtained over the test folds in the choice data experiments (section 4.2). A separate set of utility and choice model parameters was inferred for each test fold, so this should give readers a good idea of the robustness of the models. We have added this table to the PDF attached to the global response (Table 1).\n>[regarding L561: doing gradient descent multiple times] It would be helpful to show the mean and standard deviations over several random seeds instead of just the best result to demonstrate training robustness.\n\nWe should stress that we used gradient descent multiple times to infer the latent choice model and utility parameters from human data, not to train the CRCS model itself. Even on the largest choice dataset, each inference run took only a few minutes. As requested, we will add a table to the appendices showing mean and standard deviations of the average NLL obtained over several independent gradient descent runs. We have added this table to the PDF attached to the global response (Table 2).\n>How can the true utility or its generation process be observed or estimated in real-world scenarios?\n\nThis question gets to the heart of what makes human modeling difficult. As we cannot look inside people’s heads, it is impossible to observe humans’ true utilities or their true choice making process. Modeling lust follow the standard scientific process of proposing a new model based on some theory or assumptions of how this process works, and then evaluating that model against competing baselines. A choice model (including the underlying utility function) can be evaluated in two ways: i) we can check if the model correctly predicts people’s choices (experiments in section 4.2 and 4.3) and ii) we can check if the inferred utility is consistent with other observations that are derived from that same utility function (e.g., the evaluation of the implied rankings we performed in section 4.2.1).\n>How should the prior distributions in Table 3 be selected or estimated? Are there general methods for estimating these prior distributions from a given dataset?\n\nThe priors for the choice model and utility parameters have very little effect on the model itself. However, as they are used when training the model, it is important that they cover any parameter values that we may wish to use in practice. We followed standard Bayesian practice for selecting them. As we had little prior knowledge of which parameter values were most likely in practice (beyond some insights from Howes et al. [17] regarding which parameter values actually affected the model’s output), we used flat priors that were as broad as sensible (see last paragraph of appendix A.1).\n>What are the architectures of neural networks $\\hat{q}$ and $\\hat{u}$.\n\nWe will add these details to the appendix. We have provided an overview of the architecture of both networks in the general response and a visualization in Figure 1a in the accompanying PDF.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LT5xZpi5cP", "reviewer": "Reviewer_cVZM", "summary": "This paper uses a state-of-the art-models model from psychology for preference learning. To make learning tractable, the authors use variational inference to approximate unknown distributions (and expectations). The model is shown to outperform existing approaches, suggesting that it better predicts human preferences and captures contextual choice effects that are commonly observed in human behaviour.", "questions": "1. What is the number of parameters for each of the models? It will be good to have these reported in Table 1.\n2. I might have missed it, but I didn't see the network architectures for $\\hat{u}$ and $\\hat{q}$?\n3. Have you thought applying the model to LLMs for preference learning?", "limitations": "Some of the limitations of this work are discussed in Section 5. Societal impact statement is also included in Section 5.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Preference learning is a very relevant topic. This work borrows a state-of-the-art model cognitive model from psychology (Howes et al., 2016) and aims to make it computationally tractable to be applied in the context of preference learning.\n\n- The paper is well-motivated and clearly written. It effectively balances the mathematical theory with intuitive explanations.  \n\n- I find this work to be a nice example of cross-disciplinary ML project: model from psychology, use of variational methods to make it computationally feasible, diverse set of experiments (e.g. policy making, hotel/property rankings)\n\n- Thorough empirical evaluation: section 4.1 ensures that the learnt surrogate is a good approximation (when compared to MCMC); then in the following subsections, CRCS is evaluated on a diverse set of datasets (as already mentioned in the previous bullet), as well as simulated case studies. K-fold cross validation is performed, and Wilcoxon rank test  is done to check for statistical significance.", "weaknesses": "No major weaknesses. Some minor suggestions:\n\n- Line 56 and the whole paragraph after: the meaning of CRCS acronym is not introduced until line 125.\n\n- line 153: when reading this paragraph for the first time, I found it strange that the parameters $w$ are observed (known). It then became clear that they are observed only by the *user* that is being modelled. \n\n- a graphical model describing all the variables in the model can be useful; it can help visualise the differences between the modelled user and the outside observer (which variables are observed or unobserved). Clearly indicating what is given and what is learnt - e.g. the prior $p(w)$ and the marginal $p(x)$ are both assumed given?\n\n- line 221: typo though -> thought", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review of our paper and thoughtful feedback. We address your main questions and feedback below:\n>I find this work to be a nice example of cross-disciplinary ML project: model from psychology, use of variational methods to make it computationally feasible, diverse set of experiments.\n\nThank you for the positive feedback!\n>What is the number of parameters for each of the models? [what are] the network architectures for $\\hat{u}$ and $\\hat{q}$.\n\nWe will include details regarding network architectures and sizes to the appendix. We have provided a summary of these details in the general response. In the PDF attached to the general response, you can find the number of parameters for each model in Table 3, and a visualization of the network architecture in Figure 1a.\n> Have you thought [of] applying the model to LLMs for preference learning?\n\nThis is a very good point. LLMs have had a tremendous impact on preference learning research and properly aligning them to human preferences is an important societal problem. Our results suggest that modeling context effects is important when learning from human preferences, and so we do consider applying our model to LLMs important future work. Right now our model assumes that options are represented by features. Thus, given some featurization of, say, LLM text outputs for a given prompt, our model could be directly applied. However, this would ignore the reading and interpreting of these texts that human evaluators have to do. We think additionally modeling those tasks in the choice model, by extending the foundation provided by the current model, could be even more transformational. Based on what we have written here, we will add a more detailed discussion on the implications of our work to LLM fine-tuning to the related work section.\n> the meaning of CRCS acronym is not introduced until line 125.\n\nThank you for pointing this out. We will introduce the acronym earlier.\n>a graphical model describing all the variables in the model can be useful\n\nAgreed! We have added two graphical models in the PDF attached to the global response. One shows the problem from the point of view of the user making a choice (Figure 1b in the attached PDF) and the other shows it from the point of view of an outside observer trying to infer the user’s utility function (Figure 1c in the attached PDF). We will add these figures to the appendix of the paper.\n>line 153: when reading this paragraph for the first time, I found it strange that the parameters are observed (known). It then became clear that they are observed only by the user that is being modelled.\n\nThank you for pointing this out. As another reviewer also got confused by this, we will include the graphical models and also update the writing to clarify that these observations indeed happen in the user's head and are not observable from the outside.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for the time and effort dedicated to review of our work and for the helpful and constructive feedback.

## Motivation and focus of the paper
Our paper presents a cross-disciplinary approach to preference learning. Humans are known to exhibit a number of context effects when making choices. When these effects are not properly accounted for, learning from choices (preferences) may lead to incorrect inferences of the underlying utility. This has important implications for ML systems that learn from preferences.

The foundational contribution of our paper is the insight that computational rationality, which posits that humans are rational under bounds, can offer a step change in how ML systems learn about humans. Concretely, we propose a model-based approach to preference learning which leverages a SOTA cognitive model [17] derived from this computational rationality theory. The empirically validated computational rationality assumptions built into this model induce known context effects. Using this model, or rather a new tractable variant of it, for learning from preferences therefore introduces a strong inductive bias in our inferences. This type of inductive bias has not been used in preference learning before, and we show that it significantly improves utility inference and choice prediction compared to prior work.

The paper therefore offers a number of concrete contributions. We introduce CRCS, a tractable version of an existing cognitive model by Howes et al. [17] that supports efficient inference of the utility function and model parameters. We show that CRCS can be used in large-scale learning from preferences and that it makes better inferences than a set of recent baselines. This has practical benefits for any existing work that uses learning from preferences. To account for any context effects not yet captured by CRCS, we additionally introduce LC-CRCS, which is able to learn any additional context effects from human data.

## Clarification on neural network architecture
As requested by several reviewers, we provide here details on the architecture of the two neural networks we use in the paper, $\hat{u}$ and $\hat{q}$. These approximate key intractable computations in the paper: $\hat{u}$ takes as input a vector of observations $(\boldsymbol{\widetilde{u}}, \boldsymbol{\widetilde{o}})$ and predicts the expected utility of each option. $\hat{q}$ takes as input a set of options and predicts the likelihood that each option will be chosen. Both networks are multitask networks; they are additionally conditioned on the parameters of the utility function $w$ and the choice model parameters $(\varepsilon, \boldsymbol{\tau}, \sigma_{calc}^2)$.

The architecture of both networks is virtually identical, differing only in the dimension of their inputs, and in the fact that the output of $\hat{q}$ is transformed with a log-softmax function. Figure 1a in the attached PDF shows the architecture of $\hat{q}$. The network design consists of two modules. The first in an embedding module consisting of three layers that takes the utility and choice model parameters and embeds them into a latent embedding space. The output dimension of the layers and the dimension of the embedding space are given in Table 3 in the attached PDF. The second module, the main module, consists of four layers and takes as input the set of options $x$ (or the observations in the case of $\hat{u}$) and transforms these into likelihood predictions (expected utilities for $\hat{u}$). The output sizes of these layers are also listed in Table 3. To condition the main module on the utility and choice model parameters, we concatenated the embedding from the embedding module with the input to layers 2 and 3 of the main module (see figure). We trained $\hat{u}$ and $\hat{q}$ using the AdamW optimizer implemented in pytorch. We used an exponentially decaying learning rate. We list the batch size, number of epochs for training and the starting and ending learning rates in Table 3 in the attached PDF.

## Additional experimental results and clarity improvements
We have collected additional empirical results to support our response to reviewers’ comments. The new figures and tables are in the rebuttal PDF document (attached to this comment). Based on the feedback from the reviewers, we will also expand the paper and incorporate a number of clarifications. We list the most important changes below:
- We will clarify the description of the model. Specifically, we will be more explicit about which variables are observable to the user and to an outside observer. We will also more clearly delineate what part of the theory concerns the cognitive model and what part concerns learning from preferences.
- As requested by reviewer cVZM we have added to the appendix two graphical models, one showing the cognitive choice model (Figure 1b in the attached PDF) and one showing how this fits within the larger inference problem of learning from preferences (Figure 1c in the attached PDF).
- We will update the paper’s introduction and discussion sections to clearly state the foundational insights contributed by this work, based on the response we gave to reviewer rsF2.
- As requested by reviewers CwtL, hXMa and cVZM we have provided further details about the architecture and training details of the neural networks above. We will add these details to the appendix of the paper.
- As requested by reviewers hXMa and CwtL we have additionally reported averaged NLL figures with standard deviations for the choice set experiments in Table 1 in the attached PDF. These will be added to the appendix of the paper.
- As requested by reviewer hMXa we have included a table showing the mean and standard deviation of the average NLL obtained over several gradient descent runs in Table 2 in the attached PDF. We will discuss this table in the appendices.

Meta-review: The paper proposes modeling of choice behavior by using models of human choice from cognitive science as priors linked to data by neural network models. This allows fitting a broader set of possible data than is possible under intractable cognitive science models, and fitting to data that are not well explained by such models. Detailed experimental simulations demonstrate the efficacy of the solution. 

Opinions of this paper are quite split. I have read the reviews, responses, and the paper itself. I lean toward acceptance. The reviewers had a large number of legitimate questions, which seem to be answered in the responses. Addressing these will require non-trivial work on the authors' part. The remaining question is about the contribution. The authors argue that using cognitive models as a prior is in important contribution, one that is supported by their empirical results. The authors are generally less convinced that the idea is important, given that there are not significant technical advances. I lean toward acceptance because ideas that supported by data can be important contributions in general, and in this specific case, improving modeling human choice behavior at scale is an important problem.
Decision: Accept (poster)