Preference Learning of Latent Decision Utilities with a Human-like Model of Preferential Choice

Authors: sebastiaan  de peuter
shibei  zhu
yujia  guo
andrew  howes
samuel  kaski
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 17:38:40

Abstract: Preference learning methods make use of models of human choice in order to infer the latent utilities that underlie human behaviour. However, accurate modeling of human choice behavior is challenging due to a range of context effects that arise from how humans contrast and evaluate options. Cognitive science has proposed several models that capture these intricacies but, due to their intractable nature, work on preference learning has, in practice, had to rely on tractable but simplified variants of the well-known Bradley-Terry model. In this paper, we take one state-of-the-art intractable cognitive model and propose a tractable surrogate that is suitable for deployment in preference learning.  We then introduce a mechanism for fitting the surrogate to human data that cannot be explained by the original cognitive model. We demonstrate on large-scale human data that this model produces significantly better inferences on static and actively elicited data than existing Bradley-Terry variants. We further show in simulation that when using this model for preference learning, we can significantly improve a utility in a range of real-world tasks.

Main paper: 
 introduction: 
 AI systems need exact descriptions of tasks to be performed. However, humans find more complex tasks hard to describe. In response, preference learning has emerged as one way to learn from human feedback. It has been used to teach AI systems a variety of tasks from how to hand objects to humans to how to play Atari games [1] [2] [3] [4] . More recently, human feedback has been used to train large language models to summarize text [5] , answer questions in natural language [6] , and to train deep generative models to generate realistic medical images [7] .When learning from human feedback, it is generally assumed that some latent utility function f guides an individual's behavior, but that the individual cannot describe this function to the machine. Thus, preference queries are used to elicit information about f from the user. A preference query gives a user a set of options x 1 , . . . , x n and asks the user to select their preferred option, i.e., the one with the highest utility. Given a model of how people make such choices, the machine can then infer the underlying function f from the user's chosen item y. For example, Stiennon et al. [5] learned a utility function for text summaries by showing users a text with several summaries and asking them to choose the best summary.There are several models of choice that have been used for learning preferences from human choices. Some recent work on Reinforcement Learning from Human Feedback (RLHF), for example, has used a simple binary choice model [5, 6] p(y = x 1 |x 1 , x 2 ) = σ(f (x 1 ) -f (x 2 )), over choices x 1 and x 2 , though generally, most preference learning approaches have used the Bradley-Terry model [8] p(y = x i |x 1 , . . . , x n ) = exp(βf (x i )) n j=1 exp(βf (x j )).Although these models have proven to be practical, they are not realistic models of human choice behavior. Specifically, both models make choices between two options without taking into account the rank orderings of option attributes; a widely observed property of human decision-making [9] [10] [11] .As a result, these models fail to predict a number of apparent biases in human choice behavior. These include contextual choice effects [12, 13] , which occur in situations where a decision maker's choice between two options is influenced by adding more options to the choice set [14, 12] . Say, for example, we have two options A and B and a user exhibits a probability of choosing between these. When a third decoy option C is introduced which is strictly dominated by B, there is a shift in the probability of choices from A to B.Though context effects are not certain to appear in preference queries posed to users, they are known to appear in a wide range of human tasks including risky choice tasks [12] , multi-attribute choice tasks [15] and perceptual judgement tasks [11] and in many other species including jays and honeybees [16] . These effects point to a potential gap in the accuracy of the models currently used, during preference learning, to interpret the choices made by users. Moreover, this gap has the potential to lead to incorrect inferences about the latent preference utilities of observed human decision-makers.The contribution of this paper is threefold. First, we show that we can improve preference learning by leveraging computational rationality theory, a general cognitive-scientific theory which posits that human behavior is rational under cognitive bounds [17, 18] . We learn preferences from human choice behaviors using a state-of-the-art cognitive model that is based on a computational rational analysis of context-dependent choice under uncertainty and is backed by substantial empirical support in the psychology literature [19] . Like all computationally rational models, behavior under this model emerges from the latent utility function and a latent set of cognitive bounds. This provides strong inductive biases when inferring these latent factors from human behavior which -as we will show experimentally -significantly improves learning from preferences. Our second contribution lies in making this cognitive model amenable to preference learning. To this end, we generalize it, and make inference practical by approximating intractable calculations with a surrogate we call the Computationally Rational Choice Surrogate (CRCS) model. Finally, we find experimentally that CRCS can sometimes perform worse than the Linear Context Logit (LCL) model [20] . We hypothesize that human context effects are partially a consequence of cross-feature effects. These are not modeled in CRCS, but can be learnt by LCL. We therefore propose LC-CRCS, which takes advantage of these effects by combining CRCS with LCL.We report three sets of experiments. In the first, we show that CRCS matches the original model's prediction of human choice behavior. In the second, we compare preference learning with CRCS to preference learning with recently proposed variants of the Bradley-Terry choice model. Using existing human data sets, we show that CRCS outperforms these in choice prediction and utility function inference, but performs worse than LCL on some tasks. We then show that LC-CRCS can additionally outperform LCL in these tasks. In the third set of experiments, we show the applicability of CRCS in three real-world use cases and verify its parameter recovery capability 2 Background 
learning from preferences: 
 Preference learning methods aim to infer latent utility functions from human choices. Depending on the type of queries presented, there are two main streams of research: (1) learning from pairwise comparisons or (2) learning from ranking, where humans rank a set of n options. Popular methods include Gaussian Process regression that captures the preference relationships of pairwise queries [21, 22] . Other work, such as as [23] [24] [25] , uses Deep Neural Networks trained on ranked demonstrations to approximate the underlying reward functions. To reduce the computational burden created by the necessity for numerous queries, active learning techniques [26] [27] [28] [29] have been proposed for efficient query proposal with maximum information gain. However, these methods typically require consistent preference order within the ranking and do not consider any contextual effects within the query dataset. Reinforcement Learning approaches include Preference-Based Reinforcement Learning (PBRL) and Reinforcement Learning with Human Feedback (RLHF), where the reward function is inferred from the preference feedback. Work using ranking queries and human feedback can reach or even exceed human-level performance in several RL benchmarks [23, 24] .To date, preference learning research has yet to make use of plausible models of human decisionmaking such as [30, 31] . These models are inspired by extensive studies of human behavior and give rise to contextual choice effects. Consider a hypothetical choice between two sightseeing trips, one to Paris and the other to London. Both trips come with free coffee. Let's say that 70% of people prefer Paris and 30% London. Now imagine that we add a third option which is identical to the London trip but without free coffee. If, for this three-trip choice problem, we observe that 40% prefer London with free coffee, then we will have observed a contextual choice effect known as a "preference reversal" [32] . The choice frequency for London with coffee is increased by a context that includes a dominated choice. This effect has been observed both in sample averages and, more interestingly, within individuals. It has been taken as evidence that people are irrational [33] and have no stable preferences [34] . Needless to say, both instability and irrationality pose severe challenges to the viability of preference learning.More recent theories, however, demonstrate that contextual choice effects can be consequences of computationally rational processes that assume stable preferences. These theories explain contextual choice effects by modeling the fact that people compare attributes and/or utilities under uncertainty. These include Bayesian theories [35] , rational analyses [19] , and neurobiological relative encoding theories [13, 36] . These theories use comparisons between option attributes to compute expected utilities, such that these expectations are sensitive to the reliability of the comparison as an indicator of expected utility.Other work has proposed variations on the Bradley-Terry model to include these contextual effects, with the same commitment to stable preferences. Bower and Balzano [37] posit that context effects are the result of humans comparing options only on the k most salient features within a context. They propose a Bradley-Terry model where utilities are calculated only on the k most salient features, where saliency is measured by the sample variance of each feature within the current set of options.Tomlinson and Benson [20] do not propose a specific theory of context effects, but rather propose to learn them from data. They introduce the Linear Context Logit (LCL) model, a Bradley-Terry model with a linear utility function, in which context effects are modeled as a context-dependent change in the (globally stable) weights of the utility function. This change is modeled as a linear function of the average attribute values of the set of options presented to a user (the context), and is inferred from human choice data. They further introduce the Decomposed LCL model, in which each feature induces its own context effect -whereas in LCL the features jointly inducing a single context effectand where the final context effect results from a mixture of these individual effects.In the current paper, we commit to distinguishing behavioral choices, which are observable, from latent preferences, which are not. When we refer to "preferences" we are referring to the latent utility function f (x), and not to the observable choice behavior.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations of the work in the conclusion.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
modeling computationally rational choice: 
 To learn latent preferences from human choice behavior, we build on a computationally rational model of choice behaviors by Howes et al. [19] which is sensitive to the aforementioned context effects. This model assumes that humans make utility-maximizing choices, but that the option utilities are estimated from noisy observations of the true utilities and noisy comparisons between the option attributes. Here we will first describe the original model in a general form. We then extend it to a general space of utility functions and introduce our Computationally Rational Choice Surrogate (CRCS) model, a model which replaces intractable computations in the original model with learned surrogates to allow tractable inference of the latent utility function. Finally, we introduce the LC-CRCS model, an extension of the CRCS model which is able to learn additional context effects not captured by the CRCS model.Let x 1 , . . . , x n ∈ R d be a set of n options, each with d attributes. Let f : R d → R be a latent utility function that maps each option to its associated utility. As a shorthand, we will denote the utilities of a collection of options x = ⟨x 1 , . . . , x n ⟩ as u = ⟨u 1 , . . . , u n ⟩ where u i = f (x i ).The cognitive model introduced by Howes et al. [19] assumes that when making choices, humans do not observe the options x 1 , . . . , x n nor their utilities directly. Instead humans are assumed to make utility-maximizing choices based on two sets of noisy observations of the options. The first set are noisy observations u = ⟨ u 1 , . . . , u n ⟩ of the true utility of each option. These are modeled as samples from a Gaussian centered around the true utilities ∀i ∈ 1, . . . , n :u i ∼ N (f (x i ), σ 2 calc ) with noise σ 2calc which we will call the calculation noise. The second set are noisy observations of the ordinal relation between the values of each attribute for each pair of options. Given an attribute k and a pair of options (x i , x j ), this ordinal relationship is defined by the following observation function:o(x i,k , x j,k ) =    ≺ iff x i,k < x j,k -τ k ≻ iff x i,k > x j,k + τ k ≡ elsewith τ k an attribute-specific tolerance parameter. Intuitively, a larger τ k creates a greater margin within which attribute values will be considered equal. For binary attributes, we set τ k to zero. Each noisy ordinal observation o(x i,k , x j,k ) is sampled as follows: with probability1 -ε sample o(x i,k , x j,k ) = o(x i,k , x j,k), otherwise sample uniformly at random from {≺, ≻, ≡}. The probability of ordinal error ε is a parameter, and is the sole source of noise within the ordinal observations. We will denote the set of noisy ordinal observations as o = { o(x i,k , x j,k )} k=1...d,i=1...n,j=i+1...n .Given these observations u and o for options x 1 , . . . , x n , and the choice model parameters θ = (σ 2 calc , ε, τ 1 , . . . , τ d ), the above model implies a posterior distribution over the options' true utilities p(u| u, o, θ) and associated expected values E[u i | u, o, θ]. As they do not observe true utilities of the options, humans are assumed to choose the option y with the highest expected utility:y = argmax xi∈{x1,...,xn} E[u i | u, o, θ].Preference learning requires that we are able to reason about how various utilities lead to different choice behaviors. Therefore, to make the original cognitive model amenable to preference learning, we replace the fixed utility function f by a space of utility functions {f w } w∈W parameterized by a utility parameter w. We assume that the user being modeled makes choices based on some chosen parameter value w, which is known only to them, and which we represent as an additional observed random variable in the model. Necessarily, any calculation of utility therefore depends on w. Under these assumptions the user's posterior over utilities, and thus their choice y, is:EQUATIONwhere this expectation is calculated under the posteriorEQUATIONIn our description of the model above, we have taken the point of view of the user making the choices. However, we now return to a preference learning perspective, i.e. that of an outside observer such as an AI system trying to infer the utility function that underlies these choices. We assume that the AI system observes the presented options x 1 , . . . , x n , as well as the option y the user chooses. The goal is then to infer the unknown utility parameter w and choice model parameters θ from observed choices (x, y). However, the noisy observations u and o on which the user bases their choice are part of their internal perception of the options, and are therefore not observable to an AI system. This means that in evaluating the likelihood of a choice y under the above choice model we must treat the observations as latent. This yields the following choice policy for the user:EQUATIONHere, p(y| o, u, w) is a point mass on y following equation (1) . Given m pairs (x (l) , y (l) ), a prior p(w) over the space of utility parameters and a prior p(θ) over the space of choice model parameters, we can use the likelihood in equation ( 3) to infer a posterior over the parameters w and θ:p(w, θ|{(x (l) , y (l) )} m l=1 ) ∝ p(w)p(θ) m l=1p(y (l) |x (l) , w, θ).The issue we face in calculating p(w, θ|{(x (l) , y (l) )}) is that the likelihood p(y|x, w, θ) is intractable. First, the calculation of the expected values in equation ( 1) requires the evaluation of an intractable integral over x in equation (2) . The expected values can be approximated using a Monte Carlo estimate [19] , but many samples are needed to achieve a good approximation. Second, the calculation of the likelihood itself requires the evaluation of an intractable integral over all possible observations in equation (3) . As before, one could approximate this integral using a Monte Carlo estimate, but this would again require many samples.Instead, we propose to train surrogate neural networks to approximate both these quantities. We introduce a first neural network u( u, o, w, θ) trained to predict a vector of the expected values E[u| u, o, w, θ] from given observations u, o and parameters w and θ. Then u(•) is trained by minimizingEQUATIONSamples (w, θ, u, u, o) are obtained by (1) sampling w ∼ p(w), θ ∼ p(θ) and x ∼ p(x) from their respective priors, (2) calculating u i = f w (x i ) for each option, and (3) sampling the observations u i ∼ p( u i |u i , θ) and o ∼ p( o|x, θ). Note that the minimum of L util ( u) is exactly the function that assigns to each tuple ( u, o, w, θ) the vector of expectations E[u| u, o, w, θ].Next, we train a second neural network q(y|x, w, θ), which we will refer to as our CRCS model, to approximate the user's policy p(y|x, w, θ) over choice behaviors. By using the fact that u( u, o, w, θ) ≈ E[u| u, o, w, θ] we minimize the cross-entropy loss between q and choices based on utilities predicted by u. The loss function is thus:L pol ( q) = E p(w,θ,x, u, o)-ln q argmax {x1,...,xn} u( u, o, w, θ) x, w, θ .Samples (w, θ, u, u, o) are obtained as above.Although our proposed model can predict a range of context effects, it does not yet capture all.Although CRCS can model how each individual feature influences the expected utility of the options, it cannot model how features can impact each other. This is something that LCL does do: its utility weight updating mechanism changes the weight of each feature based on the mean value of all other features. Thus, features can influence how other features are valued. In the most general sense, LCL's fundamental mechanism corresponds to a function g(w, x) which maps the utility weights w and the set of options x (which make up the context) to a new set of weights w ′ . We therefore propose to integrate this same mechanism into CRCS, resulting in a new model q(y|x, g(w, x), θ). As q is differentiable, we can infer g from data using gradient descent. In the experiments that follow, we will use this approach in settings where f w is a linear function. Thus, like LCL, we will define g as a linear function of x C : the mean attribute values of the options x. We will refer to the resulting model q y x, w + (Ax C ) T , θ as LC-CRCS. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Our implementation of the experiments is available online. We provided direct links to the data needed to run the experiments, and where processing of the data is needed the required code is provided.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 We first validate the proposed CRCS model by comparing its results with the original computationally rational choice model by Howes et al. [19] . Then we compare the proposed CRCS model and our LC-CRCS variant with three baselines on human choice data, and finally study the performance of the model on three case studies: car crash structure design, water drainage network design, and retrosynthesis planning. 1We evaluate our proposed CRCS model on four datasets of human choices. These datasets are large sets of choices (x (l) , y (l) ) collected from human participants. The District-Smart dataset [38] contains pairwise preferences over voting districts, where participants were asked to choose the district they felt was most compact. The features extracted for each district are six geometric measures identified by the original authors as good measures of compactness. The Car-Alt dataset [39] contains choices between six hypothetical alternative fuel cars. Each car has 21 features, including size, range, operating cost, etc. We also use a dataset collected in [40] , which we will call the Hotels dataset, where in a user study participants were asked which of three hotels they preferred. The hotels were collected from a booking site and had as features the price per night and average review rating. For each participant, a choice was collected on one of six sets of options constructed to target three known context effects: attraction, compromise and similarity. Lastly, we use the data collected by Dumbalska et al. [36] on a property task, which we will refer to as Dumbalska. Here, participants ranked three properties in order of best to worst value. For our purposes, we will treat the top-ranked item as the choice. Value was defined as the given rental cost minus the value participants thought the house was worth (which had been elicited in an earlier stage). For each participant, responses were collected on a large collection of choices, specifically engineered to span the entire range of potential context effects. Thus, unlike the other datasets, we have multiple recorded choices per participant. This allows us to make inferences per individual, rather than at the population level, and evaluate how well our choice models fit the preferences and context effects exhibited by individuals.In this experiment, we validate our CRCS model against the original implementation of Howes et al. [19] on a risky choice task. In this task, a user is presented with a set of three options, each of which is a pair (p i , v i ) consisting of a probability p i and a payoff v i . Upon selecting option i, the user receives payoff v i with probability p i , meaning that each option has expected payofff (p i , v i ) = p i • v i .Comparing expected option values predicted by u with the Monte Carlo estimates used in [19] , we find that on sets of three options both generally agreed on the relative magnitude of the utilities, and that they agreed on the ranking of the utilities in 92.277% ± 0.165% (Agresti-Coull) of cases. Next, we verified q's ability to predict contextual preference reversals. This was tested on Range-Frequency decoy conditions [19] where two "Pareto-optimal" options with equal utility are presented along with a decoy option with slightly lower utility which is dominated by one of the other two options. Preference reversals -specifically, increased likelihood of choosing the Pareto-optimal option that dominates the decoy -have been observed in humans and are predicted by the original model. Figure 3 in the appendix shows that q reproduces the range of reversal rates of the original model. In this set of experiments, we evaluate each models' ability to generalize to unseen data. We compare our proposed CRCS model and the LC-CRCS variant against three baselines: vanilla Bradley-Terry, the variant proposed by [37] (referred to as Bower & Balzano) and LCL [20] . On four different datasets, we infer the parameters for each model on a training set of observed choices {(x (l) , y (l) )} m l=1 and calculate the negative log-likelihood (NLL) of a held-out test set under the inferred parameters. Inference was done using gradient descent on the NLL of the training set. We performed cross-validation, and report the sum of the test sets' NLLs across the folds. For Hotels, Car-Alt and District-Smart we split the choice data across 50, 20 and 10 folds respectively. By evaluating each choice model on each test fold, we obtained paired observations (one per condition) for each test fold, allowing us to perform a Wilcoxon rank test across the folds to test significance. On Dumbalska, we look at how well the choice models can fit to individuals, and thus perform cross-validation for each participant individually. We then treated the sum of the NLLs of the test sets per participants as individual measures, and tested significance using a Wilcoxon test across the participants. Following prior work, we used a linear utility function in all choice models in all datasets.Table 1 shows the total NLL achieved by each model on each dataset. We observe that our proposed LC-CRCS model achieves the highest NLL on Hotels, District-Smart and Dumbalska. This difference is significant (p < 0.01) in all three cases. On Car-Alt, we see that LCL performs better than all other models, with the difference being significant (p < 0.01) for all except the CRCS model (p > 0.2). We theorize that the poor performance of the CRCS model on Car-Alt is due to insufficient option data to train u on (see Appendix A.1), leading to poor estimates of expected utility and therefore poor choice predictions.As part of the District-Smart human subject study, Kaufman et al. [38] collected rankings on six sets of districts from small groups of participants. Ranking such large sets is quite difficult, and we should expect these rankings to be quite noisy. However, like the binary choices that were collected, these rankings are indicative of people's true preferences, and thus should be consistent with any ranking of the same districts implied by the utility function we infer from the binary choices. To test this, we use our choice models to infer utility parameters on the entire set of binary choices. For each of the six sets, we then measure -using Kendall's τ [41] -how consistent the ranking implied by the inferred utility parameters is with the ranking collected in the study. We report the average consistency across all six sets. Because the loglikelihood of CRCS and LC-CRCS is not convex, we repeat this procedure 25 times, starting from different points, to control for the effect local optima may have on the inferences. We test significance using a Wilcoxon test across the six sets of rankings.Unlike the previous experiment, during inference we regularized the choice model parameters of LCL, CRCS and LC-CRCS. This was essential to infer utility parameters that were consistent with the collected rankings. For LCL we used the L1 matrix norm of the weight adaptation mechanism's parameter matrix as a regularization term. The L1 norm enforces sparsity and thus encourages LCL to only fit to the most significant context effects [20] . For CRCS and LC-CRCS we used the probability of the choice model parameters under a chosen prior as the regularization term. Using our understanding of the model this allowed us to encode specific prior knowledge into the regularization. More details can be found in Appendix A.3. From the results in Table 2 we observe that both CRCS and LC-CRCS infer utility parameters that are significantly (p < 0.001) more consistent with the collected rankings than the baselines. LC-CRCS performs slightly worse than CRCS, though the difference was not yet significant.We now evaluate how well the choice models perform in a preference learning setting, where we actively select the queries we put to a user. Whereas in the previous set of experiments we evaluated how well the choice models perform on large amounts of data, here we are interested in how well they perform when minimal data is available. We use the wealth of data available per participant (between 530 and 1060 responses) in the Dumbalska data set to run a user experiment in silico. For each choice model, we use active learning to infer utility function and choice model parameters for each participant individually. At each time step of the experiment, we select from the set of queries recorded for the participant the most informative query using the expected information gain. The participant's response to this query is then revealed, and the posterior over the utility and choice model parameters is updated. We use a particle filter to maintain the posterior beliefs. This elicitation process is performed for 25 time steps on 75 participants. To evaluate the inferences made by the choice models, we calculate the expected likelihood of the remaining queries where the choice has not been revealed yet. As the training data is actively selected for each choice model independently, the data on which they are evaluated -the remaining queries -will differ, meaning that we cannot use a paired test as we have done so far. Instead, we test significance using an independent t-test.Figure 1a shows the mean expected utility calculated over the participants as a function of time for each choice model. We observe that the two variants of our CRCS model make significantly (p < 0.01) better predictions of the participants' choices at all time steps (except 0). Where in the previous experiment (Table 1 ) we saw that LCL was close in predictive power to our proposed models, we see that in this low data setting the difference is much more pronounced. This is because a number of the context effects observed in the Dumbalska data set are built into the CRCS model. While LCL has shown it can learn some of these effects, it needs much more data to do so. Interestingly, we also observe that the LC-CRCS model, which can learn some new context effects on top of the ones built into q, shows significant (p < 0.05) improvement on the CRCS model itself, even when very little data is available. This shows that it provides us with the best of both worlds, showing quick adaptability in low data settings and good performance when more data is available.We now test the feasibility of using our model to learn a utility function from simulated choice behaviors in real-world tasks, and to use the inferred utility function to help a designer solve a task by recommending design solutions to them. We consider a preference learning setting where we learn utility and choice model parameters by iteratively eliciting a simulated designer's preferences over sets of candidate designs, chosen to maximize the expected information gain. Using q, we infer a posterior over the unknown parameters from the observed choices. To simulate a variety of users, we run this experiment in silico, using q with utility parameters sampled from a non-informative prior and choice model parameters sampled from a prior designed to capture a wide range of behaviors exhibited by the CRCS model. At each time step we measure two things: our ability to recover the unknown parameters from the observed choices, and the utility of the design recommendations we make. The inference error is measured by the distance to the true parameters under our current posterior beliefs. The second is measured using the recommendation regret: the difference in utility between the designer's optimal design and the recommended design, which is chosen to be the design with the highest expected utility under the posterior.The first use case involves the design of the frontal crash structure of a car to optimize three separate objectives g 1 (t), g 2 (t), g 3 (t). The design is parameterized by five parameters t = ⟨t 1 , . . . t 5 ⟩ which determine the thicknesses of various metal elements. We define the utility function as the Chebyshev scalarization of the original objectives: f w (t) = max i∈{1,2,3} w i |g i (t) -z * i | where z * i denotes the ideal value of g i () and the weights w sum to one. Different choices of the utility weights w correspond to different trade-offs between the objectives, and therefore to different solutions on the Pareto frontier. Figure 1b shows the average recommendation regret as a function of the number of queries across 300 runs of this experiment. We observe that the recommendation regret reduces quickly, yielding good recommendations after as few as 10 queries. We attribute this to the utility inference error, shown in Figure 6a Our second use case is a water drainage network design problem [42] . This use case is another multiobjective problem involving six objectives g 1 (z), . . . , g 6 (z). Here, we scalarize the problem using a weighted sum f w (z) = 6 i=1 w i g i (z) where the weights w sum to one. As before, different choices of w correspond to different solutions on the Pareto frontier. We ran 300 runs of this experiment.Here too we see that recommendation regret (Figure 1c ) and utility inference error (Figure 6a in the appendix) drop quickly as we put more queries to the designer, with the largest reduction within the first 10 queries. We achieve high-quality recommendations after less than 10 queries.Retrosynthesis planning, the problem of finding feasible reaction pathways to synthesize target molecules, is a central task of synthetic chemistry. Significant progress has been made in solving it through end-to-end automatic synthesis planning [43] [44] [45] . Existing work has focused on expanding the search space of feasible reaction plans. Each route may satisfy an additional subset of properties, and different individuals or organizations may have different preferences over the properties. Chemists' preferences over these plans are often highly complex, representing trade-offs between multiple objectives informed by personal experience and company policy. However, learning their preferences in a way that can then inform AI-driven synthesis planning has not yet been done. We designed a chemist-in-the-loop retrosynthesis planning framework to generate routes with an inferred user model.To build a personalized retrosynthesis planner, we modified one of the state-of-the-art automatic retrosynthesis platforms Aizynthfinder [44] , built on Monte-Carlo Tree Search (MCTS) with a fixed utility function. Details can be found in Appendix B.2.1. First, we proposed a new utility function as the weighted combination of five feature properties g 1 (r), . . . , g 5 (r), that correspond to reactants cost, intermediates stability, reaction feasibility, total reaction success rate, poor reaction success rate, and a route score computed by a data-driven scoring model g 5 . Given the input routes, this model predicts the distance between the current route and the (latent) optimal route. We trained this model on 47,055 synthetic routes extracted from the Journal of Medicinal Chemistry.We report the inference error during preference learning in Appendix B.2.2. We integrated the inferred utility weights into our planning system and assessed the consistency of the generated route with the ground truth user utility preferences. We used inferred weights to synthesize 100 target molecules for each weight. In order to measure the recommendation quality, we evaluated the top-ranked routes from both Aizynthfinder and our model under the true utilities. Specifically, we measured the maximum true utility score among the list of top k recommendations. This is to show how far down from the recommended options list the user needs to go to find their optimal choice. Figure 7c shows that within the top k options, the reaction pathways recommended from our model reaches higher maximum utility score compared to the ones generated by Aizynthfinder. As a significance test, we use the Wilcoxon rank test across every molecule and every user utility with p < (1.61 × 10 -53 ) for all k. 
conclusion: 
 A tractable surrogate model of choice, called CRCS, inspired by theories of human decision-making, was proposed and shown to be a better basis for preference learning than some, but not all, existing models. In response, we modified the model so that it could make cross-feature observations of feature values extending the opportunity for contextual decision-making. We verified against human data from a range of tasks that the new model, called LC-CRCS, outperforms the tested models both in terms of its ability to predict choices and in its inferences of the utility function that underlies the observed choices. Moreover, we find that it corresponds well to previously reported experimental data demonstrating human susceptibility to contextual choice effects. Feasibility of using the new model for preference learning and its ability to recover parameters was also demonstrated in three case studies. Together, the results demonstrate the viability of CRCS and LC-CRCS in high performance preference learning systems.Limitations and future work We identify two primary limitations. First, training CRCS requires sufficiently many choice sets, or a sufficient well-specified task so that new sets can be generated. As we saw with Car-Alt, when insufficient choice sets are available for training, performance can suffer. Second, CRCS and CRCS-LC only work on choice sets of fixed size. Extending these surrogates to variable size choice sets is a promising direction for future work. Another promising direction for future work is the application of the current choice model to large language model (LLM) fine-tuning. Currently, given some featurization of LLM responses to a prompt, CRCS could be directly applied. However, this would ignore the reading and interpreting of these responses that human evaluators have to do. As such, we see an extension of the current choice model that integrates these cognitive processes, based on the same computational rationality theory, as potentially transformational future work. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: All hyperparameters such as the number of experimental runs, number of data splits and priors are listed in the paper. These hyperparameters are also contained in the code which is available online.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: The paper reports error bars suitably and correctly defined, providing appropriate information about the statistical significance of the experiments. We ensured that all reported results include measures of variability, such as standard deviation or confidence intervals, to accurately represent the reliability and significance of our findings.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We discuss the broader impact of the paper in the conclusion. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risk.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: For the data we use we cite the papers which originally introduced those datasets. We also provide direct links to the data and license information in the appendices.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: This paper introduces two new models, implementations for which are available online.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Though the paper uses a number of datasets collected from human subject experiments, it did not collect any new data itself.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? 
 
 Appendix: Learning. The potential societal impact is in line with the broad body of prior work on learning from preferences and modeling humans, none of which we feel must be specifically highlighted here. calc on Range-Frequency conditions for q ("surrogate") and for the original implementation of Howes et al. [19] ("surrogate"). For q, we show the mean ± std. dev. for 10 models trained with different seeds. The "reversal rate" is measured by calculating the rate at which the Pareto-optimal decoy-dominating option is chosen. To control for random variation we subtract from this the "inverse reversal rate", the rate at which the other Pareto-optimal option is chosen. For non-zero values of σ 2 calc , we see that though q is less sensitive to σ 2 calc , it reproduces the range of reversal rates of the original model.This section provides details on how the CRCS model was trained for the choice tasks corresponding to the Hotels, District-Smart, Car-Alt and Dumbalska datasets. In order to train our CRCS model on a new choice task, we need to define three priors: a prior over sets of options p(x), a prior over utility function weights p(w), and a prior over choice model parameters p(θ).The prior over sets of options p(x) is by far the most important prior for successfully training the CRCS model. It is clearly important that this prior matches the distribution of choice sets we expect to see for the choice task we target. However, it is even more important to ensure that that prior has proper support across the entire space of option sets. From equation 2 we see that in order to predict the true utilities of the options x, u essentially has to infer the option set x (which it does not observe) from the observations u and o. It can only learn to do this well it if during training we can expect it to encounter all x that could have resulted in u and o.The priors p(x) for these tasks were defined as follows:• For Hotels we had access to the set of 200 hotels the original authors had used to build their study. Thus, we generated options triplets from the prior by uniformly sampling (without replacement) three hotels from this set. • For District-Smart we had access to 21778 electoral districts collected by the original authors. We generated pairs of options by sampling them uniformly from this set without replacement. • Options in the Dumbalska task have two features, a property's rental cost and the participant's valuation of it, both of which were bounded between 0 and 2500. The human choice data suggested that both are strongly correlated. We created a prior over individual choices that reproduces this correlation by using a multivariate normal distribution, mixed with a uniform distribution over the entire option space, to ensure sufficient support even on less frequently encountered options. Figure 4 shows the empirical option distribution within the choice data, and our engineered prior over individual options. To generate option sets, we then sampled three options from this prior. • Car-Alt considers options sets consisting of hypothetical cars. Although we know that the options were created from a set of 120 cars, the features of each option are determined both by the Car that corresponds to that option and the participant who makes the choice. For example, the cost of each car is expressed as a multiple of the participant's log income.The inclusion of participant-dependent features creates correlation between the options. Unfortunately, we did not have enough information to engineer a new prior p(x) that would faithfully reproduce this correlation, and would faithfully match the empirical distributions over sets of options encountered in the original user study. Thus, we were forced to train the CRCS model on the limited number of choice sets that appear in the human choice data.As all tasks used linear utilities, and as the CRCS model is invariant to scaling of the utility function, we define p(w) in all cases as a uniform distribution over all vectors of length 1.The prior over the choice model parameters was set based on choice model parameters reported for risky choice in [19] after accounting for scale differences of utilities and feature values for each task2 They are listed in table 4. For LC-CRCS, we additionally placed independent standard normal priors on all entries of A. U (0, 1)District-Smart N ∞ 0 (0.0, 2.0) Beta(1, 3)hull N ∞ 0 (0.06, 0.12) bbox: N ∞ 0 (0.08, 0.16) reock: N ∞ 0 (0.05, 0.1) polsby: N ∞ 0 (0.08, 0.16) sym_x: N ∞ 0 (0.14, 0.28) sym_y:N ∞ 0 (0.1, 0.2) Car-Alt N ∞ 0 (0.0, 2.0) Beta(1, 3)Price divided by ln(income): Bradley-Terry 0.890 ± 8×10 -6 0.632 ± 1×10 -5 1.575 ± 3×10 -5 0.567 ± 3×10 -6 Bower & Balzano 0.890 ± 2×10 -5 0.631 ± 4×10 -6 1.575 ± 5×10 -5 0.567 ± 2×10 -6 LCL 0.831 ± 2×10 -5 0.616 ± 4×10 -6 1.560 ± 5×10 -5 0.562 ± 2×10 -6 CRCS (ours) 0.855 ± 0.014 0.628 ± 0.009 1.620 ± 0.032 0.568 ± 0.007 LC-CRCS (ours) 0.902 ± 0.063 0.616 ± 0.006 1.590 ± 0.013 0.572 ± 0.007N ∞ 0 (0.7, 1.4) Range: N ∞ 0 (45, 90) Acceleration: N ∞ 0 (1, 2) Top speed: N ∞ 0 (8.The experiments on static data were run using a cross-validation strategy. For each fold, we inferred utility parameters and choice model parameters jointly for each choice model by performing vanilla gradient descent on the train set log-likelihood. For the CRCS and LC-CRCS model the starting points were sampled from the priors defined in section A.1. For the Bradley-Terry variants the utility parameters were sampled independently from a standard Gaussian, and for LCL the learnable parameter matrix A was populated using sampled from N (0, 0.1).CRCS and LC-CRCS have non-convex likelihood functions, meaning that gradient descent is liable to get stuck in local optima. To mitigate this, we performed inference multiple times (up to 50), starting from multiple starting points, and chose the parameters that achieved the best log-likelihood on a held-out part of the training data. Table 5 shows the variance in average NLL achieved by individual inference runs on the various choice problems. We can see that there is significantly higher standard deviation when doing inference with CRCS and LC-CRCS, pointing to the existence of local minima, and confirming the necessity of repeated inference to address this.For District-Smart, we used gradient descent to infer the utility and choice model parameters for each choice model on the entire set of binary choices. For LCL, CRCS, and LC-CRCS we noticed that the inferred utility functions were highly inconsistent with the rankings that had been collected in the same user study. As explained in the main paper, we used regularization to address this. We will go into some more detail on why we think regularization was needed and how we tuned the regularizers we used. Table 6 below shows the consistency of the ranking implied by the inferred weights for each of these three choice models with the collected rankings. We can see immediately that compared Table 6 : Consistency between collected rankings and rankings implied by inferred weights with and without regularization of choice model parameter for LCL, CRCS and LC-CRCS.Without regularization -0.36 -0.17 -0.04 With regularization 0.287 0.622 0.525 to Bradley-Terry and the Bower & Balzano models, the consistency is quite poor, especially for LCL. We hypothesize that this is caused by heterogeneity in the task utilities different participants used in making their choices. The intention of Kaufman et al. [38] was to capture humans' intuitive understanding of what it means for an electoral district to be compact. Therefore, in the user study, people were encouraged to make choices "according to your own best judgement" [38] . As it is likely that the various participants in the study had slightly different intuitions about compactness, we can therefore expect that the recorded choices have been made with slightly different utility weights w.To fit a choice model using a single utility to these choices could thus prove problematic. For future work, it would be interesting to consider a hierarchical approach, where we would model the fact that any choice has been made according to an unobserved utility function drawn from some unknown distribution.For the current work, we resorted to using regularization to ensure the choice models did not overfit to the noise in the utility function. These regularization strategies were tuned using one of the six rankings collected, while for evaluating the consistency of the inferred weights with the collected rankings we used all six. For LCL, we regularized the weight update mechanism w + Ax C by using the norm of A, multiplied by 75 to get the desired regularization strength, as a regularization term.For the CRCS model, to ensure that we make sensible inferences, we had to ensure that the noise in the utility function would be explained by the right source of noise in the model. The prior we placed on the choice model parameters was designed to do just this. We used placed a Beta(1, 1000) to ensure that ε, which determines the level of noise on the attribute comparisons, would stay close to 0. Attribute comparisons are the primary source of context effects and are necessary to fit to any such effects in the data. Additionally, we note that in the experiment conducted in [19] , ε was fixed to 0. We then placed a N (25.0, 0.1) prior on σ calc , the noise on the utility observations, to help explain the heterogeneity of utility functions itself. We also placed a weak prior on τ 1 , . . . , τ d , namely the prior we also used when training the CRCS model. The same prior was used for LC-CRCS.The elicitation experiment on the Dumbalska dataset was performed for each participant in the original experiment individually. We excluded participants according to the same rule the original paper had used. For each run, we would select a participant from the dataset and treat the queries to which responses had been collected for this participant as the queries we could put to the user. In each time step, we used the posterior at that point to estimate the expected information gain of each query that had not been used yet, and selected the query with the highest information gain. The recorded response to this query would then be revealed, and the posterior would be updated with this new observation using the choice model. To maintain the posterior, we used a particle filter containing up to five million particles representing combinations of utility and choice model parameters. The particle filter was not refreshed during the experiment. At each time step we measured the expected likelihood, where the expectation was taken with regard to the posterior and a uniform distribution over the remaining set of recorded queries (those which had not yet been selected as part of the elicitation process). For completeness, we also measured the expected log-likelihood and the entropy in the marginal utility parameter posterior. Those are shown in Figure 5 .We list below here the sources for the data we use in the human data experiments for Dumbalska [36] , Car-Alt [39] , Hotels [40] and District-Smart [38] . We obtained the human choice data for District-Smart and Car-Alt from the excellent collection of choice data collected by Tomlinson and Benson [20] for their implementation. Here, we will describe additional details on the crash structure design and water drainage network design use cases. Both use cases were run with the same priors for the utility parameters and choice model parameters. The utility parameter prior was a uniform Dirichlet distribution. The choice model parameters for the CRCS model were chosen to capture the widest possible range of choice behaviors.In each step, candidate choice queries were generated by sampling 1000 queries of three design options from a uniform prior over the domain of either use case. The candidate with the highest expected information gain was chosen. For crash structure design we elicited responses toon 50 preference queries in each run of the experiment and for water drainage network design we elicited responses to 100 queries, though for space reasons we only show the first 75 on the graphs in this paper. The recommendations on which we measured the recommendation regret at each step were selected by maximizing the expected utility under the current posterior over a pre-calculated Pareto front for the user case. The user's optimal design was found by optimizing the true utility over this same front.Figure 6 shows the utility and choice model parameter inference errors for both use cases.Here we provide further details about Aizynthfinder [44] . Aizynthfinder 3 is an open-source retrosynthesis planner that uses Monte Carlo Tree Search (MCTS) and a template-based expansion policy 4to search for possible reactions and an additional filter policy that discard the infeasible reactions. The expansion policy is a multi-class classification model that predicts the most probable reaction templates. In practice, this model produces the top 50 possible templates as the possible action during the tree expansion process. Then, the infeasible reactions are filtered out with the filter policy. During the expansion and selection phase of MCTS, it uses the upper confidence bound (UCB) to select and score routes as defined:EQUATION)where n is the visitation times of a node, C is the bias hyperparameter set to 1.4, and Q is the accumulated reward that is defined as: Justification: We test our proposed models on a varied set of data sets and use cases.EQUATIONGuidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).Implementation available at https://github.com/AaltoPML/Preference-Learning-with-ahuman-like-model-of-choice.N b a denotes a normal distribution truncated to the range (a, b). Note that in order to be consistent with other baselines, within the CRCS model utilities are calculated on z-normalized features, while ordinal observations are calculated on non-normalized features.code available: http://www.github.com/MolecularAI/aizynthfinderdownload available: https://doi.org/10.6084/m9.figshare.12334577.v1 


Decision: The paper proposes modeling of choice behavior by using models of human choice from cognitive science as priors linked to data by neural network models. This allows fitting a broader set of possible data than is possible under intractable cognitive science models, and fitting to data that are not well explained by such models. Detailed experimental simulations demonstrate the efficacy of the solution. 

Opinions of this paper are quite split. I have read the reviews, responses, and the paper itself. I lean toward acceptance. The reviewers had a large number of legitimate questions, which seem to be answered in the responses. Addressing these will require non-trivial work on the authors' part. The remaining question is about the contribution. The authors argue that using cognitive models as a prior is in important contribution, one that is supported by their empirical results. The authors are generally less convinced that the idea is important, given that there are not significant technical advances. I lean toward acceptance because ideas that supported by data can be important contributions in general, and in this specific case, improving modeling human choice behavior at scale is an important problem.

Reviews: review: paper_summary: paper_summary: The submission proposes an approach to preference learning using a model inspired by findings in cognitive science. Specifically, it uses an amortized inference variant of a previously proposed model to enable tractable inference of preference values, and applies it to a number of case studies, where it is shown to outperform both the classical Bradley-Terry model, and additional more recent baselines.
strength_weakness: strength_weakness: My primary concerns with the paper are related to the actual technical implementation details, which are not articulated clearly enough, and where they are articulated give me some pause. Specifically: 
* If I understand things correctly, the policy network takes in $x, w, \tilde{u} and \tilde{o}$, and a sufficiently flexible surrogate should be able to learn arbitrary functions of x and w already. Is this correct? If so, why is the explicit feature engineering needed? The paper also discusses later "built in" context effects in CRCS and "learned" ones in LR-CRCS -- what is meant by this? 
* This also makes me wonder: what's the network architecture used and other training hyperparameters (optimizer, learning rates, etc)? In general I would expect a sufficiently flexible surrogate to also achieve better than ~92% agreement with the original model. I'm also puzzled about the claim of insufficient option data to claim the utility network -- shouldn't it be possible to simulate arbitrary amounts of observations in this case such that the network fully spans the range of possible outcomes and enabling amortized inference? I can see why sampling only from the training data would create this restriction, but I don't understand why this choice was made.
questions: questions: * Reporting summed rather than averaged NLLs (with standard errors) seems like an odd choice -- why was this done? 
* Can the authors comment about the choice of Wilcoxon signed-rank test for some hypothesis tests and t-tests for others? 
* Why are baselines not present in the regret panels of figure 1?

Additional Notes:
* Figure 1 tick labels should be larger. 
* L53 maybe "computational rationality analysis"? 
* L66 it's -> its
* The notation in section 3.3. is a bit confusing, in the following way: u is a function of x, but the notation sometimes uses u and sometimes x (e.g. in the expectations in the utility and policy loss, or how \tilde{u} is conditioned on u but \tilde{o} is conditioned on x. I think this might be because of a distinction between the observer's and user's likelihoods, but this isn't quite spelled out here. 
* I believe that the overall setup here is a bilevel optimization problem (because L_util is optimized w.r.t. w and L_pol w.r.t. x), and I'm not sure how incomplete optimization or other issues in the utility network impact the policy network (especially considering that the accuracy of the utility network seems not ideal). 
* Line 278 I believe \citet should be \citep there?
limitations: limitations: Discussion is adequate, though as noted above I'm puzzled about the limitation on requiring sufficiently many choice sets for estimation.
paper_summary: paper_summary: In this paper, the authors present a new model which approximates an existing intractable Bayesian model for preference learning. The paper describes a generative model for choice, and shows how inference can be approximated using two different neural networks. The proposed model is then augmented using a cross-feature correction inspired by another piece of existing work. Results show that the proposed model outperforms the baselines on sensible metrics like negative log likelihood of the test set.

**Conclusion:** Overall, the paper makes a neat proposal and is fairly meticulous in developing the central idea. However, the contribution doesn’t appear substantial and might be of interest to a smaller subset of the community.
strength_weakness: strength_weakness: 1. While it was easy to follow the theory, I found it a little difficult to deeply understand the application section. A little more focus on methodology on at least one experiment might have been more helpful. Specifically, it was a little tricky to cleanly connect the theory to the application for me.
2. The contribution is somewhat incremental, and might be better suited for area-specific workshop. Since the Bayesian model already exists, the main contribution is estimating the two intractable quantities using NNs, if I understand correctly. It’s useful to have such models, but I am not sure if they generate any foundational insights.
3. For synthetic data, if the generative model is closer to the inference model, it’s not very surprising that the latter does better than another model unaware of the generative process. This weakens the evidence on synthetic data a little.
questions: questions: NA
limitations: limitations: NA
paper_summary: paper_summary: The paper introduces two models for learning preferences from human choice behaviors: the Computationally Rational Choice Surrogate (CRCS) and a variant (LC-CRCS) that considers the context effect of the choice set. Both models are based on a state-of-the-art cognitive model of human decision-making. The paper replaces intractable computations in the original model with surrogates to enable feasible estimations and demonstrates the numerical performance of these models through several experiments.
strength_weakness: strength_weakness: 1. Limited Applicability: The models require data that includes the orderings of each feature (or their generation processes) and the true utilities (or their generation processes). Such data is not commonly available. Additionally, prior knowledge of several parameters must be estimated before training the models. For instance, each feature needs a distribution for the corresponding $\tau$, which becomes challenging when the number of features is large.
   
2.Experimental Focus: The experimental results primarily focus on negative log-likelihoods (NLLs). Although the proposed methods outperform benchmarks in terms of NLLs, it is unclear if this improvement is practically significant. Comparing other metrics (e.g., accuracy) would better demonstrate the contribution of this work. Moreover, the paper does not provide standard deviations for the results to indicate robustness, even though the results are statistically significant.

3. Training Robustness: Line 561 mentions that "As CRCS and LC-CRCS have non-convex likelihoods, we performed gradient descent multiple times, starting from multiple starting points, and chose the parameters that achieved the best train set log-likelihood." It would be helpful to show the mean and standard deviations over several random seeds instead of just the best result to demonstrate training robustness. Training on large datasets can be time-consuming, making multiple training runs impractical.
questions: questions: 1. How can the true utility or its generation process be observed or estimated in real-world scenarios?
2. How should the prior distributions in Table 3 be selected or estimated? Are there general methods for estimating these prior distributions from a given dataset?
3. How do the models perform on other metrics (e.g., accuracy)?
4. What are the architectures of neural networks $\hat{q}$ and $\hat{u}$?
limitations: limitations: See weaknesses.
paper_summary: paper_summary: This paper uses a state-of-the art-models model from psychology for preference learning. To make learning tractable, the authors use variational inference to approximate unknown distributions (and expectations). The model is shown to outperform existing approaches, suggesting that it better predicts human preferences and captures contextual choice effects that are commonly observed in human behaviour.
strength_weakness: strength_weakness: No major weaknesses. Some minor suggestions:

- Line 56 and the whole paragraph after: the meaning of CRCS acronym is not introduced until line 125.

- line 153: when reading this paragraph for the first time, I found it strange that the parameters $w$ are observed (known). It then became clear that they are observed only by the *user* that is being modelled. 

- a graphical model describing all the variables in the model can be useful; it can help visualise the differences between the modelled user and the outside observer (which variables are observed or unobserved). Clearly indicating what is given and what is learnt - e.g. the prior $p(w)$ and the marginal $p(x)$ are both assumed given?

- line 221: typo though -> thought
questions: questions: 1. What is the number of parameters for each of the models? It will be good to have these reported in Table 1.
2. I might have missed it, but I didn't see the network architectures for $\hat{u}$ and $\hat{q}$?
3. Have you thought applying the model to LLMs for preference learning?
limitations: limitations: Some of the limitations of this work are discussed in Section 5. Societal impact statement is also included in Section 5.

Comments: title: Update after authors' rebuttal
Still confused about CRCS vs LC-CRCS
Please respond to the authors
Please respond to the authors
Please respond to reviewers
Please respond to the authors
comment: Thank you for your response and for clarifying your remaining concerns.

> Ablation study is important. Even the choices of priors are not important, it should be validated by experiments and to see if there are some choices can fit the tasks better and improve the performances.

Yes, we naturally agree ablation studies are important when they add new insights to the paper. Our concern is that an ablation on the priors used to train the surrogate would provide a foregone conclusion. If the priors are too narrow and the surrogate is used when conditioned on parameter values it has not been trained on, the performance will be worse. In this case the prior should have been revised to be more broad. If the priors are broad, the surrogate takes longer to train.

> Besides the priors, there are still other hyperparameters such as the number of layers in the two networks. (So I don't think "Note that the only hyperparameters that need to be chosen are the prior distributions from which samples are drawn when training the surrogate.") I also do think they are important. For example, the number of layers and embedding dimensions may control the expressiveness of these two networks and thus influence the performance.

You are right about the network hyperparameters. We described their values in the rebuttal PDF. These were set automatically by performing a grid search over a range of possibilities. The set-up for the grid search was the same across the choice tasks. We will describe how this grid search was done in the appendix, and will provide a sensitivity analysis on the effect of the various choices of network hyperparameters on performance.

Thank you again for helping improve our submission. We very much appreciated the time and effort. We believe we have been able to clarify all the concerns and would appreciate if you would take that into account in the score.
Thank you for asking, and for the comments.

> I do not quite understand the assertion, `We study how to mitigate the effect context effects - a human bias widely studied in cognitive science - have on utility inference from preferences.` My understanding is context effects are widely observed and the proposed model is learning underlying utilities that can explain their emergence.

Your understanding is correct. Context effects are widely observed and the underlying utilities are a fundamental factor in their emergence. The point we were trying to make is that when inferring these underlying utilities from choices made by humans - where we expect context effects to have affected these choices - it is important that we do so with a model that correctly models context effects. If context effects are not properly modeled, there is a risk that we infer the wrong underlying utility function. The experiments provide evidence for this. The Bradley-Terry model, which does not model any context effects, is in all cases the worst of the choice models considered.

We hope this helps to clarify.
Thank you for the response, it addresses some of my concerns.

However, in my view, 

- Ablation study is important. Even the choices of priors are not important, it should be validated by experiments and to see if there are some choices can fit the tasks better and improve the performances. 
- Besides the priors, there are still other hyperparameters such as the number of layers in the two networks. (So I don't think "Note that the only hyperparameters that need to be chosen are the prior distributions from which samples are drawn when training the surrogate.") I also do think they are important. For example, the number of layers and embedding dimensions may control the expressiveness of these two networks and thus influence the performance.
I appreciate the authors taking the time to address my concerns. The clarification around how the theory connects to the experiments is quite helpful. I also concede the point that this could be an apt addition to the main conference given the recent trend of publications the authors pointed out.

However, I do not quite understand the assertion, `We study how to mitigate the effect context effects - a human bias widely studied in cognitive science - have on utility inference from preferences`. My understanding is context effects are widely observed and the proposed model is learning underlying utilities that can explain their emergence. This makes me wonder if I missed something fundamental in the paper. I am raising my score from 4 to 5, but lowering my confidence from 4 to 3.
Thank you for your explanation, and our apologies if we had been unclear.

> My primary concern is still the applicability in practice: there are too many hyperparameters to decide in the proposed approach which can influence its performance, such as the priors.

Note that the only hyperparameters that need to be chosen are the prior distributions from which samples are drawn when training the surrogate. Given that the surrogate is a highly flexible neural network, **what mainly matters here is that the priors cover all cases** which would be encountered when the surrogate is used in practice. The primary reason that there is variation in the priors we chose is that **we automatically scaled them according to the magnitude of the different features**. We could have used the same broad prior throughout but would have paid a penalty in compute time while the results of our experiments would have been identical. We believe therefore that **the priors have no negative implications for the applicability in practice** of the reported approach.

To be more specific, the choice of priors is easily achieved as long as two constraints are respected: (1) priors should not be overly narrow – because this may lead to poor performance, (2) priors should not be overly broad – because this may lead to inefficient learning of the surrogate. Within these constraints there is a wide range of priors that will produce virtually identical results to those reported in the paper. The effect of the choice of priors on the performance is easy to see. Recall that our surrogate is trained to reproduce choices predicted by an existing model. We train this surrogate to be able to make these predictions conditional on a range of values of the free parameters in the model - these are the choice model and utility parameters. During training we randomly sample parameter values form the priors and then train the surrogate against the model (both conditioned on these parameters; i.e., like sampling tasks in multi-task learning). In testing, we used this surrogate to infer the values of these parameters from human data, and to make new predictions. The concern when the priors are too narrow is that we may end up in a situation where we have to make predictions with the surrogate conditioned on parameter values on which it has not been trained. This would likely result in erroneous predictions. The concern with broadening the priors is that more training is needed to cover the greater variation in parameter values to condition on. This trade-off between computational cost and accuracy across the parameter range is subjective, but was made based on prior knowledge available from prior work (as explained in our original rebuttal and in the appendix).

> why not always use the family of truncated normal distributions with a large range (instead of using a uniform distribution in the Hotel data or varied ranges in different datasets)?

This is essentially what we did. For Hotels, which is the first experiment we performed, we used a uniform distribution with wide support. For the other choice tasks we switched to even broader priors. There, we used the same truncated normal priors (through re-scaled due to differences in feature magnitudes) and the same Beta prior for $p(\varepsilon)$ for all tasks. We did not repeat the Hotels experiment with this choice of priors because there would have been no appreciable change in performance.

**We will expand the existing discussion on prior selection in section A.1 of the appendix** to include this discussion on the factors that go into prior selection and will explain why the chosen priors differed between choice tasks.
Thank you for your comment. My primary concern is still the applicability in practice: there are too many hyperparameters to decide in the proposed approach which can influence its performance, such as the priors. In your response to the choices of the priors, you mentioned, "The priors for the choice model and utility parameters have very little effect on the model itself. However, as they are used when training the model, it is important that they cover any parameter values that we may wish to use in practice." If the only factor that matters is that "they cover any parameter values that we may wish to use in practice" and "The priors for the choice model and utility parameters have very little effect on the model itself", why not always use the family of truncated normal distributions with a large range (instead of using a uniform distribution in the Hotel data or varied ranges in different datasets)? I would expect more discussions and potentially with an ablation study on the priors (and other hyperparameters), which could help demonstrate your claims and provide guidance on how to apply this approach in practice.
Thank you for your comment. We apologize for misunderstanding your original question regarding LC-CRCS.

Fundamentally, CRCS as we defined it does not learn context effects from (human) data. Instead, context effects emerge given an empirically validated theory (choices maximise expected utility given noisy observations; section 3.1). This theory can be simulated and thus be amortised into \hat{q}. $\hat{q}$ can therefore be trained without any access to human data. There are a number of free variables in the model (the choice model and utility parameters) which are inferred from human data in our experiments, but these determine the utility function and various noise levels within the choice process, and are thus not directly responsible for the context effects. Although not learning context effects from human data has clear advantages, it also means that the model will not generate context effects not explained by the theory. This is why we introduced the additional linear mapping $g(w,x)$ in LC-CRCS. This can be learnt from human data and can therefore fit to context effects not yet explained by the theory or generated by CRCS.
I appreciate the authors' clarifications regarding the training setup. I'm still surprised that the agreement to the brute-force approach is so low, but I will also grant the point that maybe it's not that important to match to the brute force approach considering performance of the proposed model on the actual evaluation tasks is still improved. 

Regarding feature engineering, my concern is not $\tilde{u}$ and $\tilde{o}$. Rather, I mean the $g(w, x)$ learned linear mapping -- why is this not already learnable within the original CRCS? As far as I can tell, $g$ simply takes the average of the $x$'s, and transforms it linearly, both of which the $\hat{q}$ network can do already.
Thank you for your response. We thought that we had fully answered your questions. To help us understand your decision, could you please point us to which of our responses failed to address your original comments? Most importantly, in answer to your first point regarding limited applicability, our model does not require observed feature orderings or true utilities. In answer to your second point, choice predictions are probabilistic requiring NLLs, not accuracy, to fully assess.
Thank the authors for the response. I will maintain my original score.
Hello reviewer cVZM: The authors have responded to your comments. I would expect you to respond in kind.
Hello reviewer hXMa: The authors have responded to your comments. I would expect you to respond in kind.
Hello reviewer rsF2: The authors have responded to your comments. I would expect you to respond in kind.
Hello reviewer CwtL: The authors have responded to your comments. I would expect you to respond in kind.