Title: Preference Learning of Latent Decision Utilities with a Human-like Model of Preferential Choice
Venue: NeurIPS 2024
Accepted: yes
Keywords: preference learning

human-in-the-loop

AI-assistance for decision making

user modeling

cogntitive science

retrosynthesis planning
Abstract: Preference learning methods make use of models of human choice in order to infer the latent utilities that underlie human behavior. However, accurate modeling of human choice behavior is challenging due to a range of context effects that arise from how humans contrast and evaluate options. Cognitive science has proposed several models that capture these intricacies but, due to their intractable nature, work on preference learning has, in practice, had to rely on tractable but simplified variants of the well-known Bradley-Terry model. In this paper, we take one state-of-the-art intractable cognitive model and propose a tractable surrogate that is suitable for deployment in preference learning.  We then introduce a mechanism for fitting the surrogate to human data and extend it to account for data that cannot be explained by the original cognitive model. We demonstrate on large-scale human data that this model produces significantly better inferences on static and actively elicited data than existing Bradley-Terry variants. We further show in simulation that when using this model for preference learning, we can significantly improve utility in a range of real-world tasks.

Preference Learning of Latent Decision Utilities with a Human-like Model of Preferential Choice

Abstract

Preference learning methods make use of models of human choice in order to infer the latent utilities that underlie human behavior. However, accurate modeling of human choice behavior is challenging due to a range of context effects that arise from how humans contrast and evaluate options. Cognitive science has proposed several models that capture these intricacies but, due to their intractable nature, work on preference learning has, in practice, had to rely on tractable but simplified variants of the well-known Bradley-Terry model. In this paper, we take one stateof-the-art intractable cognitive model and propose a tractable surrogate that is suitable for deployment in preference learning. We then introduce a mechanism for fitting the surrogate to human data and extend it to account for data that cannot be explained by the original cognitive model. We demonstrate on large-scale human data that this model produces significantly better inferences on static and actively elicited data than existing Bradley-Terry variants. We further show in simulation that when using this model for preference learning, we can significantly improve utility in a range of real-world tasks.

1 Introduction

AI systems need exact descriptions of tasks to be performed. However, humans find more complex tasks hard to describe. In response, preference learning has emerged as one way to learn from human feedback. It has been used to teach AI systems a variety of tasks from how to hand objects to humans to how to play Atari games [1-4]. More recently, human feedback has been used to train large language models to summarize text [5], answer questions in natural language [6], and to train deep generative models to generate realistic medical images [7].

When learning from human feedback, it is generally assumed that some latent utility function f guides an individual's behavior, but that the individual cannot describe this function to the machine. Thus, preference queries are used to elicit information about f from the user. A preference query gives a user a set of options x 1 , . . . , x n and asks the user to select their preferred option, i.e., the one with the highest utility. Given a model of how people make such choices, the machine can then infer the underlying function f from the user's chosen item y . For example, Stiennon et al. [5] learned a utility function for text summaries by showing users a text with several summaries and asking them to choose the best summary.

There are several models of choice that have been used for learning preferences from human choices. Some recent work on Reinforcement Learning from Human Feedback (RLHF), for example, has used

a simple binary choice model [5, 6] p ( y = x 1 | x 1 , x 2 ) = σ ( f ( x 1 ) -f ( x 2 )) , over choices x 1 and x 2 , though generally, most preference learning approaches have used the Bradley-Terry model [8]

Although these models have proven to be practical, they are not realistic models of human choice behavior. Specifically, both models make choices between two options without taking into account the rank orderings of option attributes; a widely observed property of human decision-making [9-11]. As a result, these models fail to predict a number of apparent biases in human choice behavior. These include contextual choice effects [12, 13], which occur in situations where a decision maker's choice between two options is influenced by adding more options to the choice set [14, 12]. Say, for example, we have two options A and B and a user exhibits a probability of choosing between these. When a third decoy option C is introduced which is strictly dominated by B , there is a shift in the probability of choices from A to B .

Though context effects are not certain to appear in preference queries posed to users, they are known to appear in a wide range of human tasks including risky choice tasks [12], multi-attribute choice tasks [15] and perceptual judgement tasks [11] and in many other species including jays and honeybees [16]. These effects point to a potential gap in the accuracy of the models currently used, during preference learning, to interpret the choices made by users. Moreover, this gap has the potential to lead to incorrect inferences about the latent preference utilities of observed human decision-makers.

The contribution of this paper is threefold. First, we show that we can improve preference learning by leveraging computational rationality theory, a general cognitive-scientific theory which posits that human behavior is rational under cognitive bounds [17, 18]. We learn preferences from human choice behaviors using a state-of-the-art cognitive model that is based on a computational rational analysis of context-dependent choice under uncertainty and is backed by substantial empirical support in the psychology literature [19]. Like all computationally rational models, behavior under this model emerges from the latent utility function and a latent set of cognitive bounds. This provides strong inductive biases when inferring these latent factors from human behavior which - as we will show experimentally - significantly improves learning from preferences. Our second contribution lies in making this cognitive model amenable to preference learning. To this end, we generalize it, and make inference practical by approximating intractable calculations with a surrogate we call the Computationally Rational Choice Surrogate (CRCS) model. Finally, we find experimentally that CRCS can sometimes perform worse than the Linear Context Logit (LCL) model [20]. We hypothesize that human context effects are partially a consequence of cross-feature effects. These are not modeled in CRCS, but can be learnt by LCL. We therefore propose LC-CRCS, which takes advantage of these effects by combining CRCS with LCL.

We report three sets of experiments. In the first, we show that CRCS matches the original model's prediction of human choice behavior. In the second, we compare preference learning with CRCS to preference learning with recently proposed variants of the Bradley-Terry choice model. Using existing human data sets, we show that CRCS outperforms these in choice prediction and utility function inference, but performs worse than LCL on some tasks. We then show that LC-CRCS can additionally outperform LCL in these tasks. In the third set of experiments, we show the applicability of CRCS in three real-world use cases and verify its parameter recovery capability.

2 Background

2.1 Learning from Preferences

Preference learning methods aim to infer latent utility functions from human choices. Depending on the type of queries presented, there are two main streams of research: (1) learning from pairwise comparisons or (2) learning from ranking, where humans rank a set of n options. Popular methods include Gaussian Process regression that captures the preference relationships of pairwise queries [21, 22]. Other work, such as as [23-25], uses Deep Neural Networks trained on ranked demonstrations to approximate the underlying reward functions. To reduce the computational burden created by the necessity for numerous queries, active learning techniques [26-29] have been proposed for efficient query proposal with maximum information gain. However, these methods typically require consistent preference order within the ranking and do not consider any contextual effects within the query

dataset. Reinforcement Learning approaches include Preference-Based Reinforcement Learning (PBRL) and Reinforcement Learning with Human Feedback (RLHF), where the reward function is inferred from the preference feedback. Work using ranking queries and human feedback can reach or even exceed human-level performance in several RL benchmarks [23, 24].

2.2 Modeling contextual choice

To date, preference learning research has yet to make use of plausible models of human decisionmaking such as [30, 31]. These models are inspired by extensive studies of human behavior and give rise to contextual choice effects . Consider a hypothetical choice between two sightseeing trips, one to Paris and the other to London. Both trips come with free coffee. Let's say that 70% of people prefer Paris and 30% London. Now imagine that we add a third option which is identical to the London trip but without free coffee. If, for this three-trip choice problem, we observe that 40% prefer London with free coffee, then we will have observed a contextual choice effect known as a 'preference reversal' [32]. The choice frequency for London with coffee is increased by a context that includes a dominated choice. This effect has been observed both in sample averages and, more interestingly, within individuals. It has been taken as evidence that people are irrational [33] and have no stable preferences [34]. Needless to say, both instability and irrationality pose severe challenges to the viability of preference learning.

More recent theories, however, demonstrate that contextual choice effects can be consequences of computationally rational processes that assume stable preferences. These theories explain contextual choice effects by modeling the fact that people compare attributes and/or utilities under uncertainty. These include Bayesian theories [35], rational analyses [19], and neurobiological relative encoding theories [13, 36]. These theories use comparisons between option attributes to compute expected utilities, such that these expectations are sensitive to the reliability of the comparison as an indicator of expected utility.

Other work has proposed variations on the Bradley-Terry model to include these contextual effects, with the same commitment to stable preferences. Bower and Balzano [37] posit that context effects are the result of humans comparing options only on the k most salient features within a context. They propose a Bradley-Terry model where utilities are calculated only on the k most salient features, where saliency is measured by the sample variance of each feature within the current set of options. Tomlinson and Benson [20] do not propose a specific theory of context effects, but rather propose to learn them from data. They introduce the Linear Context Logit (LCL) model, a Bradley-Terry model with a linear utility function, in which context effects are modeled as a context-dependent change in the (globally stable) weights of the utility function. This change is modeled as a linear function of the average attribute values of the set of options presented to a user (the context), and is inferred from human choice data. They further introduce the Decomposed LCL model, in which each feature induces its own context effect - whereas in LCL the features jointly induce a single context effect and where the final context effect results from a mixture of these individual effects.

In the current paper, we commit to distinguishing behavioral choices, which are observable, from latent preferences, which are not. When we refer to 'preferences' we are referring to the latent utility function f ( x ) , and not to the observable choice behavior.

3 Modeling computationally rational choice

To learn latent preferences from human choice behavior, we build on a computationally rational model of choice behaviors by Howes et al. [19] which is sensitive to the aforementioned context effects. This model assumes that humans make utility-maximizing choices, but that the option utilities are estimated from noisy observations of the true utilities and noisy comparisons between the option attributes. Here we will first describe the original model in a general form. We then extend it to a general space of utility functions and introduce our Computationally Rational Choice Surrogate (CRCS) model, a model which replaces intractable computations in the original model with learned surrogates to allow tractable inference of the latent utility function. Finally, we introduce the LC-CRCS model, an extension of the CRCS model which is able to learn additional context effects not captured by the CRCS model.

3.1 A computationally rational model of choice

Let x 1 , . . . , x n ∈ R d be a set of n options, each with d attributes. Let f : R d → R be a latent utility function that maps each option to its associated utility. As a shorthand, we will denote the utilities of a collection of options x = ⟨ x 1 , . . . , x n ⟩ as u = ⟨ u 1 , . . . , u n ⟩ where u i = f ( x i ) .

The cognitive model introduced by Howes et al. [19] assumes that when making choices, humans do not observe the options x 1 , . . . , x n nor their utilities directly. Instead, humans are assumed to make utility-maximizing choices based on two sets of noisy observations of the options. The first set are noisy observations ˜ u = ⟨ ˜ u 1 , . . . , ˜ u n ⟩ of the true utility of each option. These are modeled as samples from a Gaussian centered around the true utilities

∀ i ∈ 1 , . . . , n : ˜ u i ∼ N ( f ( x i ) , σ 2 calc ) with noise σ 2 calc which we will call the calculation noise . The second set are noisy observations of the ordinal relation between the values of each attribute for each pair of options. Given an attribute k and a pair of options ( x i , x j ) , this ordinal relationship is defined by the following observation function:

˜ ˜ Given these observations ˜ u and ˜ o for options x 1 , . . . , x n , and the choice model parameters θ = ( σ 2 calc , ε, τ 1 , . . . , τ d ) , the above model implies a posterior distribution over the options' true utilities p ( u | ˜ u , ˜ o , θ ) and associated expected values E [ u i | ˜ u , ˜ o , θ ] . As they do not observe true utilities of the options, humans are assumed to choose the option y with the highest expected utility:

with τ k an attribute-specific tolerance parameter. Intuitively, a larger τ k creates a greater margin within which attribute values will be considered equal. For binary attributes, we set τ k to zero. Each noisy ordinal observation ˜ o ( x i,k , x j,k ) is sampled as follows: with probability 1 -ε sample ˜ o ( x i,k , x j,k ) = o ( x i,k , x j,k ) , otherwise sample uniformly at random from {≺ , ≻ , ≡} . The probability of ordinal error ε is a parameter, and is the sole source of noise within the ordinal observations. We will denote the set of noisy ordinal observations as o = { o ( x i,k , x j,k ) } k =1 ...d,i =1 ...n,j = i +1 ...n .

Preference learning requires that we are able to reason about how various utilities lead to different choice behaviors. Therefore, to make the original cognitive model amenable to preference learning, we replace the fixed utility function f by a space of utility functions { f w } w ∈W parameterized by a utility parameter w . We assume that the user being modeled makes choices based on some chosen parameter value w , which is known only to them, and which we represent as an additional observed random variable in the model. Necessarily, any calculation of utility therefore depends on w . Under these assumptions the user's posterior over utilities, and thus their choice y , is:

where this expectation is calculated under the posterior

3.2 Learning from choice behaviors

In our description of the model above, we have taken the point of view of the user making the choices. However, we now return to a preference learning perspective, i.e. that of an outside observer such as an AI system trying to infer the utility function that underlies these choices. We assume that the AI system observes the presented options x 1 , . . . , x n , as well as the option y the user chooses. The goal is then to infer the unknown utility parameter w and choice model parameters θ from observed

choices ( x , y ) . However, the noisy observations ˜ u and ˜ o on which the user bases their choice are part of their internal perception of the options, and are therefore not observable to an AI system. This means that in evaluating the likelihood of a choice y under the above choice model we must treat the observations as latent. This yields the following choice policy for the user:

Here, p ( y | ˜ o , ˜ u , w ) is a point mass on y following equation (1). Given m pairs ( x ( l ) , y ( l ) ) , a prior p ( w ) over the space of utility parameters and a prior p ( θ ) over the space of choice model parameters, we can use the likelihood in equation (3) to infer a posterior over the parameters w and θ :

3.3 Tractable inference through surrogates

The issue we face in calculating p ( w,θ |{ ( x ( l ) , y ( l ) ) } ) is that the likelihood p ( y | x , w, θ ) is intractable. First, the calculation of the expected values in equation (1) requires the evaluation of an intractable integral over x in equation (2). The expected values can be approximated using a Monte Carlo estimate [19], but many samples are needed to achieve a good approximation. Second, the calculation of the likelihood itself requires the evaluation of an intractable integral over all possible observations in equation (3). As before, one could approximate this integral using a Monte Carlo estimate, but this would again require many samples.

Instead, we propose to train surrogate neural networks to approximate both these quantities. We introduce a first neural network ̂ u ( ˜ u , ˜ o , w, θ ) trained to predict a vector of the expected values E [ u | ˜ u , ˜ o , w, θ ] from given observations ˜ u , ˜ o and parameters w and θ . Then ̂ u ( · ) is trained by minimizing

Samples ( w,θ, u , ˜ u , ˜ o ) are obtained by (1) sampling w ∼ p ( w ) , θ ∼ p ( θ ) and x ∼ p ( x ) from their respective priors, (2) calculating u i = f w ( x i ) for each option, and (3) sampling the observations ˜ u i ∼ p ( ˜ u i | u i , θ ) and ˜ o ∼ p ( ˜ o | x , θ ) . Note that the minimum of L util ( ̂ u ) is exactly the function that assigns to each tuple ( u , o , w, θ ) the vector of expectations E [ u | u , o , w, θ ] .

˜ ˜ ˜ ˜ Next, we train a second neural network ̂ q ( y | x, w, θ ) , which we will refer to as our CRCS model , to approximate the user's policy p ( y | x , w, θ ) over choice behaviors. By using the fact that ̂ u ( ˜ u , ˜ o , w, θ ) ≈ E [ u | ˜ u , ˜ o , w, θ ] we minimize the cross-entropy loss between ̂ q and choices based on utilities predicted by u . The loss function is thus:

Samples ( w,θ, u , ˜ u , ˜ o ) are obtained as above. 3.4 Modeling cross-feature influence in CRCS

Although our proposed model can predict a range of context effects, it does not yet capture all. Although CRCS can model how each individual feature influences the expected utility of the options, it cannot model how features can impact each other. This is something that LCL does do: its utility weight updating mechanism changes the weight of each feature based on the mean value of all other features. Thus, features can influence how other features are valued. In the most general sense, LCL's fundamental mechanism corresponds to a function g ( w, x ) which maps the utility weights w and the set of options x (which make up the context) to a new set of weights w ′ . We therefore propose to integrate this same mechanism into CRCS, resulting in a new model ̂ q ( y | x , g ( w, x ) , θ ) . As ̂ q is differentiable, we can infer g from data using gradient descent. In the experiments that follow, we will use this approach in settings where f w is a linear function. Thus, like LCL, we will define g as a linear function of x C : the mean attribute values of the options x . We will refer to the resulting model ̂ q ( y ∣ ∣ x , w +( Ax C ) T , θ ) as LC-CRCS.

4 Experiments

We first validate the proposed CRCS model by comparing its results with the original computationally rational choice model by Howes et al. [19]. Then we compare the proposed CRCS model and our LC-CRCS variant with three baselines on human choice data, and finally study the performance of the model on three case studies: car crash structure design, water drainage network design, and retrosynthesis planning. 1

We evaluate our proposed CRCS model on four datasets of human choices. These datasets are large sets of choices ( x ( l ) , y ( l ) ) collected from human participants. The District-Smart dataset [38] contains pairwise preferences over voting districts, where participants were asked to choose the district they felt was most compact. The features extracted for each district are six geometric measures identified by the original authors as good measures of compactness. The Car-Alt dataset [39] contains choices between six hypothetical alternative fuel cars. Each car has 21 features, including size, range, operating cost, etc. We also use a dataset collected in [40], which we will call the Hotels dataset, where in a user study participants were asked which of three hotels they preferred. The hotels were collected from a booking site and had as features the price per night and average review rating. For each participant, a choice was collected on one of six sets of options constructed to target three known context effects: attraction, compromise and similarity. Lastly, we use the data collected by Dumbalska et al. [36] on a property task, which we will refer to as Dumbalska. Here, participants ranked three properties in order of best to worst value. For our purposes, we will treat the top-ranked item as the choice. Value was defined as the given rental cost minus the value participants thought the house was worth (which had been elicited in an earlier stage). For each participant, responses were collected on a large collection of choices, specifically engineered to span the entire range of potential context effects. Thus, unlike the other datasets, we have multiple recorded choices per participant. This allows us to make inferences per individual, rather than at the population level, and evaluate how well our choice models fit the preferences and context effects exhibited by individuals.

4.1 Validation of the CRCS model: risky choice tasks with preference reversals

In this experiment, we validate our CRCS model against the original implementation of Howes et al. [19] on a risky choice task. In this task, a user is presented with a set of three options, each of which is a pair ( p i , v i ) consisting of a probability p i and a payoff v i . Upon selecting option i , the user receives payoff v i with probability p i , meaning that each option has expected payoff f ( p i , v i ) = p i · v i .

Comparing expected option values predicted by ̂ u with the Monte Carlo estimates used in [19], we find that on sets of three options, both generally agreed on the relative magnitude of the utilities, and agreed on the ranking of the utilities in 92 . 277% ± 0 . 165% (Agresti-Coull) of cases. Next, we verified ̂ q 's ability to predict contextual preference reversals. This was tested on Range-Frequency decoy conditions [19] where two 'Pareto-optimal" options with equal utility are presented along with a decoy option with slightly lower utility which is dominated by one of the other two options. Preference reversals - specifically, increased likelihood of choosing the Pareto-optimal option that dominates the decoy - have been observed in humans and are predicted by the original model. Figure 3 in the appendix shows that ̂ q reproduces the range of reversal rates of the original model.

4.2 Evaluation on static human choice data

In this set of experiments, we evaluate each models' ability to generalize to unseen data. We compare our proposed CRCS model and the LC-CRCS variant against three baselines: vanilla Bradley-Terry, the variant proposed by[37] (referred to as Bower & Balzano) and LCL [20]. On four different datasets, we infer the parameters for each model on a training set of observed choices { ( x ( l ) , y ( l ) ) } l m =1 and calculate the negative log-likelihood (NLL) of a held-out test set under the inferred parameters. Inference was done using gradient descent on the NLL of the training set. We performed cross-validation, and report the sum of the test sets' NLLs across the folds. For Hotels, Car-Alt and District-Smart we split the choice data across 50, 20 and 10 folds respectively. By evaluating each choice model on each test fold, we obtained paired observations (one per condition) for each test fold, allowing us to perform a Wilcoxon rank test across the folds to test significance. On Dumbalska, we look at how well the choice models can fit to individuals, and thus perform cross-validation for each participant individually. We then treated the sum of the NLLs of the test sets per participants as individual measures, and tested significance using a Wilcoxon test across the participants. Following prior work, we used a linear utility function in all choice models on all datasets.

Table 1 shows the total NLL achieved by each model on each dataset. We observe that our proposed LC-CRCS model achieves the highest NLL on Hotels, District-Smart and Dumbalska. This difference is significant (p < 0.01) in all three cases. On Car-Alt, we see that LCL performs better than all other models, with the difference being significant (p < 0.01) for all except the CRCS model (p > 0.2). We theorize that the poor performance of the CRCS model on Car-Alt is due to insufficient option data to train ̂ u on (see Appendix A.1), leading to poor estimates of expected utility and therefore poor choice predictions.

4.2.1 Evaluating the inferred utility function

As part of the District-Smart human subject study, Kaufman et al. [38] collected rankings on six sets of districts from small groups of participants. Ranking such large sets is quite difficult, and we should expect these rankings to be quite noisy. However, like the binary choices that were collected, these rankings are indicative of people's true preferences, and thus should be consistent with any ranking of the same districts implied by the utility function we infer from the binary choices. To test this, we use our choice models to infer utility parameters on the entire set of binary choices. For each of the six sets, we then measure - using Kendall's τ [41] - how consistent the ranking implied by the inferred utility parameters is with the ranking collected in the study. We report the average consistency across all six sets. Because the log-likelihood of CRCS and LC-CRCS is not convex, we repeat this procedure 25 times, starting from different points, to control for the effect local optima may have on the inferences. We test significance using a Wilcoxon test across the six sets of rankings.

Unlike the previous experiment, during inference we regularized the choice model parameters of LCL, CRCS and LC-CRCS. This was essential to infer utility parameters that were consistent with the collected rankings. For LCL we used the L1 matrix norm of the weight adaptation mechanism's parameter matrix as a regularization term. The L1 norm enforces sparsity and thus encourages LCL to fit only to the most significant context effects [20]. For CRCS and LC-CRCS we used the probability of the choice model parameters under a chosen prior as the regularization term. Using our understanding of the model this allowed us to encode specific prior knowledge into the regularization. More details can be found in Appendix A.3. From the results in Table 2 we observe that both CRCS and LC-CRCS infer utility parameters that are significantly (p < 0.001) more consistent with the collected rankings than the baselines. LC-CRCS performs slightly worse than CRCS, though the difference was not yet significant.

4.3 Elicitation on human choice data

We now evaluate how well the choice models perform in a preference learning setting, where we actively select the queries we put to a user. Whereas in the previous set of experiments we evaluated how well the choice models perform on large amounts of data, here we are interested in how well they perform when minimal data is available. We use the wealth of data available per participant (between 530 and 1060 responses) in the Dumbalska data set to run a user experiment in silico. For each choice model, we use active learning to infer utility function and choice model parameters for each participant individually. At each time step of the experiment, we select from the set of queries recorded for the participant the most informative query using the expected information gain. The participant's response to this query is then revealed, and the posterior over the utility and choice model parameters is updated. We use a particle filter to maintain the posterior beliefs. This elicitation process is performed for 25 time steps on 75 participants. To evaluate the inferences made by the choice models, we calculate the expected likelihood of the remaining queries where the choice has not been revealed yet. As the training data is actively selected for each choice model independently, the data on which they are evaluated - the remaining queries - will differ, meaning that we cannot use a paired test as we have done so far. Instead, we test significance using an independent t-test.

Figure 1a shows the mean expected utility calculated over the participants as a function of time for each choice model. We observe that the two variants of our CRCS model make significantly (p < 0.01) better predictions of the participants' choices at all time steps (except 0). Where in the previous experiment (Table 1) we saw that LCL was close in predictive power to our proposed models, we see that in this low data setting the difference is much more pronounced. This is because a number of the context effects observed in the Dumbalska data set are built into the CRCS model. While LCL has shown it can learn some of these effects, it needs much more data to do so. Interestingly, we also observe that the LC-CRCS model, which can learn some new context effects on top of the ones built into ̂ q , shows significant (p < 0.05) improvement on the CRCS model itself, even when very little data is available. This shows that it provides us with the best of both worlds, showing quick adaptability in low data settings and good performance when more data is available.

4.4 Simulated case studies

We now test the feasibility of using our model to learn a utility function from simulated choice behaviors in real-world tasks, and to use the inferred utility function to help a designer solve a task by recommending design solutions to them. We consider a preference learning setting where we learn utility and choice model parameters by iteratively eliciting a simulated designer's preferences over sets of candidate designs, chosen to maximize the expected information gain. Using ̂ q , we infer a posterior over the unknown parameters from the observed choices. To simulate a variety of users, we run this experiment in silico, using ̂ q with utility parameters sampled from a non-informative prior and choice model parameters sampled from a prior designed to capture a wide range of behaviors exhibited by the CRCS model. At each time step we measure two things: our ability to recover the unknown parameters from the observed choices, and the utility of the design recommendations we make. The inference error is measured by the distance to the true parameters under our current posterior beliefs. The second is measured using the recommendation regret: the difference in utility between the designer's optimal design and the recommended design, which is chosen to be the design with the highest expected utility under the posterior.

4.4.1 Case study 1: learning from preferences in structural design

The first use case involves the design of the frontal crash structure of a car to optimize three separate objectives g 1 ( t ) , g 2 ( t ) , g 3 ( t ) [42]. The design is parameterized by five parameters t = ⟨ t 1 , . . . t 5 ⟩ which determine the thicknesses of various metal elements. We define the utility function as the Chebyshev scalarization of the original objectives: f w ( t ) = max i ∈{ 1 , 2 , 3 } w i | g i ( t ) -z ∗ i | where z ∗ i denotes the ideal value of g i () and the weights w sum to one. Different choices of the utility weights w correspond to different trade-offs between the objectives, and therefore to different solutions on the Pareto frontier. Figure 1b shows the average recommendation regret as a function of the number of queries across 300 runs of this experiment. We observe that the recommendation regret reduces quickly, yielding good recommendations after as few as 10 queries. We attribute this to the utility inference error, shown in Figure 6a in Appendix B.1, which reduces equally quickly.

0

5

10

15

20

25

Number of preference queries

0.4

0.5

0.6

0.7

0.8

expected likelihood

CRCS

LC-CRCS

Bradley-Terry

LCL

Bower & Balzano

(a) Dumbalska

0

10

20

30

40

50

Number of preference queries.

0.01

0.02

0.03

0.04

0.05

Recommendation regret

(b) Crash structure

0

20

40

60

Number of preference queries.

0.005

0.010

0.015

0.020

0.025

Recommendation regret

(c) Water drainage

2

4

6

8

10

Number of top k.

0.64

0.66

0.68

0.70

0.72

0.74

0.76

0.78

Maximum true utility score

Aizynthfinder (Baseline)

Ours

(d) Retrosynthesis

4.4.2 Case study 2: learning from preferences in water drainage network design

Our second use case is a water drainage network design problem [43]. This use case is another multiobjective problem involving six objectives g 1 ( z ) , . . . , g 6 ( z ) . Here, we scalarize the problem using a weighted sum f w ( z ) = ∑ 6 i =1 w i g i ( z ) where the weights w sum to one. As before, different choices of w correspond to different solutions on the Pareto frontier. We ran 300 runs of this experiment. Here too we see that recommendation regret (Figure 1c) and utility inference error (Figure 6a in the appendix) drop quickly as we put more queries to the designer, with the largest reduction within the first 10 queries. We achieve high-quality recommendations after less than 10 queries.

4.4.3 Case study 3: improving retrosynthesis planning with preference learning

Retrosynthesis planning, the problem of finding feasible reaction pathways to synthesize target molecules, is a central task of synthetic chemistry. Significant progress has been made in solving it through end-to-end automatic synthesis planning [44-46]. Existing work has focused on expanding the search space of feasible reaction plans. Each route may satisfy an additional subset of properties, and different individuals or organizations may have different preferences over the properties. Chemists' preferences over these plans are often highly complex, representing trade-offs between multiple objectives informed by personal experience and company policy. However, learning their preferences in a way that can then inform AI-driven synthesis planning has not yet been done. We designed a chemist-in-the-loop retrosynthesis planning framework to generate routes with an inferred user model.

To build a personalized retrosynthesis planner, we modified one of the state-of-the-art automatic retrosynthesis platforms, Aizynthfinder [45], built on Monte Carlo Tree Search (MCTS) with a fixed utility function. Details can be found in Appendix B.2.1. First, we proposed a new utility function as the weighted combination of five feature properties g 1 ( r ) , . . . , g 5 ( r ) , that correspond to reactants cost, intermediates stability, reaction feasibility, total reaction success rate, poor reaction success rate , and a route score computed by a data-driven scoring model g 5 . Given the input routes, this model predicts the distance between the current route and the (latent) optimal route. We trained this model on 47,055 synthetic routes extracted from the Journal of Medicinal Chemistry.

We report the inference error during preference learning in Appendix B.2.2. We integrated the inferred utility weights into our planning system and assessed the consistency of the generated route with the ground truth user utility preferences. We used inferred weights to synthesize 100 target molecules for each weight. In order to measure the recommendation quality, we evaluated the top-ranked routes from both Aizynthfinder and our model under the true utilities. Specifically, we measured the maximum true utility score among the list of top k recommendations. This is to show how far down from the recommended options list the user needs to go to find their optimal choice. Figure 7c shows that within the top k options, the reaction pathways recommended from our model reaches higher maximum utility score compared to the ones generated by Aizynthfinder. As a significance test, we use the Wilcoxon rank test across every molecule and every user utility with p < (1 . 61 × 10 -53 ) for all k .

5 Conclusion

In this paper we have proposed a tractable surrogate model of choice, called CRCS, inspired by theories of human decision-making. This model was shown to be a better basis for preference learning than some, but not all, existing models. In response, we modified the model so that it could make cross-feature observations of feature values extending the opportunity for contextual decision-making. We verified against human data from a range of tasks that the new model, called LC-CRCS, outperforms the tested models both in terms of its ability to predict choices and in its inferences of the utility function that underlies the observed choices. Moreover, we find that it corresponds well to previously reported experimental data demonstrating human susceptibility to contextual choice effects. Feasibility of using the new model for preference learning and its ability to recover parameters was also demonstrated in three case studies. Together, the results demonstrate the viability of CRCS and LC-CRCS in high performance preference learning systems.

Limitations and future work We identify two primary limitations. First, training CRCS requires sufficiently many choice sets, or a sufficient well-specified task so that new sets can be generated. As we saw with Car-Alt, when insufficient choice sets are available for training, performance can suffer. Second, CRCS and CRCS-LC only work on choice sets of fixed size. Extending these surrogates to variable size choice sets is a promising direction for future work. Another promising direction for future work is the application of the current choice model to large language model (LLM) fine-tuning. Currently, given some featurization of LLM responses to a prompt, CRCS could be directly applied. However, this would ignore the reading and interpreting of these responses that human evaluators have to do. As such, we see an extension of the current choice model that integrates these cognitive processes, based on the same computational rationality theory, as potentially transformational future work.

Societal impact This paper presents work motivated by the goal to advance the field of Machine Learning. The potential societal impact is in line with the broad body of prior work on learning from preferences and modeling humans, none of which we feel must be specifically highlighted here.

Acknowledgements This work was supported by the Research Council of Finland (flagship programme: Finnish Center for Artificial Intelligence, FCAI; grants 345604, 341763 and 359207), and the UKRI Turing AI World-Leading Researcher Fellowship, EP/W002973/1. Computational resources were provided by the Aalto Science-IT Project.

References

Christian Wirth, Riad Akrour, Gerhard Neumann, Johannes Fürnkranz, et al. A survey of Preference-Based Reinforcement Learning Methods. Journal of Machine Learning Research , 18(136):1-46, 2017.

Paul F Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforcement Learning from Human Preferences. In Proceedings of the 31st International Conference on Neural Information Processing Systems , pages 4302-4310, 2017.

Andras Kupcsik, David Hsu, and Wee Sun Lee. Learning dynamic robot-to-human object handover from human feedback. Robotics Research: Volume 1 , pages 161-176, 2018.

Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems , 31, 2018.

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems , 33:3008-3021, 2020.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

Shenghuan Sun, Gregory M Goldgof, Atul Butte, and Ahmed M Alaa. Aligning synthetic medical images with clinical knowledge using human feedback. arXiv preprint arXiv:2306.12438 , 2023.

Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3/4):324-345, 1952.

John W Payne, James R Bettman, and Eric J Johnson. Adaptive strategy selection in decision making. Journal of experimental psychology: Learning, Memory, and Cognition , 14(3):534, 1988.

Takao Noguchi and Neil Stewart. Multialternative decision by sampling: A model of decision making constrained by process data. Psychological review , 125(4):512, 2018.

Andrea M Cataldo and Andrew L Cohen. The comparison process as an account of variation in the attraction, compromise, and similarity effects. Psychonomic Bulletin & Review , 26(3): 934-942, 2019.

Douglas H Wedell. Distinguishing among models of contextually induced preference reversals. Journal of Experimental Psychology: Learning, Memory, and Cognition , 17(4):767, 1991.

Konstantinos Tsetsos, Rani Moran, James Moreland, Nick Chater, Marius Usher, and Christopher Summerfield. Economic irrationality is optimal during noisy decision making. Proceedings of the National Academy of Sciences , 113(11):3102-3107, 2016.

Joel Huber, John W Payne, and Christopher Puto. Adding asymmetrically dominated alternatives: Violations of regularity and the similarity hypothesis. Journal of consumer research , 9(1): 90-98, 1982.

Konstantinos Tsetsos, Marius Usher, and Nick Chater. Preference reversal in multiattribute choice. Psychological review , 117(4):1275, 2010.

Sharoni Shafir, Tom A Waite, and Brian H Smith. Context-dependent violations of rational choice in honeybees (apis mellifera) and gray jays (perisoreus canadensis). Behavioral Ecology and Sociobiology , 51:180-187, 2002.

Richard L Lewis, Andrew Howes, and Satinder Singh. Computational rationality: Linking mechanism and behavior through bounded utility maximization. Topics in cognitive science , 6 (2):279-311, 2014.

Falk Lieder and Thomas L Griffiths. Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources. Behavioral and brain sciences , 43:e1, 2020.

Andrew Howes, Paul A Warren, George Farmer, Wael El-Deredy, and Richard L Lewis. Why contextual preference reversals maximize expected value. Psychological review , 123(4):368, 2016.

Kiran Tomlinson and Austin R Benson. Learning interpretable feature context effects in discrete choice. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 1582-1592, 2021.

Wei Chu and Zoubin Ghahramani. Preference learning with gaussian processes. In Proceedings of the 22nd international conference on Machine learning , pages 137-144, 2005.

Neil Houlsby, Ferenc Huszar, Zoubin Ghahramani, and Jose Hernández-lobato. Collaborative gaussian processes for preference learning. Advances in neural information processing systems , 25, 2012.

Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning , pages 783-792. PMLR, 2019.

Daniel S Brown, Wonjoon Goo, and Scott Niekum. Better-than-demonstrator imitation learning via automatically-ranked demonstrations. In Conference on robot learning , pages 330-359. PMLR, 2020.

Vivek Myers, Erdem Biyik, Nima Anari, and Dorsa Sadigh. Learning multimodal rewards from rankings. In Conference on robot learning , pages 342-352. PMLR, 2022.

Ashish Kapoor, Kristen Grauman, Raquel Urtasun, and Trevor Darrell. Active learning with gaussian processes for object categorization. In 2007 IEEE 11th international conference on computer vision , pages 1-8. IEEE, 2007.

Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745 , 2011.

Kevin G Jamieson and Robert Nowak. Active ranking using pairwise comparisons. Advances in neural information processing systems , 24, 2011.

Erdem Bıyık, Nicolas Huynh, Mykel J Kochenderfer, and Dorsa Sadigh. Active preferencebased gaussian process regression for reward learning. arXiv preprint arXiv:2005.02575 , 2020.

Sudeep Bhatia, Graham Loomes, and Daniel Read. Establishing the laws of preferential choice behavior. Judgment and Decision Making , 16(6):1324-1369, 2021.

Jerome R Busemeyer, Sebastian Gluth, Jörg Rieskamp, and Brandon M Turner. Cognitive and neural bases of multi-attribute, multi-alternative, value-based decisions. Trends in cognitive sciences , 23(3):251-263, 2019.

Amos Tversky, Paul Slovic, and Daniel Kahneman. The causes of preference reversal. The American Economic Review , pages 204-217, 1990.

Paul W Glimcher. Efficiently irrational: deciphering the riddle of human choice. Trends in cognitive sciences , 26(8):669-687, 2022.

Petter Johansson, Lars Hall, and Nick Chater. Preference change through choice. In Neuroscience of preference and choice , pages 121-141. Elsevier, 2012.

Francesco Rigoli, Christoph Mathys, Karl J Friston, and Raymond J Dolan. A unifying bayesian account of contextual effects in value-based choice. PLoS computational biology , 13(10): e1005769, 2017.

Tsvetomira Dumbalska, Vickie Li, Konstantinos Tsetsos, and Christopher Summerfield. A map of decoy influence in human multialternative choice. Proceedings of the National Academy of Sciences , 117(40):25169-25178, 2020.

Amanda Bower and Laura Balzano. Preference modeling with context-dependent salient features. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 1067-1077. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/ bower20a.html .

Aaron R Kaufman, Gary King, and Mayya Komisarchik. How to measure legislative district compactness if you only know it when you see it. American Journal of Political Science , 65(3): 533-550, 2021.

David Brownstone, David S Bunch, Thomas F Golob, and Weiping Ren. A transactions choice model for forecasting demand for alternative-fuel vehicles. Research in Transportation Economics , 4:87-129, 1996.

David Ronayne and Gordon DA Brown. Multi-attribute decision by sampling: An account of the attraction, compromise and similarity effects. Journal of Mathematical Psychology , 81: 11-27, 2017.

Maurice G Kendall. A new measure of rank correlation. Biometrika , 30(1/2):81-93, 1938.

Xingtao Liao, Qing Li, Xujing Yang, Weigang Zhang, and Wei Li. Multiobjective optimization for crash safety design of vehicles using stepwise regression model. Structural and multidisciplinary optimization , 35:561-569, 2008.

Ryoji Tanabe and Hisao Ishibuchi. An easy-to-use real-world multi-objective optimization problem suite. Applied Soft Computing , 89:106078, 2020.

Ola Engkvist, Per-Ola Norrby, Nidhal Selmi, Yu-hong Lam, Zhengwei Peng, Edward C Sherer, Willi Amberg, Thomas Erhard, and Lynette A Smyth. Computational prediction of chemical reactions: current status and outlook. Drug discovery today , 23(6):1203-1218, 2018.

Samuel Genheden, Amol Thakkar, Veronika Chadimová, Jean-Louis Reymond, Ola Engkvist, and Esben Bjerrum. Aizynthfinder: a fast, robust and flexible open-source software for retrosynthetic planning. Journal of cheminformatics , 12(1):70, 2020.

Weihe Zhong, Ziduo Yang, and Calvin Yu-Chian Chen. Retrosynthesis prediction using an end-to-end graph generative architecture for molecular graph editing. Nature Communications , 14(1):3009, 2023.

u

i

w

x

i

u

i

˜

o

θ

n

u

i

w

x

(

l

)

i

˜

u

(

l

)

i

y

(

l

)

˜

o

(

l

)

θ

n

m

(a) Our choice model, originally introduced in [19], posits that humans make utility-maximizing choices (for some utility function parameters w and choice model parameters θ ) based only on observations ( ˜ u , ˜ o ) . The options x 1 , . . . , x n and their true utilities u 1 , . . . , u n are not observed.

˜

(b) An outside observer observes a set of choices y ( l ) made over associated options x ( l ) 1 , . . . , x ( l ) n . From this data set, the objective is to infer the parameters w and θ . The noisy observations ( ˜ u ( l ) , ˜ o ( l ) ) that are central to each of the user's choices are internal to the user and are therefore unobserved.

0.30

reversal

0.25

0.20

inverse

0.15

0.10

I

0.05

reversal

0.00

refrence

-0.05

surrogate

0

1

2

3

4

5

calculationnoise SD

A Human data experiments

A.1 Priors

This section provides details on how the CRCS model was trained for the choice tasks corresponding to the Hotels, District-Smart, Car-Alt and Dumbalska datasets. In order to train our CRCS model on a new choice task, we need to define three priors: a prior over sets of options p ( x ) , a prior over utility function weights p ( w ) , and a prior over choice model parameters p ( θ ) .

The prior over sets of options p ( x ) is by far the most important prior for successfully training the CRCS model. It is clearly important that this prior matches the distribution of choice sets we expect to see for the choice task we target. However, it is even more important to ensure that that prior has proper support across the entire space of option sets. From equation 2 we see that in order to predict the true utilities of the options x , ̂ u essentially has to infer the option set x (which it does not observe) from the observations ˜ u and ˜ o . It can only learn to do this well if during training we can expect it to encounter all x that could have resulted in u and o .

For Hotels we had access to the set of 200 hotels the original authors had used to build their study. Thus, we generated option triplets from the prior by uniformly sampling (without replacement) three hotels from this set.

˜ ˜ The priors p ( x ) for these tasks were defined as follows:

(a) Empirical option distribution

(b) Prior p(x)

2500

2500

participant valuation

2000

participant valuation

2000

1500

1500

1000

1000

500

500

0

500

1000150020002500

0

500

1000150020002500

rentalcost

rentalcost

For District-Smart we had access to 21778 electoral districts collected by the original authors. We generated pairs of options by sampling them uniformly from this set without replacement.

Options in the Dumbalska task have two features, a property's rental cost and the participant's valuation of it, both of which were bounded between 0 and 2500. The human choice data suggested that both are strongly correlated. We created a prior over individual choices that reproduces this correlation by using a multivariate normal distribution, mixed with a uniform distribution over the entire option space, to ensure sufficient support even on less frequently encountered options. Figure 4 shows the empirical option distribution within the choice data, and our engineered prior over individual options. To generate option sets, we then sampled three options from this prior.

Car-Alt considers options sets consisting of hypothetical cars. Although we know that the options were created from a set of 120 cars, the features of each option are determined both by the Car that corresponds to that option and the participant who makes the choice. For example, the cost of each car is expressed as a multiple of the participant's log income. The inclusion of participant-dependent features creates correlation between the options. Unfortunately, we did not have enough information to engineer a new prior p ( x ) that would faithfully reproduce this correlation, and would faithfully match the empirical distributions over sets of options encountered in the original user study. Thus, we were forced to train the CRCS model on the limited number of choice sets that appear in the human choice data.

As all tasks used linear utilities, and as the CRCS model is invariant to scaling of the utility function, we define p ( w ) in all cases as a uniform distribution over all vectors of length 1.

The prior over the choice model parameters was set based on choice model parameters reported for risky choice in [19] after accounting for scale differences of utilities and feature values for each task. They are listed in table 4 2 . For LC-CRCS, we additionally placed independent standard normal priors on all entries of A .

A.2 Additional details on static data experiments

The experiments on static data were run using a cross-validation strategy. For each fold, we inferred utility parameters and choice model parameters jointly for each choice model by performing vanilla gradient descent on the train set log-likelihood. For the CRCS and LC-CRCS model the starting points were sampled from the priors defined in section A.1. For the Bradley-Terry variants the utility parameters were sampled independently from a standard Gaussian, and for LCL the learnable parameter matrix A was populated using sampled from N (0 , 0 . 1) .

CRCS and LC-CRCS have non-convex likelihood functions, meaning that gradient descent is liable to get stuck in local optima. To mitigate this, we performed inference multiple times (up to 50), starting from multiple starting points, and chose the parameters that achieved the best log-likelihood on a held-out part of the training data. Table 5 shows the variance in average NLL achieved by individual inference runs on the various choice problems. We can see that there is significantly higher standard deviation when doing inference with CRCS and LC-CRCS, pointing to the existence of local minima, and confirming the necessity of repeated inference to address this.

A.3 Additional details on rank consistency experiments

For District-Smart, we used gradient descent to infer the utility and choice model parameters for each choice model on the entire set of binary choices. For LCL, CRCS, and LC-CRCS we noticed that the inferred utility functions were highly inconsistent with the rankings that had been collected in the same user study. As explained in the main paper, we used regularization to address this. We will go into some more detail on why we think regularization was needed and how we tuned the regularizers

0

5

10

15

20

25

Number of preference queries

3.0

2.5

2.0

1.5

1.0

expected log-likelihood

CRCS

LC-CRCS

Bradley-Terry

LCL

Bower & Balzano

we used. Table 6 below shows the consistency of the ranking implied by the inferred weights for each of these three choice models with the collected rankings. We can see immediately that compared

to Bradley-Terry and the Bower & Balzano models, the consistency is quite poor, especially for LCL. We hypothesize that this is caused by heterogeneity in the task utilities different participants used in making their choices. The intention of Kaufman et al. [38] was to capture humans' intuitive understanding of what it means for an electoral district to be compact. Therefore, in the user study, people were encouraged to make choices "according to your own best judgement" [38]. As it is likely that the various participants in the study had slightly different intuitions about compactness, we can therefore expect that the recorded choices have been made with slightly different utility weights w . To fit a choice model using a single utility to these choices could thus prove problematic. For future work, it would be interesting to consider a hierarchical approach, where we would model the fact that any choice has been made according to an unobserved utility function drawn from some unknown distribution.

For the current work, we resorted to using regularization to ensure the choice models did not overfit to the noise in the utility function. These regularization strategies were tuned using one of the six rankings collected, while for evaluating the consistency of the inferred weights with the collected rankings we used all six. For LCL, we regularized the weight update mechanism w + Ax C by using the norm of A , multiplied by 75 to get the desired regularization strength, as a regularization term. For the CRCS model, to ensure that we make sensible inferences, we had to ensure that the noise in the utility function would be explained by the right source of noise in the model. The prior we placed on the choice model parameters was designed to do just this. We used placed a Beta (1 , 1000) to ensure that ε , which determines the level of noise on the attribute comparisons, would stay close to 0. Attribute comparisons are the primary source of context effects and are necessary to fit to any such effects in the data. Additionally, we note that in the experiment conducted in [19], ε was fixed to 0. We then placed a N (25 . 0 , 0 . 1) prior on σ calc , the noise on the utility observations, to help explain the heterogeneity of utility functions itself. We also placed a weak prior on τ 1 , . . . , τ d , namely the prior we also used when training the CRCS model. The same priors were used for LC-CRCS.

A.4 Additional details on elicitation experiment

The elicitation experiment on the Dumbalska dataset was performed for each participant in the original experiment individually. We excluded participants according to the same rule the original paper had used. For each run, we would select a participant from the dataset and treat the queries to which responses had been collected for this participant as the queries we could put to the user. In each time step, we used the posterior at that point to estimate the expected information gain of each query that had not been used yet, and selected the query with the highest information gain. The recorded response to this query would then be revealed, and the posterior would be updated with this new observation using the choice model. To maintain the posterior, we used a particle filter containing up to five million particles representing combinations of utility and choice model parameters. The particle filter was not refreshed during the experiment. At each time step we measured the expected likelihood, where the expectation was taken with regard to the posterior and a uniform distribution over the remaining set of recorded queries (those which had not yet been selected as part of the elicitation process). For completeness, we also measured the expected log-likelihood and the entropy in the marginal utility parameter posterior. Those are shown in Figure 5.

A.5 Additional information on the datasets used

We list below here the sources for the data we use in the human data experiments for Dumbalska [36], Car-Alt [39], Hotels [40] and District-Smart [38]. We obtained the human choice data for DistrictSmart and Car-Alt from the excellent collection of choice data collected by Tomlinson and Benson [20] for their implementation.

B Use cases

B.1 Structural design and water drainage network design

Here, we will describe additional details on the crash structure design and water drainage network design use cases. Both use cases were run with the same priors for the utility parameters and choice model parameters. The utility parameter prior was a uniform Dirichlet distribution. The choice model parameters for the CRCS model were chosen to capture the widest possible range of choice behaviors.

In each step, candidate choice queries were generated by sampling 1000 queries of three design options from a uniform prior over the domain of either use case. The candidate with the highest expected information gain was chosen. For crash structure design we elicited responses to 50 preference queries in each run of the experiment and for water drainage network design we elicited responses to 100 queries, though for space reasons we only show the first 75 on the graphs in this paper. The recommendations on which we measured the recommendation regret at each step were selected by maximizing the expected utility under the current posterior over a pre-calculated Pareto

0

10

20

30

40

50

Number of preference queries.

0.2

0.3

0.4

0.5

Mean utility inference error

(a) Crash structure

0

10

20

30

40

50

Number of preference queries.

0.19

0.20

0.21

0.22

0.23

0.24

Mean model parameter inference error

(b) Crash structure

0

20

40

60

Number of preference queries.

0.30

0.35

0.40

0.45

Mean utility inference error

(c) Water drainage

0

20

40

60

Number of preference queries.

0.18

0.20

0.22

0.24

Mean model parameter inference error

(d) Water drainage

front for the user case. The user's optimal design was found by optimizing the true utility over this same front.

Figure 6 shows the utility and choice model parameter inference errors for both use cases.

B.2 Retrosynthesis planning

B.2.1 Aizynthfinder

Here we provide further details about Aizynthfinder [45]. Aizynthfinder 3 is an open-source retrosynthesis planner that uses Monte Carlo Tree Search (MCTS) and a template-based expansion policy 4 to search for possible reactions and an additional filter policy that discard the infeasible reactions. The expansion policy is a multi-class classification model that predicts the most probable reaction templates. In practice, this model produces the top 50 possible templates as the possible action during the tree expansion process. Then, the infeasible reactions are filtered out with the filter policy. During the expansion and selection phase of MCTS, it uses the upper confidence bound (UCB) to select and score routes as defined:

where n is the visitation times of a node, C is the bias hyperparameter set to 1.4, and Q is the accumulated reward that is defined as:

0

10

20

30

40

50

Number of preference queries

0.15

0.20

0.25

0.30

0.35

0.40

0.45

Mean inference error

utilty parameters

model parameters

where N molecules in stock is the current number of molecules in stock according to a given database, N molecules is the total number of the molecules in the current search tree and depth is the depth of the search tree.

B.2.2 Inference results

Now, we report the inference results over the preference weighs. First, we simulate the synthetic user weights by sampling 100 different weight combinations from a uniform Dirichlet distribution Dir( α ) with α = (1 , 1 , 1 , 1 , 1 , 1) . Figure 7 shows that both inference error and recommendation regret of the utility function and choice model parameters reduce during the inference using 50 preference queries.

In addition, we report the average number of solved routes from both Aizynthfinder and our model. For Aizynthfinder, we synthesize just 100 molecules, while for ours, we collect the statistics over 100 the target molecules for 100 inferred user utilities. The average number of solved routes is 45 . 62 ± 28 . 99 and 42 . 75 ± 28 . 89 for Aizynthfinder and ours respectively. These results are justified, as the synthetic user utilities are randomly sampled from non-informative prior.

Hidden Layer E2

Hidden Layer E3

Embedding

x

Hidden  M2

Hidden  M3

w

C Surrogate architecture and training

We provide details here on the architecture of the two neural networks ̂ u and ̂ q . Both networks are multi-task networks; they make their predictions conditioned on the utility parameters w and the choice model parameters θ = ( σ calc , ε, τ ) . ̂ u takes as input a vector of observations ˜ u , ˜ o and predicts the expected utility of each option. ̂ q takes as input a set of options x and predicts the likelihood that each option will be chosen.

Both networks have a number of hyperparameters such as layer output sizes. Other hyperparameters include training details such as learning rates and batch sizes. These hyperparameters were selected using a grid search over a predefined set of possible values for each hyperparameter. We selected the hyperparameters that minimized the loss of each surrogate neural network on each individual choice task. Table 7 lists the selected network hyperparameters: the layer output sizes for both the embedding module and main module, and the size of the embedding produced by the embedding module. The last three columns list training hyperparameters: the batch size, the number of training epochs, and the learning rate at the start and end of training.

The architecture of both networks is virtually identical, differing only in their inputs, and in the fact that the output of ̂ q is transformed with a log-softmax function, while ̂ u 's is not. Figure 8 visualizes the architecture of both networks. The networks consist of two modules. The first in an embedding module consisting of three layers that takes the utility and choice model parameters and embeds them into a latent embedding space. The second module, the main module, consists of four layers and takes as input the set of options (or the observations in the case of ̂ u ) and transforms these into log-likelihood predictions (expected utilities for ̂ u ). To condition the main module on the utility and choice model parameters, we concatenated the embedding from the embedding module with the input to layers 2 and 3 of the main module (see Figure 8). We trained ̂ u and ̂ q using the AdamW optimizer implemented in pytorch with an exponentially decaying learning rate.

D Computational resources

In table 8 we report the computational cost of the experiments reported in this paper. We only report the resources used to obtain the results presented. We estimate that if we were to include all testing and preliminary runs, the total compute time used would double or triple. We do not report the cost of validating the CRCS model on risky choice as the runtime was negligible.

All the experiments were run on either CPUs on a cluster or on a MacBook Pro. The cluster CPU jobs were single core unless otherwise mentioned. MacBook CPU jobs used multiple cores (typically less than 2.5 cores) with 16GB of memory. We estimate the maximum power consumption of a single core on our cluster to be around 7.5W, and the power usage of a single core on a MacBook to be significantly less. Assuming the worst case scenario where all computation was run on the cluster, the total energy used to obtain the reported experiments would be 146KWh. Given an average carbon

intensity of 94g CO2eq/KWh for the electricity supplied to our cluster in 2023, this means that we estimate the carbon footprint of these experimental results to be around 14kg CO2eq.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We test our proposed models on a varied set of data sets and use cases.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of the work in the conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper includes no theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide additional details on the experiments in the appendices, such as sources for the data used, neural network architectures and priors used. The code for our experiments is additionally available online.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our implementation of the experiments is available online. We provided direct links to the data needed to run the experiments, and where processing of the data is needed the required code is provided.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All hyperparameters such as the number of experimental runs, number of data splits and priors are listed in the paper. These hyperparameters are also contained in the code which is available online.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The paper reports error bars suitably and correctly defined, providing appropriate information about the statistical significance of the experiments. We ensured that all reported results include measures of variability, such as standard deviation or confidence intervals, to accurately represent the reliability and significance of our findings.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All information about compute resources can be found in Appendix D, including the type of compute workers, memory, execution time and infrastructure.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in this paper adheres to the NeurIPS Code of Ethics in all respects. We adhered to privacy standards, obtaining explicit consent for data usage where applicable. Deprecated datasets were avoided, and data licensing terms were respected. We conducted thorough assessments to prevent biases and considered the societal and environmental impacts of our research, ensuring it does not facilitate harm, discrimination, or privacy violations. Detailed documentation and secure data practices were followed to support transparency, reproducibility, and legal compliance.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the broader impact of the paper in the conclusion.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: For the data we use we cite the papers which originally introduced those datasets. We also provide direct links to the data and license information in the appendices.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: This paper introduces two new models, implementations for which are available online.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Though the paper uses a number of datasets collected from human subject experiments, it did not collect any new data itself.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper performed no new human subject experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper uses a state-of-the art-models model from psychology for preference learning. To make learning tractable, the authors use variational inference to approximate unknown distributions (and expectations). The model is shown to outperform existing approaches, suggesting that it better predicts human preferences and captures contextual choice effects that are commonly observed in human behaviour.\n\nSTRENGTHS:\n- Preference learning is a very relevant topic. This work borrows a state-of-the-art model cognitive model from psychology (Howes et al., 2016) and aims to make it computationally tractable to be applied in the context of preference learning.\n\n- The paper is well-motivated and clearly written. It effectively balances the mathematical theory with intuitive explanations.  \n\n- I find this work to be a nice example of cross-disciplinary ML project: model from psychology, use of variational methods to make it computationally feasible, diverse set of experiments (e.g. policy making, hotel/property rankings)\n\n- Thorough empirical evaluation: section 4.1 ensures that the learnt surrogate is a good approximation (when compared to MCMC); then in the following subsections, CRCS is evaluated on a diverse set of datasets (as already mentioned in the previous bullet), as well as simulated case studies. K-fold cross validation is performed, and Wilcoxon rank test  is done to check for statistical significance.\n\nWEAKNESSES:\nNo major weaknesses. Some minor suggestions:\n\n- Line 56 and the whole paragraph after: the meaning of CRCS acronym is not introduced until line 125.\n\n- line 153: when reading this paragraph for the first time, I found it strange that the parameters $w$ are observed (known). It then became clear that they are observed only by the *user* that is being modelled. \n\n- a graphical model describing all the variables in the model can be useful; it can help visualise the differences between the modelled user and the outside observer (which variables are observed or unobserved). Clearly indicating what is given and what is learnt - e.g. the prior $p(w)$ and the marginal $p(x)$ are both assumed given?\n\n- line 221: typo though -> thought\n\nQUESTIONS:\n1. What is the number of parameters for each of the models? It will be good to have these reported in Table 1.\n2. I might have missed it, but I didn't see the network architectures for $\\hat{u}$ and $\\hat{q}$?\n3. Have you thought applying the model to LLMs for preference learning?\n\nLIMITATIONS:\nSome of the limitations of this work are discussed in Section 5. Societal impact statement is also included in Section 5.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces two models for learning preferences from human choice behaviors: the Computationally Rational Choice Surrogate (CRCS) and a variant (LC-CRCS) that considers the context effect of the choice set. Both models are based on a state-of-the-art cognitive model of human decision-making. The paper replaces intractable computations in the original model with surrogates to enable feasible estimations and demonstrates the numerical performance of these models through several experiments.\n\nSTRENGTHS:\nThe paper is well-written and presents a new model, which is based on Howes et al. (2016), for understanding human choice behaviors. Compared to the original model, the new model is more computationally tractable. Also it uses several real-world datasets to test the models.\n\nAndrew Howes, Paul A Warren, George Farmer, Wael El-Deredy, and Richard L Lewis. Why contextual preference reversals maximize expected value. Psychological review, 123(4):368, 2016.\n\nWEAKNESSES:\n1. Limited Applicability: The models require data that includes the orderings of each feature (or their generation processes) and the true utilities (or their generation processes). Such data is not commonly available. Additionally, prior knowledge of several parameters must be estimated before training the models. For instance, each feature needs a distribution for the corresponding $\\tau$, which becomes challenging when the number of features is large.\n   \n2.Experimental Focus: The experimental results primarily focus on negative log-likelihoods (NLLs). Although the proposed methods outperform benchmarks in terms of NLLs, it is unclear if this improvement is practically significant. Comparing other metrics (e.g., accuracy) would better demonstrate the contribution of this work. Moreover, the paper does not provide standard deviations for the results to indicate robustness, even though the results are statistically significant.\n\n3. Training Robustness: Line 561 mentions that \"As CRCS and LC-CRCS have non-convex likelihoods, we performed gradient descent multiple times, starting from multiple starting points, and chose the parameters that achieved the best train set log-likelihood.\" It would be helpful to show the mean and standard deviations over several random seeds instead of just the best result to demonstrate training robustness. Training on large datasets can be time-consuming, making multiple training runs impractical.\n\nQUESTIONS:\n1. How can the true utility or its generation process be observed or estimated in real-world scenarios?\n2. How should the prior distributions in Table 3 be selected or estimated? Are there general methods for estimating these prior distributions from a given dataset?\n3. How do the models perform on other metrics (e.g., accuracy)?\n4. What are the architectures of neural networks $\\hat{q}$ and $\\hat{u}$?\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors present a new model which approximates an existing intractable Bayesian model for preference learning. The paper describes a generative model for choice, and shows how inference can be approximated using two different neural networks. The proposed model is then augmented using a cross-feature correction inspired by another piece of existing work. Results show that the proposed model outperforms the baselines on sensible metrics like negative log likelihood of the test set.\n\n**Conclusion:** Overall, the paper makes a neat proposal and is fairly meticulous in developing the central idea. However, the contribution doesn’t appear substantial and might be of interest to a smaller subset of the community.\n\nSTRENGTHS:\n1. The paper provides ample background and motivating examples, clearly elucidating the relevance of the problems.\n2. The development of the model is systematic, and key steps are formally defined making it easy to follow the progression of the ideas.\n3. The model appears more expressive in general since it takes into account different sources of noise, and more independent variables (such as the cross-feature influence or the error tolerance of comparing features).\n4. The results seem to be generally promising on both real and synthetic data.\n\nWEAKNESSES:\n1. While it was easy to follow the theory, I found it a little difficult to deeply understand the application section. A little more focus on methodology on at least one experiment might have been more helpful. Specifically, it was a little tricky to cleanly connect the theory to the application for me.\n2. The contribution is somewhat incremental, and might be better suited for area-specific workshop. Since the Bayesian model already exists, the main contribution is estimating the two intractable quantities using NNs, if I understand correctly. It’s useful to have such models, but I am not sure if they generate any foundational insights.\n3. For synthetic data, if the generative model is closer to the inference model, it’s not very surprising that the latter does better than another model unaware of the generative process. This weakens the evidence on synthetic data a little.\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submission proposes an approach to preference learning using a model inspired by findings in cognitive science. Specifically, it uses an amortized inference variant of a previously proposed model to enable tractable inference of preference values, and applies it to a number of case studies, where it is shown to outperform both the classical Bradley-Terry model, and additional more recent baselines.\n\nSTRENGTHS:\nThe paper sets up its problem well and provides a suitable review of past work. I particularly appreciated the very crisp distinction between user's and outside observer's model in section 3.2. Performance is competitive relative to the baselines chosen, which include relatively recent methods.\n\nWEAKNESSES:\nMy primary concerns with the paper are related to the actual technical implementation details, which are not articulated clearly enough, and where they are articulated give me some pause. Specifically: \n* If I understand things correctly, the policy network takes in $x, w, \\tilde{u} and \\tilde{o}$, and a sufficiently flexible surrogate should be able to learn arbitrary functions of x and w already. Is this correct? If so, why is the explicit feature engineering needed? The paper also discusses later \"built in\" context effects in CRCS and \"learned\" ones in LR-CRCS -- what is meant by this? \n* This also makes me wonder: what's the network architecture used and other training hyperparameters (optimizer, learning rates, etc)? In general I would expect a sufficiently flexible surrogate to also achieve better than ~92% agreement with the original model. I'm also puzzled about the claim of insufficient option data to claim the utility network -- shouldn't it be possible to simulate arbitrary amounts of observations in this case such that the network fully spans the range of possible outcomes and enabling amortized inference? I can see why sampling only from the training data would create this restriction, but I don't understand why this choice was made.\n\nQUESTIONS:\n* Reporting summed rather than averaged NLLs (with standard errors) seems like an odd choice -- why was this done? \n* Can the authors comment about the choice of Wilcoxon signed-rank test for some hypothesis tests and t-tests for others? \n* Why are baselines not present in the regret panels of figure 1?\n\nAdditional Notes:\n* Figure 1 tick labels should be larger. \n* L53 maybe \"computational rationality analysis\"? \n* L66 it's -> its\n* The notation in section 3.3. is a bit confusing, in the following way: u is a function of x, but the notation sometimes uses u and sometimes x (e.g. in the expectations in the utility and policy loss, or how \\tilde{u} is conditioned on u but \\tilde{o} is conditioned on x. I think this might be because of a distinction between the observer's and user's likelihoods, but this isn't quite spelled out here. \n* I believe that the overall setup here is a bilevel optimization problem (because L_util is optimized w.r.t. w and L_pol w.r.t. x), and I'm not sure how incomplete optimization or other issues in the utility network impact the policy network (especially considering that the accuracy of the utility network seems not ideal). \n* Line 278 I believe \\citet should be \\citep there?\n\nLIMITATIONS:\nDiscussion is adequate, though as noted above I'm puzzled about the limitation on requiring sufficiently many choice sets for estimation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}