Uncertain Decisions Facilitate Better Preference Learning

Existing observational approaches for learning human preferences , such as inverse reinforcement learning , usually make strong assumptions about the observability of the human ’ s environment . However , in reality , people make many important decisions under uncertainty . To better understand preference learning in these cases , we study the setting of inverse decision theory ( IDT ) , a previously proposed framework where a human is observed making non-sequential binary decisions under uncertainty . In IDT , the human ’ s preferences are conveyed through their loss function , which expresses a tradeoff between different types of mistakes . We give the first statistical analysis of IDT , providing conditions necessary to identify these preferences and characterizing the sample complexity—the number of decisions that must be observed to learn the tradeoff the human is making to a desired precision . Interestingly , we show that it is actually easier to identify preferences when the decision problem is more uncertain . Furthermore , uncertain decision problems allow us to relax the unrealistic assumption that the human is an optimal decision maker but still identify their exact preferences ; we give sample complexities in this suboptimal case as well . Our analysis contradicts the intuition that partial observability should make preference learning more difficult . It also provides a first step towards understanding and improving preference learning methods for uncertain and suboptimal humans . 1 Introduction . The problem of inferring human preferences has been studied for decades in fields such as inverse reinforcement learning ( IRL ) , preference elicitation , and active learning . However , there are still several shortcomings in existing methods for preference learning . Active learning methods require query access to a human ; this is infeasible in many purely observational settings and may lead to inaccuracies due to the description-experience gap [ 1 ] . IRL is an alternative preference learning tool which requires only observations of human behavior . However , IRL suffers from underspecification , i.e . preferences are not precisely identifiable from observed behavior [ 2 ] . Furthermore , nearly all IRL methods require that the observed human is optimal or noisily optimal at optimizing for their preferences . However , humans are often systematically suboptimal decision makers [ 3 ] , and accounting for this makes IRL even more underspecified , since it is hard to tell suboptimal behavior for one set of preferences apart from optimal behavior for another set of preferences [ 4 ] . IRL and preference learning from observational data are generally applied in situations where a human is acting under no uncertainty . Given the underspecification challenge , one might expect that adding in the possibility of uncertainty in decision making ( known as partial observability ) would only make preference learning more challenging . Indeed , Choi and Kim [ 5 ] and Chinaei and Chaib-Draa [ 6 ] , who worked to apply IRL to partially observable Markov decision processes ( POMDPs , where agents 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . act under uncertainty ) , remarked that the underspecification of IRL combined with the intractability of POMDPs made for a very difficult task . In this work , we find that , surprisingly , observing humans making decisions under uncertainty actually makes preference learning easier ( see Figure 1 ) . To show this , we analyze a simple setting , where a human decision maker observes some information and must make a binary choice . This is somewhat analogous to supervised learning , where a decision rule is chosen to minimize some loss function over a data distribution . In our formulation , the goal is to learn the human decision maker ’ s loss function by observing their decisions . Often , in supervised learning , the loss function is simply the 0-1 loss . However , humans may incorporate many other factors into their implicit “ loss functions ” ; they may weight different types of mistakes unequally or incorporate fairness constraints , for instance . One might call this setting “ inverse supervised learning , ” but it is better described as inverse decision theory ( IDT ) [ 7 , 8 ] , since the objective is to reverse-engineer only the human ’ s decision rule and not any learning process used to arrive at it . IDT can be shown to be a special case of partially observable IRL ( see Appendix B ) but its restricted assumptions allow more analysis than would be possible for IRL in arbitrary POMDPs . However , we believe that the insights we gain from studying IDT should be applicable to POMDPs and uncertain decision making settings in general . We introduce a formal description of IDT in Section 3 . While we hope to provide insight into general reward learning , IDT is also a useful tool in its own right ; even in this binary , non-sequential setting , human decisions can reveal important preferences . For example , during a deadly disease outbreak , a government might pass a law to quarantine individuals with a chance of being sick . The decision rule the government uses to choose who to quarantine depends on the relative costs of failing to quarantine a sick person versus accidentally quarantining an uninfected one . In this way , even human decisions where there is a “ right ” answer are revealing if they are made under uncertainty . This example could distinguish a preference for saving lives versus one for guaranteeing freedom of movement . These preferences on the tradeoff between costs of mistakes are expressed through the loss function that the decision maker optimizes . In our main results on IDT in Section 4 , we find that the identifiability of a human ’ s loss function is dependent on whether the decision we observe them making involves uncertainty . If the human faces sufficient uncertainty , we give tight sample complexity bounds on the number of decisions we must observe to identify their loss function , and thus preferences , to any desired precision ( Theorem 4.2 ) . On the other hand , if there is no uncertainty—i.e. , the correct decision is always obvious—then we show that there is no way to identify the loss function ( Theorem 4.11 and Corollary 4.12 ) . Technically , we show that learning the loss function is equivalent to identifying a threshold function over the space of posterior probabilities for which decision is correct given an observation ( Figure 2 ) . This threshold can be determined to precision in Θ ( 1/ ( pc ) ) samples , where pc is the probability density of posterior probabilities around the threshold . In the case where there is no uncertainty in the decision problem , pc = 0 and we demonstrate that the loss function can not be identified . These results apply to optimal human decision makers—that is , those who completely minimize their expected loss . When a decision rule or policy is suboptimal , in general their loss function can not be learned [ 4 , 9 ] . However , we show that decisions made under uncertainty are also helpful in this case ; under certain models of suboptimality , we can still exactly recover the human ’ s loss function . We present two such models of suboptimality ( see Figure 3 ) . In both , we assume that the decision maker is restricting themselves to choosing a decision rule h in some hypothesis classH , which may not include the optimal decision rule . This framework is similar to that of agnostic supervised learning [ 10 , 11 ] , but solves the inverse problem of determining the loss function given a hypothesis class and decision samples . If the restricted hypothesis classH is known , we show that the loss function can be learned similarly to the optimal case ( Theorem 4.7 ) . Our analysis makes a novel connection between Bayesian posterior probabilities and binary hypothesis classes . However , assuming thatH is known is a strong assumption ; for instance , we might suspect that a decision maker is ignoring some data features but we may not know exactly which features . We formalize this case by assuming that the decision maker could be considering the optimal decision rule in any of a number of hypothesis classes in some family H. This case is more challenging because we may need to identify which hypothesis class the human is using in order to identify their loss function . We show that , assuming a smoothness condition on H , we can still obtain the decision maker ’ s loss function ( Theorem 4.10 ) . We conclude with a discussion of our results and their implications in Section 5 . We extend IDT to more complex loss functions that can depend on certain attributes of the data in addition to the chosen decision ; we show that this extension can be used to test for the fairness of a decision rule under certain criteria which were previously difficult to measure . We also compare the implications of IDT for preference learning in uncertain versus clear decision problems . Our work shows that uncertainty is helpful for preference learning and suggests how to exploit this fact . 2 Related Work . Our work builds upon that of Davies [ 8 ] and Swartz et al . [ 7 ] , who first introduced inverse decision theory . They describe how to apply IDT to settings in which a doctor makes treatment decisions based on a few binary test outcomes , but provide no statistical analysis . In contrast , we explore when IDT can be expected to succeed in more general cases and how many observed decisions are necessary to infer the loss function . We also analyze cases where the decision maker is suboptimal for their loss function , which are not considered by Davies or Swartz et al . Inverse reinforcement learning ( IRL ) [ 2 , 12 , 13 , 14 , 15 ] , also known as inverse optimal control , aims to infer the reward function for an agent acting in a Markov decision process ( MDP ) . Our formulation of IDT can be considered as a special case of IRL in a partially observable MDP ( POMDP ) with two states and two actions ( see Appendix B ) . Some prior work explored IRL in POMDPs [ 5 , 6 ] by reducing the POMDP to a belief-state MDP and applying standard IRL algorithms . Our main purpose is not to present improvements to IRL algorithms ; rather , we give an analysis of the difference between observable and partially observable settings for preference learning . We begin with the restricted setting of IDT but hope to extend to sequential decision making in the future . We also consider cases where the human decision maker is suboptimal , which previous work did not explore . Performance metric elicitation ( ME ) aims to learn a loss function ( aka performance metric ) by querying a human [ 16 , 17 , 18 ] . ME and other active learning approaches [ 19 , 20 , 21 , 22 ] require the ability to actively ask a user for their preference among different loss or reward functions . In contrast , IDT aims to learn the loss function purely by observing a decision maker . Active learning is valuable for some applications , but there are many cases where it is infeasible . Observed decisions are often easier to obtain than expert feedback . Also , active learning may suffer from the description-experience gap [ 1 ] ; that is , it may be difficult to evaluate in the abstract the comparisons that these methods give as queries to the user , leading to biased results . In contrast , observing human decision making “ in the wild ” with IDT could lead to a more accurate understanding of human preferences . Preference and risk elicitation aim to identify people ’ s preferences between different uncertain or certain choices . A common tool is to ask a person to choose between a lottery ( i.e. , uncertain payoff ) and a guaranteed payoff , or between two lotteries , varying parameters and observing the resulting choices [ 23 , 24 , 25 ] . In our analysis of IDT , decision making under uncertainty can be cast as a natural series of choices between lotteries . If we observe enough different lotteries , the decision maker ’ s preferences can be identified . On the other hand , if there is no uncertainty , then we only observe choices between guaranteed payoffs and there is little information to characterize preferences .

Summary: The authors study the problem of learning the preferences of a human decision-maker from past decisions. The setting is that feature–decision pairs (X, Y) are drawn from a known distribution $\mathcal{D}$. Initially the authors assume the Bayes-optimal case, where the decision-maker perfectly knows P(Y | X), and the objective is to identify the decision boundary separating positive labels from negative labels in terms of P(Y | X) conditional belief. The authors take a PAC approach, give bounds on the number of samples required to identify the boundary with probability $1 - \delta$. They later relax the Bayes-optimal rationality assumption, considering the case of a limited hypothesis class and the case of a family of hypothesis classes.