# A brief tutorial on preferences in AI

Preferences are important both [for rationality](/lw/8q8/urges_vs_goals_the_analogy_to_anticipation_and/) and [for Friendly AI](http://commonsenseatheism.com/wp-content/uploads/2011/11/Muehlhauser-Helm-The-Singularity-and-Machine-Ethics-draft.pdf), so preferences are a [major](http://wiki.lesswrong.com/wiki/Preference) [topic](/lw/xy/the_fun_theory_sequence/) [of](/lw/lq/fake_utility_functions/) [discussion](/lw/l3/thou_art_godshatter/) [on](/lw/lb/not_for_the_sake_of_happiness_alone/) [Less](/lw/71x/a_crash_course_in_the_neuroscience_of_human/) [Wrong](/lw/9jh/the_humans_hidden_utility_function_maybe/). We've discussed preferences in the context of economics and decision theory, but I think AI has a more robust set of tools for working with preferences than either economics or decision theory has, so I'd like to introduce Less Wrong to some of these tools. In particular, I think AI's toolset for working with preferences may help us think more clearly about [CEV](http://intelligence.org/upload/CEV.html).

In AI, we can think of working with preferences in four steps:

1.  **Preference acquisition**: In this step, we aim to extract preferences from a user. This can occur either by _preference learning_ or by _preference elicitation_. Preference learning occurs when preferences are acquired from data about the user's past behavior or past preferences. Preference elicitation occurs as a result of an interactive process with the user, e.g. a question-answer process.
2.  **Preferences modeling**: Our next step is to mathematically express these acquired preferences as preferences between pairwise choices. The properties of a preferences model are important. For example, is the relation _transitive_? (If the model tells us that choice c~1~ is preferred to c~2~, and c2 is preferred to c~3~, can we conclude that c~1~ is preferred to c~3~?) And is the relation _complete_? (Is any choice comparable to any other choice, or are there some incomparabilities?)
3.  **Preference representation**: Assuming we want to capture and manipulate the user's preferences robustly, we'll next want to represent the preferences model in a _preference representation language_.
4.  **Preferences reasoning**: Once a user's preferences are represented in a preference representation language, we can do cool things like _preferences aggregation_ (involving the preferences of multiple agents) and _preference revision_ (a user's new preferences being added to her old preferences). We can also perform the usual computations of decision theory, game theory, and more.

### Preference acquisition

Preference learning is typically an application of [supervised machine learning](http://en.wikipedia.org/wiki/Supervised_learning) (classification). Throw the algorithm at a database containing a user's preferences, and it will learn that user's preferences and make predictions about the preferences not listed in the database, including preferences about pairwise choices the user may never have faced before.

Preference elicitation involves asking a user a series of questions, and extracting their preferences from the answers they give. [Chen & Pu (2004)](http://hci.epfl.ch/members/lichen/IC_TECH_REPORT_200467.pdf) survey some of the methods used for this.

In studying CEV, I am interested in methods built for learning a user's utility function from inconsistent behavior (because [humans make inconsistent choices](/lw/6da/do_humans_want_things/)). [Nielsen & Jensen (2004)](http://people.cs.aau.dk/~tdn/papers/nielsen-jensen-04-ai.pdf) provided two computationally tractable algorithms which handle the problem by interpreting inconsistent behavior as random deviations from an underlying "true" utility function. As far as I know, however, nobody in AI has tried to solve the problem with an algorithm informed by the [latest data from neuroeconomics](/lw/9jh/the_humans_hidden_utility_function_maybe/) on how human choice is the product of at least three valuation systems, only one of which looks anything like an "underlying true utility function."

### Preference Modeling

A model of a user's preferences describes one of three relations between any two choices ("objects"): a _strict preference_ relation which says that one choice is preferred to another, an _indifference_ relation, and an _incomparability_ relation. Kaci (2011), chapter 2 provides a brief account of preference modeling.

### Preference Representation

In decision theory, a preference relation is represented by a numerical function with associates a utility value with each choice. But this may not be the best representation. We face an exponential number of choices whose explicit enumeration and evaluation is time-consuming. Moreover, users can't compare all pairwise choices and evaluate how satisfactory each choice is.

Luckily, choices are often made on the basis of a set of attributes, e.g. cost, color, price, etc. You can use a preference representation language to represent partial descriptions of preferences and rank-order possible choices. The challenge of a preference representation language is that it should (1) cope with a user's preferences, (2) faithfully represent the user's preferences such that it rank-orders choices in a way similar to how the user would specify choices if they were able to provide preferences for every pairwise comparison, (3) cope with possibly inconsistent preferences, and (4) offer attractive complexity properties, i.e. the spatial cost of representing partial descriptions of preferences and the time cost of comparing pairwise choices or computing the best choices.

One popular method of preference representation is with the graphical representation language of _conditional preference networks_ or "CP-nets." They look like [this](http://commonsenseatheism.com/wp-content/uploads/2012/02/CP-net.gif).

### Preferences Reasoning

There are a multitude of ways in which one might want to reason algorithmically about preferences. I point the reader to Part II of Kaci (2011) for a very incomplete overview.

### General Sources:

Domshlak et al. (2011). [Preferences in AI: An Overview](http://commonsenseatheism.com/wp-content/uploads/2011/08/Domshlak-et-al-Preferences-in-AI-an-overview.pdf). _Artificial Intelligence 175_: 1037-1052.

Fürnkranz & Hüllermeier (2010). _[Preference Learning](http://www.amazon.com/Preference-Learning-Johannes-F%C3%BCrnkranz/dp/3642141242/)_. Springer.

Kaci (2011). _[Working with Preferences: Less is More](http://www.amazon.com/Working-Preferences-Less-Cognitive-Technologies/dp/3642172792/)_. Springer.