# Inferring Our Desires

Related: [Cached Selves](/lw/4e/cached_selves/), [The Neuroscience of Desire](/lw/4z7/the_neuroscience_of_desire/)

> You don't know your own mind.  
>     \- Jonathan Swift, _[Polite Conversation](http://www.amazon.com/conversation-dialogues-Jonathan-introduction-Saintsbury/dp/1176297252/)_

Researchers showed subjects two female faces for a few seconds and asked which face was more attractive. Researchers then placed the photos face down and handed subjects the face they had chosen, asking them to explain the motives behind their choice. But sometimes, researchers used a sleight-of-hand trick to switch the photos, showing viewers the face they had _not_ chosen. Very few subjects noticed the face they were given was not the one they had chosen. Moreover, they happily explained why they preferred the face they had actually rejected, inventing reasons like "I like her smile" even though they had _actually_ chosen the solemn-faced picture.^1^

The idea that we lack good introspective access to our own desires - that we often have no idea what we want^2^ \- is a key [lemma](http://dictionary.reference.com/browse/lemma) in [naturalistic metaethics](http://wiki.lesswrong.com/wiki/No-Nonsense_Metaethics), so it seems worth a post to collect the science by which we know that.

Early warnings came from split-brain research, which identified an 'interpreter' in the left hemisphere that invents reasons for beliefs and actions. When the command 'walk' was flashed to split-brain subjects' right hemispheres, they got up from their chairs and start walking away. When asked _why_ they suddenly started walking away, they replied (for example) that they got up because they wanted a Coke.^3^

####   

#### The overjustification effect

Common sense suggests that we infer others' feelings from their appearance and actions, but we have a different, more direct route to our _own_ feelings: direct perception or _introspection_.^4^ In contrast, [self-perception theory](http://en.wikipedia.org/wiki/Self-perception_theory)^5^ suggests that our knowledge of ourselves is exactly like our knowledge of others.^6^ One famous result explained by self-perception theory is the overjustification effect.

In a famous 1973 study,

> nursery school children drew pictures with a magic marker, a presumably intrinsically interesting activity, under one of three reward conditions. In the first condition the children expected to receive a reward (a fancy 'good player' award) for drawing, in the second they received the reward unexpectedly, and children in a third group received no reward. Only the expected reward produced a decrement in performance, during a later 'free play' period, as compared with the other two groups. \[This\] overjustification effect seemed to be due not to the reward itself but to the implication that the reward was the reason for the behavior. Only if the participants knew a reward was coming when they performed the behavior would it undermine their intrinsic interest in the task.^7^

It seems that subjects initially drew pictures because of intrinsic motivation in that activity, but the payment led them to unconsciously 'conclude' that their behavior did not represent their _actual _desires. Thus, they performed more poorly in the subsequent 'free play' period. This is known as the [overjustification effect](http://en.wikipedia.org/wiki/Overjustification_effect).

After dozens of similar studies, two meta-analyses confirmed that the overjustification effect occurs when (1) subjects are led to expect rewards before performing the behavior, (2) the rewards are tangible, and (3) the rewards are independent of the subjects’ level of performance.^8^

####   

#### Implicit motivation

If we can be wrong about our own desires, then presumably many of our desires are activated unconsciously and operate unconsciously. Such _implicit motivations_ have been amply confirmed.^9^

In one study, subjects were primed with achievement-related words ('strive', 'win', 'attain') during a word-finding task. During a second word-finding task, subjects were interrupted by an intercom announcement asking them to stop working. Those who had been primed with achievement-related words kept working more often than those who had not been so primed. Subjects were unable to identify the effect of this priming on their own motivation.^10^

This demonstrates that priming unconsciously affects the accessibility or strength of existing goals.^11^ Do we unconsciously _form_ goals, too?

We do, as shown by decades of research on [operant conditioning](http://en.wikipedia.org/wiki/Operant_conditioning). When a neutral potential goal is associated with a stimulus of positive affect, we acquire new goals, and we can be unaware that this has happened:

> Watching someone smile while eating blueberry muffins may, for instance, link that activity to positive affect, which creates a goal representation. Indeed, such observational or social learning is thought to be a basic way in which infants learn which behavioral states are desired and which ones are not.^12^

####   

#### Implications

This research is how we know about the [hidden complexity of wishes](/lw/ld/the_hidden_complexity_of_wishes/), a key lemma in the [fragility of human value](/lw/y3/value_is_fragile/). We don't know what many of our desires are, we don't know where they come from, and we can be wrong about our own motivations.

As such, we'd be unlikely to get what we _really_ want if the world was [re-engineered](http://en.wikipedia.org/wiki/Friendly_artificial_intelligence) in accordance with a description of what we want that came from verbal introspective access to our motivations. [Less naive proposals](http://intelligence.org/upload/CEV.html) would involve probing the [neuroscience of motivation](/lw/4z7/the_neuroscience_of_desire/) at the [algorithmic level](http://en.wikipedia.org/wiki/David_Marr_(neuroscientist)#Levels_of_analysis).^13^

#### Notes

^1^ Johansson et al. (2005).

^2^ Several experiments have established that we infer rather than perceive the moment we decided to act: Rigoni et al (2010); Banks & Isham (2009, 2010); Moore & Haggard (2008); Sarrazin et al. (2008); Gomes (1998, 2002). But do not infer that conscious thoughts do not affect behavior. As on recent review put it: "The evidence for conscious causation of behavior is profound, extensive, adaptive, multifaceted, and empirically strong. However, conscious causation is often indirect and delayed, and it depends on interplay with unconscious processes. Consciousness seems especially useful for enabling behavior to be shaped by nonpresent factors and by social and cultural information, as well as for dealing with multiple competing options or impulses" (Baumeister et al. 2011). We can even be wrong about _whether_ we intended to act at all: Lynn et al. (2010); Morsella et al (2010). If we don't have direct introspective access even to our decisions to act, why think we have introspective access to our _desires_?

^3^ Gazzaniga (1992), pp. 124-126.

^4^ But widespread findings of self-ignorancechallenge this view. See, for example, Wilson (2004).

^5^ Zanna & Cooper (1974) seemed to have disproved self-perception theory in favor of cognitive dissonance theory, but Fazio et al (1977) showed that the two co-exist. This remains the modern view.

^6^ Laird (2007), p. 7.

^7^ Laird (2007), p. 126. The study described is Lepper et al. (1973).

^8^ Cameron & Pierce (1994); Tang & Hall (1995); Eisenberger & Cameron (1996).

^9^ Aarts & Dijksterhuis (2000); Bargh (1990); Bargh & Gollwitzer (1994); Chartrand & Bargh (1996, 2002); Fishbach et al. (2003); Fitzsimons & Bargh (2003); Glaser & Kihlstrom (2005); Gollwitzer et al. (2005); Hassin (2005); Shah (2003). For reviews, see Ferguson et al. (2007); Kruglanski & Kopetz (2008); Moskowitz et al. (2004). Unconscious motivations can even adapt to novel and changing circumstances: see Ferguson et al. (2007), pp. 155-157.

^10^ Bargh et al. (2001).

^11^ Shah (2003); Aarts et al. (2004).

^12^ Custers (2009).

^13^ Inferring desires from behavior alone probably won't work, either: Soraker & Brey (2007). Also: My thanks to Eliezer Yudkowsky for his feedback on a draft of this post.

#### References

Aarts & Dijksterhuis (2000). [Habits as knowledge structures: Automaticity in goal-directed behavior](http://goallab.nl/publications/documents/Aarts,%20Dijksterhuis%20(2000)%20-%20habits%20as%20knowlegde%20structures.pdf). _Journal of Personality and Social Psychology, 78_: 53–63.

Aarts, Gollwitzer, & Hassin (2004). [Goal contagion: Perceiving is for pursuing](http://commonsenseatheism.com/wp-content/uploads/2011/02/Aarts-Goal-contagion-Inferring-goals-from-others-actions.pdf). _Journal of Personality and Social Psychology, 87_: 23–37.

Bargh (1990). Auto-motives: Preconscious determinants of social interaction. In Higgins & Sorrentino (eds.), _Handbook of motivation and cognition: Foundations of social behavior_ (Vol. 2, pp. 93–130). Guilford.

Bargh & Gollwitzer (1994). Environmental control of goal-directed action: Automatic and strategic contingencies between situations and behavior. In Spaulding (ed.), _Nebraska Symposium on Motivation_ (Vol. 41, pp. 71–124). University of Nebraska Press.

Bargh, Gollwitzer, Lee-Chai, Barndollar, & Troetschel (2001). [The automated will: Nonconscious activation and pursuit of behavioral goals](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3005626/pdf/nihms256255.pdf). _Journal of Personality and Social Psychology, 81_: 1014–1027.

Baumeister, Masicampo, & Vohs (2011). [Do conscious thoughts cause behavior?](http://commonsenseatheism.com/wp-content/uploads/2011/05/Baumeister-et-al-Do-conscious-thoughts-cause-behavior.pdf) _Annual Review of Psychology, 62_: 331-361.

Banks & Isham (2009). [We infer rather than perceive the moment we decided to act](http://commonsenseatheism.com/wp-content/uploads/2010/12/Banks-Isham-We-infer-rather-than-perceive-the-moment-we-decided-to-act.pdf). _Psychological Science, 20_: 17–21.

Banks & Isham (2010). [Do we really know what we are doing? Implications of reported time of decision for theories of volition](http://books.google.com/books?hl=en&lr=&id=FMPkA08Cjq8C&oi=fnd&pg=PA47&ots=eRBTpd3GnZ&sig=tm-serEd27xIK7xlkxxXnEl3AyM#v=onepage&q&f=false). In Sinnott-Armstrong & Nadel (eds.), _Conscious Will and Responsibility: A Tribute to Benjamin Libet_ (pp. 47-60).

Cameron & Pierce (1994). Reinforcement, reward and intrinsic motivation: A meta-analysis. _Review of Educational Research, 64_: 363–423.

Chartrand & Bargh (1996). [Automatic activation of impression formation and memorization goals: Nonconscious goal priming reproduces effects of explicit task instructions](http://www.yale.edu/acmelab/articles/chartrand_bargh_1996.pdf). _Journal of Personality and Social Psychology, 71_: 464–478.

Chartrand & Bargh (2002). Nonconscious motivations: Their activation, operation, and consequences. In Tesser & Stapel (eds.), _Self and motivation: Emerging psychological perspectives_ (pp. 13–41). American Psychological Association.

Custers (2009). How does our unconscious know what we want? The role of affect in goal representations. In Moskowitz & Grant (eds.), The Psychology of Goals. Guilford.

Eisenberger & Cameron (1996). [Detrimental effects of reward: Reality or myth?](http://www.psychology.uh.edu/faculty/Eisenberger/files/16_Detrimental_Effects_of_Reward_Reality_or_Myth.pdf) _American Psychologist, 51_: 1153–1166.

Fazio, Zanna, & Cooper (1977). Dissonance and self-perception: An integrative view of each theory's proper domain of application. _Journal of Experimental Social Psychology, 13_: 464-479.

Ferguson, Hassin, & Bargh (2007). Implicit Motivation. In Shah & Gardner (eds.), _Handbook of Motivation Science_ (pp. 150-166). Guilford.

Fishbach, Friedman, & Kruglanski (2003). [Leading us not unto temptation: Momentary allurements elicit automatic goal activation](http://commonsenseatheism.com/wp-content/uploads/2011/05/Fishbach-Leading-us-not-unto-temptation-Momentary-allurements-elicit-automatic-goal-activation.pdf). _Journal of Personality and Social Psychology, 84_: 296–309.

Fitzsimons & Bargh (2003). [Thinking of you: Nonconscious pursuit of interpersonal goals associated with relationship partners](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3011819/pdf/nihms256261.pdf). _Journal of Personality and Social Psychology, 84_: 148–163.

Gazzaniga (1992). _[Nature's mind: The biological roots of thinking, emotion, sexuality, language, and intelligence](http://www.amazon.com/Natures-Mind-Biological-Sexuality-Intelligence/dp/0465048633/)_. Basic Books.

Glaser & Kihlstrom (2005). Compensatory automaticity: Unconscious volition is not an oxymoron. In Hassin, Uleman, & Bargh (eds.), _The new unconscious_ (pp. 171–195). Oxford University Press.

Gollwitzer, Bayer, & McCullouch (2005). The control of the unwanted. In Hassin, Uleman, & Bargh (eds.), _The new unconscious_ (pp. 485–515). Oxford University Press.

Gomes (1998). The timing of conscious experience: a critical review and reinterpretation of Libet’s research. _Consciousness and Cognition, 7:_ 559–595.

Gomes (2002). [Problems in the timing of conscious experience](http://commonsenseatheism.com/wp-content/uploads/2011/05/Gomes-Problems-in-the-timing-of-conscious-experience.pdf). _Consciousness and Cognition, 11_: 191–97.

Hassin (2005). Non-conscious control and implicit working memory. In Hassin, Uleman, & Bargh (eds.), The new unconscious (pp. 196–224). Oxford University press.

Johansson, Hall, Silkstrom, & Olsson (2005). [Failure to detect mismatches between intention and outcome in a simple decision task](http://thebrowser.com/files/Mismatches-between-intention-and-outcome.pdf). _Science, 310_: 116-119.

Kruglanski & Kopetz (2008). The role of goal systems in self-regulation. InMorsella, Bargh, & Gollwitzer (eds.), _Oxford Handbook of Human Action_ (pp. 350-369). Oxford University Press.

Laird (2007). _[Feelings: The Perception of Self](http://www.amazon.com/Feelings-Perception-Self-Affective-Science/dp/0195098897/)_. Oxford University Press.

Lepper, Green, & Nisbett (1973). [Undermining children’s intrinsic interest with extrinsic rewards: A test of the 'overjustification' hypothesis](http://fitaba.com/page16/assets/Overjustification%20Study%20-%20Lepper.pdf). _Journal of Personality and Social Psychology, 28_: 129–137.

Lynn, Berger, Riddle, & Morsella (2010). [Mind control? Creating illusory intentions through a phony brain–computer interface](http://commonsenseatheism.com/wp-content/uploads/2011/05/Lynn-et-al-Mind-control-Creating-illusory-intentions-through-a-phony-brain–computer-interface.pdf). _Consciousness and Cognition, 19_: 1007-1012.

Moore & Haggard (2008). [Awareness of action: inference and prediction](http://commonsenseatheism.com/wp-content/uploads/2011/05/Moore-Haggard-Awareness-of-action-inference-and-prediction.pdf). _Consciousness and Cognition, 17_: 136–144.

Morsella, Berger, & Krieger (2010). [Cognitive and neural components of the phenomenology of agency](http://commonsenseatheism.com/wp-content/uploads/2011/05/Morsella-et-al-Cognitive-and-neural-components-of-the-phenomenology-of-agency.pdf). _Neurocase._

_Moskowitz, Li, & Kirk (2004). _The implicit volition model: On the preconscious regulation of temporarily adopted goals. In Zanna (ed.), _Advances in experimental social psychology_ (Vol. 36, pp. 317–404). Academic Press.

Rigoni, Brass, & Sartori (2010). [Post-action determinants of the reported time of conscious intentions](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2876876/pdf/fnhum-04-00038.pdf). _Frontiers in Human Neuroscience, 4_: 38.

Sarrazin, Cleeremans, & Haggard (2008). [How do we know what we are doing? Time, intention, and awareness of action](http://commonsenseatheism.com/wp-content/uploads/2011/05/Sarrazin-et-al-How-do-we-know-what-we-are-doing-Time-intention-and-awareness-of-action.pdf). _Consciousness and cognition, 17_: 602–615.

Shah (2003). The motivational looking glass: How significant others implicitly affect goal appraisals. _Journal of Personality and Social Psychology, 85_: 424–439.

Soraker & Brey (2007). [Ambient Intelligence and Problems with Inferring Desires from Behaviour](http://commonsenseatheism.com/wp-content/uploads/2011/05/Soraker-Brey-Ambient-intelligence-and-problems-with-inferring-desires-from-behavior.pdf). _International Review of Information Ethics, 8_: 7-12.

Tang & Hall (1995). The overjustification effect: A meta-analysis. _Applied Cognitive Psychology, 9_: 365–404.

Wilson (2004). _[Strangers to Ourselves: Discovering the Adaptive Unconscious](http://www.amazon.com/Strangers-Ourselves-Discovering-Adaptive-Unconscious/dp/0674013824/)_. Belknap.

Zanna & Cooper (1974). Dissonance and the pill: An attribution approach to studying the arousal properties of dissonance. _Journal of Personality and Social Psychology, 29: 703-709_.