Cognition 143 (2015) 141–162

Contents lists available at ScienceDirect

Cognition
journal homepage: www.elsevier.com/locate/COGNIT

Affective cognition: Exploring lay theories of emotion
Desmond C. Ong ⇑, Jamil Zaki, Noah D. Goodman
Department of Psychology, Stanford University, United Statesa r t i c l e

i n f oArticle history:
Received 19 April 2015
Revised 12 June 2015
Accepted 19 June 2015
Available online 7 July 2015
Keywords:
EmotionInference
Lay theories
Bayesian models
Emotion perception
Cue integrationa b s t r a c t
Humans skillfully reason about others’ emotions, a phenomenon we term affective cognition. Despite its
importance, few formal, quantitative theories have described the mechanisms supporting this phenomenon. We propose that affective cognition involves applying domain-general reasoning processes
to domain-speciﬁc content knowledge. Observers’ knowledge about emotions is represented in richand coherent lay theories, which comprise consistent relationships between situations, emotions, andbehaviors. Observers utilize this knowledge in deciphering social agents’ behavior and signals (e.g., facial
expressions), in a manner similar to rational inference in other domains. We construct a computationalmodel of a lay theory of emotion, drawing on tools from Bayesian statistics, and test this model acrossfour experiments in which observers drew inferences about others’ emotions in a simple gambling paradigm. This work makes two main contributions. First, the model accurately captures observers’ ﬂexible
but consistent reasoning about the ways that events and others’ emotional responses to those events
relate to each other. Second, our work models the problem of emotional cue integration—reasoning aboutothers’ emotion from multiple emotional cues—as rational inference via Bayes’ rule, and we show that
this model tightly tracks human observers’ empirical judgments. Our results reveal a deep structural relationship between affective cognition and other forms of inference, and suggest wide-ranging applications
to basic psychological theory and psychiatry.
Ó 2015 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND
license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

1. Introduction
It is easy to predict that people generally react positively tosome events (winning the lottery) and negatively to others (losing
their job). Conversely, one can infer, upon encountering a crying
friend, that it is more likely he has just experienced a negative,
not positive, event. These inferences are examples of reasoning
about another’s emotions: a vital and nearly ubiquitous humanskill. This ability to reason about emotions supports countlesssocial behaviors, from maintaining healthy relationships to scheming for political power. Although it is possible that some features of
emotional life carries on with minimal inﬂuence from cognition,
reasoning about others’ emotions is clearly an aspect of cognition.
We propose terming this phenomenon affective cognition—the collection of cognitive processes that involve reasoning about emotion.
For decades, scientists have examined how people manage tomake complex and accurate attributions about others’ psychological states (e.g., Gilbert, 1998; Tomasello, Carpenter, Call, Behne, &Moll, 2005; Zaki & Ochsner, 2011). Much of this work converges⇑ Corresponding author at: Department of Psychology, Stanford University,
Stanford, CA 94305, United States.
E-mail address: dco@stanford.edu (D.C. Ong).

on the idea that individuals have lay theories about how others
react to the world around them (Flavell, 1999; Gopnik &Wellman, 1992; Heider, 1958; Leslie, Friedman, & German, 2004;
Pinker, 1999). Lay theories—sometimes called intuitive theories
or folk theories—comprise structured knowledge about the world(Gopnik & Meltzoff, 1997; Murphy & Medin, 1985; Wellman &Gelman, 1992). They provide an abstract framework for reasoning,and enable both explanations of past occurrences and predictionsof future events. In that sense, lay theories are similar to scientiﬁctheories—both types of theories are coherent descriptions of how
the world works. Just as a scientist uses a scientiﬁc theory todescribe the world, a lay observer uses a lay theory to make senseof the world. For instance, people often conclude that if Sally was inanother room and did not see Andy switch her ball from the basket
to the box, then Sally would return to the room thinking that herball was still in the basket: Sally holds a false belief, where herbeliefs about the situation differs from reality (Baron-Cohen,
Leslie, & Frith, 1985). In existing models, this understanding ofothers’ internal states is understood as a theory that can be used
ﬂexibly and consistently to reason about other minds. In this paper,
we propose a model of how people likewise reason about others’
emotions using structured lay theories that allow complex
inferences.

http://dx.doi.org/10.1016/j.cognition.2015.06.010
0010-0277/Ó 2015 The Authors. Published by Elsevier B.V.
This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

142

D.C. Ong et al. / Cognition 143 (2015) 141–162

Within the realm of social cognition, lay theories compriseknowledge about how people’s behavior and mental states relate
to each other, and allow observers to reason about invisible but
important factors such as others’ personalities and traits (Chiu,
Hong, & Dweck, 1997; Heider, 1958; Jones & Nisbett, 1971; Ross,
1977; Ross & Nisbett, 1991), beliefs and attitudes (Kelley &Michela, 1980), and intentions (Jones & Davis, 1965; Kelley,
1973; Malle & Knobe, 1997). Crucially, lay theories allow social
inference to be described by more general principles of reasoning.
For example, Kelley (1973)’s Covariational Principle describes how
observers use statistical co-variations in observed behavior todetermine whether a person’s behavior reﬂects a feature of that
person (e.g., their preferences or personality) or a feature of the situation in which they ﬁnd themselves. There are many similar
instances of lay-theory based social cognition: Fig. 1 lists just several such examples, such as how lay theories of personality (e.g.,
Chiu et al., 1997), race (e.g., Jayaratne et al., 2006), and ‘‘theoriesof mind’’ (e.g., Gopnik & Wellman, 1992) inform judgments andinferences—not necessarily made consciously—about traits andmental states. Although lay theories in different domains containvastly different domain-speciﬁc content knowledge, the same common principles of reasoning—for example, statistical co-variation,
deduction, and induction—are domain-general, and can be applied
to these lay theories to enable social cognitive capabilities suchas inferences about traits or mental states.
Lay theories can be formalized using Bayesian statistics using
ideal observer models (Geisler, 2003). This approach has been usedsuccessfully to model a wide range of phenomena in vision, memory, decision-making (Geisler, 1989; Liu, Knill, & Kersten, 1995;
Shiffrin & Steyvers, 1997; Weiss, Simoncelli, & Adelson, 2002),
and, more recently, social cognition (e.g., Baker, Saxe, &Tenenbaum, 2009). An ideal observer analysis describes the optimal conclusions an observer would make given (i) the observed
evidence and (ii) the observer’s assumptions about the world.
Ideal observer models describe reasoning without making claims
as to the mechanism or process by which human observers drawthese conclusions (cf. Marr, 1982), and provide precise, quantitative hypotheses through which to explore human cognition.
We propose that affective cognition, too, can be understood asreasoning with a lay theory: that is, affective cognition comprises
domain-general cognitive processes applied to domain-speciﬁcknowledge about emotions (Fig. 1). Domain-speciﬁc knowledge
comprises the observers’ lay theory of emotion, and includes, forexample, beliefs about what emotions are, how they are caused,and how people behave in response to emotions. We propose that
this complex knowledge can be captured in a causal model, andthat observers use domain-general reasoning and inference processes to draw conclusions from this knowledge, similar to thoseused in perception and other domains. We make these ideas precise below by constructing an ideal observer model of emotional
reasoning: we describing the domain-speciﬁc knowledge in a statistical causal model, and the domain-general reasoning as anapplication of Bayesian inference.
1.1. Attributing emotional reactions
How does an observer infer that agents (the targets of affective
cognition) who spill a cup of coffee, miss the bus, or fall off a bicycle, likely feel similar (negative) emotions? One problem that any
model of affective cognition must deal with is the combinatorial
explosion of outcomes and emotional states that people can experience. It would be both inefﬁcient and impractical for observers tostore or retrieve knowledge about the likely affective consequencesof every possible situation. We hypothesize that people circumvent
this complexity by evaluating situations based on a smaller number of ‘‘active psychological ingredients’’ those situations contain.
For instance, many emotion-inducing situations share key common features (e.g., the attainment or nonattainment of goals) thatconsistently produce particular emotions (Barrett, Mesquita,
Ochsner, & Gross, 2007; Ellsworth & Scherer, 2003). An individual
in a situation can take advantage of this commonality by appraising
the situation along a small number of relevant appraisal dimensions: that is, reducing a situation to a low-dimensional set of
emotion-relevant features (Ortony, Clore, & Collins, 1988;
Schachter & Singer, 1962; Scherer, Schorr, & Johnstone, 2001;
Smith & Ellsworth, 1985; Smith & Lazarus, 1993).
We propose that observers similarly reduce others’ experience
to a small number of emotionally relevant features when engaging
in affective cognition. The examples above—spilling coffee, missing
the bus, and falling off a bicycle—could all be associated, forinstance, with unexpectedly losing something (e.g. coffee, time,and health). Note that the features relevant to the person’s actual
emotions (identiﬁed by appraisal theories) may not be identicalto the features used by the observer (which are part of the observer’s lay theory). The latter is our focus when studying affective
cognition. Thus, we will ﬁrst elucidate the situation features relevant for attributing emotion to another person. We operationalize
this in Experiment 1 by studying a simple family of scenarios—agambling game—and considering a variety of features such as
amount of money won, prediction error (the amount won relative
to the expected value of the wheel), and distance from a better orworse outcome.

Fig. 1. Lay theories within social cognition comprise domain-speciﬁc knowledge about behavior and mental states. Inferences about traits and beliefs occur when observers
apply domain-general reasoning processes to these lay theories. In an analogous fashion, we propose that affective cognition is domain-general reasoning over domainspeciﬁc knowledge in a lay theory of emotions.

D.C. Ong et al. / Cognition 143 (2015) 141–162

1.2. Reasoning from emotional reactions
A lay theory should support multiple inferences that are coherently related to each other: we can reason from a cause to its
effects, but also back from an effect to its cause, and so on. For
example, we can intuit that missing one’s bus makes one feelsad, and we can also reason, with some uncertainty, that thefrowning person waiting forlornly at a bus stop might have justmissed their bus. If affective cognition derives from a lay theory,
then it should allow observers to both infer unseen emotions basedon events, and also to infer the type of event that a social target has
experienced based on that person’s emotions. In the framework ofstatistical causal models, these two types of inference—from emotions to outcomes and from outcomes to emotions—should be
related using the rules of probability. In Experiment 2, we explicitly test this proposal: do people reason ﬂexibly back and forth
between emotions and the outcomes that cause them? Do forward
and reverse inferences cohere as predicted by Bayesian inference?

1.3. Integrating sources of emotional evidence
Domain-general reasoning should also explain more complex
affective cognition. For instance, observers often encounter multiple cues about a person’s emotions: They might witness anotherperson’s situation, but also the expression on the person’s face,
their body posture, or what they said. Sometimes these cues evenconﬂict—for instance, when an Olympic athlete cries after winning
the gold medal. This seems to be a pair of cues that individuallysuggest conﬂicting valence. A comprehensive theory of affective
cognition should address how observers translate this deluge ofdifferent information types into an inference, a process we callemotional cue integration (Zaki, 2013).
Prior work suggests two very different approaches that observers might take to emotional cue integration. On the one hand,the facial dominance hypothesis holds that facial expressions universally broadcast information about emotion to external observers (Darwin, 1872; Ekman, Friesen, & Ellsworth, 1982; Smith,
Cottrell, Gosselin, & Schyns, 2005; Tomkins, 1962; for more extensive reviews, see Matsumoto, Keltner, Shiota, O’Sullivan, & Frank,
2008; Russell, Bachorowski, & Fernández-Dols, 2003). This suggests that observers should draw primarily on facial cues in determining social agents’ emotions (Buck, 1994; Nakamura, Buck, &Kenny, 1990; Wallbott, 1988; Watson, 1972). On the other hand,contextual cues often appear to drive affective cognition evenwhen paired with facial expressions. For instance, observers often
rely on written descriptions of a situation (Carroll & Russell, 1996;
Goodenough & Tinker, 1931) body postures (Aviezer, Trope, &Todorov, 2012; Aviezer et al., 2008; Mondloch, 2012; Mondloch,
Horner, & Mian, 2013; Van den Stock, Righart, & de Gelder,2007), background scenery (Barrett & Kensinger, 2010; Barrett,Mesquita, & Gendron, 2011; Lindquist, Barrett, Bliss-Moreau, &Russell, 2006), and cultural norms (Masuda et al., 2008) when
deciding how agents feel.
Of course, both facial expressions and contextual cues inﬂuenceaffective cognition. It is also clear that neither type of cue ubiquitously ‘‘wins out,’’ or dominates inferences about others’ emotions.
An affective cognition approach suggests that observers shouldsolve emotional cue integration using domain-general inference
processes. There are many other settings—such as binocular vision(Knill, 2007) and multisensory perception (Alais & Burr, 2004;
Shams, Kamitani, & Shimojo, 2000; Welch & Warren, 1980)—that
require people to combine multiple cues into coherent representations. These ideal observer models assume that observers combine
cues in an optimal manner given their prior knowledge and uncertainty. In such models, sensory cue integration is modeled as

143

Bayesian inference (for recent reviews, see de Gelder & Bertelson,
2003; Ernst & Bülthoff, 2004; Kersten, Mamassian, & Yuille, 2004).
Our framework yields an approach to emotional cue integrationthat is analogous to cue integration in object perception: a rational
information integration process. Observers weigh available cues toan agent’s emotion (e.g., the agent’s facial expression, or the context the agent is in) and combine them using statistical principlesof Bayesian inference. This prediction naturally falls out of ourclaim that affective cognition resembles other types oftheory-driven inference, with domain-speciﬁc content knowledge:
the lay theory of emotion describes the statistical and causal relations between emotion and each cue; joint reasoning over thisstructure is described by domain-general inference processes.
We empirically test the predictions of this approach inExperiments 3 and 4. We aim to both extend the scope of our lay
theory model and resolve the current debate in emotion perception
by predicting how different cues are weighted as observers makeinferences about emotion.
1.4. Overview
We ﬁrst describe the components of our model and how it canbe used to compute inferences about others’ emotions. We formalize this model in the language of Bayesian modeling (Goodman &Tenenbaum, 2014; Goodman, Ullman, & Tenenbaum, 2011;
Grifﬁths, Kemp, & Tenenbaum, 2008). Speciﬁcally, we focus on
the ways that observers draw inferences about agents’ emotions
based on the situations and outcomes those agents experience. In
all our experiments, we restricted the types of situations thatagents experience to a simple gambling game. Although this paradigm does not capture many nuances of everyday affective cognition, its simplicity allowed us to quantitatively manipulate featuresof the situation and isolate situational features that best trackaffective cognition.
Experiment 1 sheds light on the process of inferring an agent’s
emotions given a situation, identifying a set of emotion-relevant
situation features that observers rely on to understand others’
affect. Experiment 2 tests the ﬂexibility of emotional lay theories,
by testing whether they also track observers’ reasoning about theoutcomes that agents’ likely encountered based on their emotions;
our model’s predictions tightly track human judgments.
We then expand the set of observable evidence that our model
considers, and describe how our model computes inferences frommultiple cues—emotional cue integration. Experiment 3 tests themodel against human judgments of emotions from both situation
outcomes and facial expressions; Experiment 4 replicates this withsituation outcomes and verbal utterances. In particular, we showthat the Bayesian model predicts human judgments accurately,
outperforming the baseline single cue dominance (e.g. facial orcontext dominance) models. Together, the results support theclaim that reasoning about emotion represents a coherent set ofinferences over a lay theory, similar to reasoning in other domainsof psychology.
Finally, we describe some limitations of our model, motivate
future work, and discuss the implications of an affective cognition
approach for emotion theory, lay theories in other domains, andreal-world applications.
2. Exploring the ﬂexible reasoning between outcomes andemotions
Our model of a lay theory of emotion is shown schematically in
Fig. 2. An observer (i.e. the reasoner) uses this lay theory to reason
about an agent (i.e. the target of the reasoning). There are
emotion-eliciting situations in the world, and the outcomes of144

D.C. Ong et al. / Cognition 143 (2015) 141–162

Fig. 2. Model of a lay theory that an observer could use during affective cognition.
Using the notation of Bayesian networks, we represent variables as circles, andcausal relations from a causal variable to its effect as arrows. Shaded variables
represent unobservable variables. Although causal ﬂows are unidirectional, as
indicated by the arrows, information can ﬂow the other way, as when making
inferences about upstream causes from downstream outcomes. In this model,
observers believe that situation outcomes cause an agent to feel an emotion, whichthen causes certain behavior such as speech, facial expressions, body language or
posture, and importantly, actions that potentially result in new outcomes and a
new emotion cycle. From the observable variables—which we call ‘‘cues’’—we caninfer the agent’s latent, or unobservable, emotion. Other mental states could be
added to this model. One such extension includes the agent’s motivational states orgoals, which would interact with the outcome of a situation to produce emotions;such goals would also inﬂuence the actions taken.

these situations, interacting with other mental states such as goals,
cause an agent to feel emotions. The agent’s emotions in turn produce external cues including facial expressions, body language,
and speech, as well as further actions. All of these variables, except
mental states such as emotion and goals, are potentially observable variables; in particular, emotion is a latent variable that is
unobservable because it is an internal state of the agent.
Each of these directed causal relationships can be represented
as a probability distribution. For example, we can write different
levels of happiness and anger given the outcome of winning the lottery as P(happy|won lottery) and P(angry|won lottery). In ourmodel, we represent the relationship between general outcomeso and emotions e as P(e|o). Similarly, the causal relationshipbetween an agent’s emotions and his resultant facial expressions
f can be written as P(f|e), and so forth.
As we discussed above, it would be impractical for observers tostore the affective consequences of every possible situation outcome (i.e., P(e|o) for every possible outcome o). We hypothesizethat observers reduce the multitude of possible outcomes into a
low-dimensional set of emotion-relevant features via, for example,
appraisal processes (e.g., Ortony et al., 1988). One potentially
important outcome feature is value with respect to the agent’s
goals. Indeed, a key characteristic of emotion concepts, as compared to non-emotion concepts, is their inherent relation to a psychological value system (Osgood, Suci, & Tannenbaum, 1957), thatis, representations of events’ positive or negative affective valence(Clore et al., 2001; Frijda, 1988). As economists and psychologists
have long known, people assess the value of events relative to their
expectations: winning $100 is exciting to the person who expected$50 but disappointing to the person who expected $200 (Carver &Scheier, 2004; Kahneman & Tversky, 1979). Deviations from anindividual’s expectation are commonly termed prediction errors,
and prediction errors in turn are robustly associated with the experience of positive and negative affect (e.g., Knutson, Taylor,
Kaufman, Peterson, & Glover, 2005). Responses to prediction errors
are not all equal, however; individuals tend to respond morestrongly to negative prediction errors, as compared to positive prediction errors of equal magnitude, a property commonly referred
to as loss aversion (Kahneman & Tversky, 1984). These value computations are basic and intuitively seem linked to emotion concepts, and we propose that they form an integral part ofobservers’ lay theory of emotion. Thus, we hypothesize thatreward, prediction error, and loss aversion constitute key outcomefeatures that observers will use to theorize about others’ emotions,and facilitate affective cognition. Other, less ‘‘rational’’ features
likely also inﬂuence affective cognition. Here we consider one suchfactor: the distance from a better (or worse) outcome, or how close
one came to achieving a better outcome. In Experiment 1 we
explore the situation features that parameterize P(e|o) in a simple
gambling scenario.
Although the lay theory we posit is composed of directed causal
relationships—signaled by arrows in Fig. 2—people can also draw‘‘reverse inferences’’ about causes based on the effects they observe(for instance, a wet front lawn offers evidence for the inference
that it has previously rained.). In addition to reasoning about emotions e given outcomes o, observers can also draw inferences about
the posterior probability of different outcomes o having occurred,given the emotion e. This posterior probability is written as
P(o|e) and is speciﬁed by Bayes’ Rule:PðojeÞ ¼

PðejoÞPðoÞ
PðeÞ

ð1Þ

where P(e) and P(o) represent the prior probabilities of emotion eand outcome o occurring, respectively. By way of example, imagine
that you walk into your friend’s room and ﬁnd her crying uncontrollably; you want to ﬁnd the outcome that made her sad—likely the owith the highest P(o|sad)—so you start considering possible candidate outcomes using your knowledge of your friend. She is a conscientious and hardworking student, and so if she fails a class exam,
she would be sad, i.e., P(sad|fail exam) is high. But, you also recall
that she is no longer taking classes, and so the prior probability offailing an exam is small, i.e., P(fail exam) is low. By combining those
two pieces of information, you can infer that your friend probably
did not fail an exam, i.e., P(fail exam|sad) is low, and so you canmove on and consider other possible outcomes. In Experiment 2
we consider whether people’s judgments from emotions to outcomes are predicted by the model of P(e|o) identiﬁed inExperiment 1 together with Bayes’ rule.
2.1. Experiment 1: Emotion from outcomes1
Our ﬁrst goal is to understand how P(e|o) relates to the featuresof a situation and outcome. We explore this in a simple gamblingdomain (Fig. 3) where we can parametrically vary a variety of outcome features, allowing us to understand the quantitative relationship between potential features and attributed emotions.
2.1.1. Participants
We recruited one hundred participants through Amazon’s
Mechanical Turk and paid them for completing the experiment.
All experiments reported in this paper were conducted according
to guidelines approved by the Institutional Review Board at
Stanford University.
2.1.2. Procedures
Participants played the role of observers and watched characters play a simple gambling game. These characters each spun awheel and won an amount of money depending on where thewheel landed (Fig. 3). On each trial, one character spun one wheel.
We pre-generated a total of 18 wheels. Each wheel comprised
three possible outcomes, all of which were non-negative. We systematically de-correlated the probabilities of the outcomes and thevalues of the outcomes of each wheel, allowing for separate
1
Materials, data, and code can be found at: http://www.github.com/desmond-ong/
affCog

D.C. Ong et al. / Cognition 143 (2015) 141–162

145

Fig. 3. Sample screenshots from Experiment 1. Participants see a character spin a wheel that lands, in this case, on $60. Participants then attribute emotions to the characteron eight separate Likert scales.

calculation of reward amount and expected value. We used a standard deﬁnition of expected value: the averaged reward that thecharacter expected to win. For each ‘‘wedge’’, or sector of thewheel, we take the reward amount on that sector, and multiply
it with the notational probability—based on the size—of that sector.
The expected value is the sum of these products over all the sectors. The ﬁnal correlation of the amount won with the expectedvalue was 0.3. The 18 wheels with 3 outcomes each resulted in(after discarding one to make a number that could be evenly
divided by total trials) 50 scenarios, where each scenario corresponds to a particular outcome on a particular wheel. On each trial,
the experimental program selected the scenario (the particularoutcome to be won) randomly from the total set of 50 scenarios,and not proportional to the sector size within a wheel.2 This wasa design choice to ensure that small sectors (with low notational
probability) were equally represented in the data. This fact was
not obvious to the participants, and we assume that participants
treated the wheel as being fairly spun. The exact position within
the sector where the spinner lands—for example, whether the spinner lands in the center of the sector, or near the edge of the sector, as
in the example shown in Fig. 3—was a real number in the interval (0,1) drawn from a uniform distribution.
Each participant completed 10 trials. On each trial, they saw a
stick ﬁgure character with a randomized male name (e.g. ‘‘Bob’’)
spin a wheel. The names of the characters were randomized on
every trial. After the result of the spin, participants had to rate
how the character feels, on a set of 9 point Likert scales. There wereeight different Likert scales for each of eight emotion adjectives:
happy, sad, angry, surprised, fearful, disgusted, content, and disappointed. The ﬁrst six were the classic ‘‘basic emotions’’ describedby Ekman and colleagues (e.g., Ekman et al., 1982). We added ‘‘content’’ and ‘‘disappointed’’ to capture emotion concepts related tocounterfactual comparisons with outcomes that could have, but
did not occur (Gilovich & Medvec, 1995; Sweeny & Vohs, 2012).
2
The alternative, which we did not do, would be to randomly choose a wheel withall its possible outcomes from a set of all possible wheels, and then uniformly choose
where it lands; each sector would then be sampled proportional to its notational
probability.

2.1.3. Forward regression model
We expected that affective cognition should depend on speciﬁcsituation features—especially those related to value computation—that often predict an agent’s emotions. As such, we chose several apriori features based on previous work in decision theory as model
regressors to predict observers’ affective judgments. It is worth
reiterating here that our prediction is not that these features actually affect the ways that agents feel; that point has been made by
decades of economic and psychological research. Rather, our prediction here is that observers spontaneously rely on these features
when inferring how others feel—in essence applying sophisticatedlay theories to affective cognition.
We predicted that reward, prediction error, and loss aversion
form key features of affective cognition. We operationalized theseas regressors representing the amount won by the agent in a trial(‘‘win’’), the prediction error (‘‘PE’’; the difference between what
the agent won and the expected value of the wheel), and the absolute value of the prediction error (‘‘|PE|’’), respectively. Using both
PE and |PE| in the same model allows the coefﬁcient on PE to differ
when PE is positive and negative, modeling loss aversion.3
We additionally evaluated several other a priori plausible regressors (none of which survived model selection below). People compare their results with salient possible counterfactuals (e.g. what
they ‘‘could have’’ won; Medvec, Madey, & Gilovich, 1995). To examine whether observers weigh such comparisons during affective
cognition, we computed—for each gamble—a score representing
regret (the difference between how much the agent won and the
maximum he could have won4; Loomes & Sugden, 1982) and relief(the difference between how much the agent won and the minimum
3
Note that researchers model loss aversion as the difference between the
coefﬁcient of PE when PE is negative and when PE is positive, often using a piecewisefunction. It is often modeled using the piecewise equations: y = a PE, for PE > 0, andy = b PE, for PE < 0. We chose to adopt a mathematically equivalent formulation using
both PE and |PE| as regressors in the same model: y = c PE + d |PE| across all values of
PE. We can easily show that a = c + d, and b = c  d.
4
Note that this ‘game-theoretic’ deﬁnition of regret (over outcomes given a
speciﬁed choice) is slightly distinct from most psychological deﬁnitions that involve
regret over choices. We chose not to call this variable disappointment (usuallycontrasted with regret), as that was one of the emotions we measured.

146

D.C. Ong et al. / Cognition 143 (2015) 141–162

he could have won). Finally, we drew on previous work on ‘‘luck’’ (e.g.
Teigen, 1996), where observers tend to attribute more ‘‘luck’’ to anagent who won when the chances of winning were low, even after
controlling for expected payoffs. To model whether observers accountfor this when attributing other emotions, we included a regressor toaccount for the probability of winning, i.e., the size of the sector thatthe wheel landed on. Since a probability is bounded in [0,1] and theother regressors took on much larger domains of values (e.g. win varied from 0 to 100), we used a logarithm to transform the probability tomake the values comparable to other regressors (‘‘logWinProb’’).
The ﬁnal regressor we included was a ‘‘near-miss’’ term tomodel agents’ affective reactions to outcomes as a function of theirdistance from other outcomes. Such counterfactual reasoning often
affects emotion inference. For instance, people reliably judge
someone to feel worse after missing a ﬂight by 5 min, as compared
to 30 min (Kahneman & Tversky, 1982). Near-misses ‘‘hurt’’ morewhen the actual outcome is close (e.g., in time) to an alternative,
better outcome. In our paradigm, since outcomes are determinedby how much the wheel spins, ‘‘closeness’’ can be operationalized
in terms of the angular distance between (i) where the wheel
landed (the actual outcome; as deﬁned by the pointer) and (ii)the boundary between the actual outcome and the closest sector.
We deﬁned a normalized distance, which ranged from 0 to 0.5,
with 0 being at the boundary edge, and 0.5 indicating the exact
center of the current sector. Near-misses have much greater
impact at smaller distances, so we took a reciprocal transform5(1/x) to introduce a non-linearity that favors smaller distances.
Finally, we scaled this term by the difference in payment amounts
from the current sector to the next-nearest sector, to weigh thenear-miss distance by the difference in utility in the two payoffs.
In total, we tested seven outcome variables: win, PE, |PE|, regret,relief, logWinProb and nearMiss. We ﬁt mixed-models predicting eachemotion using these regressors as ﬁxed effects, and added a random
intercept by subject. We performed model selection by conducting
backward stepwise regressions to choose the optimal subset ofregressors that predicted a majority of observers’ ratings of theagents’ emotions. This was done using the step function in the R package lmerTest. Subsequently, we used the optimal subset of regressors
as ﬁxed effects with the same random effect structure. Full details of
the model selection and results are given in Appendix A.

2.1.4. Results
Model selection (in Appendix A) revealed that participants’emotion ratings were signiﬁcantly predicted only by three of theseven regressors we initially proposed: amount won, the prediction
error (PE), and the absolute value of the prediction error (|PE|) (seealso Section 3.3 for a re-analysis with more data). Crucially, PE and|PE| account for signiﬁcant variance in emotion ratings after
accounting for amount won. This suggests that affective cognition
is remarkably consistent with economic and psychological models
of subjective utility. In particular, emotion inferences exhibitedreference-dependence—tracking prediction error in addition toamount won—and loss aversion—in that emotion inferences weremore strongly predicted by negative, as opposed to positive prediction error. These features suggest that lay observers spontaneously
use key features of prospect theory (Kahneman & Tversky, 1979,
1984) in reasoning about others’ emotions: a remarkable connection between formal and everyday theorizing. It is worth noting
as well that the signiﬁcant regressors for surprise followed aslightly different pattern from the rest of the other emotions,where the win probability, as well as regret and relief, seem justas important as the amount won, PE, and |PE|.
5

We tried other non-linear transforms such as exponential and log transforms,
which all performed comparably.

The aforementioned analysis suggests that amount won, PE, and|PE| are necessary to model emotion inferences in a gambling context and suggest a low dimensional structure for the situation features. Next, we explored the underlying dimensionality ofparticipants’ inferences about agents’ emotions via an a priori
planned Principal Component Analysis (PCA). Previous work on
judgments of facial and verbal emotions (e.g., Russell, 1980;
Schlosberg, 1954) and self-reported emotions (e.g., Kuppens,
Tuerlinckx, Russell, & Barrett, 2012; Watson & Tellegen, 1985) havesuggested a low-dimensional structure, and we planned this analysis to see if a similar low-dimensional structure might emerge inattributed emotions in our paradigm.
The ﬁrst principal component (PC) accounted for 59% of the variance in participants’ ratings along all 8 emotions, while the second
PC accounted for 16%; subsequent PCs individually explained lessthan 10% of the variance. The ﬁrst PC accounted for most of the variance in the emotion ratings, although the second PC accounted for a
far lower, but still noteworthy, amount of variance. Full details of
the PCA procedure and loading results are given in Appendix A.
Post-hoc exploratory analysis of the ﬁrst two PCs revealed thatthe ﬁrst PC positively tracked happiness and contentment, while
negatively tracking all negative emotions; by contrast, the second
PC positively tracked the intensity of both positive and negative
emotions (Fig. 4A). Interestingly, this connects with classic concepts of valence and arousal, respectively, which feature centrally
in emotion science6 (e.g., Kuppens et al., 2012; Russell, 1980;
Schlosberg, 1954). In particular, some theorists view emotional
valence as a crucial form of feedback to the agent: positively
valenced emotions like happiness signal a positive prediction
error—that the agent is doing better than expected—hence, positively reinforcing successful behavior. Conversely, negatively
valenced emotions could signal to the agent to change some behavior to achieve a better outcome (e.g., Carver & Scheier, 2004; Ortony
et al., 1988). In line with this, we ﬁnd that the ﬁrst PC (‘‘valence’’) of
emotions attributed by the observer correlated strongly with the PEof the situation (r = 0.737, 95% C.I. = [0.707; 0.764]). Additionally, we
ﬁnd that the second PC (‘‘arousal’’) correlated with |PE| (r = 0.280[0.222, 0.364]; Fig. 4B).
We started with a priori predictions for a low-dimensional summary of outcome features, and followed up with a post hoc dimensionality reduction analysis of the emotion ratings. It is intriguingthat the low-dimensional value computations (e.g. PE, |PE|) are
intimately tied with the principal components of the emotion ratings (‘‘valence’’ and ‘‘arousal’’). However, note that the second PC(‘‘arousal’’) accounts for much less variance than the ﬁrst PC (‘‘valence’’). One possibility is that the paradigm we used is limited in
the range of emotions that it elicits in an agent, which restricts
the complexity of emotion inferences in this paradigm. A second
possibility is that emotional valence is the central feature of affective cognition, and valence would carry most of the variance inemotion inferences across more complex scenarios. Although weare not able to address the second possibility in this paper, there
is much theoretical evidence from affective science (e.g., Barrett& Russell, 1999; Russell, 1980; Schlosberg, 1954) in favor of thesecond possibility; future work is needed to explore this further.
Together, these ﬁndings provide two key insights into the structure of affective cognition: (i) lay theories of emotion are low
dimensional, consistent with affective science concepts of valence
and arousal, and (ii) these core dimensions of emotion inference
also track aspects of the situation and outcome that reﬂect value
computation parameters described by economic models such asprospect theory. Although the speciﬁc structure of affective
6
We invite the reader to compare the striking similarity between our Fig. 4A withsimilar valence-arousal ﬁgures in the literature, such as Fig. 1 from Russell (1980) and
Fig. 2A from Kuppens et al. (2012).

D.C. Ong et al. / Cognition 143 (2015) 141–162

147

Fig. 4. (A) Participants’ emotion ratings projected onto the dimensions of the ﬁrst two principal components (PCs), along with the loadings of the PCs on each of the eightemotions. The loading of the PCs onto the eight emotions suggests a natural interpretation of the ﬁrst two PCs as ‘‘valence’’ and ‘‘arousal’’ respectively. The labels for disgust
and anger are overlapping. (B) Participant’s emotion ratings projected onto the dimensions of the ﬁrst two PCs, this time colored by the prediction error (PE = amountwon  expected value of wheel). (For interpretation of the references to colour in this ﬁgure legend, the reader is referred to the web version of this article.)cognition likely varies depending on the complexity and details ofa given context, we believe that observers’ use of low-dimensional‘‘active psychological ingredients’’ in drawing inferences constitutes a core feature of affective cognition.
2.2. Experiment 2: Outcomes from emotions
Experiment 1 established the features that allow observers toreason about agents’ emotions given a gamble outcome: representing P(e|o) in a simple linear model. In Experiment 2 (Fig. 5), we test
the hypothesis that participants’ lay theories are ﬂexible enough toalso allow ‘‘backward’’ inferences about the situation outcome
based on emotions. These backward inferences are predicted fromthe forward regression model by Bayes’ rule. To evaluate our model’s predictions, we show participants the emotions of characters
who have played the same game show—importantly, without
showing the eventual outcome—and elicit participants’ judgments
about the likely outcome. We then compare the empirical judgments (in Experiment 2) to the posterior probabilities predictedby the model (based on data from Experiment 1).
2.2.1. Participants
We recruited one hundred twenty-ﬁve participants via Amazon
Mechanical Turk. We excluded three participants because theyreported technical difﬁculties with the animation, resulting in a
ﬁnal sample size of 122.
2.2.2. Stimuli
We generated graphical emotion proﬁles that ostensibly represented the emotions that the agent feels after each outcome(Fig. 5B). These emotion proﬁles were shown on continuous slider
scales that corresponded to each individual emotion. For each of
the outcomes, we used the average emotion ratings given by participants in Experiment 1 to generate the emotion proﬁles seen
by participants in Experiment 2. Speciﬁcally, for each outcome,we drew eight emotion values from Gaussians with means andstandard deviations equal to the means and standard deviationsof the emotion ratings given by participants in response to thatoutcome in Experiment 1. This was meant to provide naturalistic
emotion proﬁles for the participants in Experiment 2.

2.2.3. Procedures
Each participant completed 10 trials, randomly drawn from thesame 50 pre-generated scenarios used in Experiment 1. On eachtrial, participants were shown a stick ﬁgure and the game wheel,as before. This time, as the wheel spun, a gray square covered
the wheel, occluding the outcome (Fig. 5A). Participants were thenshown a graphical representation of the agent’s emotions after seeing the outcome (Fig. 5B). For instance, on a particular trial, a participant might see three possible gamble outcomes of $25, $60, and$100. Without being able to see where the wheel landed, theymight then be told that the agent who spun the wheel, following
its outcome, feels moderately happy (e.g., a slider at about a 6 on
a 9 point scale), not at all sad (e.g., another slider that shows
between a 1 and 2 on a 9 point scale), and so forth; see Fig. 5 for
an illustration. Participants were then asked to infer how likely it
was that each of the three possible outcomes had occurred. Theygave probability judgments on 9 point Likert scales. Hence, on eachtrial, they gave three likelihood judgments, which corresponded toeach of the three possible outcomes.
2.2.4. Bayesian model details
In order to generate model-based predictions about observers’‘‘reverse inference’’ about outcomes given emotions, the posterior
P(o|e), one needs three components: P(o), P(e), and P(e|o) (Eq.
(1)). The prior probabilities of the outcomes P(o) here are givenby the notational probability of each outcome on the wheel—i.e.
the relative size of each outcome—and are transparent to an observer in our paradigm. Larger sectors have higher probabilities ofthat outcome occurring. One actually does not need to explicitly
calculate the prior probability of the emotion P(e) as long as we
properly normalize P(o|e).7
The crucial step lies in calculating the likelihood P(e|o). To calculate P(e|o), we drew on the ‘‘forward reasoning’’ data collected in
7
Consider calculating the posteriors of the three possible outcomes given an
observed value of happiness, h: P(o1|h), P(o2|h), and P(o3|h), which are proportional
to [P(h|o1)P(o1)], [P(h|o2)P(o2)], and [P(h|o3)P(o3)] respectively. The sum of the latter
three quantities is simply the prior on emotion P(h). Thus, as long as proper
normalization of the probabilities is carried out (i.e. ensuring that the posteriors all
sum to 1), we do not need to explicitly calculate P(h) in our calculation of P(o|h). Thisis true for the other emotions in the model.

148

D.C. Ong et al. / Cognition 143 (2015) 141–162

A

Less

More

B

Fig. 5. Screenshots from Experiment 2. (A) Participants were shown a character that spins a wheel, but a gray square then occludes the outcome on the wheel. (B) Participants
then saw a graphical representation of the character’s emotions, and inferred separate probabilities for each of the three possible outcomes having occurred.

Experiment 1. In particular, we leveraged Experiment 1’s regression model to calculate the extent to which observers would be
likely to infer different emotions of an agent based on each outcome. To do so, we applied the variables identiﬁed as most relevant
to affective cognition in Experiment 1—win, PE, and |PE|—to predict
the emotions observers would assign to agents given the novel outcomes of Experiment 2. For instance, in modeling happiness, thisapproach produces the following equation for each wheel
outcome:happy ¼ c0;happy þ c1;happy win þ c2;happy PE þ c3;happy jPEj þ ehappy

ð2Þ

We employed similar regression equations to estimate P(e|o) for
the other seven emotions, where eemotion (with zero mean and standard deviation eemotion ) represents the residual error terms of theregressions. The coefﬁcients ci;emotion are obtained numerically by ﬁtting the data from Experiment 1; the linear model is ﬁt across all
participants and scenarios to obtain one set of coefﬁcients per emotion. For a new scenario with {win, PE, |PE|}, the likelihood of00
observing a certain happy value h is simply the probability that h
is drawn from the linear model. In other words, it is the probability
that the error0h  c0;happy þ c1;happy win þ c2;happy PE þ c3;happy jPEj

ð3Þis drawn from the residual error distribution for ehappy .
The error distribution ehappy resulting from the above regression
captures intrinsic noise in the relation between outcomes andemotions of the agent—uncertainty in the participant’s lay theory.
However, in addition to this noise, there are several other sourcesof noise that may enter into participants’ judgments. First, participants may not take the graphical sliders as an accurate representation of the agent’s true emotions (and indeed we experimentally
generated these displays with a small amount of noise, as
described above). Secondly, participants might have some uncertainty around reading the values from the slider. Thirdly, participants may also have some noise in the prior estimates they use
in each trial.
Instead of having multiple noise parameters to model these andother external sources of noise, we instead modiﬁed the intrinsic

noise in the regression model. We added a likelihood smoothing
parameter f (zeta), which ampliﬁes the intrinsic noise in the
regression model, such that the likelihood P(h0 |o) is the probability
0that h  c0;happy þ c1;happy win þ c2;happy PE þ c3;happy jPEj is drawn from
Nð0; ðfrhappy Þ2 Þ, i.e. a normal distribution with mean 0 and standard
deviation frhappy .
Using Eq. (3) with the additional noise parameter, we can calculate the likelihood of observing a value h0 as a result of an outcomeo, i.e. P(h0 |o). We then calculate the joint likelihood of observing acertain combination of emotions e0 for a particular outcome o as
the product of the individual likelihoods,000

Pðe0 joÞ ¼ Pðhappy joÞPðsad joÞ . . . Pðdiapp joÞ

ð4Þ

A note on Eq. (4): The only assumption we make is that the outcome o is the only cause of the emotions, i.e., there are no otherhidden causes that might inﬂuence emotions. The individual emotions are conditionally independent given the outcome (common
cause), and thus the joint likelihood is proportional to the productof the individual emotion likelihoods.
Next, to calculate the posterior as speciﬁed in Eq. (1), we multiply the joint likelihood P(e|o) with the prior probability of the outcome o occurring, P(o), which is simply the size of the sector. We
performed this calculation for each individual outcome, before normalizing to ensure the posterior probabilities P(o|e) for a particularwheel sum to 1 (by dividing each probability by the sum of theposteriors). The normalization removes the need to calculate P(e)explicitly (see Footnote 3). The resulting model has only one freenoise parameter, otherwise being ﬁxed by hypothesis and theresults of Experiment 1.
Up to this point, we had only used data from participants inExperiment 1 to build the model. To verify the model, we collapsed
the empirical judgments that participants gave in Experiment 2 (anindependent group compared to participants in Experiment 1) for
each individual outcome. We then compared the model’s predicted
posterior probabilities for each outcome to the empirical
judgments.
We optimized the one free noise parameter f in the model tominimize the root-mean-squared-error (RMSE) of the model

149

D.C. Ong et al. / Cognition 143 (2015) 141–162

residuals. We conducted a bootstrap with 5000 iterations to estimate the noise parameter, RMSE, and the model correlation, as
well as their conﬁdence interval.
2.2.5. Results
Model-based posterior probabilities tightly tracked the observer judgments in Experiment 2 (Fig. 6B). The optimal noise parameter was 3.2 [2.9, 3.6], which resulted in a model RMSE of 0.116[0.112, 0.120]. The model’s predictions explained much of the variance in participants’ judgments, achieving a high correlation of
0.806 [0.792, 0.819]. For comparison, the bootstrapped split-halfcorrelation of the posterior probability estimates in Experiment 2
is 0.895 [0.866, 0.920]. The split-half correlation for the emotion
attributions in Experiment 1, the data that this model is ﬁt to, is0.938 [0.925, 0.950]. Together these two split-half reliabilities givean upper-bound for model performance, and our model performs
very comparably to these upper limits.
This results suggest that a Bayesian framework can accurately
describe how observers make reverse inferences, P(o|e), given
how they make forward inferences, P(e|o). At a broader level,
the results imply that the causal knowledge in an observer’s lay
theory of emotion is abstract enough to use for multiple patternsof reasoning. In the next section, we extend this work further byconsidering inferences about emotion from multiple sources of
information.

3. Emotional cue integration
Thus far, we have examined how observers reason from outcomes to emotions, and likewise use emotions to draw inferences
about the outcomes that caused those emotions. However, unlike
the observers in Experiment 2 who were given a graphical readout of agents’ emotional states, real-world observers rarely havedirect access to others’ emotions. Instead observers are forced to
draw inferences about invisible but crucial emotional states basedon agents’ outward, observable cues. In fact, observers often aretasked with putting together multiple, sometimes competing emotional cues, such as facial expression, body language, situation outcome, and verbal utterances. How does the observer integrate thisinformation—performing emotional cue integration—to infer theagent’s underlying emotional state? We propose that as with otherforms of cognition, observers’ performance might be similar to an
ideal observer that rationally integrates cues to an agent’s emotion
using Bayesian inference.
The model we introduced earlier (Fig. 2) can be extended tointegrate multiple cues to emotion. Assume that the observer hasaccess to both the outcome o and the facial expression f. The posterior probability of an underlying emotion e given both o and f isgiven by the following equation:Pðejo; fÞ /PðejfÞPðejoÞ
PðeÞ

ð5Þ

A complete derivation of this equation is given in Appendix B.
The joint-cue posterior—that is, P(e|o, f)—is proportional to the product of the individual cue likelihoods P(e|f) and P(e|o), normalized
by the prior probability of the emotion occurring P(e). This mathematical expression captures the intuition that judgments using twocues should be related to the judgments using each individual cue.
Not all cues provide equal information about the latent emotion—some cues are more reliable than others. Consider P(e|f) or
the probabilities of observing different intensities of emotion e,
given a face f. A big frown, for instance, often signals low happiness; thus, if we operationalize happiness as a variable ranging
from 1 to 9, we can represent P(happy|frown) as a curve that isFig. 6. (A) Simple causal model. The solid arrow represents a causal relationship(outcomes ‘‘cause’’ emotion; P(e|o) from Experiment 1), while the dashed arrow
represents an inference that can be made (P(o|e) from Experiment 2). (B)Comparison of participants’ estimates of the posterior probability from
Experiment 2 with the predictions of the model built in Experiment 1. There is astrong correlation of 0.806 [0.792, 0.819]. The dashed red line has intercept 0 andslope 1 and is added for reference. (For interpretation of the references to colour inthis ﬁgure legend, the reader is referred to the web version of this article.)

sharply peaked at a low value of happy, like 3 (see the red curvein the top panel in Fig. 7.). In this case, when the observer can be
fairly conﬁdent of the inference of emotion given the face, we
say that this face is a reliable cue.
By contrast, if a second face has an ambiguous expression, thisface might not give the observer much information about the unobserved emotion. In this case, we represent P(happy|ambiguous
expression) as a curve with a larger variance. If asked to make aninference about an agent’s emotion based on this expression, observers make an inference on a value of happiness, but they may not be
conﬁdent in the accuracy of that estimate; this face is an unreliablecue (see the red curve in the bottom panel in Fig. 7).
Consider the two faces described above, now each paired with amedium-reliability context. Imagine an observer who sees anagent unwrap a parcel from home, a context that usually elicits
some joy. If the observer sees the context paired with an ambiguous facial expression (a relatively low-reliability cue), the observer’s judgments would tend to rely more on the context, because
the facial expression contributes little information (bottom panel,Fig. 7). In contrast, if the observer sees instead a big frown (a relatively high-reliability cue) with the context, the observer’s judgments would tend to favor the facial expression over the context(top panel, Fig. 7). Importantly, reliability gives a quantiﬁable
way of measuring how much each cue should be weighted withrespect to each other.
We operationalize the reliability of a cue by calculating theinformation theoretic entropy (Shannon & Weaver, 1949), H, ofthe emotion-given-cue (e.g. e|f) distribution:

X
X
H½PðejfÞ ¼  PðfÞ PðejfÞ log PðejfÞ
f2F

ð6Þ

e

where the sum over e is taken over the different values of theemotion (in our model, emotions take discrete values from 1 to

150

D.C. Ong et al. / Cognition 143 (2015) 141–162

1.5

2. P(e|o)

1.5

1.0

0.5

Distribution
1. P(e|f)

0.0

Low Reliability

Probability

0.5High Reliability

1.0

3. P(e|f)*P(e|o)

0.0Fig. 7. Illustration of the effect of the reliability of a cue in cue combination. The same contextual outcome cue distribution (green solid line) is given in both panels. Top: theemotion distribution for a reliable face (red solid line). The distribution is relatively sharply peaked. Bottom: the emotion distribution for a less reliable face with the samemean (red solid line), which results in a larger uncertainty in the distribution as compared to the top panel. The blue dashed line in both panels shows the product of the two
distributions (note that this is only the numerator in the cue integration equation, Eq. (5), and does not include the normalizing prior term in the denominator). One can seethat the low reliability cue has a relatively smaller effect on the distribution of the product: the distribution P(e|f) ⁄ P(e|o) in the bottom panel is very similar to the
distribution P(e|o). (For interpretation of the references to color in this ﬁgure legend, the reader is referred to the web version of this article.)

9 inclusive).8 The entropy of a distribution captures the amount of
information that can be obtained from the distribution, with amore informative (‘‘sharply-peaked’’) distribution having lower
entropy. Thus, the higher the reliability of the cue, the lower theentropy of the emotion-given-cue probability distribution.
The resulting model predictions are sensitive to the reliability ofa given cue. To illustrate, consider two faces, f1 and f2, whose distributions P(e|f) have the same mean. Let f1 have a much smaller
variance so that P(e|f1) is very sharply peaked (red curve, top
panel, Fig. 7), and let f2 have a much larger variance so that
P(e|f2) is almost uniform (red curve, bottom panel, Fig. 7) i.e., f1
is a more reliable cue and has less entropy than f2. In a single
cue emotion-perception scenario, where a observer has to makean inference about the emotion e given only either f1 or f2, the distribution of the observer’s inferences would follow P(e|f); in bothcases, the observer would give the same mean, but perhaps withmore variance in the case of f2. Consider next a multi-cue integration scenario, where the observer is given two cues, the outcome
context o (green curve, Fig. 7), and either f1 or f2. Because f2 is lessreliable than f1, f2 will be weighted less with respect to o, than f1
will be weighted relative to o. Mathematically, this follows from
Eq. (5): multiplying the distribution P(e|o)/P(e) by P(e|f2) will havelittle effect because P(e|f2) has very similar values across a large
domain of emotion values, and so would modify each potential
emotion by a similar amount. Conversely, multiplying by P(e|f1)
will have a larger effect (see Fig. 7 for an illustration).
Finally, we consider two approximations—or simpliﬁcations—that observers could use during affective cognition, as alternatives
to our fully Bayesian cue integration model. Under a face-cue-onlymodel, a observer presented with a face f and an outcome o would
rely only on the face, and would thus use P(e|f) to approximate
P(e|o, f). Under a second outcome-cue-only model, the observer
instead relies exclusively on the outcome and uses P(e|o) toapproximate P(e|o, f). These approximations are meant to model
the predictions made by ‘‘face dominance’’ and ‘‘context dominance’’ accounts (see above).

8
In our experiments, faces are chosen at random, hence we use a uniform
distribution for the prior on faces, P(f).

In Experiments 3 and 4, we calculate the performance of these
two approximate models in addition to the full Bayesian model. In
Experiment 3, we examined cue integration with outcomes andfaces, and in Experiment 4, we examined integration of evidence
from outcomes and verbal utterances, showing the generalizabilityof the model to other types of emotional cues.
3.1. Experiment 3: Cue Integration from outcomes and faces
In Experiment 3, we tested our cue integration model by examining its correspondence with human judgments of combinationsof facial expressions and situation outcomes. On one third of thetrials, participants saw the outcome of a gamble that a character
plays, as in Experiment 1. On another third of the trials, participants saw only the facial expression following the outcome, but
not the outcome itself. On the ﬁnal third of ‘‘joint-cue’’ trials, participants saw both the facial expression and the outcome. On alltrials, participants attributed emotions to the character, as inExperiment 1.
3.1.1. Participants
We recruited four hundred sixty-ﬁve participants throughAmazon’s Mechanical Turk. We planned for a larger number thanwere included in Studies 1 and 2 because of the large number offace stimuli tested.
3.1.2. Stimuli
The gambles were identical to Experiments 1 and 2, except that
we used only 10 possible scenarios for this experiment. We generated eighteen facial expressions, shown in Fig. 8, using the software FaceGen. The ﬁrst 12 faces varied in emotional valence andarousal. Here, we operationalized emotional valence by parametrically varying the curvature of the lips—for positive valence: an
upward turn in the lips for a smile; and for negative valence: adownward turn in the lips for a frown or scowl—and the shapeof the eyes and eyebrows, with larger eyes and relaxed eyebrowssignaling positive valence, and smaller eyes and furrowed eyebrows signaling negative valence. We operationalized emotionalarousal by varying the gap between the lips, with low arousal faces
having no gap between the lips and high arousal faces showing a

D.C. Ong et al. / Cognition 143 (2015) 141–162

wide gap (a wide mouthed smile or scowl with teeth). We designed
the ﬁnal 6 faces to be ‘‘ambiguous’’, i.e. a mix of different emotions
like sad and angry, or sad and happy. We made these using a combination of FaceGen’s preset discrete emotion settings. Exact combinations are given in Fig. 8 caption.
3.1.3. Procedures
Participants completed ten trials. On each trial, participants
watched a character play a gamble in the form of a wheel(Fig. 9A). Participants saw the character spin the wheel, and werethen shown one of three possibilities. On Outcome-Only trials, participants saw only the outcome on the wheel (similar toExperiment 1). On Face-Only trials, a gray square would occlude
the outcome on the wheel, and participants saw only the facial
expression of the character after the character sees the outcomeon the wheel (we told participants that the character still could
see the outcome). Finally, on Joint-Cue trials, participants saw boththe outcome and the facial expression. See Fig. 9B for an illustration. These three types of trials (two types of single cue trials
and one joint-cue trial) occurred with equal probability, so on average each participant encountered about three of each type of trial,out of ten total trials. On the joint-cue trials where both facial andoutcome cues are shown, we randomly matched the outcome andface, so there was no correlation between the emotions typicallyconveyed by the two cues. This ensured a random mix of congruent(e.g., a large outcome on the wheel accompanied by a positive face)
and incongruent combinations (e.g., a large outcome on the wheel
accompanied by a negative face). On all trials, participants rated
the extent to which the character felt each of eight discrete emotions—happy, sad, angry, surprised, fearful, disgusted, content, anddisappointed—using 9-point Likert scales.
3.1.4. Model details
We used participants’ responses to the single-cue Outcome-Onlyand Face-Only trials to construct empirical distributions for P(e|o)
and P(e|f) respectively (Fig. 9C). For the outcome model, P(e|o), we
used the set of features isolated in Experiment 1 (win, PE, |PE|). The
ratings for the outcome-only trials were used to model P(e|o), in asimilar fashion to Experiment 1. The set of features isolated inExperiment 1 enables a reduction in dimensionality of the situation
features, which allows the estimation of a less noisy statisticalmodel. For faces, it is less clear what a corresponding set of low
dimensional features would be, and so we estimated P(e|f) fromthe raw density-smoothed9 empirical ratings to the face-only trials.
Using the face-only model P(e|f) and the wheel-only model P(e|o),we can construct the full Bayesian cue integration model (Eq. (5)), as
well as the approximate face-only and wheel-only models.
We compared these three models against participants’
responses to the Joint-Cue trials. First, we used participants’
responses to the Joint-Cue trials to generate an empirical(density-smoothed) probability distribution of emotion given a
particular face and outcome combination P(e|o, f). From this, we
calculated the expected emotion rating (i.e., a real value from 1to 9) for each emotion for each combination. Finally, we compared
the empirical expectations with the expectations of the different
models. To clarify, though this was a within-subject paradigm,
we did not have enough statistical power to build a cue integration
model for each individual participant, Instead, we constructed
these models collapsed across participants.
To compare the performance of the models, we calculated twoquantities, the root-mean-squared-error (RMSE) and the correlation, both with respect to the empirical data. We bootstrapped9
We used R’s density function with its default settings to perform densitysmoothing using a Gaussian kernel.

151

the two values, along with their 95% conﬁdence intervals, frombootstrap calculations with 5000 iterations.
3.1.5. Results
We present a summary of the results in Fig. 10. Comparing the
RMSE on all the trials (Fig. 10A), the Bayesian model performed thebest with the lowest RMSE of 1.218 [1.149, 1.287], outperforming10
both the Face-only model with 1.386 [1.314, 1.464] (t(49) = 3.30,
p = .002) and the Outcome-only model with 1.524 [1.462, 1.589](t(49) = 6.46, p < .001). Correspondingly, the Bayesian model also
had the highest correlation with the empirical data (r = 0.812[0.790, 0.833]), performing signiﬁcantly better than the Face-only(r = 0.733 [0.706, 0.759]; t(49) = 4.55, p < .001) and the
Outcome-only (r = 0.692 [0.664, 0.717]; t(49) = 7.06, p < .001) models (Fig. 10B).
Next, we analyzed two subsets of the data that contained incongruent cue combinations: Joint-Cue trials that had anegatively-valenced face presented with a ‘‘positive’’ wheel(deﬁned as winning the largest amount on the wheel), andJoint-Cue trials that had a positively-valenced face presented witha ‘‘negative’’ wheel (deﬁned as winning the smallest amount on
the wheel). Although the pattern of RMSEs did not differ fromthe set of all trials to the incongruent combinations (Fig. 10A),
we found an interesting result when we examined the correlationsof the single-cue-only models in the incongruent cue-combination
trials (Fig. 10B). When a negative face is presented with a positive
wheel, the Face-only model produced a lower correlation ofr = 0.343 [0.213, 0.468]) than the Outcome-only model (r = 0.465[0.332, 0.576]; t(49) = 1.31, p = .20, not signiﬁcant) and the full
Bayesian model (r = 0.571 [0.454, 0.668]; t(49) = 2.61, p = .012,
signiﬁcant with some overlap of the 95% CIs). Conversely, when apositive face is presented with a negative wheel, theOutcome-only model did signiﬁcantly worse with a correlation of
0.297 [0.165, 0.422], as compared to the Face-only model(r = 0.678 [0.551, 0.778]; t(49) = 4.19, p < .001) and the full
Bayesian model (r = 0.667 [0.568, 0.748]; t(49) = 4.55, p<.001).
In other words, if we compare the single-cue-only models, whena negative cue is presented with a positive cue, the model that only
considers the positive cue better approximated observer judgments than the model than only considers the negative cue.
This ‘‘valence-dominance’’ result is surprising, and in fact, is notpredicted by the literature, which, as laid out earlier, predicts
speciﬁc-cue-dominance. The Bayesian model, however, seems toaccount for this valence effect extremely well. In particular, theBayesian model automatically weights the positive cues to agreater extent than the negative cues. Because the Bayesian model
weights cues according to their reliability (see discussion above,
especially Fig. 7), this suggests that the positive cues have a higher
reliability than the negative cues. If this is true, then the positivecues distributions (both P(e|o) and P(e|f)) should have a lower
entropy than the negative cue distributions. Post-hoc analyses conﬁrmed this predicted difference in distribution entropies. The negative faces have a signiﬁcantly higher entropy (mean entropy ofthe emotion given negative face distributions = 2.66 bits,
SD = 0.43 bits) as compared to the positive faces (meanentropy = 1.60 bits, SD = 0.97 bits; t(7) = 2.70, p = .03). Similarly,
the negative wheels have a signiﬁcantly higher entropy (meanentropy = 2.39 bits, SD = 0.37 bits) than the positive wheels (meanentropy = 1.99 bits, SD = 0.43 bits; t(7) = 4.10, p = .005). This highlights an interesting result: the reason why there seems to be some10
In this section we bootstrapped parameter estimates and 95% conﬁdence
intervals. This allows judgment statistical signiﬁcance of differences by CI
non-overlap, a non-parametric approach. For ease of comparison, we additionallyestimate and report t-statistics, bearing in mind the hidden parametric assumptions
that may not be satisﬁed.

152

D.C. Ong et al. / Cognition 143 (2015) 141–162

Fig. 8. Face stimuli used in Experiment 1, created using FaceGen. The 12 faces in the top and middle rows vary in both valence and arousal. Top row: positively valenced faces,
increasing in valence and arousal from left to right. Middle row: negatively valenced faces, decreasing in valence and arousal from left to right. The top and middle right-mostfaces are neutral valence high arousal and neutral valence low arousal, respectively. Bottom row: set of ‘‘ambiguous’’ faces made using combinations of FaceGen’s pre-deﬁned
discrete emotions. From left to right: (Sad, Surprised, and Happy), (Angry, Happy, and Surprised), (Fear, Happy, and Disgust), (Disgust, Surprised, and Happy), (Sad, Happy,
Disgust and Fear), and (Sad, Happy, and Angry).

Fig. 9. (A) Screenshot from a trial from Experiment 3. Participants saw a character about to spin a wheel with three possible outcomes. (B) Each trial resulted in one of threepossibilities: the participant is shown (i) only the outcome, (ii) only the character’s facial expression, or (iii) both the outcome and the facial expression. Following this, theparticipant is asked to judge the character’s emotions. (C) The single cue trials are used to model P(e|o) and P(e|f) respectively, which serve as single-cue only models. The
single-cue models are used to calculate the Bayesian cue-integration model. These three models are evaluated using empirical judgments made by participants in the jointcue trials.

evidence for a ‘‘positive-valence-dominance’’ is because positivecues tend to have higher reliability. This result implies that—atleast in the context of our task—participants tend to be more certain when making emotion attributions to agents given positive,
as compared to negative, cues.
In sum, the results from Experiment 3 showed that the Bayesianmodel best predicts participants’ judgments of emotions in
multiple-cue scenarios. In addition, this quantitative paradigm
allowed us to examine participants’ emotion attributions in incongruent cue combinations, and uncovered evidence for a different
type of dominance: in our paradigm, positively-valenced cues havegreater reliability and tend to dominate negatively-valenced cues.
However, we do not want the take home message to be that ‘‘posi
tive-valence-dominance’’ is a better rule than face or context dominance to resolve conﬂicts; in fact, this is antithetical to the spirit of

the model. The Bayesian model makes one simple assumption: thatobservers weigh cues according to the cues’ reliability. In this gambling paradigm, positive cues have higher reliability, but we do notwant to generalize that positive cues in other contexts are morereliable as well. The Bayesian model accounted for this valence
effect even without an explicit assumption, further suggesting thata rational approach to emotional cue integration is well able tocapture these intricacies in affective cognition.
3.2. Experiment 4: Cue integration from outcomes and utterances
Experiment 3 examined combinations of facial expressions andsituation outcomes. In Experiment 4, we show that our model generalizes to other cues by examining combinations of verbal utterances and situation outcomes.

D.C. Ong et al. / Cognition 143 (2015) 141–162

153

Fig. 10. Results of Experiment 3. Error bars indicate bootstrapped 95% conﬁdence intervals. (A) Root-mean-squared-error for the different models. From left to right,
performance on all data; performance on the negative face, positive wheel trials; and performance on the positive face, negative wheel trials. (B) Correlations of the different
models with empirical judgments. (C) Performance of the Bayesian model on all trials. On the vertical axis are participants’ judgments of all eight emotions in the joint-cue
trials, and the horizontal axis indicates the models’ predicted emotions. The red-dashed line has an intercept of 0 and a slope of 1, and is added for reference. The model’scorrelation with the empirical joint-cue predictions was 0.812 [0.790, 0.833].

3.2.1. Participants
We recruited one hundred ﬁfty participants through Amazon’s
Mechanical Turk.
3.2.2. Stimuli
We used identical gambles to Experiment 3. We replaced thefacial expression with an utterance, ostensibly made by the character after seeing the outcome of the wheel. The list of the 10 utterances (‘‘cool’’, ‘‘awesome’’, ‘‘yay’’, ‘‘wow’’, ‘‘man’’, ‘‘oh’’, ‘‘damn’’,‘‘dang’’, ‘‘meh’’ and ‘‘yikes’’) included a mix of clearly valenced
utterances and ambiguous utterances.
3.2.3. Procedures
We used identical procedures to Experiment 3, except thatinstead of face stimuli, participants saw an utterance that ostensibly was made by the character after seeing the outcome of thewheel. Each participant completed 10 trials. On each trial, participants saw either (i) an Outcome-Only trial, (ii) an Utterance-Only

154

D.C. Ong et al. / Cognition 143 (2015) 141–162

Fig. 11. Results of Experiment 4. (A) Root-mean-squared-error for the different models. From left to right, performance on all data; performance on the negative utterance
and positive wheel trials; and performance on the positive utterance and negative wheel trials. (B) Correlations of the different models. (C) Performance of the Bayesian modelon all trials, showing a correlation with the empirical joint-cue predictions of 0.733 [0.687, 0.770].

trial, or (iii) a Joint-Cue trial with both outcomes and utterances,randomly paired. Participants then attributed emotions to thecharacter as in Experiment 3.
3.2.4. Results
We repeated the same analysis procedure as in Experiment 3.
We used participants’ responses to the Utterance-Only andOutcome-Only trials to construct the empirical distributions forP(e|u) and P(e|o) respectively. Next, we used these two models
to construct the full Bayesian model P(e|o, u). Replicating theresults of Experiment 3, on all trials, the Bayesian model performs
the best with the lowest RMSE of 1.494 [1.391, 1.612], performing

signiﬁcantly better as compared to the Utterance-only model at
1.847 [1.731, 1.969] (t(49) = 4.18, p < .001), and better, but notsigniﬁcantly, as compared to the Outcome-only model at 1.619[1.512, 1.724] (t(49) = 1.54, p = .13). When we examined the correlation with the empirical judgments, the Bayesian model
achieved a correlation of r = 0.733 [0.687, 0.770], again, signiﬁcantly better compared with the Utterance-only model at
r = 0.570 [0.518, 0.618] (t(49) = 4.81, p < .001), and the
Outcome-only model at r = 0.668 [0.623, 0.710] (t(49) = 2.05,
p = .046, signiﬁcant with some overlap of the 95% CIs) (Fig. 11).
When we repeated the analysis for the incongruent combination subsets, we ﬁnd that the Bayesian model again tightly predicts

D.C. Ong et al. / Cognition 143 (2015) 141–162

155

Fig. 12. Scatterplot results of Experiment 4, only on the trials with the incongruent combination of Negative Utterance and Positive Outcomes. All emotions are plotted.
Participants’ judgments are plotted on the y-axis on all three graphs. The Utterance-only model, in the bottom right, has a negative correlation with the data, due to the larger
spread, possibly due to potential pragmatic interpretations that the utterance-only model cannot account for.

observers’ judgments, with a high correlation with the empirical
judgments (Negative Utterance and Positive Outcome, r = 0.784[0.702, 0.845]; Positive Utterance and Negative Outcome,r = 0.517 [0.367, 0.620]), although it did not signiﬁcantly outperform the Outcome-only model (Negative Utterance and Positive
Outcome, r = 0.775 [0.684, 0.844]; t(49) = 0.16, p = .87; Positive
Utterance and Negative Outcome, r = 0.486 [0.321, 0.628];
t(49) = 0.29, p = .77) in this experiment (see Fig. 11).
Unexpectedly, the Utterance-only model showed a negative
correlation of r = 0.147 [0.276, 0.012] with the empirical judgments in trials in which a negative utterance was presented with apositive outcome. We suspect that this is because utterances could
be interpreted pragmatically, i.e., in a non-literal way. Consider anutterance like ‘‘dang’’: this utterance carries a negative literal connotation (e.g. expected happiness given ‘‘dang’’ alone was 2.66 on a
1–9 scale). However, when paired with a positive outcome—an
example outcome which, rated alone, produced an expected happiness rating of 8.18—the combination of ‘‘dang’’ and the outcome
could be interpreted as being even more positive (expected happiness of the combination of ‘‘dang’’ and the outcome is 8.75). Thus,
although observers might interpret an individual verbal cue to be
negative, they might interpret the same cue, presented with a positive outcome, to be positive, perhaps as a form of sarcasm orhyperbole. Note that there is much heterogeneity in the way thatparticipants interpret these combinations: not everyone treats
them pragmatically, and not every such combination is treatedpragmatically. This heterogeneity results in a larger spread inparticipant responses to the joint combinations relative to theutterance-only results (see Fig. 12). This effect likely drives thenegative correlation of the (literal) Utterance-only model withthe judgments of the combinations given by participants. It is
worth noting that there has been recent success in applying
Bayesian models to study interpretation of non-literal language(Frank & Goodman, 2012; Kao, Wu, Bergen, & Goodman, 2014).
Future work could extend our model of affective cognition to
include these pragmatic effects.
Finally, as in Experiment 3, we ﬁnd an effect of valence on cue
reliability: positive cues have higher reliability and lower entropy
than negative cues. For outcomes alone, the negative wheels show
a signiﬁcantly higher entropy (mean entropy = 2.38 bits, SD = 0.37
bits) than the positive wheels (mean entropy = 1.98 bits, SD = 0.40
bits; t(7) = 5.01, p = .002). This was the case for utterances as well;entropy for negative utterances (mean entropy = 2.69 bits,
SD = 0.45 bits) was greater than for the positive utterances (meanentropy = 1.77 bits, SD = 0.80 bits; t(7) = 2.83, p = .03). This reinforces the ﬁnding from Experiment 3 that participants seem to
be more certain in their judgments given positive cues than negative cues, at least in this gambling context.
The results of this experiment show that the emotional cue
integration model we propose, derived from a Bayesian treatmentof lay theories, generalizes beyond facial expressions to other cues.
Experiment 4 also replicates many of the critical results from
Experiment 3, including the consistent performance of theBayesian model.

156

D.C. Ong et al. / Cognition 143 (2015) 141–162

3.3. Revisiting emotions from outcomes: a combined analysis
Experiments 3 and 4 were designed to test our cue integration
predictions, using analyses that built upon the results ofExperiment 1, which explored the important outcome features
for predicting participants’ affective judgments. However,Experiments 3 and 4 also produced additional data with which
we can re-examine Experiment 1’s results, that is, the statisticalcausal model of outcomes to emotions. This combined analysis
would allow us to test the reliability of the results of the statisticalmodel that we built in Experiment 1 (and used in the cue integration analyses above). In order to do this, we proceeded to isolate
the wheel-only trials from Experiments 3 and 4, together with all
the trials from Experiment 1, creating a dataset of 3048 observations from 690 participants.
With this larger dataset, we repeated the model selection, withthe full set of regressors (as in Appendix A), to determine which outcome features signiﬁcantly predict attributed emotions. As
expected, the amount won, PE, and |PE| again came out as signiﬁcantpredictors for the various emotions (details in Appendix A), withseveral additional patterns of interest: e.g., fear is only predictedby the amount won. Surprise now strongly depends on |PE|, regret,and the winning probability, which suggests a different structure
for surprise than the rest of the emotions (additional evidence for thisis also given by the different loadings in the PCA analysis).
One additional regressor of note became signiﬁcant with thislarger dataset: the near-miss term predicted happiness(b = 3.32e05 [5.51e05 1.12e05], t(682) = 2.963, p = 0.003).
We tested for, but did not ﬁnd a signiﬁcant difference between positive and negative near-misses (v2(1) = 0.089, p = 0.77), i.e., whether
the next-nearer outcome was a larger or smaller payoff, although
this could just be from a lack of power. To better understand themagnitude of the near-miss term, let us consider a concretenear-miss example using the slopes on win (b = 0.0405 [0.029,
0.052], t(682) = 7.08, p < .001), PE (b = 0.036 [0.024, 0.048], t(682) =5.95, p < .001), and |PE| (b = 0.015 [0.025, 0.005] t(682) = 2.84,
p = .006), and the $25/$60/$100 wheel in Fig. 2. Not considering thenear-miss term, and all else being equal, if the result had changed
from $60 to $100, there would be an increase in happiness of
40 ⁄ (.0405 + 0.036  0.015) = 2.46 points on a 9 point Likert scale.
By contrast, if the outcome result (the exact point the black pointerindicated) moved from the center of the $60 sector to a near-missdistance of 1% of the sector size away from the $60/$100 boundary,
there would be a decrease in happiness of 40 ⁄ (1/0.5  1/0.01) ⁄(3.32 ⁄ 105) = 0.130 points on a 9 point scale. Thus, in this gambling scenario, the effect of a near-miss on subjective happiness
attributed is on the order of 5% of the relative happiness of winning
the next higher amount. Getting a near-miss on the $60 wheel inFig. 2 and narrowly missing the $100 sector (narrowly missing winning $40 more) has a subjective cost equivalent to losing about $2,
compared with a far-miss (landing in the center of the sector). Thisis a small effect relative to actually winning—and indeed, we could
not detect it with Experiment 1 data alone—yet it is a large andnot insigniﬁcant effect considering that it does not depend on changing actual payoffs, but merely relative closeness. This result builds
upon the results of Experiment 1 by investigating what additionalsituation features might factor (perhaps more weakly) into affective
cognition in this paradigm.

4. Discussion
We have proposed a framework for studying affective cognition
as domain-general reasoning applied to a domain-speciﬁc lay theory of emotions; the lay theory is described as a statistical causal
knowledge of others’ emotions, and reasoning as Bayesianinference based on this knowledge. Observers’ lay theories consistof a consistent structure that captures causal relationships
between situation outcomes (or emotion-eliciting events), theagent’s emotions, and the agent’s observable behaviors. Each ofthese causal relationships contains complex knowledge: for example, an observer incorporates appraisal processes when reasoning
about how an agent feels after the outcome of a situation. This
framework makes detailed quantitative predictions that wereborne out in a series of experiments. We demonstrated that observers are able to consistently reason both forward, along the causal
direction, and ‘‘backwards’’, about causes based on emotional reaction. The forward causal model relied on a small set of situation
features (Experiment 1), and backward reasoning was
well-described as Bayesian inference given this forward model(Experiment 2). This approach provides further traction in understanding how observers infer unobservable emotional states fromdiverse observable cues. In particular, we have shown that integrating multiple sources of information as prescribed by Bayesianstatistics explains human judgments of emotions from facial
expressions and outcomes (Experiment 3), and from utterancesand outcomes (Experiment 4). Our results showed an interesting‘‘valence dominance’’ effect, whereby positively-valenced emotional cues tended to have higher reliabilities and were weighted
more so than negatively-valenced cues. Our model was able toaccount for this valence effect without a priori speciﬁcation, attesting to the robustness of a probabilistic model of affective cognition.
Our studies contrast with previous studies that have found face orcontext dominance (e.g., Aviezer et al., 2008; Nakamura et al.,
1990; Russell et al., 2003) in that our paradigm involves a
restricted gambling context; it is possible that our valence result
may not generalize to other contexts, though we believe that theunderlying model of cue integration will still be applicable.
Emotions are numerous and complex, and we have only examined emotional reasoning in very constrained gambling scenarios.
These scenarios constitute a reasonable starting point for studying
affective cognition, because they afford consistent assumptions
and quantitative manipulations. For instance, an agent playing agamble almost certainly wants to win more money, giving observers a clear cornerstone for understanding the affective meaningof different gamble outcomes. Even in these simple scenarios, withostensibly simple economic motivations, participants were sensitive to subtle ‘‘irrationalities’’ that they expected the agent toevince. For instance, by pooling data from three of our experiments, we found that the nearness of alternative, better outcomes(or worse outcomes) factored into our participants’ inferences
about agents’ emotion. Further work will likely uncover additionalfeatures that factor into affective cognition.
Not all emotions are relevant in our scenarios—fear, for example, may be irrelevant, as may a more complex emotion likepride—and furthermore, we might reason about emotions like happiness differently in a gambling context than in other contexts. One
avenue of further research includes extending our model to more
complex situations. For instance, people do not live in a social vacuum, and emotions often depend on interactions with others: acomprehensive model would include how observers reason about
emotions in social interactions, leading to emotions like prideand jealousy.
The work presented here was concerned with high-level reasoning, and abstracted out, for example, the process by which weperceive facial expressions (How does an observer look at a faceand decide that the agent is happy? Which facial features areimportant?), or how we interpret the linguistic content of emotional utterances (How do we interpret ‘‘dang’’ as negative?). In
this work, we measured the connection between these percepts
and emotional states, rather than explaining them, then investigated how observers utilized them to make higher-level

D.C. Ong et al. / Cognition 143 (2015) 141–162

inferences. The details of these processes are important in a fulldescription of affective cognition, and each of these will require
further, domain-speciﬁc study into the lay theory of emotion.
That said, we believe that the framework considered here—treating
affective cognition using domain-general reasoning—is generalizable and will extend to more detailed knowledge about emotions.
4.1. Relation to modeling of social cognition
In Fig. 2 and throughout this paper, we alluded to other mental
states, such as the agent’s goals, that are important in a lay theoryof emotion but that we did not consider in detail. This tight link
between emotions and other mental states translates into a strong
parallel of our work with other models of social cognition (e.g.,Goodman, Baker, & Tenenbaum, 2009). For example, Baker et al.
(2009) proposed a computational model of Theory of Mind thatincorporates an agent’s beliefs and desires into an observer’s lay
theory of behavior. This was a formalization of earlier work inbelief-desire psychology, describing a lay theory of how an agent
rationally chooses actions given his beliefs and desires (Dennet,
1989; Gopnik & Meltzoff, 1997). In a similar fashion, the recently
proposed Rational Speech Act (RSA) model (e.g., Frank &Goodman, 2012; Goodman & Stuhlmüller, 2013; Kao et al., 2014)
treats language understanding between agents as rational social
cognition. Again, this work has its roots in earlier work on communication between rational agents (e.g., Clark, 1996). The RSA model
assumes that the speaker rationally chooses his utterance to convey his desired inference to the listener, and the listener recursively reasons about the speaker’s mental state to infer the goalof the speaker. These and other models of social cognition are helping to shed light on how naïve observers reason about the cognitive
states of those around them.
Integrating the lay theories of cognitive states (beliefs anddesires) and lay theories of emotions will have explanatory beneﬁts for both lines of work. For instance, in this paper we implicitly
assumed the goals of the agent, and showed how observers reason
about an agent’s emotions with respect to the prediction error of
the situation, or how well the situation compared to the agent’s
expectations. In our paradigm it was a safe assumption that agentswished to win larger amounts of money; in more general situations
the observer would have to make an inference about the agent’s
goals in order to evaluate the ‘‘prediction error’’. Thus, one would
have to incorporate inferences over goals into a comprehensive
theory of affective cognition. In the other direction—adding emotions into other models of social cognition—attributed emotions
become a potential cause for otherwise ‘‘irrational’’ actions thatare observed. This is an important extension to current models ofsocial cognition, which must explain actions as the result of
rational, purposive behavior or of simple noise.
4.2. Relation to scientiﬁc theories of emotion
Here we have focused on lay theories that observers apply tounderstand the emotions of those around them, but people likely
apply an analogous reasoning process to reason about their own
emotions as well. This is consistent with evidence that people
use similar strategies when reasoning about their own traits andpreferences (Bem, 1972; Gopnik, 1993; Nisbett & Wilson, 1977).
Contrary to many people’s intuition, it appears that individuals
do not have privileged access to their ‘‘true’’ mental states; rather,
they use observations of their own behaviors and situations to reason about themselves using lay theories—similar to how theymight reason about other people.
A similar paradigm shift in affective science increasingly suggests that people do not have ‘‘privileged access’’ to their emotional experience, but instead reason about their experiences in a

157

contextualized manner. For instance, under Feldman Barret
et al.’s Conceptual Act Theory (Barrett, 2006; Lindquist & Barrett,2008), a typical emotion experience starts with the agent experiencing a stimulus. The agent collects information about the context
as well as visceral signals of valence (e.g. pleasurable feelings) andarousal (e.g. heart rate, clammy skin)—what Barrett calls ‘‘core
affect’’ (Russell & Barrett, 1999). The agent then uses these pieces
of information as inputs into a conceptualization process wherein
the agent labels their emotion using emotion concepts—the agent’s
own lay theory of emotion.
Our approach to understanding third-person affective cognition
mirrors the emerging prominence of concepts and lay theories inﬁrst-person emotional experience. There is much room for overlap
and progress down these parallel and complementary paths. On
the one hand, studying ﬁrst-person emotional experience might
give insight into what factors go into a third-person lay theory.
In our work, for example, we build off prior work in appraisal theory (e.g., Ortony et al., 1988) and behavioral economics (e.g.
Kahneman & Tversky, 1979) to inform our a priori assumptionsof PE and loss aversion, which we can then test in a third-person
context. Conversely, studying third-person lay theories mightallow testing of predictions that might be hard to manipulate ina ﬁrst-person context. For example, in a ﬁrst-person context, it is
difﬁcult to experimentally separate the affective component of a
response from the cognitive processes of reasoning and categorization. If a researcher ﬁnds a phenomenon in a ﬁrst-person context
and cannot distinguish whether it might be due to a cognitive process, then translating the paradigm into a third-person context
might allow the researcher to isolate the cognitive componentsof that reasoning. Thus our work may prove useful in understanding emotional experience in addition to emotional reasoning.

4.3. Applications
An affective cognition approach also holds a great deal of
applied potential. One natural application is to artiﬁcial agents capable of interacting with emotional users (e.g., Breazeal, 2004;
Gratch & Marsella, 2004; Hudlicka, 2003; Picard, 1997; Wehrle &Scherer, 2001). There are many speciﬁc applications of emotionallyaware agents. Researchers have started using ‘‘virtual humans’’ intraining physicians in patient interaction (Raij et al., 2007; see also
Medical Cyberworlds, Inc.), and using realistic avatars to populate
immersive virtual reality environments in order to improve users’social cognition (Bailenson, Yee, Merget, & Schroeder, 2006;
Swartout et al., 2006). Other researchers have worked on buildings
robots that can provide companionship—often termed Social
Robotics, for example, Kismet, from the MIT Media Lab (Brooks,
Breazeal, Marjanović, Scassellati, & Williamson, 1999). Building
on our framework here, we can imagine endowing artiﬁcial observers with a human-like theory of emotions. If one uses a psychologically validated computational model to allow artiﬁcial
observers to reason about emotions, this could result innear-human-level ability to attribute emotions, but importantly,
do so in a way that a human partner would expect. One could
imagine incorporating a lay theory of emotion into personal digitalassistants (like Apple’s Siri and Google’s Google Now) which, whencombined with the enormous amount of information they ‘‘observe’’ about their users, would allow digital assistants to reason
about and react to their users’ emotions. This would have far reaching implications for the efﬁciency of these products (digital assistants pre-empting users’ emotions and choices) and also improveusers’ likeability of the product. We might also start seeing these
digital assistants doubling up as conversation partners, and perhaps providing basic advice and counseling services (e.g., modern
successors to the ‘‘computer therapist’’ ELIZA).

158

D.C. Ong et al. / Cognition 143 (2015) 141–162

This segues into another potential area in which a theory affective cognition could prove useful: the diagnosis and treatment of
psychopathology. The idea that disorders might arise from a bias
in affective cognition dovetails with popular approaches in clinicalpsychology, such as Cognitive Behaviorism, which conceptualizes
affective disorders as stemming from maladaptive cognitive biases(Beck, 1979). Our work might serve as an additional tool that thecognitive behavior therapist may use to help identify patients’biases in reasoning about others’, and possibly the patients’ own,
emotions. One noteworthy and emerging ﬁeld is that ofComputational Psychiatry (Huys et al., 2012; Montague, Dolan,
Friston, & Dayan, 2012; Sharp, Monterosso, & Montague, 2012),
which assumes that social and emotional functioning can be characterized by computational ‘‘phenotypes’’, which comprise genetic,
neural, and behavioral characteristics described in a computationalmodel of cognitive functioning. Similarly, our work provides a way
to model behavioral attributions of emotions that could form partof these models of psychiatric disorders. One could imagine using amodel of affective cognition to help identify characteristic biases incertain psychiatric populations: for example, perhaps patients who
suffer from anxiety might also attribute increased emotional reactivity to agents (greater reliance on PE, |PE|) as compared to a typical observer. Finally, combining the applications to technology
and psychopathology, our work could enable novel technologies
for scalable diagnosis and treatment. Our work could inform automated cognitive monitoring applications and automated dialog
agents that can measure early warning diagnostic signs or serve
as artiﬁcial therapists (e.g., Barak & Grohol, 2011; Helgadóttir,
Menzies, Onslow, Packman, & O’Brian, 2009; Kenny, Parsons,
Gratch, & Rizzo, 2008). In sum, building a computational modelof affective cognition has many potential applications, and here
we have listed only a few that we feel are most promising.

5. Conclusion
Humans are lay psychologists that use theories to understandthe people around them, and often how, when, and why these people experience emotions. In this paper, we propose a framework forstudying affective cognition—reasoning about emotion—using acomputational, lay theory approach. This approach concentrates

on how observers reason about emotions, and our results show asurprising rationality and ﬂexibility in the way human observers
perform affective cognition. Speciﬁcally, the way that observers
reason between pairs of variables (emotions and the outcomes thatcause them), and the way that observers combine informationfrom multiple sources to infer emotion (emotional cue integration), can both be described using domain-general reasoning similar to other forms of cognition. Emotions are immenselycomplicated, and have complex interactions with other psychological states—these relationships are captured in a domain-speciﬁclay theory of emotions. Yet, the way that an observer reasons about
these complex emotions can be described and understood using
the same inferential processes that underlie reasoning in otherpsychological domains. It is our hope that the study of affective
cognition—treated as another form of cognition—will drive forward
theory and empirical studies in affective science, social cognition,and the crucial bridge between them.

Acknowledgements
This work was supported in part by an A⁄STAR National Science
Scholarship to D.C.O. and by a James S. McDonnell Foundation
Scholar Award and ONR Grant N00014-13-1-0788 to N.D.G.

Appendix A. Model selection and Principal Component Analysis
A.1. Experiment 1 model selection details
For the analysis in Experiment 1 (Section 2.1.3), we performedmodel selection from the full set of a priori speciﬁed features(win, PE, |PE|, Regret, Relief, logWinProb and nearMiss). In the tablebelow, we report coefﬁcients for the full model (with all 7 predictors). We can see from the full model that win, PE and |PE| predicts
the majority of all emotions. In particular, PE and |PE| predict all
except fear, and win predicts all except fear and anger. Fear is a
strange emotion in this paradigm, and so not much inference
should be drawn from the results for fear. The other notable emotion to note is surprise, which has signiﬁcant loadings on some ofthe other predictors. Note that we can also see interesting

Table A1
Table of coefﬁcients (with p-values in parentheses) for the models with all seven regressors. Total signiﬁcant refers to the total count of signiﬁcant predictors from the full models(i.e. from the table). Total after model selection indicates number of regressors remaining after model selection.

*
**
***

Win

PE

|PE|

RegretRelieflogWinProb

nearMissHappy

0.036 (<0.001***)0.027 (<0.001***)0.0055 (0.36)

0.0056 (0.24)

0.31 (0.032*)

1.5e05 (0.23)

Sad0.024 (<0.001***)

0.018 (0.022**)

0.020(<0.001***)0.028 (<0.001***)

0.0018 (0.80)

0.0026(0.64)
0.0011(0.81)
0.015 (0.016*)

0.10 (0.54)

2.6e06 (0.86)

0.10 (0.48)

2.1e06 (0.87)

1.94(<0.001***)
0.029 (0.82)

2.2e05 (0.22)

0.013 (0.87)

Anger0.0041 (0.31)

Surprise0.019 (0.029*)

Disgust0.0084 (0.022*)

Fear0.0021 (0.36)***

0.0013 (0.83)

0.027 (<0.001

0.027 (<0.001***)0.024 (0.003**)

0.026(<0.001***)
0.0039 (0.31)

0.022 (<0.001***)0.0075 (0.16)

0.0034 (0.063.)*

0.014(0.0025**)
0.023 (<0.001***)

0.00031(0.93)
0.0032 (0.71)

0.034 (<0.001

Disappointment

0.025 (<0.001***)

0.035(<0.001***)

Total signiﬁcantTotal after model
selection

6
5 (ex. anger, disgust,
fear)

7
8

p < .05.
p < .01.
p < .001.

)

0.025(<0.001***)
0.019 (0.035*)

Content)***

0.024 (0.0135 )78

0.0097 (0.19)
1
1 surprise

0.00063(0.88)
0.0012 (0.64)

3.3e06 (0.78)

0.0001(0.98)
0.0036(0.52)

0.26 (0.20)

1.3e05(0.071.)1.3e05 (0.48)

0.17 (0.32)

1.6e05 (0.33)

1
1 surprise2
2 surprise,
happy

0
0

159

D.C. Ong et al. / Cognition 143 (2015) 141–162
Table A2
Table of coefﬁcients (with p-values in parentheses) for the ﬁnal model used in thetext, with only win, PE, |PE|.

Happy
Sad
Anger
Surprise
DisgustFear
Content
Disappointment

Intercept

Win

PE

|PE|

4.515
(<0.001***)
3.380(<0.001***)1.721(<0.001***)
3.251(<0.001***)1.764(<0.001***)1.381(<0.001***)3.709(<0.001***)
4.724(<0.001***)0.0386(<0.001***)
0.0245(<0.001***)
0.00375(0.251)
0.0222(<0.001***)
0.00536(0.066.)
0.00225(0.23)
0.0321(<0.001***)
0.0292(<0.001***)0.0350(<0.001***)
0.0217(<0.001***)
0.0251(<0.001***)
0.012(0.013*)
0.0211(<0.001***)
0.00325(0.09.)
0.0221(<0.001***)
0.0441(<0.001***)

0.0177(<0.001***)0.0268(<0.001***)0.0267(<0.001***)0.0373(<0.001***)
0.0217(<0.001***)0.00342(0.037*)
0.0124(0.0025**)
0.0219(<0.001***)*

p < .05.
p < .01.
***p < .001.
**

differences between surprise and the other emotions, from thePrincipal Component Analysis (see below in Appendix A.3)
Next, we performed stepwise backward elimination using thestep function in the lmerTest package in R. Starting from the fullmodel, the variables are considered one at a time for elimination, and the function calculates an anova to compare themodel with and without the variable. According to the ﬁnal
model selection, the prediction error (PE) and its absolute value(|PE|) predicted was signiﬁcant in predicting all of the emotions(last row of Table A1). For the amount won, it was signiﬁcant
after selection for ﬁve emotions, and trending for one, so we
included it in the ﬁnal set of regressors. The last four regressors
did not have much explanatory power. Finally, in Table A2, wereport the coefﬁcients in the reduced model that we used inExperiments 1–2. (Note also that although we use the same
regressors (win, PE, |PE|) in Experiments 3 and 4, the coefﬁcients were ﬁtted to the data in Experiments 3 and 4
respectively.)

A.2. Combined analysis model selection details
We repeated this model selection analysis with the data from
Experiments 1, 3 and 4 (discussed in Section 3.3). The table forthe combined data is given in Table A3, with the coefﬁcients forthe full model as well as the total after model selection. The results
are qualitatively similar to those reported in Table A1 above. In thefull model, the amount won and |PE| predict 7 emotions, while PE
strangely predicts only 1 emotion, but after model selection,amount won and |PE| predict 7, and PE predicts 5. For illustration,
we have indicated in Table A3 the regressors that survived model
selection. Similar to above, we noticed that the pattern of coefﬁcient loadings on surprise and fear formed a different proﬁle ascompared to the other emotions.
The model selection results for Contentment was initially puzzling to us: if we include just win, PE and |PE| in the regression
model (i.e., the formula we used in Experiments 3 and 4), all three
are signiﬁcant predictors, as we expect from Table A1 (model
log-likelihood = 6439.2, AIC = 12,894, BIC = 12,943). However,starting from the full set of regressors and performing backward
step selection (which maximizes log-likelihood) yields win, |PE|,
Regret and Relief as signiﬁcant predictors (model log-likelihood =6438.7, AIC = 12,895, BIC = 12,950). It seems that in this particularcase for contentment, the two models are very similar in performance (with the log-likelihood of the latter model barely outperforming the former by 0.5, but the AIC and BIC of the former
model also barely outperforms the latter).
A.3. Principal Component Analysis details
We performed a Principal Component Analysis (PCA) of participants’ emotion ratings using R’s princomp function (discussed in
Section 2.1.4). The loadings of each of the PCs, as well as their standard deviations and proportion of total variance explained, aregiven in Table A4. In the main text, we limit the discussion to
the ﬁrst two PCs (see Fig. 4). The ﬁrst PC, with 59.1% of the variance, loads positively with positively-valenced emotions, and negatively with negatively-valenced emotions, resulting in ourinterpretation of PC1 representing ‘‘emotional valence’’. Although

Table A3
Similar to Table A1, now with combined data from Experiments 1, 3 and 4. Table of coefﬁcients (with p-values in parentheses) for the models with all seven regressors. Total
signiﬁcant refers to the total count of signiﬁcant predictors from the full models (i.e. from the table). Total after model selection indicates number of regressors remaining aftermodel selection.
Win

PE

|PE|

RegretRelieflogWinProb

nearMissHappy

0.036 (<0.001***)

0.021 (0.094.)

0.013 (0.23)

0.0067 (0.43)

0.32 (0.21)

Sad0.027 (0.012**)

0.014 (0.32)

0.017(0.0057**)
0.028(<0.001***)
0.028(<0.001***)
0.031(<0.001***)
0.023(<0.001***)0.0027 (0.13)

0.0038 (0.75)

0.0039(0.67)
0.0030(0.72)
0.016(0.052.)
0.0014(0.85)
0.0001(0.95)
0.0089(0.25)
0.0065(0.44)

0.19 (0.52)

3.3e05(0.003**)4.4e06 (0.73)

0.022 (0.93)

4.7e06 (0.69)

1.66(<0.001***)0.016 (0.95)

3.9e05 (0.80)
5.9e07 (0.96)

0.056 (0.46)

5.2e06 (0.47)

0.25 (0.31)

2.0e05 (0.21)

0.14 (0.58)

2.1e06 (0.88)

1
1 surprise

1
1 happyAnger0.0015 (0.038 )0.017 (0.171)

Surprise0.0042 (0.532)

0.019 (0.107)

Disgust0.015 (0.026*)

0.020 (0.093.)

Fear0.0083(<0.001***)0.028 (<0.001***)

0.0020 (0.58)

Disappointment

0.027(<0.001***)

0.030 (0.020*)

Total signiﬁcantTotal after model
selection

7
7 (ex. surprise)

1
5 (ex. fear, content,
surprise)

Content**

0.011 (0.34)

p < .05.
p < .01.
***p < .001.
Indicates the regressor was signiﬁcant after model selection.
**

0.015(0.011*)
0.023(<0.001***)77 (ex. fear)

0.0014 (0.89)**

0.033 (0.002 )0.0051 (0.59)
0.0031 (0.26)
0.010 (0.28)
0.013 (0.21)
1
2 surprisecontent

0
1 content

160

D.C. Ong et al. / Cognition 143 (2015) 141–162

Table A4
Results of the Principal Component Analysis. The top half shows the loadings of the various Principal Components (PCs) on the emotions. The values shown indicate the loading ofa particular PC on a particular emotion. Missing values were non-signiﬁcant (p > .05) loadings. The bottom half of the table shows the standard deviation and the proportion of
total variance explained by each PC.
Loadings

PC1

PC2

PC3

PC4

PC5

PC6

PC7

PC8

Happy
Sad
Anger
Surprise
DisgustFear
Content
Disappointment

0.473
0.394
0.2890.202
0.2430.121
0.127

0.215

0.507
0.316

0.775
0.309
0.115
0.199
0.226
0.1740.112
0.445
0.7300.520
0.127
0.103
0.808
0.113

0.250
0.181
0.451
0.2900.320
0.259
0.216
0.6330.260
0.654

0.219
0.451

0.6090.750

0.409
0.515

0.126
0.2600.337
0.745
0.308
0.151
0.3400.125

0.360

0.406

Standard deviation
Proportion of variance explained

4.68
0.591

2.42
0.158

1.91
0.098

1.39
0.052

1.15
0.035

1.06
0.030

0.84
0.019

0.77
0.016

the second PC explains much less (15.8%) of the variance, analyzing
this does provide some insight. The second PC loads in the same
direction with all the emotions; hence we interpret it to be tracking the magnitude of all the emotions. Note that in Table A4, theloadings of the second PC are all negative; however, the sign of a
PC is arbitrary—what is relevant are the relative signs of the different loadings. Thus, we might interpret PC2 as being proportional to‘‘emotional arousal’’ (or PC2 as ‘‘negative emotional arousal’’).
Subsequent PCs explain less than 10% of the variance.

Appendix B. Derivation of the multi-cue combination equation
Recall that in our model (Fig. 2), we assume that outcome ocauses emotion e, and emotion causes facial expressions f. If wetreat this model as a Bayesian network, we can factor the network
to get the probability of observing a certain combination of variables (o, e, f) into:

Pðo; e; fÞ ¼ PðfjeÞPðejoÞPðoÞ:W e are interested in inferring e given o and f, P(e|o, f). Using theidentity that:PðAjBÞ ¼PðA; BÞ
;PðBÞ

we can write the desired quantity as:Pðejo; fÞ ¼Pðo; e; fÞ PðfjeÞPðejoÞPðoÞ
¼:
Pðo; fÞ
Pðo; fÞ

Using Bayes’ rule, we can rewrite P(f|e) in terms of P(e|f):PðejfÞ ¼

PðfjeÞPðeÞ
:PðfÞ

Making this substitution for P(f|e) and rearranging, we arrive at:Pðejo; fÞ ¼PðejfÞPðfÞ PðejoÞPðoÞ PðejfÞPðejoÞ PðfÞPðoÞ
¼
PðeÞPðo; fÞ
PðeÞPðo; fÞ

The terms in the right-most parenthesis do not depend on e, and is aconstant for a ﬁxed (o, f) combination. Thus, we arrive at the crucial
cue-integration equation:Pðejo; fÞ /PðejfÞPðejoÞ
;PðeÞwhere the joint-cue posterior P(e|o, f) is proportional to the individual (emotion|cue) probabilities P(e|f) and P(e|o), and normalized by
the prior probability of the emotion P(e).

References
Alais, D., & Burr, D. (2004). The ventriloquist effect results from near-optimal
bimodal integration. Current Biology, 14(3), 257–262.
Aviezer, H., Hassin, R. R., Ryan, J., Grady, C., Susskind, J., Anderson, A., et al. (2008).
Angry, disgusted, or afraid? Studies on the malleability of emotion perception.
Psychological Science, 19(7), 724–732.
Aviezer, H., Trope, Y., & Todorov, A. (2012). Body cues, not facial expressions,discriminate between intense positive and negative emotions. Science,
338(6111), 1225–1229.
Bailenson, J. N., Yee, N., Merget, D., & Schroeder, R. (2006). The effect of behavioral
realism and form realism of real-time avatar faces on verbal disclosure,nonverbal disclosure, emotion recognition, and copresence in dyadic
interaction. Presence: Teleoperators and Virtual Environments, 15(4), 359–372.
Baker, C. L., Saxe, R., & Tenenbaum, J. B. (2009). Action understanding as inverse
planning. Cognition, 113(3), 329–349.
Barak, A., & Grohol, J. M. (2011). Current and future trends in Internet-supported
mental health interventions. Journal of Technology in Human Services, 29(3),155–196.
Baron-Cohen, S., Leslie, A. M., & Frith, U. (1985). Does the autistic child have a‘‘theory of mind’’? Cognition, 21(1), 37–46.
Barrett, L. F. (2006). Solving the emotion paradox: Categorization and theexperience of emotion. Personality and social psychology review, 10(1), 20–46.
Barrett, L. F., & Kensinger, E. A. (2010). Context is routinely encoded during emotion
perception. Psychological Science, 21(4), 595–599.
Barrett, L. F., Mesquita, B., & Gendron, M. (2011). Context in emotion perception.
Current Directions in Psychological Science, 20(5), 286–290.
Barrett, L. F., Mesquita, B., Ochsner, K. N., & Gross, J. J. (2007). The experience ofemotion. Annual Review of Psychology, 58, 373.
Barrett, L. F., & Russell, J. A. (1999). The structure of current affect controversies andemerging consensus. Current Directions in Psychological Science, 8(1), 10–14.
Beck, A. T. (1979). Cognitive therapy and the emotional disorders. Penguin.
Bem, D. J. (1972). Self-perception theory. Advances in Experimental Social Psychology,
6, 1–62.
Breazeal, C. L. (2004). Designing sociable robots. MIT press.
Brooks, R. A., Breazeal, C., Marjanović, M., Scassellati, B., & Williamson, M. M. (1999).
The Cog project: Building a humanoid robot. In Computation for metaphors,analogy, and agents (pp. 52–87). Berlin Heidelberg: Springer.
Buck, R. (1994). Social and emotional functions in facial expression andcommunication: The readout hypothesis. Biological Psychology, 38(2), 95–115.
Carroll, J. M., & Russell, J. A. (1996). Do facial expressions signal speciﬁc emotions?
Judging emotion from the face in context. Journal of Personality and Social
Psychology, 70(2), 205.
Carver, C. S., & Scheier, M. F. (2004). Self-regulation of action and affect. Handbook of
self-regulation: Research, theory, and applications, 13–39.
Chiu, C. Y., Hong, Y. Y., & Dweck, C. S. (1997). Lay dispositionism and implicit
theories of personality. Journal of Personality and Social Psychology, 73(1), 19.
Clark, H. H. (1996). Using language. Cambridge: Cambridge University Press.
Clore, G. L., Wyer, R. S., Jr., Dienes, B., Gasper, K., Gohm, C., & Isbell, L. (2001).
Affective feelings as feedback: Some cognitive consequences. Theories of moodand cognition: A users handbook, 27–62.
Darwin, C. (1872). The expression of the emotions in man and animals. London:
Murray.
de Gelder, B., & Bertelson, P. (2003). Multisensory integration, perception andecological validity. Trends in Cognitive Sciences, 7(10), 460–467.
Dennet, D. C. (1989). The intentional stance. MIT Press.
Ekman, P., Friesen, W. V., & Ellsworth, P. (1982). What emotion categories ordimensions can observers judge from facial behavior? In Emotion in the humanface (pp. 39–55). New York: Cambridge University Press.
Ellsworth, P. C., & Scherer, K. R. (2003). Appraisal processes in emotion. Handbook of
Affective Sciences, 572, V595.
Ernst, M. O., & Bülthoff, H. H. (2004). Merging the senses into a robust percept.
Trends in Cognitive Sciences, 8(4), 162–169.
Flavell, J. (1999). Cognitive development: Children’s knowledge about other minds.
Annual Review of Psychology, 50, 21–45.

D.C. Ong et al. / Cognition 143 (2015) 141–162
Frank, M. C., & Goodman, N. D. (2012). Predicting pragmatic reasoning in language
games. Science, 336(6084). 998-998.
Frijda, N. H. (1988). The laws of emotion. American Psychologist, 43(5), 349–358.
Geisler, W. S. (1989). Sequential ideal-observer analysis of visual discriminations.
Psychological Review, 96(2), 267.
Geisler, W. S. (2003). Ideal observer analysis. The Visual Neurosciences, 825–837.
Gilbert, D. (1998). Ordinary personology. In D. Gilbert, S. T. Fiske, & G. Lindzey (Eds.),
The handbook of social psychology (4th ed., pp. 89–150). New York: McGraw Hill.
Gilovich, T., & Medvec, V. H. (1995). The experience of regret: What, when, and why.
Psychological Review, 102(2), 379.
Goodenough, F. L., & Tinker, M. A. (1931). The relative potency of facial expression
and verbal description of stimulus in the judgment of emotion. Journal ofComparative Psychology, 12(4), 365.
Goodman, N. D. & Tenenbaum, J. B. (electronic). Probabilistic models of cognition.
<http://probmods.org> Retrieved 14.07.14.
Goodman, N. D., Baker, C. L., & Tenenbaum, J. B. (2009). Cause and intent: Social
reasoning in causal learning. In Proceedings of the thirty-ﬁrst annual conference ofthe cognitive science society.
Goodman, N. D., & Stuhlmüller, A. (2013). Knowledge and Implicature: Modelinglanguage understanding as social cognition. Topics in Cognitive Science, 5,
173–184.
Goodman, N. D., Ullman, T. D., & Tenenbaum, J. B. (2011). Learning a theory of
causality. Psychological Review, 118(1), 110.
Gopnik, A. (1993). Theories and illusions. Behavioral and Brain Sciences, 16(01),90–100.
Gopnik, A., & Meltzoff, A. N. (1997). Words, thoughts, and theories. The MIT Press.
Gopnik, A., & Wellman, H. (1992). Why the child’s theory of mind really is a theory.
Mind and Language, 7(1–2), 145–171.
Gratch, J., & Marsella, S. (2004). A domain-independent framework for modelingemotion. Cognitive Systems Research, 5(4), 269–306.
Grifﬁths, T. L., Kemp, C., & Tenenbaum, J. B. (2008). Bayesian models of cognition. In
Cambridge handbook of computational cognitive modeling (pp. 59–100).
Heider, F. (1958). The psychology of interpersonal relations. New York: Wiley.
Helgadóttir, F. D., Menzies, R. G., Onslow, M., Packman, A., & O’Brian, S. (2009).
Online CBT I: Bridging the gap between Eliza and modern online CBT treatment
packages. Behaviour Change, 26(04), 245–253.
Hudlicka, E. (2003). To feel or not to feel: The role of affect in human–computerinteraction. International Journal of Human-Computer Studies, 59(1), 1–32.
Huys, Q. J., Eshel, N., O’Nions, E., Sheridan, L., Dayan, P., & Roiser, J. P. (2012). Bonsai
trees in your head: How the Pavlovian system sculpts goal-directed choices bypruning decision trees. PLoS Computational Biology, 8(3), e1002410.
Jayaratne, T. E., Ybarra, O., Sheldon, J. P., Brown, T. N., Feldbaum, M., Pfeffer, C. A.,
et al. (2006). White Americans’ genetic lay theories of race differences andsexual orientation: Their relationship with prejudice toward Blacks, and gay
men and lesbians. Group Processes & Intergroup Relations, 9(1), 77–94.
Jones, E. E., & Davis, K. E. (1965). From acts to dispositions the attribution process inperson perception. Advances in Experimental Social Psychology, 2, 219–266.
Jones, E. E., & Nisbett, R. E. (1971). The actor and the observer: Divergent perceptions of
the causes of behavior. Morristown, NJ: General Learning Press, p. 16.
Kahneman, D., & Tversky, A. (1979). Prospect theory: An analysis of decision underrisk. Econometrica, 47(2), 263–292.
Kahneman, D., & Tversky, A. (1984). Choices, values, and frames. American
Psychologist, 39(4), 341.
Kahneman, D., & Tversky, A. (1982). The simulation heuristic. In D. Kahneman, P.
Slovic, & A. Tversky (Eds.), Judgment under uncertainty: Heuristics and biases(pp. 208). 201: Cambridge University Press.
Kao, J. T., Wu, J. Y., Bergen, L., & Goodman, N. D. (2014). Nonliteral understanding of
number words. Proceedings of the National Academy of Sciences. http://dx.doi.org/10.1073/pnas.1407479111.
Kelley, H. H. (1973). The processes of causal attribution. American Psychologist,
28(2), 107.
Kelley, H. H., & Michela, J. L. (1980). Attribution theory and research. Annual Review
of Psychology, 31(1), 457–501.
Kenny, P., Parsons, T., Gratch, J., & Rizzo, A. (2008). Virtual humans for assisted
health care. In Proceedings of the 1st international conference on Pervasive
Technologies related to assistive environments (pp. 6). ACM.
Kersten, D., Mamassian, P., & Yuille, A. (2004). Object perception as Bayesianinference. Annual Review of Psychology, 55, 271–304.
Knill, D. C. (2007). Robust cue integration: A Bayesian model and evidence fromcue-conﬂict studies with stereoscopic and ﬁgure cues to slant. Journal of Vision,
7(7), 5.
Knutson, B., Taylor, J., Kaufman, M., Peterson, R., & Glover, G. (2005). Distributedneural representation of expected value. The Journal of Neuroscience, 25(19),4806–4812.
Kuppens, P., Tuerlinckx, F., Russell, J., & Barrett, L. (2012). The relation between
valence and arousal in subjective experience. Psychological Bulletin.
Leslie, A. M., Friedman, O., & German, T. P. (2004). Core mechanisms in ‘‘theory ofmind’’. Trends in Cognitive Sciences, 8(12), 528–533.
Lindquist, K. A., & Barrett, L. F. (2008). Constructing emotion the experience of fear
as a conceptual act. Psychological Science, 19(9), 898–903.
Lindquist, K. A., Barrett, L. F., Bliss-Moreau, E., & Russell, J. A. (2006). Language andthe perception of emotion. Emotion, 6(1), 125.
Liu, Z., Knill, D. C., & Kersten, D. (1995). Object classiﬁcation for human and ideal
observers. Vision Research, 35(4), 549–568.
Loomes, G., & Sugden, R. (1982). Regret theory: An alternative theory of rational
choice under uncertainty. The Economic Journal, 92(368), 805–824.

161

Malle, B. F., & Knobe, J. (1997). The folk concept of intentionality. Journal of
Experimental Social Psychology, 33(2), 101–121.
Marr, D. (1982). Vision. San Francisco: W. H. Freeman.
Masuda, T., Ellsworth, P. C., Mesquita, B., Leu, J., Tanida, S., & Van de Veerdonk, E.
(2008). Placing the face in context: Cultural differences in the perception of
facial emotion. Journal of Personality and Social Psychology, 94(3), 365.
Matsumoto, D., Keltner, D., Shiota, M. N., O’Sullivan, M., & Frank, M. (2008). Facial
expressions of emotion. Handbook of emotions, 3, 211–234.
Medvec, V. H., Madey, S. F., & Gilovich, T. (1995). When less is more: Counterfactual
thinking and satisfaction among Olympic medalists. Journal of Personality andSocial Psychology, 69. 603-603.
Mondloch, C. J. (2012). Sad or fearful? The inﬂuence of body posture on adults’ andchildren’s perception of facial displays of emotion. Journal of Experimental Child
Psychology, 111(2), 180–196.
Mondloch, C. J., Horner, M., & Mian, J. (2013). Wide eyes and drooping arms: Adultlike congruency effects emerge early in the development of sensitivity toemotional faces and body postures. Journal of Experimental Child Psychology,
114(2), 203–216.
Montague, P. R., Dolan, R. J., Friston, K. J., & Dayan, P. (2012). Computational
psychiatry. Trends in Cognitive Sciences, 16(1), 72–80.
Murphy, G. L., & Medin, D. L. (1985). The role of theories in conceptual coherence.
Psychological Review, 92(3), 289.
Nakamura, M., Buck, R., & Kenny, D. A. (1990). Relative contributions of expressive
behavior and contextual information to the judgment of the emotional state ofanother. Journal of Personality and Social Psychology, 59(5), 1032.
Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports
on mental processes. Psychological Review, 84(3), 231.
Ortony, A., Clore, G. L., & Collins, A. (1988). The cognitive structure of emotions. New
York: Cambridge University Press.
Osgood, C. E., Suci, G. J., & Tannenbaum, P. H. (1957). The measurement of meaning(Vol. 47). Urbana: University of Illinois Press.
Picard, R. W. (1997). Affective computing. MIT press.
Pinker, S. (1999). How the mind works. Annals of the New York Academy of Sciences,882(1), 119–127.
Raij, A. B., Johnsen, K., Dickerson, R. F., Lok, B. C., Cohen, M. S., Duerson, M., et al.
(2007). Comparing interpersonal interactions with a virtual human to thosewith a real human. IEEE Transactions on Visualization and Computer Graphics,
13(3), 443–457.
Ross, L. (1977). The intuitive psychologist and his shortcomings: Distortions in theattribution process. Advances in Experimental Social Psychology, 10, 173–220.
Ross, L., & Nisbett, R. E. (1991). The person and the situation: Perspectives of social
psychology. Mcgraw-Hill Book Company.
Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social
Psychology, 39(6), 1161.
Russell, J. A., Bachorowski, J. A., & Fernández-Dols, J. M. (2003). Facial and vocal
expressions of emotion. Annual Review of Psychology, 54(1), 329–349.
Russell, J. A., & Barrett, L. F. (1999). Core affect, prototypical emotional episodes, andother things called emotion: Dissecting the elephant. Journal of Personality andSocial Psychology, 76(5), 805.
Schachter, S., & Singer, J. (1962). Cognitive, social, and physiological determinants ofemotional state. Psychological Review, 69(5), 379.
Scherer, K. R., Schorr, A. E., & Johnstone, T. E. (2001). Appraisal processes in emotion:
Theory, methods, research. Oxford University Press.
Schlosberg, H. (1954). Three dimensions of emotion. Psychological Review,61(2), 81.
Shams, L., Kamitani, Y., & Shimojo, S. (2000). What you see is what you hear. Nature.
Shannon, C. E., & Weaver, W. (1949). The mathematical theory of communication.
Urbana, Illinois: The University of Illinois Press.
Sharp, C., Monterosso, J., & Montague, P. R. (2012). Neuroeconomics: A bridge fortranslational research. Biological Psychiatry, 72(2), 87–92.
Shiffrin, R. M., & Steyvers, M. (1997). A model for recognition memory: REM—
retrieving effectively from memory. Psychonomic Bulletin & Review, 4(2),145–166.
Smith, M. L., Cottrell, G. W., Gosselin, F., & Schyns, P. G. (2005). Transmitting anddecoding facial expressions. Psychological Science, 16(3), 184–189.
Smith, C. A., & Ellsworth, P. C. (1985). Patterns of cognitive appraisal in emotion.
Journal of Personality and Social Psychology, 48(4), 813.
Smith, C. A., & Lazarus, R. S. (1993). Appraisal components, core relational themes,
and the emotions. Cognition & Emotion, 7(3–4), 233–269.
Swartout, W. R., Gratch, J., Hill, R. W., Jr, Hovy, E., Marsella, S., Rickel, J., et al. (2006).
Toward virtual humans. AI Magazine, 27(2), 96.
Sweeny, K., & Vohs, K. D. (2012). On near misses and completed tasks the nature of
relief. Psychological Science, 23(5), 464–468.
Teigen, K. H. (1996). Luck: The art of a near miss. Scandinavian Journal of Psychology,
37, 156–171.
Tomasello, M., Carpenter, M., Call, J., Behne, T., & Moll, H. (2005). Understanding andsharing intentions: The origins of cultural cognition. Behavioral and Brain
Sciences, 28(5), 675–691. discussion 691–735.
Tomkins, S. S. (1962). Affect, imagery, consciousness (Vol. 1). New York: Springer.
Van den Stock, J., Righart, R., & de Gelder, B. (2007). Body expressions inﬂuence
recognition of emotions in the face and voice. Emotion, 7(3), 487.
Wallbott, H. G. (1988). Faces in context: The relative importance of facial expression
and context information in determining emotion attributions. In K. R. Scherer(Ed.), Facets of emotion (pp. 139–160). Hillsdale, NJ: Erlbaum.
Watson, S. G. (1972). Judgment of emotion from facial and contextual cue
combinations. Journal of Personality and Social Psychology, 24(3), 334.

162

D.C. Ong et al. / Cognition 143 (2015) 141–162

Watson, D., & Tellegen, A. (1985). Toward a consensual structure of mood.
Psychological Bulletin, 98(2), 219.
Wehrle, T., & Scherer, K. R. (2001). Towards computational modeling of appraisal
theories.
Weiss, Y., Simoncelli, E. P., & Adelson, E. H. (2002). Motion illusions as optimal
percepts. Nature Neuroscience, 5(6), 598–604.
Welch, R. B., & Warren, D. H. (1980). Immediate perceptual response to intersensory
discrepancy. Psychological Bulletin, 88(3), 638.

Wellman, H. M., & Gelman, S. A. (1992). Cognitive development: Foundational
theories of core domains. Annual Review of Psychology, 43(1), 337–375.
Zaki, J. (2013). Cue integration: A common framework for physical perception andsocial cognition. Perspectives on Psychological Science, 8(3), 296–312.
Zaki, J., & Ochsner, K. (2011). Reintegrating the study of accuracy into social
cognition research. Psychological Inquiry, 22(3), 159–182.