Oct 08, 2013

American Economic Association

Cognitive Reflection and Decision Making
Author(s): Shane Frederick
Reviewed work(s):Source: The Journal of Economic Perspectives, Vol. 19, No. 4 (Autumn, 2005), pp. 25-42
Published by: American Economic Association
Stable URL: http://www.jstor.org/stable/4134953 .
Accessed: 13/11/2011 13:59
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .
http://www.jstor.org/page/info/about/policies/terms.jsp
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range ofcontent in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new formsof scholarship. For more information about JSTOR, please contact support@jstor.org.

American Economic Association is collaborating with JSTOR to digitize, preserve and extend access to The
Journal of Economic Perspectives.

http://www.jstor.org

Journal of EconomicPerspectives-Volume19, Number4--Fall 2005-Pages 25-42

Cognitive Reflection and Decision Making

Shane Frederick

People with higher cognitiveability (or "IQ") differ from those with lower

cognitive ability in a variety of important and unimportant ways. On
average, they live longer, earn more, have larger working memories, faster
reaction times and are moresusceptible to visual illusions (Jensen, 1998). Despite
the diversity of phenomena related to IQ, few have attempted to understand-or
even describe-its influences on judgment and decision making. Studies on time
preference, risk preference, probability weighting, ambiguity aversion, endowment
effects, anchoring and other widely researched topics rarely make any reference tothe possible effects of cognitive abilities (or cognitive traits).
Decision researchers may neglect cognitive ability because they are moreinterested in the averageeffect of some experimental manipulation. On this view,
individual differences (in intelligence or anything else) are regarded as a
nuisance-as just another source of "unexplained" variance. Second, most studies
are conducted on college undergraduates, who are widely perceived as fairly
homogenous. Third, characterizing performance differences on cognitive tasks
requires terms ("IQ"and "aptitudes"and such) that many object to because of theirassociation with discriminatory policies. In short, researchers may be reluctant tostudy something they do not find interesting, that is not perceived to vary muchwithin the subject pool conveniently obtained, and that will just get them into
trouble anyway.
But as Lubinski and Humphreys (1997) note, a neglected aspect does notcease to operate because it is neglected, and there is no good reason for ignoring
the possibilitythat general intelligence or various more specific cognitive abilities areimportant causal determinants of decision making. To provoke interest in thisSShane Frederickis Assistant Professorof ManagementScience,Sloan Schoolof ManageMassachusetts.His e-mailaddressis
ment,MassachusettsInstituteof Technology,Cambridge(shanefre@mit.edu).

26

Journal of EconomicPerspectives

neglected topic, this paper introduces a three-item "Cognitive Reflection Test"(CRT) as a simple measure of one type of cognitive ability. I will show that CRT
scores are predictive of the types of choices that feature prominently in tests ofdecision-making theories, like expected utility theory and prospect theory. Indeed,
the relation is sometimes so strong that the preferences themselves effectivelyfunction as expressions of cognitive ability-an empirical fact begging for a theoretical explanation.
After introducing the CRT, I examine its relations with two important decisionmaking characteristics: time preference and risk preference. The CRT is thencompared with other measures of cognitive ability or cognitive "style,"including the
Wonderlic Personnel Test (WPT), the Need For Cognition scale (NFC) and selfreported SAT and ACT scores. The CRT exhibits considerable difference between
men and women, and I discuss how this relates to sex differences in time and risk
preferences. The final section discusses the interpretation of correlations between
cognitive abilities and decision-making characteristics.

The Cognitive Reflection Test (CRT)
Many researchers have emphasized the distinction between two types of cognitive processes: those executed quickly with little conscious deliberation and thosethat are slower and more reflective (Epstein, 1994; Sloman, 1996; Chaiken andTrope, 1999; Kahneman and Frederick, 2002). Stanovich and West (2000) called
these "System 1" and "System 2" processes, respectively. System 1 processes occur
spontaneously and do not require or consume much attention. Recognizing thatthe face of the person entering the classroom belongs to your math teacher involves
System 1 processes-it occurs instantly and effortlessly and is unaffected by intellect, alertness, motivation or the difficulty of the math problem being attempted at
the time. Conversely, finding /19163 to two decimal places without a calculator
involves System 2 processes-mental operations requiring effort, motivation, concentration, and the execution of learned rules.'The problem /19163 allows no role for System 1. No number spontaneously
springs to mind as a possible answer. Someone with knowledge of an algorithm andthe motivation to execute it can arrive at the exact answer (138.43), but the
problem offers no intuitive solution.
By contrast, consider this problem:
A bat and a ball cost $1.10. The bat costs $1.00 more than the ball.
How much does the ball cost? _ cents
Here, an intuitive answer doesspring quickly to mind: "10 cents." But this "impulsive" answer is wrong. Anyone who reflects upon it for even a moment would
1 For a discussion of the distinction between System 1 and System 2 in the context of choice heuristics,
see Frederick (2002).

Shane Frederick 27

Figure 1
The Cognitive Reflection Test (CRT)(1) A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball.
How much does the ball cost?
cents(2) If it takes 5 machines 5 minutes to make 5 widgets, how long would it take
100 machines to make 100 widgets?
minutes(3) In a lake, there is a patch of lily pads. Every day, the patch doubles in size.
If it takes 48 days for the patch to cover the entire lake, how long would ittake for the patch to cover half of the lake?
days

recognize that the difference between $1.00 and 10 cents is only 90 cents, not $1.00
as the problem stipulates. In this case, catching that error is tantamount to solving
the problem, since nearly everyone who does not respond "10 cents" does, in fact,give the correct response: "5 cents."
In a study conducted at Princeton, which measured time preferences using
both real and hypothetical rewards, those answering "10 cents" were found to besignificantly less patient than those answering "5 cents."2 Motivated by this result,two other problems found to yield impulsive erroneous responses were included
with the "bat and ball" problem to form a simple, three-item "Cognitive Reflection
Test" (CRT), shown in Figure 1. The three items on the CRT are "easy"in the sensethat their solution is easily understood when explained, yet reaching the correct
answer often requires the suppression of an erroneous answer that springs "impulsively" to mind.
The proposition that the three CRT problems generate an incorrect "intuitive"
answer is supported by several facts. First, among all the possible wrong answers
people could give, the posited intuitive answers (10, 100 and 24) dominate. Second,even among those responding correctly, the wrong answer was often considered
first, as is apparent from introspection, verbal reports and scribbles in the margin(for example, 10 cents was often crossed out next to 5 cents, but never the otherway around). Third, when asked to judge problem difficulty (by estimating the
proportion of otherrespondents who would correctly solve them), respondents who
missed the problems thought they were easier than the respondents who solved
them. For example, those who answered 10 cents to the "bat and ball" problem
estimated that 92 percent of people would correctly solve it, whereas those who
answered "5 cents" estimated that "only" 62 percent would. (Both were considerable overestimates.) Presumably, the "5 cents" people had mentally crossed out
10 cents and knew that not everyone would do this, whereas the "10 cents" people
2 The "bat and ball" problem was

subsequently used by Nagin and Pogarsky (2003) in a laboratory
experiment on cheating. When respondents could obtain a $20 reward for correctly answering six trivia
questions, those answering 10 cents were significantly more likely to defy the experimenter's request tocomplete the task without looking at the answers.

28

Journal of EconomicPerspectives

thought the problem was too easy to miss. Fourth, respondents do much better on
analogous problems that invite more computation. For example, respondents missthe "bat and ball" problem far more often than they miss the "banana and bagel"problem: "Abanana and a bagel cost 37 cents. The banana costs 13 cents more thanthe bagel. How much does the bagel cost?"
The CRT was administered to 3,428 respondents in 35 separate studies over a
26-month period beginning in January 2003. Most respondents were undergraduates at various universities in the midwest and northeast who were paid $8 tocomplete a 45-minute questionnaire that included the CRT and measures ofvarious decision-making characteristics, like time and risk preferences.3 On thepage on which the CRT appeared, respondents were told only: "Below are several
problems that vary in difficulty. Try to answer as many as you can."Table 1 shows the mean scores at each location and the percentage answering
0, 1, 2 or 3 items correctly. Most of the analyses that follow compare the "low"group(those who scored 0 out of 3) with the "high" group (those who scored 3 out of 3).
The two "intermediate" groups (those who scored a 1 or 2) typically fell between
the two extreme groups on whatever dependent measure was analyzed. Thus,
focusing attention on the two "extreme" groups simplifies the exposition andanalysis without affecting the conclusions.
Since more of the respondents were college students from selective schools,
the two "extreme" groups that formed the basis for most statistical comparisonswere far more similar in cognitive abilities than two extreme groups formed fromthe general population. Thus, the group differences reported here likely understate
the differences that would have been observed if a more representative sample had
been used.

Cognitive Reflection and Time Preferences
The notion that more intelligent people are more patient-that they devalueor "discount" future rewards less-has prevailed for some time. For example, in his
New Principlesof Political Economy(1834, pp. 57), Rae writes: "The strength of theintellectual powers, giving rise to reasoning and reflective habits... brings before us
the future... in its legitimate force, and urge the propriety of providing for it."
The widely presumed relation between cognitive ability and patience has beentested in several studies, although rather unsystematically. Melikian (1959) asked
children from five to twelve years of age to draw a picture of a man, which they
could exchange for either 10 fils (about 3 cents) or for a "promissory note"redeemable for 20 fils two days later. Those who opted for the promissory notescored slightly higher on an intelligence test based on an assessment of those

3 There were three exceptions to this: 1) the participants from Carnegie Mellon University completed
the survey as part of class; 2) the 4th ofJuly participants received "only"a frozen ice cream bar; and 3) the
participants from the web study were unpaid, although they were entered into a lottery for iPods andother prizes.

CognitiveReflectionand DecisionMaking 29

Table 1CRT Scores, by Location
Percentagescoring0, 1, 2 or 3

Locationsat which data werecollected

Mean CRT score

"Low"
0

1

2

"High"
3N =Massachusetts Institute of Technology
Princeton University
Boston fireworks displaya
Carnegie Mellon University
Harvard Universityb
University of Michigan: Ann Arbor
Web-based studiesc
Bowling Green University
University of Michigan: Dearborn
Michigan State University
University of Toledo

2.18
1.63
1.53
1.51
1.43
1.18
1.10
0.87
0.83
0.79
0.57

7%
18%
24%
25%
20%
31%
39%
50%
51%
49%
64%

16%
27%
24%
25%
37%
33%
25%
25%
22%
29%
21%

30%
28%
26%
25%
24%
23%
22%
13%
21%
16%
10%

48%
26%
26%
25%
20%
14%
13%
12%
6%
6%
5%

61
121
195
746
51
1267
525
52
154
118
138

1.24

33%

28%

23%

17%

3428

Overall

Notes:a Respondents in this study were people picnicking along the banks of the Charles River prior to
the July 4th fireworks display. Their ages ranged from 15 to 63, with a mean of 24. Many of the younger
participants were presumably students at a college in the Boston or Cambridge area. Most completed thesurvey in small groups of friends or family. Although they were requested not to discuss it until everyonein their group had completed it, some may have. (This, presumably, would elevate the CRT scores
relative to most of the other studies in which participation was more closely supervised.)b The participants in this study were all members of a student choir group, which was predominately
female. Unlike the other locations in which the numbers of men and women were comparable, 42 of 51
participants in this study were women.
cThese were participants in two online studies, consisting of both college students and others whosee-mail addresses were obtained from online retailers.

drawings.4 Funder and Block (1989) paid 14 year-olds to participate in six experimental sessions. For each of the first five sessions, they could choose between
receiving $4 or foregoing ("investing") their $4 payment for $4.80 in the sixth andfinal session. The teenagers with higher IQs chose to invest more of their money.
In a follow-up to an extensive series of experiments investigating the ability ofpreschool children to delay gratification (Mischel, 1974), Shoda, Mischel andPeake (1990) found that the children who had waited longer before succumbing to
the impulse to take an immediately available inferior reward scored higher on their
SATs taken over a decade later. Similarly, Parker and Fischhoff (2005) found thatscores on a vocabulary test taken around age eleven predicted the individual's
tendency, at around age 18, to prefer a larger later reward over a smaller sooner
one (for example, $120 in four weeks to $100 tomorrow). Using small real rewards,
Benjamin and Shapiro (2005) found that respondents with higher SAT math scores
4 Given the relativelywide range of ages in this study, it remains unclear whether this relation is

attributableto intelligence,perse,or to age, whichmight correlatewith the developmentof artisticskill
or patience or trustor some other specifictraitthat can be distinguishedfrom cognitiveability.

30

Journal of EconomicPerspectives(or their Chilean equivalent) were more likely to choose a larger later reward overa smaller sooner one (for example, to prefer a postdated check for $5.05 over a$5.00 check that can be immediately cashed). However, Monterosso et al. (2001)
found no relation between the IQ of cocaine addicts and their imputed discountrates, and Kirby, Winston and Santiesteban (2005) found no reliable relation
between students' SAT scores and the amount they would bid for a delayedmonetary reward (although they did find that college grade point averages correlated positively with those bids).
Collectively, these studies support the view that cognitive ability and time
preference are somehow connected, though they have not generally focused on thetypes of intertemporal decisions over which cognitive ability exerts influence, nor
explained why it does so.5 Toward this end, I examined the relation between CRTscores and various items intended to measure different aspects of "time preference." As shown in Table 2, these included several hypothetical choices between an
immediate reward and a larger delayed reward (items a through e), an immediate
reward and a sequence of delayed rewards (items f through h), a shorter more
immediate massage and longer more delayed massage (item i) and a smaller
immediate loss or a larger delayed loss (items j and k).6 Item 1 asked respondents
to state their maximum willingness to pay to have a book shipped overnight rather
than waiting two weeks. Item m involved real money. Through a series of choices,
respondents specified the smallest amount of money in four days that they would
prefer to $170 in two months, and one of them was selected to actually receive oneof their choices. Items n through q asked respondents to report their impulsivity,
procrastination, preoccupation with their future and concerns about inflation on
an 11-point scale ranging from -5 (much less than the average person taking thissurvey today) to + 5 (much more than the average person taking this survey today).7
Table 2 shows the responses of the low and high CRT groups for each of the
17 items. The reported value is either the percentage choosing the patient option
or the mean response. The subscripts are the total number of respondents in the
low and high CRT groups who answered that item. The rightmost column reports
the level of statistical significance of group differences-the p-values from a chisquare test (for dichotomous responses) or a t-test (for continuous responses).
Those who scored higher on the CRT were generally more "patient"; theirdecisions implied lower discount rates. For short-term choices between monetary
rewards, the high CRT group was much more inclined to choose the later larger
5 Shoda, Mischel and Peake (1990) examined preschoolers' willingness to wait (for additional marshmallows and pretzels and such) under four experimental conditions. They found that patience predicted SAT scores in only one of their four conditions-when the attractive but inferior reward wasvisually exposed and no distraction technique (such as "think fun") was suggested. In the other three
conditions, patient behavior was actually negativelycorrelated with subsequent SAT scores.
6 I assumed that delaying the extraction of a tooth involved a largerdelayed loss, because during theintervening two weeks, one will suffer additional toothache pain, or additional disutility from dreading
the forthcoming extraction pain, and that the only reason for not doing it immediately was that future
pain was discounted relative to immediate pain.

7Among the itemsin Table2, men weremore patientfor itemsc, k and 1,and theyworriedmore about
inflation.There were no significantdifferencesbetweenmen and women for any other item.

Shane Frederick 31

Table2
Intertemporal Behavior for Low and High CRT Groups(percentagechoosingpatient option or mean response)
CRT group
Item

IntertemporalChoiceorJudgmentab$3400 this month or $3800 next month$100 now or $140 next year
$100 now or $1100 in 10 years$9 now or $100 in 10 years$40 immediately or $1000 in 10 years$100 now or $20 every year for 7 years$400 now or $100 every year for 10 years$1000 now or $100 every year for 25 years
30 min. massage in 2 weeks or 45 min. massage in Nov.
Lose $1000 this year or lose $2000 next year
Tooth pulled today or tooth pulled in 2 weeks
Willingness to pay for overnight shipping of chosen book
Smallest amount in 4 days preferred to $170 in 2 months
How impulsive are you?
How much do you tend to procrastinate?
How much do you think about your future?
How much do you worry about inflation?

c

d
e
f
g
h
i

j
k
1
m
n
o
p
q

Low
35%61,
22%40947%28340%364

50%13528%6o
64%44
52%29528%272

78%16659%43o$4.54150$11672+1.01110+1.05110+2.49110-1.16110

High

Stat. Signif

p < 0.0001
p < 0.0001
57%208p < 0.05
46%277p < 0.10
n.s.
59%83
n.s.
43%28
n.s.
72%43
n.s.
49%99
n.s.
27%126
n.s.
73%86
n.s.
65%242$2.18163 p < 0.0001$13382p < 0.01-0.2147p < 0.001
n.s.
+1.0647+1.6447 p < 0.01
+0.1147p < 0.01
60%19637%297

reward (see items a and b). However, for choices involving longer horizons (items
c through h), temporal preferences were weakly related or unrelated to CRT scores.
A tentative explanation for these results is as follows: a thoughtful respondent
can find good reasons for discounting future monetary outcomes at rates exceeding
the prevailing interest rate-the promiser could default, one may be predictably
wealthier in the future (with correspondingly diminished marginal utility forfurther wealth gains), interest rates could increase (which increases the opportunity cost of foregoing the immediate reward), and inflation could reduce the future
rewards' real value (if the stated amounts are interpreted as being denominated in
nominal units)." Collectively, these reasons could, for example, justify choosing $9
now over $100 in 10 years (item d), even though the implied discount rate of sucha choice (27 percent), exceeds market interest rates. However, such reasons are notsufficiently compelling tojustify choosing $3400 this month over $3800 next month(which implies an annual discount rate of 280 percent). Hence, one observesconsiderable differences between CRT groups for choices like those in items a andb, where more careful deliberation or "cognitive reflection" should argue strongly
in favor of the later larger reward, but negligible differences for many of the otheritems, for which additional reflection would not make such a strong case for thelarger later reward (although one might argue that additional reflection shoulds Frederick, Loewenstein and O'Donoghue (2002) offer a detailed and extended discussion of theconceptual dissection of imputed discount rates and discuss many reasons why choices between
monetary rewards are problematic for measuring pure time preference.

32

Journal of EconomicPerspectives

reveal the wisdom of choosing the delayed 45-minute massage, since one will likely
still be alive, still be stressed and sore, still like massages, and still derive greater
benefits from longer ones).
It appears that greater cognitive reflection fosters the recognition or appreciation of considerations favoring the later larger reward (like the degree to which
the implied interest rate exceeds the rate offered by the market). However, itremains unclear whether cognitive reflection also influences other determinants ofintertemporal choices (like pure time preference). CRT scores were unrelated topreferences for the massage and tooth-pull items, which were intended as measuresof pure time preference. On the other hand, those in the low CRT group (the"cognitively impulsive") were willing to pay significantly more for the overnight
shipping of a chosen book (item 1), which doesseem like an expression of an aspectof puretime preference (the psychological "pain"of waiting for something desired).
Thus, despite the wide variety of items included to help address this issue,
further resolution of the types of psychological characteristics associated withcognitive reflection (and other cognitive abilities) is still required. Toward this goal,
respondents in some of the later studies were also asked to report several personality characteristics that seemed relevant to intertemporal choices (items n throughq). The self-perceived tendency to procrastinate was unrelated to CRT scores (bothgroups thought that they procrastinate more than their peers). However, the high
CRT group perceived themselves to be significantly lessimpulsive, moreconcerned
about inflation and (curiously) less preoccupied with their future. The inflation
result supports the idea that the high-scoring groups are more likely to consider
such background factors in their choices between temporally separated monetary
rewards. Its interpretation, however, is ambiguous, since it implies a considerationof future conditions, but would be a justification for choosing the proximate
reward.

Cognitive Reflection and Risk Preferences
In the domain of risk preferences, there is no widely shared presumption
about the influences of cognitive ability and almost no research on the topic.
Donkers, Melenberg and van Soest (2001) found that more educated respondents
were more tolerant of risk in hypothetical gambles: for example, they were more
likely to prefer an 80 percent chance of45 florins (about $23) over a sure 30 florins(about $15). Benjamin and Shapiro (2005) found that students with higher scoreson the math section of the SAT (or its Chilean equivalent) were more likely tochoose according to expected value for real decisions involving small stakes (forexample, they were more likely to prefer a 50 percent chance to win $1.05 over asure 50 cents).
To assess the relation between CRT and risk preferences, I included several
measures of risk preferences in my questionnaires, including choices between acertain gain (or loss) and some probability of a larger gain (or loss). For some

CognitiveReflectionand DecisionMaking 33

items, expected value was maximized by choosing the gamble, and for some it was
maximized by choosing the certain outcome.
The results are shown in Table 3a. In the domain of gains, the high CRT group
was more willing to gamble-particularly when the gamble had higher expectedvalue (top panel), but, notably, even when it did not (middle panel). If all five items
from the middle panel of Table 3a are aggregated, the high CRT group gambled
significantly more often than the low CRT group (31 percent versus 19 percent;
X2 = 8.82; p < 0.01). This suggests that the correlation between cognitive ability
and risk taking in gains is not due solely to a greater disposition to compute
expected value or to adopt that as the choice criterion.9 For items involving losses(lower panel), the high CRT group was less risk seeking; they were more willing
accept a sure loss to avoid playing a gamble with lower (more negative) expectedvalue.
Two pairs of items (d versus o and h versus r) were reflections of one anotherin the domain of gains and losses. Prospect theory predicts that people will be more
willing to take risks to avoid losses than to achieve gains; that respondents willswitch from risk aversion to risk seeking when the valence of a gamble (or "prospect") changes from positive to negative (Kahneman and Tversky, 1979). Though
this is spectacularly true for the low CRT group, who are much more willing togamble in the domain of losses than in the domain of gains, there is no such
reflection effect among the high CRT group, as shown in Table 3b. This result
starkly shows the importance of considering cognitive ability when evaluating thedescriptive validity of a theory of decision making.1'

Is the CRT Just Another IQ test?
Of the 3,428 respondents who completed the three-item CRT, many also
completed one or more additional cognitive measures: 921 completed the Wonderlic Personnel Test (WPT)-a 12-minute, 50-item test used by the National

the gamble was not popular among eithergroup for any of the "anti-expected-value"
gambles, since risk aversion and expected value both militate against it. However, any factors favoring
the gamble over the sure thing (for example, valuing the excitement of gambling or dismissing the sureamount as negligibly small) would be more likely to tip preferences in favor of the gamble among thoseless averse to it (the high CRT group, asjudged from items a through h). The gambles in items i throughm were designed, in part, to have some chance of being chosen (the sure amounts were small, and the
expected values of the gambles were typically close to the sure amount). Including choices in which thegambles lacked these properties (for example, offering a choice between $4,000 for sure and a 50
percent chance of $5000) would be pointless, because nearly everyone would reject the gamble, leaving
no response variance to analyze. Item i comes close to illustrating this point.
9 As expected,

10 Although

the descriptive

accuracy of expectedutility theory markedly improves for respondents

withhigher scores, it cannot explain why a 75 percent chance of $200 is frequently rejected in favor of a sure
$100, across all levels of cognitive ability, since this is a small fraction of one's wealth, and even a concave
utility function is approximately linear over small changes (Rabin, 2000).

34

Journal of EconomicPerspectives

Table3aRisk Seeking Behavior among Low and High CRT Groups
Percentagechoosingriskieroption
Item
a
b
c
d
e
f
g
h

Certaingains vs. Higher expectedvalue gambles$1,000 for sure or a 90% chance of $5,000$100 for sure or a 90% chance of $500
$1,000 for sure or a 75% chance of $4,000$100 for sure or a 75% chance of $200
$100 for sure or a 75% chance of $150
$100 for sure or a 50% chance of $300
$500 for sure or a 15% chance of $1,000,000$100 for sure or a 3% chance of $7,000
Certaingains vs. Lowerexpectedvalue gambles

i

j
k
1
m

$100 for sure or a 25% chance of $200
$100 for sure or a 25% chance of $300
$5 for sure or a 4% chance of $80$5 for sure or a 1% chance of $80$60 for sure or a 1% chance of $5000
Certainlossesvs. Lowerexpectedvalue gambles

n
o
p
q
r

Lose $10 for sure or a 90% chance to lose $50
Lose $100 for sure or a 75% chance to lose $200
Lose $100 for sure or a 50% chance to lose $300
Lose $50 for sure or a 10% chance to lose $800
Lose $100 for sure or a 3% chance to lose $7000

CRTgroup
Low

High

Stat. Signif

52%28074%225

56%95

78%92

37%26457%102

19%84338%475

10%21734%94

p < 0.0001
p < 0.01
p < 0.001
p < 0.0001
p < 0.0001
p < 0.05
p < 0.0001
p < 0.01

47%68

75%20

31%341
8%139

60%135
21%70

Low

High

7%68

10%20

14%137

18%39

29%84

36%50

27%37
19%153

37%38
32%31

Low

High

n.s.
n.s.
n.s.
n.s.
n.s.

24%29

6%16

n.s.

54%33931%14161%33544%18o55%109

63%68

28%57

p < 0.0001
n.s.
p < 0.01
p < 0.0001

23%56

Table3b
The Reflection Effect for Low and High CRT Groups
CRT group
Item

Percentagechoosinggamblein the domain of gains and losses

d
o
h
r

$100 for sure or a 75% chance of $200
Lose $100 for sure or a 75% chance to lose $200
$100 for sure or a 3% chance of $7,000
Lose $100 for sure or a 3% chance to lose $7000

Low

High

19%84338%47554%33931%141

8%139

21%70
28%57

63%68

Football League" and other employers to assess the intellectual abilities of theirprospective hires; 944 completed an 18-item "need for cognition" scale (NFC),
which measures the endorsement of statements like "the notion of thinking ab11Pat

McInally, a Harvard graduate who later became a punter for the Cincinnati Bengals, was the only
college football player to score a perfect 50 out of 50 on the Wonderlic-a score attained by only one
person in 30,000. Of the 921 respondents who took it in these studies, the highest score was a 47.

Shane Frederick 35

Table4
Correlations Between Cognitive Measures
CRT
CRT
SAT
SATM
SATv
ACT
WPT
NFC

.44
.46
.24
.46
.43
.22

SAT

SAT,

SAT,

ACT

WPT

NFC

434

434
434

434
434
434

667
152
152
152

921
276
276
276
466

944
64
64
64
190
276

.77
.81
.77
.49
.30

.28
.63
.40
.21

.67
.37
.28

.48
.30

.19

stractly is appealing to me" (Cacioppo, Petty and Kao, 1984). Several hundred
respondents also reported their scores on the Scholastic Achievement Test (SAT)or the American College Test (ACT), the two most common college entrance
examinations.
Table 4 shows the correlations between cognitive measures. The numbers
above the diagonal are the sample sizes from which these correlations were computed (the number of surveys that included both measures). For example,
152 respondents reported both SAT and ACT scores, and their correlation was 0.77.
As expected, all measures correlate positively and significantly with one another.
The moderate correlations suggest that all five tests likely reflect common factors,
but may also measure distinct characteristics, as they purport to. I have proposedthat the CRT measures "cognitive reflection"-the ability or disposition to resist
reporting the response that first comes to mind. The need for cognition scale(NFC) is advanced as a measure of someone's "tendency to engage in and enjoy
thinking" (Cacioppo and Petty, 1982), but relies on self-reports rather than observed behavior. The Wonderlic Personnel Test (WPT) is intended to measure aperson's general cognitive ability, and the ACT and SAT are described as measuresof academic "achievement."
Although the various tests are intended to measure conceptually distinguishable traits, there are many likely sources of shared variance. For example, though
the CRT is intended to measure cognitive reflection, performance is surely aided byreading comprehension and mathematical skills (which the ACT and SAT alsomeasure). Similarly, though NFC and intelligence are distinguishable, the list of
ways in which those with high NFC differ from those with low NFC (see Cacioppi
et al., 1996) sounds very much like the list one would create if people were sorted
on any measure of cognitive ability. Namely, those with higher NFC were found to
do better on arithmetic problems, anagrams, trivia tests and college coursework, tobe more knowledgeable, more influenced by the quality of an argument, to recall
more of the information to which they are exposed, to generate more "taskrelevant
thoughts" and to engage in greater "information-processing activity."
The empirical and conceptual overlap between these tests suggests that theywould all predict time and risk preferences and raises the question of their relative

36

Journal of EconomicPerspectives

Table5
Correlations Between Cognitive Measures and Decision-Making Indices
Choiceunder uncertainty(Preferencesfor gamblesacrossdomains)
Intertemporalchoice
Cognitive
measure
CRT
SAT
SATM
SATv
ACT
WPT
NFC

Gains

LossesPreferenceforpatient option

Expectedvalue
favors gamble

Expectedvalue
favors suregain

Expectedvalue
favors sure loss

+0.12****3099+0.22****3150+0.07387-0.04387+0.09368+0.08**1014+0.07149+0.05149+0.06149+0.13*367-0.12****1366-0.12*275-0.11275+0.08287+0.03497-0.24****546-0.00215+0.15**387+0.10*577+0.00837+0.06755+0.19***368-0.03368+0.14**549+0.13***904+0.13****875-0.08275-0.01358

predictive validities. To assess this issue, I correlated the scores on the various
cognitive measures with composite indices of decision-making characteristics
formed from the time preference items in Table 2 or the risk preference items inTable 3. The composite scores registered the proportion of patient (or risk seeking) responses. For example, respondents might have been asked whether they
prefer $3,400 this month or $3,800 next month, whether they would prefer ashorter massage in two weeks or a longer one in November and how much theywould pay for overnight shipping of a book. Respondents who preferred the $3800,
the longer later massage and who were willing to pay less than the median personfor express shipping would be coded as "patient" on all three items and would
receive a score of 1. If they were patient on two of the three items, they would
receive a score of 0.66, and so on. Thus, the indices are scores ranging from 0 to
1, in coarse or fine increments depending on how many questions the respondent
answered.12

As shown in Table 5, the CRT was either the best or second-best predictor
across all four decision-making domains and the only test related to them all. Thus,
12

Composite indices were used to measure respondents' general tendencies within a given decisionmaking domain and to permit aggregation across studies. However, unless respondents received
identical items, their scores are not perfectly comparable. This issue is not vital for establishing thepredictive validity of the CRT, because the correlations reflect the pattern plainly observable from theindividual items. However, for the purpose of comparing the cognitive measures, composite indices are
more problematic, because the full battery of cognitive tests was not typically given, and different studies
involved different items. For example, at Carnegie Mellon University, respondents answered items b, dand 1 from Table 2 and items a and d from Table 3. The CRT was the only cognitive measure obtained
for these respondents. Thus, these particular items will be disproportionately represented in thecomposite decision-making indices with which the CRT is correlated. This problem can be overcome bydoing a pairwise comparison of cognitive measures only for those respondents who were given both.
This more painstaking analysis generally confirms the implications of Table 5-namely, the different
tests often function similarly, but the CRT is a bit more highly correlated with the characteristics ofinterest.

CognitiveReflectionand DecisionMaking

37

for researchers interested in separating people into cognitive groups, the CRT is an
attractive test: it involves only three items and can be administered in a minute or
two, yet its predictive validity equals or exceeds other cognitive tests that involve up
to 215 items and take up to 3 /2hours to complete (or which involve self-reports thatcannot be readily verified).

Sex Differences
Men scored significantly higher than women on the CRT, as shown in Table 6.
The difference is not likely due to a biased sampling procedure, because there wereno significant sex differences for any other cognitive measure, except SATmathscores, for which there was a modest difference corresponding to national averages.
Nor can it be readily attributed to differences in the attention or effort expended
on the survey, since women scored slightly higheron the Wonderlic test, which wasgiven under identical circumstances (included as part of a 45-minute survey that
recruited respondents were paid to complete).
It appears, instead, that these items measure something that men have moreThatof.
something may be mathematical ability or interest, since the CRT items
have mathematical content, and men generally score higher than women on mathtests (Benbow and Stanley, 1980; Halpern, 1986; Hyde, Fennema and Lamon, 1990;
Hedges and Nowell, 1995). However, men score higher than women on the CRT,
even controlling for SAT math scores. Furthermore, even if one focuses only on
respondents who gave the wrong answers, men and women differ. Women's mistakes tend to be of the intuitive variety, whereas men make a wider variety of errors.
For example, the women who miss the "widgets" problem nearly always give theerroneous intuitive answer "100," whereas a modest fraction of the men give
unexpected wrong answers, such as "20"or "500" or "1."For every CRT item (andseveral other similar items used in a longer variant of the test) the ratio of"intuitive" mistakes to "other" mistakes is higher for women than for men. Thus,
the data suggest that men are more likely to reflect on their answers and lessinclined to go with their intuitive responses.'3
Because men score higher, the "high" CRT group is two-thirds men, whereas
the "low" CRT group is two-thirds women. Thus, the differences between CRT
groups may be revealing other male/female differences besides cognitive reflection. To remove this confound, Table 7 presents results split by both sex and CRT
score for selected items, including a heretofore undiscussed item involving thewillingness to pay for a coin flip in which "heads"pays $100 and "tails"pays nothing.
Four facts are noteworthy. First, CRT scores are more highly correlated withtime preferences for women than for men; the low and high groups differ more.
Second, as suggested by most prior research (Byrnes, Miller and Schafer, 1999,
13 One might draw the opposite conclusion from self-reports. Using the scale describedearlier, respondents were asked "How long do you deliberate before reaching a conclusion?" Women reported higherscores than men (1.16 vs. 0.45; t186= 2.32; p<0.05).

38

Journal of EconomicPerspectives

Table6
Sex Differences in Cognitive Measures
Test

Men

Women

Significanceof group difference

CRT
SAT

1.47
1334

1.03
1324

p < 0.0001
n.s.

SATmath688

666

p < 0.01

646
26.7
26.2
0.91

658
26.3
26.5
0.85

n.s.
n.s.
n.s.
n.s.

SATverbal

ACT
Wonderlic
NFC

present an overview), women were considerably more risk averse than men, andthis remains true even after controlling for CRT score. Third, for the selected risk
items, CRT is as important as sex. In other words, high-scoring women behave
almost identically to low-scoring men (compare the upper left and lower right cells
within each of the five items in the lower panel). Fourth, in contrast to the patternobserved for the time preference items, CRT scores are more highly correlated withrisk preferences for men than for women.
The curious finding that CRT scores are more tightly linked with time preferences for women than for men, but are more tightly linked with risk preferences
for men than for women held for the other tests of cognitive ability, as well.
Expressed loosely, being smart makes women patient and makes men take morerisks.14 This result was unanticipated and suggests no obvious explanation. The only
related finding of which I am aware is in a study by Shoda, Mischel and Peake(1990), who found that the patience of preschool girls was strongly related to theirsubsequent SAT scores, but the patience of preschool boys was not.

Discussion
The instructions in studies of decision making commonly reassure respondents that "there are no right or wrong answers." If this line is sincere, it implies
that researchers will interpret such preferences as they would a choice between

14 This conclusion can also be expressed less loosely. First, when faced with three mathematicalreasoning problems ("bat and ball," "widgets" and "lilypads"), certain responses that are plausibly
construed as manifestations of intelligence ("5," "5"and "47") tend to correlate positively with certain
other responses that are plausibly construed as expressions of patience (namely, an expressed willingness to wait for larger later rewards), and this tendency is more pronounced in women than men.
Second, the production of the canonically correct responses tends also to correlate positively withcertain responses that are plausibly construed as expressions of risk tolerance (namely, an expressed
willingness to forego a smaller certain reward in favor of a probabalistic larger one), and this tendency
is more pronounced in men than in women. Third, sex differences in risk seeking and in the degree ofrelation to CRT scores was true only in the domain of gains. For the selected loss items (n through r in
Table 3), there were no sex differences.

Shane Frederick 39

Table 7
Results Split by Both CRT and Sex(percentagechoosingpatient option or mean response)
CRT group

Significanceofgroup
Intertemporalchoiceorjudgment$3400 this month or $3800 next month$100 this year or $140 next year

$100 now or $1100 in 10 years$9 now or $100 in 10 years

Willingness to pay for overnight shippingof chosen bookRiskychoiceorjudgment(percentagechoosingriskyoption or mean
response)$100 for sure or a 75% chance of $200

$500 for sure or a 15% chance of $1,000,000$1000 for sure or a 90% chance of $5000$100 for sure or a 3% chance of $7000

Willingness to pay for a coin flip, where"HEADS"pays $100 and "TAILS"paysnothing.

Sex

Low

Men
Women

39%17o60%84

39%252

67%51

Men
Women

21%10634%161p<0.0525%19449%70

p<0.001

Men
Women

58%88

56%11043%18657%68

p<0.05Highdifferencep<0.01p<0.001n.s.

Men

40%123

Women

41%22943%17853%89

p<0.10Men
Women$4.0541$4.5495$1.9484$2.1940p<0.001p<0.001

CRTgroupn.s.

Significanceof group
difference

Sex

Low

High

Men
Women

26%23943%244p < 0.0001

16%39829%130p < 0.01

Men
Women

40%68

80%41

p < 0.0001

25%109

38%37

n.s.

Men
Women

59%10,

81%151

p < 0.001

46%166

59%65

Men
Women

6%36
8%99

30%44
8%24

p < 0.01
n.s.

Men
Women

$13.0054$20.0059$12.0036

p < 0.001
n.s.

$11.0012

p < 0.10

apples and oranges-as a primitive that neither requires nor permits further
scrutiny.
However, unlike a preference between apples and oranges, time and risk
preferences are sometimes tied so strongly to measures of cognitive ability that theyeffectively function as such a measure themselves.'5 For example, when a choice
15 To encourage respondents to consider each choice carefully, andindependently from the otheritems, several "filler" choices were inserted between the "focal items." An analysis of these responses
shows that CRT scores are unrelated to preferences between apples and oranges, Pepsi and Coke, beer

40

Journal of EconomicPerspectives

between a sure $500 and a 15 percent chance of $1,000,000 was presented torespondents along with an eight-item version of the CRT, only 25 percent of those
who missed all eight problems chose the gamble, compared to 82 percent among
those who solved them all. Should this result be interpreted to mean that choosing
the gamble is the "correct"response for this item?
The position that some preferences are better than others and that cognitive
ability is one indicator of the "better" preference is not unprecedented. Savage(1954) argued that increased understanding ought to increase the frequency of the"truly"normative response; that preferences that initially contradict some normative principle may not survive thorough deliberation (what he termed "reflective
equilibrium"). 16 Stanovich and West (2000) extended these views, by arguing that
increased understanding may arise from superior intellect (as well as from extended deliberation or reflection or instruction). In response to those contendingthat judgments commonly labeled as errors or biases are actually equally goodanswers to different interpretations of the question (for example, Hilton, 1995),
Stanovich and West argued that if smarter respondents were more likely to givecanonically correct answers, the other answers must not be equally good after all.17
Some, however, reject the notion that a correlation between (some measureof) cognitive ability and some particular response identifies the "better"response.
For example, Sternberg (2000, pp. 697-698) argues: "[T]o characterize people
with high SAT scores as those who should set the norm for what is somehow trueor right seems to be off target. People with high SAT scores have high levels ofcertain kinds of cognitive abilities. They have no monopoly on quality of thinking
and certainly no monopoly on truth."
The prevalence of this view could be directly tested. Respondents could beshown the respective test scores of those who chose the sure $500 and those whochose the 15 percent chance of $1,000,000. If Sternberg's view is widely shared, thismanipulation would have no effect. If, on the other hand, the correlation between
cognitive ability and preference held normative force, making respondents awareof it would cause many of them to choose the gamble.
Of course, the weight that should be placed on the opinions of those withhigher cognitive abilities clearly depends on the type of decision in question. If one
were deciding between a fixed- and variable-interest mortgage, imitating one's

and wine or rap concertsand ballet.However,CRTscoresare stronglypredictiveof the choice between
AmongtheAmongthe low CRTgroup,67 percentpreferredPeople.
Peoplemagazineand the NewYorker.
high CRTgroup, 64 percent preferredthe NewYorker.
16Slovic and Tversky (1974) use an eloquent and entertaining mock debate between Allais and Savage
to illustrate opposing views on the related issue of whether the opinions of people who have deliberated
longer over an issue ought to count more.
17Along similar lines, Bar Hillel (1991, p. 413) comments: "Many writers have attempted to defend
seemingly erroneous responses by offering interpretations of subjects' reasoning that rationalizes their
responses. Sometimes, however, this charitable approach has been misguided, either because thesubjects are quick to acknowledge their error themselves once it is pointed out to them, or because theinterpretationrequiredto justify the response is even more embarrassingthan the error it seeks to
excuse.

CognitiveReflectionand DecisionMaking

41

brilliant neighbor seems prudent. However, if one were deciding between an apple
and an orange, Einstein's preference for apples seems irrelevant.
Thus, a relation between cognitive ability and preference does not, by itself,
establish the correct choice for any particular individual. Two individuals withdifferent cognitive abilities may experience outcomes differently, which may warrant different choices (for example, what magazines to read or movies to attend).
But with respect to the example motivating this discussion, one must ask whether
it is plausible that people of differing cognitive abilities experience increments ofwealth as differently as their choices suggest. It seems exceedingly unlikely that the
low CRT group has a marked kink in their utility function around $W + 500,
beyond which an extra $999,500 confers little additional benefit. It seems morereasonable, instead, to override the conventional caveat about arguing with tastes(Becker and Stigler, 1977) and conclude that choosing the $500 is the "wrong
answer"-much as 10 cents is the wrong answer in the "bat and ball" problem.
Whatever stance one adopts on the contentious normative issues of whether apreference can be "wrong" and whether more reflective people make "better"choices, respondents who score differently on the CRT make differentchoices, andthis demands someexplanation.
mI thankDan Ariely,ScottArmstrong,Daniel Benjamin,BrettBoshco,EricBradlow,Craig
Fox, Kerri Frederick,Steve Garcia, TimothyHeath, James Hines, Eric Johnson, Daniel
Kahneman, Robyn LeBoeuf GeorgeLoewenstein,Leif Nelson, Nathan Novemsky,Greg
Pogarsky,Drazen Prelec,Daniel Read, Eldar Shafir, TimothyTaylor, CatherineTucker,
Michael WaldmanandJaclyn Ziresfor commentsreceivedon earlierdrafts.A specialthanks
to SteveGarcia,who coordinatedmostof thesurveysgeneratingthedata summarizedhere.As
always (butparticularlyin this case), the views expressedor impliedare thoseof the author
alone.

References
Bar-Hillel, Maya. 1991. "Commentary on Wolford, Taylor, and Beck: The Conjunction Fallacy?"Memoryand Cognition.19:4, pp. 412-14.
Becker, Gary and George Stigler. 1977. "De
Gustibus Non est Disputandum." AmericanEconomicReview.67:2, pp. 76-90.
Benbow, Camilla P. and J. C. Stanley. 1980.
"Sex Differences in Mathematical Ability: Fact orArtifact?"Science.210:4475, pp. 1262-264.
Benjamin, Daniel J. and Jesse M. Shapiro.
2005. "Who is 'Behavioral?' Cognitive Ability
and Anomalous Preferences." Working paper,
Harvard University.

Byrnes, James P., David C. Miller and William
D. Schafer. 1999. "Gender Differences in Risk
Taking: A Meta-Analysis." PsychologicalBulletin.
125:3, pp. 367-83.
Cacioppo, John T. and Richard E. Petty. 1982.
"The Need for Cognition." Journal of Personality
and SocialPsychology.42:1, pp. 116-31.
Cacioppo, John T., Richard E. Petty and
Chuan Feng Kao. 1984. "The Efficient Assessment of Need for Cognition." Journal of Personality Assessment.48:3, pp. 306-07.
Cacioppo, John T., Richard E. Petty, Jeffrey
A. Feinstein and W. Blair G. Jarvis. 1996. "Dis-42

Journal of EconomicPerspectives

positional Differences in Cognitive Motivation:
The Life and Times of Individuals Varying in
Need for Cognition." PsychologicalBulletin. 119:2,
pp. 197-253.
Chaiken, Shelly and Yaacov Trope. 1999.
Dual-Process Theories in Social Psychology.New
York: Guilford Press.
Donkers, Bas, Bertrand Melenberg and Arthur
van Soest. 2001. "Estimating Risk Attitudes Using Lotteries: A Large Sample Approach." Journal of Risk and Uncertainty.22:2, pp. 165-95.
Epstein, Seymour. 1994. "Integration of the
Cognitive and Psychodynamic Unconscious."
AmericanPsychologist.49:8, pp. 709-24.
Frederick, Shane. 2002. "Automated Choice
Heuristics," in Heuristicsand Biases:ThePsychology
of IntuitiveJudgment.T. Gilovich, D. Griffin and
D. Kahneman, eds. New York: Cambridge University Press, pp. 548-58.
Frederick, Shane, George Loewenstein and
Ted O'Donoghue. 2002. "Time Discounting andTime Preference: A Critical Review."Journal of
EconomicLiterature.40:2, pp. 351-401.
Funder, David C. and Jack Block. 1989. "TheRole of Ego-Control, Ego-Resiliency, and IQ in
Delay of Gratification in Adolescence." Journal of
Personalityand Social Psychology.57:6, pp. 1041050.
Halpern, Diane F. 1986. Sex Differencesin Cognitive Abilities.Hillsdale, N.J.: Erlbaum.
Hedges, Larry V. and Amy Nowell. 1995. "SexDifferences in Mental Test Scores, Variability,
and Numbers of High-Scoring Individuals." Science.July 7, 269, pp. 41-45.
Hilton, Denis J. 1995. "The Social Context of
Reasoning: Conversational Inference and Rational Judgment." PsychologicalBulletin. September,
118, pp. 248-71.
Hyde, Janet Shibley, Elizabeth Fennema andSusan J. Lamon. 1990. "Gender Differences inMathematics Performance: A Meta-Analysis."
PsychologicalBulletin. 107:2, pp. 139-55.
Jensen, Arthur R. 1998. ThegFactor:TheScienceof Mental Ability.Westport, Conn.: Praeger.
Kahneman, Daniel and Shane Frederick. 2002.
"Representativeness Revisited: Attribute Substitution in Intuitive Judgment," in Heuristics andBiases: The Psychologyof Intuitive Judgment. T.
Gilovich, D. Griffin and D. Kahneman, eds. New
York: Cambridge University Press, pp. 49-81.
Kahneman, Daniel and Amos Tversky. 1979.
"Prospect Theory: An Analysis of Decision Under Risk." Econometrica.47:2, pp. 263-91.
Kirby, Kris N., Gordon C. Winston and Mariana Sentiesteban. 2005. "Impatience andGrades: Delay-Discount Rates Correlate Negatively with College GPA."Learningand Individual
Differences.Forthcoming.

Lubinski, David and Lloyd Humphreys. 1997.
"Incorporating General Intelligence into Epidemiology and the Social Sciences." Intelligence.
24:1, pp. 159-201.
Melikian, Levon. 1959. "Preference for Delayed Reinforcement: An Experimental Studyamong Palestinian Arab Refugee Children."Journal of SocialPsychology.50, pp. 81-86.
Mischel, Walter. 1974. "Processes in Delay ofGratification," in Advances in ExperimentalSocial
Psychology.L. Berkowitz, ed. San Diego, Calif.:
Academic Press, pp. 249-92.
Monterosso, John, Ronald Ehrman, Kimberly
L. Napier, Charles P. O'Brien and Anna Rose
Childress. 2001. "Three Decision-Making Tasks
in Cocaine-Dependent Patients: Do They Measure the Same Construct?" Addiction. 96:12,
pp. 1825-837.
Nagin, Daniel S. and Greg Pogarsky. 2003.
"An Experimental Investigation of Deterrence:
Cheating, Self-Serving Bias, and Impulsivity."
Criminology.41:1, pp. 501-27.
Parker, Andrew M. and Baruch Fischhoff.
2005. "Decision-Making Competence: External
Validation through an Individual-Differences
Approach." Journal of BehavioralDecisionMaking.
18:1, pp. 1-27.
Rabin, Matthew. 2000. "Risk Aversion andExpected-Utility Theory: A Calibration Theorem." Econometrica.68:5, pp. 1281-292.
Rae, John. 1834. The New Principlesof Political
Economy.Reprinted in 1905 as The Sociological
Theoryof Capital.New York: Macmillan.
Savage, Leonard J. 1954. The Foundations of
Statistics.New York: Wiley.
Shoda, Yuichi, Walter Mischel and Philip K.
Peake. 1990. "Predicting Adolescent Cognitive
and Self-Regulatory Competencies from Preschool Delay of Gratification: Identifying Diagnostic Conditions." Developmental
Psychology.26:6,
pp. 978-86.
Sloman, Steven A. 1996. "The Empirical Case
for Two Systems of Reasoning." PsychologicalBulletin. 119:1, pp. 3-22.
Slovic, Paul and Amos Tversky. 1974. "Who
Accepts Savage's Axiom?" BehavioralScience.19:4,
pp. 368-73.
Stanovich, Keith E. and Richard F. West. 2000.
"Individual Differences in Reasoning: Implications for the Rationality Debate?" BehavioralandBrain Sciences.22:5, pp. 645-726.
Sternberg, Robert J. 2000. "The Ability is
not General, and Neither are the Conclusions. [Response to K. E. Stanovich and R.F.
West.]" Behavioral and Brain Sciences. 23:5,
pp. 697-98.