ARTICLE IN PRESS

Journal of Theoretical Biology 231 (2004) 107–120
www.elsevier.com/locate/yjtbi

How should we deﬁne goodness?—reputation dynamics inindirect reciprocity
Hisashi Ohtsuki, Yoh Iwasa
Department of Biology, Faculty of Sciences, Kyushu University, Hakozaki 6-10-1, Fukuoka 812-8581, Japan
Received 4 February 2004; received in revised form 8 June 2004; accepted 11 June 2004

Abstract
Theory of indirect reciprocity is important in explaining cooperation between humans. Since a partner of a social interaction oftenchanges, an individual should assess his partner by using social information such as reputation and make decisions whether to help
him or not. To those who have ‘good’ social reputation does a player give aid as reciprocation, whereas he has to refuse to help those
who have ‘bad’ reputation. Otherwise beneﬁts of altruism is easily exploited by them. Little has been known, however, about thedeﬁnition of ‘goodness’ in reputation. What kind of actions are and should be regarded as good and what kind of actions bad? And
what sort of goodness enables sustaining exchange of altruism? We herein challenge this question with an evolutionary perspective.
We generalize social reputation as ‘Honor-score’ (H-score) and examine the conditions under which individuals in a group stably
maintain cooperative relationships based on indirect reciprocity. We examine the condition for evolutionarily stable strategies(ESSs) over 4096 possible cases exhaustively. Mathematical analysis reveals that only eight cases called ‘leading eight’ are crucial tothe evolution of indirect reciprocity. Each in the leading eight shares two common characteristics: (i) cooperation with good persons
is regarded as good while defection against them is regarded as bad, and (ii) defection against bad persons should be regarded as agood behavior because it works as sanction. Our results give one solution to the deﬁnition of goodness from an evolutionary
viewpoint. In addition, we believe that the formalism of reputation dynamics gives general insights into the way social information is
generated, handled, and transmitted in animal societies.
r 2004 Elsevier Ltd. All rights reserved.
Keywords: Indirect reciprocity; Reputation dynamics; Goodness; Leading eight; ESS

1. Introduction
Humans are distinct in their social ability to get alongwith others. It is true that eusocial animals such as
honeybees or wasps are prominently social, but their wellorganized societies are based upon close kinship amongindividuals. Humans, on the other hand, exhibit remarkable cooperative tendency towards unrelated individuals.
Indirect reciprocity (Alexander, 1987; Fehr andFischbacher, 2003, Nowak and Sigmund, 1998b) gives
one clear-cut explanation of how this preference forCorresponding author. Tel.: +81-92-642-2641; fax: +81-92-6422645.
E-mail address: ohtsuki@bio-math10.biology.kyushu-u.ac.jp(H. Ohtsuki).

0022-5193/$ - see front matter r 2004 Elsevier Ltd. All rights reserved.
doi:10.1016/j.jtbi.2004.06.005

cooperation has evolved. When an individual behaves
altruistically towards another, he suffers the cost of thehelp, such as time, energy, or risks, hence his action does
not seem to pay for the moment. However, if a third
person who knows of his good deed recompenses him
with cooperation, the cost he paid will be cancelled, andconsequently he may result in getting positive net
beneﬁt. Indirect reciprocity differs from direct reciprocity or reciprocal altruism (Trivers, 1971) in that thedonor of the help receives the return not from the
beneﬁciary himself but from another individual. Due tothis nature of indirect reciprocity, the availability ofsocial information on each individual is necessary inorder to sustain cooperation. Without such informationindividuals cannot discriminate cooperative persons
from cheaters or social parasites, who enjoy beneﬁt of

ARTICLE IN PRESS108

H. Ohtsuki, Y. Iwasa / Journal of Theoretical Biology 231 (2004) 107–120

altruism with paying nothing. Since nobody assures thatthose who are cooperated never fail to offer help toanother, there must be a way to know who is worth
cooperating and who is not. Therefore social information such as reputation or rumor must have greatly
contributed to the evolution of indirect reciprocity byrealizing discriminate altruism. Verbal communication
is a sophisticated way to transfer social information(Enquist and Leimar, 1993; Nakamaru and Kawata,
2004; Pollock and Dugatkin, 1992).
Several previous studies have revealed the characteristics of indirect reciprocity. Nowak and Sigmund(1998b) is a milestone on this issue. They introduced‘image score’ as an index which measures goodness orcooperativeness of each individual and examined what
behavioral rule may evolve. Image score is of an integer
value, and it increases by one unit every time one
cooperates, whereas it decreases by one when one
refuses to cooperate. Hence, a player’s image score isan appropriate portrait of his preference for cooperation. Using computer simulation, Nowak and Sigmund(1998b) showed that ‘discriminator’ strategy is an
outcome of the evolution. This strategy prescribes
cooperation only with an individual with nonnegative
image score. Since nonnegative image score suggeststhat the focal person tends to cooperate rather than todefect, the success of discriminators shows that notunconditional altruism but conditional cooperation isnecessary for the evolution of indirect reciprocity. They
also considered strategies which take their own image
score into account in making decisions. For example,‘And’ strategy cooperates if his score is low and theopponent’s score is high. ‘Or’ strategy, on the other
hand, cooperates if his score is low or the opponent’s
score is high. Nowak and Sigmund (1998b) found thatsuch strategies also promote cooperation.
Leimar and Hammerstein (2001) critically reexamined
Nowak and Sigmund’s study. They pointed out that a
discriminator receives no beneﬁt by caring about theimage score of others. When an individual cooperates,
he gains good reputation irrespective of his opponent’s
reputation. Thus there is no need for a player to consider
other’s reputation. What a player is really interested in is
his own reputation because it directly affects the future
beneﬁt of him. Leimar and Hammerstein (2001) took Dq
strategy for an example to demonstrate that strategies
which smartly calculate the inﬂuence of giving behavior
on other’s decisions defeat discriminators. They claimed
that the scope for discriminators are very narrow.
Instead they examined an alternative called ‘standing
strategy’, which Sugden (1986) had previously proposed.
A player who uses this strategy adopts a criterion
different from image score in assessing others: thiscriterion is called ‘standing’. Standing of each individualis either good or bad and is supposed to be good at theinitial state. An individual in bad standing can regain his

good standing only by cooperating with others, which isthe same rule as image score. However, a person in goodstanding falls into bad if and only if he fails to cooperatewith an opponent in good standing. Even if he refuses tohelp an individual in bad standing, he does not lose his
good standing. This is because the refusal is interpreted
as punishment against a selﬁsh individual (for studies onpunishment, see Brandt and Sigmund (2003), Fehr andGächter (2000), Fehr and Rockenbach (2003), andHenrich and Boyd (2001)). It is true that withdrawalof cooperation from a person with bad reputation might
be thought as bad behavior if people pay attention only
to the action itself. However, if they consider the
motivation of the action; noting that the purpose of
the defection is to impose sanctions on cheaters, thewithdrawal can be labeled as good.
Difference between discriminator and standing strategy lies also in the way of using social reputation. A
discriminator, who adopts image score as a reputationassignment rule, considers only the reputation of his
opponent in making his decisions and will cooperatewith those who have a good image score. A standing
strategist, on the other hand, uses his own reputation as
well in order to do better in a group. He always
monitors his own status within the population andsometimes tries to enhance his standing so as to further
increase the possibility that he receives cooperation fromothers. Hence he offers help if he is in bad standing or if
his opponent is in good standing, which is similar to the‘Or’ strategy noted above.
In short, discriminator and standing strategy differs in
the notion of goodness. Ohtsuki (2004) demonstrated
that discriminate altruism cannot evolve, if everyone in agroup adopts the image-score criterion. However, if
everyone adopts the standing criterion, discriminate
altruism can be maintained (Panchanathan and Boyd,2003). We can also imagine many other variants ofreputation-assignment rules. For example, Sugden(1986) considered standing of a different kind (we callit ‘strict-standing’). When a population uses strictstanding, an individual in bad standing can regain his
good standing only by helping a good person; helping aperson in bad standing is not enough because he may be
regarded as a traitor. A reputation rule of the similar
property was brieﬂy mentioned in Panchanathan andBoyd (2003), who suggested that it have a different
evolutionary outcome. The question now rises what
reputation-assignment rule is plausible and what realizes
discriminate altruism. As far as we know, no study has
systematically examined evolutionary basis of thoserules so far. Here, we will formulate such reputationassignment rules as ‘reputation dynamics’ and challenge
the question. How should we deﬁne goodness? We
examine evolutionary origin of the notion of goodness
that allows the sustaining exchange of altruism. Our
exhaustive ESS analysis gives an answer.

ARTICLE IN PRESS
H. Ohtsuki, Y. Iwasa / Journal of Theoretical Biology 231 (2004) 107–120

2. The model
2.1. Indirect reciprocity game
Consider an inﬁnitely large population in which two
players randomly form a pair and engage in one-shot‘indirect reciprocity game’. In this game, two players
simultaneously determine whether to help the opponent(=Cooperation, denoted by C) or not (=Defection, D),considering social reputation of both self and theopponent. Help costs the donor c while the recipientof the help receives the beneﬁt bð4cÞ. Neither gains if
help is not offered. Mutual help yields b  cð40Þ toboth, whereas unilateral help brings the best outcome to
the recipient and the worst to the donor. Hence, thisgame represents social dilemma. After one-round of thegame, each player changes his partner and will interact
with a different opponent. We assume that the entire
period of social interactions is sufﬁciently long, and that
players engage in the indirect reciprocity game manytimes with many different opponents. Since the population is assumed to be inﬁnitely large, a player does notmeet the same opponent again. Therefore, direct
reciprocity is excluded from the model.

2.2. H-score representation of social reputation
If a player wants to assess another with respect to his
cooperative tendency, it is useful to know the whole
history of his actions, such as how many times and withwhom he has cooperated. It is, however, difﬁcult forplayers to monitor and memorize all the events in a large
population due to costs, time, or capacity. In order toavoid such difﬁculties, individuals instead use a simpler
form of information storage, such as ‘he is nice’ or ‘he isuntrustworthy’. Thanks to the great ﬂuidity of language,
reputation or rumor (Nakamaru and Kawata, 2004) cancontribute to achieving such a purpose.
Here, we consider social reputation of the simplest
kind; that is, one’s social reputation is binary, like ‘goodor bad’ or ‘nice or wicked’. In order to make thefollowing mathematical analyses easy, we introduce‘Honor-score’ (H-score) as an index of one’s social
reputation. H-score is a binary digit (either ‘1’ or ‘0’)and represents its owner’s social reputation. In the109

following, we are to regard H-score of 1 as good
reputation and 0 as bad for simplicity. However, if weswitch H-score of 1 and 0 and regard 1 as bad and 0 as
good, this does not affect the basic property of ourresults at all. It is only for making interpretation of ourresults easy that we relate H-score of 1 to good
reputation. H-score classiﬁes players in the population
into two groups with different social reputation. Notethat H-score realizes theoretical generalization of such
criteria of social reputation as image score (binary
version) (Nowak and Sigmund, 1998a) or standing(Sugden, 1986).

2.3. Reputation dynamics
How do players in a population assign H-score toeach other? For example, when a player cooperates witha person whose social reputation is bad, he may beconsidered good if an observer pays attention to his
cooperative action itself. However, if an observer notes
the motivation of his action, i.e. if the observer thinksthat his cooperation is based on his vicious motivation
to give aid to asocial conspirators, he may be regarded
as bad. Assignment of reputation comprises a matter ofdeﬁnition of goodness as this. Here we model the notionof goodness in a population as ‘reputation dynamics’.
We assume that everyone in a population shares thesame reputation dynamics, like public opinion in asociety.
Reputation dynamics is represented by d ¼ ðd ijX Þ i;j¼1;0 .
X ¼C; D
Each component is either 1 or 0. A society with
reputation dynamics d assigns H-score d ijX to a focal
player E 1 if (i) H-score of E 1 is i, (ii) H-score of
E 1 ’s opponent, say E 2 , is j, and (iii) E 1 ’s action towards
E 2 is X (=C or D). For example, reputation dynamics ðd 11C ; d 11D ; d 10C ; d 10D ; d 01C ; d 01D ; d 00C ; d 00D Þ ¼
ð1; 0; 1; 0; 1; 0; 1; 0Þ assigns H-score of 1 only to those
who cooperate. Hence this is image score. We abbreviate
image score as d ¼ IMAGE. Reputation dynamics
ðd 11C ; d 11D ; d 10C ; d 10D ; d 01C ; d 01D ; d 00C ; d 00D Þ ¼ ð1; 0;
1; 1; 1; 0; 1; 0Þ corresponds to standing. The abbreviation of this is given by d ¼ STAND. There can be eight
different situations ði; j; X Þ, so it follows that we have
28 ¼ 256 different reputation dynamics within our

Table 1
A few examples of reputation dynamics
d 11C

d 11D

d 10C

d 10D

d 01C

d 01D

d 00C

d 00D

Name

Abbreviation

1
1
1
1

0
0
0
0

1
1
1
0

0
1
1
1

1
1
1
1

0
0
0
0

1
1
0
0

0
0
0
0

Image score
Standing
Strict-standing
Judging

IMAGE
STAND
S-STANDJUDGEARTICLE IN PRESS
H. Ohtsuki, Y. Iwasa / Journal of Theoretical Biology 231 (2004) 107–120

110

formalism (see also Brandt and Sigmund (2004)). Table 1
lists up several examples ofreputation dynamics.
2.4. Behavioral strategies
Under a given reputation dynamics, how should eachplayer handle social information? We assume that eachplayer has his own reputation usage as his ‘behavioral
strategy’. A player’s behavioral strategy prescribes how
to behave towards others, considering social reputationof both self and his opponent. It can be a conditional
strategy in general. It is represented by a vectorp ¼ ðpij Þi;j¼1;0 . Four components pij are either C (=Cooperation) or D (=Defection). A player with behavioral
strategy p chooses the action pij ð¼ C or DÞ when his Hscore is i and that of his present opponent is j. For
example, behavioral strategy ðp11 ; p10 ; p01 ; p00 Þ ¼
ðC; D; C; DÞ prescribes cooperation if and only if his
opponent’s H-score is 1. We call it ‘Co-strategy’ andabbreviate it as p ¼ CO, because it pays attention only
to the reputation of its co-player. In contrast, behavioral
strategy ðp11 ; p10 ; p01 ; p00 Þ ¼ ðD; D; C; CÞ considers
only H-score of the player himself and gives help only
when his own H-score is 0. We call it ‘Self-strategy’ðp ¼ SELFÞ. A simple rule ðp11 ; p10 ; p01 ; p00 Þ ¼
ðC; C; C; CÞ always prescribes cooperation, which wecall ‘AllC-strategy’ ðp ¼ AllCÞ, while another ruleðp11 ; p10 ; p01 ; p00 Þ ¼ ðD; D; D; DÞ is named ‘AllD-strategy’ ðp ¼ AllDÞ. There are four different cases in ði; jÞ, so
we have 24 ¼ 16 different behavioral strategies in total(see also Brandt and Sigmund (2004)). For several
examples of behavioral strategies, seeTable 2.
When one reputation dynamics is ﬁxed in the
population, there are 24 ¼ 16 different behavioral
strategies. A combination of reputation dynamics anda behavioral strategy, ðd; pÞ, is called ‘ESS pair’ when p
is an evolutionarily stable strategy among those 16
possible behavioral strategies under the reputation
dynamics d. In the following, we search for ESS pairs
exhaustively. Since there are 28 ¼ 256 kinds of reputation dynamics, we need to investigate 28  24 ¼ 212 ¼
4096 pairs in total. Novelty of our framework is that it
includes not only several famous strategies proposed so
far but also all the possible cases.

Table 2
Several examples of behavioral strategies and their abbreviations
p11

p10

p01

p00

Name

Abbreviation

C
D
D
C
C
D

D
D
D
D
C
D

C
C
C
C
C
D

D
C
D
C
C
D

Co-strategy
Self-strategy
And-strategy
Or-strategy
AllC-strategy
AllD-strategy

CO
SELF
AND
OR
AllC
AllD

For example, discriminators (Nowak and Sigmund,1998a) use image score and help only those who havegood
reputation,sothey
correspondstoðd; pÞ ¼ ðIMAGE; COÞ. In contrast, a standing strategist(Leimar and Hammerstein, 2001) adopts standing
criterion and cooperates if standing of the opponent isgood or if standing of self is bad. Hence, standing
strategy corresponds to ðd; pÞ ¼ ðSTAND; ORÞ.
2.5. Mirror symmetry
Recall that we introduced H-score only to classify
individuals into two social states. This property causes‘mirror symmetry’ in H-scores. If we switch 1 and 0 in
H-score in all the places in the model, predictions
derived from the model remain unchanged at all. Almost
all pairs ðd; pÞ have their mirror images ðd 0 ; p0 Þ, which
are equivalent to original ones provided that H-score of
1 and 0 are switched. For further details of the mirror
symmetry, see Appendix A. Considering the equivalence, the number of effective pairs we need to study is
reduced from 4096 to 2080.

3. Method
3.1. Indirect observation model
Consider the situation where player A meets andinteracts with player B. Most other players, say player
C, in a large population do not observe what A does.
Only one player, say player D, who luckily observes theFig. 1. (a) Indirect observation model: Even if player C wants to know
the H-score of player A, he cannot observe what A does in theinteraction with B due to the large population size. Instead C listens to
D’s opinion, who observed the interaction, either directly or indirectly.
Therefore, if there are two individuals who are in the same situation as
player C, say C1 and C2 , they always have the same opinion on player
A. (b) Direct observation model: Player C directly observes what A doesin interaction with B, and assigns H-score to A by himself. If there are
two individuals who are in the same situation as player C, say C1 andC2 , they may have different opinion on A.

ARTICLE IN PRESS
H. Ohtsuki, Y. Iwasa / Journal of Theoretical Biology 231 (2004) 107–120

interaction, knows A’s behavior. Hence other players in
the population would listen to player D’s evaluation
about player A, either directly from D himself orindirectly by rumor. Fig. 1(a) sketches this situation. We
assume that (i) D never tells a lie and (ii) C always
believes what D says. As a consequence everyone in the
population has the same opinion on a focal player, andhence H-score of each player can be treated as his
internal state. We call this ‘indirect observation model’.
In contrast, Leimar and Hammerstein (2001) postulated
a different model in which each player observes everysocial interaction in the population by himself. In orderto clarify the difference, we call Leimar and Hammerstein’s model ‘direct observation model’ (Fig. 1(b)).
Later in this paper, we will discuss the difference
between those two models. In the following, we study
indirect observation model.
3.2. Roles of small errors
In examining evolutionary stability of strategies, we
introduce two kinds of errors into the model; ‘execution
error’ and ‘assignment error’. Importance of error ornoise in the analysis of indirect reciprocity has beenstressed (Fishman, 2003; Leimar and Hammerstein,
2001; Lotem et al., 1999; Panchanathan and Boyd, 2003;
Sherratt and Roberts, 2001; Sugden, 1986). If no one
commits an error at all, difference between several
strategies is lost, which results in many strategies being
neutral with respect to payoffs. For example, if a
discriminate altruist, who offers help only to good
persons, invades a population of indiscriminate altruists,then only mutual help will occur and it will be repeated
forever in the population. This will prevent us fromcomparing two strategies. If error is included,
however, there will be some players who fail tocooperate and have bad reputation. Difference inbehavior towards those accidental cheaters reveals thedifference of strategies itself, and we can clearly
distinguish these two.
The ﬁrst kind of error, execution error, occurs when aplayer is to cooperate with others. He may sometimes
fail to achieve cooperation against his will due to thiserror. We set the probability that a player commits
execution error as me ð40Þ. Note that we do not consider
the opposite situation in which a player mistakenly
cooperates in spite that he wants to defect. The error of
the second type, assignment error, happens when an
observer of an interaction is to assign reputation to a
focal person. He may misperceive an action or misuse
reputation dynamics in assigning reputation. The probability that this error occurs is given by ma ð40Þ. We
incorporate both cases: (i) a player mistakenly assigns
H-score of 1 when he should assign 0, and (ii) a player
mistakenly assigns H-score of 0 when he should assign 1.
Note that Leimar and Hammerstein (2001) considered111

the error caused only by perception of actions and calledit ‘perception error’. In contrast, assignment error
includes causes other than perception of actions,because the mistake of assignment can occur even if
perception of the action is correct.
3.3. Dynamics of H-score and ESS analysis
3.3.1. Dynamics of H-score in continuous time
We postulate that reputation dynamics d is ﬁxed
in the population. In other words, all in the population
share the same criterion about goodness. Imagine
that a small number of mutants with behavioral strategy
p0 invades a population dominated by p strategists.
Let ht ðpÞ be a proportion of wild-type individuals
whose H-score is 1 among wild-type players at timet, and ht ðp0 Þ be that among mutants. Both ht ðpÞ andht ðp0 Þ change over time, and their dynamics dependon behavioral strategies (p and p0 ) and reputation
dynamics d.
First, we consider the change of ht ðp0 Þ in a shorttime interval ½t; t þ Dt . We assume that within thisinterval a fraction of Dt players in the population engage
in the indirect reciprocity game once. There are three
types in p0 strategists, differing in the transition of their
H-score.
(i) Players with H-score 1 at time t, who engage in a
game: The fraction of these players among p0
strategists is Dt ht ðp0 Þ. Each of them meets a wildtype player and plays one-shot indirect reciprocity
game. With probability ht ðpÞ, H-score of his wildtype opponent is 1. In this case, he takes action
X ¼ p011 and he will be assigned H-score of d 11p011 by
an observer. With probability ð1  ht ðpÞÞ, H-scoreof his wild-type opponent is 0. In that case, he takes
action X ¼ p010 and will be assigned a new H-score
d 10p010 .
(ii) Players with H-score 0 at time t, who engage in a
game: The fraction of these players among p0
strategists is Dt ð1  ht ðp0 ÞÞ. Similarly to (i), theytake action X ¼ p001 and become assigned H-scoreof d 01p001 (with probability ht ðpÞ), or take action X ¼
p000 and become assigned H-score d 00p000 (withprobability 1  ht ðpÞ).
(iii) Players who do not play a game: The fraction ofthese players among p0 strategists is ð1  DtÞ, afraction ht ðp0 Þ of them have H-score of 1 at time t.
Their H-score remains unchanged.
From these we have the following equation:hihtþDt ðp0 Þ ¼ Dt ht ðp0 Þ ht ðpÞd 11p011 þ ð1  ht ðpÞÞd 10p010
hþ Dt ð1  ht ðp0 ÞÞ ht ðpÞd 01p001iþ ð1  ht ðpÞÞd 00p000 þ ð1  DtÞht ðp0 Þ:(1)

ARTICLE IN PRESS
H. Ohtsuki, Y. Iwasa / Journal of Theoretical Biology 231 (2004) 107–120

112

Taking the limit of Dt ! 0, we have the following
differential equation:hidht ðp0 Þ ¼ ht ðp0 Þ ht ðpÞd 11p011 þ ð1  ht ðpÞÞd 10p010
dthþ ð1  ht ðp0 ÞÞ ht ðpÞd 01p001iþ ð1  ht ðpÞÞd 00p000  ht ðp0 Þ:(2)
3.3.2. Effects of two kinds of errors
In addition, we need to consider errors of two kinds.
First, due to execution error a player fails to cooperate
against his intention with probability me . In other words,
he is compelled to choose defection with probability me .
Only with probability ð1  me Þ can he perform his
intended action. As a result, d ijp0ij ’s in Eq. (2) are
rewritten as ð1  me Þd ijp0ij þ me d ijD .
Next, consider assignment error. Imagine that the true
H-score to be assigned is d ijX . Because of this error,
however, one is assigned the wrong H-score ð1  d ijX Þwith probability ma . Hence, the expected proportion ofindividuals who are assigned H-score of 1 is given byð1  ma Þd ijX þ ma ð1  d ijX Þ ¼ ð1  2ma Þd ijX þ ma .
Combining those two errors, d ijp0ij ’s in Eq. (2) are
replaced byð1  me Þfð1  2ma Þd ijp0ij þ ma g þ me fð1  2ma Þd ijD þ ma g
¼ ð1  2ma Þfð1  me Þd ijp0ij þ me d ijD g þ ma :Thanks to these properties, we can derive the ESS
condition of each behavioral strategy analytically. Leth ðpÞ and h ðp0 Þ the equilibrium of Eq. (4) expressed interms of parameters p; p0 ; d; me and ma . When time t is
sufﬁciently large, the probability that a focal wild-type
player receives donation in one social interaction is
given by
yðp; pÞ ¼ dðp11 Þh ðpÞh ðpÞ þ dðp10 Þh ðpÞð1  h ðpÞÞþ dðp01 Þð1  h ðpÞÞh ðpÞ þ dðp00 Þ ð1  h ðpÞÞð1  h ðpÞÞ;where dðCÞ ¼ 1 and dðDÞ ¼ 0. The probability that a
focal wild-type player helps his opponent in one social
interaction is the same as above. The probability that a
focal mutant player receives donation from a wild-type
player in one social interaction isyðp; p0 Þ ¼ dðp11 Þh ðpÞh ðp0 Þ þ dðp10 Þh ðpÞð1  h ðp0 ÞÞþ dðp01 Þð1  h ðpÞÞh ðp0 Þ þ dðp00 Þ ð1  h ðpÞÞð1  h ðp0 ÞÞThus Eq. (2) is rewritten asdht ðp0 Þdt

hi¼ ht ðp0 Þht ðpÞ ð1  2ma Þfð1  me Þd 11p011 þ me d 11D g þ mahþ ht ðp0 Þð1  ht ðpÞÞ ð1  2ma Þfð1  me Þi
 d 10p010 þ me d 10D g þ mah
þ ð1  ht ðp0 ÞÞht ðpÞ ð1  2ma Þfð1  me Þi
 d 01p001 þ me d 01D g þ mah
þ ð1  ht ðp0 ÞÞð1  ht ðpÞÞ ð1  2ma Þfð1  me Þd 00p000iþ me d 00D g þ ma  ht ðp0 Þ:(4)

3.3.3. Invasibility condition
Now we are interested in whether the reputation of pstrategist and p0 -strategist are good or bad after the longrun. Based on Eq. (4), we can prove analytically thatboth ht ðpÞ and ht ðp0 Þ converge to stationary values h ðpÞand h ðp0 Þ, respectively, when t becomes inﬁnitely large,
and that the limits are independent of their initial values
h0 ðpÞ and h0 ðp0 Þ. Hence, initial reputation has no effecton the evolutionary outcome. For mathematical detailsof the analysis, see Appendix B.

(6)and the one that a focal mutant player helps his wildtype opponent in one social interaction isyðp0 ; pÞ ¼ dðp011 Þh ðp0 Þh ðpÞ þ dðp010 Þh ðp0 Þð1  h ðpÞÞþ dðp001 Þð1  h ðp0 ÞÞh ðpÞ þ dðp000 Þ ð1  h ðp0 ÞÞð1  h ðpÞÞ:

(3)

(5)

(7)

Let W ðpjpÞ be the average payoff of a wild-type andW ðp0 jpÞ be that of a mutant. Then we haveW ðpjpÞ ¼ b yðp; pÞ  c yðp; pÞ;000W ðp jpÞ ¼ b yðp; p Þ  c yðp ; pÞ:

(8)
(9)

The ESS criterion for behavioral strategy p under
reputation dynamics d is
W ðpjpÞ4W ðp0 jpÞ ðfor all p0 apÞ:ð10Þ

Hence the procedure for searching for ESS pairs is asfollows:(1) Fix one reputation dynamics d. Take a behavioral
strategy p, the stability of which we want to check.
(2) Take a mutant strategy p0 which is different from p.
(3) Calculate average H-scores h ðp) and h ðp0 Þ, by using
the procedure in Appendix B.
(4) Calculate average payoffs W ðpjpÞ and W ðp0 jpÞ, byusing Eqs. (5)–(9).
(5) If W ðpjpÞ4W ðp0 jpÞ holds, then p is stable against
invasions by p0 .
(6) Repeat (2) to (5) for each of 15 behavioral strategies
p0 ðapÞ. If p is stable against all the others, then p is
an ESS under reputation dynamics d. In otherwords, ðd; pÞ is an ESS pair.
We assumed that everyone in the population agrees touse common reputation dynamics d. Also assumed isthat all players have the same opinion about one person.

ARTICLE IN PRESS
H. Ohtsuki, Y. Iwasa / Journal of Theoretical Biology 231 (2004) 107–120

Hence one’s ‘reputation’ is given by the society as awhole, not by each individual. Roughly speaking, ourmathematical approach seeks the best way to handle
public opinion. When ðd; pÞ is found to be an ESS pair,
it means that behavioral rule p is the best under acriterion of goodness d. Of course under a different
criterion d 0 may the best behavioral strategy p0 be
different.

for details). Hence, we will need to study only one of thetwo. This conﬁnes our scope to 2080 cases out of 4096.
Fig. 2(a) shows the distribution of the relative payoffof ESS pairs, when the beneﬁt is b ¼ 2 and the cost is
c ¼ 1. Since AllD-strategy is always evolutionarily stable
under any reputation dynamics, we regard the pairs
which behavioral strategy is p ¼ AllD as trivial and do
not consider them in Fig. 2(a). In contrast, ESS pairs
which behavioral strategy is not p ¼ AllD earn positive
payoffs, so we pay attention to them in the following.
We ﬁnd 25 non-trivial ESS pairs. Their details areshown in Table 3. Because the beneﬁt of the help isb ¼ 2 and the cost is c ¼ 1, the average payoff can beb  c ¼ 1 at its maximum in principle. We expect that insuch an ideal group all individuals would both give andreceive help without committing any errors. On the

4. Results of exhaustive search
We have exhaustively searched for ESS pairs among
4096 cases of ðd; pÞ. When ðd; pÞ is an ESS pair, itsmirror image ðd 0 ; p0 Þ is also an ESS pair (see Appendix A
25

25

20

20

15

15

10

10

5

5
0

number of ESS pairs

(a)

0.2

0.4

0.6

0.8

1

(b)

25

25

20

20

15

15

10

10

5

5

(c)

113

0

0.2

0.4

0.6

0.8

1

0

0.2

0.4

0.6

0.8

1

(d)

0

0.2

0.4

0.6

0.8

1

0

0.2

0.4

0.6

0.8

1

25
20
15
10
5

(e)

relative payoff : W(p|p)/(b-c)
Fig. 2. Distribution of the relative payoff of ESS pairs, with different cost-beneﬁt ratio. The horizontal axis represents the relative payoff, which isdeﬁned as the ratio of the average payoff to its theoretical maximum, W ðpjpÞ=ðb  cÞ. The vertical axis represents the number of ESS pairs. (a) Beneﬁtof help is b ¼ 2 and its cost is c ¼ 1. If all in the population never failed to cooperate with one another, the relative payoff would be W ðpjpÞ=ðb  cÞ ¼
1 (perfectly cooperative population), which is the theoretical maximum. However, due to execition error and assignment error it is lower than 1 ingeneral. We ﬁnd 161 ESS pairs (When ðd; pÞ is an ESS pair, then its mirror image ðd 0 ; p0 Þ is also an ESS pair. In such a case we count only one of thetwo), behavioral strategies in 136 of which are AllD-strategy. We pay attention to the other 25 ESS pairs, which are shown here. Roughly speaking
there are two groups differing in the relative payoff, which is above 0.6 (60% of theoretical maximum) for the ﬁrst group and is below 0.4 (below40%) for the second group. Table 3 shows further details of this ﬁgure. The highest eight ESS pairs (colored in black in the ﬁgure) realize the relative
payoff of more than 0.94, which is close to the theoretical maximum. We call those leading eight in the text. (b)–(e) Distributions of ESS pairs undervarious cost–beneﬁt ratios. Cost is ﬁxed at c ¼ 1 throughout those ﬁgures whereas the beneﬁt of help is (b) b ¼ 4, (c) b ¼ 3, (d) b ¼ 1:2, and (e)b ¼ 1:1, respectively. The number of ESS pairs except for AllD-strategy pairs (p ¼ AllD) decreases as the beneﬁt declines; (b) 73 pairs, (c) 59 pairs, (d)15 pairs, and (e) 13 pairs. The leading eight (in black in each ﬁgure), however, always rank the highest among them. Other parameters:me ¼ ma ¼ 0:02.

ARTICLE IN PRESS
H. Ohtsuki, Y. Iwasa / Journal of Theoretical Biology 231 (2004) 107–120

114

Table 3All ESS pairs that are non-trivial when b ¼ 2 and c ¼ 1
ðd 11C

d 11D

d 10C

d 10D

d 01C

d 01D

d 00C

d 00D Þ

ðp11

p10

p01

p00 Þ

Relative payoff

ð1
0

1
1
0
1
0Þ
ðC
D
C
CÞ
0.943
ð1
0

1
1
0

1Þ
ðC
D
C
DÞ
0.942
ð1
0

1
1
0
0
0Þ
ðC
D
C
DÞ
0.940
yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
ð1
0
0
0
1
0
1
0Þ
ðC
D
C
CÞ
0.838
ð1
0
0
0
1
0

0Þ
ðC
D
C
DÞ
0.809
ð1
0

1
0
0
1
0Þ
ðC
D
D
CÞ
0.705
ð1
0

1
0
0

1Þ
ðC
D
D
DÞ
0.680
ð1
0
0
0
1
0
0
0Þ
ðC
D
C
DÞ
0.331
ð0
0

1
1
0
0
0Þ
ðD
D
C
DÞ
0.244
ð1
1
0
1
1
1

0Þ
ðD
C
D
DÞ
0.232
ð1
0

1
0
0
0
0Þ
ðC
D
D
DÞ
0.170
ð1
1
0
1
1
1
1
1Þ
ðD
C
D
DÞ
0.101
We have 25 non-trivial ESS pairs (we omit their mirror images). The symbol ‘’ represents a wild card (i.e. either 1 or 0). Average payoff is calculated
as W ðpjpÞ=ðb  cÞ.

other hand, the average payoff is zero in the group where
nobody cooperates. Hence the relative payoff, which isdeﬁned as the ratio of average payoff to the maximum
payoff, W ðpjpÞ=ðb  cÞ, can be used as a measure ofcooperative tendency in the group. It is 1 for perfectly
cooperative population and is 0 for AllD-population.
Due to errors, however, even for the most cooperative
strategy the relative payoff is a little lower than 1. There
are two large groups in the distribution. ESS pairs in one
group have the relative payoff of around 0.2 and those in
the second group are around 0.8. The most successfuleight ESS pairs earn more than 0.94 (colored in black in
Fig. 2(a)), next nine between 0.68 and 0.84, and the others
less than 0.34. Here we pay attention to those best eight
ESS pairs and call them ‘leading eight’.
Next, we examine effects of cost-beneﬁt ratio. Wechange the beneﬁt b while the cost c remains unchanged.
Figs. 2(b)–(e) show the distributions of the relative
payoff of ESS pairs. The parts colored in black in Fig. 2
represent the leading eight. Although the number of ESS
pairs decreases as b declines, the leading eight not only
keep their evolutionary stability but also always rank
the highest. When the beneﬁt is large enough (Fig. 2(b)),
there are 73 non-trivial ESS pairs. As the efﬁciency ofhelp declines, many ESS pairs except for the leadingeight lose their evolutionary stability. At b ¼ 1:1 (Fig.
2(e)), there remain only 13 ESS pairs, 8 of which are in
the leading eight and they still yield large beneﬁt whereas
the other 5 less than 40% of the theoretical maximum.
Compared with other ESS pairs, the leading eight have aremarkable robustness against the decline in the
efﬁciency of help, provided b4c.
Changes in the error rates do not undermine theirsupremacy either. Fig. 3 show distributions of ESS pairs
with various me and ma values. The leading eight are
evolutionarily stable regardless of error rates. Also they
are always the highest in the rankings.

The leading eight share a few common characteristics.
First, d 1C ¼ 1 and d 1D ¼ 0 (note that  is a wild card)
hold over the leading eight. The former equation statesthat if a player offers help to a good player, the donor
will also be given good reputation. The latter notes that
if a player refuses to help a good player, he will beassigned bad reputation. In short, behavior towards a
good person critically affects the actor’s reputation:
good reputation to cooperation and bad reputation todefection. This is not true to behavior towards a bad
person; sometimes cooperation towards a bad person
yields bad reputation to the actor, or vice versa.
Second, d 10D ¼ 1 always holds over the leading eight.
This rule indicates that refusal of help against a bad
individual does not undermine reputation of a good
person. If we pay attention to the action of refusal itself,
it may be socially bad. However, not cooperating with abad person would be helpful for sustaining discriminate
altruism in a society: if individuals were to donate evento social parasites, the cooperative relationship would be
lost easily. Hence, a society as a whole should regard
such refusal as a good action. This realizes the notion of
sanction. Even defection itself is beneﬁcial to the actor
himself, in other words, even if the refusal saves a player
the cost of cooperation, others in a society must notblame him for his defection, provided that it is aimed ata bad person. In contrast, defection towards a good
player should of course be blamed, which is stated by
d 11D ¼ d 01D ¼ 0.

5. Conditions for sustained cooperation
Mathematical analysis supports the conclusion fromnumerical search in the last section. First, we can derive
the following inequality as the condition of evolutionary

ARTICLE IN PRESS

number of ESS pairs

H. Ohtsuki, Y. Iwasa / Journal of Theoretical Biology 231 (2004) 107–120
25

25

20

20

15

15

10

10

5

5
0

(a)

0.2

0.4

0.6

0.8

1

(b)

25

25

20

20

15

15

10

10

5

5

(c)

0

0.2

0.4

0.6

0.8

1

(d)

115

0

0.2

0.4

0.6

0.8

1

0

0.2

0.4

0.6

0.8

1

relative payoff : W(p|p)/(b-c)
Fig. 3. Distribution of the relative payoff of ESS pairs, with different error rates, me and ma . The horizontal axis represents the relative payoff,
W ðpjpÞ=ðb  cÞ. The vertical axis represents the number of ESS pairs. We set b ¼ 2 and c ¼ 1 in these ﬁgures. The part colored in black represents theleading eight. (a) When error rates are too small (me ¼ ma ¼ 0:005). We ﬁnd 25 ESS pairs, which are the same as those in Fig. 2(a). The relative payoffof the leading eight is above 0.985. (b) When errors frequently occur (me ¼ ma ¼ 0:08). We ﬁnd 23 ESS pairs. Although the relative payoff of theleading eight is around 0.8, the leading eight keep their evolutionary stability and remain the highest in the ranking. (c),(d) When one of two error
rates are large compared with the other ((c) me ¼ 0:05 and ma ¼ 0:01, (d) me ¼ 0:01 and ma ¼ 0:05). In both cases we ﬁnd 25 ESS pairs, which are thesame as those in Fig. 2(a). We see that the leading eight are still the best.

stability of each in the leading eight against AllD:b
41 þ OðmÞ;c

ð11Þ

where m is a parameter of the same order of magnitude
as me and ma (see Appendix B for derivation). This tells
us that cooperation by the leading eight is evolutionarily
stable whenever b is a little greater than c, if errors occur
only infrequently. Second, the average payoff in the
population where everyone adopts one of the leadingeight is calculated as
W ðpjpÞ ¼ ðb  cÞf1  OðmÞg:ð12Þ

The leading eight can maintain almost maximum
average payoff in the society they dominate. We ﬁndthat Eqs. (11) and (12) hold simultaneously only for theleading eight. ESS pairs other than the leading eight do
not have this good property. Hence, we conclude thatthe leading eight are the most suitable for sustaining
cooperation, especially when the yield of altruism ispositive but tiny.
According to the characteristic of ESS pairs, weclassify the leading eight into three groups, named
Group I, Group II, and Group III, respectively, as in
Table 4.
Two ESS pairs belong to Group I. Note that one ofthese, ðd; pÞ ¼ ðSTAND; ORÞ, corresponds to the standing strategy (Leimar and Hammerstein, 2001; Panchanathan and Boyd, 2003; Sugden, 1986). They arecharacterized by p ¼ OR, d 00C ¼ 1, and d 00D ¼ 0. First,
p ¼ OR suggests that players are interested in their own
reputation. They try to enhance their reputation when
they are with bad reputation. Rules d 00C ¼ 1 and d 00D ¼
0 say that a bad individual can regain his good
reputation by cooperating with cheaters, who are almost
always with bad reputation. In such a society, cooperation done by a bad individual is regarded as a goodaction even if it is aimed at social parasites. Taylor
expansion with respect to two error rates me and mashows that strategies in Group I are stable against
invasions by AllD ifb41 þ ðme þ 2ma Þ þ ðm2e þ 4me ma þ 6m2a Þ þ
c

;

ð13Þ

which is the most strict condition among the threegroups, and their average payoff is given byðb  cÞ 1  ð2me þ ma Þ þ ð3m2e þ 6me ma þ m2a Þ þ;
ð14Þ
which is the largest among the three.
Four ESS pairs belong to Group II. They arecharacterized by d 00D ¼ 1. In a society dominated bythem, defection done by a bad person against socialparasites is regarded as good. It is true that those four
ESS pairs are stable against a few AllD-strategists, butthey may be susceptible to a cluster of AllD-strategists
since defection between two cheaters is considered goodunder those rules. Taylor expansion with respect to two

ARTICLE IN PRESS
H. Ohtsuki, Y. Iwasa / Journal of Theoretical Biology 231 (2004) 107–120

116
Table 4The leading eightGroupðd 11C

d 11D

d 10C

d 10D

d 01C

d 01D

d 00C

d 00D Þ

ðp11

p10

p01

p00 Þ

a

ðC
D
C
C Þd
ð1
0
1
1
1
0
1
0Þ
ð1
0
0
1
1
0
1
0Þ
ðC
D
C
C Þd
yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
Group II
ð1
0
1
1
1
0
1
1Þ
ðC
D
C
D Þe
ð1
0
1
1
1
0
0
1Þ
ðC
D
C
D Þe
ð1
0
0
1
1
0
1
1Þ
ðC
D
C
D Þe
ð1
0
0
1
1
0
0
1Þ
ðC
D
C
D Þe
yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
ðC
D
C
D Þe
Group III
ð1
0
1
1
1
0
0
0 Þb
c
ð1
0
0
1
1
0
0
0Þ
ðC
D
C
D Þe
Group IaStanding d ¼ STAND.
Strict-standing d ¼ S-STAND.
c
Judging d ¼ JUDGE.
d
Or-strategy p ¼ OR.
e
Co-strategy p ¼ CO. We omit their mirror images.
berror rates me and ma shows that strategies in Group II
are stable against invasions by AllD ifb41 þ ðme þ 2ma Þ þ ðm2e þ 2me ma þ 4m2a Þ þ
cand their average payoff is given byðb  cÞ 1  ð2me þ ma Þ þ ð2m2e þ 4me ma Þ þ

;


:

ð15Þ

ð16Þ

The other two ESS pairs are in Group III. We see thestrict-standing d ¼ S-STAND in this group. The groupis characterized by d 00C ¼ d 00D ¼ 0. It says that a bad
player can never regain his good reputation throughsocial interactions with other bad persons. The only way
for him to acquire a good reputation is to meet a good
person and helps him (d 01C ¼ 1). Once one falls into bad
reputation, it is hard for him to recover his previous
social state. Hence, those rules in Group III are the moststrict against bad players among the three groups; goodsocial reputation is highly valuable there. Taylor
expansion with respect to two error rates me and mashows that strategies in Group III are stable against
invasions by AllD ifb41 þ ðma Þ þ ð2me ma þ 3m2a Þ þ
c

;

ð17Þ

which is the weakest condition among the three groups,
and their average payoff is given by; ð18Þðb  cÞ 1  ð2me þ ma Þ þ ðm2e þ 2me ma  m2a Þ þwhich is the smallest among the three.

6. Discussion
Altruism brings welfare at the sacriﬁce of its donors.
Seen from the beneﬁt of a group, it should be greatly
encouraged. For donors, however, it is only costly.

There is a conﬂict between group’s interest and that ofan individual. Since it is not a group but an individualthat natural selection works on, everyone tries not to
pay any costs, which leads to the situation where no one
is motivated at all to participate in cooperation. If
return for the help were always assured, people would be
willing to pay the cost of help for they can surely acquire
the beneﬁt which exceeds the cost. But it is impossible innature without such powerful deterrence as legal
institutions.
Indirect reciprocity gives us an afﬁrmative answer tothis issue. If discriminate altruists, who carefully
distinguish individuals by their social reputation,
selectively choose potential receivers of their help, it
becomes possible to build a society based on altruism
without any external enforcement. We stress thatindiscriminate altruists, who help everyone, will neverachieve such a purpose because they are easily exploitedby social parasites. Conditional cooperation is necessary. That is why the leading eight have conditional
strategies such as p ¼ CO and OR.
However, the above discussion misses an importantquestion; what goodness is. As Fehr and Fischbacher(2003) mention, the notion of goodness in social
reputation is one of the most fundamental issues inindirect reciprocity. Our intuition tells us that all
cooperators are good and all defectors are bad, as
recommended by the image-score criterion d ¼ IMAGE.
Previous studies (Leimar and Hammerstein, 2001;
Ohtsuki, 2004; Panchanathan and Boyd, 2003) have,
however, shown that this simple criterion does not work
well for maintaining general exchange of altruism:
things are not so simple. As Nowak and Sigmund(1998a) mentioned in their discussion, if every defection
is regarded as bad a player who happens to meet a socialparasite will not want to refuse help, otherwise his
reputation would be undermined. This leads to the

ARTICLE IN PRESS
H. Ohtsuki, Y. Iwasa / Journal of Theoretical Biology 231 (2004) 107–120

117

standing criterion (Sugden, 1986), which justiﬁes defection against bad persons as punishment. Some haverevealed that the standing is important (Leimar andHammerstein, 2001; Panchanathan and Boyd, 2003).
Takahashi and Mashima (2003) broadened theirscope to criteria other than image-score or standing.
They assumed that a player’s reputation is determinedby (i) his action (ﬁrst-order information) and (ii) his
opponent’s reputation (second-order information).
Hence in their model there are 16 reputation dynamics(though they called those ‘strategies’). Using mixedpopulation model and island model they showed thatstanding strategy is highly susceptible to misperception.
They casted serious doubts on the effectiveness of thestanding strategy.
Although several previous studies have examined theperformance of a couple of reputation dynamics, no one
has systematically studied evolutionary plausibility of
the notion of goodness so far. For that reason we
studied all possible reputation dynamics exhaustively.
To fulﬁll our purpose, we had to ﬁrst exclude
subjectivity or bias we have as to the notion of goodness.
We introduced a simple binary digit called Honor-scoreas a substitute for good or bad. H-score system makes it
possible for us to study and compare a large number of
reputation dynamics. We believe that this is the ﬁrst
paper that the notion of goodness, namely who should
be regarded good in social exchange, is studied systematically as evolutionary games.

reputation of themselves cannot achieve steadysocial exchange. In addition, our result has shownthat such queer behavioral strategies as ‘cooperate
only if reputation of both players agree’ or ‘cooperate
only if reputation of both players disagree’ arehopeless.
Reputation dynamics within ESS pairs in the leadingeight bring fruitful perspective on the notion of goodness. The leading eight have roughly speaking two
features in common. One is that a player interactingwith good persons are assessed by what he does.
Cooperation with good individuals should be goodand defection against good ones should be bad. The
second feature should we consider with much emphasis:a good player who refused to help a bad person must belabeled good. This enables players facing cheaters torefuse help without worrying about the inﬂuence of theaction on their own good reputation. Defection of thiskind should be regarded as righteous and retaliatory,
not selﬁsh. It is because the image-score criterion does
not have this property that the society based on thatfails to maintain indirect reciprocity. It is true, however,that players adopting the leading eight should be
required high cognitive ability. We expect that indirect
reciprocity has evolved owing to several capabilities ofhumans.

6.1. Leading eight

There is difference between indirect observation
model studied in the present paper and direct observation model by Leimer and Hammerstein (2001) concerning the way to obtain social information of others.
These two models are the same when there is no error in
the assignment of H-score. However, the existence ofassignment error drastically changes evolutionary stability of the leading eight.
Consider, for example, the stability of the standing
strategy, ðd; pÞ ¼ ðSTAND; ORÞ (Leimar and Hammerstein, 2001; Panchanathan and Boyd, 2003; Sugden,1986). For ðd; pÞ ¼ ðSTAND; ORÞ to be an ESS pair, itis necessary that ‘Or-strategy’ (p ¼ OR) should be stableagainst the invasion by AllC-strategy (p ¼ AllC). This
holds for indirect observation model studied in thepresent paper. However, in direct observation model it is
no more true and the standing strategy is not an ESS
pair. The reason is given as follows.
Imagine a population in which Or-strategists arecommon and there are a few AllC-mutants. Or-strategist
and AllC-strategist take different actions only when theymeet an opponent whose H-score is 0; Or-strategist
defects against him whereas AllC-strategist cooperates.
Note that the opponent’s H-score of 0 is always due toerrors, because he is usually cooperative. Difference inpayoffs between two strategies results from errors.

Our ESS analysis has shown that the leading eight are
the most plausible candidates which have contributed tothe evolution of indirect reciprocity. Even when there
are errors of various kinds and when the beneﬁt of help
exceeds its cost only by a small margin, a society basedon the leading eight can maintain social proﬁts stably;
those who help others surely receive greater return. In
contrast, when the beneﬁt b is much greater than thecost c we have a number of ESS pairs (see Fig. 2(b)).
Several reputation dynamics can secure indirect reciprocity if the help yields much.
Behavioral strategies in the leading eight are either
p ¼ OR or CO. For reputation dynamics in Group I,
behavioral strategy p ¼ OR, which considers reputationof its own, does slightly better than p ¼ CO. This result
agrees with what Panchanathan and Boyd (2003)
discussed. We have conﬁrmed that even with assignment
error (that is similar to ‘perception error’ (Leimar andHammerstein, 2001)) does the behavioral strategy p ¼
OR earn much beneﬁt than p ¼ CO. In contrast, for
reputation dynamics in Group II and III, we ﬁnd thatthe simple behavioral strategy p ¼ CO is evolutionarily
stable. This result for ESSs tells us that not only the
AllC-strategy but also those which care only about the6.2. Indirect observation model vs. direct observation
model

ARTICLE IN PRESS
118

H. Ohtsuki, Y. Iwasa / Journal of Theoretical Biology 231 (2004) 107–120

In indirect observation model, H-score of a focal
person is common knowledge among players in the
population; all have the same opinion on him. Therefore, when a player E 1 is mistakenly assigned bad
reputation (H-score ¼ 0) by an observer, everyone
mistakenly believes that E 1 ’s H-score is 0, though heshould have assigned 1. In this case, Or-strategist
outcompetes AllC-strategist, because defection aimed
at E 1 by Or-strategist is regarded as good (d 10D ¼ 1)
thus he can save the cost of help ðþcÞ without
undermining his good reputation.
In contrast, each may have different opinion on a
person in direct observation model. In this model, eachplayer has his subjective H-score towards each person.
H-score is no more shared among individuals. Therefore, even when a player mistakenly assigns (hissubjective) H-score of 0 to player E 1 due to assignment
error, others will not agree with his opinion. The Orstrategist’s belief that E 1 is not worth cooperating is not
approved by others, so he will miss the futurecooperation (b) in exchange for the refusal of the help
(þc). Because b þ co0 holds, Or-strategists are
invaded by AllC-mutants.
The evolutionary outcomes of two models differ inthe presence of errors. Which is more plausible may
depend on the situation considered. We think thatindirect observation model may be more suitable thanthe direct observation model in human interactions.
Fluidity of language must have greatly contributed tothe evolution of indirect reciprocity (Nakamaru and
Kawata, 2004). Thanks to the convenience and highavailability of language as a source of information, eachsaves the cost of monitoring others and keeps aware ofsocial information, thus it results in robust cooperative
relationships. In contrast, a player in the direct
observation model observes every social interaction byhimself. We predict such an excessive investment tomonitoring costs makes indirect reciprocity difﬁcult to
be maintained in a large group. The difference between
indirect observation and direct observation suggests a
merit of human language. On the other hand, direct
observation model may be more suitable for a society of
non-human animals where language is not available.
6.3. When reputation dynamics differ
In this paper, players have different behavioral
strategies but they all have the same reputation
dynamics. Instead, we may consider an evolutionary
game among players which differ in both reputation
dynamics and behavioral strategy. However, such aformalism is not necessarily wider in scope than thegame in which only behavioral strategy differs with
reputation dynamics ﬁxed, as shown below. The strategy
is then a pair of ðd ; pÞ. The condition for invasibility
should be calculated from the ﬁtness of a mutant actor

ðd 0 ; p0 Þ in a population dominated by residents ðd; pÞ.
First, we consider behavioral strategies are different(p0 ap). In indirect observation model, reputation iscontrolled by observers. The ﬁtness of the rare mutant
depends on the opponents and the observers, which are
all of dominant type ðd; pÞ. To calculate the ﬁtness of theinvader, we need the behavioral strategy of the mutant
p0 , that of the opponent p, and the reputation dynamics
of the observer d. Note that the reputation dynamics ofthe rare mutant d 0 has no effect to the mutant’s ﬁtness.
As a result, the invasibility of ðd 0 ; p0 Þ is exactly the sameas the invasibility of a mutant p0 playing with the
opponent p when reputation dynamics is ﬁxed as d.
Hence, the leading eight are the only possible ESSs evenwhen we regard ðd; pÞ as a strategy.
This argument excludes the case in which mutant andresidents have exactly the same behavioral strategy p but
differ in reputation dynamics ðd 0 adÞ. In such a case, themutant is perfectly neutral. Mutant frequency iscontrolled only by random genetic drift. A directcomputer simulation might show an apparent coexistence of multiple reputation dynamics in the population, which is in fact a neutral mixture. However, thisdoes not imply the absence of selection on reputation
dynamics in real societies. We suspect that reputation
dynamics in reality are very likely to be determined bygroup selection—a reputation dynamics that realizes asociety with a higher cooperation level is able to spreadand to drive away alternatives that realize a society witha lower level of cooperation. If so, discussing theevolution of the moral judgment in a game with ðd; pÞ
as a ‘strategy’ can be misleading, because it implicitly
assumes that the reputation dynamics should be formed
by intra-group selection.
Hence, treating reputation dynamics d as a ﬁxed ruleand considering a game between different behavioral
strategies under the rule d is a valid approach, which isexactly what we have done in our paper.
After we submitted our paper, we were informed that
Hannelore Brandt and Karl Sigmund had developed asimilar idea independently and also submitted theirpaper to this journal almost simultaneously (Brandt andSigmund, 2004; published in the same issue). In thefollowing, we brieﬂy explain the major differences
between two works. (1) Brandt and Sigmund adopted
direct observation model as a manner of introducing
errors. Hence, in their model different players may
have different ‘score’ of the same person. Players do
not exchange the information on the ‘score’ itself.
In contrast our H-score is ‘reputation’ formed insocial exchange of information among players. (2)Brandt and Sigmund developed an individual
based computer simulation model. They focused 3
reputation dynamics (they called ‘assessment modules’)ðd ¼ IMAGE; STAND; JUDGEÞ and 6 behavioral
strategies(theycalled‘actionmodules’)ARTICLE IN PRESS
H. Ohtsuki, Y. Iwasa / Journal of Theoretical Biology 231 (2004) 107–120

ðp ¼ CO; SELF; AND; OR; AllC; AllDÞ. Since AllC
and AllD do not need reputation dynamics, they ran
games among 3  4 þ 2 ¼ 14 combinations. In contrast,
we have examined all 4096 possible pairs and derived
ESS condition mathematically. As a result we succeeded
in discovering eight successful pairs of reputation
dynamics and behavioral strategies. (3) There aredifferences in terminology, some of which comenaturally from the difference in the basic idea of thetwo models.

119

Appendix B. Analytical solution of Eq. (4)In this section, we prove that (i) both ht ðpÞ and ht ðp0 Þ
converge to stationary values h ðpÞ and h ðp0 Þ, respectively, in the limit of t ! 1 and (ii) those limits are
independent of their initial ones h0 ðpÞ and h0 ðp0 Þ.
Let us begin the proof. Deﬁne Dij ðp0 Þ as
Dij ðp0 Þ  ð1  2ma Þfð1  me Þd ijp0ij þ me d ijD g þ ma
ði; j ¼ 1; 0Þ:(B.1)

We easily see that 0oma %Dij ðp0 Þ%1  ma o1 holds. Then
Eq. (4) becomes
Acknowledgements
We thank Hannelore Brandt, Toshikazu Hasegawa,Rie Mashima, Martin Nowak, Karl Sigmund, andNobuyuki Takahashi for their fruitful comments.

There exist 212 ¼ 4096 pairs ðd; pÞ to consider in ourframework. However, switching 1 and 0 in H-scorecauses mirror symmetry among them. If we completely
switch 1 and 0, a pair represented by ðd; pÞ becomes
equivalent to the pair ðd 0 ; p0 Þ, where¼

1  d 00X

d 010X¼

1  d 01X

d 001X¼

1  d 10X

d 000X¼

1  d 11X

p011

¼

p00

p010

¼

p01

p001

¼

p10

p000

¼

p11 :

¼

1  d 00X

d 10X

¼

1  d 01X

p11

¼

p00

p10

¼

p01

(B.2)

Substituting p for p0 in Eq. (B.2) yields

(B.3)

where
A2 ðpÞ ¼ D11 ðpÞ  D10 ðpÞ  D01 ðpÞ þ D00 ðpÞ;ðB:4aÞ

A1 ðpÞ ¼ D10 ðpÞ þ D01 ðpÞ  2D00 ðpÞ  1;

ðB:4bÞ

A0 ðpÞ ¼ D00 ðpÞ:ðB:4cÞ

Here we note the following proposition.
Proposition B.1. Consider the dynamical system over
½0; 1 , given byðX ¼ C; DÞðA:1Þ

In other words, ðd 0 ; p0 Þ is an mirror image of ðd; pÞ. The
mirror image shares the same properties as an originalone. Hence, we can examine only one pair of eachsymmetric dyad.
A pair ðd; pÞ is symmetric to itself whend 11X ð1  ht ðpÞÞD00 ðp0 Þ  ht ðp0 Þ:dht ðpÞ ¼ A2 ðpÞ fht ðpÞg2 þ A1 ðpÞ ht ðpÞ þ A0 ðpÞdt Iðht ðpÞÞ;Appendix A. Mirror symmetry

d 011Xdht ðp0 Þ ¼ ht ðp0 Þht ðpÞD11 ðp0 Þ þ ht ðp0 Þð1  ht ðpÞÞD10 ðp0 Þdtþ ð1  ht ðp0 ÞÞht ðpÞD01 ðp0 Þ þ ð1  ht ðp0 ÞÞðX ¼ C; DÞðA:2Þ

holds. Since 26 ¼ 64 cases satisfy Eq. (A.2), the total
number of effective pairs that need to be studied is now
26 þ ð212  26 Þ=2 = 2080.

d
x ¼ F ðxÞ ¼ ax2 þ bx þ c
dt

ðB:5Þ

If F ð0Þ40 and F ð1Þo0 hold, the limit x ¼ limt!1 xðtÞalways uniquely exists and it is independent of the initialvalue xðt ¼ 0Þ. To put it precisely, x is given as follows:
8
pﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ< b b2 4ac
if aa0;
2a
x ¼
(B.6)
:
 bc
if a ¼ 0:This can be proved by standard analysis of differential
equations.
We
note
that
Ið0Þ ¼ D00 ðpÞ^ma 40
andIð1Þ ¼ D11 ðpÞ  1%ð1  ma Þ  1 ¼ ma o0. Using thesefacts and Proposition B.1, we conclude that the valueh ðpÞ ¼ limt!1 ht ðpÞ exists and that it is independent ofh0 ðpÞ.
Let us consider ht ðp0 Þ next. Eq. (B.2) is rewritten as
follows:dht ðp0 Þ ¼ B1;t ðp; p0 Þ ht ðp0 Þ þ B0;t ðp; p0 Þ;
dt

ðB:7Þ

ARTICLE IN PRESS
H. Ohtsuki, Y. Iwasa / Journal of Theoretical Biology 231 (2004) 107–120

120

where
0

0

0

B1;t ðp; p Þ ¼ 1  fht ðpÞD11 ðp Þ þ ð1  ht ðpÞÞD10 ðp Þgþ fht ðpÞD01 ðp0 Þ þ ð1  ht ðpÞÞD00 ðp0 Þg;(B.8a)
B0;t ðp; p0 Þ ¼ ht ðpÞD01 ðp0 Þ þ ð1  ht ðpÞÞD00 ðp0 Þ:

ðB:8bÞ

When time t is large, ht ðpÞ approaches h ðpÞ, so Eq. (B.7)
becomesdht ðp0 Þ ¼ B1 ðp; p0 Þ ht ðp0 Þ þ B0 ðp; p0 Þ;
dt
where

ðB:9Þ

B1 ðp; p0 Þ ¼ 1  fh ðpÞD11 ðp0 Þ þ ð1  h ðpÞÞD10 ðp0 Þgþ fh ðpÞD01 ðp0 Þ þ ð1  h ðpÞÞD00 ðp0 Þg;(B.10a)B0 ðp; p0 Þ ¼ h ðpÞD01 ðp0 Þ þ ð1  h ðpÞÞD00 ðp0 Þ:0ðB:10bÞ

0

We see that 0oB0 ðp; p ÞoB1 ðp; p Þ holds. Therefore,
from Eq. (B.9) we have
lim ht ðp0 Þ ¼

t!1

B0 ðp; p0 Þð h ðp0 ÞÞ:
B1 ðp; p0 ÞðB:11Þ

Obviously, h ðp0 Þ is independent of its initial value h0 ðp0 Þ.
This ends the proof.
Since all the calculations above are analytic, thevalues of h ðpÞ and h ðp0 Þ can be expressed in terms ofseven parameters d; p; p0 ; b; c; me and ma . Hence, wehave the Taylor expansion of the payoff with respect toerror rates me and ma . Because their expressions are very
complicated and messy, we do not show them here.
Setting p0 ¼ AllD and me ¼ ma ¼ m, we have Eqs. (11)and (12). Eqs. (13) to (18) are derived in a similarmanner.
References
Alexander, R., 1987. The Biology of Moral Systems. Aldine de
Gruyter, New York.

Brandt, H., Hauert, C., Sigmund, K., 2003. Punishment andreputation in spatial public goods games. Proc. R. Soc. London
B 270, 1099–1104.
Brandt, H., Sigmund, K., 2004. The logic of reprobation: assessment
and action rules for indirect reciprocation. J. Theor. Biol., thisissue.
Enquist, M., Leimar, O., 1993. The evolution of cooperation in mobile
organisms. Anim. Behav. 45, 747–757.
Fehr, E., Fischbacher, U., 2003. The nature of altruism. Nature 425,
785–791.
Fehr, E., Gächter, S., 2000. Cooperation and punishment in public
goods experiments. Am. Econ. Rev. 90, 980–994.
Fehr, E., Rockenbach, B., 2003. Detrimental effects of sanctions onhuman altruism. Nature 422, 137–140.
Fishman, M., 2003. Indirect reciprocity among imperfect individuals.
J. Theor. Biol. 225, 285–292 (doi:10.1006/S0022-5193(03)00246-7).
Henrich, J., Boyd, R., 2001. Why people punish defectors; weak
conformist transmission can stabilize costly enforcement of norms
in cooperative dilemmas. J. Theor. Biol. 208, 79–89 (doi:10.1006/
jtbi.2000.2202).
Leimar, O., Hammerstein, P., 2001. Evolution of cooperation throughindirect reciprocity. Proc. R. Soc. Lond. B 268, 745–753.
Lotem, A., Fishman, M., Stone, L., 1999. Evolution of cooperation
between individuals. Nature 400, 226–227.
Nakamaru, M., Kawata, M., 2004. Evolution of rumors that
discriminate lying defectors. Evol. Ecol. Res. 6, 261–283.
Nowak, M., Sigmund, K., 1998a. The dynamics of indirect reciprocity.
J. Theor. Biol. 194, 561–574.
Nowak, M., Sigmund, K., 1998b. Evolution of indirect reciprocity by
image scoring. Nature 393, 573–577.
Ohtsuki, H., 2004. Reactive strategies in indirect reciprocity. J. Theor.
Biol. 227, 299–314 (doi:10.1016/j.jtbi.2003.11.008).
Panchanathan, K., Boyd, R., 2003. A tale of two defectors theimportance of standing for evolution of indirect reciprocity.
J. Theor. Biol. 224, 115–126 (doi:10.1016/S0022-5193(03)00154-1).
Pollock, K., Dugatkin, L., 1992. Reciprocity and the emergence ofreputation. J. Theor. Biol. 159, 25–37.
Sherratt, T., Roberts, G., 2001. The importance of phenotypic
defectors in stabilizing reciprocal altruism. Behav. Ecol. 12 (3),313–317.
Sugden, R., 1986. The Economics of Rights, Co-operation andWelfare. Blackwell, Oxford, UK.
Takahashi, N., Mashima, R., 2003. The emergence of indirect
reciprocity: is the standing strategy the answer? Center for the
study of cultural and ecological foundations of the mind,
Hokkaido University, Japan, Working paper series No. 29.
Trivers, R., 1971. The evolution of reciprocal altruism. Quart. Rev.
Biol. 46, 35–57.