April 20, 2005 12:56 00008

International Journal of Neural Systems, Vol. 15, Nos. 1&2 (2005) 93–99
c World Scientific Publishing Company

DYNAMICS OF BETTING BEHAVIOR UNDER FLATREWARD CONDITIONAYAKO ONZO∗ and KEN MOGI†Sony Computer Science Laboratories, Takanawa Muse Bldg, 3-14-13,
Higashigotanda, Shinagawa-ku, Tokyo, 141-0022, Japan
andTokyo Institute of Technology, 4259, Nagatsuta-cho, Midori-ku,
Yokohama, 226-8502, Japan∗
onzo@olive.ocn.ne.jp†
kenmogi@csl.sony.co.jp
One of the missions of the cognitive process of animals, including humans, is to make reasonable judgments and decisions in the presence of uncertainty. The balance between exploration and exploitation
investigated in the reinforcement-learning paradigm is one of the key factors in this process. Recently,
following the pioneering work in behavioral economics, growing attention has been directed to humanbehaviors exhibiting deviations from the simple maximization of external reward. Here we study the
dynamics of betting behavior in a simple game, where the probability of reward and the magnitude ofreward are designed to give a “zero” expected net reward (“flat reward condition”). No matter how thesubject behaves, there is on average no change in one’s resources, and therefore every possible sequenceof action has the same value. Even in such a situation, the subjects are found to behave not in a randommanner, but in ways showing characteristic tendencies, reflecting the dynamics of brain’s reward system.
Our results suggest that brain’s reward system is characterized by a rich and complex dynamics only
loosely coupled with external reward structure.
Keywords: Reward; neuroeconomics; uncertainty; betting; dopamine.

alternative sources of reward, and might work unfavorably for one’s survival. The balance between
exploitation and exploration is thus importantfor survival, and has been investigated in thereinforcement-learning paradigm.5
In the developmental process, the psychologicalsafe base provided by caretakers is considered to be anecessary basis for the infant’s voluntary explorationof novel stimuli.6–7 Perception of safe base as a basis
for exploration is likely to be relevant also in mature
humans.
Shultz and his colleagues revealed that dopamine
neurons code uncertainty itself.8 There was asustained increase in activity that grew from theonset of the conditioned stimulus to the expected
time of reward. The peak of the sustained activation occurred at the time of potential reward, whichcorresponds to the moment of greatest uncertainty.
These results suggest that dopamine neurons might

1. Introduction
Animals, including humans, encounter novel stimuli in the course of life, incurring cognitive uncertainty. How animals coordinate their actions in suchan uncertain environment is one of the crucial questions to be asked of cognition.
Metacognition is considered to be essential in
the robust handling of uncertainty.1–4 Hampton
reported on the metacognitive ability of rhesus
monkeys.3 It was found that the monkey has a“metacognition” of its internal state, i.e., its own
assessment of the likelihood of conducting the task
successfully. This kind of cognitive process is likely
to contribute to a more stable handling of uncertain
situations.
Avoiding uncertainty is not necessarily adaptive.
To keep exploiting only certain sources of reward istantamount to dismissing opportunities to explore
93April 20, 2005 12:56 00008

94

A. Onzo & K. Mogi

respond to uncertainty itself and that uncertainty
could be regarded as a secondary reward, providing a possible explanation for addiction to gambling.
The temporal parameters involved in the learningof action-reward association, e.g., the discount rate,
and their correlate in the dopamine system,9–11 are
expected to be important in the metacognition of
uncertainty and related cognitive processes.
A variety of studies in economics have shownthat humans make decisions under uncertainty based
not on the expected value but on the subjective
expected utility.12–16 The tendencies of risk seeking or aversion12–13 and of loss aversion14–16 havebeen explained by taking the magnitude relationshipof preference between the options into account. No
matter how well a theory is developed to account forpeople’s decision making, there is always a certain
degree of uncertainty involved. People do not always
make the same choice even under exactly the same
condition. In this respect, it is important to clarify
the nature of brain’s internal dynamics related to
uncertainty handling as is reflected in people’s decision making.
Investigating people’s betting behavior is one of
the simple and effective tools for uncovering thenature of the cognition of uncertainty. In most studies, the game is designed in such a way that a change
in betting behavior leads to a change in the expected
gain, reflecting real life situations. On the other
hand, it is possible that people’s betting behavior
exhibit internal dynamics of its own, in dissociation
with externally defined reward structure.
Here we investigate the dynamics of betting
behavior in a simple game where the expected
net reward value is constant (zero) regardless of
the action chosen (flat reward condition). We
conjectured that this particular paradigm reveals
the nature of internal neural dynamics involvedin people’s judgment under uncertainty, which
is only loosely coupled with external defined
reward.
2. Methods
We conducted a betting game that consisted of
20 trials. The participants were 12 healthy young
adults (ages 22–29, 6M & 3F). The subjects were
undergraduate and graduate students in several
universities in Tokyo area, and possessed basic

knowledge of probability. The subjects were awarethat the outcomes of betting would be determined by
the simulated random process in the computer, andthat there was no way to influence the outcome by
any system of betting. The subjects were instructed
about the general conditions involved in the gamebeforehand and gave informed consent. The subjects
started with an initial resource of 5 units, and tried
to increase the amount as best as they could. The
buttons and letters were displayed on a laptop computer (Sony VAIO PCG-505G/B).
In each trial, the subject had a choice ofeither betting or escaping, by pushing the “Bet”
or “Escape” button that became active 5 seconds
after pushing the “Next” button (Fig. 1). The trial
number, probability of winning (=0.25, always thesame), and the current resource were displayed. For
the very first turn, the buttons became active 5 seconds after launching the game software. There was no
explicit time pressure, and the subjects were allowed
to use as much time as they liked when making
the decisions. The delay from the activation of the“Bet” and “Escape” buttons to the moment when
the subjects made the choice was recorded. Whenthe subject pushed the “Bet” button, 1 unit was
taken from the subject’s resource. The probabilityof winning was fixed at 0.25. If the subject won,
4 units were added to the resource. The expected
net gain when betting was therefore zero. When thesubject pushed the “Escape” button, the resource
remained the same. Thus, there was no change inthe expected value of gain no matter what choice(“Bet” or “Escape”) the subject made (flat rewardcondition). From the probabilistic point of view, thesubject had no rational motivation to bet or escapewith a particular strategy, and any deviations froma random betting pattern can only be explained interms of cognitive bias and/or illusion, with no actual
contribution to the net gain.
The feedback (“You Win!” “You Lose!” or “YouEscape!”) was displayed immediately after the subject pushed the “Bet” or “Escape” button. The
screen remained the same until the subject pushed
the “Next” button. The game was over when the
resource became 0 or the trial number reached 20.
Each subject repeated 30 games. When the 20 trials
were over, the resource was set back to the initialvalue of 5, and the gains and losses were not carriedover to the next session.

April 20, 2005 12:56 00008

Dynamics of Betting Behavior Under Flat Reward Condition

Fig. 1.

95

Experimental protocol.

The behavior of the subjects can be characterized by
the probability of the subject to bet (betting rate),
Nbet /Ntrial , where Nbet and Ntrial are the numberof betting and trials, respectively. By obtaining the
betting rate under various criteria, we characterized
the behavior of the subjects.
From the probabilistic point of view, there is
no rationale for the subjects to bet using any specific strategy. Any tendency away from the random betting behavior would suggest the existenceof (unconscious or conscious) strategy that the subjects employed, in which the internal dynamics ofthe brain’s reward and decision-associated areas isreflected.
Figure 2 shows the relation between the amountof resource and betting rate. The rate is flat for
amount of resource up to ∼16, and then again stays
at a higher fixed level for larger amounts of resource.

Betting rate

3. Results

1

1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

2
3
4
5
6
7
8
9
10

0

5

10

15

20

25

Resource

Fig. 2.

30

11
12
All

Resource and betting rate.

As there are considerable individual differences, weplot the average as well as individual data in thisand following figures. The subject numbers are usedconsistently throughout. Since the game is over when
the resource becomes zero, it is psychologically natural to bet more frequently when more resource isApril 20, 2005 12:56 00008

Betting ratio

accumulated through winning (“safe base” effect).
Note that there is a slight increase of betting
rate when the resource is small, which cannot be
explained by the safe base effect.
The trial number could also influence the betting
rate. A subject could, for example, take a greater risk
towards the end of the 20 trials set within a game.
Note that the subjects were aware of the trial number, as it is displayed clearly on the game screen (seeMethods). Figure 3 shows the relation between thetrial number and betting rate. There is no apparent tendency for the subjects to bet more frequently
towards the end of the session. The absence of theeffect of trial number is consistent throughout the
subjects, although the betting rate differs among
subjects.
Since the probability of winning in a trial is independent of past outcomes, there can be no rational
explanation for the dependence of betting behavior,
if any, on the previous outcomes. Figure 4 shows thedependence of betting rate on the immediately previous outcome (Lose, Win, or Escape). There is a statistically significant tendency to bet more frequently
after losing, than winning (p < 0.05). Note the considerable individual differences in the betting rate.
When averaged over the subjects, there was no
significant difference in the reaction time no matter which choice (“Bet” or “Escape”) the subject
made. In addition, no significant dependence of thereaction time on the previous outcome (“Escape”,“Lose”, or “Win”) was found (Fig. 5). This statistical insignificance, however, is due to the large standard deviation in the reaction time. The individual plots for average reaction time for making the“Bet” or “Escape” choice indicate that the subjects
do actually take longer when choosing to bet than

Mean betting rate

A. Onzo & K. Mogi
All

1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

1
2
3
4
5
6
7
8
9
10

Lose

Fig. 4.

Win

11

Escape12Previous outcome and betting behavior.

3.5
Mean reaction time [sec]96

Previous
outcome

3
2.5

All
Lose
Win
Escape

2
1.5
1
0.5
0
EscapeFig. 5.

Bet

Betting behavior and reaction time.

















 

 

 





"!
# 
%$
&#')(

Fig. 6. Reaction time of individual subjects for “Bet”
and “Escape” choices.

1

1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

2
3
4
5
6
7
8
9
10

0

Fig. 3.

5

10
Trial

15

20

Trial number and betting behavior.

11
12
All

to escape (Fig. 6). The longer delay before deciding
to “bet” likely reflects the nature of cortical processing involved. Note that two subjects (number
2 and 7) opted to keep betting and never escaped(often resulting in the termination of the gamebefore 20 trials). Therefore, their data do not appear
in Fig. 6.
Data analysis so far suggests that there are considerable individual differences in betting behavior.
There are important aspects of betting dynamics
that are not apparent when averaged. The dynamics

April 20, 2005 12:56 00008

Dynamics of Betting Behavior Under Flat Reward Condition

1

1

2
3

Betting rate

0.8

4

0.6

5
6

0.4

7
8

0.2

1
Betting rate

0.8
Betting streak

0.6

Losing streak

0.4

Winning streak

0.2
0

0

1

2

3

4

5

Length of each streak

Fig. 8. Betting rate in betting, losing, and winningstreaks.
1

0.6

2

0.5
Winning rate

of betting behavior might be heterogeneous even in
an individual. It seemed possible that the betting
behavior as is indicated in Fig. 2, which can beapproximated as a Markov process, is actually composed of heterogeneous modes of betting dynamics,
with the subjects behaving in distinct manners awayfrom the random behavior.
Since there are 3k possible outcomes for k consecutive trials, it is difficult to analyze all possible trajectories of betting behavior. After some preliminary
analysis, we focused on a particular mode of betting
behavior, where the subject keeps betting for several
trials without escaping (“betting streak”).
Figure 7 shows the “cornering effect” in a betting streak. Here, the betting rate is plotted against
the number of consecutive choices of betting alreadymade. The betting rate keeps increasing, away fromthe random behavior where the betting rate should
stay at a constant level. When a subject keeps betting, it appears that he or she is “cornered” intoa state where there the betting rate continues toincrease, resource permitting.
It is possible that a betting streak is driven by
consecutive winnings or losses. In consecutive winnings, the subjects might “feel good” and keep betting. In consecutive losses, they might be motivatedto make up for the loss. We analyzed the betting ratein “winning streaks” (where the subject keeps betting and winning) and “losing streaks” (where thesubject keeps betting and losing). Note that winning streaks and losing streaks are subsets of betting streaks. Comparison between the data suggeststhat the outcome does not have significant effecton the subject’s behavior in a betting streak (Fig. 8).
The subject seems to be determined to keep betting,regardless of the result.

97

3
4

0.4

5

0.3

6

0.2

7
8

0.1

9
10

0
0

5

10

15

Length of betting streak

Fig. 9.

20

11
12
All

Cumulative winning rate in a betting streak.

Although no statistically significant difference inbetting rate was found in the comparison between
the three kinds of streaks, there indeed was a small
tendency that a betting streak is induced by consecutive winnings. Figure 9 provides evidence that
supports the hypothesis that a betting streak is
induced by winning repeatedly. When we calculate
the cumulative winning rate (i.e., the average winning rate within a betting streak), it becomes significantly higher as the length of the betting streakis increased. This result suggests that at least somebetting streaks are induced by an event where thesubject had the fortune to win above the chance levelof 0.25. Note that when averaged over all trials, thebetting rate is higher after losing, rather than winning (Fig. 2). Thus, the behavior of the subjects ina betting streak is a phenomenon separate from theaverage tendency.

9
10

0
0

5

10

15

Length of betting streak

20

11
12
All

Fig. 7.

Betting streak.

4. Discussions
The robust handling of uncertainty is important forsurvival. It is interesting to ask how we recognize andjudge uncertainty, and choose appropriate actions.

April 20, 2005 12:56 00008

98

A. Onzo & K. Mogi

Subjective bias and illusions are integral part of thebrain’s cognitive process of uncertainty handling,
and merits investigation apart from the context of
external reward optimization
A large variety of studies in economics and neuroeconomics have taken subjective factors that influence a subject’s decision making.13 For instance,
Kahneman and his colleagues studied the introspective value of lotteries.16 In their experiment, the subjects had to choose between lottery A and B. In A,
there is a sure loss of $750. In B, there is a 75%
chance to lose $1000 and a 25% chance to lose nothing. They found that although both lotteries had an
identical expected value, a clear majority of respondents preferred B (13% of the subjects chose A and
87% chose B). This result suggests that there isa risk seeking preference on this kind of negative
choice. They obtained a hypothetical value function
by investigating people’s preference. The value function is (a) defined on gains and losses rather than on
total wealth, (b) concave in the domain of gains andconvex in the domain of losses, and (c) considerably
steeper for losses than for gains. Social factors canalso influence people’s perception of uncertainty.
It is important to study the cognition of uncertainty in the full richness of its dynamics, since our
behavior is embedded in the constantly changing situations in daily life. In our experiment reported here,
the subjects exhibited characteristic betting dynamics in a simply designed game. Since the expected
gain is constant regardless of the subject’s behavior,differential behavior cannot be explained on the basisof reward optimization. The differential behavior of
the subject can only arise from the brain’s internal
dynamics, reflecting neural mechanism for evaluating
reward and making decisions in an uncertain environment. Informal interviews with the subjects after
the experiment suggested that they were in generalunaware of the fact that they were behaving differentially depending on the previous outcome. Therefore,
the betting dynamics is likely to reflect unconscious
tendencies, rather than an application of conscious
strategies.
Several factors seem to affect the betting
behavior. The perception of the resource as safe
base, memory of recent results, and perception of
the probability of winning are some elements affecting the betting behavior observed. The complex neural computation involving the representation of thesefactors, finally culminating in a winner-take-all typedecision-making, is likely to be an integral part of therobust handling of uncertainty. The betting dynamics under the flat reward condition is neutral toreward, and the evolution of the neural dynamics
might be understood in a context neutral to theimmediate reward.17 At least some aspects of betting dynamics (e.g., higher betting rate after losing compared to winning) cannot be explained by
the immediate optimization of gain. Needless to say,these cognitive processes need to contribute to, orat lease be compatible with, the efficient utilizationof external reward in the environment. The enrichment of neural dynamics in a context neutral toexpected gain might contribute to the final fitness.
The exact logic behind such a development of neutral dynamics, however, needs to be clarified in future
investigations.
Finally, there was a considerable individual difference of betting behavior. As already mentioned,
two subjects chose to keep betting and never escape.
Apart from these extreme cases, there is a wide variety of betting patterns among the subjects. Anecdotal evidence suggest that such heterogeneity ofstrategy is typically observed in gaming under thepresence of uncertainty, and might reflect a general tendency of the neural system involved in therobust handling of uncertain situations. In particular, in situations involving interaction with anotheragent, such as the ultimatum game,18 and prisoner’s dilemma,19 heterogeneity of strategies might
induce rich interpersonal dynamics, increasing thecomplexity of social interaction and contributing tothe overall fitness of the group of people involved.
Acknowledgments
We thank Ruggiero Cavallo, Hisayuki Nagashima,
Kei Omata, Takayasu Sekine, Tamami Sudo, Fumiko
Tanabe, Fumihiko Taya, Takanori Yokoyama,
Toru Yanagawa, and Zhang Qi for their helpful
discussions.
References
1. G. Graham and J. Neisser, Probing for relevance:
What metacognition tells us about the power ofconsciousness, Conscious Cogn. 9(2 Pt 1) (2000)
172–177.
2. D. Griffiths, A. Dickinson and N. Clayton, Episodic
memory: What can animals remember about their
past? Trends Cogn. Sci. 3(2) (1999) 74–80.

April 20, 2005 12:56 00008

Dynamics of Betting Behavior Under Flat Reward Condition

3. R. R. Hampton, Resus monkeys know when they
remember, PNAS 98(9) (2001) 5359–5362.
4. A. Koriat, The feeling of knowing: Some metatheoretical implications for consciousness and control,
Conscious Cogn. 9(2 Pt 1) (2000) 149–171.
5. R. S. Sutton and A. G. Barto, Reinforcement Learning (MIT Press, 1998).
6. M. D. Ainsworth, Object relations, dependency,
and attachment: A theoretical review of the infantmother relationship, Child. Dev. 40(4) (1969)
969–1025.
7. J. Bowlby, Attachment (Perseus Books, 1982).
8. C. D. Fiorillo, P. N. Tobler and W. Schultz, Discrete coding of reward probability and uncertainty
by dopamine neurons, Science 299(5614) (2003)
1898–1902.
9. W. Schultz, Multiple reward signals in the brain(Review) Nat. Rev. Neurosci. 1(3) (2000) 199–207.
10. W. Schultz, P. Dayan and P. R. Montague, A
neural substrate of prediction and reward, Science
275(5306) (1997) 1593–1599.
11. P. Shizgal and A. Arvanitogiannis, Neuroscience.
Gambling on dopamine, Science 299(5614) (2003)
1856–1858.

99

12. D. Bernoulli, Exposition of a new theory on the measurement of risk, Econometrica 22 (1954) 23–36.
13. P. W. Glimcher and A. Rustichini, Neuroeconomics:
The consilience of brain and decision, Science306(5695) (2004) 447–452.
14. D. Kahneman and A. Tversky, Prospect theory: An
analysis of decision under risk, Econometrica 47(1979) 263–291.
15. D. Kahneman, P. Slovic and A. Tversky, Judgment
Under Uncertainty (Cambridge University Press,
1982).
16. D. Kahneman and A. Tversky, Choices, Values, andFlames (Cambridge University Press, 2000).
17. M. Kimura, Evolutionary rate at the molecular level,
Nature 217 (1968) 624–626.
18. M. Milinski and C. Wedekind, Working memory
constraints human cooperation in the prisoner’s
dilemma, in Proc. Natl. Acad. Sci. 95(23) (USA,1998), pp. 13755–13758.
19. A. G. Sanfey, J. K. Rilling, J. A. Aronson,
L. E. Nystrom and J. D. Cohen, The neural basis of
economic decision-making in the Ultimatum Game,
Science 300(5626) (2003) 1755–1758.

April 20, 2005 12:56 00008