Discounting, Values, and Decisions∗John K.-H. Quah

Bruno Strulovici

Oxford University

Northwestern University

March 24, 2013

Abstract
How do discount rates affect agents’ decisions and valuations? This paper provides acommon methodology and a systematic analysis of this question, considering stochastic
and managed cash-flows, stochastic discount rates, time inconsistency, and including
arbitrary learning and payoff or utility processes. We show that some of these features
can lead to counter-intuitive answers (e.g., “a more patient agent stops earlier”), but wealso establish, under simple conditions, theorems yielding robust and unifying answers
on the impact of discount rates on control and stopping decisions and on valuations. We
apply our theory to models of search, experimentation, bankruptcy, optimal growth,
investment, and social learning.

∗

We are grateful for the suggestions and comments from the Editor and three anonymous referees,
as well as comments from Chiaki Hara, Johannes Hörner, Paul Milgrom, Frank Riedel, Kareen Rozen,
Larry Samuelson, Philipp Strack, Jean Tirole, and seminar participants at the World Congress of the
Econometric Society (2010), Kyoto University, Kobe University, Konan University, Brown University, Yale
University, and the SAET Conference (2011). Email addresses: john.quah@economics.ox.ac.uk and bstrulovici@northwestern.edu.

1Introduction

How do discount rates affect agents’ decisions and valuations? As economists, we are routinely confronted with this question: whether they concern savings and refinancing decisions,the exercise of options, or experimentation with reforms or new technologies, discount rates
are at the heart of economic analysis. Given its importance, one might expect there to bea unified and general treatment of this question, but that is not the case. While there exist
papers that deal with the general relationship between net present value, discount rates,and optimal stopping (in particular, Samuelson (1937), Arrow and Levhari (1969), and Sen(1975)), those works only consider the problem in a non-stochastic context.1 When the environment is stochastic, answers have been given in different models using different methods,
but there have been few attempts at an overarching and systematic analysis; the aim of thispaper is to do precisely that.
With the multitude of phenomena that could arise in a general setting, it is not clear thatrobust comparative statics for discount rates are even possible. Indeed, it is known that, evenin a deterministic context, a lower discount rate can reduce the present value of a projectand precipitate its termination if it has a negative termination value (Sen, 1975) or if there
are restrictions on the dates at which the project can be interrupted (Samuelson, 1937).
Arrow and Levhari (1969) have shown that for deterministic pure stopping problems withno termination values and no restrictions on stopping times, both the stopping time and thenet present value are increasing with patience. However, most economic applications do not
fit in that framework. We consider more realistic environments where (i) cash/utility flows
are stochastic and there are stochastic (possibly correlated) termination values; (ii) the agent
can manage or influence in some way his cash flows (so his decision problem involves both
stopping and control); (iii) discount rates are also stochastic and correlated with payoff orutility flows; and (iv) the agent is time inconsistent.2 For reasons explained in Section 2.1,
standard comparative statics techniques are ill suited to analyze the impact of discounting
on decisions. In this paper, we develop a unifying methodology that achieves precisely that.
General results on discounting, stopping, and control
1

These papers were concerned with finding conditions guaranteeing the monotonicity of the value ofcapital with respect to interest rates and the uniqueness of an internal rate of return. In addition to theapplications provided in the present paper, our results also provide a general answer to these questions.
2
Our analysis focuses on a single agent, but the methodology promises fruitful applications to the analysis
of dynamic games (see the brief discussion in the Conclusion (Section 6)). For example, in a game where
players choose effort and efforts are strategic complements, a change in the discount rate that leads to greater
effort from one player will lead to higher equilibrium efforts for all players.

2For pure stopping problems with deterministic discounting, we prove that even with stochastic cash/utility flows and stochastic (but non-negative) termination values, the agent withthe lower discount rate stops later almost surely; similarly the net present value of the
project also increases with a lower discount rate. Many economic decisions (such as searchand experimentation, of which more below) can be modeled as pure stopping problems, so
our general result will apply in all of those contexts. There are also problems where theagent has the ability to manipulate cash flows or utility flows; in formal terms these areproblems with both stochastic control and stopping. For these problems, one could showthat when the utility flow is a function of some Markov process, the continuation domain(i.e., the set of states at which stopping is suboptimal) is always shrinking with the discountrate. However, this does not necessarily mean that the more patient agent stops later and
we give an example of a counter-intuitive situation where the opposite happens. To restore
a positive result, more structure has to be imposed on the agent’s decision environment. Ina large class of models, utility depends on some one-dimensional state variable that follows a
Markov process, such as the wealth of the agent, or the stock of capital available. We show
that, as long as the utility of the agent is increasing in the state, a lower discount rate leadsto (a) higher “effort” to increase the state, (b) a higher state trajectory (pointwise, almost
surely), and (c) later project interruption.
In his presidential address to the American Finance Association, Cochrane (2011) argues
that changes in discount rates matter much more than changes in cash flows in explaining
asset return variation. Cochrane’s address focuses on stochastic discount rates, which arepervasive in financial and macroeconomic analysis. In this context, we may consider an agent
to be more patient than another if he has a lower discount rate almost surely. However, evenwith this strong definition of greater patience, one could build examples where the correlation
between cash flows and discount rates leads to the more patient agent stopping sooner. To
restore a positive result, additional conditions are needed; we show it suffices that (loosely
speaking) lower discount rates are correlated with higher cash flows.
An oft-studied theme in behavioral economics is time inconsistent behavior, which can havea major impact on the timing of decisions (see, e.g., Akerlof (1991), O’Donoghue and Rabin (1999)). However, the issue of how time inconsistent agents may be ranked by theirpatience, and whether such a ranking is related to decisions in a way similar to that between
the discount rate and decisions for time consistent agents, is a largely unexplored issue. We
define an agent to be more patient than another if every one of his future selves is morepatient than the corresponding future self of the other agent. We prove that for all stochastic pure stopping problems, the more patient agent stops earlier, provided the agent (and3

everyone of his future selves) uses a discount function obeying a certain condition satisfied
by hyperbolic and also quasi-hyperbolic discount functions. This result is easily shown for
naive agents; for sophisticated agents whose actions result from an equilibrium amongst theagent’s multiple selves, the proof is more challenging. Using the same methods, we also showthat for a given time inconsistent agent, increasing his sophistication (in the sense definedby O’Donoghue and Rabin (2001)) leads to earlier stopping almost surely.
Applications to specific models
Our results can be applied to yield general results on the impact of discounting in a numberof important economic models.
(1) The standard economic model of search is an example of a pure stopping problem. For
certain highly stylized search models, it is known that a more patient agent will search fora longer period. However, in many applications (e.g., to labor and housing markets), it isreasonable to suppose that the agent learns about conditions affecting search outcomes along
the search process and he can also become better at searching, which reduces his search cost.
In formal terms, this means that search costs may be stochastic and non-stationary overtime, and the distribution of the termination value can also be state and time dependent. In
some situations, objects encountered in the past can be recalled while in other situations, it
may be reasonable to exclude this possibility; does this influence the impact of discount rates
on optimal stopping? Our general result tells us that, with or without these complexities,
the more patient agent stops searching later.
(2) It is standard in economics to model experimentation as a bandit problem (see Rothschild (1974)). In its simplest form, the agent has to decide when to switch from playing
the ‘risky’ arm whose payoff distribution is unknown to playing the ‘safe’ arm, which hasa known payoff distribution. This is a stochastic pure stopping problem and it follows immediately from our result that the more patient agent will switch to the safe arm, i.e., stop
experimenting, later. In a multi-armed bandit problem with independent arms, this result
can be extended to show that the duration of experimentation on each risky arm (and hence
the total time spent on experimentation) also increases with patience.
(3) The recent literature on corporate default has emphasized the role of shareholders’optimizing behavior on the timing of bankruptcies. These models typically assume thatshareholders’s only decision is to decide on when to default; in cases where the setup issufficiently simple, an explicit solution to this problem can be obtained (see, for example,
Leland (1994)). More realistically, debt payments may depend in sophisticated ways on past
performance and shareholders also make decisions on (say) project technologies that have4

an impact on cash flow. Thus the shareholders’ problem involves both optimal control andstopping. Whether or not a model with these features can be solved explicitly, our results
guarantee that a lower discount rate leads to later bankruptcy.
(4) A basic question in growth theory is whether a more patient representative agent leadsto ‘capital deepening’, i.e., a higher capital stock at all times (see, for example, Amir et al.
(1991)). Models giving this result are typically deterministic, involve strong assumptions on
the form of the utility and production functions, and restrict the agent’s decision to that ofdeciding on the level of consumption/saving at each period. We show, in a continuous time
economy with a stochastic production technology, that a more patient representative agent
leads to capital deepening almost surely; this result does not require shape restrictions on
the utility and production functions (thus permitting increasing returns to scale) and also
allows the agent’s control variable to be multi-dimensional (thus allowing for other decisionvariables, besides consumption, to affect capital accumulation).
(5) In consumption-based asset pricing models, the cash flow of a project is discounted bya stochastic discount function that is proportional to the marginal utility of the economy’s
representative agent at each time and state. We establish a systematic relation between an
agent’s risk attitude and the “patience” implied by his discount function: the less risk averse
agent, the more patient is the discount function, which in turn means that projects will havehigher net present value and terminate later.
Organization of the paper
Section 2 is devoted to the pure stopping problem, beginning with a discussion of the deterministic case to provide motivation. Problems involving both stopping and control aredealt with in Section 3, while Section 4 considers pure stopping problems with stochastic
discounting (in addition to stochastic payoffs). Most of our results do not depend on discrete
time versus continuous time specifications. For simplicity, counterexamples are mostly given
in discrete time, while general results are mostly stated in continuous time. The exception
is Section 3.2 and applications of its main result; the continuous time formulation seems
crucial in that case. Time inconsistency is discussed in Section 5; to keep things technicallymanageable, we use a discrete time formulation in that section. All the substantive sections
begin with a formulation of the general theory followed by applications to widespread economic models. Some of the simpler proofs are contained in the main part of the paper; allothers are found in the Appendix. Section 6 concludes.

5

2

The Stochastic Pure Stopping Problem

In this section we consider how the value and optimal stopping time of a pure stopping
problem varies with the discount rate. This issue has been fully solved in the case where
payoffs are deterministic but not when it is stochastic. Nonetheless, we shall begin witha brief discussion of the deterministic case because this simpler setting allows us both tohighlight some of the difficulties involved in establishing a general result and also to develop
some intuition for why a general result may be possible.

2.1

Background: the deterministic case

Suppose there is a project that gives a cash flow – which can be either positive or negative– of π(s) at time s over some interval [0, t̄]. The project can be stopped at no cost at sometime before t̄. An agent with a discount rate of r chooses the stopping time t to maximize
Rt
the discounted cash flow U (t, r) = 0 e−rs π(s)ds. The value of the project, as a function ofr is V (r) = U (τ (r), r), where τ (r) is the optimal stopping time at the discount rate r. The
question we wish to ask is the following: how do τ and V (r) vary with r?
The first answer to this problem was provided by Arrow and Levhari (1969), who showedthat τ and V are both decreasing in r. The proof given by these authors is rather complicatedand involves ancillary assumptions, even though the reader may at first blush wonder why
this problem is not completely straightforward. It is not straightforward because common
methods for dealing with comparative statics problems are not applicable in this case. To
see this note that U (t, r) is not typically concave in t; indeed, since ∂U(t, r) = π(t), U has∂ta local maximum at t∗ whenever π(t∗ ) = 0 and π is locally decreasing. What is worse, thelocal maxima do not vary with r, so there is no straightforward way of determining how
τ (r), which is the time at which U is globally maximized, varies with r.
A more sophisticated approach may be to check for single crossing differences (see Milgrom
and Shannon (1994)). Using this approach, a sufficient condition for τ (r) to be decreasing
in r is the following: for all t2 > t1 and r0 > r,
U (t1 , r) ≥ (>) U (t2 , r) =⇒ U (t1 , r0 ) ≥ (>) U (t2 , r0 )(1)

We now provide an example which illustrates why single crossing differences can fail and also
hints at a way out of the problem. Consider a project, active over the time interval [0, 3],
which has the following payoff: π(t) = A > 0 for t ∈ [0, 1], π(t) = −B < 0 for t ∈ (1, 2] andπ(t) = C > 0 for t ∈ (2, 3]. The local maxima of U (t, r) are at t = 1 and 3, and depending6

on r, either could be the global maximum. Now it is straightforward to check that
U (0, r) − U (2, r) = r−1 (1 − e−r ) Be−r − A .
which, when B is large, goes from positive to negative values as r increases, so there is a
failure of single crossing differences. Put another way, if the agent can only stop at t = 0 or
2, then less patience can lead to later stopping, which is the opposite of what we want. But
in fact, the agent can stop at any time between 0 and 3, so it does not matter that t = 2
becomes preferred to t = 0 as r increases because t = 0 is never optimal. On the other hand,U (1, r) − U (3, r) = r−1 e−r (1 − e−r ) −Ce−r + B ,which does go from negative to positive as r increases (and thus obeys (1)). This meansthat if stopping at t = 1 is preferred to stopping at t = 3 for some value of r, it will remain
preferred as r increases: the impatient stops earlier.
This example suggests that what is needed to establish the monotonicity of τ (r) is something
weaker than single crossing differences — a concept that restricts payoff comparisons to thosepairs of choice variables that are relevant for global optimality. In Quah and Strulovici (2009),
we develop a way of ranking payoff functions (called the interval dominance order) which
has this feature and we use it to establish the monotonicity of τ (r). The techniques in that
paper can be developed (in a non-trivial way) to obtain a general result relating patienceand stochastic optimal stopping, even though the stochastic nature of the problem precludes
a straightforward application of the interval dominance order.

2.2

Stochastic optimal stopping

For T = [0, t̄] or T = [0, ∞), let {ut }t∈T and {Gt }t∈T denote real-valued stochastic processes
adapted to the filtration F of some filtered probability space (Ω, P, F = {Ft }t∈T ). The valuefunction of a pure stopping problem is given byZ τV (α) = sup U (τ ; α) = E
α(s)us ds + α(τ )Gτ
(2)
τ ∈T

0

where T denotes the set of stopping times adapted to F and taking values in T . We assume
that the discount function α : T → (0, ∞) is deterministic (for now) and strictly positive,
R t̄
with the normalization α(0) = 1. We also assume that E[ 0 α(s)|us |ds] is finite, α(t) and|Gt (ω)| are uniformly bounded, α and G have bounded variation, that all processes are rightcontinuous and have left limits, and that the filtration F is right continuous and satisfies theusual conditions (see, e.g., Karatzas and Shreve (1991)). These assumptions guarantee that7

all expectations (including conditional expectations) are (almost everywhere) well-defined,that Fubini’s Theorem can be used, and that Lebesgue-Stieltjes integrals with respect to α
and G are well defined.
The discount function α gives, at each time t, the discount factor between time 0 and time t.
By our definition, the stopping time τ that solves (2) is optimal from the perspective of anagent at t = 0. The agent is time-consistent, i.e., will not revise at time s > 0 the optimal
stopping time chosen initially, if his discount function at time s is given by ᾱ(s, t) = α(t)/α(s)where t (with t > s) refers to the time that has elapsed from time 0. It would be reasonable
to restrict discount functions to be decreasing in t, though our results do not in fact requirethat assumption.
For expositional simplicity, we assume that the agent stops if and only if he is indifferent
between stopping and continuing, and denote by τ (α) the resulting stopping time. All of
our results can be stated more generally by endowing the set of stopping times with a lattice
structure and making statements in terms of the strong set order. Formally, τ (α) is theessential infimum of all stopping times that are optimal when the discount function is α.
The right continuity of G and of the filtration {Ft }t∈T guarantee that τ (α) is an optimal
stopping time.3
The stopping problem as stated assumes that the flow payoff after stopping is zero. Whenthe agent is making a purely financial decision (such as a firm deciding on when to invest in
a project, or a landlord deciding on when to sell an apartment, etc.), then the zero payoff
simply means that the cash flow ceases when the project stops. The agent’s objective in (2)is to maximize the discounted cash flow and V (α) can then be interpreted as the present
value of the project. In other settings, the post-stopping zero flow payoff is a normalization
and what happens after stopping depends on the setting. For example, imagine an agent
who is engaging in some activity X that gives him a stochastic consumption of xt at timet. He has the option of switching, without cost but only once, away from this activity toanother activity Y that gives a stochastic consumption of yt at time t.4 The agent has a
discount function α and is risk averse, with the Bernoulli utility function w. Thus the agent’s
problem is to choose a stopping (or, in this context, a switching) time τ that is adapted to3

The essential infimum of an arbitrary family of random variables is uniquely defined up to a set ofmeasure zero (see Durrett (2004)). The right continuity of G and of the filtration {Ft }t∈T guarantee thatthe infimum is itself optimal (see Lamberton and Lapeyre (2007)).
4
Note that yt does not depend on the time at which the agent switches from X to Y .

8F and maximizes his lifetime utility
#
"Z
Z t̄
τ
α(s)w(ys )ds .
E
α(s)w(xs )ds +

(3)

τ

0

Clearly, we can reformulate this problem by treating the agent as choosing τ to maximizeZ τEα(s) (w(xs ) − w(ys )) ds ;(4)0

this problem then takes precisely the form (2), with us = w(xs ) − w(ys ), Gt ≡ 0, and apayoff of zero after stopping. Note also that the optimal stopping time is invariant to affine
transformations of w and, consequently, to linear – but not affine – transformations of us .

2.3

Ordering discount functions

We endow discount functions with the following relation, which (as one could easily check)is a partial order.
Definition 1 (Patience Order) The discount function β exhibits greater patience thananother discount function α (denoted by α ≺ β) ifβ(s)is weakly increasing in s.
α(s)If, for instance, α(s) = exp(−rs)
and β(s) = exp(−r̄s), then α ≺ β whenever r̄ ≤ r. MoreRtgenerally, if α(t) = exp 0 −rα (s)ds for some discount rate function rα , with a similar
expression for β, then α ≺ β whenever rβ (t) ≤ rα (t) for all t. We shall refer to an agent
with discount function α as simply the α agent (or agent α) and we say that agent β is morepatient than agent α if α ≺ β. The next result motivates our definition of the patience order.
Proposition 1 5 Two agents with discount functions α and β are choosing between receiving π1 at time t1 and π2 at time t2 , where both payoffs and times can vary freely, subject to
π1 > 0, π2 > 0, and t2 > t1 . Then α ≺ β if and only if, whenever agent α chooses π2 thenagent β also chooses π2 .
Proof. If α ≺ β and α(t2 )π2 > α(t1 )π1 , then clearly β(t2 )π2 > β(t1 )π1 . Thus agent β ismore patient in the sense that whenever agent α is prepared to wait longer to receive π2then so is agent β. Conversely, suppose that α 6≺ β, so there is t2 > t1 such thatβ(t2 )α(t2 )<.
β(t1 )α(t1 )
5

We are very grateful to an anonymous referee for suggesting this characterization.

9

If we can choose π1 to be a number strictly between the two ratios and let π2 = 1, then theagent with discount function β will strictly prefer to receive π1 at t1 to π2 at t2 (becauseβ(t1 )π1 > β(t2 )) while the agent with discount function α will have the opposite preference
between those alternatives (because α(t2 ) > π1 α(t1 )).

Notice that this choice problem can also be understood as a problem of optimal stopping,where the flow payoffs are identically zero, and there are termination payoffs of π1 at t1 , π2 attime t2 , and nothing at other times. Therefore, the proposition tells us that, for τ (α) ≤ τ (β)
to hold generally, it is necessary that α ≺ β. The next result tells us that it is also sufficient.

2.4

Patience and Stopping

Theorem 1 Suppose that G is a nonnegative process in the stopping problem (2). Then
V (β) ≥ V (α) and τ (β) ≥ τ (α) a.s. whenever α ≺ β.
A key condition for the result is the agent’s ability to stop at any time in the interval [0, t̄].
In order to get some intuition for this, suppose on the contrary that the agent cannot stop,and is thus condemned to incur all cash flows till t̄. In this case, there is a perfect symmetry
in the agent’s value function corresponding to a cash flow stream and the opposite of thatstream. In particular, U (t̄; β) > U (t̄; α) for some stream π if and only if U (t̄; β) < U (t̄; α)
for the stream −π. Thus, if the agent is limited to either stopping immediately or incurring
the entire stream of cash flows, it is easy to build examples for which only the more patient(agent β) stops immediately.
What breaks this symmetry is the endogenous interruption of cash flows at any time in [0, t̄]:if a cash flow sequence is first positive and then negative, both a patient and impatient
agent will stop after getting the positive cash-flows, and the patient agent will experience ahigher discounted utility from it. By contrast, if the cash flow stream is first negative andthen positive, the present value may be negative for the impatient agent and positive for thepatient agent, but the reverse can never hold. A patient agent may therefore find it optimal
to endure the negative cash flows in order to enjoy the positive ones in the future when an
impatient decides to stop immediately, but the reverse situation can never occur.
Theorem 1 generalizes to a stochastic environment the results of Arrow and Levhari (1969),
Sen (1975), and Quah and Strulovici (2009). The result, like all results of this section, has astraightforward counterpart in discrete time. The proof of this theorem proceeds by showing
that, at agent α’s optimal stopping time τ (α) (but not at other stopping times),
U (τ (α); β) ≥ U (τ (α); α) = V (α).
10(5)

Since V (β) ≥ U (τ (α); β), we obtain V (β) ≥ V (α). Given this result, it is intuitive thatτ (β) ≥ τ (α): at a given time s > 0 and state, agent β’s value of the future payoff stream ishigher than agent α’s; thus if the latter does not stop neither will the former.
The nonnegativity of G is needed in Theorem 1 for the following reason: if the terminal value
gets increasingly negative (e.g., a chore that worsens over time), a more impatient agent may
postpone the chore until the last minute, while a more patient agent, more sensitive to theworsening of the chore, may decide to stop immediately. Recalling an example emphasized
by Sen (1975), a more patient agent may decide to stop polluting earlier.
Lastly, we consider three agents with discount functions α, β, and γ, with α ≺ β ≺ γ. The
next result, which is a corollary of Theorem 1 says that agent γ prefers the stopping rule ofagent β agent to that of agent α. In particular, if both α and β exhibit impatience, in thesense that they are decreasing in t, then a perfectly patient agent, i.e., an agent with discountfunction γ(t) ≡ 1 for all t ∈ [0, t̄], will prefer the stopping decision of the less impatient agent
β to that of agent α.
Proposition 2 6 Suppose that G is a nonnegative process in the stopping problem (2) andα ≺ β ≺ γ. Then U (τ (β), γ) ≥ U (τ (α), γ), where τ (α) and τ (β) are the optimal stopping
times of agents α and β respectively.

2.5

Application: Costly Search

Search models play a fundamental role in economics and have numerous applications to laborand housing markets, macroeconomics and monetary economics (see, for example, Rogerson
et al. (2005)). In these models, it is often assumed that that marginal cost of search isconstant and the item being sought has qualities that are drawn from a distribution that isfixed over time (either because it is exogenous, or because it is an equilibrium distributionthat is common knowledge). In practice, however, agents learn about this distribution as
their search progresses. For example, a worker learns about labor market conditions or about
the idiosyncratic marketability of his skills. In housing markets, a household’s search strategy
may evolve over time, by narrowing or directing its focus on specific housing locations or
categories. Similarly, search cost can evolve over time: it can increase as a result of scarcer
opportunities (e.g., a job candidate has to look farther for a job), or decrease as a result ofexperience or sunk costs (the household hears of more efficient or cheaper real estate agentsalong the search processes). Indeed, there are realistic scenarios where search costs andsearch outcomes are stochastic and correlated.
6

We would like to thank an anonymous referee for suggesting this result.

11

Standard search problems are instances of a pure stopping problem where the utility flowis the search cost cs ≥ 0 and the termination value Gτ is the value of the item eventuallychosen by the agent. Formally, the agent chooses a stochastic stopping time τ to maximize
 Z τ

E −
α(s)cs ds + α(τ )Gτ ;0

if there is no recall, Gt = Mt , where Mt ≥ 0 is the value of the item uncovered at time t and,if there is perfect recall, Gt = maxs≤t {Ms }. Theorem 1 tells us that even when the searchcost is time-varying and stochastic, and even when the value distribution of the item beingsought changes over time, the duration of search increases with patience.

2.6

Application: Experimentation

The most widely used model of endogenous learning is probably the experimentation model
based on the multi-armed bandit framework.7 In this section we consider how the lengthof experimentation in a multi-armed bandit model depends on the discount rate of theagent. The problem is formalized as follows. Let xt = (x1t , x2t , . . . , xnt ) denote the state at
instant t, where n is the number of arms. A strategy for the agent is a stochastic processι : [0, +∞) → {1, . . . , n} that is right continuous and has left limits. Denoting by I the setof all such processes, the value function of the agent is given by

Z ∞
−rt
ιt
e π(xt , ιt )dt ,V (x0 ) = sup Eι∈I

0where the instantaneous payoff π depends on the arm chosen and on the state of that arm.8
When applied to learning models, the state and payoffs have the following structure: xit
corresponds to the agent’s belief, at time t, about the underlying state, θi , of arm i. In
this case, we have π(xit , i) = E[Π(θi )|xit ], where Π(θi ) is the expected payoff of arm i if its
underlying state is θi , and xit evolves according to Bayesian updating. The underlying states{θi }i∈{1,...,n} are assumed to be independently distributed, so that the arms are uncorrelated:pulling one arm does not tell the agent anything about the underlying state of the otherarms.9 Lastly, we assume that arm 1 is a safe arm in that it has a known payoff distribution,
7

One of its earliest applications to economics dates back to Rothschild (1974), who uses a one-armed
bandit to model market learning and pricing by a monopolist. See Bergemann and Välimäki (2006) for a
recent review of the multi-armed bandit theory and its economic applications.
8This formalization excludes lump sum payoffs. At the expense of introducing more notation, the model
could easily be modified to include jumps in payoffs and beliefs, as in Keller et al. (2005).
9
In the case of learning with Brownian noise, dxιtt = κιt xιtt (1 − xιtt )dBtιt and dxit = 0 for i 6= ιt , where{Bti }i,t is a vector of standard Brownian motions and κ = (κ1 , . . . , κn ) is a vector of constants capturing thesignal-to-noise ratio of each arm (see Bolton and Harris (1999) for the derivation).

12and therefore entails no learning (x1t = x10 for all t ≥ 0); all other arms have unknown payoff
distributions and will be called risky.
It is easy to show that if the agent chooses the safe arm at some instant, then he will playthat arm forever after, because the problem is Markovian and the agent learns nothing newabout the arms, implying that the state becomes constant. Therefore, the experimentation
problem may be re-expressed as follows:

Z τ
Z ∞
−rt
1
−rt
ιt
e π(x0 , 1)dt ,V (x0 ) = sup Ee π(xt , ιt )dt +ι∈I−1 ,τ ∈Tτ0where I−1 is the subset of processes in I taking values in {2, . . . , n}, and T is the set of
all stopping times. Without loss of generality, the payoff of the safe arm can be normalized to zero, because the agent’s optimization problem is unchanged if all cash-flows aretranslated upwards or downwards by a constant amount. This leads to the following further
simplification of the problem:

Z τ
−rt
ιt
e π̄(xt , ιt )dt ,(6)V̄ (x0 ) = sup Eι∈I−1 ,τ0

where π̄(xi , i) = π(xi , i) − π(x10 , 1) for all xi and i ≥ 2. Stopping experimentation now meansthat the agent receives a payoff stream of zero forever after stopping.
Whittle (1980)10 shows the following key result: the optimal stopping time, τ , of problem(6) is equal to the sum of the stopping times {τi }ni=2 , where τi is the solution to (6) in thecase where only arm i and (the safe) arm 1 are available. This result is intuitive: how long
the agent tries a given arm should be independent of other risky arms, because the states ofindividual arms are uncorrelated. Therefore, when the agent retires, he must have tried eachrisky arm exactly as long as what he would have done if only that arm had been available(in addition to the safe arm). Now, each bandit problem with one risky and one safe arm isa pure stopping problem, and Theorem 1 says that τi is increasing in patience. This is truePof every i, and thus of τ = i τi . We have shown the following result.
Proposition 3 In a multi-armed bandit with uncorrelated risky arms and a safe arm, amore patient agent will, almost surely, experiment longer on each risky arm (and thus stop
experimentation later).
When there is a single risky arm whose distribution is governed by a binary (“High” or“Low”) unknown parameter , Cohen and Solan (2012) have characterized the optimal cutoff
10

See also Frostig and Weiss (1999, p. 13) for a concise exposition.

13

policy, from which it seems easy to derive comparative statics on the discount rate.11 Proposition 3 provides a general result where the agent can experiment with multiple risky arms
and the parameter governing each arm can take arbitrarily many values.
This application illustrates that a more patient can stop later even when he controls hislearning process (here, by choosing which arm to pull at each instant). How general is thisresult? In the next section we shall examine the impact of discounting and patience inmodels with both control and stopping.12

3

A General Model of Stochastic, Managed Cash Flows

We now extend the setting of the last section to combine both stopping and control decisions.
The optimization problem is now the following:Z τ
λα(s)u(x (s), λ(s), s)ds + α(τ )Gτ ,V (α) = sup U (τ, λ; α) = E
(7)
τ,λ

0

where {xλt }, {λt }, and τ are adapted to F and {xλt }t≥0 is controlled by λ. {xλt }t∈T takes values
in some topological space X and {λt }t∈T takes value in some set Λ.
hR {λt }t∈T is assumed to beit̄
λadmissible, i.e., such that x is uniquely defined and such that E 0 |α(s)u(x(s), λ(s), s)|ds
is finite.13 It is easy to see, given our earlier results, that the value of this combined control
and stopping problem remains monotonic in the discount rate.
Proposition 4 Suppose that G is a nonnegative process and that α and β are discountfunctions for problem (7) with α ≺ β. Then V (β) ≥ V (α).
Proof. Suppose (τ (α), λ(α)) is jointly optimal for agent α. Then τ (α) is also optimal for thisagent, conditional on λ(α). Therefore, by (5), we obtain U (τ (α), λ(α); β) ≥ U (τ (α), λ(α); α).
Since V (β) ≥ U (τ (α), λ(α); β), we have V (β) ≥ V (α).
It would be tempting to conclude that the optimal stopping time is also increasing withpatience. However, that is generally false, because optimal controls may differ under α and β,11Their result generalizes the characterization of optimal policies (for the one-agent case) in the Brownian
setting of Bolton and Harris (1999) and the Poisson setting of Keller, Rady, and Cripps (2005).
12
In the standard multi-armed bandit model, the agent chooses only one arm at each instant. If instead hecan choose the speed of experimentation (for example, by allocating resources between risky and safe arms),
the actual time spent experimenting is not necessarily increasing with patience. As illustrated in the more
abstract setting of the counterexample in Section 3.2, a more patient decision maker may experiment at afaster rate than a less patient one, resulting in the experimentation boundary being reached faster.
13
See Fleming and Soner (1993) for a general definition of admissible controls.

14

resulting in different paths for the state variables. An example of this phenomenon is provided
in Section 3.2. When the state has Markov dynamics, however, the next section provesin full generality that the continuation domain, i.e., the set of states for which stopping is
suboptimal, increases with patience. Section 3.2 will then provide an additional monotonicity
condition under which the optimal stopping time increases in patience.

3.1

Continuation Domain

Suppose now that {xt } is a Markov process: for any t0 and control {λs }s≥t0 , the distributionof {xs }s≥t0 depends on past history only through xt0 . Also suppose that the termination
value process is of the form Gt = G(xt , t) for some function G of state and time. In thiscase, it is well known that the value function depends, at any time t, only on the current
state xt and time t: Vt = V (t, xt ) for some function V .
For any discount function α, define the continuation domain at time t byC(α, t) = {x : V (α, t, x) > G(t, x)}, where

Z τ
α(s)
α(τ )
λ
u(x (s), λ(s), s)ds +Gτ .
V (α, t, x) = sup Eα(t)τ ≥t,λt α(t)C(α, t) is the set of states at which stopping at time t (and, thus, getting termination
value G(t, x)) is strictly suboptimal. In general, C(α, t) varies with time, but if x has timehomogeneous dynamics and G(x, t) only depends on x, C(α, t) is a constant subset of X . For
example, if x is a one-dimensional controlled diffusion, then C(α, t) is typically (though notnecessarily) an interval (a, b) such that it is optimal to stop exactly when xt hits a boundaryof the interval.14 The next result says that the continuation domain expands with patience.
Theorem 2 Suppose that, for problem (7), x is a Markov process and Gt = G(xt , t) is
nonnegative. Then C(α, t) ⊂ C(β, t) for all t, whenever α ≺ β.
Proof. Suppose that x ∈ C(α, t). This means that V (α, t, x) > G(t, x). Theorem 4, applied
at time t, implies that V (β, t, x) ≥ V (α, t, x), so x ∈ C(β, t).


3.2

Patience, Control and Stopping

Theorem 2 does not imply that a more patient agent stops later. Indeed, agents with different
patience levels apply different controls to the state, and a more patient agent may accelerate
14

In Sections 3.2 and 3.3, the interval is of the form (a, ∞).

15

the collision of the state with the boundary of the continuation domain, even though thatdomain always expands with patience. This is illustrated in the following example.
A counterexample
Consider a Markov setting with initial state x0 = 0 and with two control levels: Λ = {1, 2}.
The utility flow is equal to u(x, λ) = M for x ∈ [1, 10], u(x, λ) = −M for all x > 10 where
M is a large positive constant, and u(x, 1) = 1 and u(x, 2) = −0.01 for x ∈ [0, 1). Finally,
suppose thatdx
= λt .
dt
The state xt can only go up. Moreover, it is optimal to stop at x = 10, and not before, since
there is always a control yielding positive utility before that level. Thus, the continuation
domain is C = [0, 10] for all discount functions. Finally, it is optimal to spend as muchtime as possible in the region with payout rate M , i.e. set λ(x) = 1 for x ∈ [1, 10]. The
only question, therefore, is how fast to get to x = 1. A very impatient agent will never
use the control λ(x) = 2 for x sufficiently close to 0, because that control yields negative
instantaneous utility, while the reward (M ) is “far away,” arriving only when xt = 1. Bycontrast, a patient agent puts more value on future cash flows; from his perspective, thesmall negative cash flow 0.01 is only incurred for a short time, and leads to high cash flows
sooner. Thus it seems possible that for sufficiently high values of M , the more patient agent
will choose λ(x) = 2 for all x ∈ [0, 1) and therefore stop sooner than a less patient agent.
This intuition is proved formally in the Appendix.
Positive Results: Markovian State with Increasing Utility
To obtain a positive result relating stopping and patience, we consider the following control
problem which is a special case of (7):
Z τ

−rs
maximize U (τ, λ, ; α) = Ee u(x(s), λ(s), s)ds ,(8)0

subject tosubject to dxt = µ(xt , λt , t)dt + σ(xt , t)dBt

x0 = x,(9)where {xt } is one dimensional and B is the standard Brownian motion. We assume that σ isbounded below and away from zero and that the value function is well-defined and smooth,so that it obeys its Hamilton-Jacobi-Bellman equation.15 As stated, the problem assumes
15

For time-homogeneous dynamics, Strulovici and Szydlowski (2012) prove that the value function issmooth as long as the control space is compact and the drift, volatility, and return functions satisfy standard
Lipschitz and growth conditions.

16

constant discounting; this is to simplify the exposition and the case of general discountfunctions is treated in the Appendix. Notice also that the termination value is set at zero;
it is not hard to see that allowing it to take arbitrary positive values can lead to the morepatient agent stopping earlier, which is the opposite of the result we want.16 Last but notleast, we assume that the instantaneous utility function, u(x, λ, t) is increasing in x for all(λ, t). Since it is always possible to think of the state as −x rather than x, any results
obtained with this assumption will have an analog in the case where u(x, λ, t) is decreasing
in x for all (λ, t). In effect, we are excluding cases where u is not monotone in x or where
it increases in x for some values (λ, t) and decreases in x for others. This assumption issatisfied in many applications, though, obviously, not in the counterexample we gave.
The Markov setting of problem (8) means that the continuation value at any time depends
on the history only through the current state and so could be written as V (x, t, r) and the
optimal control λ can similarly be thought of as a function of just (x, t, r). Because we
assume u is increasing in x, so is V . Indeed, starting from some state y, with y > x, one canalways replicate the control applied when starting from x and get a higher payoff flow at all
times before and upon stopping. The continuation domain at time t consists of the states x
such that V (x, t, r) > 0; therefore, there exists a process a(t) such that the optimal stopping
time is given by τ = inf{xt ∈/ (at , ∞)}.
Given the initial condition, let X(r) ∈ C ([0, t̄], R) denote17 the path obtained when using
the optimal control at each time t. In other words, for each realization ω of the underlying Brownian motion, X(r) is the trajectory of the state over time. We say that X(r) is
decreasing in r if, for all r00 > r0 , we have P r ({ω : X(r00 )t ≤ X(r0 )t for all t ∈ [0, t̄]}) = 1.
Theorem 3 Let λ(x, t, r) and τ (r) be the control and stopping time that solve problem (8).18
Then µ(x, λ(x, t, r), t), X(r), and τ (r) are all decreasing in r, provided u(x, λ, t) is increasing
in x for all (λ, t).
Theorem 3 says i) the optimal drift, ii) the state path, and iii) the optimal stopping time are
all increasing in patience. There is an intuitive explanation for this result. The key property
is that when u is increasing in x, V is submodular in (x, r); this means that, starting from a
16

For example, if the termination value is very high at some time T̄ if the state x ≥ x̄ and zero otherwise,
and if it is costly to raise the value of x over time, a patient agent may incur the cost of raising x to x̄ in orderto get to the high termination value and stop at T̄ , while a very impatient agent, focused only on the highcost of raising x, may reject this option and stop later (or never), provided the utility flow is nonnegative.
17
C ([0, t̄], R) denotes the set of real-valued continuous functions defined on [0, t̄] (or [0, ∞), if t̄ = +∞).
18
In the case of multiple solutions we refer here to the solution leading to the smallest drift and the earliest
stopping time; see the Appendix for details.

17common state, the agent with the lower discount rate (i.e., the more patient agent), values
an increase in x more than the less patient agent. Consequently, whenever they are in thesame state, the more patient agent will (using his control) drive the state higher than theless patient. Thus the more patient agent has the higher drift and his state path (which isa continuous function of time) may touch but will never cross the corresponding path of theless patient agent. Recall that the stopping boundary takes the form of a lower bound on
the state, and by Theorem 2, that bound has to be lower for the more patient agent. Theseobservations guarantee that the state trajectory of the more patient agent hits his stopping
boundary later than the trajectory of a less patient agent.
Notice that while Theorem 3 says the optimal drift is decreasing in r, it does not say thatthe same holds for the optimal control. Of course, when the control λ is scalar and thedrift function µ is increasing in λ, then the optimal control must be decreasing in r sinceµ(x, λ, t) = m. The next result gives conditions under which a multi-dimensional optimal
control is decreasing in r.
Proposition 5 Suppose that Λ (the domain of control λ) is a lattice, µ is increasing in λ,and u and µ are supermodular in λ. Then λ(x, t, r) is decreasing in r.

3.3

Application: Bankruptcy

Recent models of corporate bankruptcy, starting with Leland (1994), take the view that thetiming of bankruptcy is endogenously chosen by equityholders. Until then, equityholders
firm must pay a coupon rate c(x) to debtholders, where c is decreasing in some performancemeasure x of the firm, and receive a payout rate δ(x), with δ increasing in x.19 Following
Leland (1994) and Manso et al. (2009), we assume that the performance measure {xt } is atime-homogeneous diffusion (for example, a geometric Brownian motion, or a mean-reverting
process). The shareholder problem is thus the pure stopping problem of solving
Z τ

−rt
V (x, r) = sup Exe (δ(xt ) − c(xt ))dt .
(10)
τ ∈T

0

Given the time-homogeneous Markov structure of the problem, and given that δ − c is anincreasing function of x, it is easy to show that optimal default takes the form of a hitting
19

For standard debt, c is a constant. However, in many contracts such as performance-pricing loans or
step-up bonds, c increases as some performance measure of the firm deteriorates. This measure maybe thecredit rating, or directly related to the earnings (EBITDA, price-earning ratio, etc.) of the issuing firm (seeManso, et al. (2009) for examples).

18

time τ (r) = inf{t : xt ≤ AB (r)}; AB (r) is called the default-triggering level of the firm, andis independent of the initial asset level x. If δ(x) = δx, c(x) = c, and x is the geometric
Brownian motion with drift µ and volatility σ, standard computations (see Leland (1994))
imply thatγ(r) µ c
AB (r) =1−,γ(r) + 1
r δ√
where γ(r) = (m + m2 + 2rσ 2 )/σ 2 and m = µ − σ 2 /2. Since AB increases in γ and r, andγ(r) is also increasing in r, AB must be increasing in r. More generally, Theorems 1 and 2
together give us the following result.
Proposition 6 For problem (11), the default-triggering level AB (r) is increasing in r while
the optimal default time τ (r) and the firm’s value V (x, r) are both decreasing in r.
Proposition 6 ensures that the monotonicity of optimal default time with respect to theinterest rate holds for general asset processes and coupon and payout profiles. This result
can be made even more general by allowing the firm to influence state xt through an action
λt that costs f (λt ) ≥ 0, with its effect determined by (9). The objective function is then

Z τ
−rt
V (x, r) = sup Exe (δ(xt ) − c(xt ) − f (λt ))dt .
(11)
τ ∈T

0

The instantaneous payoff δ(xt ) − c(xt ) − f (λt ) is increasing in the state xt , and so we canapply Theorem 3, which guarantees that all the conclusions of Proposition 6 hold even inthis more general setting (with both control and stopping).20

3.4

Application: Optimal Growth and Capital Deepening

One of the very first and most important applications of optimal control to economics is tothe optimal growth problem over an infinite horizon. Sensitivity of the optimal path withrespect to discounting has been studied, in a deterministic context, by Brock and Mirman(1972), Mendelsohn and Sobel (1981), Amir, Mirman, and Perkins (1991), Amir (1996),
and Quah and Strulovici (2009), amongst others. In our earlier paper (Quah and Strulovici(2009)), we showed in a deterministic, continuous-time model, that a lower discount rate
leads to capital deepening, i.e., a higher level of capital stock at all times. The result relies
on minimal assumptions; in particular, the representative agent’s utility function need not
20

Using the results of the present paper, Manso (2013, Proposition 9) obtains an equivalent of Proposition 6
for the equilibrium default boundary arising in a game between the borrower and credit-rating agencies.
Proposition 6 can also be used to study when households default on their mortgages.

19

be concave and there are no shape restrictions on the production function, which can exhibit
increasing returns to scale. We now extend this result to a stochastic context.21
An agent with initial capital k0 > 0 manages his capital through consumption, savings, andefforts so as to maximize his expected lifetime utility. Formally, the agent solves
Z ∞

−rs
max E
e u(cs , ks , s)ds
(12)
c

0

subject to dkt = H(kt , ct , t)dt + σ(kt , t)dBt ,where B is the standard Brownian motion, c is a finite dimensional control, and the drift
and volatility functions H and σ are such that k is nonnegative.22 The control c can capture
not only consumption, but also leisure and effort, technological choices, and other realistic
components. kt is the capital available to the agent at time t. Note that we have stated the
problem as one of pure control, without stopping (as is customary for growth models); to fit
this within our framework, we assume that u > 0, so that stopping is never optimal. In thesimplest case where utility depends only on consumption, we can write u(ks , cs , s) = ū(cs ks ),where ū : R+ → R+ is the felicity function, so cs ∈ [0, 1] is consumption expressed as
proportion of the capital stock and u is increasing in ks provided ū is increasing. Since thedrift H is completely arbitrary (subject to the usual integrability conditions), we are allowing
for arbitrary returns to scale in technology.
An application of Theorem 3 immediately yields the following result.
Proposition 7 Suppose that u(c, k, t) is increasing in k for all c, t. For any initial capitallevel k0 , the trajectory of capital {kt }t≥0 at the solution to (12) decreases in r almost surely.

4

Stochastic Discounting

Dynamic stochastic models in macroeconomics and finance typically involve stochastic discounting, where the stochastic discount is a “state price deflator,” representing some (random) intertemporal rate of substitution between (in discrete time) consecutive periods.23
Another, related, reason for considering stochastic discounting is to think of discounting
as the borrowing rate of a firm. If there is procyclicality between the interest rate and21

Riedel and Su (2010) study the stochastic irreversible investment decision of single firm and obtaincomparative statics of investment with respect to the cost of capital.
22
For example, kt = exp(Kt ) for some Itô process.
23
See, e.g., Duffie (2001).

20

the firm profit (or if on the contrary, its profits are countercyclical with respect to thegeneral borrowing rate, or lending rate), the discounting rate and cash flows are correlated.
Discounting may also be stochastic because the borrowing rate of the firm depends on its
credit rating. In this case, there is an obvious connection between cash flows and discounting.
Performance pricing loans and step up bonds are examples of debt contracts where theinterest rate depends on some performance measure of the borrower.
The strongest extension of our order on discount functions to a stochastic setting is torequire the ratio process β(s)/α(s) to be increasing in s path by path. However, we provide
an example to show that even this is not sufficient to guarantee that the more patient agent
stops later. To obtain positive results linking patience and stopping times, we need to
introduce certain restrictions on cash flows; we provide two results of this type. The first
theorem is based on some positive correlation condition. The second theorem assumes asingle crossing condition on the utility function.

4.1

Counterexample: Discount-Cash Flow Correlation

There are three periods {0, 1, 2} and two states of the world, “high” and “low,” which areequally likely. The cash-flows in Periods 0 and 1 are −1 and −2, respectively. The cash-flowin Period 2 is 21 if the state is high, and -4 if the state is low.
t
Agent A uses the discount function α(t) = 21 and strictly prefers to make the investment,yielding a present value equal to −1 + 12 (−2) + 41 12 (21) + 12 (−4) > 0.
Agent B faces the stochastic discount function β(0) = 1 and β(1) = 21 , and β(2) = 41 if thestate is high, and β(2) = 21 if the state is low. Agent B is more patient than agent A: theratio β/α is increasing over time for each realization of uncertainty, and strictly so if thestate is low. Plainly it makes no sense for anyone to invest in this project and stop after
periods 0 or 1. Assuming that the project is terminated after period 2, the present value toagent B is −1 + 12 (−2) + 14 21 (21) + 21 12 (−4) which is still less than zero. So agent B will notinvest in the project.
The key feature driving this counterexample is the negative correlation between B’s patienceand cash flows. It is possible to obtain a positive result once this phenomenon is excluded.

4.2

Positive Results with Stochastic Discounting

A stochastic discount function α is a strictly positive process adapted to the filtration Fthat is defined on [0, t̄], with α0 = 1 almost surely. An agent solves the following optimal

21

stopping problem (using the notation of Section 2):Z τα(s)us ds.
V (α) = sup E
τ ∈T

(13)

0

We would like to compare the optimal stopping behavior under α with that under anotherstochastic discount function β, also adapted to F. Since F represents the information thatagents receive, the fact that both are adapted to F prevents an agent from learning more
about future cash flows through his discount process than another agent with the otherdiscount process. Otherwise, a less patient agent could for example get more positive news
about future cash flows than a more patient one, which could clearly revert the result. For
simplicity, the termination value is set to zero.
Our first result gives a sufficient condition for comparative statics, under the assumptionthat α ≺ β almost surely, i.e.,


β(s)
Pr
ω:
is increasing on [0, t̄]
= 1;
α(s)
we shall denote this simply by α ≺ β. We know from the counterexample that this alone
is not sufficient to obtain a result, but this assumption along with the following positive
correlation condition will suffice:


β(s)
Corr
, α(s)us 1s<τ (α) Ft ≥ 0 a.s., for all 0 ≤ t < s < t̄,
(14)
α(s)
(where τ (α) refers to the optimal stopping time under α). The condition (14) means thatagent β is more likely to value future cash flows more relative to agent α when i) thesecash-flows are higher, ii) agent α values these cash flows more, and iii) agent α is less likely
to have stopped already.
Theorem 4 Suppose that α ≺ β and that the positive correlation condition (14) holds.
Then, for problem (13), τ (α) ≤ τ (β) and V (α) ≤ V (β).
Assumption 14 involves the optimal stopping time τ (α). The next result relies on assumptions directly expressed on primitives. Suppose that us = u(xs ) where x ∈ X ⊂ R is someone-dimensional state and suppose, also, that the discount factors are stochastic only throughthe state x, i.e., αs = α(xs , s). Such a specification, where payoffs and the discount factor
depend on some common underlying process arises, for instance, in affine term structure
models (see Duffie (2001, Chapter 7)). In such a context the following result holds.
Theorem 5 Suppose that the discount functions α and β obey
22

(a) α(x, ·) ≺ β(x, ·) for all x ∈ X and(b) β(x, s)/α(x, s) is increasing in x for all s ∈ [0, t̄].
Then for any problem (13) such that us = ū(xs ), where ū : X → R has the single crossing
property,24 we obtain τ (α) ≤ τ (β) and V (α) ≤ V (β).
Note that conditions (a) and (b) in Theorem 5 are together equivalent to requiring β(x, s)/α(x, s)
to be increasing in (x, s). Condition (a) simply says that agent β is more patient than α inevery state x. Condition (b) says that agent β’s valuation of future cash flows relative toagent α’s is greater in the higher states, which (because of the single crossing property onū) are also the states with a positive payoff.

4.3

Application: Investment in a Consumption-Based Asset Pricing ModelIn consumption-based asset-pricing models, cash flows are discounted according to a stochastic discount function that is proportional to the marginal utility of the representative agent.25
Our results enable us to connect the patience of the representative agent with his level ofrisk aversion.26
In formal terms, the present value of a project available at time 0 isZ τ
0−ρt u (xt )eV (u) = max E
πt dt ,τ ∈T
u0 (x0 )0where u(·) is the utility of the representative agent, ρ is his discount rate, xt > 0 is theaggregate consumption at time t, πt is the payoff generated by the project at time t, andτ is the termination time of the project, which is optimally chosen by its owner. Consider
a project Π that, while active, entails an operating cost of c per unit of time, with the
expected profit at time t depending on the state of the economy: π̄(xt ) = p(xt ) − c, where p
is increasing.27 It is clear that π̄ has the single crossing property; let x̄ denote the crossing
point, i.e., p(x̄) − c = 0. Finally, suppose that the project has an initial fixed cost I > 0, so
it will be initiated if and only if V (u) ≥ I; in this case we say that the project is u-valuable.
24

We mean that, whenever ū(x) ≥ (>) 0, then ū(x̃) ≥ (>) 0.
See, for continuous-time versions of this result, Breeden (1979) and Duffie and Zame (1989).
26
Nachman (1975) studies the impact of time and risk preferences in a setting with a single stochastic
lump sum.
27
It is assumed that the project is small compared to the size of the economy and, hence, has a negligible
impact on {xt }.
25

23

Without making any assumption on the aggregate consumption process xt , the next resultshows that Π is less likely to be undertaken if the representative agent is more risk averse
and, assuming the project is underway, higher risk aversion leads to an earlier stopping time.
Proposition 8 Suppose that u is more risk averse to v in the sense that u has the higher
coefficient of risk aversion, i.e., −u00 /u0 (x) ≥ −v 00 /v 0 (x) for all x > 0. Then project Π isv-valuable if it is u-valuable and the project will be abandoned later under v than under u.
Proof. In the case where the representative agent’s utility is u, the stochastic discountfactor is given by αt = α(xt , t), whereu0 (x).
α(x, t) = e−ρt 0
u (x0 )
Similarly, in the case where it is v, we have β(x, t) = e−ρt v 0 (x)/v 0 (x0 ). Thus
β(x, t)u0 (x0 ) v 0 (x)= 0.
α(x, t)v (x0 ) u0 (x)This ratio is independent of time and increasing in x. The latter holds because u is morerisk averse than v and so v 0 /u0 is increasing in x. Conditions (a) and (b) in Theorem 5 areboth satisfied and thus the claim follows from an application of that theorem.

Note that a straightforward extension of this proposition holds if the “u” agent, in addition
to being less risk averse than than the “v” agent, also has a lower discount rate.

5Time Inconsistency and Optimal Stopping

One of the best documented and widely researched behavioral departures from the “standard
model” is time inconsistency (see Frederick et al. (2002) for a survey). The agent’s time
preference is now characterized by a discount bifunction α(s, t) > 0, which is defined for
all t ≥ s, with α(s, s) = 1. At time s, the agent’s time-preference for future cash flows isgoverned by the discount function α(s, ·).28 An agent is time consistent if α(s, t0 )/α(s, t) is
independent of s for all s ≤ min{t, t0 }. If this fails to hold, the agent is time inconsistent; thisgives rise to a game between the selves of an agent at different times, who have misaligned
preferences over cash flows. A naive agent ignores the fact that he is time inconsistent,
while a sophisticated agent fully takes this problem into account. It is common in models oftime inconsistency to assume that the different selves have the same discount function, so
28

We use the term ‘bifunction’ to emphasize the dependence on two variables and to distinguish the more
complex discount function of a time-inconsistent agent from the discount function of a time consistent agent.

24α(s, s + k) = α(s0 , s0 + k) for all s, s0 , and k > 0, though (unless otherwise stated) we shallnot be making this assumption.
Definition 2 (Generalized Patience Order) Let β and α be two discount bifunctions.
We say that β exhibits greater patience than α (denoted by α ≺inc β) if, at each time s,β(s, t)is increasing in t, for all t ≥ s.
α(s, t)This definition simply says that one agent is more patient than another if his s-self is morepatient than the s-self of the other agent (in the sense given by Definition 1), for all s. Thissection first establishes that the results of Section 2 extend to the case of naive agents. It
then shows by example that when two agents are sophisticated, the more patient agent canstop earlier than the less patient one. However, when time inconsistency takes the form of
decreasing patience, a condition satisfied by hyperbolic and quasi-hyperbolic discounting, theequilibrium stopping time of a sophisticated agent is increasing with patience, which restoresthe positive result obtained in Section 2. We also show that a naive agent stops later thana sophisticated one with the same discount bifunction, and that the stopping time falls withincreasing sophistication. Our analysis will focus on pure stopping problems.

5.1

Naive agents

Consider the pure stopping problem introduced in Section 2. Given a discount function
γ : T → R++ , we denote by Vs (γ) the value function at time s of a time-consistent agent
with discount function γ. Now consider a (possibly) time-inconsistent and naive agent withdiscount bifunction α : (s, t) 7→ α(s, t) ∈ R for s ≤ t. This agent believes at time s that his
future selves all have the discount function α(s, ·). Therefore, he believes that stopping attime s is optimal if and only if Vs (α(s, ·)) = 0. We assume that the agent stops wheneverVs (α(s, ·)) = 0 and denote the stopping time arising from this rule by τ N (α).
It is easy to show that if A and B are naive agents with discount bifunctions α and β
respectively and that B is more patient than A, then τ N (β) ≥ τ N (α) with probability 1.
Indeed, for any s < τ N (α), we have Vs (α(s, ·)) > 0. That value function is exactly the sameas the one computed in Section 2. Since α(s, ·) ≺ β(s, ·), Theorem 1 applied to time s anddiscount functions β(s, ·) and α(s, ·) implies that Vs (β(s, ·)) ≥ Vs (α(s, ·)). Therefore, B finds
it strictly optimal to continue at time s whenever A does.

25

5.2

Sophisticated agents: A counterexample

With sophisticated agents, the multiple selves (one for each time) of an agent play an equilibrium where, at each time, exactly one of the selves makes a single move (a formal definitionis given in Section 5.3). Consider the following sequence of cash flows
1, −M,M M
M,, . . . , , 0, 0, . . .
n nnwhere M = 10, say, n is a large integer, and there are exactly n + 1 periods where the cash.
flow equals M
nAgent A is time–consistent with discount function α(t) = 21t . As is easily checked, A optimally takes the first cash flow and stops before getting −M .
Agent B has the same discount function as A at period 0 but suddenly becomes perfectly
patient after period 1. Formally, β(0, t) = 21t for all t ≥ 0 and β(s, t) = 1 for all s > 0 andt ≥ s. If B does not stop at time zero, his Period-1 self optimally incur the loss −M in orderto get future total benefits close to (n + 1)M/n > M . This continuation is strictly worse,
seen from the perspective of Period-0 B, than stopping at time zero. The more patient agent,
B, stops at Period 0, while the less patient, A, stops at Period 1.

5.3

Sophisticated agents with Decreasing Patience

We now assume that time is discrete, with final period t̄. Cash flows {u(t)}t≤t̄ follow any
arbitrary (possibly non Markovian) process with finite unconditional and conditional expectations. We focus on pure-strategy equilibria: at each time s, the s-self of the agent optimallydecides whether or not to interrupt the cash flows, if no previous self has already done so,given the strategy profile of his future selves. Formally, let T = {0, 1, . . . , t̄} denote the setof periods, and Ht denote the set of cash-flow histories from time 0 to time t excluded.29
The set of outcomes is then given by Ω = Ht̄+1 . A pure-strategy equilibrium is a binary
map e : {(t, h) : t ∈ T, h ∈ Ht } 7→ {C, S} that determines, for each t and h ∈ Ht , whether
the t-self of the agent continues (C) or stops (S), given that previous selves have continued.
Optimality requires that e(t, h) = C only if the value function conditional on continuing is
positive. (For simplicity, we rule out here any termination value, so that stopping yields
zero.) We also assume, to pin down the equilibrium, that any indifferent agent stops. With
these assumptions, the equilibrium is unique and can be computed by backward induction.
29

By convention, we assume that at time t, the agent must decide whether to continue before observing thetime t cash flow. The opposite convention, where current-period uncertainty is resolved before the agent’s
decision to continue, does not affect the analysis.

26

For s ≤ t, let α(s, t) denote the discount factor of the s-self at time t. Let S(α) denotethe set of pairs (t, h), with h ∈ Ht , at which the agent stops, in equilibrium. This set iscompletely determined by backward induction, and is independent of play history up to timet (excluded): whether the t-self decides to continue or stop depends only on the behaviorof his future selves, not one the decisions or time preferences of his past selves. If ω ∈ Ωdenotes the realized sequence of cash flows, let ω t ∈ Ht denotes the truncation of ω up to
time t. The equilibrium stopping time τ α of an agent with discount function α is given byτ α (ω) = min{t : (t, ω t ) ∈ S(α)}.
Let also τtα denote the stopping time of the agent restricted to τtα ≥ t, i.e., when all selves
before time t are forced to continue. From the previous observation, we have0τtα (ω) = min{t0 : (t0 , ω t ) ∈ S(α) and t0 ≥ t}.
The stopping times are stochastic because the cash-flow histories (ω t ) are stochastic.
For any s ≤ t and history h ∈ Ht , let V̄st (h) denote the continuation value of the s-self of theagent, from time t onwards, if all selves before time t are forced to continue, and cash-flow
history h has been realized. In particular,#"τ α −1tXα(s, t0 )u(t0 )|h ,V̄st (h) = E
t0 =t

where by convention the sum equals zero if the index set is empty, and where τ = t̄ + 1 if
the agent never stops.
Given our counterexample, a positive result on the stopping behavior of sophisticated agents
must involve additional restrictions. We shall impose the following condition.
Definition 3 (Decreasing Patience) The discount bifunction α exhibits decreasing
patience if, for any s < s0 , α(s0 , ·) ≺ α(s, ·) on the time interval T ∩ [s0 , t̄]:α(s, t)is increasing in t for t ≥ s0 .
α(s0 , t)
It is easy to see that the property hold for the two most popular ways of modeling timeinconsistency.30
30Prelec (2004) introduces a similar condition which he calls “decreasing impatience.” The difference interminology comes from our different viewpoints: the agent exhibits more impatience between two dates
as he gets closer to the earlier of the two dates, according to our patience ordering (that is what we calldecreasing patience). However, fixing the time at which the agent sits, he exhibits less impatience between
any two dates separated by a fixed duration, as these dates get further away in time. The conditions are
consistent whenever α(s, t) = α̃(t − s) for some function α̃ : T → R+ .

27

Example 1 (Hyperbolic Discounting) Decreasing patience holds for hyperbolic discounting, defined asα(s, t) = 1/(1 + k(t − s))for t > s, where k is some positive parameter.
Example 2 (Quasi-Hyperbolic Discounting) Decreasing patience holds for quasi-hyperbolic
discounting (also referred to as β-δ discounting), defined asα(s, t) = 1 for t = s andα(s, t) = β δ t−s for t > s,where δ ∈ (0, 1) and β ∈ (0, 1).
To clarify the notation, we let V̄st (h; α) denote the value function V̄st (h) for agent α.
Theorem 6 Suppose that α and β exhibit decreasing patience and β inc α. Then,
S(β) ⊂ S(α)(thus τ β ≥ τ α ) and, for all (s, h) with h ∈ Hs , V̄ss (h; β) ≥ V̄ss (h; α).

5.4Comparing naive, sophisticated, and partially sophisticatedagents

It is easy to see that between a sophisticated and a naive agent with the same discountbifunction α, the sophisticated agent always stops before the naive agent. This is because thenaive agent thinks that he is time consistent. Equivalently, he believes that his future selves
will do exactly what he wants them to do, in contrast to the sophisticated agent, who realizes
that he has to compromise with his future selves. Therefore, the naive agent’s perceived valuefunction must be weakly higher than the sophisticated agent’s, because it is optimal fromhis perspective. In particular, whenever the sophisticated agent has a positive continuation
value, so will the naive agent. So he only wants to stop if the sophisticated agent wants
to stop.31 The result below formalizes this observation; τ N (α) (τ S (α)) denote the optimal(equilibrium) stopping time of a naive (sophisticated) agent with discount function α.
31This argument is reminiscent of the control-sharing effects in Strulovici (2010): an individual incentives
to pursue experimentation are always higher if he is a dictator than if he shares control with other membersof society (see Theorems 1 and 8 in that paper). The relevant interpretation here is that the naive agent
erroneously believes that he is a dictator over his future selves.

28

Proposition 9 Given any discount bifunction α(s, t), τ S (α) ≤ τ N (α) almost surely.32
Using Theorem 6, we could extend this result to partial sophistication. Suppose that atime-inconsistent agent has a β-δ discount function. O’Donoghue and Rabin (2001) propose
a model of partial sophistication in which the agent believes, at any time s, that his future0
self at time s0 > s has the discount function α̂(s0 , t) = 1 for t = s0 and α̂(s0 , t) = β̂δ t−s fort > s0 and some β̂ ∈ [β, 1]. The case β̂ = 1 corresponds to a naive agent, while β̂ = β (thetrue value) corresponds to a fully sophisticated agent. Thus, sophistication varies inverselywith β̂. We denote by τβ̂ the stopping time of an agent whose sophistication coefficient is β̂.
Proposition 10 If two agents have the same quasi-hyperbolic discount function α but different levels of sophistication, the more sophisticated agent will stop earlier, i.e., if β̂ 0 > β̂ 00 ,then τβ̂ 0 ≥ τβ̂ 00 .
Proof. At any time s, the agent with sophistication coefficient β̂ ∈ [β, 1] perceives thevalue of the stopping process in the same way as a (fully) sophisticated agent with thediscount function γβ̂ , where γβ̂ (s, t) = α(s, t) for all t ≥ s, and γβ̂ (s0 , t) = 1 for t = s0 > s
0and γβ̂ (s0 , t) = β̂δ t−s for t > s0 . Note that γβ̂ exhibits decreasing patience in the domain{(s0 , t) : t ≥ s0 ≥ s} and that γβ̂ 0 inc γβ̂ 00 on the same domain. Therefore, by Theorem 6,
if V̄ss (h; γβ̂ 00 ) > 0 then V̄ss (h; γβ̂ 0 ) > 0. So the less sophisticated will continue if the moresophisticated agent decides to continue.


5.5

Application: Social Experimentation and Altruism

It is well known in the dynamic social choice literature that pure altruism (i.e., a generation
takes into account the fact that later generations care about even later generations) results intime inconsistency.33 For example, consider a dynastic model where each generation lives for
one period, and let xt denote the vector of goods consumed by the time-t generation. This
vector could include, among other items, the pollution level in period t, and hence embed
trade-offs between the production of perishable goods and the production of long-lasting
pollution. Suppose that each generation cares about the utility of future generations at a
decaying exponential rate: using Ut to denote the total utility of the time-t generation, and32

In both this result and in Proposition 10, we assume that the payoffs received over time do not varywith the sophistication of the agents. In particular, we assume that the termination values equal zero inboth cases. When stopping involves the agent moving on to some other activity, and if that activity is also
subject to time consistency issues, then we have a situation where the termination value of a project is itself
a dependent on the agent’s sophistication; in such a situation the result can be overturned.
33
See, e.g., Andreoni (1989) and Saez-Marti and Weibull (2005).

29

u(·) the immediate utility that it receives from immediate consumption of xt , we have
X 0
Ut = u(xt ) +γ t −t Ut0
t0 >tfor some discount factor γ. This formula, which holds for each t, implies a time-inconsistentβ-δ discount function in terms of the consumption stream, given by
Ut = u(xt ) +1X
0(2γ)t −t u(xt0 ).
2 t0 >t(See Saez-Marti and Weibull (2005) for the derivation.) This utility function is well-definedprovided γ < 1/2. Therefore, a decision maker who, at each period, represents the interestsof the current generation, will make decisions taking into account the interests of future
generations: the situation is formally equivalent to dynamic decision making by a timeinconsistent social planner.
Social discounting is of primary importance in evaluating the optimal level of experimentation
in social/public policy. An application of particular interest concerns policies towards global
warming (see, e.g., Karp (2005)). What is the equilibrium amount of social experimentation
when the social planner exhibits time inconsistency, and how does it vary with patience? As
we pointed out in Section 2.6, experimentation is a stochastic pure stopping problem when
it is modeled as a bandit problem with one risky arm and one save arm; thus Theorem 6
provides a general answer: if the social planner exhibits decreasing patience, such as β-δ
discounting, then increasing his overall patience level will result in an increase in social
experimentation.34

6

Conclusion

Whether they are financial, economic, or psychological, time preferences play a central rolein economic decisions. This paper provides a new methodology to systematically analyze theimpact of discount rates on agents’ values and decisions, yielding insights into the validityof common intuition on this issue. We provide examples with counter-intuitive outcomesarising in some plausible economic environments, as well as new conditions guaranteeing
34

Another widely studied source of time inconsistency, in dynamic political models, comes from the factthat elected representatives and pivotal voters typically change over time (see Persson and Svensson (1989)
and Tabellini and Alesina (1990)). This time inconsistency is particularly important in the context of social
experimentation, as noted by Strulovici (2010) and Callander and Hummel (2013). Theorem 6 shows thatas all decision makers become more patient, social experimentation lasts longer, provided that the sequenceof discount functions used by the successive decision makers in office exhibits decreasing patience.

30

positive results in other, at least as natural, environments. Therefore, the main lesson of thepaper is not simply that “lower discount rates imply higher valuation and later stopping.”
Instead, we provide a road map for economists that identify environments under which such
intuitive statements hold, as well as key mechanisms that could lead to its violation.
Our analysis has been mostly limited to a single agent, although the arguments used for time
inconsistent agents illustrate how our methodology can be employed to analyze multi-agent
settings. In particular, our techniques provide a way of analyzing supermodular games inwhich the timing decisions of all agents exhibit complementarities, with or without cash-flow
management. One promising application concerns the analysis of partnerships in stochastic
matching models. For example, McAdams’ (2011) description of how discounting affects thelength of partnerships is consistent with our results, while Fujiwara-Greve and Yasuda (2011)
obtain similar comparative statics results with respect to outside options in dynamic partnerships.35 More generally, our techniques could be applied to analyze models of multi-agent
search and experimentation; exploring those connections is a natural and promising area for
future research.

7

Appendix

Proof of Theorem 1
Zero Termination Value We first prove Theorem 1 when the termination value is zero. The
proof is based on several lemmas.
Lemma 1 Let τ be any optimal stopping time for the stopping problem (2). Then

Z τ


P ω ∈ Ω : t ≤ τ and E
α(s)us ds|Ft < 0 = 0
t

R τ

Proof. Let Aε = {ω ∈ Ω : t ≤ τ and E t α(s)us ds|Ft ≤ −ε} for ε ≥ 0. We need toshow that P (A0 ) = 0. It suffices to show that P (Aε ) = 0 for all ε > 0 since this implies thatP∞
P (A0 ) = P (∪∞n=1 A1/n ) ≤n=1 P (A1/n ) = 0. Suppose on the contrary that P (Aε ) > 0 for∗some ε > 0, and let τ ∗ = t1ω∈Aε + τ 1ω∈A
/ ε . Since Aε is Ft measurable, τ is a stopping time.
35

The authors also observe that the comparative statics of the present paper holds in their setting.

31

Denoting Ω \ Aε by Acε , we have
hR ∗
i
hR ∗
i
hR ∗
i
τ
τ
τ
E 0 α(s)us ds = P (Acε )E 0 α(s)us ds|Acε + P (Aε )E 0 α(s)us ds|Aε
hR ∗
i

R τ
τ
= P (Acε )E 0 α(s)us ds|Acε + P (Aε )E 0 α(s)us ds|Aε
R τ

R τ

≥ P (Acε )E 0 α(s)us ds|Acε + P (Aε )(E 0 α(s)us ds|Aε + ε)R τ
= E 0 α(s)us ds + P (Aε )ε,which contradicts the optimality of τ .

Given an optimal stopping time τ for problem (2), we define v : R+ → R byv(s) = E[us 1s<τ ];(15)v(s) is the expected payoff rate at time s, where the payoff is zero in the event that one
has stopped before s (i.e., s ≥ τ ). By the optimality of τ , Fubini’s theorem applied toR τV (α) = E 0 α(s)us ds implies thatZ t̄V (α) = U (τ (α); α) =α(s)v(s)ds.

(16)

0

The next result is a simple consequence of the fact that, at every point in time, the expected
payoff of an optimizing agent looking forward must be non-negative.
Lemma 2 For all t in [0, t̄),R t̄tα(s)v(s)ds ≥ 0.

Proof. By definition of v,
"Z
t̄

Z t̄
α(s)v(s)ds = Et#α(s)us 1s<τ ds

t

" "Z
##
t̄
= E E
α(s)us 1s<τ 1t<τ ds|Ftt Z τ
= E E
α(s)us ds|Ft 1t<τ .
tOptimality of τ and Lemma 1 imply that the inner expectation is almost surely nonnegativeR t̄
if t < τ . Therefore, the random variable E[ t α(s)us 1s<τ ds|Ft ]1t<τ is always nonnegative,
and so is its expectation.

The next lemma is proved in Quah and Strulovici (2009).

32

Lemma 3 Suppose that γ and h are integrable real-valued functions defined on some compact
R x00
interval [x0 , x00 ] of R, with γ increasing. If x h(s)ds ≥ 0 for all x in [x0 , x00 ], then
Z x00
Z x00
0
h(s)ds.
(17)γ(s)h(s)ds ≥ γ(x )x0x0We now conclude the proof of Theorem 1 for the case of a zero termination value. First,
note that V (β) ≥ V (α) since V (β) ≥ U (τ (α); β), V (α) = U (τ (α); α) and
Z t̄
Z t̄
Z t̄
β(s)
U (τ (α); β) =β(s)v(s)ds =α(s)v(s)ds = U (τ (α); α). (18)α(s)v(s)ds ≥00 α(s)
0
The inequality follows from Lemmas 2 and 3 and the last equality from (16).
It remains for us to establish that the smallest optimal stopping times are ranked, i.e.,τ (β) ≥ τ (α) almost surely. Suppose on the contrary that the set Ω0 = {ω : τ (β) < τ (α)}
has strictly positive probability. By mimicking at time τ (β) the argument that gave us (18),
we have on Ω0 (which is Fτ (β) -measurable)
"Z
#
"Z
#
τ (α)
τ (α)
β(τ (β))E
β(s)us ds Fτ (β) ≥Eα(s)us ds Fτ (β)
a.s.
α(τ (β))
τ (β)
τ (β)
This gives a lower bound for the continuation value with discount β at time τ (β), wheneverτ (β) < τ (α). Since τ (α) is the essential infimum of the optimal stopping times for theR τ (α)discount function α, E[ τ (β) α(s)us |Fτ (β) ] > 0 almost surely on Ω0 . Otherwise the agent
would be weakly better off stopping at τ (β) when the event Ω0 occurs, on a set of positive
probability. Since the ratio βt /αt is also strictly positive for all t, this implies that
#
"Z
τ (α)
(19)
E
β(s)us ds 1Ω0 > 0.
τ (β)

Therefore, the stopping time max{τ (α), τ (β)} strictly dominates τ (β) for the discount function β, contradicting optimality of τ (β).
General Case Here we consider the case of an arbitrary nonnegative termination value
process Gt . By absolute continuity of α(t)Gt , the objective can be reexpressed as
Z τ


dαs
max E
α(s) us ds + dGs + Gs,ταs0where the last two integrals are Lebesgue-Stieltjes integral.36 This objective is similar to thecase of a zero termination value, except that the new utility flow now directly depends on α36See, e.g., Royden (2007) for a definition. The integrals are well defined whenever the integrating process
has bounded variation and the integrand is measurable and bounded.

33

sthrough the third term Gs dα. Optimality of τ implies that
αs

Z τ
E
t



dαs
α(s) us ds + dGs + Gs
≥0αs

for all t < τ . We use the following generalization of Lemma 3:
Lemma 4 Let H be a stochastic process of bounded variation such that E[Ht ] ≤ E[Ht̄ ] forall t ∈ [0, t̄).37 Then, for any positive, increasing, deterministic process γ, we have"Z#t̄γs dHs ≥ γ(0)E[H(t̄) − H(0)].

E
0The proof is by integration by parts: we have
Z t̄

Z t̄
γs dHs = γ(0)(H(t̄) − H(0)) +0(Ht̄ − Hs )dγs .
0Taking expectations proves the lemma. Applying the lemma to γ = β/α and to the process
H defined (up to a constant) bydαs
1s<τ ,
dHs = α(s) us ds + dGs + Gs
αs
we obtain
"Z
"Z

#

#
t̄
t̄
dαs
dαs
E
≥E
.
β(s) us ds + dGs + Gs
α(s) us ds + dGs + Gs
αs
αs
0
0

(20)

s
s
Since β/α is increasing, dβ≥ dα
for all s (the inequality holds even when β and α areβsαs
discontinuous). This, combined with (20) and the fact that G is nonnegative, implies that
Z τ


Z τ


dβs
dαs
E
β(s) us ds + dGs + Gs≥Eα(s) us ds + dGs + Gs
≥ 0.
βs
αs
0
0

Therefore, V (β) ≥ V (α). The rest of the proof is similar to the case of a zero termination
value: on the set Ω0 , we can show that continuing until at least τ (α) is strictly beneficial, using
an inequality similar to (20) for continuation values evaluated at time τ (β) and contradict
optimality of τ (β).
Proof of Proposition 2
37

If t̄ is infinite, Ht̄ is the limit of Ht as t → ∞, and is assumed to exist a.s.

34

Suppose that α ≺ β. Theorem 1 implies that τ (α) ≤ τ (β) with probability 1. This means
that, at time τ (α), continuing until τ (β) is optimal for agent β and in particular yields
positive value. Since β ≺ γ, by an argument analogous to that leading to (18), we know thatthe value of continuing until τ (β) is positive for the γ agent, and thus preferred to stoppingat τ (α).
Counterexample of Section 3.2
Suppose that the agent discounts future cash flows at rate r. The Bellman equation forx ∈ [0, 1] is given by
0 = max{−rV (x) + 1 + V 0 (x); −rV (x) − 0.01 + 2V 0 (x)},where the first term corresponds to the control λ = 1 and the second term to λ = 2. Thus
it is optimal to choose λ = 2 if and only if
V 0 (x) ≥ 1.01.

(21)

We will check that for r small enough, the solution to the HJB equation is maximized by
the second term, which corresponds to control λ = 2. The general solution to−rV (x) − 0.01 + 2V 0 (x) = 0
is rx 
−0.01+ c exp.
(22)
r2
The boundary condition is V (1) = M [1 − exp (−rT )]/r, where T is the time it takes for x
to go from 1 to 10 under control λ = 1 (so T = 9). This implies that
V (x) =c=
e−r/2 M (1 − e−rT + 0.01 ,r(23)

and hence that
 exp (−r/2) 
exp (r(x − 1)/2) M 1 − e−rT + 0.01 ≥M 1 − e−rT + 0.01 ,22(24)
which is uniformly, arbitrarily large for M large and r fixed. In particular, at a given discountrate r̃, there is M̃ such that (21) is satisfied. This shows that the function V defined by (22),with c defined in (23), and the control λ = 2 solve the HJB equation, and hence setting
λ(x) = 2 for all x ∈ [0, 1) is optimal for this agent.38
V 0 (x) =38

From straightforward inspection of the righthand side of (24), one can show the following, sharper result:for all M > 1, there exists r̄ such that for all r0 < r̄, it is optimal to set λ = 2 for all x ≤ 1.

35

On the other hand, for a fixed x̂ ∈ (0, 1) and with M = M̃ , it is clear from (24) that (21)will be violated if r is sufficiently high above r̃. Such an agent’s optimal control will involve
choosing λ(x) = 1 for at least some x ∈ [0, 1); consequently, she will stop later than theagent with discount rate r.

Proof of Theorem 3
Recall that V (x, r) is the continuation value at time 0 when the state is x and the discountrate is r. Our first objective is to prove the following result.
Lemma 5 V (x, r) is submodular in (x, r).
Proof. Given the discount rate r, we denote the optimal control at state x̂ and time t byλ(x̂, t) and let τ (x) denote the optimal stopping time with initial state x. We claim that if
y > x, then
"Z
#
"Z
#
τ (y)
τ (x)
E
se−rs u(ys , λ(ys , s), s)ds ≥ E
se−rs u(xs , λ(xs , s), s)ds
(25)
0

0

To see this, definehs = E u(ys , λ(ys , s), s)1τ (y)≥s − u(xs , λ(xs , s), s)1τ (x)≥s ,(26)

where the expectation is taken with respect to the Wiener measure (the probability measureR∞
generated by the paths of Brownian Motion). Notice that for all t, t e−rs hs ds ≥ 0. Thisis because (proceeding as in the proof of Lemma 2) we can writeZ ∞e−rs hs ds = E V (yt , t, r)1t≤τ (y) − V (xt , t, r)1t≤τ (x) ;tsince V (yt , t, r) ≥ V (xt , t, r) ≥ 0 and 1t≤τ (y) ≥ 1t≤τ (x) (the latter because it takes moretime to hit the lower boundary when starting from a higher level), the difference inside
the expectation is nonnegative almost surely and, therefore, so is the expectation. Using
Lemma 3, this time with the function γ(s) = s, we conclude thatZ ∞se−rs hs ds ≥ 0.
0This inequality can in turn be rewritten as (25).
By a generalized envelope theorem (see Milgrom and Segal, 2002),39
Z τ

∂
−rt
E
e u(xt , λt , t)dt ,Vr (x, r) =∂r
0
39

The parameter r does not affect the distribution of the underlying process, which justifies the applicationof the theorem.

36

evaluated at the optimal controls λ and τ . Computing the derivative explicitly,

Z τ
−rt
(−t)e u(xt , λt , t)dt .
Vr (x, r) = E
0This implies that for y > x,"Z
Vr (y, r) − Vr (x, r) = −E

#

τ (y)

se

−rs

"Z

u(ys , λ(ys , s))ds + E

0

#

τ (x)
−rs

se

u(xs , λ(xs , s))ds ,0which is less than zero from (25).



We can now conclude the proof of Theorem 3.
Proof.
The smoothness of the value function V implies that it satisfies the following
Hamilton-Jacobi-Bellman (HJB) equation.

1 2
0 = sup u(x, λ, t) + µ(x, λ, t)Vx (x, t, r) + Vt (x, t, r) + σ (x)Vxx (x, t, r) − rV (x, t, r)2
λ(27)
for all x > a(t). We first make a change of variable, replacing the initial control variable bya direct determination of the drift. Let υ(x, m, t) = sup{u(x, λ, t) : µ(x, λ, t) = m}, withυ(x, m, t) = −∞ if the defining set is empty. Thus, υ(x, m, t) is the maximal utility one
can achieve at (x, t) while providing a drift m to the state. We assume that this maximum
is achieved whenever υ is finite, which is easy to guarantee by simple assumptions on theprimitives.40 The optimal control problem can be re-expressed in terms of m:
1 2
0 = sup υ(x, m, t) + mVx (x, t, r) + Vt (x, t, r) + σ (x)Vxx (x, t, r) − rV (x, t, r) . (28)2m
Any drift-control process {mt } that achieves at all times the supremum in (28) is an optimal
control. Reciprocally, a control process that fails to achieve the supremum in (28) on a set of
positive measure is suboptimal.41 We assume that the set of maximizers of (28) is nonempty
and compact for all (x, t).42 Let m(x, t, r) be the smallest maximizer of (28) for state x, timet, and discount r. By Lemma 5, V (x, t, r) is submodular in (x, r), so the objective functionin (28) is submodular in (m, r). By standard monotone comparative statics arguments (seeMilgrom and Shannon (1994)), we know that m(x, t, r) is decreasing in r.
40

For example, if µ and u are continuous in λ and λ lies in a compact set, then the set {λ : µ(x, λ, t) = m}is closed and therefore compact, and λ 7→ u(x, λ, t) achieves its maximum on that set.
41
Precisely, one may show that any HJB-maximizing control strictly dominates any control that does not
solve (28) for a time set of positive measure.
42
This is guaranteed if υ is continuous in m and m lies in a compact set for each (x, t). This, in turn, isguaranteed, by the maximum theorem, if u and µ are continuous in λ and λ lies in a compact set.

37

We now claim that X(r), the path obtained when using the control m, is decreasing in
r. Given an initial state x, let x0t be some realized optimal trajectory for rate r0 (andxt the corresponding optimal path for r). For r < r0 and for all t > 0, we know thatm(x0t , t, r) ≥ m(x0t , t, r0 ). Given this, the paths of xt and x0t cannot cross because, given thatthey are continuous, they must first touch, and the moment they touch (i.e. have the samevalue of the state variable at the same time), the path of the lower discount rate receives
a higher drift.43 Thus X(r) is decreasing in r. Lastly, we know from Theorem 2 that theboundary a(r) is increasing in r. The previous observations imply that xt (r) hits a(r) later
than xt (r0 ) hits a(r0 ), path by path, so τ (r) ≥ τ (r0 ).
Proof of Proposition 5
We shall prove a bit more than what is stated in the proposition. We shall identify an
optimal control function λ(x, t, r) that is decreasing in r and also (in order to properly linkup this result with Theorem 3) we shall also show that µ(x, λ(x, t, r), t) = m(x, t, r), wherem is defined in the proof of Theorem 3.
It is straightforward to check that the objective function in (27) is supermodular in λ and has
decreasing differences in (λ, r), given our assumptions and using the submodularity of V in(x, r) (Lemma 5). Therefore, the set of maximizers of (27) form a lattice and are decreasingin the strong set order (see Topkis (1978) or Milgrom and Shannon (1994)). Let λ(x, t, r) be
the smallest element in the set of maximizers; λ(x, t, r) is then decreasing in r.
To see that µ(x, λ(x, t, r), t) = m(x, t, r), note that, given (x, t), there must be λ̃ such thatµ(x, λ̃, t) = m(x, t, r) and λ̃ must also maximize the objective function in (27). Therefore,λ̃ ≥ λ(x, t, r), by the definition of λ(x, t, r). Since µ is increasing, m(x, t, r) = µ(x, λ̃, t) ≥µ(x, λ(x, t, r), t). On the other hand, because λ(x, t, r) maximizes the objective in (27),µ(x, λ(x, t, r), t) must maximize the objective in (28), so µ(x, λ(x, t, r), t) ≥ m(x, t, r) by thedefinition of m.

Extension of Section 3.2 to General Discount Functions
 RtSuppose that α and β have absolutely continuous logarithms, so α(t) = α(0) exp − 0 rα (s)ds ,with a similar expression for β. In this case, α ≺ β is equivalent to rβ (t) ≤ rα (t) for allt ∈ T . Let
ν

β(t)
,
γν (t) = α(t)α(t)
43Continuity plays an important role in this argument. In discrete time, or if jumps were allowed, thepaths could cross each other.

38

for ν ∈ [0, 1]. It is easy to check that γ0 = α, γ1 = β, and γν ≺ γν 0 for all ν ≤ ν 0 .
Lemma 5 can then modified as follows: let V (x, ν) denote the value function of an agent
with discount function γν , starting from state x; then, V (x, ν) is submodular. Indeed, theenvelope theorem implies thatZ τ ν β(t)∂V (x, ν)− log=Eγν (t)u(xt , λν (t))dt .
∂ν
α(t)
0
To show that Vν (y, ν) − Vν (x, ν) ≤ 0 for y ≥ x, we use the same argument as in Lemmas 5,R∞with the difference that, instead of 0 se−rs hs ds ≥ 0, we show that


Z ∞
β(s)
log
γν (s)hs ds ≥ 0.
α(s)0The result is obtained by an application of Lemma 3 to the increasing function s 7→log (β(s)/α(s)).
The HJB equations are modified only by replacing r with the time varying rate rν (t) =rα (t) + ν(rβ (t) − rα (t)). The claims in Theorem 3 and Proposition 5 can then be extended
to the case of general discount functions, with the proof proceeding in the same way as theproof we have already given for the case of constant discount rate.
Proof of Theorem 4
We first show that if it is strictly optimal for A to continue beyond t = 0, then it is alsoR τ (α)strictly optimal for B to do so. Let Vt = E[ tα(s)us ds|Ft ] denote the value function of
agent α at time t < τ (α). Optimality of τ (α) implies that E[Vt 1t<τ (α) ] ≥ 0 and, hence (seeLemma 2)Z t̄E[α(s)us 1s<τ (α) ]ds ≥ 0tfor all t ≤ t̄. Since Agent B can always choose the stopping time τ (α), we have (using, again,
the same manipulation as in the proof of Theorem 1),
Z t̄



Z t̄ 
β(s)β(s)
V (β) ≥
Eα(s)us 1s<τ (α) ds ≥E× E[α(s)us 1s<τ (α) ]ds,
α(s)
α(s)
0
0


where the second inequality comes from condition (14) evaluated at t = 0. The function E[β(s)/α(s)] is increasing in s, since β(s)/α(s) is increasing path by path. Applying
Lemma 3, we conclude that V (β) ≥ V (α). In particular, if V (α) is strictly positive, thenso is V (β); thus the β agent will not stop at t = 0 if agent α does not stop at t = 0. For
any time t̃ > 0 and realization such that τ (α) > t̃, one shows similarly that τ (β) > t̃, using
39

again (14), this time evaluated at t = t̃, and the fact that E[β(s)/α(s)|Ft̃ ] is increasing in s.
We conclude that τ (β) ≥ τ (α) almost surely.

Proof of Theorem 5
The proof is identical to that of Theorem 4, except for the following. Single-crossing of theutility function u and monotonicity of the ratio β/α with respect to x (condition (b)) implythat
β(x̄, s)β(xs , s)α(xs , s)ū(xs )1s<τ ≥× E[α(xs , s)ū(xs )1s<τ ],Eα(xs , s)α(x̄, s)where x̄ is a crossing point of u. Condition (a) then implies that we can apply Lemma 3 asin the proof of Theorem 4, and reach the same conclusion.
Proof of Theorem 6
We start with the following lemma.
Lemma 6 Suppose that α exhibits decreasing patience. Then for any s ≤ t and historyh ∈ Ht , we have V̄st (h) ≥ 0.
Proof. We show this by backward induction on s that for any t ≥ s and h ∈ Ht . For s = t̄(and thus, t = s), the final self can always avoid the last cash flow and get 0, which proves
the condition, since τt̄α is chosen by the last self. Suppose now that the induction hypothesis
holds for all s0 > s. We will show that it holds for s. For t = s, the inequality holds because
the s self can always interrupt cash flows immediately and get 0, and would indeed do so
if his present value, conditional on continuing, were negative. Now suppose that t > s andh ∈ Ht . We havet̄XtV̄s (h) =α(s, t0 )v(t0 ),t0 =twhere v(t0 ) = E u(t0 )1t0 <τtα h . We now show that for all t̃ ≥ t,t̄Xα(t, t0 )v(t0 ) ≥ 0.

(29)

t0 =t̃

We can write this expression as"
E 1t̃≤τtα

τtα −1X α τt̃ −1Xα(t, t0 )u(t0 ) h = E  1t̃≤τtα E α(t, t0 )u(t0 ) h̃ h ,#

t0 =t̃

t0 =t̃

40

where h̃ is the history at time t̃, and we use the fact that τtα = τt̃α whenever t̃ ≤ τtα (seeobservation above). The inner expectation is equal to V̄tt̃ (h̃) and is nonnegative, by the
induction hypothesis. This shows (29).
Applying the discrete version of Lemma 3 (see Quah and Strulovici (2009)), and using thatα(s, ·)  α(t, ·), we conclude that
t̄
X

0

0

α(s, t )v(t ) ≥ α(s, t)

t0 =t

t̄
X

α(t, t0 )v(t0 ) ≥ 0,

t0 =t

which proves the induction step and the lemma.



We can now prove the theorem.
Proof. Let S s (α) denote the set of pairs (t, h) in S(α) such that t ≥ s, i.e., the pairs oftime-histories, after time s, at which the agent stops. We show by backward induction on sthat S s (β) ⊂ S s (α). The inclusion holds as an equality for s = t̄, since both agents continue
at t̄ if and only if the expected utility of that period is positive, and get that expected utility,
multiplied by their discount factor. Suppose that the inclusion holds for all s0 > s. We willshow that it also holds for s. Consider any h ∈ Hs . If A (the α agent) stops at (s, h), thenthere is nothing to show. Suppose that A continues at (s, h). We need to show that B (theα, andβ agent) also continues. Since A continues, we have, conditional on h, τsα = τs+1#
" τ α −1sXEα(s, t)u(t) h > 0.
t=s

If B continues, he gets βτs+1 −1XE
β(s, t)u(t) h .

(30)

t=s
β
α
By the induction hypothesis, τs+1
≥ τs+1= τsα . Therefore, (30) may be reexpressed as" τ α −1#sXαEβ(s, t)u(t) + V̄sτs (h(τsα ); β) h ,t=s

where h(τsα ) is the (random) realized history up until time τsα . We show that the firstterm is strictly positive and that the second term is nonnegative. The first term equalsPt̄0
t=s β(s, t)v(t), where v(t) = E[u(t)1t<τsα |h]. We now show that for all s ≥ s,t̄Xα(s, t)v(t) ≥ 0.

t=s0

41

(31)

Proceeding as in Lemma 6, that expression is equal to
 

 α
τs0 −1XE  1s0 ≤τsα E α(s, t)u(t) h0  h ,t=s00where h0 is the history at time s0 . The inner expectation is equal to V̄ss (h0 ; α), and is nonnegative by Lemma 6. This shows (31). An application of the discrete version of Lemma 3,
along with the fact that β(s, ·)  α(s, ·), then shows thatt̄Xβ(s, t)v(t) ≥

t=s

t̄
X

α(s, t)v(t)

(32)

t=s

and, therefore, that the first term is strictly positive. The second term is nonnegative,
because V̄st (h; β) is always nonnegative, from Lemma 6.44 Therefore, it is strictly optimalfor B to continue at (s, h), which shows the induction step. Finally, (32) implies that
Vss (h; β) ≥ Vss (h, α), which shows the second part of the theorem.

References
Akerlof, G. (1991) “Procrastination and Obedience,” American Economic Review,Vol. 81, No. 2, pp. 1–19.
Amir, R., Mirman, L., and Perkins, W. (1991) “One-Sector Nonclassical Optimal
Growth: Optimality Conditions and Comparative Dynamics,” International Economic Review, Vol. 32, pp. 625–644.
Amir, R. (1996) “Sensitivity Analysis of Multisector Optimal Economic Dynamics,” Journalof Mathematical Economics, Vol. 25, pp. 123–141.
Andreoni, J. (1989) “Giving with Impure Altruism: Applications to Charity and Ricardian
Equivalence,” Journal of Political Economy, Vol. 97, pp. 1447–1458.
Arrow, K., Levhari, D. (1969) “Uniqueness of the Internal Rate of Return with Variable
Life of Investment,” Economic Journal, Vol. 79, pp. 560-566.
Bergemann, D., Välimäki, J. (2006) “Bandit Problems,” Discussion Paper No. 1551,
Cowles Foundation, Yale University.
44

Recall that nonnegativity of V̄st was obtained by backward induction, and holds for all histories leading
up to time t.

42

Bolton, P., Harris, C. (1999) “Strategic Experimentation,” Econometrica, Vol. 67, No. 2,
pp. 349–374.
Breeden, D. (1979) “An Intertemporal Asset Pricing Model with Stochastic Consumption
and Investment Opportunities,” Journal of Financial Economics, Vol. 7, pp. 265–296.
Brock, W., Mirman, W. (1972) “Optimal Economic Growth and Uncertainty: The Discounted Case,” Journal of Economic Theory, Vol. 4, pp. 479–513.
Callander, S., Hummel, P. (2013) “Preemptive Experimentation under Alternating
Political Power,” Working Paper, Stanford University.
Cochrane, J. (2011) “Presidential Address: Discount Rates,” Journal of Finance, Vol. 46,
No. 4, pp. 1047–1108.
Cohen, A., Solan, E. (2012) “Bandit Problems with Lévy Payoff Processes,” forthcomingin Mathematics of Operations Research.
Duffie, D (2001) Dynamic Asset Pricing Theory, Third Edition. Princeton University
Press.
Duffie, D., Zame, W. (1989) “The Consumption-Based Capital Asset Pricing Model,”Vol. 57, No. 6, pp. 1279–1297.
Durrett, R. (2004) Probability: Theory and Examples, Third Edition. Duxbury Press.
Fleming, W. H., Soner, H.M. (1993) Controlled Markov Processes and Viscosity Solutions, Springer-Verlag, New York.
Frederick, S., Loewenstein, G., and O’Donoghue, T. (2002) “Time Discounting
and Time Preferences: A Critical Review,” Journal of Economic Literature, Vol. 40, pp. 351–
401.
Frostig, E., Weiss, G (1999) “Four proofs of Gittins multiarmed bandit theorem, Applied
Probability Trust.
Fujiwara-Greve, T., Yasuda, Y. (2011) “Repeated Cooperation with Outside Options,”
Working Paper, Keio University.
Karp, L. (2005) “Global Warming and Hyperbolic Discounting,” Journal of Public Economics, Vol. 89, pp. 261–282.

43

Keller, G., Rady, S., and Cripps, M. (2005) “Strategic Experimentation with Exponential Bandits,” Econometrica, Vol. 73, No. 1, pp. 39–68.
Lamberton, D., Lapeyre, B. (2007) Introduction to Stochastic Calculus Applied to Finance, Second Edition. Chapman and Hall.
Leland, H. (1994) “Corporate Debt Value, Bond Covenants, and Optimal Capital Structure,” Journal of Finance, Vol. 49, pp. 1213-1252.
Manso, G. (2013) “Feedback Effects of Credit Ratings,” forthcoming in the Journal of
Financial Economics.
Manso, G., Strulovici, B., and Tchistyi, A. (2010) “Performance-Sensitive Debt,”
Review of Financial Studies, Vol. 23, No. 5, pp. 1819–1854.
McAdams, D. (2011) “Performance and Turnover in a Stochastic Partnership,” American
Economic Journal: Economics, Vol. 3, No. 4, pp. 107–142.
Mendelsohn, R., Sobel, M. (1981) “Capital Accumulation and the Optimization of
Renewable Resource Models,” Journal of Economic Theory, Vol. 23, pp. 243–260.
Milgrom, P., Segal, I. (2002) “Envelope Theorems for Arbitrary Choice Sets,” Econometrica, Vol. 70, No. 2, pp. 583–601.
Milgrom, P., Shannon, C. (1994) “Monotone Comparative Statics,” Econometrica,
Vol. 62, No. 1, pp. 157–180.
Nachman, D. (1975) “Risk Aversion, Impatience, and Optimal Timing Decisions,” Journalof Economic Theory, Vol. 11, pp. 1960–246.
O’Donoghue, T., Rabin, M. (1999) “Doing It Now or Later,” American Economic Review, Vol. 89, No. 1, pp. 103–124.
O’Donoghue, T., Rabin, M. (2001) “Choice and Procrastination,” Quarterly Journal of
Economics, Vol. 116, No. 1, pp. 121–160.
Persson, T., Svensson, L. (1989) “Why a Stubborn Conservative would Run a Deficit:
Policy with Time-Inconsistent Preferences,” Quarterly Journal of Economics, Vol. 104,
pp. 325–345.
Prelec, D. (2004) “Decreasing Impatience: A Criterion for Non-Stationary Time Preference and “Hyperbolic” Discounting,” Scandinavian Journal of Economics, Vol. 106, No. 3,
pp. 511–532.
44

Quah, J., Strulovici, B. (2009) “Comparative Statics, Informativeness, and the Interval
Dominance Order,” Econometrica, Vol. 77. No. 6, pp. 1949–1992.
Riedel, F., Su, X. (2010) “On Irreversible Investment,” forthcoming, Finance and Stochastics.
Rogerson, R., Shimer, R., and Wright, R. (2005) “Search-Theoretic Models of theLabor Market: A Survey,” Journal of Economic Literature, Vol. 18, pp. 959–988.
Rothschild, M. (1974) “A Two-Armed Bandit Theory of Market Pricing” Journal of
Economic Theory, Vol. 9, pp. 185–202.
Royden, H. (2007) Real Analysis, 4th Edition, Prentice Hall.
Saez-Marti, M, Weibull, J. (2005) “Discounting and Altruism to Future DecisionMakers,” Journal of Economic Theory, Vol. 122, pp. 254–266.
Samuelson, P. (1937) “Some Aspects of the Pure Theory of Capital,” Quarterly Journalof Economics, pp. 469–496.
Sen, A. (1975) “Minimal conditions for monotonicity of capital value,” Journal of Economic
Theory, Vol. 11, pp. 340-355.
Strulovici, B. (2010) “Learning While Voting: Determinants of Collective Experimentation,” Econometrica, Vol. 78, No. 3, pp. 933–971.
Strulovici, B., Szydlowski, M. (2012) “On the Smoothness of Value Functions and the
Existence of Optimal Strategies,” Working Paper, Northwestern University.
Tabellini, G., Alesina, A. (1990) “Voting on the Budget Deficit,” American Economic
Review, Vol. 80, pp. 37–49.
Topkis, D. (1978) “Minimizing a Submodular Function on a lattice,” Operations Research,
Vol. 26, pp. 305–321.
Whittle, P. (1980) “Multi-armed bandits and the Gittins index,” Journal of the Royal
Statistical Society, Series B, Vol. 42, pp. 143–149.

45