# A Consumption and Investment Problem via a Markov Decision Processes Approach with Random Horizon

Authors: Octavio Paredes Pérez; Víctor Hugo Vázquez Guevara; Hugo Cruz-Suárez
Journal: Advances in Operations Research
 (2022)

---

## Abstract

This work is devoted to a consumption and investment problem, in which there is an investor with certain initial wealth with the possibility of deciding how much of such wealth will be consumed and how much will be invested in each of a series of successive times. The key issue is to find a wealth assignation rule in order to maximize the performance criteria; such dilemma will be achieved by the dynamic programming technique for the Markov decision processes with random horizon.


---

## Body

## 1. Introduction

Markov decision processes (MDPs) provide a very useful system for creating and implementing a decision-making process whose results are partially random. MDPs are useful stochastic processes for boarding a wide range of optimization problems of continuous or discrete nature (In this paper, it will be only considered the discrete framework). In all the sequel, at each step, the process is in some state and the decision maker may choose any action that is available for such a state. The process responds at the next stage by randomly moving to a new state and giving a reward to the decision maker. The central problem of MDPs is to find an “optimal policy”; i.e., a function that specifies some mechanism for selecting actions optimally at each stage.MDPs can be solved by dynamic programming. For example, in [1], a comprehensive and theoretical treatment of the mathematical foundations of optimal stochastic control of discrete-time systems is given; meanwhile, in [2], interest is mostly limited to MDPs with a Borel space of states and possibly unlimited costs. In [3], it is explained that the theory of the stochastic dynamic programming method is easily applicable to many practical problems, even for nonstationary models.However, there exist another method which may be considered for solving stochastic optimization problems, for example: In [4], an emended minimax method is developed based on the semi-autonomized multiobjective optimization algorithm by amending the classical minimax method, which leads to desirable optimal values in the certitude state and to find another Pareto optimal solution under fuzziness in the incertitude state. In [5], the author focuses on one general fractile criterion iterative-interactive optimization process in order to obtain the preferable Pareto optimal solution, subject to a specified main objective function to multiobjective stochastic linear programming problems in a fuzzy environment. In addition, in [6], one real-life-based cost-effective and customer-centric closed-loop supply chain management model is considered together with the T set that represents the inherent impreciseness to objective functions which conducts to find that optimal values are superior than stipulated goals to both the objective functions in the T environment. In [7], the effects of setup cost reduction and quality improvement in a two-echelon supply chain model with deterioration are developed. The objective is to minimize the total cost of the entire supply chain model by simultaneously optimizing setup cost, process quality, number of deliveries, and lot size. In [8], a set of very interesting situations coming from mobile and wireless networks, connection management, and Internet is considered in which optimal decisions are required and it is necessary to provide a side view about control problems and the theory behind them.In this paper, the possibility will be taken into account that external factors may force the process to be completed earlier than planned. In this way, it is necessary to consider the horizon as a random variable, which may be independent of the state-action space [9]. Such an idea has been explored; for example, in [10], where the optimal selection strategy for the Armed Bandit paradigm with random horizon and possibly random discount factors is found.Hence, it will be considered and investor with certain initial capital which; in each of a random number of times, may reinvest into risky assets, consume or invest in a risk-free bond. The goal is to conceive a strategy of consumption and investment in order to maximize the expected sum of an utility coming from; exclusively, the spent capital at each stage. Hence, in this paper, via the theory of MDPs with a finite random horizon, an optimal policy of consumption and investment will be established, in the case in which utility function responsible for evaluating consumption is of the exponential type. Although these kind of utility functions are rather classical, they are as well useful since such functions consider a constant absolute aversion to risk and are the only risk-averse increasing utility function whose risk premium is invariant with respect to wealth [11, 12]. The term risk aversion refers to the preference for stochastic realizations with limited deviation from the expected value. In risk-averse optimal control, one may prefer a policy with a higher cost in expectation but lower deviations than one with a lower cost but possibly higher deviations [13].This work is organized as follows: in section two, fundamental ideas around MDP with random horizon jointly with an equivalence between performance criteria of an MDP with a finite random horizon and the one associated with an MDP with deterministic horizon are analyzed. Section three will be addressed to a consumption and investment scenario with random horizon together with a numerical experiment. Additionally, this section contains the main contribution of this text: the finding of the optimal policy for the consumption and investment problem with a finite random horizon and exponential utility function. Finally, two appendices are included. On the one hand, Appendix A deals with basic definitions around the MDP theory together with some useful assumptions for solving the consumption and investment problem by means of the dynamic programming technique. On the other hand, the concept of the financial market will be discussed in Appendix B.


## 2. Markov Decision Process with a Random Horizon

In literature, one may find references where discrete-time control problems with a random horizon are discussed, for example: [9, 10]. Therefore, let τ be a discrete random variable associated with some probability space Ω′,ℱ′,ℙ. Suppose that the mass function of τ is known and given by ρn≔ℙτ=n, with n=0,1,2,…,N, where N is a natural number or N=∞. Consider now a Markov decision model E,A,D,Q,rn,g and define the following performance criteria:(1)Vτπ,x≔E∑n=0τ−1rnXn,an+gXτ,where π∈Δ, x∈E, and E denotes the expected value with respect to the joint distribution of the process xn,an and τ. In order to introduce the corresponding optimal control problem, we define the optimal value function as follows:(2)Vτx≔supπ∈ΔVτπ,x,x∈E.In this way, the optimal control problem with a random horizon consists of finding a policyπ∗∈Δ such that Vτπ∗,x=Vτx, ∀x∈E. The following assumption will be considered for simplifying the performance criteria under a discrete random horizon [9].Assumption 1.
For eachx∈E and π∈Δ induced process Xn,an|n=0,1,2,… is independent of τ.So, under Assumption1 and equation (1), reward performance criteria are reduced to [9].(3)Vτπ,x=Exπ∑n=0N−1ℙnrXn,an+ℙNgXN,x∈E,π∈Δ,where for each n=0,1,…,N, ℙn≔ℙτ≥n.Thus, the optimal control problem with a random horizonτ is equivalent to the optimal control problem with planning horizon N+1, a nonhomogeneous reward function and an equally zero terminal reward. Hence, Theorem A.6 may be considered under conditions of Assumption A.5. An alternative approach discussed in [9] considers a different set of assumption on the reward function (which remain fixed at each stage) and the transition kernel Q.


## 3. Consumption and Investment Problem with a Random Horizon

An investor has an initial wealth ofx>0 and at the beginning of at most N periods (implicitly a random horizon τ with support on 0,1,…,N is contemplated) he/she can determine which part of his/her wealth will be consumed and which part will be invested in the financial market given in Appendix B. Amount cn denotes the consumed amount at time n and will be evaluated by an utility function Uc. Remaining money will be invested in d risky assets and in a risk-free bond. Terminal riches Xτ is judged via another utility function Up. The main problem is designing a strategy of sequential decisions of consumption and investment in order to maximize the sum of his/her expected gains.In all the sequel, AssumptionB.9 will be supposed and no arbitrage opportunities are available. In addition, it is supposed that the domain of both utility functions Uc and Up is 0,∞. In this context, dynamics of the wealth is as follows:(4)Xn+1=1+in+1Xn−cn+an⋅Rn+1,where cn,an is a consumption and investment strategy, i.e., an and cn are ℱn adapted, 0≤cn≤Xn and X0=x.The consumption and investment problem previously described may be associated with a Markov decision model with the following components:E≔0,∞,A≔ℝ+×ℝd,Dx≔c,a∈A|0≤c≤xand1+in+1x−c+a⋅Rn+1≥0,ℙ−a.s. for x≥0,Z≔−1,∞d where z∈Z denotes the relative risk,Tx,c,a,z≔1+in+1x−c+a⋅z is the transition function,QZ is the distribution of Rn+1,rx,c,a≔Ucc is the reward function,gx≔Upx is the terminal reward function,τ is the random horizon with support on 0,1,…,N.In this framework, value function is defined as follows:(5)Vx=supπExπ∑n=0τ−1UccnXn+UpXτ,where the supremum is taken over all policies π=f0,…,fN−1 with fnx=cnx,anx.Sufficient conditions will be given to propose the solution of the consumption and investment problem with a random horizon with a finite support. Under assumption1, the proof of the following result can be obtained by using Theorem A.6 and the wealth dynamics given in (4) [14]. Its conclusion allows to associate a MDP with a random horizon with support on 0,1,…,N with another MDP with a nonhomogenous reward function, deterministic horizon N+1, and equally zero terminal cost.Theorem 1.
In the multiperiodic consumption and investment problem, we define functionsV0,V1,…,VN+1 on E by the following:(6)VN+1x≔0,VNx≔ℙNUpx,VNx≔supc,a∈DnxℙnUcc+EVn+11+in+1x−c+a⋅Rn+1.
Then, there exist maximizers fn∗ of Vn and the strategy f0∗,…,fN∗ is optimal for the consumption and investment problem.


### 3.1. Exponential Utility Function

This section deals with a version of the consumption and investment problem with exponential utility functions and a finite random horizon. In this setting, the process that describes the evolution of investor’s capital may end before some fixed horizon due to external causes. However, Assumption1 prevents the decision maker to finish such process because of bad investments, which may lead to drop this process below zero.Utility functions arise naturally in economics and finance, for example:On the mean-variance approach of Merton and Samuelson, it has already found that a quadratic utility provides a closed-form solution for the portfolio selection under very general conditions; however, on the case of power and the exponential utility function, there is no possibility to find closed-form solutions without information on the distribution of the return process [15]. In addition, in [16], by assuming that a portfolio’s returns follow an approximate log-normal distribution, the closed-form expressions of the optimal portfolio weights were obtained for both power and logarithmic utility functions.In portfolio optimization, in order to maximize the widespread logarithmic utility of some investor, assets whose prices depend on their past values in a non-Markovian way are taken into account [17]. On the same topic [18], Chapter 9 provides a very interesting contribution on the treatment of utility functions, in particular the risk aversion is deeply addressed.On a similar matter, in [19], it is possible to review a self-contained survey of utility functions (exponential and power utilities of the first and second kind) together with some of their applications in finance. This reference also discusses the Pareto optimal risk exchanges and presents very illustrative examples dealing with earlier mentioned utility functions.Exponential utility functions are widely employed because they consider a constant absolute aversion to risk [20–22], and they are the only risk-averse increasing utility functions whose risk premium is invariant respect to wealth [11, 12]. The term risk aversion refers to the preference for stochastic realizations with limited deviation from the expected value. In risk-averse optimal control, one may prefer a policy with a higher cost in expectation but lower deviations than one with lower cost but possibly higher deviations [13]. In addition, from the technical point of view, if both utility functions Uc and Up are of the following form:(7)−1γexp−γy,γ>0andy∈E,then they are bounded superiorly, and hence, Assumption B.9 is directly satisfied.The following assumption is needed by ensuring that optimal consumptions do not exceed available capital.Assumption 2.
For eachn=0,1,…,N and x>0, suppose that(8)1<ℙnkn+1bn+11+in+1vn<expxγ,where(9)kn+1=ℙn+1γkn+2bn+21+in+2vn+1ℙn+1γ/bn+21+in+2+γ+kn+2vn+1kn+2bn+21+in+2vn+1ℙn+1bn+21+in+2/bn+21+in+2+γ,(10)bn+1=bn+21+in+2γbn+21+in+2+γ,with kN=ℙN1/γ, bN=γ, ℙn≔∑i=nNρi=ℙτ≥n, n=0,1,2,…,N and(11)vn≔infα∈ℝdEexp−γSN0Sn0α⋅Rn+1,which exists, thanks to Theorem 4.1.1 of [14].Then, it is possible to deduce the following result for the consumption and investment problem with a (finite) random horizon and risk-averse increasing utility function whose risk premium is invariant with respect to wealth (exponential utility function). This theorem provides a mechanism for acting optimally with respect to the consumption and risk investments at each stage, such optimal decisions come from the optimization of equations expressed in Theorem1 which are relatively simple; in this case, thanks to the definitions of coefficients bn and kn.Theorem 2.
Suppose that bothUcc and Upx are exponential utility functions of the form (5) with γ>0. Then, it holds that:(12)VN+1x=0,VNx=ℙN−1γexp−γx=−kNexp−bNx,VNx==−knexp−bnx,n∈0,1,…,N−1,∀x≥0, where kn, bn, and vn are given by equations (9)–(11), respectively. In addition, optimal consumption at stage n is cn∗=ζn∗xn, where(13)ζn∗=lnℙn+bn+11+in+1xn−lnkn+1bn+11+in+1vnxnbn+11+in+1+γ,and the optimal investment at stage n is an∗=SN0/Sn01+in+1αn∗, where αn∗ is the solution of (11), which may be found via αn∗=1/γSn0/SN0α˜n where α˜n is the minimum point of(14)α↦Eexp−α⋅Rn+1,α∈ℝd.Remark 1.
Preceding theorem supplies the pursued consumption and investment optimal policyfn∗ which is such that(15)fn∗x=ζn∗xn,an∗,with ζn∗ and an∗ as in Theorem 2.Proof 1.
First of all, consider the following sets of functions in order to test AssumptionA.5(16)M≔v:E⟶ℝ|vx≡0orvx=−kexp−bx,fork,b>0,Ψ≔f∈F|fx=ζx,α,withα∈ℝd,ζ∈ℝ+.(i)
VN+1x≡0 and VNx=gNx=Upx, and both belong to M.(ii)
Ifvx≡0, then it is straightforward to see that Tnvx∈M. Assume now that, vx=−kexp−bx for k,b>0. In consequence,(17)Tnvx=supc,a∈DxℙnUcc+Ev1+in+1x−c+a⋅Rn+1=supc,a∈Dxℙn−1γexp−γc+E−kexp−b1+in+1x−c+a⋅Rn+1=supc,a∈Dx−ℙnγexp−γc−kexp−b1+in+1x−cEexp−b1+in+1a⋅Rn+1.Consider the transformation:c=ζx and a=SN0/Sn0b1+in+1α, hence:(18)Tnvx=sup0≤ζ≤1−ℙnγexp−γζx−kexp−b1+in+1x−ζxEα∈ℝdexp−b1+in+1SN0Sn0b1+in+1α⋅Rn+1=sup0≤ζ≤1−ℙnγexp−γζx−kexp−b1+in+1x−ζxvn,wherevn is given in (11).(iii)
The existence of a maximizer in the setΨ will be proven. For this, we examine the real function ℓζ stated as(19)ℓζ≔−ℙnγexp−γζx−kexp−b1+in+1x−ζxvn.It is possible to discover its maximum through standard optimization techniques. Hence, from Assumption2, it is observed that the only critical point of ℓ in 0,1 is as follows:(20)ζ=lnℙn+b1+in+1x−lnkb1+in+1vnxb1+in+1+γ.which is a relative maximum via the criterion of the second derivative. By substituting the value ofζ in Tnvx, it is found that:(21)Tnvx=−ℙnγkb1+in+1vnℙnγ/b1+in+1+γkkb1+in+1vnℙn−b1+in+1/b1+in+1+γvnexp−b1+in+1γb1+in+1+γx.Therefore, the maximizer forv is of the form ζx,a and Tnvx∈M(iv)
Expressions forVn and their corresponding maximizers by utilizing Theorem 1 will be attained.
For eachx>0,(22)VNx=ℙN−1γexp−γx=−kNexp−bNx.
By an inductive process and following essentially the same lines as those inii and iii, it may be found for n=1,…,N−1 that:(23)VN−nx=−kN−nexp−bN−nx,∀x∈E,where kN−n and bN−n are expressed in (9) and (10). Additionally, optimal consumption is given by cN−n=ζN−nx with:(24)ζN−n=lnℙN−n+bN−n−11+iN−n−1x−lnkN−n−1bN−n−11+iN−n−1vN−nxbN−n−11+iN−n−1+γ,and the optimal investment is a∗=SN0/SN−n0γ1+iN−n−1α∗, where α∗ is the solution of(25)vN−n=infα∗∈ℝdEexp−γSN0SN−n0α⋅RN−n−1.Remark 2.
The earlier theorem states that under its own assumptions, it is possible to explicitly find the optimal strategy; hence, it is not necessary to perform numerical methods for solving the dynamic programming equation at each stage. However, if this was not the case, there exist several papers dealing with complexity of solution algorithms for MDPs for finite state and action spaces. Nevertheless, we refer to the reader to the contribution of Chow and Tsitsiklis [23] where tight lower bounds on the computational complexity of dynamic programming for the case where the state space is continuous, and the problem is to be solved approximately, within a specified accuracy. On the same direction, Section 12.5 of [24] is also relevant for the framework studied in this article.


### 3.2. Numerical Example

In this section, results of Section3.1 will be illustrated. For this, consider d=2 (two risky assets) and that distribution of relative risk random vectors Rn may be approximated by a bivariate normal distribution with parameters μn and Σn. In this case, it may be found that, for each n(26)vn=e−μn+1T⋅Σn+1−1μn+1+1/2Σn+1−1μn+1TΣn+1Σn+1−1μn+1,and that(27)an∗=1γ1+in+1Σn+1−1μn+1.For the sake of simplicity, we consider that random vectorsRn are independent and identically distributed with parameters Σn+1=1/8001/8 and μn+1=0.10.2. Additionally, set γ=0.6 for having a not so flat utility function; over the horizontal axis (bigger values of γ lead to utility functions closer to the x-axis), in addition a constant interest rate will be contemplated: in=0.05. Finally, it will be established that a random horizon will behave as a binomially distribution with parameters 10 and 0.5.Now, we split the simulation example into two stages:


#### 3.2.1. Stage I: Before Implementing Dynamics of the Wealth Process

At this stage, it is possible to find corresponding values ofvn and an∗. Given the simplifications considered above, it is achievable to find constant values of(28)vn=0.877204,an∗=1.26984,2.53968.And the values ofbn and kn (that will help on construction of optimal consumptions) observed in Table 1.Table 1
Values ofbn and kn
nbnkn00.07400.45610.08040.543320.08840.535830.09880.445440.11260.313250.13200.185260.16110.090470.20980.035080.30730.009990.60000.0016This is possible since these parameters do not depend on the initial capital or the wealth process.


#### 3.2.2. Stage II: Performing the Wealth Process

At this stage, the initial capital becomes relevant; hence, Tables2–4 expose the evolution of relative and absolute optimal consumptions as well as a trajectory of Xn with an initial capital equal $100,$500, and $1,000. A decreasing behavior is observed on the wealth process accompanied by an increasing behavior of relative consumption, which leads to an “almost” constant performance of absolute consumption. On the same fashion, Figures 1–3 illustrate same observations but allow to compare trajectories of ζn,cn, and xn.Table 2
Evolution of optimal consumptions and wealth processes withx0=$100.
nζncx00.09839.8310010.125511.731293.475820.151113.235287.592730.177513.9678.657640.202713.661067.395650.239714.067158.686460.281613.172846.778670.330312.513337.884780.401410.893627.139190.51997.236013.8624Table 3
Evolution of optimal consumptions and wealth processes withx0=$500.
nζncx00.111455.750010.123857.6066465.320320.137559.1076429.873830.153459.8102389.897240.172959.8306346.041750.197759.9246303.109260.231359.0432255.267070.279858.3780208.642080.358656.7587158.278690.513253.0666103.4034Table 4
Evolution of optimal consumptions and wealth processes withx0=$1000.
nζncx00.1144114.4100010.1233114.6273929.662520.1341114.7768855.905030.1476114.8599778.183740.1650114.9040696.388450.1881114.8943610.815060.2205114.8113520.686570.2691114.7476426.412580.3501114.5821327.284290.5122114.2206223.0001Figure 1
Capital evolution.Figure 2
Relative consumption evolution.Figure 3
Absolute consumption evolution.In Figure4, the dynamics of relative consumption are observed by considering different values of γ with remaining parameters fixed as before. It may be noticed that practically, no effect is observed.Figure 4
Relative consumption evolution by varyingγ.In Figure5, sensibility analysis of optimal relative consumption with respect to interest rate is schematized. It is possible to appreciate an increasing performance as the constant interest rate increases; nevertheless, Figure 6 exhibits a similar shape on absolute optimal consumptions than Figure 3. In addition, (27) implies a decreasing behavior of optimal investments.Figure 5
Relative consumption evolution by varying interest rate.Figure 6
Wealth evolution by varying interest rate.Furthermore, Figure7 shows a concave behavior on wealth dynamics as interest rate increases, however, one may remark that: naturally, wealth is bigger for huger values of i.Figure 7
Wealth evolution by varying interest rate.In order to perform an analysis of execution time by implementing the strategy dictated in Theorem2, an initial capital of $1,000,000 with a fixed interest rate of .05 and a horizon with discrete uniform distribution will be taken into account. Hence, in Table 5, the support of random horizon is increased as well as the value of parameter γ. From such a table, together with Figure 8, it is observed that the value of γ has no influence on execution time and that it poses a slow increasing rate (below the identity function) as maximum value of random horizon grows.Table 5
Execution time when the initial capital is$1×106 with interest rate i=0.05.
Nγ=0.1γ=0.5γ=3γ=6γ=60γ=600100.0500620.0487070.0543270.0516120.0506010.0534601000.0560880.0541500.0562770.0554590.0585220.0558455000.0991380.0762280.0784270.0761780.0865100.07639610000.1008920.1009310.1024640.1235350.1018020.09961950000.3105580.3097410.3103880.3108150.3411300.3330531×1040.6184410.6151980.6087870.6374770.6218240.6422651×10513.09300313.23226813.16681913.68787713.14108113.3136202×105122.116643119.764554120.755948118.576175117.964425121.3252804×105672.962272672.426565666.151385672.633024691.408851664.8689046×1051748.2922271664.1206371661.0679921674.3609701736.7903311655.1106828×1053088.8963573096.601483076.6101243220.8608913111.9671923083.4802881×1064937.2313324897.3888114900.3087285013.4958084955.1273754916.801511Figure 8
Execution time when the initial capital is$1×106 with interest rate i=0.05.On the same fashion, a discretely uniform horizon will be considered again, an initial capital of$1,000,000 but now γ=0.3 and interest rate will vary from 0.01 to 0.75. In this setting, it is possible to observe in Table 6 as well as in Figure 9 that the interest rate has no repercussion on execution time and that it follows a moderate rise as a random horizon owns a larger support.Table 6
Execution time when the initial capital is$1×106 with γ=0.3.
Ni=0.01i=0.03i=0.05i=0.25i=0.5i=0.75100.0492720.0521320.0494650.0491040.0436700.0497451000.0544730.0553180.0576880.0578670.0583720.0553055000.0755730.0778490.0777990.0768460.0763830.07557110000.1037490.1019130.1020750.1008610.1039450.10510050000.3185980.3093530.3150170.3225250.3242650.3312671×1040.6330600.6644030.6339800.6339520.6354050.6595291×10513.33949013.37760413.35255013.47517413.37924813.5997372×105121.145216122.143420122.275267122.301754122.777018126.0048074×105703.916410673.852490677.379424681.878394719.429588678.4658436×1051667.0687111678.7132651664.5572721692.6025571678.6394821716.2137048×1053152.0068123139.2879813125.0319733083.3424233087.6339513092.6782431×1064883.2653784877.6924004873.4822254885.9970064992.8161114870.649850Figure 9
Execution time when initial capital is$1×106 with γ=0.3.By summarizing, it may be seen that neither the value of parameter of exponential utility function nor the interest rate have an effect on performance time and that implementation time grows slower than the maximum value of horizon.


## 3.1. Exponential Utility Function

This section deals with a version of the consumption and investment problem with exponential utility functions and a finite random horizon. In this setting, the process that describes the evolution of investor’s capital may end before some fixed horizon due to external causes. However, Assumption1 prevents the decision maker to finish such process because of bad investments, which may lead to drop this process below zero.Utility functions arise naturally in economics and finance, for example:On the mean-variance approach of Merton and Samuelson, it has already found that a quadratic utility provides a closed-form solution for the portfolio selection under very general conditions; however, on the case of power and the exponential utility function, there is no possibility to find closed-form solutions without information on the distribution of the return process [15]. In addition, in [16], by assuming that a portfolio’s returns follow an approximate log-normal distribution, the closed-form expressions of the optimal portfolio weights were obtained for both power and logarithmic utility functions.In portfolio optimization, in order to maximize the widespread logarithmic utility of some investor, assets whose prices depend on their past values in a non-Markovian way are taken into account [17]. On the same topic [18], Chapter 9 provides a very interesting contribution on the treatment of utility functions, in particular the risk aversion is deeply addressed.On a similar matter, in [19], it is possible to review a self-contained survey of utility functions (exponential and power utilities of the first and second kind) together with some of their applications in finance. This reference also discusses the Pareto optimal risk exchanges and presents very illustrative examples dealing with earlier mentioned utility functions.Exponential utility functions are widely employed because they consider a constant absolute aversion to risk [20–22], and they are the only risk-averse increasing utility functions whose risk premium is invariant respect to wealth [11, 12]. The term risk aversion refers to the preference for stochastic realizations with limited deviation from the expected value. In risk-averse optimal control, one may prefer a policy with a higher cost in expectation but lower deviations than one with lower cost but possibly higher deviations [13]. In addition, from the technical point of view, if both utility functions Uc and Up are of the following form:(7)−1γexp−γy,γ>0andy∈E,then they are bounded superiorly, and hence, Assumption B.9 is directly satisfied.The following assumption is needed by ensuring that optimal consumptions do not exceed available capital.Assumption 2.
For eachn=0,1,…,N and x>0, suppose that(8)1<ℙnkn+1bn+11+in+1vn<expxγ,where(9)kn+1=ℙn+1γkn+2bn+21+in+2vn+1ℙn+1γ/bn+21+in+2+γ+kn+2vn+1kn+2bn+21+in+2vn+1ℙn+1bn+21+in+2/bn+21+in+2+γ,(10)bn+1=bn+21+in+2γbn+21+in+2+γ,with kN=ℙN1/γ, bN=γ, ℙn≔∑i=nNρi=ℙτ≥n, n=0,1,2,…,N and(11)vn≔infα∈ℝdEexp−γSN0Sn0α⋅Rn+1,which exists, thanks to Theorem 4.1.1 of [14].Then, it is possible to deduce the following result for the consumption and investment problem with a (finite) random horizon and risk-averse increasing utility function whose risk premium is invariant with respect to wealth (exponential utility function). This theorem provides a mechanism for acting optimally with respect to the consumption and risk investments at each stage, such optimal decisions come from the optimization of equations expressed in Theorem1 which are relatively simple; in this case, thanks to the definitions of coefficients bn and kn.Theorem 2.
Suppose that bothUcc and Upx are exponential utility functions of the form (5) with γ>0. Then, it holds that:(12)VN+1x=0,VNx=ℙN−1γexp−γx=−kNexp−bNx,VNx==−knexp−bnx,n∈0,1,…,N−1,∀x≥0, where kn, bn, and vn are given by equations (9)–(11), respectively. In addition, optimal consumption at stage n is cn∗=ζn∗xn, where(13)ζn∗=lnℙn+bn+11+in+1xn−lnkn+1bn+11+in+1vnxnbn+11+in+1+γ,and the optimal investment at stage n is an∗=SN0/Sn01+in+1αn∗, where αn∗ is the solution of (11), which may be found via αn∗=1/γSn0/SN0α˜n where α˜n is the minimum point of(14)α↦Eexp−α⋅Rn+1,α∈ℝd.Remark 1.
Preceding theorem supplies the pursued consumption and investment optimal policyfn∗ which is such that(15)fn∗x=ζn∗xn,an∗,with ζn∗ and an∗ as in Theorem 2.Proof 1.
First of all, consider the following sets of functions in order to test AssumptionA.5(16)M≔v:E⟶ℝ|vx≡0orvx=−kexp−bx,fork,b>0,Ψ≔f∈F|fx=ζx,α,withα∈ℝd,ζ∈ℝ+.(i)
VN+1x≡0 and VNx=gNx=Upx, and both belong to M.(ii)
Ifvx≡0, then it is straightforward to see that Tnvx∈M. Assume now that, vx=−kexp−bx for k,b>0. In consequence,(17)Tnvx=supc,a∈DxℙnUcc+Ev1+in+1x−c+a⋅Rn+1=supc,a∈Dxℙn−1γexp−γc+E−kexp−b1+in+1x−c+a⋅Rn+1=supc,a∈Dx−ℙnγexp−γc−kexp−b1+in+1x−cEexp−b1+in+1a⋅Rn+1.Consider the transformation:c=ζx and a=SN0/Sn0b1+in+1α, hence:(18)Tnvx=sup0≤ζ≤1−ℙnγexp−γζx−kexp−b1+in+1x−ζxEα∈ℝdexp−b1+in+1SN0Sn0b1+in+1α⋅Rn+1=sup0≤ζ≤1−ℙnγexp−γζx−kexp−b1+in+1x−ζxvn,wherevn is given in (11).(iii)
The existence of a maximizer in the setΨ will be proven. For this, we examine the real function ℓζ stated as(19)ℓζ≔−ℙnγexp−γζx−kexp−b1+in+1x−ζxvn.It is possible to discover its maximum through standard optimization techniques. Hence, from Assumption2, it is observed that the only critical point of ℓ in 0,1 is as follows:(20)ζ=lnℙn+b1+in+1x−lnkb1+in+1vnxb1+in+1+γ.which is a relative maximum via the criterion of the second derivative. By substituting the value ofζ in Tnvx, it is found that:(21)Tnvx=−ℙnγkb1+in+1vnℙnγ/b1+in+1+γkkb1+in+1vnℙn−b1+in+1/b1+in+1+γvnexp−b1+in+1γb1+in+1+γx.Therefore, the maximizer forv is of the form ζx,a and Tnvx∈M(iv)
Expressions forVn and their corresponding maximizers by utilizing Theorem 1 will be attained.
For eachx>0,(22)VNx=ℙN−1γexp−γx=−kNexp−bNx.
By an inductive process and following essentially the same lines as those inii and iii, it may be found for n=1,…,N−1 that:(23)VN−nx=−kN−nexp−bN−nx,∀x∈E,where kN−n and bN−n are expressed in (9) and (10). Additionally, optimal consumption is given by cN−n=ζN−nx with:(24)ζN−n=lnℙN−n+bN−n−11+iN−n−1x−lnkN−n−1bN−n−11+iN−n−1vN−nxbN−n−11+iN−n−1+γ,and the optimal investment is a∗=SN0/SN−n0γ1+iN−n−1α∗, where α∗ is the solution of(25)vN−n=infα∗∈ℝdEexp−γSN0SN−n0α⋅RN−n−1.Remark 2.
The earlier theorem states that under its own assumptions, it is possible to explicitly find the optimal strategy; hence, it is not necessary to perform numerical methods for solving the dynamic programming equation at each stage. However, if this was not the case, there exist several papers dealing with complexity of solution algorithms for MDPs for finite state and action spaces. Nevertheless, we refer to the reader to the contribution of Chow and Tsitsiklis [23] where tight lower bounds on the computational complexity of dynamic programming for the case where the state space is continuous, and the problem is to be solved approximately, within a specified accuracy. On the same direction, Section 12.5 of [24] is also relevant for the framework studied in this article.


## 3.2. Numerical Example

In this section, results of Section3.1 will be illustrated. For this, consider d=2 (two risky assets) and that distribution of relative risk random vectors Rn may be approximated by a bivariate normal distribution with parameters μn and Σn. In this case, it may be found that, for each n(26)vn=e−μn+1T⋅Σn+1−1μn+1+1/2Σn+1−1μn+1TΣn+1Σn+1−1μn+1,and that(27)an∗=1γ1+in+1Σn+1−1μn+1.For the sake of simplicity, we consider that random vectorsRn are independent and identically distributed with parameters Σn+1=1/8001/8 and μn+1=0.10.2. Additionally, set γ=0.6 for having a not so flat utility function; over the horizontal axis (bigger values of γ lead to utility functions closer to the x-axis), in addition a constant interest rate will be contemplated: in=0.05. Finally, it will be established that a random horizon will behave as a binomially distribution with parameters 10 and 0.5.Now, we split the simulation example into two stages:


### 3.2.1. Stage I: Before Implementing Dynamics of the Wealth Process

At this stage, it is possible to find corresponding values ofvn and an∗. Given the simplifications considered above, it is achievable to find constant values of(28)vn=0.877204,an∗=1.26984,2.53968.And the values ofbn and kn (that will help on construction of optimal consumptions) observed in Table 1.Table 1
Values ofbn and kn
nbnkn00.07400.45610.08040.543320.08840.535830.09880.445440.11260.313250.13200.185260.16110.090470.20980.035080.30730.009990.60000.0016This is possible since these parameters do not depend on the initial capital or the wealth process.


### 3.2.2. Stage II: Performing the Wealth Process

At this stage, the initial capital becomes relevant; hence, Tables2–4 expose the evolution of relative and absolute optimal consumptions as well as a trajectory of Xn with an initial capital equal $100,$500, and $1,000. A decreasing behavior is observed on the wealth process accompanied by an increasing behavior of relative consumption, which leads to an “almost” constant performance of absolute consumption. On the same fashion, Figures 1–3 illustrate same observations but allow to compare trajectories of ζn,cn, and xn.Table 2
Evolution of optimal consumptions and wealth processes withx0=$100.
nζncx00.09839.8310010.125511.731293.475820.151113.235287.592730.177513.9678.657640.202713.661067.395650.239714.067158.686460.281613.172846.778670.330312.513337.884780.401410.893627.139190.51997.236013.8624Table 3
Evolution of optimal consumptions and wealth processes withx0=$500.
nζncx00.111455.750010.123857.6066465.320320.137559.1076429.873830.153459.8102389.897240.172959.8306346.041750.197759.9246303.109260.231359.0432255.267070.279858.3780208.642080.358656.7587158.278690.513253.0666103.4034Table 4
Evolution of optimal consumptions and wealth processes withx0=$1000.
nζncx00.1144114.4100010.1233114.6273929.662520.1341114.7768855.905030.1476114.8599778.183740.1650114.9040696.388450.1881114.8943610.815060.2205114.8113520.686570.2691114.7476426.412580.3501114.5821327.284290.5122114.2206223.0001Figure 1
Capital evolution.Figure 2
Relative consumption evolution.Figure 3
Absolute consumption evolution.In Figure4, the dynamics of relative consumption are observed by considering different values of γ with remaining parameters fixed as before. It may be noticed that practically, no effect is observed.Figure 4
Relative consumption evolution by varyingγ.In Figure5, sensibility analysis of optimal relative consumption with respect to interest rate is schematized. It is possible to appreciate an increasing performance as the constant interest rate increases; nevertheless, Figure 6 exhibits a similar shape on absolute optimal consumptions than Figure 3. In addition, (27) implies a decreasing behavior of optimal investments.Figure 5
Relative consumption evolution by varying interest rate.Figure 6
Wealth evolution by varying interest rate.Furthermore, Figure7 shows a concave behavior on wealth dynamics as interest rate increases, however, one may remark that: naturally, wealth is bigger for huger values of i.Figure 7
Wealth evolution by varying interest rate.In order to perform an analysis of execution time by implementing the strategy dictated in Theorem2, an initial capital of $1,000,000 with a fixed interest rate of .05 and a horizon with discrete uniform distribution will be taken into account. Hence, in Table 5, the support of random horizon is increased as well as the value of parameter γ. From such a table, together with Figure 8, it is observed that the value of γ has no influence on execution time and that it poses a slow increasing rate (below the identity function) as maximum value of random horizon grows.Table 5
Execution time when the initial capital is$1×106 with interest rate i=0.05.
Nγ=0.1γ=0.5γ=3γ=6γ=60γ=600100.0500620.0487070.0543270.0516120.0506010.0534601000.0560880.0541500.0562770.0554590.0585220.0558455000.0991380.0762280.0784270.0761780.0865100.07639610000.1008920.1009310.1024640.1235350.1018020.09961950000.3105580.3097410.3103880.3108150.3411300.3330531×1040.6184410.6151980.6087870.6374770.6218240.6422651×10513.09300313.23226813.16681913.68787713.14108113.3136202×105122.116643119.764554120.755948118.576175117.964425121.3252804×105672.962272672.426565666.151385672.633024691.408851664.8689046×1051748.2922271664.1206371661.0679921674.3609701736.7903311655.1106828×1053088.8963573096.601483076.6101243220.8608913111.9671923083.4802881×1064937.2313324897.3888114900.3087285013.4958084955.1273754916.801511Figure 8
Execution time when the initial capital is$1×106 with interest rate i=0.05.On the same fashion, a discretely uniform horizon will be considered again, an initial capital of$1,000,000 but now γ=0.3 and interest rate will vary from 0.01 to 0.75. In this setting, it is possible to observe in Table 6 as well as in Figure 9 that the interest rate has no repercussion on execution time and that it follows a moderate rise as a random horizon owns a larger support.Table 6
Execution time when the initial capital is$1×106 with γ=0.3.
Ni=0.01i=0.03i=0.05i=0.25i=0.5i=0.75100.0492720.0521320.0494650.0491040.0436700.0497451000.0544730.0553180.0576880.0578670.0583720.0553055000.0755730.0778490.0777990.0768460.0763830.07557110000.1037490.1019130.1020750.1008610.1039450.10510050000.3185980.3093530.3150170.3225250.3242650.3312671×1040.6330600.6644030.6339800.6339520.6354050.6595291×10513.33949013.37760413.35255013.47517413.37924813.5997372×105121.145216122.143420122.275267122.301754122.777018126.0048074×105703.916410673.852490677.379424681.878394719.429588678.4658436×1051667.0687111678.7132651664.5572721692.6025571678.6394821716.2137048×1053152.0068123139.2879813125.0319733083.3424233087.6339513092.6782431×1064883.2653784877.6924004873.4822254885.9970064992.8161114870.649850Figure 9
Execution time when initial capital is$1×106 with γ=0.3.By summarizing, it may be seen that neither the value of parameter of exponential utility function nor the interest rate have an effect on performance time and that implementation time grows slower than the maximum value of horizon.


## 3.2.1. Stage I: Before Implementing Dynamics of the Wealth Process

At this stage, it is possible to find corresponding values ofvn and an∗. Given the simplifications considered above, it is achievable to find constant values of(28)vn=0.877204,an∗=1.26984,2.53968.And the values ofbn and kn (that will help on construction of optimal consumptions) observed in Table 1.Table 1
Values ofbn and kn
nbnkn00.07400.45610.08040.543320.08840.535830.09880.445440.11260.313250.13200.185260.16110.090470.20980.035080.30730.009990.60000.0016This is possible since these parameters do not depend on the initial capital or the wealth process.


## 3.2.2. Stage II: Performing the Wealth Process

At this stage, the initial capital becomes relevant; hence, Tables2–4 expose the evolution of relative and absolute optimal consumptions as well as a trajectory of Xn with an initial capital equal $100,$500, and $1,000. A decreasing behavior is observed on the wealth process accompanied by an increasing behavior of relative consumption, which leads to an “almost” constant performance of absolute consumption. On the same fashion, Figures 1–3 illustrate same observations but allow to compare trajectories of ζn,cn, and xn.Table 2
Evolution of optimal consumptions and wealth processes withx0=$100.
nζncx00.09839.8310010.125511.731293.475820.151113.235287.592730.177513.9678.657640.202713.661067.395650.239714.067158.686460.281613.172846.778670.330312.513337.884780.401410.893627.139190.51997.236013.8624Table 3
Evolution of optimal consumptions and wealth processes withx0=$500.
nζncx00.111455.750010.123857.6066465.320320.137559.1076429.873830.153459.8102389.897240.172959.8306346.041750.197759.9246303.109260.231359.0432255.267070.279858.3780208.642080.358656.7587158.278690.513253.0666103.4034Table 4
Evolution of optimal consumptions and wealth processes withx0=$1000.
nζncx00.1144114.4100010.1233114.6273929.662520.1341114.7768855.905030.1476114.8599778.183740.1650114.9040696.388450.1881114.8943610.815060.2205114.8113520.686570.2691114.7476426.412580.3501114.5821327.284290.5122114.2206223.0001Figure 1
Capital evolution.Figure 2
Relative consumption evolution.Figure 3
Absolute consumption evolution.In Figure4, the dynamics of relative consumption are observed by considering different values of γ with remaining parameters fixed as before. It may be noticed that practically, no effect is observed.Figure 4
Relative consumption evolution by varyingγ.In Figure5, sensibility analysis of optimal relative consumption with respect to interest rate is schematized. It is possible to appreciate an increasing performance as the constant interest rate increases; nevertheless, Figure 6 exhibits a similar shape on absolute optimal consumptions than Figure 3. In addition, (27) implies a decreasing behavior of optimal investments.Figure 5
Relative consumption evolution by varying interest rate.Figure 6
Wealth evolution by varying interest rate.Furthermore, Figure7 shows a concave behavior on wealth dynamics as interest rate increases, however, one may remark that: naturally, wealth is bigger for huger values of i.Figure 7
Wealth evolution by varying interest rate.In order to perform an analysis of execution time by implementing the strategy dictated in Theorem2, an initial capital of $1,000,000 with a fixed interest rate of .05 and a horizon with discrete uniform distribution will be taken into account. Hence, in Table 5, the support of random horizon is increased as well as the value of parameter γ. From such a table, together with Figure 8, it is observed that the value of γ has no influence on execution time and that it poses a slow increasing rate (below the identity function) as maximum value of random horizon grows.Table 5
Execution time when the initial capital is$1×106 with interest rate i=0.05.
Nγ=0.1γ=0.5γ=3γ=6γ=60γ=600100.0500620.0487070.0543270.0516120.0506010.0534601000.0560880.0541500.0562770.0554590.0585220.0558455000.0991380.0762280.0784270.0761780.0865100.07639610000.1008920.1009310.1024640.1235350.1018020.09961950000.3105580.3097410.3103880.3108150.3411300.3330531×1040.6184410.6151980.6087870.6374770.6218240.6422651×10513.09300313.23226813.16681913.68787713.14108113.3136202×105122.116643119.764554120.755948118.576175117.964425121.3252804×105672.962272672.426565666.151385672.633024691.408851664.8689046×1051748.2922271664.1206371661.0679921674.3609701736.7903311655.1106828×1053088.8963573096.601483076.6101243220.8608913111.9671923083.4802881×1064937.2313324897.3888114900.3087285013.4958084955.1273754916.801511Figure 8
Execution time when the initial capital is$1×106 with interest rate i=0.05.On the same fashion, a discretely uniform horizon will be considered again, an initial capital of$1,000,000 but now γ=0.3 and interest rate will vary from 0.01 to 0.75. In this setting, it is possible to observe in Table 6 as well as in Figure 9 that the interest rate has no repercussion on execution time and that it follows a moderate rise as a random horizon owns a larger support.Table 6
Execution time when the initial capital is$1×106 with γ=0.3.
Ni=0.01i=0.03i=0.05i=0.25i=0.5i=0.75100.0492720.0521320.0494650.0491040.0436700.0497451000.0544730.0553180.0576880.0578670.0583720.0553055000.0755730.0778490.0777990.0768460.0763830.07557110000.1037490.1019130.1020750.1008610.1039450.10510050000.3185980.3093530.3150170.3225250.3242650.3312671×1040.6330600.6644030.6339800.6339520.6354050.6595291×10513.33949013.37760413.35255013.47517413.37924813.5997372×105121.145216122.143420122.275267122.301754122.777018126.0048074×105703.916410673.852490677.379424681.878394719.429588678.4658436×1051667.0687111678.7132651664.5572721692.6025571678.6394821716.2137048×1053152.0068123139.2879813125.0319733083.3424233087.6339513092.6782431×1064883.2653784877.6924004873.4822254885.9970064992.8161114870.649850Figure 9
Execution time when initial capital is$1×106 with γ=0.3.By summarizing, it may be seen that neither the value of parameter of exponential utility function nor the interest rate have an effect on performance time and that implementation time grows slower than the maximum value of horizon.


## 4. Conclusions

In this paper, a consumption and investment problem was studied through a Markov decision process with random horizon of finite support. In this framework, the optimal consumption and investment was obtained via a dynamic programming approach by evaluating consumptions via an exponential utility function.



---

*Source: 3184610-2022-06-07.xml*