Global Concavity and Optimization in a Class of Dynamic Discrete Choice Models

Discrete choice models with unobserved heterogeneity are commonly used Econometric models for dynamic Economic behavior which have been adopted in practice to predict behavior of individuals and firms from schooling and job choices to strategic decisions in market competition . These models feature optimizing agents who choose among a finite set of options in a sequence of periods and receive choice-specific payoffs that depend on both variables that are observed by the agent and recorded in the data and variables that are only observed by the agent but not recorded in the data . Existing work in Econometrics assumes that optimizing agents are fully rational and requires finding a functional fixed point to find the optimal policy . We show that in an important class of discrete choice models the value function is globally concave in the policy . That means that simple algorithms that do not require fixed point computation , such as the policy gradient algorithm , globally converge to the optimal policy . This finding can both be used to relax behavioral assumption regarding the optimizing agents and to facilitate Econometric analysis of dynamic behavior . In particular , we demonstrate significant computational advantages in using a simple implementation policy gradient algorithm over existing “ nested fixed point ” algorithms used in Econometrics . 1 INTRODUCTION . Dynamic discrete choice model with unobserved heterogeneity is , arguably , the most popular model that is currently used for Econometric analysis of dynamic behavior of individuals and firms in Economics and Marketing ( e.g . see surveys in Eckstein and Wolpin ( 1989 ) , Dubé et al . ( 2002 ) Abbring and Heckman ( 2007 ) , Aguirregabiria and Mira ( 2010 ) ) . Even most recent Econometric papers on single-agent dynamic decision-making use this setup to showcase their results ( e.g . Arcidiacono and Miller , 2011 ; Aguirregabiria and Magesan , 2016 ; Müller and Reich , 2018 ) .In this model , pioneered in Rust ( 1987 ) , the agent chooses between a discrete set of options ( typically 2 ) in a sequence of discrete time periods to maximize the expected cumulative discounted payoff . The reward in each period is a function of the state variable which follows a Markov process and is observed in the data and also a function of an idiosyncratic random variable that is only observed by the agent but is not reported in the data . The unobserved idiosyncratic component is designed to reflect heterogeneity of agents that may value the same choice differently . Despite significant empirical success in prediction of dynamic economic behavior under uncertainty , dynamic discrete choice models frequently lead to seemingly unrealistic optimization problems that economic agents need to solve . For instance , Hendel and Nevo ( 2006 ) features an elaborate functional fixed point problem with constraints , which is computationally intensive , especially in continuous state spaces , for consumers to buy laundry detergent in the supermarket . Common approach for this functional fixed point problem is value function iteration ( See Section 2.3 for more discussion ) . At the same time , rich literature on Markov Decision Processes ( cf . Sutton and Barto , 2018 ) have developed several effective optimization algorithms , such as the policy gradient algorithm and its variants , that do not require solving for a functional fixed point . However , the drawback of the policy gradient is that the value function in a generic Markov Decision problem is not concave in the policy . This means that gradient-based algorithms have no guarantees for global convergence for a generic MDP . While for some specific and simple models where closed-form characterizations exist , the convergence results are shown by model-specific technique which is hard to generalize ( e.g . Fazel et al. , 2018 , for linear quadratic regulator ) . In this paper our main goal is to resolve the dichotomy in empirical social science literature that the rationality of consumers requires for them to be able to solve the functional fixed point problem which is computationally intensive . Our main theoretic contribution is the proof that , in the class of dynamic discrete choice models with unobserved heterogeneity , the value function of the optimizing agent is globally concave in the policy . This implies that a large set of policy gradient algorithms that have a modest computational power requirement for the optimizing agents have a fast convergence guarantee in our considered class of dynamic discrete choice models . The importance of this result is twofold . First , it gives a promise that seemingly complicated dynamic optimization problems faced by consumers can be solved by relatively simple algorithms that do not require fixed point computation or functional optimization . This means that the policy gradient-style methods have an important behavioral interpretation . As a result , consumer behavior following policy gradient can serve as a behavioral assumption for estimating consumer preferences from data which is more natural for consumer choice settings than other assumptions that have been used in the past for estimation of preferences ( e.g . -regret learning in Nekipelov et al . ( 2015 ) ) . Second , more importantly , our result showing fast convergence of the policy gradient algorithm makes it an attractive alternative to the search for the functional fixed point in this class of problems . While the goal of the Econometric analysis of the data from dynamically optimizing consumers is to estimate consumer preferences by maximizing the likelihood function , it requires to sequentially solve the dynamic optimization problem for each value of utility parameters along the parameter search path . Existing work in Economics prescribes to use fixed point iterations for the value function to solve the dynamic optimization problem ( see Rust ( 1987 ) , Aguirregabiria and Mira ( 2007 ) ) . The replacement of the fixed point iterations with the policy gradient method significantly speeds up the maximization of the likelihood function . This makes the policy gradient algorithm our recommended approach for use in Econometric analysis , and establishes practical relevance of many newer reinforcement learning algorithms from behavioral perspective for social sciences . 2 PRELIMINARIES . In this section , we introduce the concepts of the Markov decision process ( MDP ) with choice-specific payoff heterogeneity , the conditional choice probability ( CCP ) representation and the policy gradient algorithm . 2.1 MARKOV DECISION PROCESS . A discrete-time Markov decision process ( MDP ) with choice-specific heterogeneity is defined as a 5-tuple 〈S , A , r , , P , β〉 , where S is compact convex state space with diam ( S ) ≤ S̄ < ∞ , A is the set of actions , r : S×A → R+ is the reward function , such that r ( s , a ) is the immediate non-negative reward for the state-action pair ( s , a ) , are independent random variables , P is a Markov transition model where where p ( s′|s , a ) defines the transition density between state s and s′ under action a , and β ∈ [ 0 , 1 ) is the discount factor for future payoff . We assume that random variables are observed by the optimizing agent and not recorded in the data . These variables reflect idiosyncratic differences in preferences of different optimizing agents over choices . In the following discussion we refer to these variables as “ random choice-specific shocks . '' In each period t = 1 , 2 , . . . , ∞ , the nature realizes the current state st based on the Markov transition P given the state-action pair ( st−1 , at−1 ) in the previous period t− 1 , and the choice-specific shocks t = { t , a } a∈A drawn i.i.d . from distribution . The optimizing agent chooses an action a ∈ A , and her current period payoff is sum of the immediate reward and the choice-specific shock , i.e. , r ( s , a ) + t , a . Given initial state s1 , the agent ’ s long-term payoff is E 1 , s2 , 2 , ... [ ∑∞ t=1 β t−1r ( st , at ) + t , at ] . This expression makes it clear that random shocks play a crucial role in this model by allowing us to define the ex ante value function of the optimizing agent which reflects the expected reward from agent ’ s choices before the agent observes realization of t. When the distribution of shocks is sufficiently smooth ( differentiable ) , the corresponding ex ante value function is smooth ( differentiable ) as well . This allows us to characterize the impact of agent ’ s policy on the expected value by considering functional derivatives of the value function with respect to the policy . In the remainder of the paper , we rely on the following assumptions . Assumption 2.1 . The state space S is compact in R and the action spaceA is binary , i.e. , A = { 0 , 1 } . Assumption 2.2 . For all states s , the immediate reward r ( s , 0 ) for the state-action pair ( s , 0 ) is zero i.e. , r ( s , 0 ) = 0 , and the immediate reward r ( s , 1 ) for the state-action pair ( s , 1 ) is bounded between [ Rmin , Rmax ] . Assumption 2.3 . Choice-specific shocks are Type I Extreme Value random variables with location parameter 0 ( cf . Hotz and Miller , 1993 ) which are independent over choices and time periods . Assumption 2.1 , 2.2 , 2.3 are present in most of the papers on dynamic decision-making in economics , marketing and finance , ( e.g . Dubé et al. , 2002 ; Aguirregabiria and Mira , 2010 ; Arcidiacono and Miller , 2011 ; Aguirregabiria and Magesan , 2016 ; Müller and Reich , 2018 ) The policy and the value function A stationary Markov policy is a function σ : S × RA → A which maps the current state s and choice-specific shock to an action . In our further discussion we will show that there is a natural more restricted definition of the set of all feasible policies in this model . Given any stationary Markov policy σ , the value function Vσ : S → R is a mapping from the initial state to the long-term payoff under policy σ , i.e. , Vσ ( s1 ) = E 1 , s2 , 2 , ... [ ∞∑ t=1 βt−1 { r ( st , σ ( st , t ) ) + t , σ ( st , t ) } ] . Since the reward is non-negative and bounded , and the discount β ∈ [ 0 , 1 ) , value function Vσ is well-defined and the optimal policy σ̃ ( i.e. , Vσ̃ ( s ) ≥ Vσ ( s ) for all policies σ and states s ) exists . Furthermore , the following Bellman equation holds Vσ ( s ) = E [ r ( s , σ ( s , ) ) + σ ( s , ) + β Es′ [ Vσ ( s′ ) |s , σ ( s , ) ] ] for all policies σ ( 1 ) 2.2 CONDITIONAL CHOICE PROBABILITY REPRESENTATION . Based on the Bellman equation ( 1 ) evaluated at the optimal policy , the optimal Conditional Choice Probability δ̃ ( a|s ) ( i.e. , the probability of choosing action a given state s in the optimal policy σ̃ ) can be defined as δ̃ ( a|s ) = E [ 1 { r ( s , a ) + a + β Es′ [ Vσ̃ ( s′ ) |s , a ] ≥ r ( s , a′ ) + a′ + β Es′ [ Vσ̃ ( s′ ) |s , a′ ] , ∀a′ } ] The optimal policy σ̃ can , therefore , be equivalently characterized by threshold function π̃ ( s , a ) = r ( s , a ) + β Es′ [ Vσ̃ ( s′ ) |s , a ] , such that the optimizing agent chooses action a† which maximizes the sum of the threshold and the choice-specific shock , i.e. , a† = argmaxa { π̃ ( s , a ) + a } . Similarly , all non-optimal policies can be characterized by the corresponding threshold functions denoted π . Under Assumption 2.3 the conditional choice probability δ can be explicitly expressed in terms of the respective threshold π as ( cf . Rust , 1996 ) δ ( a|s ) = exp ( π ( s , a ) ) / ( ∑ a′∈A exp ( π ( s , a ′ ) ) ) . We note that this expression induces a one-to-one mapping from the thresholds to the conditional choice probabilities . Therefore , all policies are fully characterized by their respective conditional choice probabilities . For notational simplicity , since we consider the binary action space A = { 0 , 1 } , and the reward r ( s , 0 ) is normalized to 0 we denote the immediate reward r ( s , 1 ) as r ( s ) ; denote the conditional choice probability δ ( 0|s ) as δ ( s ) ; and denote π ( s , 1 ) as π ( s ) . In the subsequent discussion given that the characterization of policy σ via its threshold is equivalent to its characterization by conditional choice probability δ , we interchangeably refer to δ as the “ policy . '' Then we rewrite the Bellman equation for a given policy δ as Vδ ( s ) = ( 1− δ ( s ) ) r ( s ) − δ ( s ) log ( δ ( s ) ) − ( 1− δ ( s ) ) log ( 1− δ ( s ) ) + β E , s′ [ Vδ ( s ) ( s ′ ) ∣∣∣s ] ( 2 ) Now we make two additional assumptions that are compatible with standard assumptions in the Econometrics literature . Assumption 2.4 . For all states s ∈ S , the conditional distribution of the next period Markov state p ( ·|s , 1 ) first-order stochastically dominates distribution p ( ·|s , 0 ) , i.e. , for all ŝ ∈ S , Prs′ [ s′ ≤ ŝ|s , 1 ] ≤ Prs′ [ s′ ≤ ŝ|s , 0 ] . Assumption 2.5 . Under the optimal policy δ̃ , the value function is non-decreasing in states , i.e. , Vδ̃ ( s ) ≤ Vδ̃ ( s′ ) for all s , s′ ∈ S s.t . s < s′ . Consider a myopic policy δ̄ ( s ) = ( exp ( r ( s ) ) + 1 ) −1 which uses threshold π̄ ( s ) = r ( s ) . This policy corresponds to agent optimizing the immediate reward without considering how current actions impact future rewards . Under Assumption 2.4 and Assumption 2.5 , the threshold for optimal policy is at least the threshold of myopic policy , i.e. , π̃ ( s ) ≥ π̄ ( s ) . Hence , Lemma 2.1 holds . Lemma 2.1 . The optimal policy δ̃ chooses action 0 with weakly lower probability than the myopic policy δ̄ in all states s ∈ S , i.e. , δ̃ ( s ) ≤ δ̄ ( s ) .

Summary: This paper considers reinforcement learning for discrete choice models with unobserved heterogeneity, which is useful for analyzing dynamic Economic behavior.  Random choice-specific shocks in reward is accommodated, which are only observed by the agent but not recorded in the data. Existing optimization approaches rely on finding a functional fixed point, which is computationally expensive.  The main contribution of the paper lies in formulating discrete choice models into an MDP, and showing that the value function is concave with respect to the policy (represented by conditional choice probability).  So policy gradient algorithm can provably converge to the global optimal.  Conditions on the parameters for global concavity are identified and rates of convergences are established.  Finally, significant advantages in computation were demonstrated on the data from Rust (1987), compared with “nested fixed point” algorithms that is commonly used in Econometrics.