The Gambler's Problem and Beyond

We analyze the Gambler ’ s problem , a simple reinforcement learning problem where the gambler has the chance to double or lose their bets until the target is reached . This is an early example introduced in the reinforcement learning textbook by Sutton & Barto ( 2018 ) , where they mention an interesting pattern of the optimal value function with high-frequency components and repeating nonsmooth points . It is however without further investigation . We provide the exact formula for the optimal value function for both the discrete and the continuous cases . Though simple as it might seem , the value function is pathological : fractal , self-similar , derivative taking either zero or infinity , not smooth on any interval , and not written as elementary functions . It is in fact one of the generalized Cantor functions , where it holds a complexity that has been uncharted thus far . Our analyses could lead insights into improving value function approximation , gradientbased algorithms , and Q-learning , in real applications and implementations . 1 INTRODUCTION . We analytically investigate a deceptively simple problem , the Gambler ’ s problem , introduced in the reinforcement learning textbook by Sutton & Barto ( 2018 ) , on Example 4.3 , Chapter 4 , page 84 . The problem setting is natural and simple enough that little discussion was given in the book apart from an algorithmic solution by value iteration . A close inspection will however show that the problem , as a representative of the entire family of Markov decision processes ( MDP ) , involves a level of complexity and curiosity uncharted in years of reinforcement learning research . The problem discusses a gambler ’ s casino game , where they places multiple rounds of betting . The gambler gains the bet amount if they win a round or loses the bet if they lose the round . The probability of losing each round is p ≥ 0.5 , independently . The game ends when the gambler ’ s capital reaches either their goal of N or 0 . In each round , the gambler must decide what portion of the capital to stake . In the discrete setting this bet amount must be an integer , but it can be a real number in the continuous setting . To formulate it as an MDP , we denote state s be the current capital and action a the bet amount . The reward is +1 when the state reaches s = N , and zero otherwise . Our goal is to solve the optimal value function of the problem . We first give the solution to the discrete Gambler ’ s problem . Denote N as the target capital , n as the current capital ( which is the state in the discrete setting ) , p > 0.5 as the probability of losing a bet , and γ as the discount factor . The special case of N = 100 , γ = 1 corresponds to the original setting in Sutton and Barto ’ s book . Proposition 1 . Let 0 ≤ γ ≤ 1 and p > 0.5 . The optimal value function z ( n ) is v ( n/N ) in the discrete setting of the Gambler ’ s problem , where v ( · ) is the optimal value function under the continuous case defined in Theorem 12 . The above statement amounts the discrete problem to the continuous problem by a uniform discretization . The rest of the discussion will be on the more general continuous setting . In the setting , the target capital is 1 , the state space is [ 0 , 1 ] , and the action space is 0 < a ≤ min { s , 1− s } at state s , meaning that the bet can be any fraction of the current capital as long as the capital after winning does not exceed 1 . We state the optimal value function below and intuitive description of the value function later in this section . Theorem 12 . Let 0 ≤ γ ≤ 1 and p > 0.5 . Under the continuous setting of the Gambler ’ s problem , the optimal state-value function is v ( 1 ) = 1 , and v ( s ) = ∞∑ i=1 ( 1− p ) γibi i−1∏ j=1 ( ( 1− p ) + ( 2p− 1 ) bj ) ( 1 ) for 0 ≤ s < 1 , where s = 0.b1b2 . . . b ` . . . ( 2 ) is the binary representation of the state s. Next , we solve the Bellman equation of the continuous Gambler ’ s problem . In the strictly discounted setting 0 ≤ γ < 1 , the solution of the Bellman equation f ( 0 ) = 0 , f ( 1 ) = 1 , f ( s ) = max 0 < a≤min { s,1−s } ( 1− p ) γf ( s+ a ) + pγf ( s− a ) is f ( s ) = v ( s ) the optimal value function ( Proposition 21 ) . This uniqueness does not hold in general . If the rewards are not discounted , the solution of the Bellman equation is either the optimal value function , or a constant function larger than 1 . Theorem 22 . Let γ = 1 , p > 0.5 , and f ( · ) be a real function on [ 0 , 1 ] . f ( s ) solves the Bellman equation if and only if either • f ( s ) is v ( s ) defined in Theorem 12 , or • f ( 0 ) = 0 , f ( 1 ) = 1 , and f ( s ) = C for all 0 < s < 1 , for some constant C ≥ 1 . Under the corner case of γ = 1 , p = 0.5 ( where the gambler do not lose capital in bets in expectation ) , the problem involves the midpoint concavity ( Sierpiński , 1920a ; b ) and Cauchy ’ s functional equation . The measurable function that solves the Bellman equation is uniquely f ( s ) = C ′s + B′ on s ∈ ( 0 , 1 ) , for some constants C ′+B′ ≥ 1 . Additionally , under Axiom of Choice , f ( s ) can also be some non-constructive , non Lebesgue measurable function described by a Hamel basis ( Theorem 27 and its lemmas ) . Though the description of the Gambler ’ s problem seems natural and simple , Theorem 12 shows that its simpleness is deceptive . The optimal value function is fractal and self-similar , and non-rectifiable ( see Corollary 14 and Lemma 8 ) . It is thus not smooth on any interval , which can be unexpected when a significant line of reinforcement learning studies is based on function approximation like discretization and neural networks . The value function ( 1 ) can neither be simplified into a formula of elementary functions , which introduces difficulties in understanding it . The function is monotonically increasing with v ( 0 ) = 0 and v ( 1 ) = 1 , but its derivative is 0 almost everywhere , which is counterintuitive . This is known as singularity , a famous pathological property of functions . v ( s ) is continuous almost everywhere but not absolutely continuous . Also when γ is strictly smaller than 1 , it is discontinuous on a dense and compact set of infinitely many points . These properties indicate that assumptions like smoothness , continuity , and approximability are not satisfied in this problem . In general , it is reasonable to doubt if these assumptions can be imposed in reinforcement learning . To better understand the pathology of v ( s ) , we analogize it to the Cantor function , which is well known in analysis as a counterexample of many seemingly true statements ( Dovgoshey et al. , 2006 ) . In fact , v ( s ) is a generalized Cantor function , where the above descriptions are true for both v ( s ) and the Cantor function . Intuitive description of v ( s ) . All the statements above require the definition of v ( s ) . In fact , in this paper , v ( s ) is important enough such that its definition will not change with the context . The function can not be written as a combination of elementary functions . Nevertheless , we give an intuitive way to understand the function for the original , undiscounted problem . The function can be regarded as generated by the following iterative process . First we fix v ( 0 ) = 0 and v ( 1 ) = 1 , and compute v ( 1 2 ) = pv ( 0 ) + ( 1− p ) v ( 1 ) = ( 1− p ) . Here , v ( 12 ) is the weighted average of the two “ neighbors ” v ( 0 ) and v ( 1 ) that have been already evaluated . Further , the same operation applies to v ( 14 ) and v ( 3 4 ) , where v ( 1 4 ) = pv ( 0 ) + ( 1 − p ) v ( 12 ) = ( 1 − p ) 2 and v ( 34 ) = pv ( 1 2 ) + ( 1 − p ) v ( 1 ) = ( 1 − p ) + p ( 1 − p ) , and so forth to v ( 18 ) , v ( 3 8 ) , etc . This process evaluates v ( s ) on the dense and compact set ⋃ ` ≥1G ` of the dyadic rationals , where G ` = { k2− ` | k ∈ { 1 , . . . , 2 ` − 1 } } . With the fact that v ( s ) is monotonically strictly increasing , this dyadic rationals determines the function v ( s ) uniquely . This iterative process can also be explained from the analytical formula of v ( s ) . Starting with the first bit , a bit of 0 will not change the value , while a bit of 1 will add ( 1−p ) ∏i−1 j=1 ( ( 1−p ) + ( 2p−1 ) bj ) to the value . This term can also be written as ( 1 − p ) ( ( 1 − p ) # 0 bits · p # 1 bits ) , where the number of bits is counted over all previous bits . The value ( 1 − p ) # 0 bits · p # 1 bits decides the gap between two neighbor existing points in the above process , when we insert a new point in the middle . This insertion corresponds to the iteration on G ` over ` . We provide high resolution plots of z ( n ) , N = 100 and v ( s ) in Figure 1 and Figure 2 , respectively . The non-smoothness and the self-similar fractal patterns can be clearly observed from the figures . Also , v ( s ) is continuous when γ = 1 while v ( s ) is not continuous on infinitely many points when γ < 1 . In fact , when γ < 1 , the function is discontinuous on the dyadic rationals ⋃ ` ≥1G ` while continuous on its complement , as we will rigorously show later . Self-similarity . The function v ( s ) on [ s̄ , s̄ + 2− ` ] for any s̄ = 0.b1b2 . . . b ` ( 2 ) , ` ≥ 1 is self-similar to the function itself on [ 0 , 1 ] . Let s = 0.b1b2 . . . b ` . . . ( 2 ) ∈ [ s̄ , s̄+ 2− ` ] , this can be observed by v ( s ) = ∞∑ i=1 ( 1− p ) γibi i−1∏ j=1 ( ( 1− p ) + ( 2p− 1 ) bj ) = ∑̀ i=1 ( 1− p ) γibi i−1∏ j=1 ( ( 1− p ) + ( 2p− 1 ) bj ) + γ ` ( ∏̀ j=1 ( ( 1− p ) + ( 2p− 1 ) bj ) ) × ∞∑ i=1 ( 1− p ) γib ` +i i−1∏ j=1 ( ( 1− p ) + ( 2p− 1 ) b ` +j ) = v ( s̄ ) + γ ` ∏̀ j=1 ( ( 1− p ) + ( 2p− 1 ) bj ) · v ( 2 ` ( s− s̄ ) ) . ( 2 ) The self-similarity can be compared with the Cantor function ( Dovgoshey et al. , 2006 ; Mandelbrot , 1985 ) , which uses the ternary of s instead in the formula . The Cantor function is self-similar to itself on [ s̄ , s̄ + 3− ` ] , when s̄ = 0.b1b2 . . . b ` ( 3 ) and b ` 6= 1 . Both v ( s ) and the Cantor function can be uniquely described by their self-similarity , the monotonicity , and the boundary conditions . Optimal policies . It is immediate by Theorem 12 and Lemma 8 that π ( s ) = min { s , 1− s } is one of the ( Blackwell ) optimal policies . Here , Blackwell optimality is defined as the uniform optimality under any 0 ≤ γ ≤ 1 . This policy agrees with the intuition that under a game that is in favor of the casino ( p > 0.5 ) , the gambler desires to bet the maximum to finish the game in as little cumulative bet as possible . In fact , the probability of reaching the target is the expected amount of capital by the end of the game , which is negative linear to the cumulative bet . The optimality is not unique though , for example , π ( 1532 ) = 1 32 is also optimal ( for any γ ) . Under γ = 1 the original , undiscounted setting , small amount of bets can also be optimal . Namely , when s can be written in finite many bits s = b1b2 . . . b ` ( 2 ) in binary ( assume b ` = 1 ) , π ( s ) = 2−l is also an optimal policy . This policy is by repeatedly rounding the capital to carryover the bits , keeping the game to be within at most ` rounds of bets .

Summary: The paper gave a very detailed analysis of the value function of the gamblers problem. In this problem, the agent decides how much to bet in each step, doubling or losing the chosen amount. There is a chance of loss of greater than 0.5 and as a result the agent would like to minimize the number of steps. While one optimal policy is extremely simple (just bet the maximum), this paper shows that the value function is deceptively complicated due to its fractal structure. In particular, the value function is continuous only when there is no discounting and does not have well behaved derivatives. The exposition is clear and well structured. The author argues such behaviors have consequences for valued function based RL such as value iterations or Q-learning, since one of the simplest RL problem has a weird value function.