Mathematical Analysis of Reinforcement Learning — Bellman Optimality Equation | by Vaibhav Kumar | Towards Data Science

Reinforcement learning has achieved remarkable results in playing games like StarCraft (AlphaStar) and Go (AlphaGO). At the core of all these successful projects lies — The Bellman optimality equation for Markov decision processes (MDPs).
The Bellman optimality equation is a recursive equation that can be solved using dynamic programming (DP) algorithms to find the optimal value function and the optimal policy. In this article, I will try to explain why the Bellman optimality equation can solve every MDP by providing an optimal policy and perform an easy (hopefully) mathematical analysis of the same.
S is state space.
V is the value function.
V* is the optimal value function.
V(s) is the value of state s.
π is a policy.
π* is the optimal policy.
π(s) returns the action for state s under the policy π.
P is the transition probability matrix.
A is the action space.
Despite my best efforts, the analysis is going to be fairly rigorous and I assume the readers to be familiar with the following perquisites:
Markov decision process (MDP)Bellman Equation and how it’s solved using iterative methods.Basics of RL — Value functions, rewards, policies, discount factor etc.Linear algebraVector calculus
Markov decision process (MDP)
Bellman Equation and how it’s solved using iterative methods.
Basics of RL — Value functions, rewards, policies, discount factor etc.
Linear algebra
Vector calculus
If you have studied RL and MDPs, you must’ve encountered the claim: “For each MDP, there is always at least one policy that is better than or equal to all other policies.” This occurs both in Sutton and Barto’s book as well as in the lecture series by David Silver. Reading/Hearing this makes the claim pretty intuitive, however, I had to dig in deeper and understand this in a more concrete manner. Therefore, in this article, I will mathematically prove the following theorem:
Theorem: For any finite MDP, there exists an optimal policy π* such that it is better than or equal to every other possible policy π.
Before finding the best policy, we need to understand the ordering of policies. When is one policy (π1) considered better than the other (π2)?
If the value of a state derived using π1 is better than or equal the value of a state derived using π2 for every state in the environment, then the policy π1 is said to be better than policy π2. Mathematically, this can be written as follows:
Now that we know how to compare policies, we need to prove that there always exists a policy that is better than all other policies.
We are going to prove this using the Banach fixed point theorem by showing that the Bellman optimality operator is a contraction over a complete metric space of real numbers with metric L-infinity norm. For this, we will first discuss the fixed point problem and complete metric spaces with respect to the Cauchy sequence.
The above paragraph sounds very intimidating but its actually going to be pretty easy and intuitive once we get past the basic terminologies. We will be discussing everything that is in bold in the above paragraph. Let’s follow a bottom-up approach and learn each concept and conquer our fears:
I am sure most of us are familiar with the problem of finding the roots of an equation. We solve for x such that a function f(x) = 0. However, in a fixed point problem, we solve for x such that f(x) = x. As the name suggests, x is a fixed point, it does not change even on the application of the function. A fixed point problem can be converted into a problem of finding roots by forming another function g(x) = f(x)-x = 0. In fact, even root finding problems can also be converted back to fixed point problems. However, it’s really easy to solve fixed point problems (for special cases) and that is what makes them incredibly interesting and useful (sans the computational overhead).
To solve a fixed point problem, choose a random starting value of x and repeatedly apply f(x) infinite times. If the function is convergent and you’re lucky, you will find the solution.
Mathematically, it’s pretty simple, lets first describe a notation:
Now, if the function is convergent then it must converge to some value, say, x*. This value, x* is indeed the solution to the fixed point problem as shown below:
Let's choose some arbitrary value x0 and apply the function f(.) on x0 for infinite times to get x*, and then use that to solve the fixed point problem:
The intuition behind this is pretty simple, if a function has converged at some point then the value of this function at that convergent point will be the convergent point itself. Therefore, the convergent point is the fixed point itself.
This can also be observed empirically through the notebook below:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import warnings
warnings.simplefilter("ignore")

def f(x):
    return np.sin(x)/x

def plot(lower=-1, upper=200):
    ys, xs = [], []
    plt.figure(figsize=(6, 4))
    for x in range(lower, upper):
        try:
            ys.append(f(x))
            xs.append(x)
        except:
            continue
    plt.xlabel("x")        
    plt.ylabel("f(x)")        
    plt.plot(xs, ys)
    plt.show()

plot()

The function above is clearly converging
Note: Instead of infinity, we just apply it 10000 times
def fixed_point(INF = 10000):
    x0 = np.random.randn()
    for _ in range(INF):
        x0 = f(x0)
    return x0    

solution = fixed_point()
print(solution)

0.8767262153950625

def verify(solution):
    print(f"Values of function: f{f(solution)} | Value of x: {solution}")
    if f(solution) == solution:
        return True

verify(solution)

Values of function: f0.8767262153950625 | Value of x: 0.8767262153950625

True
A metric space is simply a set with a metric defined to measure the distance between any two elements of the set. For example, the Euclidean space is a metric space with distance defined as the Euclidean distance in the set of real numbers. Therefore, a metric space M is represented as (X, d) where X is the set and d is some sort of metric. The metric d must satisfy the following properties:
Identify d(x,x) = 0Non-Negativity d(x, y) >0Symmetry: d(x,y) = d(y,x)Triangular inequality: d(x,z) ≤ d(x,y)+d(y,z)
Identify d(x,x) = 0
Non-Negativity d(x, y) >0
Symmetry: d(x,y) = d(y,x)
Triangular inequality: d(x,z) ≤ d(x,y)+d(y,z)
For a metric space (X, d) the sequence of elements of the set X, (x1, x2, x3.... xn) is a Cauchy sequence if, for every positive real number ε, there exists an integer N such that the following equation holds:
The mathematical explanation here isn’t very intuitive and needlessly complex. In simple words, a sequence of elements of a metric space is Cauchy if this sequence converges at some point (the distance between them becomes constant). Another great explanation for this is given by this website and goes as follows: “for any small distance, there is a certain index past which any two terms are within that distance of each other, which captures the intuitive idea of the terms becoming close.” The idea of “terms become close” is the basic intuition behind convergence or the limit of the series.
A metric space (X, d) is complete if every possible Cauchy sequence of the elements in the set X converges to an element that also belongs to the set X. That is to say, the convergent limit of every Cauchy sequence of the elements of set lies in the set itself. Which is why it’s called “complete”.
A function (or operator or mapping) defined on the elements of the metric space (X, d) is a contraction (or contractor) if there exists some constant γ∈[0,1) such that for any two elements of the metric space x1 and x2, the following condition holds:
This means that after applying the mapping f(.) on the elements x1 and x2, they got closer to each other by at least a factor γ. Also, the smallest value of such a constant γ is called the Lipschitz constant (this is an important constant for generative adversarial networks). Also, if γ=1, the mapping is no more a contraction but rather a short mapping. Intuitively, it can be observed that the sequential values of the elements are getting closer after applying the contraction mapping.
This is the heart and soul of our proof. Informally, this theorem says that for a complete metric space, the application of a contractor on the elements of the set, again and again, would eventually get us to an optimal, unique value. We know:
Contractors bring the elements of the set together.Applying this contractor, again and again, would get us a sequence. (Cauchy?)The Cauchy sequences in a complete metric space always converge to a value that is part of the metric space.
Contractors bring the elements of the set together.
Applying this contractor, again and again, would get us a sequence. (Cauchy?)
The Cauchy sequences in a complete metric space always converge to a value that is part of the metric space.
Formally, this theorem can be formulated as:
Theorem: Let (X, d) be a complete metric space and a function f: X->X be a contractor then, f has a unique fixed point x*∈ X (i.e. f(x*)=x*) such that the sequence f(f(f(...f(x)))) converges to x*.
Now, to prove this mathematically, we need to prove both the uniqueness and existence of x*.
Uniqueness: We will prove this by contradiction. Let us assume that the convergence value is not unique and x1* and x2* are two values at which the sequence of contractor converges then, we will have:
Uniqueness: We will prove this by contradiction. Let us assume that the convergence value is not unique and x1* and x2* are two values at which the sequence of contractor converges then, we will have:
Also, note that f is a contractor so it must hold the following property:
Now since γ∈[0,1), it’s impossible to satisfy both equation 1 and 2 simultaneously. Therefore our assumption must be wrong. Hence, by contradiction, x* must be unique.
2. Existence: Now that we’ve proved that x* is unique, we need to prove that x* exists. Let (x1, x2, x3, .... xn) be the sequence formed by repeatedly applying the contractor.
If we assume that the sequence (x1, x2, x3, .... xn) is Cauchy, we know for certain that this sequence will converge to some point, say, x*. Also, since the metric space is complete, this convergent point,x* will belong to the metric space (X,d). Now, we just need to prove that this sequence is Cauchy. We will do this by taking two elements of the set xn and xm such that m>>n and, m is very large, then by repeatedly applying the triangular inequality property of the metric d, we have:
Now, since f is a contractor, we know that:
We can further reduce d(xm, xn) as follows:
Now, by choosing n to be sufficiently large, we can make the RHS of the above equation less than any positive real number ε. Hence, the sequence (x1, x2, x3, .... xn) is Cauchy and an optimal x*∈X exists. This concludes the proof of the Banach fixed point theorem.
For the value function, V(s) we define a new operator, the optimal Bellman operator,B which takes in a value function and returns another value function. We define this operator as follows:
It can easily be observed, B is a recursive operator. Therefore, this will generate a sequence of value functions. If we can show that B is indeed a contractor for some metric space (X,d) then by the Banach fixed point theorem, we can conclude that the repeated application of the optimal Bellman operator will eventually give a unique optimal value function using which an optimal (best) policy can be derived. Therefore, all our work now reduces to proving that B is a contractor. First, let's define the metric space as follows:
Metric space (X,d): The set X is the set of real numbers defined as follows:
For the metric, we use the L-infinity norm defined as follows:
According to this metric, the distance between the two value functions will be equal to the highest element-wise absolute difference between the two. Also, for finite MDPs with finite rewards, the value functions will always stay in the real space. It is impossible for the value function to not be in the real space, therefore, this finite space will always be complete.
Theorem: Bellman operator B is a contraction mapping in the finite space (R, L-infinity)
Proof: Let V1 and V2 be two value functions. Then:
In the second step above, we introduce inequality by replacing a’ by a for the second value function. This is because by replacing its optimal action a’ by some other action a, we have reduced its overall value thereby introducing an inequality.
In the fourth step, we remove the L-infinity norm by taking the max over s’ (recall the definition of L-infinity with respect to value functions in our setting)
In the final step, we remove the sigma because the sum of probabilities is always 1.
Finally, for the Bellman optimality equation, since γ∈[0,1) (let's ignore the possibility of γ=1 for now), the Bellman operator is, therefore, a contractor.
Now we know:
(R, L-infinity) is a complete metric spaceBellman operator B is a contractor
(R, L-infinity) is a complete metric space
Bellman operator B is a contractor
Hence, by the Banach fixed point theorem, we conclude that there exists a unique optimal value function V* for every MDP. Using this V*, we can derive the optimal policy π*.
Hence proved, for any finite MDP, there exists an optimal policy π* such that it is better than or equal to every other possible policy π.
Now, how to find this optimal policy and value function? One way is to just repeatedly apply the Bellman operator to a random initial value function to get the optimal function. But, this is computationally very expensive and often downright infeasible. Therefore, we use iterative methods like value and policy iteration or temporal difference methods like Q-Learning or SARSA. For more on this, please refer to my blog Reinforcement learning: Temporal-Difference, SARSA, Q-Learning & Expected SARSA in python or just view these algorithms on my Github: RL_from_scratch.
We learned some basic mathematical tools like metric spaces, complete metric spaces, Cauchy sequences, contraction mapping and the Banach fixed point theorem. Building upon all of this, we mathematically proved the unique optimality of the Bellman optimality equation for every MDP.
I used https://latex.codecogs.com/eqneditor/editor.php to generate images for LaTeX. (I wish medium allowed LaTeX)
Wikipedia for providing all the formal definitions and theorems.
The CMU 15–781 course
The notes and video by Dr Daniel Murfet
The free NPTEL course by Dr Balaraman Ravindran and IIT-Madras
I have tried to explain everything in as simple words as possible. If I was unclear or if I made some mistakes, please let me know in the responses. Also, there might be some mistakes in the latex equations. Please let me know if you find them. Last but not least, thanks for reading!