Bellman Equations

Bellman Equations are a set of equations used in dynamic programming and reinforcement learning to recursively calculate the value function for a Markov Decision Process (MDP).
The value function gives the expected cumulative reward that an agent can achieve from a particular state by following a particular policy. The Bellman equations express the value function in terms of the expected immediate reward and the expected value of the next state.
There are two versions of the Bellman equations: the Bellman expectation equation and the Bellman optimality equation.
The Bellman expectation equation expresses the value function V(s) for a state s in terms of the expected immediate reward R(s,a) and the expected value of the next state V(s'):
V(s) = E[R(s,a) + γV(s')],
where a is an action, s' is the next state, and γ is a discount factor that determines the importance of future rewards.
The Bellman optimality equation expresses the optimal value function V*(s) for a state s in terms of the maximum expected immediate reward and the expected value of the next state under the optimal policy:
V*(s) = max_a {E[R(s,a) + γV*(s')]},
where max_a is the maximum over all possible actions a.
The Bellman equations are used in algorithms such as value iteration and policy iteration to iteratively calculate the value function and find the optimal policy for an MDP.
Category:Mathematics