Object-based Reinforcement Learning
By: Luca Anzalone
Tags: Graph Neural Networks, Deep Learning, Reinforcement Learning, Unsupervised Learning, Computer Vision

An Informal Introduction to RL

Reinforcement learning provides a set of tools to train an agent to take optimal actions within an environment (a real or simulated world) by trial and error (i.e. executing an action and then experiencing its effects), guided only by rewards. A reward is a scalar feedback signal that tells the agent how good the taken actions are. The agent collects rewards in order to learn which actions are the best to execute in a given situation or state.

RL lies at the intersection of many well-known fields. [Source]

In brief, an agent learns how to solve a task that is decision-making problem. Atask is formally defined as a Markov Decision Process (MDP) described in terms of an environment and a reward function. The environment determines the dynamics of the world, i.e. what happens next when an action is executed in a particular situation (named a state), whilst the reward function tells the agent how much it has earned (or lost) by executing an action, given the current situation of the world (its state).

At each timestep, the agent alters the environment’s state by executing an action. The effect of the action is provided by the environment as the next state and reward. The agent uses the reward or both feedback to improve its behaviour. [Source]

The solution to a MDP is a policy 𝛑 that determines the agent’s behaviour. The optimal policy is one that “maximises the (discounted) sum of (expected) rewards”, also known as the RL objective:

The RL objective. The optimal agent is the one that achieves the highest possible reward. When actions are sampled from a stochastic policy, we weight the cumulative rewards by actions probabilities.

Notice that this is a generic formula because the policy can be represented in many ways: as a table, linear or shallow combination of hand-engineered features, or as a deep neural network. Moreover, the policy can be stochastic or deterministic. A stochastic policy is a probability distribution over actions, in this case the RL objective maximises the expected sum of rewards. Conversely, a deterministic policy is a deterministic function of the state s in which only one action has nonzero probability.

Deep RL favours deep neural network policies as they are both expressive (we can easily increase the model capacity) and computationally efficient in space and time, because computing the forward pass of a neural network is usually cheap. Expressed in these terms, the RL objective aims to find the policy’s parameters θ that yields the highest possible (discounted) sum of cumulative rewards:

The RL objective when the policy is a neural network with parameters θ. Note that the expectation is over trajectories 𝜏, i.e. pairs of states and actions (s, a), obtained by interacting with the environment and acting according to a policy with parameters θ.

Notice that in both objectives there is a discount factor 𝛄, i.e. a real number between 0 and 1 that “scales” the future rewards. The discount factor is mandatory when the time-horizon T is infinite because it prevents the sum of rewards to diverge. In general, we always want to discount the rewards commonly by a factor of 0.9, 0.95, or 0.99.

In order to compute the optimal policy we need a way to compare policies. The intuition is that a better policy reaches “more valuable” states as well as picking out “more rewarding” actions:

The state-value function V(s) , quantifies how much reward in average we can get by taking optimal actions from state s. If s is the initial state its value measures how much reward the current policy will get in average from the beginning. Thus, the optimal policy is the one that achieves the highest value. Similarly,

, quantifies how much reward in average we can get by taking optimal actions from state s. If s is the initial state its value measures how much reward the current policy will get in average from the beginning. Thus, the optimal policy is the one that achieves the highest value. Similarly, The action-value function Q(s, a), says how good is to take an action a when being in a state s. The action-value function is useful to understand which specific action is best to take while being in state s. We know that the optimal policy executes only optimal actions, i.e. actions whose Q-value are maximal when facing a given state.

These two functions are the basic ingredients of the Bellman Optimality Equation that finds some optimal policy. For small problems the equation can be easily solved with dynamic programming. Instead, for high-dimensional and continuous MDPs we often use neural networks to approximate V and/or Q used by model-free methods (e.g. Q-Learning, TD-Learning, Policy Gradients, Actor-Critic) and model-based methods (e.g. Dyna) to find optimal policies for the given task.