Temporal-Difference (TD) Learning
By: Sagi Shaier
Tags: Rl Tutorial Series, Data Science, Reinforcement Learning Tutorial Series, Machine Learning, Reinforcement Learning

Chapter 6 — Temporal-Difference (TD) Learning

Key concepts in this chapter:

- TD learning

- SARSA

- Q Learning

- Expected SARSA

- Double Q Learning

The key is behind TD learning is to improve the way we do model-free learning. To do this, it combines the ideas from Monte Carlo and dynamic programming (DP):

Similarly to Monte Carlo methods, TD methods can work in a model-free learning. Similarly to DP, TD methods update the value estimates based partially on other estimates, without having to go through the entire episode (they bootstrap).

TD Prediction

Monte Carlo methods update the value of a state only after the reward is returned (only at the end of the episode Gt is known).

MC update rule [ref]

Whereas TD method only need to wait until the next time step to create an update using the observed reward Rt+1 and the estimate V (St+1).

The simplest TD method is using the following update:

TD update rule [ref]

as soon as it move to state St+1 and receiving reward Rt+1.

From the above equations we can see that the target for the Monte Carlo update is Gt, whereas the target for the TD update is Rt+1 + V (St+1).

We call the part in the brackets TD error, as it measure the difference between V(St) — the estimated value, and Rt+1 + V (St+1) — the better estimate.

The above TD update is called TD(0) or one-step TD. That is, because it updates the value following the next state (St+1).

It’s a special case of TD(λ)

Since the above TD(0) algorithm is updating based on another estimate, we say that it’s a bootstrapping method, like DP.

Backup diagram for TD(0) [ref]

The estimated value of a state (top node in the diagram) is updated based on one sample of the immediate next state (bottom node)

Example:

To see how TD and Monte Carlo differ consider the following image

Driving home example [ref]

In this example, we are driving home from work and we try to estimate how long it will take us. Initially we assumed that it will take us 30 minutes, but as each point on the X axis shows, we re-estimated our time because of complications (e.g. car doesn’t work, highway is busy, etc).

How can we update our estimate of the time it takes to get home for next time we leave work?

The Monte Carlo way to do it (left), will be to wait until we get home and then update our initial estimate. That is because we don’t know the true return (Gt — the final value).

The TD way to do it (right), will be to continuously shift our estimate as we experience the environment.

Advantages of TD Prediction Methods

As mentioned before, TD learn estimates from other estimates. In other words, they learn a guess based on another guess — they bootstrap. But is it a good thing to do? The book talks about this question later on.

One clear advantage of TD methods is that they can work in a model-free learning. Another advantage is that they can be used online. That is, we don’t need to wait until the end of an episode — just until the next time step.

Sarsa: On-Policy TD Control

This section discusses an on-policy TD control method.

Similarly to before, instead of learning a state-value function, we will learn an action-value function — qπ(s, a). We can do this similarly to the TD method above to learn Vπ.

Remember that a single episode is comprised of a sequence of states and state–action pairs:

If before we moved from state S to the next state S’ and learned the value of states — Vπ, now we’ll move from state–action pair to state–action pair, and learn the values of state–action pairs — qπ.

The update:

is done after moving from a state S to a state S’ (as long as S’ is not a terminal state. In that case Q(St+1, At+1) = 0).

This update uses several events — (St, At, Rt+1, St+1, At+1), which is why the algorithm is called Sarsa.

Sarsa backup diagram [ref]

Similarly to all on-policy methods, we continuously estimate qπ while taking the greedy actions in policy π.

Q-learning: Off-policy TD Control

Q- learning is very similar to Sarsa, except that in the update stage of Q(S,A), we take the maximum over next state–action pairs:

Q learning update [ref]

Doing such, Q — the learned action-value, directly approximates q* — the optimal action-value function, independently of the policy being followed.

Expected Sarsa

Expected Sarsa is very similar to Q learning, except that instead of taking the maximum over next state–action pairs it uses the expected value, considering the probability of each action to happened in the policy

Expected Sarsa update [ref]

Sarsa is easier to compute than expected Sarsa, but using the expected Sarsa we eliminate the variance that happen due to the random selected of At+1.

In Q learning we take the max over actions — denoted by the “(“. In expected Sarsa we take the expected value over actions [ref]

Maximization Bias and Double Learning

As we’ve seen, in Q learning the update is taking the action that has the current highest value estimate. We call this a maximization bias or positive bias.

For example, consider a state S with many actions (a1, a2…) whole true value q(s, a), are all zero. Since the estimated values Q(s, a) are as the name suggest, estimates. Some of them will be below 0 and some above. So we can see that the max of the true values is 0, but the max of the estimated value is positive

This is solved with double Q learning:

Double Q learning [ref]

As shown above, the double Q learning maintains 2 separate tables, Q1 & Q2 of the state-action pairs. Then with 0.5 probability, we either going to take the maximum action using Q1, but evaluate that state-action pair using Q2 (line below “with 0.5 prob.”), or vice versa — taking action with Q2 and estimating with Q1.

Until next time,

Cheers.