SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kavosh Asadi*
Shoham Sabach*
Amazon & Technion
Yao Liu
Omer Gottesman
Rasool Fakoor
We study the convergence behavior of the celebrated temporal-difference (TD) learning algorithm. By looking at the algorithm through the lens of optimization, we first argue that TD can be viewed as an iterative optimization algorithm where the function to be minimized changes per iteration. By carefully investigating the divergence displayed by TD on a classical counter example, we identify two forces that determine the convergent or divergent behavior of the algorithm. We next formalize our discovery in the linear TD setting with quadratic loss and prove that convergence of TD hinges on the interplay between these two forces. We extend this optimization perspective to prove convergence of TD in a much broader setting than just linear approximation and squared loss. Our results provide a theoretical explanation for the successful application of TD in reinforcement learning.
Temporal-difference (TD) learning is arguably one of the most important algorithms in reinforcement learning (RL), and many RL algorithms are based on principles that TD embodies. TD is at the epicenter of some of the recent success examples of RL [1, 2], and has influenced many areas of science such as AI, economics, and neuroscience. Despite the remarkable success of TD in numerous settings, the algorithm is shown to display divergent behavior in contrived examples [3, 4, 5]. In practice, however, divergence rarely manifests itself even in situations where TD is used in conjunction with complicated function approximators. Thus, it is worthwhile to obtain a deeper understanding of TD, and to generalize existing convergence results to explain its practical success.
In this paper, our desire is to study TD through the lens of optimization. We argue that TD could best be thought of as an iterative optimization algorithm, which proceeds as follows:
$$
\theta^ {t + 1} \approx \arg \min  _ {w} H \left(\theta^ {t}, w\right). \tag {1}
$$
Here, the objective  $H$  is constructed by a) choosing a function approximator such as a neural network and b) defining a discrepancy measure between successive predictions, such as the squared error. We will discuss more examples later.
This process involves two different parameters, namely the target parameter  $\theta^t$  that remains fixed at each iteration  $t$ , and the optimization parameter  $w$  that is adjusted during each iteration to minimize the corresponding loss function. Using the more familiar deep-RL terminology,  $\theta^t$  corresponds to the parameters of the target network, whereas  $w$  corresponds to the parameters of the online network. Many RL algorithms can be described by this iterative process with the main difference being the approach taken to (approximately) perform the minimization. At one side of the spectrum lies the original TD algorithm [6], which proceeds by taking a single gradient-descent step to adjust  $w$  and immediately updating  $\theta$ . More generally, at each iteration  $t$  we can update the  $w$  parameter  $K$  times
using gradient-descent while freezing  $\theta$ , a common technique in deep RL [1]. Finally, Fitted Value Iteration [7] lies at the other extreme and solves each iteration exactly when closed-form solutions exist. Therefore, a deeper understanding of the iterative optimization process (1) can facilitate a better understanding of TD and related RL algorithms.
The iterative process (1) is well-studied in the setting where  $H$  is constructed using linear function approximation and squared error. In particular, with  $K = 1$ , meaning TD without frozen target network, the most notable result is due to the seminal work of Tsitsiklis and Van Roy [3] that proved the convergence of the algorithm. More recently, Lee and He [8] focused on the more general case of  $K \geq 1$ . While they took a major step in understanding the role of a frozen target-network in deep RL, they leaned on the more standard optimization tools to show that gradient-descent with a fixed  $K$  can be viewed as solving each iteration approximately. Therefore, in their analysis, each iteration results in some error. This error is accumulated per iteration and needs to be accounted for in the final result. Therefore, Lee and He [8] fell short of showing convergence to the TD fixed-point, and managed to show that the final iterate will be in the vicinity of the fixed-point defined by the amount of error accumulated over the iterations.
In this work, our primary contribution is to generalize and improve existing results on TD convergence. In particular, in our theory, we can support the popular technique of employing frozen target networks with general  $K \geq 1$ , and we also support the use of a family of function approximators and error-measure pairs that includes linear function-approximation and squared error as a special case. To the best of our knowledge, this is the first contraction result that does not restrict to the case of  $K = 1$ , and can also generalize the well-explored linear case with quadratic error.
To build intuition, we start by taking a deeper dive into one of the classical examples where TD displays divergent behavior [3]. In doing so, we identify two key forces, namely a target force and an optimization force, whose interplay dictates whether TD is guaranteed to converge or that a divergent behavior may manifest itself. If the optimization force can dominate the target force, the process is convergent even when we operate in the famous deadly triad [5], namely in the presence of bootstrapping, function approximation, and off-policy updates. Overall, our results demonstrate that TD is a sound algorithm in a much broader setting than understood in previous work.
Reinforcement learning (RL) is the study of artificial agents that can learn through trial and error [5]. In this paper, we focus on the setting where the agent is interested in predicting the long-term goodness or the value of its states. Referred to as the prediction setting, this problem is mathematically formulated by the Markov reward process (MRP) [9]. In this paper, we consider the discounted infinite-horizon case of MRPs, which is specified by the tuple  $\langle S, \mathcal{R}, \mathcal{P}, \gamma \rangle$ , where  $\mathcal{S}$  is the set of states. The function  $R: \mathcal{S} \to \mathbb{R}$  denotes the reward when transitioning out of a state. For any set, we denote the space of probability distributions over it by  $\Delta$ . The transition  $\mathcal{P}: \mathcal{S} \to \Delta(\mathcal{S})$  defines the conditional probabilities over the next states given the current state, and is denoted by  $\mathcal{P}(s' \mid s)$ . Finally, the scalar  $\gamma \in (0, 1)$  geometrically discounts rewards that are received in the future steps.
The primary goal of this paper is to understand the behavior of RL algorithms that learn to approximate the state value function defined as  $v(s) \coloneqq \mathbb{E}\big[\sum_{t=0}^{\infty} \gamma^{t} r_{t} \big| s_{0} = s\big]$ . To this end, we define the Bellman operator  $\mathcal{T}$  as follows:
$$
\left[ \mathcal {T} v \right] (s) := \mathcal {R} (s) + \sum_ {s ^ {\prime} \in \mathcal {S}} \gamma \mathcal {P} \left(s ^ {\prime} \mid s\right) v \left(s ^ {\prime}\right),
$$
which we can write compactly as:  $\mathcal{T}v\coloneqq R + \gamma Pv$ . In large-scale RL problems the number of states  $|S|$  is enormous, which makes it unfeasible to use tabular approaches. We focus on the setting where we have a parameterized function approximator, and our desire is to find a parameter  $\theta$  for which the learned value function  $v(s;\theta)$  results in a good approximation of the value function  $v(s)$ .
A fundamental and quite popular approach to finding a good approximation of the value function is known as temporal difference (TD) learning [6]. Suppose that a sample  $\langle s, r, s' \rangle$  is given where  $s' \sim \mathcal{P}(\cdot | s)$ . In this case, TD learning algorithm updates the parameters of the approximate value function as follows:
$$
\theta^ {t + 1} \leftarrow \theta^ {t} + \alpha \left(r + \gamma v \left(s ^ {\prime}; \theta^ {t}\right) - v (s; \theta^ {t})\right) \nabla_ {\theta} v (s; \theta^ {t}), \tag {2}
$$
where  $\theta^t$  denotes the parameters of our function approximator at iteration  $t$ . Also, by  $\nabla_{\theta}v(s;\theta^{t})$  we are denoting the partial gradient of  $v(s;\theta)$  with respect to the parameter  $\theta$ . Note that TD uses
the value estimate obtained by one-step lookahead  $\left(r + \gamma v(s';\theta^t)\right)$  to update its approximate value function  $v(s;\theta^t)$  in the previous step. This one-step look-ahead of TD could be thought of as a sample of the right-hand-side of the Bellman equation. Our focus on TD is due to the fact that many of the more modern RL algorithms are designed based on the principles that TD embodies. We explain this connection more comprehensively in section 3.
To understand the existing results on TD convergence, we define the Markov chain's stationary state distribution  $d(\cdot)$  as the unique distribution with the following property:  $\forall s' \sum_{s \in S} d(s) \mathcal{P}(s' \mid s) = d(s')$ . Then, Tsitsiklis and Van Roy show in [3] that, under linear function approximation, TD will be convergent if the states  $s$  are sampled from the stationary-state distribution. We will discuss this result in more detail later on in section 5.
However, it is well-known that linear TD can display divergent behavior if states are sampled from an arbitrary distribution rather than the stationary-state distribution of the Markov chain. A simple yet classical counter example of divergence of linear TD is shown in Figure 1 and investigated in section 4. First identified in [3], this example is a very simple Markov chain with two non-terminal states and zero rewards. Moreover, little is known about convergence guarantees of TD under alternative function approximators, or even when the update rule of TD is modified slightly.
In this paper, we focus on the Markov reward process (MRP) setting which could be thought of as a Markov decision process (MDP) with a single action. TD can display divergence even in the MRP setting [3, 5], which indicates that the presence of multiple actions is not the core reason behind TD misbehavior [5] — Divergence can manifest itself even in the more specific case of single action. Our results can naturally be extended to multiple actions with off-policy updates where in update (2) of TD, actions are sampled according to a target policy but states are sampled according to a behavior policy. That said, akin to Tsitsiklis and Van Roy [3], as well as chapter 11 in the book of Sutton and Barto [5], we focus on MRPs to study the root cause of TD in the most clear setting.
3 TD Learning as Iterative Optimization
In this section, we argue that common approaches to value function prediction can be viewed as iterative optimization algorithms where the function to be minimized changes per iteration. To this end, we recall that for a given experience tuple  $\langle s,r,s^{\prime}\rangle$ , TD performs the update presented in (2). A common augmentation of TD is to decouple the parameters to target  $(\theta)$  and optimization  $(w)$  parameters [1] and update  $\theta$  less frequently. In this case, at a given iteration  $t$ , the algorithm performs multiple  $(K)$  gradient steps as follows:
$$
w ^ {t, k + 1} \leftarrow w ^ {t, k} + \alpha \left(r + \gamma v \left(s ^ {\prime}; \theta^ {t}\right) - v (s; w ^ {t, k})\right) \nabla_ {\theta} v (s; w ^ {t, k}), \tag {3}
$$
and then updates the target parameter  $\theta^{t + 1}\gets w^{t,K}$  before moving to the next iteration  $t + 1$ . Here  $K$  is a hyper-parameter, where  $K = 1$  takes us back to the original TD update (2). Observe that the dependence of  $v(s';\theta^t)$  to our optimization parameter  $w$  is ignored in this update, despite the fact that an implicit dependence is present due to the final step  $\theta^{t + 1}\gets w^{t,K}$ . This means that the objective function being optimized is made up of two separate input variables<sup>2</sup>. We now define:
$$
H (\theta , w) = \frac {1}{2} \sum_ {s} d (s) \bigl (\mathbf {E} _ {r, s ^ {\prime}} [ r + \gamma v (s ^ {\prime}; \theta) ] - v (s; w) \bigr) ^ {2},
$$
where we allow  $d(\cdot)$  to be an arbitrary distribution, not just the stationary-state distribution of the Markov chain. Observe that the partial gradient of  $H$  with respect to the optimization parameters  $w$  is equivalent to the expectation of the update in (3). Therefore, TD, DQN, and similar algorithms could best be thought of as learning algorithms that proceed by approximately solving for the following sequence of optimization problems:
$$
\theta^ {t + 1} \approx \arg \min _ {w} H (\theta^ {t}, w),
$$
using first-order optimization techniques. This optimization perspective is useful conceptually because it accentuates the unusual property of this iterative process, namely that the first argument of the objective  $H$  hinges on the output of the previous iteration. It also has important practical ramifications when designing RL optimizers [11]. Moreover, the general form of this optimization
2In fact, [10] shows that there cannot exist any objective function  $J(\theta)$  with a single input variable whose gradient would take the form of the TD update.
process allows for using alternative forms of loss functions such as the Huber loss [1], the logistic loss [12], or the entropy [13], as well as various forms of function approximation such as linear functions or deep neural networks. Each combination of loss functions and function approximators yields a different  $H$ , but one that is always comprised of a function  $H$  with two inputs.
A closely related optimization process is one where each iteration is solved exactly:
$$
\theta^ {t + 1} \leftarrow \arg \min  _ {w} H \left(\theta^ {t}, w\right), \tag {4}
$$
akin to Fitted Value Iteration [7, 14]. Exact optimization is doable in problems where the model of the environment is available and that the solution takes a closed form. A pseudo-code of both algorithms is presented in Algorithms 1 and 2.
Algorithm 1 Value Function Optimization with Exact Updates
Input:  $\theta^0$ ,  $T$
$$
\theta^ {t + 1} \leftarrow \arg \min  _ {w} H \left(\theta^ {t}, w\right)
$$
Return  $\theta^T$
Algorithm 2 Value Function Optimization with Gradient Updates
Input:  $\theta^0$ ,  $T$ ,  $K$ ,  $\alpha$
$$
w ^ {t, 0} \leftarrow \theta^ {t}
$$
for  $k = 0$  to  $K - 1$  do
$$
w ^ {t, k + 1} \leftarrow w ^ {t, k} - \alpha \nabla_ {w} H \left(\theta^ {t}, w ^ {t, k}\right)
$$
$$
\theta^ {t + 1} \leftarrow w ^ {t, K}
$$
In terms of the difference between the two algorithms, notice that in Algorithm 1 we assume that we have the luxury of somehow solving each iteration exactly. This stands in contrast to Algorithm 2 where we may not have this luxury, and resort to gradient updates to find a rough approximation of the actual solution. Thus, Algorithm 1 is more difficult to apply but easier to understand, whereas Algorithm 2 is easier to apply but more involved in terms of obtaining a theoretical understanding.
Note that if convergence manifests itself in each of the two algorithms, the convergence point denoted by  $\theta^{\star}$  must have the following property:
$$
\nabla_ {w} H \left(\theta^ {\star}, \theta^ {\star}\right) = \mathbf {0}. \tag {5}
$$
This fixed-point characterization of TD has been explored in previous work [15, 16]. Whenever it exists, we refer to  $\theta^{\star}$  as the fixed-point of these iterative algorithms. However, convergence to the fixed-point is not always guaranteed [5] even when we have the luxury of performing exact minimization akin to Algorithm 1. In this paper, we study both the exact and inexact version of the optimization process. In doing so, we identify two forces that primarily influence convergence. To begin our investigation, we study a counter example to build intuition on why divergence can manifest itself. We present a formal analysis of the convergence of the two algorithms in section 6.
4 Revisiting the Divergence Example of TD
In this section, we focus on a simple counter example where TD is known to exhibit divergence. First identified by [3], investigating this simple example enables us to build some intuition about the root cause of divergence in the most clear setting.
Shown in Figure 1, this example is a Markov chain with two non-terminal states and zero rewards. A linear function approximation,  $v(s; \theta) = \phi(s)\theta$ , is employed with a single feature where  $\phi(s_1) = 1$  and  $\phi(s_2) = 2$ . The third state is a terminal one whose value is always zero. The true value function (0 in all states) is realizable with  $\theta = 0$ .
To build some intuition about the root cause of divergence, we discuss the convergence of exact TD with a few state distributions in this example. We desire to update all but the terminal state with non-zero probability. However, to begin with, we focus on a particular extreme case where we put all of our update probability behind the second state:
$$
\theta^ {t + 1} \leftarrow \arg \min  _ {w} H \left(\theta^ {t}, w\right) = \arg \min  _ {w} \frac {1}{2} \left((1 - \epsilon) (\gamma 2 \theta^ {t}) - 2 w\right) ^ {2}.
$$
We thus have  $\nabla_w H(\theta^t, w) = 2(2w - (1 - \epsilon)\gamma 2\theta^t)$ , and since  $\nabla_w H(\theta^t, \theta^{t+1}) = 0$ , we can write:  $\theta^{t+1} \gets (2)^{-1}(1 - \epsilon)\gamma 2\theta^t$ . The process converges to the fixed-point  $\theta = 0$  for all values of  $\gamma < 1$
and  $\epsilon \in [0,1]$ . This is because the target force of  $\theta_t$  (namely  $\big((1 - \epsilon)\gamma 2\big)$ ) is always dominated by the optimization force of  $w$  (namely 2). Note that updating this state is thus not problematic at all, and that the update becomes even more conducive to convergence when  $\gamma$  is smaller and  $\epsilon$  is larger.
We now juxtapose the first extreme case with the second one where we put all of our update probability behind the first state, in which case at a given iteration  $t$  we have:
$$
\theta^ {t + 1} \leftarrow \arg \min  _ {w} H (\theta^ {t}, w) = \arg \min  _ {w} \frac {1}{2} (\gamma 2 \theta^ {t} - w) ^ {2}.
$$
We thus have  $\nabla_w H(\theta^t, w) = (w - \gamma 2\theta^t)$  and so  $\theta^{t+1} \gets (1)^{-1}\gamma 2\theta^t$ . Unlike the first extreme case, convergence is no longer guaranteed. Concretely, to ensure convergence, we require that the target force of  $\theta^t$  (namely  $\gamma 2$ ) be larger than the optimization force of  $w$  (namely 1).
To better understand why divergence manifests itself, note that the two states  $s_1$  and  $s_2$  have very similar representations (in the form of a single feature). Note also that the value of the feature is larger
for the target state than it is for the source state,  $\phi(s_2) > \phi(s_1)$ . This means that any change in the value of the source state  $s_1$ , would have the external effect of changing the value of the target state  $s_2$ . Further, the change is in the same direction (due to the positive sign of the feature in both states) and that it is exactly  $2\gamma$  larger for the target state relative to the source state. Therefore, when  $\gamma > 1/2$ , the function  $H$  will be more sensitive to the target parameter  $\theta$  relative to the optimization parameter  $w$ . This is the root cause of divergence.
Moving to the case where we update both states with non-zero probability, first note that if the two updates were individually convergent, then the combination of the two updates would also have been convergent in a probability-agnostic way. Stated differently, convergence would have been guaranteed regardless of the probability distribution  $d$  and for all off-policy updates. However, in light of the fact that the optimization force of  $s_1$  does not dominate its target force, we need to choose  $d(s_1)$  small enough so as to contain the harmful effect of updating this state.
We compute the overall update (under the case where we update the two states with equal probability) by computing the sum of the two gradients in the two extreme cases above:
$$
\left(w - \gamma 2 \theta^ {t}\right) + 2 \left(2 w - \gamma 2 (1 - \epsilon) \theta^ {t}\right) = 0,
$$
and so  $\theta^{t + 1}\gets (5)^{-1}\gamma (6 - 4\epsilon)\theta^t$  . Note that even with a uniform update distribution, the two states are contributing non-uniformly to the overall update, and the update in the state  $s_2$  is more influential because of the higher magnitude of the feature in this state  $(\phi (s_{2}) = 2$  against  $\phi (s_1) = 1)$
To ensure convergence, we need to have that  $\gamma < \frac{5}{6 - 4\epsilon}$ . Notice, again, that we are combining the update in a problematic state with the update in the state that is conducive for convergence. We can contain the negative effect of updating the first state by ensuring that our update in the second state is even more conducive for convergence (corresponding to a larger  $\epsilon$ ). In this case, the update of  $s_2$  can serve as a mitigating factor.
We can further characterize the update with a general distribution. In this case, we have:
$$
d \left(s _ {1}\right) \left(\left(w - \gamma 2 \theta^ {t}\right)\right) + \left(1 - d \left(s _ {1}\right)\right) \left(2 \left(2 w - \gamma 2 (1 - \epsilon) \theta^ {t}\right)\right) = 0,
$$
which gives us a convergent update if:  $d(s_1) < \frac{4 - 4\gamma(1 - \epsilon)}{3 + \gamma 2 - 4\gamma(1 - \epsilon)}$ . Both the denominator and the numerator are always positive, therefore regardless of the values of  $\epsilon$  and  $\gamma$ , there always exists a convergent off-policy update, but one that needs to assign less probability to the problematic state as we increase  $\gamma$  and decrease  $\epsilon$ .
This means that using some carefully chosen off-policy distributions is not only safe, but that doing so can even speed up convergence. This will be the case if the chosen distribution makes the objective function  $H$  more sensitive to changes in its first argument (target parameter  $\theta$ ) than its second argument (the optimization parameter  $w$ ). We next formalize this intuition.
Figure 1: Divergence example of TD [3].
5 Convergence of Linear TD with Quadratic Loss
We now focus on the case of TD with linear function approximation. In this case, the expected TD update (2) could be written as the composition of two separate operators (with  $v_{t} = \Phi \theta_{t}$ ):
$$
v _ {t + 1} \leftarrow \Pi_ {D} \left(\mathcal {T} (v _ {t})\right),
$$
where the projection operator  $\Pi_D$  and the Bellman operator  $\mathcal{T}$  are defined as follows:
$$
\Pi_ {D} = \Phi (\Phi^ {\top} D \Phi) ^ {- 1} \Phi^ {\top} D, \qquad \text {a n d} \qquad \mathcal {T} (v) = R + \gamma P v.
$$
Here,  $D$  is a diagonal matrix with diagonal entries  $d(s_{1}),\ldots ,d(s_{n})$ , where  $n$  is the number of states. The projection operator  $\Pi_D$  is non-expansive under any distribution  $d$ . However, the Bellman operator is a  $\gamma$ -contraction under a specific  $d$ , namely the stationary-state distribution of the Markov chain specified by the transition matrix  $P$  as shown by Tsitsiklis and Van Roy [3]. Therefore, the composition of the two operators is a  $\gamma$ -contraction when the distribution  $d$  is the stationary-state distribution of the Markov chain.
In light of this result, one may think that TD is convergent in a very narrow sense, specifically when the updates are performed using the stationary-state distribution. But, as we saw with the counter example, in general there may exist many other distributions  $d$  that are in fact very conducive for TD convergence. In these cases, the proof technique above cannot provide a tool to ensure convergence because of the reliance of the non-expansive and contraction properties of these operators. Is this a limitation of the TD algorithm itself, or a limitation of this specific operator perspective of TD? Further, if this operator perspective is limited, is there a different perspective that can give rise to a more general understanding of TD convergence?