SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our goal for the rest of the paper is to develop an alternative optimization perspective that can be applied in a broader setting relative to the operator perspective. To this end, our first task is to show that the optimization perspective can give us new insights even in the linear case and with squared loss functions. We generalize our optimization view of TD in the counter example by defining the objective function  $H$  for this case:
$$
H (\theta , w) = \frac {1}{2} \| R + \gamma P \Phi \theta - \Phi w \| _ {D} ^ {2}, \tag {6}
$$
where  $\| x \|_D = \sqrt{x^\top Dx}$ . Recall that in the exact case, the TD algorithm could succinctly be written as:  $\theta^{t+1} \gets \arg \min_w H(\theta^t, w)$ . Following the steps taken with the counter example, we first compute the gradient to derive the target and optimization forces:
$$
\nabla_ {w} H (\theta , w) = \Phi^ {\top} D (\Phi w - R - \gamma P \Phi \theta) = \underbrace {\Phi^ {\top} D \Phi} _ {\mathbf {M} _ {w}} w - \underbrace {\gamma \Phi^ {\top} D P \Phi} _ {\mathbf {M} _ {\theta}} \theta - \Phi^ {\top} D R. \tag {7}
$$
Here we have similar dynamics between  $\theta$  and  $w$  except that in this more general case, rather than scalar forces as in the counter example, we now have the two matrices  $\mathbf{M}_w$  and  $\mathbf{M}_{\theta}$ . Note that  $\mathbf{M}_w$  is a positive definite matrix,  $\lambda_{\mathrm{min}}(\mathbf{M}_w) = \min_x x^\top \mathbf{M}_wx / ||x||^2 >0$ , if  $\Phi$  is full rank. Below we can conveniently derive the update rule of linear TD by using these two matrices. All proofs are in the appendix.
Proposition 1. Let  $\{\theta^t\}_{t\in \mathbb{N}}$  be a sequence of parameters generated by Algorithm 1. Then, for the fixed-point  $\theta^{\star}$  and any  $t\in \mathbb{N}$ , we have
$$
\theta^ {t + 1} - \theta^ {\star} = \mathbf {M} _ {w} ^ {- 1} \mathbf {M} _ {\theta} \left(\theta^ {t} - \theta^ {\star}\right).
$$
This proposition characterizes the evolution of the difference between the parameter  $\theta^t$  and the fixed-point  $\theta^\star$ . Similarly to our approach with the counter example, we desire to ensure that this difference converges to 0. The following corollary gives us a condition for convergence [17].
Corollary 2. Let  $\{\theta^t\}_{t\in \mathbb{N}}$  be a sequence of parameters generated by Algorithm 1. Then,  $\{\theta^t\}_{t\in \mathbb{N}}$  converges to the fixed-point  $\theta^{\star}$  if and only if the spectral radius of  $\mathbf{M}_w^{-1}\mathbf{M}_\theta$  satisfies  $\rho (\mathbf{M}_w^{-1}\mathbf{M}_\theta) < 1$ .
We can employ this Corollary to characterize the convergence of TD in the counter example from the previous section. In this case, we have  $\mathbf{M}_w = 5$  and  $\mathbf{M}_{\theta} = \gamma (6 - 4\epsilon)$ , which gives us  $(5)^{-1}\gamma (6 - 4\epsilon) < 1$ . This is exactly the condition obtained in the previous section.
Notice that if  $d$  is the stationary-state distribution of the Markov chain then  $\rho (\mathbf{M}_w^{-1}\mathbf{M}_\theta) < 1$  [3], and so the algorithm is convergent. However, as demonstrated in the counter example, the condition can also hold for many other distributions. The key insight is to ensure that the distribution puts more weight behind states where the optimization force of  $w$  is dominating the target force due to  $\theta$ .
Note that the operator view of TD becomes inapplicable as we make modifications to  $H$ , because it will be unclear how to write the corresponding RL algorithm as a composition of operators. Can we demonstrate that in these cases the optimization perspective is still well-equipped to provide us with new insights about TD convergence? We next answer this question affirmatively by showing that the optimization perspective of TD convergence naturally extends to a broader setting than considered here, and therefore, is a more powerful perspective than the classical operator perspective.
6 Convergence of TD with General  $H$
Our desire now is to show convergence of TD without limiting the scope of our results to linear approximation and squared loss. We would like our theoretical results to support alternative ways of constructing  $H$  than the one studied in existing work as well as our previous section. In doing so, we again show that convergence of TD hinges on the interplay between the two identified forces.
Before presenting our main theorem, we discuss important concepts from optimization that will be used at the core of our proofs. We study convergence of Algorithms 1 and 2 with a general function  $H: \mathbb{R}^n \times \mathbb{R}^n \to \mathbb{R}$  that satisfies the following two assumptions:
I. The partial gradient  $\nabla_w H$ , is  $F_{\theta}$ -Lipschitz:
$$
\forall \theta_ {1}, \forall \theta_ {2} \quad \| \nabla_ {w} H (\theta_ {1}, w) - \nabla_ {w} H (\theta_ {2}, w) \| \leq F _ {\theta} \| \theta_ {1} - \theta_ {2} \|.
$$
II. The function  $H(\theta, w)$  is  $F_w$ -strongly convex in  $w$ :
$$
\forall w _ {1}, \forall w _ {2} \qquad \left(\nabla_ {w} H (\theta , w _ {1}) - \nabla_ {w} H (\theta , w _ {2})\right) ^ {\top} (w _ {1} - w _ {2}) \geq F _ {w} \| w _ {1} - w _ {2} \| ^ {2}.
$$
Note that, in the specific linear case and quadratic loss (our setting in the previous section) these assumptions are easily satisfied [8]. More specifically, in that case  $F_{\theta} = \lambda_{max}(\mathbf{M}_{\theta})$  and  $F_{w} = \lambda_{min}(\mathbf{M}_{w})$ . But the assumptions are also satisfied in a much broader setting than before. We provide more examples in section 6.3. We are now ready to present the main result of our paper:
Theorem 3. Let  $\{\theta^t\}_{t\in \mathbb{N}}$  be a sequence generated by either Algorithm 1 or 2. If  $F_{\theta} < F_{w}$ , then the sequence  $\{\theta^t\}_{t\in \mathbb{N}}$  converges to the fixed-point  $\theta^{\star}$ .
In order to prove the result, we tackle the two cases of Algorithm 1 and 2 separately. We first start by showing the result for Algorithm 1 where we can solve each iteration exactly. This is an easier case to tackle because we have the luxury of performing exact minimization, which is more stringent and difficult to implement but easier to analyze and understand. This would be more pertinent to Fitted Value Iteration and similar algorithms. We then move to the case where we approximately solve each iteration (Algorithm 2) akin to TD and similar algorithms. The proof in this case is more involved, and partly relies on choosing small steps when performing gradient descent.
6.1 Exact Optimization (Algorithm 1)
In this case, convergence to the fixed-point  $\theta^{\star}$  can be obtained as a corollary of the following result:
Proposition 4. Let  $\{\theta^t\}_{t\in \mathbb{N}}$  be a sequence generated by Algorithm 1. Then, we have:
$$
\left\| \theta^ {t + 1} - \theta^ {\star} \right\| \leq F _ {w} ^ {- 1} F _ {\theta} \left\| \theta^ {t} - \theta^ {\star} \right\|.
$$
From this result, we immediately obtain that the relative strength of the two forces, namely the conducive force  $F_{w}$  due to optimization and the detrimental target force  $F_{\theta}$ , determines if the algorithm is well-behaved. The proof of Theorem 3 in this case follows immediately from Proposition 4.
6.2 Inexact Optimization (Algorithm 2)
So far we have shown that the optimization process is convergent in the presence of exact optimization. This result supports the soundness of algorithms such as Fitted Value Iteration, but not TD yet, because in the case of TD we only roughly approximate the minimization step. Can this desirable
convergence result be extended to the more general setting of TD-like algorithms where we in exactly solve each iteration by a few gradient steps, or is exact optimization necessary for obtaining convergence? Answering this question is very important because in many settings it would be a stringent requirement to have to solve the optimization problem exactly.
We now show that indeed convergence manifests itself in the inexact case as well. In the extreme case, we can show that all we need is merely one single gradient update at each iteration. This means that even the purely online TD algorithm, presented in update (2), is convergent with general  $H$  if the optimization force can dominate the target force.
However, it is important to note that because we are now using gradient information to crudely approximate each iteration, we need to ensure that the step-size parameter  $\alpha$  is chosen reasonably. More concretely, in this setting we need an additional assumption, namely that there exists an  $L > 0$  such that:
$$
\forall w _ {1}, \forall w _ {2} \quad \| \nabla_ {w} H (\theta , w _ {1}) - \nabla_ {w} H (\theta , w _ {2}) \| \leq L \| w _ {1} - w _ {2} \|.
$$
Notice that such an assumption is quite common in the optimization literature (see, for instance, [18]). Moreover, it is common to choose  $\alpha = 1 / L$ , which we also employ in the context of Algorithm 2. We formalize this in the proposition presented below:
Proposition 5. Let  $\{\theta^t\}_{t\in \mathbb{N}}$  be a sequence generated by Algorithm 2 with the step-size  $\alpha = 1 / L$ . Then, we have:
$$
\left\| \theta^ {t + 1} - \theta^ {\star} \right\| \leq \sigma_ {K} \left\| \theta^ {t} - \theta^ {\star} \right\|,
$$
$$
\sigma_ {K} ^ {2} \equiv (1 - \kappa) ^ {K} (1 - \eta^ {2}) + \eta^ {2}.
$$
with  $\kappa \equiv L^{-1}F_w$  and  $\eta \equiv F_w^{-1}F_\theta$ .
Notice that  $\kappa$ , which is sometimes referred to as the inverse condition number in the optimization literature, is always smaller than 1. Therefore, we immediately conclude Theorem 3. Indeed, since the optimization force dominates the target force (meaning  $\eta < 1$ ), Algorithm 2 is convergent. Notice that a surprisingly positive consequence of this theorem is that we get convergent updates even if we only perform one gradient step per iteration ( $K = 1$ ). In deep-RL terminology, this corresponds to the case where we basically have no frozen target network, and that we immediately use the new target parameter  $\theta$  for the subsequent update.
To further situate this result, notice that as  $K$  approaches  $\infty$  then  $\sigma_K$  approaches  $\eta$ , which is exactly the contraction factor from Proposition 4 where we assumed exact optimization. So this proposition should be thought of as a tight generalization of Proposition 4 for the exact case. With a finite  $K$  we are paying a price for the crudeness of our approximation.
Moreover, another interesting reduction of our result is to the case where the target force due to bootstrapping in RL is absent, meaning that  $F_{\theta} \equiv 0$ . In this case, the contraction factor  $\sigma_{K}$  reduces to  $(1 - \kappa)^{K / 2}$ , which is exactly the known convergence rate for the gradient-descent algorithm in the strongly convex setting [18].
6.3 Examples
We focus on two families of loss functions where our assumptions can hold easily. To explain the first family, recall that TD could be written as follows:
$$
\theta^ {t + 1} \leftarrow \operatorname * {a r g   m i n} _ {w} H (\theta^ {t}, w).
$$
Now suppose we can write the function  $H(\theta, w)$  as the sum of two separate functions  $H(\theta, w) = G(\theta, w) + L(w)$ , where the function  $L(w)$  is strongly convex with respect to  $w$ . This setting is akin to using ridge regularization [19], which is quite common in deep learning (for example, the very popular optimizer AdamW [20]). This allows us to now work with functions  $G$  that are only convex (in fact, weakly convex is enough) with respect to  $w$ . We provide two examples:
- Suppose we would like to stick with linear function approximation. Then, the function  $G$  could be constructed using any convex loss where  $\nabla_w G(\theta, w)$  is Lipschitz-continuous with respect to  $\theta$ . Examples that satisfy this include the Huber loss [1] or the Logistic loss [12].
- Suppose we want to use the more powerful convex neural networks [21]. We need the error to be convex and monotonically increasing so that  $G$  is still convex. This is due to the classical result on the composition of convex functions. One example is the quadratic error where we restrict the output of the function approximator to positive values. Such neural nets are also Lipschitz continuous given proper activation functions such as ReLU.
Beyond this family, we have identified a second family, namely the control setting where a greedification operator is needed for bootstrapping. For example, with the quadratic error we could have:
$$
H (\theta , w) = \frac {1}{2} \sum_ {s} d (s) \sum_ {a} \pi (a | s) \left(\mathbb {E} _ {s ^ {\prime}} [ r + \gamma \max  _ {a ^ {\prime}} q \left(s ^ {\prime}, a ^ {\prime}, \theta\right) ] - q (s, a, w)\right) ^ {2},
$$
where  $q$  is the state-action value function parameterized by either  $\theta$  or  $w$ .
We again need the two assumptions, namely strong-convexity with respect to  $w$  and Lipschitzness of  $\nabla_w H(\theta, w)$  with respect to  $\theta$  to hold. Actually, Lee and He [8] already showed the strong convexity with respect to  $w$ , but we need to still show the Lipschitz property of  $\nabla_w H(\theta, w)$  with respect to  $\theta$ . Note that they showed the Lipschitz property only with respect to  $w$  and not with respect to  $\theta$ . We are now able to show this result. Please see Proposition 8 and its proof in the appendix. Our proof also supports other greedification operators, such as softmax [22], so long as its non-expansive.
In this paper, we studied convergence of TD through the lens of optimization. The underlying principles of TD are so central to RL that a large number of RL algorithms can be thought of as versions of TD, and the availability of convergence results varies between different types of algorithms. We chose a setting we believe is as elementary as possible to highlight key principles of TD convergence. Closest to our work is the work of Tsitsiklis and Van Roy [3] who proved convergence of TD with linear function approximation, squared error, and the stationary-state distribution of the Markov chain. Later work, especially that of Bhandari et al. [23], further analyzed the case of TD with stochastic gradients and non-iid samples in the on-policy case. Extensions to the control setting also exist [24, 25].
In particular, Melo et al. [24] presented a condition in their equation (7) which may look similar to our condition at first glance, but it is in fact quite different than the condition identified in our paper. In their equation (7), they require that the eigenvalues of  $\Sigma_{\pi}$ , the policy-conditioned covariance matrix  $\Phi^{\top}D\Phi$ , dominate the eigenvalues of a second matrix  $\gamma^{2}\Sigma_{\pi}^{\star}(\theta)$ . Here  $\Sigma_{\pi}^{\star}(\theta)$  is a similar covariance matrix for features, but one that is computed based on the action-greedification step.
We believe to be the first paper to show a exact contraction for TD with frozen target network and general  $K$ . To the best of our knowledge, existing results prior to Lee and He [8] mainly considered the case where we either never freeze the target network (corresponding to the value of  $K = 1$ ), or the somewhat unrealistic case where we can exactly solve each iteration. Lee and He showed guarantees pertaining to  $K > 1$ , but notice that, while their result is quite innovative, they leaned on the more standard optimization tools for ensuring that gradient descent with a fixed  $K$  can only solve each iteration approximately. Therefore, each iteration results in some error. In their theory, this error is accumulated per iteration and needs to be accounted for in the final result. Therefore, they fell short of showing 1) contraction and 2) exact convergence to the TD fixed-point, and only show that the final iterate is in the vicinity of the fixed-point defined by the amount of error accumulated over the trajectory. With finite  $K$ , they need to account for errors in solving each iteration (denoted by  $\epsilon_{k}$  in their proofs such as in Theorem 3), which prevent them from obtaining a clean convergence result. In contrast, we can show that even approximately solving each iteration (corresponding to finite  $K$ ) is enough to obtain contraction (without any error), because we look at the net effect of  $K$  updates to the online network and the single update to the target network, and show that this effect is on that always take us closer to the fixed-point regardless of the specific value of  $K$ .
Modifications of TD are presented in prior work that are more conducive to convergence analysis [26, 27, 28, 29, 30]. They have had varying degrees of success both in terms of empirical performance [31], and in terms of producing convergent algorithms [32]. TD is also studied under over-parameterization [33, 34], with learned representations [35], proximal updates [36], and auxiliary tasks [37]. Also, the quality of TD fixed-point has been studied in previous work [16].
A large body of literature focuses on finding TD-like approaches that are in fact true gradient-descent approaches in that they follow the gradient of a stationary objective function [38, 15, 39, 40, 41, 42].
In these works, the optimization problem is formulated in such a way that the minimizer of the loss will be the fixed-point of the standard TD. Whereas TD has been extended to large-scale settings, these algorithms have not been as successful as TD in terms of applications.
A closely related algorithm to TD is that of Baird, namely the residual gradient algorithm [4, 43]. This algorithm has a double-sampling issue that needs to be addressed either by assuming a model of the environment or by learning the variance of the value function [43, 44]. However, even with deterministic MDPs, in which the double sampling issue is not present [45], the algorithm often finds a fixed-point that has a lower quality than that of the TD algorithm [46]. This is attributed to the fact that the MDP might still look stochastic in light of the use of function approximation [38].
TD could be thought of as an incremental approach to approximate dynamic programming and Fitted Value Iteration [7] for which various convergence results based on the operator view exits [14, 47, 48]. Also, these algorithms are well-studied in terms of their error-propagation behavior [49, 50, 51].
Many asymptotic or finite-sample results on Q-learning (the control version of TD) with function approximation make additional assumptions on the problem structure, with a focus on the exploration problems [52, 53, 54, 55, 56]. Like mentioned before, our focus was on the prediction setting where exploration is not relevant.
8 Open Questions
To foster further progress in this area we identify key questions that still remain open.
I. We showed that TD converges to the fixed-point  $\theta^{\star}$  characterized by  $\nabla_wH(\theta^\star ,\theta^\star) = 0$  and so the sequence  $\{\| \nabla_wH(\theta^t,\theta^t)\| \}_{t\in \mathbb{N}}$  converges to 0. Can we further show that this sequence monotonically decreases with  $t$ , namely that  $\| \nabla_wH(\theta^{t + 1},\theta^{t + 1})\| \leq$ $\| \nabla_wH(\theta^t,\theta^t)\|$  for all  $t\in \mathbb{N}$ ? Interestingly, a line of research focused on inventing new algorithms (often called gradient TD algorithms) that possess this property [57]. However, if true, this result would entail that when our assumptions hold the original TD algorithm gives us this desirable property for free.
II. Going back to the counter example, we can show convergent updates by constraining the representation as follows:  $\phi(s) \leq \gamma \phi(s')$  whenever  $\mathcal{P}(s' \mid s) > 0$ . This result can easily be extended to all MDPs with a single-dimensional feature vector. A natural question then is to identify a multi-dimensional representation, given a fixed MDP and update distribution  $D$ , leads into convergent updates. Similarly, we can pose this question in terms of fixing the MDP and the representation matrix  $\Phi$ , and identifying a set of distributions that lead into convergent updates. Again, we saw from the counter example that putting more weight behind states whose target force is weak (such as states leading into the terminal state) is conducive to convergence. How do we identify such distributions systematically?
III. We studied the setting of deterministic updates. To bridge the gap with RL practice, we need to study TD under stochastic-gradient updates. The stochasticity could for instance be due to using a minibatch or using non-iid samples. Bhandari et al. [23] tackle this for the case of  $K = 1$ . Can we generalize our convergence result to this more realistic setting?
IV. We have characterized the optimization force using the notion of strong convexity. It would be interesting to relax this assumption to handle neural networks and alternative loss functions. This can serve as a more grounded explanation for the empirical success of TD in the context of deep RL. Can we generalize our result to this more challenging setting?
In this paper, we argued that the optimization perspective of TD is more powerful than the well-explored operator perspective of TD. To demonstrate this, we generalized previous convergence results of TD beyond the linear setting and squared loss functions. We believe that further exploring this optimization perspective can be a promising direction to design convergent RL algorithms.
Our general result on the convergent nature of TD is consistent with the empirical success and the attention that this algorithm has deservedly received. The key factor that governs the convergence of TD is to ensure that the optimization force of the algorithm is well-equipped to dominate the more harmful target force. This analogy is one that can be employed to explain the convergent nature of TD even in the presence of the three pillar of the deadly triad.