Title: TD Convergence: An Optimization Perspective
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement Learning

Temporal Difference Learning

Value Function Optimization

Convergence
Abstract: We study the convergence behavior of the celebrated temporal-difference (TD) learning algorithm. By looking at the algorithm through the lens of optimization, we first argue that TD can be viewed as an iterative optimization algorithm where the function to be minimized changes per iteration. By carefully investigating the divergence displayed by TD on a classical counter example, we identify two forces that determine the convergent or divergent behavior of the algorithm. We next formalize our discovery in the linear TD setting with quadratic loss and prove that convergence of TD hinges on the interplay between these two forces. We extend this optimization perspective to prove convergence of TD in a much broader setting than just linear approximation and squared loss. Our results provide a theoretical explanation for the successful application of TD in reinforcement learning.

TD Convergence: An Optimization Perspective

Abstract

We study the convergence behavior of the celebrated temporal-difference (TD) learning algorithm. By looking at the algorithm through the lens of optimization, we first argue that TD can be viewed as an iterative optimization algorithm where the function to be minimized changes per iteration. By carefully investigating the divergence displayed by TD on a classical counter example, we identify two forces that determine the convergent or divergent behavior of the algorithm. We next formalize our discovery in the linear TD setting with quadratic loss and prove that convergence of TD hinges on the interplay between these two forces. We extend this optimization perspective to prove convergence of TD in a much broader setting than just linear approximation and squared loss. Our results provide a theoretical explanation for the successful application of TD in reinforcement learning.

1 Introduction

Temporal-difference (TD) learning is arguably one of the most important algorithms in reinforcement learning (RL), and many RL algorithms are based on principles that TD embodies. TD is at the epicenter of some of the recent success examples of RL [1, 2], and has influenced many areas of science such as AI, economics, and neuroscience. Despite the remarkable success of TD in numerous settings, the algorithm is shown to display divergent behavior in contrived examples [3, 4, 5]. In practice, however, divergence rarely manifests itself even in situations where TD is used in conjunction with complicated function approximators. Thus, it is worthwhile to obtain a deeper understanding of TD, and to generalize existing convergence results to explain its practical success.

In this paper, our desire is to study TD through the lens of optimization. We argue that TD could best be thought of as an iterative optimization algorithm, which proceeds as follows:

Here, the objective H is constructed by a) choosing a function approximator such as a neural network and b) defining a discrepancy measure between successive predictions, such as the squared error. We will discuss more examples later.

This process involves two different parameters, namely the target parameter θ t that remains fixed at each iteration t , and the optimization parameter w that is adjusted during each iteration to minimize the corresponding loss function. Using the more familiar deep-RL terminology, θ t corresponds to the parameters of the target network, whereas w corresponds to the parameters of the online network. Many RL algorithms can be described by this iterative process with the main difference being the approach taken to (approximately) perform the minimization. At one side of the spectrum lies the original TD algorithm [6], which proceeds by taking a single gradient-descent step to adjust w and immediately updating θ . More generally, at each iteration t we can update the w parameter K times

Shoham Sabach ∗ Amazon & Technion

using gradient-descent while freezing θ , a common technique in deep RL [1]. Finally, Fitted Value Iteration [7] lies at the other extreme and solves each iteration exactly when closed-form solutions exist. Therefore, a deeper understanding of the iterative optimization process (1) can facilitate a better understanding of TD and related RL algorithms.

The iterative process (1) is well-studied in the setting where H is constructed using linear function approximation and squared error. In particular, with K = 1 , meaning TD without frozen target network, the most notable result is due to the seminal work of Tsitsiklis and Van Roy [3] that proved the convergence of the algorithm. More recently, Lee and He [8] focused on the more general case of K ≥ 1 . While they took a major step in understanding the role of a frozen target-network in deep RL, they leaned on the more standard optimization tools to show that gradient-descent with a fixed K can be viewed as solving each iteration approximately. Therefore, in their analysis, each iteration results in some error. This error is accumulated per iteration and needs to be accounted for in the final result. Therefore, Lee and He [8] fell short of showing convergence to the TD fixed-point, and managed to show that the final iterate will be in the vicinity of the fixed-point defined by the amount of error accumulated over the iterations.

In this work, our primary contribution is to generalize and improve existing results on TD convergence. In particular, in our theory, we can support the popular technique of employing frozen target networks with general K ≥ 1 , and we also support the use of a family of function approximators and error-measure pairs that includes linear function-approximation and squared error as a special case. To the best of our knowledge, this is the first contraction result that does not restrict to the case of K = 1 , and can also generalize the well-explored linear case with quadratic error.

To build intuition, we start by taking a deeper dive into one of the classical examples where TD displays divergent behavior [3]. In doing so, we identify two key forces, namely a target force and an optimization force, whose interplay dictates whether TD is guaranteed to converge or that a divergent behavior may manifest itself. If the optimization force can dominate the target force, the process is convergent even when we operate in the famous deadly triad [5], namely in the presence of bootstrapping, function approximation, and off-policy updates. Overall, our results demonstrate that TD is a sound algorithm in a much broader setting than understood in previous work.

2 Problem Setting

Reinforcement learning (RL) is the study of artificial agents that can learn through trial and error [5]. In this paper, we focus on the setting where the agent is interested in predicting the long-term goodness or the value of its states. Referred to as the prediction setting, this problem is mathematically formulated by the Markov reward process (MRP) [9]. In this paper, we consider the discounted infinite-horizon case of MRPs, which is specified by the tuple ⟨S , R , P , γ ⟩ , where S is the set of states. The function R : S → R denotes the reward when transitioning out of a state. For any set, we denote the space of probability distributions over it by ∆ . The transition P : S → ∆( S ) defines the conditional probabilities over the next states given the current state, and is denoted by P ( s ′ | s ) . Finally, the scalar γ ∈ (0 , 1) geometrically discounts rewards that are received in the future steps.

The primary goal of this paper is to understand the behavior of RL algorithms that learn to approximate the state value function defined as v ( s ) := E [ ∑ ∞ t =0 γ t r t ∣ ∣ s 0 = s ] . To this end, we define the Bellman operator T as follows:

which we can write compactly as: T v := R + γPv . In large-scale RL problems the number of states |S| is enormous, which makes it unfeasible to use tabular approaches. We focus on the setting where we have a parameterized function approximator, and our desire is to find a parameter θ for which the learned value function v ( s ; θ ) results in a good approximation of the value function v ( s ) .

A fundamental and quite popular approach to finding a good approximation of the value function is known as temporal difference (TD) learning [6]. Suppose that a sample ⟨ s, r, s ′ ⟩ is given where s ′ ∼ P ( ·| s ) . In this case, TD learning algorithm updates the parameters of the approximate value function as follows:

where θ t denotes the parameters of our function approximator at iteration t . Also, by ∇ θ v ( s ; θ t ) we are denoting the partial gradient of v ( s ; θ ) with respect to the parameter θ . Note that TD uses

the value estimate obtained by one-step lookahead ( r + γv ( s ′ ; θ t ) ) to update its approximate value function v ( s ; θ t ) in the previous step. This one-step look-ahead of TD could be thought of as a sample of the right-hand-side of the Bellman equation. Our focus on TD is due to the fact that many of the more modern RL algorithms are designed based on the principles that TD embodies. We explain this connection more comprehensively in section 3.

To understand the existing results on TD convergence, we define the Markov chain's stationary state distribution d ( · ) as the unique distribution with the following property: ∀ s ′ ∑ s ∈S d ( s ) P ( s ′ | s ) = d ( s ′ ) . Then, Tsitsiklis and Van Roy show in [3] that, under linear function approximation, TD will be convergent if the states s are sampled from the stationary-state distribution. We will discuss this result in more detail later on in section 5.

However, it is well-known that linear TD can display divergent behavior if states are sampled from an arbitrary distribution rather than the stationary-state distribution of the Markov chain. A simple yet classical counter example of divergence of linear TD is shown in Figure 1 and investigated in section 4. First identified in [3], this example is a very simple Markov chain with two non-terminal states and zero rewards. Moreover, little is known about convergence guarantees of TD under alternative function approximators, or even when the update rule of TD is modified slightly.

In this paper, we focus on the Markov reward process (MRP) setting which could be thought of as a Markov decision process (MDP) with a single action. TD can display divergence even in the MRP setting [3, 5], which indicates that the presence of multiple actions is not the core reason behind TD misbehavior [5] - Divergence can manifest itself even in the more specific case of single action. Our results can naturally be extended to multiple actions with off-policy updates where in update (2) of TD, actions are sampled according to a target policy but states are sampled according to a behavior policy. That said, akin to Tsitsiklis and Van Roy [3], as well as chapter 11 in the book of Sutton and Barto [5], we focus on MRPs to study the root cause of TD in the most clear setting.

3 TD Learning as Iterative Optimization

In this section, we argue that common approaches to value function prediction can be viewed as iterative optimization algorithms where the function to be minimized changes per iteration. To this end, we recall that for a given experience tuple ⟨ s, r, s ′ ⟩ , TD performs the update presented in (2). A common augmentation of TD is to decouple the parameters to target ( θ ) and optimization ( w ) parameters [1] and update θ less frequently. In this case, at a given iteration t , the algorithm performs multiple ( K ) gradient steps as follows:

and then updates the target parameter θ t +1 ← w t,K before moving to the next iteration t +1 . Here K is a hyper-parameter, where K = 1 takes us back to the original TD update (2). Observe that the dependence of v ( s ′ ; θ t ) to our optimization parameter w is ignored in this update, despite the fact that an implicit dependence is present due to the final step θ t +1 ← w t,K . This means that the objective function being optimized is made up of two separate input variables 2 . We now define:

where we allow d ( · ) to be an arbitrary distribution, not just the stationary-state distribution of the Markov chain. Observe that the partial gradient of H with respect to the optimization parameters w is equivalent to the expectation of the update in (3). Therefore, TD, DQN, and similar algorithms could best be thought of as learning algorithms that proceed by approximately solving for the following sequence of optimization problems:

using first-order optimization techniques. This optimization perspective is useful conceptually because it accentuates the unusual property of this iterative process, namely that the first argument of the objective H hinges on the output of the previous iteration. It also has important practical ramifications when designing RL optimizers [11]. Moreover, the general form of this optimization

process allows for using alternative forms of loss functions such as the Huber loss [1], the logistic loss [12], or the entropy [13], as well as various forms of function approximation such as linear functions or deep neural networks. Each combination of loss functions and function approximators yields a different H , but one that is always comprised of a function H with two inputs.

A closely related optimization process is one where each iteration is solved exactly:

akin to Fitted Value Iteration [7, 14]. Exact optimization is doable in problems where the model of the environment is available and that the solution takes a closed form. A pseudo-code of both algorithms is presented in Algorithms 1 and 2.

Algorithm 1 Value Function Optimization with Exact Updates

Algorithm 2 Value Function Optimization with Gradient Updates

In terms of the difference between the two algorithms, notice that in Algorithm 1 we assume that we have the luxury of somehow solving each iteration exactly. This stands in contrast to Algorithm 2 where we may not have this luxury, and resort to gradient updates to find a rough approximation of the actual solution. Thus, Algorithm 1 is more difficult to apply but easier to understand, whereas Algorithm 2 is easier to apply but more involved in terms of obtaining a theoretical understanding.

Note that if convergence manifests itself in each of the two algorithms, the convergence point denoted by θ ⋆ must have the following property:

This fixed-point characterization of TD has been explored in previous work [15, 16]. Whenever it exists, we refer to θ ⋆ as the fixed-point of these iterative algorithms. However, convergence to the fixed-point is not always guaranteed [5] even when we have the luxury of performing exact minimization akin to Algorithm 1. In this paper, we study both the exact and inexact version of the optimization process. In doing so, we identify two forces that primarily influence convergence. To begin our investigation, we study a counter example to build intuition on why divergence can manifest itself. We present a formal analysis of the convergence of the two algorithms in section 6.

4 Revisiting the Divergence Example of TD

In this section, we focus on a simple counter example where TD is known to exhibit divergence. First identified by [3], investigating this simple example enables us to build some intuition about the root cause of divergence in the most clear setting.

Shown in Figure 1, this example is a Markov chain with two non-terminal states and zero rewards. A linear function approximation, v ( s ; θ ) = ϕ ( s ) θ , is employed with a single feature where ϕ ( s 1 ) = 1 and ϕ ( s 2 ) = 2 . The third state is a terminal one whose value is always zero. The true value function (0 in all states) is realizable with θ = 0 .

To build some intuition about the root cause of divergence, we discuss the convergence of exact TD with a few state distributions in this example. We desire to update all but the terminal state with non-zero probability. However, to begin with, we focus on a particular extreme case where we put all of our update probability behind the second state:

We thus have ∇ w H ( θ t , w ) = 2 ( 2 w -(1 -ϵ ) γ 2 θ t ) , and since ∇ w H ( θ t , θ t +1 ) = 0 , we can write: θ t +1 ← (2) -1 (1 -ϵ ) γ 2 θ t . The process converges to the fixed-point θ = 0 for all values of γ < 1

and ϵ ∈ [0 , 1] . This is because the target force of θ t (namely ( (1 -ϵ ) γ 2) ) is always dominated by the optimization force of w (namely 2). Note that updating this state is thus not problematic at all, and that the update becomes even more conducive to convergence when γ is smaller and ϵ is larger.

We now juxtapose the first extreme case with the second one where we put all of our update probability behind the first state, in which case at a given iteration t we have:

We thus have ∇ w H ( θ t , w ) = ( w -γ 2 θ t ) and so θ t +1 ← (1) -1 γ 2 θ t . Unlike the first extreme case, convergence is no longer guaranteed. Concretely, to ensure convergence, we require that the target force of θ t (namely γ 2 ) be larger than the optimization force of w (namely 1).

To better understand why divergence manifests itself, note that the two states s 1 and s 2 have very similar representations (in the form of a single feature). Note also that the value of the feature is larger

for the target state than it is for the source state, ϕ ( s 2 ) > ϕ ( s 1 ) . This means that any change in the value of the source state s 1 , would have the external effect of changing the value of the target state s 2 . Further, the change is in the same direction (due to the positive sign of the feature in both states) and that it is exactly 2 γ larger for the target state relative to the source state. Therefore, when γ > 1 / 2 , the function H will be more sensitive to the target parameter θ relative to the optimization parameter w . This is the root cause of divergence.

Moving to the case where we update both states with non-zero probability, first note that if the two updates were individually convergent, then the combination of the two updates would also have been convergent in a probability-agnostic way. Stated differently, convergence would have been guaranteed regardless of the probability distribution d and for all off-policy updates. However, in light of the fact that the optimization force of s 1 does not dominate its target force, we need to choose d ( s 1 ) small enough so as to contain the harmful effect of updating this state.

We compute the overall update (under the case where we update the two states with equal probability) by computing the sum of the two gradients in the two extreme cases above:

and so θ t +1 ← (5) -1 γ (6 -4 ϵ ) θ t . Note that even with a uniform update distribution, the two states are contributing non-uniformly to the overall update, and the update in the state s 2 is more influential because of the higher magnitude of the feature in this state ( ϕ ( s 2 ) = 2 against ϕ ( s 1 ) = 1 ).

To ensure convergence, we need to have that γ < 5 6 -4 ϵ . Notice, again, that we are combining the update in a problematic state with the update in the state that is conducive for convergence. We can contain the negative effect of updating the first state by ensuring that our update in the second state is even more conducive for convergence (corresponding to a larger ϵ ). In this case, the update of s 2 can serve as a mitigating factor.

We can further characterize the update with a general distribution. In this case, we have:

which gives us a convergent update if: d ( s 1 ) < 4 -4 γ (1 -ϵ ) 3+ γ 2 -4 γ (1 -ϵ ) . Both the denominator and the numerator are always positive, therefore regardless of the values of ϵ and γ , there always exists a convergent off-policy update, but one that needs to assign less probability to the problematic state as we increase γ and decrease ϵ .

This means that using some carefully chosen off-policy distributions is not only safe, but that doing so can even speed up convergence. This will be the case if the chosen distribution makes the objective function H more sensitive to changes in its first argument (target parameter θ ) than its second argument (the optimization parameter w ). We next formalize this intuition.

5 Convergence of Linear TD with Quadratic Loss

We now focus on the case of TD with linear function approximation. In this case, the expected TD update (2) could be written as the composition of two separate operators (with v t = Φ θ t ):

where the projection operator Π D and the Bellman operator T are defined as follows:

Here, D is a diagonal matrix with diagonal entries d ( s 1 ) , ..., d ( s n ) , where n is the number of states. The projection operator Π D is non-expansive under any distribution d . However, the Bellman operator is a γ -contraction under a specific d , namely the stationary-state distribution of the Markov chain specified by the transition matrix P as shown by Tsitsiklis and Van Roy [3]. Therefore, the composition of the two operators is a γ -contraction when the distribution d is the stationary-state distribution of the Markov chain.

In light of this result, one may think that TD is convergent in a very narrow sense, specifically when the updates are performed using the stationary-state distribution. But, as we saw with the counter example, in general there may exist many other distributions d that are in fact very conducive for TD convergence. In these cases, the proof technique above cannot provide a tool to ensure convergence because of the reliance of the non-expansive and contraction properties of these operators. Is this a limitation of the TD algorithm itself, or a limitation of this specific operator perspective of TD? Further, if this operator perspective is limited, is there a different perspective that can give rise to a more general understanding of TD convergence?

Our goal for the rest of the paper is to develop an alternative optimization perspective that can be applied in a broader setting relative to the operator perspective. To this end, our first task is to show that the optimization perspective can give us new insights even in the linear case and with squared loss functions. We generalize our optimization view of TD in the counter example by defining the objective function H for this case:

where || x || D = √ x ⊤ Dx . Recall that in the exact case, the TD algorithm could succinctly be written as: θ t +1 ← arg min w H ( θ t , w ) . Following the steps taken with the counter example, we first compute the gradient to derive the target and optimization forces:

Here we have similar dynamics between θ and w except that in this more general case, rather than scalar forces as in the counter example, we now have the two matrices M w and M θ . Note that M w is a positive definite matrix, λ min ( M w ) = min x x ⊤ M w x/ || x || 2 > 0 , if Φ is full rank. Below we can conveniently derive the update rule of linear TD by using these two matrices. All proofs are in the appendix.

Proposition 1. Let { θ t } t ∈ N be a sequence of parameters generated by Algorithm 1. Then, for the fixed-point θ ⋆ and any t ∈ N , we have

This proposition characterizes the evolution of the difference between the parameter θ t and the fixed-point θ ⋆ . Similarly to our approach with the counter example, we desire to ensure that this difference converges to 0 . The following corollary gives us a condition for convergence [17].

Corollary 2. Let { θ t } t ∈ N be a sequence of parameters generated by Algorithm 1. Then, { θ t } t ∈ N converges to the fixed-point θ ⋆ if and only if the spectral radius of M -1 w M θ satisfies ρ ( M -1 w M θ ) < 1 .

We can employ this Corollary to characterize the convergence of TD in the counter example from the previous section. In this case, we have M w = 5 and M θ = γ (6 -4 ϵ ) , which give us (5) -1 γ (6 -4 ϵ ) < 1 . This is exactly the condition obtained in the previous section.

Notice that if d is the stationary-state distribution of the Markov chain then ρ ( M -1 w M θ ) < 1 [3], and so the algorithm is convergent. However, as demonstrated in the counter example, the condition can also hold for many other distributions. The key insight is to ensure that the distribution puts more weight behind states where the optimization force of w is dominating the target force due to θ .

Note that the operator view of TD becomes inapplicable as we make modifications to H , because it will be unclear how to write the corresponding RL algorithm as a composition of operators. Can we demonstrate that in these cases the optimization perspective is still well-equipped to provide us with new insights about TD convergence? We next answer this question affirmatively by showing that the optimization perspective of TD convergence naturally extends to a broader setting than considered here, and therefore, is a more powerful perspective than the classical operator perspective.

6 Convergence of TD with General H

Our desire now is to show convergence of TD without limiting the scope of our results to linear approximation and squared loss. We would like our theoretical results to support alternative ways of constructing H than the one studied in existing work as well as our previous section. In doing so, we again show that convergence of TD hinges on the interplay between the two identified forces.

Before presenting our main theorem, we discuss important concepts from optimization that will be used at the core of our proofs. We study convergence of Algorithms 1 and 2 with a general function H : R n × R n → R that satisfies the following two assumptions:

The partial gradient ∇ w H , is F θ -Lipschitz:

The function H ( θ, w ) is F w -strongly convex in w :

Note that, in the specific linear case and quadratic loss (our setting in the previous section) these assumptions are easily satisfied [8]. More specifically, in that case F θ = λ max ( M θ ) and F w = λ min ( M w ) . But the assumptions are also satisfied in a much broader setting than before. We provide more examples in section 6.3. We are now ready to present the main result of our paper:

Theorem 3. Let { θ t } t ∈ N be a sequence generated by either Algorithm 1 or 2. If F θ < F w , then the sequence { θ t } t ∈ N converges to the fixed-point θ ⋆ .

In order to prove the result, we tackle the two cases of Algorithm 1 and 2 separately. We first start by showing the result for Algorithm 1 where we can solve each iteration exactly. This is an easier case to tackle because we have the luxury of performing exact minimization, which is more stringent and difficult to implement but easier to analyze and understand. This would be more pertinent to Fitted Value Iteration and similar algorithms. We then move to the case where we approximately solve each iteration (Algorithm 2) akin to TD and similar algorithms. The proof in this case is more involved, and partly relies on choosing small steps when performing gradient descent.

6.1 Exact Optimization (Algorithm 1)

In this case, convergence to the fixed-point θ ⋆ can be obtained as a corollary of the following result:

Proposition 4. Let { θ t } t ∈ N be a sequence generated by Algorithm 1. Then, we have:

From this result, we immediately obtain that the relative strength of the two forces, namely the conducive force F w due to optimization and the detrimental target force F θ , determines if the algorithm is well-behaved. The proof of Theorem 3 in this case follows immediately from Proposition 4.

6.2 Inexact Optimization (Algorithm 2)

So far we have shown that the optimization process is convergent in the presence of exact optimization. This result supports the soundness of algorithms such as Fitted Value Iteration, but not TD yet, because in the case of TD we only roughly approximate the minimization step. Can this desirable

convergence result be extended to the more general setting of TD-like algorithms where we inexactly solve each iteration by a few gradient steps, or is exact optimization necessary for obtaining convergence? Answering this question is very important because in many settings it would be a stringent requirement to have to solve the optimization problem exactly.

We now show that indeed convergence manifests itself in the inexact case as well. In the extreme case, we can show that all we need is merely one single gradient update at each iteration. This means that even the purely online TD algorithm, presented in update (2), is convergent with general H if the optimization force can dominate the target force.

However, it is important to note that because we are now using gradient information to crudely approximate each iteration, we need to ensure that the step-size parameter α is chosen reasonably. More concretely, in this setting we need an additional assumption, namely that there exists an L > 0 such that:

Notice that such an assumption is quite common in the optimization literature (see, for instance, [18]). Moreover, it is common to choose α = 1 /L , which we also employ in the context of Algorithm 2. We formalize this in the proposition presented below:

Proposition 5. Let { θ t } t ∈ N be a sequence generated by Algorithm 2 with the step-size α = 1 /L . Then, we have:

with κ ≡ L -1 F w and η ≡ F -1 w F θ .

Notice that κ , which is sometimes referred to as the inverse condition number in the optimization literature, is always smaller than 1. Therefore, we immediately conclude Theorem 3. Indeed, since the optimization force dominates the target force (meaning η < 1 ), Algorithm 2 is convergent. Notice that a surprisingly positive consequence of this theorem is that we get convergent updates even if we only perform one gradient step per iteration ( K = 1 ). In deep-RL terminology, this corresponds to the case where we basically have no frozen target network, and that we immediately use the new target parameter θ for the subsequent update.

To further situate this result, notice that as K approaches ∞ then σ K approaches η , which is exactly the contraction factor from Proposition 4 where we assumed exact optimization. So this proposition should be thought of as a tight generalization of Proposition 4 for the exact case. With a finite K we are paying a price for the crudeness of our approximation.

Moreover, another interesting reduction of our result is to the case where the target force due to bootstrapping in RL is absent, meaning that F θ ≡ 0 . In this case, the contraction factor σ K reduces to (1 -κ ) K/ 2 , which is exactly the known convergence rate for the gradient-descent algorithm in the strongly convex setting [18].

6.3 Examples

We focus on two families of loss functions where our assumptions can hold easily. To explain the first family, recall that TD could be written as follows:

Now suppose we can write the function H ( θ, w ) as the sum of two separate functions H ( θ, w ) = G ( θ, w ) + L ( w ) , where the function L ( w ) is strongly convex with respect to w . This setting is akin to using ridge regularization [19], which is quite common in deep learning (for example, the very popular optimizer AdamW [20]). This allows us to now work with functions G that are only convex (in fact, weakly convex is enough) with respect to w . We provide two examples:

Suppose we would like to stick with linear function approximation. Then, the function G could be constructed using any convex loss where ∇ w G ( θ, w ) is Lipschitz-continuous with respect to θ . Examples that satisfy this include the Huber loss [1] or the Logistic loss [12].

where:

Suppose we want to use the more powerful convex neural networks [21]. We need the error to be convex and monotonically increasing so that G is still convex. This is due to the classical result on the composition of convex functions. One example is the quadratic error where we restrict the output of the function approximator to positive values. Such neural nets are also Lipschitz continuous given proper activation functions such as ReLU.

Beyond this family, we have identified a second family, namely the control setting where a greedification operator is needed for bootstrapping. For example, with the quadratic error we could have:

where q is the state-action value function parameterized by either θ or w .

Weagain need the two assumptions, namely strong-convexity with respect to w and Lipschitzness of ∇ w H ( θ, w ) with respect to θ to hold. Actually, Lee and He [8] already showed the strong convexity with respect to w , but we need to still show the Lipschitz property of ∇ w H ( θ, w ) with respect to θ . Note that they showed the Lipschitz property only with respect to w and not with respect to θ . We are now able to show this result. Please see Proposition 8 and its proof in the appendix. Our proof also supports other greedification operators, such as softmax [22], so long as its non-expansive.

7 Related Work

In this paper, we studied convergence of TD through the lens of optimization. The underlying principles of TD are so central to RL that a large number of RL algorithms can be thought of as versions of TD, and the availability of convergence results varies between different types of algorithms. We chose a setting we believe is as elementary as possible to highlight key principles of TD convergence. Closest to our work is the work of Tsitsiklis and Van Roy [3] who proved convergence of TD with linear function approximation, squared error, and the stationary-state distribution of the Markov chain. Later work, especially that of Bhandari et al. [23], further analyzed the case of TD with stochastic gradients and non-iid samples in the on-policy case. Extensions to the control setting also exist [24, 25].

In particular, Melo et al. [24] presented a condition in their equation (7) which may look similar to our condition at first glance, but it is in fact quite different than the condition identified in our paper. In their equation (7), they require that the eigenvalues of Σ π , the policy-conditioned covariance matrix Φ ⊤ D Φ , dominate the eigenvalues of a second matrix γ 2 Σ ⋆ π ( θ ) . Here Σ ⋆ π ( θ ) is a similar covariance matrix for features, but one that is computed based on the action-greedification step.

We believe to be the first paper to show a exact contraction for TD with frozen target network and general K . To the best of our knowledge, existing results prior to Lee and He [8] mainly considered the case where we either never freeze the target network (corresponding to the value of K = 1 ), or the somewhat unrealistic case where we can exactly solve each iteration. Lee and He showed guarantees pertaining to K > 1 , but notice that, while their result is quite innovative, they leaned on the more standard optimization tools for ensuring that gradient descent with a fixed K can only solve each iteration approximately. Therefore, each iteration results in some error. In their theory, this error is accumulated per iteration and needs to be accounted for in the final result. Therefore, they fell short of showing 1) contraction and 2) exact convergence to the TD fixed-point, and only show that the final iterate is in the vicinity of the fixed-point defined by the amount of error accumulated over the trajectory. With finite K , they need to account for errors in solving each iteration (denoted by ϵ k in their proofs such as in Theorem 3), which prevent them from obtaining a clean convergence result. In contrast, we can show that even approximately solving each iteration (corresponding to finite K ) is enough to obtain contraction (without any error), because we look at the net effect of K updates to the online network and the single update to the target network, and show that this effect is on that always take us closer to the fixed-point irregardless of the specific value of K .

Modifications of TD are presented in prior work that are more conducive to convergence analysis [26, 27, 28, 29, 30]. They have had varying degrees of success both in terms of empirical performance [31], and in terms of producing convergent algorithms [32]. TD is also studied under over-parameterization [33, 34], with learned representations [35], proximal updates [36], and auxiliary tasks [37]. Also, the quality of TD fixed-point has been studied in previous work [16].

Alarge body of literature focuses on finding TD-like approaches that are in fact true gradient-descent approaches in that they follow the gradient of a stationary objective function [38, 15, 39, 40, 41, 42].

In these works, the optimization problem is formulated in such a way that the minimizer of the loss will be the fixed-point of the standard TD. Whereas TD has been extended to large-scale settings, these algorithms have not been as successful as TD in terms of applications.

A closely related algorithm to TD is that of Baird, namely the residual gradient algorithm [4, 43]. This algorithm has a double-sampling issue that needs to be addressed either by assuming a model of the environment or by learning the variance of the value function [43, 44]. However, even with deterministic MDPs, in which the double sampling issue is not present [45], the algorithm often finds a fixed-point that has a lower quality than that of the TD algorithm [46]. This is attributed to the fact that the MDP might still look stochastic in light of the use of function approximation [38].

TDcould be thought of as an incremental approach to approximate dynamic programming and Fitted Value Iteration [7] for which various convergence results based on the operator view exits [14, 47, 48]. Also, these algorithm are well-studied in terms of their error-propagation behavior [49, 50, 51].

Many asymptotic or finite-sample results on Q-learning (the control version of TD) with function approximation make additional assumptions on the problem structure, with a focus on the exploration problems [52, 53, 54, 55, 56]. Like mentioned before, our focus was on the prediction setting where exploration is not relevant.

8 Open Questions

To foster further progress in this area we identify key questions that still remain open.

We showed that TD converges to the fixed-point θ ⋆ characterized by ∇ w H ( θ ⋆ , θ ⋆ ) = 0 and so the sequence {∥∇ w H ( θ t , θ t ) ∥} t ∈ N converges to 0. Can we further show that this sequence monotonically decreases with t , namely that |∥∇ w H ( θ t +1 , θ t +1 ) ∥ ≤ ∥∇ w H ( θ t , θ t ) ∥ for all t ∈ N ? Interestingly, a line of research focused on inventing new algorithms (often called gradient TD algorithms) that possess this property [57]. However, if true, this result would entail that when our assumptions hold the original TD algorithm gives us this desirable property for free.

Going back to the counter example, we can show convergent updates by constraining the representation as follows: ϕ ( s ) ≤ γϕ ( s ′ ) whenever P ( s ′ | s ) > 0 . This result can easily be extended to all MDPs with a single-dimensional feature vector. A natural question then is to identify a multi-dimensional representation, given a fixed MDP and update distribution D , leads into convergent updates. Similarly, we can pose this question in terms of fixing the MDP and the representation matrix Φ , and identifying a set of distributions that lead into convergent updates. Again, we saw from the counter example that putting more weight behind states whose target force is weak (such as states leading into the terminal state) is conducive to convergence. How do we identify such distributions systematically?

We studied the setting of deterministic updates. To bridge the gap with RL practice, we need to study TD under stochastic-gradient updates. The stochasticity could for instance be due to using a minibatch or using non-iid samples. Bhandari et al. [23] tackle this for the case of K = 1 . Can we generalize our convergence result to this more realistic setting?

We have characterized the optimization force using the notion of strong convexity. It would be interesting to relax this assumption to handle neural networks and alternative loss functions. This can serve as a more grounded explanation for the empirical success of TD in the context of deep RL. Can we generalize our result to this more challenging setting?

9 Conclusion

In this paper, we argued that the optimization perspective of TD is more powerful than the wellexplored operator perspective of TD. To demonstrate this, we generalized previous convergence results of TD beyond the linear setting and squared loss functions. We believe that further exploring this optimization perspective can be a promising direction to design convergent RL algorithms.

Our general result on the convergent nature of TD is consistent with the empirical success and the attention that this algorithm has deservedly received. The key factor that governs the convergence of TD is to ensure that the optimization force of the algorithm is well-equipped to dominate the more harmful target force. This analogy is one that can be employed to explain the convergent nature of TD even in the presence of the three pillar of the deadly triad.

References

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature , 2015.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature , 2016.

John Tsitsiklis and Benjamin Van Roy. Analysis of temporal-diffference learning with function approximation. Advances in neural information processing systems , 1996.

Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Machine Learning . Elsevier, 1995.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Richard S Sutton. Learning to predict by the methods of temporal differences. Journal of Machine Learning Research , 1988.

Geoffrey J Gordon. Stable function approximation in dynamic programming. In Machine learning . Elsevier, 1995.

Donghwan Lee and Niao He. Target-based temporal-difference learning. In International Conference on Machine Learning , 2019.

Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons, 2014.

Hamid Reza Maei. Gradient temporal-difference learning algorithms . PhD thesis, University of Alberta, 2011.

Kavosh Asadi, Rasool Fakoor, and Shoham Sabach. Resetting the optimizer in deep RL: An empirical study. Advances in Neural Information Processing Systems , 2023.

Joan Bas-Serrano, Sebastian Curi, Andreas Krause, and Gergely Neu. Logistic Q-learning. In International Conference on Artificial Intelligence and Statistics , 2021.

Doina Precup and Richard S Sutton. Exponentiated gradient methods for reinforcement learning. In International Conference on Machine Learning , 1997.

Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research , 2005.

Hamid Reza Maei, Csaba Szepesv´ ari, Shalabh Bhatnagar, and Richard S Sutton. Toward offpolicy learning control with function approximation. International Conference on Machine Learning , 2010.

J Kolter. The fixed points of off-policy td. Advances in Neural Information Processing Systems , 2011.

Carl D Meyer. Matrix analysis and applied linear algebra . SIAM, 2000.

Amir Beck. First-order methods in optimization . SIAM, 2017.

Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological) , 1996.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019.

Brandon Amos, Lei Xu, and J Zico Kolter. Input convex neural networks. In International Conference on Machine Learning , 2017.

Kavosh Asadi and Michael L Littman. An alternative softmax operator for reinforcement learning. In International Conference on Machine Learning , 2017.

Jalaj Bhandari, Daniel Russo, and Raghav Singal. A finite time analysis of temporal difference learning with linear function approximation. In Conference on learning theory , 2018.

Francisco S Melo, Sean P Meyn, and M Isabel Ribeiro. An analysis of reinforcement learning with function approximation. In Proceedings of the 25th international conference on Machine learning , 2008.

Shaofeng Zou, Tengyu Xu, and Yingbin Liang. Finite-sample analysis for sarsa with linear function approximation. Advances in neural information processing systems , 2019.

Richard S Sutton, A Rupam Mahmood, and Martha White. An emphatic approach to the problem of off-policy temporal-difference learning. Journal of Machine Learning Research , 2016.

Shangtong Zhang, Hengshuai Yao, and Shimon Whiteson. Breaking the deadly triad with a target network. In International Conference on Machine Learning , 2021.

Raghuram Bharadwaj Diddigi, Chandramouli Kamanchi, and Shalabh Bhatnagar. A convergent off-policy temporal difference algorithm. arXiv , 2019.

Han-Dong Lim, Do Wan Kim, and Donghwan Lee. Regularized Q-learning. arXiv , 2022.

Sina Ghiassian, Andrew Patterson, Shivam Garg, Dhawal Gupta, Adam White, and Martha White. Gradient temporal-difference learning with regularized corrections. In International Conference on Machine Learning , 2020.

Sina Ghiassian, Banafsheh Rafiee, and Richard S Sutton. A first empirical study of emphatic temporal difference learning. arXiv , 2017.

Gaurav Manek and J Zico Kolter. The pitfalls of regularization in off-policy TD learning. Advances in Neural Information Processing Systems , 2022.

Qi Cai, Zhuoran Yang, Jason D Lee, and Zhaoran Wang. Neural temporal-difference learning converges to global optima. Advances in Neural Information Processing Systems , 2019.

Chenjun Xiao, Bo Dai, Jincheng Mei, Oscar A Ramirez, Ramki Gummadi, Chris Harris, and Dale Schuurmans. Understanding and leveraging overparameterization in recursive value estimation. In International Conference on Learning Representations , 2021.

Dibya Ghosh and Marc G Bellemare. Representations for stable off-policy reinforcement learning. In International Conference on Machine Learning , 2020.

Kavosh Asadi, Rasool Fakoor, Omer Gottesman, Taesup Kim, Michael Littman, and Alexander J Smola. Faster deep reinforcement learning with slower online network. Advances in Neural Information Processing Systems , 2022.

Clare Lyle, Mark Rowland, Georg Ostrovski, and Will Dabney. On the effect of auxiliary tasks on representation dynamics. In International Conference on Artificial Intelligence and Statistics , 2021.

Richard S Sutton, Hamid Reza Maei, Doina Precup, Shalabh Bhatnagar, David Silver, Csaba Szepesv´ ari, and Eric Wiewiora. Fast gradient-descent methods for temporal-difference learning with linear function approximation. In International Conference on Machine Learning , 2009.

Bo Liu, Sridhar Mahadevan, and Ji Liu. Regularized off-policy TD-learning. Advances in Neural Information Processing Systems , 2012.

Bo Liu, Ji Liu, Mohammad Ghavamzadeh, Sridhar Mahadevan, and Marek Petrik. Proximal gradient temporal difference learning algorithms. In International Joint Conferences on Artificial Intelligence , 2016.

Bo Liu, Ji Liu, Mohammad Ghavamzadeh, Sridhar Mahadevan, and Marek Petrik. Finitesample analysis of proximal gradient TD algorithms. arXiv , 2020.

Yihao Feng, Lihong Li, and Qiang Liu. A kernel loss for solving the Bellman equation. Advances in Neural Information Processing Systems , 2019.

Andr´ as Antos, Csaba Szepesv´ ari, and R´ emi Munos. Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path. Journal of Machine Learning Research , 2008.

Bo Dai, Albert Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, and Le Song. Sbeed: Convergent reinforcement learning with nonlinear function approximation. In International Conference on Machine Learning , 2018.

Ehsan Saleh and Nan Jiang. Deterministic Bellman residual minimization. In Proceedings of Optimization Foundations for Reinforcement Learning Workshop at NeurIPS , 2019.

Michail G Lagoudakis and Ronald Parr. Least-squares policy iteration. Journal of Machine Learning Research , 2003.

Daniel Lizotte. Convergent fitted value iteration with linear function approximation. Advances in Neural Information Processing Systems , 2011.

Jianqing Fan, Zhaoran Wang, Yuchen Xie, and Zhuoran Yang. A theoretical analysis of deep Q-learning. In Learning for Dynamics and Control , 2020.

R´ emi Munos. Performance bounds in l p-norm for approximate value iteration. SIAM journal on control and optimization , 2007.

R´ emi Munos and Csaba Szepesv´ ari. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research , 2008.

Amir-massoud Farahmand, Csaba Szepesv´ ari, and R´ emi Munos. Error propagation for approximate policy and value iteration. Advances in Neural Information Processing Systems , 2010.

Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory , 2020.

Simon S Du, Yuping Luo, Ruosong Wang, and Hanrui Zhang. Provably efficient Q-learning with function approximation via distribution shift error checking oracle. Advances in Neural Information Processing Systems , 2019.

Simon S Du, Jason D Lee, Gaurav Mahajan, and Ruosong Wang. Agnostic Q-learning with function approximation in deterministic systems: Near-optimal bounds on approximation error and sample complexity. Advances in Neural Information Processing Systems , 2020.

Alekh Agarwal, Sham Kakade, Akshay Krishnamurthy, and Wen Sun. Flambe: Structural complexity and representation learning of low rank mdps. Advances in neural information processing systems , 2020.

Shuhang Chen, Adithya M Devraj, Fan Lu, Ana Busic, and Sean Meyn. Zap q-learning with nonlinear function approximation. Advances in Neural Information Processing Systems , 2020.

Richard S Sutton, Hamid Maei, and Csaba Szepesv´ ari. A convergent o ( n ) temporal-difference algorithm for off-policy learning with linear function approximation. Advances in neural information processing systems , 2008.

10 Appendix

Proposition 1. Let { θ t } t ∈ N be a sequence of parameters generated by Algorithm 1. Then, for the fixed-point θ ⋆ and any t ∈ N , we have

Proof. Since θ t +1 ← arg min w H ( θ t , w ) , we have ∇ w H ( θ t , θ t +1 ) = 0 . Using (7) we have:

where the last equality follows from θ ⋆ being the fixed-point and therefore ∇ w h ( θ ⋆ , θ ⋆ ) = 0 , which in light of (7) translates to Φ ⊤ DR = ( M w -M θ ) θ ⋆ . Multiplying both sides by M -1 w , we get:

Rearranging the equality leads into the desired result.

Proposition 4. Let { θ t } t ∈ N be a sequence generated by Algorithm 1. Then, we have:

Proof. First notice that θ t +1 ← arg min w H ( θ t , w ) , so we have ∇ w H ( θ t , θ t +1 ) = 0 . Now, using the F w -strong convexity of w → H ( θ t , w ) , we get that

Now, since θ ⋆ is a fixed-point, it follows that ∇ w H ( θ ⋆ , θ ⋆ ) = 0 . Therefore, we have:

̸

where in the second line we used to Cauchy-Schwartz inequality, and the last inequality follows from the F θ -Lipschitz property of ∇ w H ( · , θ ⋆ ) . Since this inequality holds true for any t ∈ N , we get that if θ t = θ ⋆ , then we also have that θ t +1 = θ ⋆ . Thus, if θ T = θ ⋆ for some T ∈ N , then θ t = θ ⋆ for any t ≥ T and so the algorithm has converged. On the other hand, if θ t = θ ⋆ for all t ∈ N , the desired result follows after dividing both sides by ∥ θ t +1 -θ ⋆ ∥ .

To prove Proposition 5 we need to provide two new Propositions to later make use of. We present these two proofs next, and then restate and prove Proposition 5.

Proposition 6. Let { θ t } t ∈ N be a sequence generated by Algorithm 2. Then, for all t ∈ N and 0 ≤ k ≤ K -1 , we have

Proof. Let t ∈ N . From the F w -strong convexity of w → H ( θ t , w ) , we obtain that

which means that

Moreover, in light of the fixed-point characterization of TD, namely that ∇ w H ( θ ⋆ , θ ⋆ ) = 0 , we can write

Here, the first inequality follows from the fact that for any two vectors a and b we have a ⊤ b ≤ (1 / 2 d ) ∥ a ∥ 2 +( d/ 2) ∥ b ∥ 2 for any d > 0 . In this case, we chose d = F w . Also the last inequality follows from the F θ -Lipschitz property of ∇ w H , which is our assumption.

Now, by combining (8) with (9) we obtain that

From the Lipschitz assumption we can write, due to the Descent Lemma [18] applied to the function w → H ( θ t , w ) , that:

Now, notice that according to Algorithm 2 we have w t,k +1 = w t,k -α ∇ w H ( θ t , w t,k ) , and so we can write:

Adding both sides of (10) with (11) yields:

which proves the desired result.

Now, we can prove the following result.

Proposition 7. Let { θ t } t ∈ N be a sequence generated by Algorithm 2. Then, for all t ∈ N and 0 ≤ k ≤ K -1 , we have

In particular, when α = 1 /L , we have

Proof. Let t ∈ N . From the definition of steps of Algorithm 2, that is, w t,k +1 = w t,k -α ∇ w H ( θ t , w t,k ) , for any 0 ≤ k ≤ K -1 , we obtain that

Using the F w -strong convexity of w → H ( θ t , w ) , we have

Combining (12) and (13), we get:

Hence, from Proposition 6, we obtain

which completes the first desired result.

Moreover, by specifically choosing the step-size α = 1 /L we obtain that:

This concludes the proof of this proposition.

Using these two results, we are ready to present the proof of Proposition 5

Proposition 5. Let { θ t } t ∈ N be a sequence generated by Algorithm 2 with the step-size α = 1 /L . Then, we have:

where:

with κ ≡ L -1 F w and η ≡ F -1 w F θ .

Proof. Let t ∈ N . From Proposition 7 (recall that α = 1 /L ) and the fact that θ t +1 = w t,K , we have

where the last inequality follows from the fact that w t, 0 = θ t . Because κ ∈ [0 , 1] , the geometric series on the right hand side is convergent, and so we can write:

which completes the desired result.

Now, following our discussion in Section 6.3 about examples of loss functions that satisfy our assumption, we recall that here we focus on the following loss

As mentioned in Section 6.3 all is left is to show that Lipschitzness of ∇ w H ( θ, w ) with respect to θ .

Proposition 8. Assume that for any ( s, a ) , the function θ → q ( s, a, θ ) is L q ( s, a ) -Lipschitz. Then, there exists F θ > 0 such that

Proof. First, we compute the gradient ∇ w H :

For the simplicity of the proof, we denote Q ( θ 1 , θ 2 ) = max a ′ q ( s ′ , a ′ , θ 2 ) -max a ′ q ( s ′ , a ′ , θ 1 ) . Hence

where we first cancelled the common parts in both gradient terms and the inequality follows immediately from the Cauchy-Schwartz inequality. Now, by using Jensen's inequality on the expectation, we obtain that

Moreover, without the loss of generality we can assume that max a ′ q ( s ′ , a ′ , θ 2 ) ≥ max a ′ q ( s ′ , a ′ , θ 1 ) , which means that we can remove the absolute value and obtain the following

Now, by using a simple property of the maximum operation, we obviously have that

Since the function θ → q ( s ′ , a ′ , θ ) is Lipschitz, we have that

It is also well-known that for such functions the gradient is bounded and therefore ∥∇ q ( s, a, w ) ∥ ≤ L q ( s, a ) . Combining all these facts yield

where the last equality follows from the basic properties of the distribution matrix D . This proves the desired result.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the TD learning algorithm from an optimization point of view which differs from the more classical fixed point Bellman operator point of view. The goal of the paper is to argue that this other viewpoint permits a better understanding of TD learning and a generalization of its convergence results explaining its practical success. The investigation of the known counterexample of TD divergence allows to identify the interplay of two forces that determine the  convergence behavior of TD learning, namely a so called target force and an optimization force. It is shown that TD shows a convergent behavior when the optimization force dominates the target one. These insights are then instantiated for linear function approximation with square loss and beyond under strong convexity and smoothness assumptions, even under the celebrated deadly triad.\n\nSTRENGTHS:\n- Understanding the behavior of the celebrated TD learning algorithm in the deadly triad setting and beyond the linear function approximation setting  is an important research goal given the popularity of the algorithm and its potential impact in RL. \n\n- Interesting insights starting from the simple counter example in Section 4 are provided and clearly explained. \n\n- The paper is very well-written, well-organized and overall easy to follow. To the best of my knowledge, proofs (including the appendix) are correct and cleanly presented.\n\nWEAKNESSES:\n1. It is not made very clear that the scheme proposed is actually different from the classical TD learning which was analyzed in [3] since the iterate $\\theta_t$ of the target network is frozen. The paper considers a ‘target-based’ version of TD learning which was inspired by the DQN algorithm using target networks [1].  \n\n2. Since one of the motivations of the paper is to show that the optimization point of view allows to address more general settings than the linear function approximation setting with square loss, I would expect a more detailed discussion of these cases in Section 6 giving the definitions of the $H$ function in that case and verifying the uniform assumptions 1 and 2 of Section 6. The discussion in l. 277 to 280 is quite minimal. The generalization does still seem a bit restrictive and the analysis provides sufficient conditions that do not close the question of the understanding of the divergence behavior of TD learning. See also the Questions section. \n\n3. Although the interpretation in terms of ‘target force’ and ‘optimization force’ has not been described as such in prior work to the best of my knowledge, arguments to show the results are quite classical and technical novelty is very limited in my opinion. The proofs of section 5 rely on the classical stability criteria in control for linear systems and was also used in other works ([16], see also discussions below about related works) even if the possibility of using other distributions instead of the stationary one was not described.The proofs for Section 6 follow standard analysis of gradient descent like algorithms for smooth and strongly convex objectives up to the drifting parameter $\\theta_t$ which is periodically synchronized with the online iterate. \n\n4. **Related work**: Closely related works are not discussed in detail, especially those analyzing the ‘target-based’ version of TD learning which is central in this work. \n\n**(a)** While the present work indeed provides some new insights, the optimization perspective proposed in this work is not completely new and I believe some additional discussion regarding this would be welcome. As a matter of fact [16] is only briefly mentioned in l. 275 whereas the optimization point of view is alluded to in the remark in Section 2.4  of [16] (see also sections 3 and 5 therein) where the modified version of the Mean Square Bellman Operator with two variables  (target and online) clearly appear. While the generalization beyond linear function approximation and the squared loss (under some Lipschitzness and strong-convexity assumptions) and the flexibility to consider a different distribution from the Markov chain’s stationary distribution are interesting  insights, the results of Proposition 1 and Corollary 2 are not very novel. For instance, instead of periodically synchronizing the target network with the online one as in Algorithm 2, one could also consider the online moving average update rule proposed in the popular DDPG algorithm (Lillicrap et al. 2016) which was analyzed in the linear function approximation on-policy setting in [16, sections 2, 3, 5] and in Barakat et al. 2022 (see Sections 4.2, 5.1, 6.1).  The aforementioned results also provide almost convergence results and sample complexity analysis accounting for noisy settings unlike the present work. \n\nLillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning, ICLR 2016.\n\nBarakat, A., Bianchi, P., Lehmann, J. Analysis of a Target-Based Actor-Critic Algorithm with Linear Function Approximation, AISTATS 2022.\n\n**(b)** If one of the main motivations or consequences of this work is to show that the ‘optimization viewpoint’ can explain the possible convergent nature of TD learning in the presence of the deadly triad as mentioned in the conclusion (under some assumptions and some suitable choice of the sampling distribution in the off-policy case), then, the ability of the target-based updates (which is actually the ‘optimization point of view’) was also advocated for in Zhang et al. 21 [25] at least in the linear function approximation setting (see Section 4 for off-policy evaluation with Q functions which could be easily adapted to V functions). \n\n**(c)** Further works such as Liu and Olshevsky 2021 could also be relevant to mention. This work points out that original TD learning (i.e. Eq. (2), as proposed in [6] and analyzed in [3]) can be seen as what they call a ‘gradient splitting’ (see Section 3 therein) even if it is known that the TD learning update rule does not correspond to any gradient descent over any function.   \n\nLiu, R., Olshevsky, A. Temporal Difference Learning as Gradient Splitting, ICML 2021.\n\n**(d)** Several works have also considered the analysis of TD learning with nonlinear function approximation beyond the linear setting (see e.g., Brandfonbrener and Bruna 2020; Agazzi and Lu 2021 to name a few). A discussion about these works seems also relevant given the generalization motivation of the present work.  \n\nBrandfonbrener, D., Bruna, J. Geometric insights into the convergence of nonlinear TD learning, ICLR 2020.  \n\nAgazzi, A., Lu, J. Temporal-difference learning with nonlinear function approximation: lazy training and mean field regimes, MSML 2021.\n\n**Minor typos:** \nl. 104: ‘the root cause of TD’, of divergence?  \nl. 242: capital $H$ instead of $h$\nl. 541-542: $\\theta^{\\star}$ instead of  $\\theta^{*}$\n\nQUESTIONS:\nIn the following, I list a few questions of which the first two are the main ones, focusing on the limit point definition and the strength of the assumptions. \n\n1. All the theoretical results show convergence to the fixed-point $\\theta^{\\star}$. How is this point defined in those results?  Eq. (5) mentions that if convergence happens then $\\nabla_{w} H(\\theta^{\\star}, \\theta^{\\star}) = 0$.  Such a characterization (which is also the fixed-point characterization of the TD solution) is used in all the proofs. Then l. 142-143 precise that ‘Whenever it exists, we refer to $\\theta^{\\star}$ as the fixed point of these iterative algorithms’. For the convergence results to be meaningful, the existence of the fixed point should be guaranteed. I guess you define $\\theta^{\\star}$ to be the fixed point of the projected Bellman operator which is indeed a contraction under some conditions but this is not very clear in the paper. However, as alluded to in the paper in l. 212-218, it is not clear whether the projected Bellman operator is still a contraction when one considers a distribution different from the stationary state distribution of the Markov Chain. What would then be the limit point(s) in the results in the case where existence is not guaranteed by the fixed point arguments relying on the operator viewpoint? Do you just suppose in that case that there exists a unique point such that $\\nabla_{w} H(\\theta^{\\star}, \\theta^{\\star}) = 0$ (which is what is required and used to conduct the proofs)? \n\n2. Concerning the assumptions and the examples provided in Section 6, input-convex neural networks [18] provide convex functions with respect to the inputs and not with respect to their weights (parameters). How these would help to guarantee that the strong convexity assumption 2 holds? Satisfying both assumptions does not seem straightforward. The constants $F_{\\theta}$ and $F_{\\omega}$ are uniform constants over $\\theta$ and $w$ that are not easy to define and compute in practice which makes the core stability condition $F_{\\theta} < F_{w}$ difficult to verify. I understand though that the focus of the paper is theoretical. Even in the linear setting and assuming that the feature matrix is given and known, can we for example suggest some distributions beyond the stationary state distribution for which the condition holds?  \n\n3. Regarding section 4, when you mention that the counter example was identified by [3], are you referring to Section IX in Tsitsiklis and Van Roy 97 (TAC) for this? Is it a simplified/modified example of that one? \n\n4. Could you mention the stationary state distribution of the Markov chain for the counter example? This could be added if relevant to show that this stationary distribution is indeed valid and leads to a convergent behavior as expected.\n\nLIMITATIONS:\nBeyond the points raised above, one limitation that is not clearly mentioned is that the convergence results are limited to the deterministic setting.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies convergence of the TD algorithm from the perspective of solving a shifting optimization problem. Through a classic failure case, the authors uncover two forces, whose interplay reveals TD's convergence properties. These two forces both depend on the state visitation distribution, the state features, and the transition kernel. The authors point out that while the stationary-state distribution ensures convergence, this does not mean no other state visitation distributions can and seek to establish sufficient properties. They generalize their analysis to TD error defined with more general functions and provide these sufficient conditions. Assuming the TD error $H$ has a value function gradient that is Lipschitz in the target function parameters and is also strongly convex in the value function parameters, a more general convergence criterion can be derived. The authors extend this result to the setting where the shifting optimization problem is only solved approximately at each step using $K$ gradient updates.\n\nSTRENGTHS:\nI found this paper to be very clearly written. To my knowledge, the claim in Proposition 1 and subsequent results are novel and make significant progress towards understanding convergence of the TD algorithm.\n\nWEAKNESSES:\nThe paper does a great job explaining and motivating the novel TD convergence analysis, and I believe it can stand on its own as a \"theory paper\". But I think experimental support for the approach in a complex (e.g., deep RL) setting could have made the paper much stronger.\n\nI am also not entirely clear which results are novel and can be attributed to the authors vs which were known beforehand. Equation (7) does not appear to be novel; some form exists in [35], sec 4. The paper could benefit from better signposting to explain where others got stuck and this paper advances.\n\nQUESTIONS:\n- line 164: where does the $1/2$ come from in the equation? If that is there for convenience, can you add a $1/2$ to the def of $H$ under line 117?\n- line 237: $M_w$ is positive definite if $\\Phi$ is full rank **AND** $D$ is full-rank, i.e., every state has positive probability under $d(s)$, no?\n- line 320: $L$ here is typically known as the strong-smoothness parameter, no? In contrast to the strong-convexity parameter.\n- line 330: the condition number I'm familiar with (see [1]) is always greater than or equal to $1$ (max eig / min eig). In this case, I think you mean the \"inverse condition number\". Also, this technically means, $\\sigma_K^2 = 1$ when $\\kappa=1$, in which case, the analysis does not imply convergence. Can you please discuss this corner case?\n- line 370: \"exits\" --> \"exists\"\n\n[1] Guille-Escuret, Charles, et al. \"A study of condition numbers for first-order optimization.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2021.\nhttp://proceedings.mlr.press/v130/guille-escuret21a/guille-escuret21a.pdf\n\nLIMITATIONS:\nI see no need for discussion of negative societal impact in this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies TD-learning with target network update. The authors recast the TD-learning algorithm into a time-varying optimization problem. The authors proves convergence for a function class with strong convexity and smoothness.\n\nSTRENGTHS:\nThe paper is easy to follow and the motivation of the work is well explained by a simple example form $\\theta\\to2\\theta$. Moreover, the viewpoint of target force and optimization force seems to novel viewpoint, and the theoretical result seems to be solid.\n\nWEAKNESSES:\n1. Assuming strong convexity and lipschitzness is too restrictive to argue for a general function class. Moreover, regarding the condition $F_{\\theta}<F_{w}$ in Theorem 3, I believe this is the key condition for the convergence but the discussion seems to be missing whether it is a common condition to be met or not. \n\n2. The analysis on iterative optimization objective for tabular and linear case has been studied in Lee et al. and Zhang et al.. The comparison with the existing work seems to be insufficient.\n\nQUESTIONS:\n1. How strict is the condition $F_{\\theta}<F_{w}$ in Theorem 3? Can we find any examples other than tabular or linear setting to show the convergence under general setting? \n\n2. Can we also find conditions for to ensure the convergence for the Baird example?\n\nLIMITATIONS:\nIn summary, the impact of theoretical result is not sufficient for the following reasons:\n\n- Assumption on strong convexity is too strong.\n\n- There are no examples or experiments showing convergence of general function class other than tabular or linear setting.\n\n- There are no compariosn between the existing works Lee et al., and Chen et al..\n\nHence, I am leaning towards rejection as for now.\n\nLee, Donghwan, and Niao He. \"Target-based temporal-difference learning.\" International Conference on Machine Learning. PMLR, 2019.\n\nChen, Zaiwei, John Paul Clarke, and Siva Theja Maguluri. \"Target Network and Truncation Overcome The Deadly Triad in $ Q $-Learning.\" arXiv preprint arXiv:2203.02628 (2022).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the convergence conditions for Temporal Difference (TD) learning utilizing a target network, and further extends its findings to scenarios where TD minimizes alternative losses beyond mean square errors. The study is conducted by formulating TD updates as iterative optimizations, under the help of a target network. Notably, the paper provides an intuitive description of the coefficients before the student's TD network parameter and the target network parameter as \"optimization force\" and \"target force,\" respectively. The paper reaches the conclusion that when optimization force dominants, the algorithm converges.\n\nSTRENGTHS:\n(1) The paper is of high quality and clarity. The authors have provided a clear setting, accompanied by well-presented proofs and well-defined assumptions. The demonstration of the counterexample is worth mentioning, as it effectively and intuitively introduces the main idea of the paper.\n\n(2) The paper extends current studies of TD to cases where alternative losses, such as Huber loss, are used, closing a gap between the theory and practical algorithms.\n\nWEAKNESSES:\n(1) The paper focuses on Markov Reward Process, which is not a common setting for TD convergence proof. Why authors did not focus on expected updates? Could authors provide more reasoning for their choice?\n\n(2) The paper re-forms TD updates as iteration optimizations with the help of a target network. Could authors add more comparisons to the convergence results of TD with a target network? For example, \nBreaking the Deadly Triad with a Target Network, Zhang et al. (2021)\nTarget-Based Temporal-Difference Learning, Lee and He (2019)\n\n(3) Some empirical results to show that the empirical contraction factor aligns with their theory findings would be great. \n\n(4) The paper proposes an interesting point: for some safe state distribution, TD converges in the off-policy case. Could authors provide the analytical form of safe distributions? Also, how should we compute these distributions in practice?\n\nQUESTIONS:\n(1) Do Huber loss, logistic loss and entropy loss satisfy all three assumption stated in Section 6 for inexact approximation?\n\nLIMITATIONS:\nThe paper did not discuss the limitations. Some discussions on an extension to stochastic updates and non-linear settings would be fascinating.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-23"}