Title: A Unifying View of Linear Function Approximation in Off-Policy RL Through Matrix Splitting and Preconditioning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Policy Evaluation

Temporal Difference Learning

Fitted Q-iteration

Target Network

Offline RL

Linear Function Approximation

Matrix Splitting
Abstract: In off-policy policy evaluation (OPE) tasks within reinforcement learning, Temporal Difference Learning(TD) and Fitted Q-Iteration (FQI) have traditionally been viewed as differing in the number of updates toward the target value function: TD makes one update, FQI makes an infinite number, and Partial Fitted Q-Iteration (PFQI) performs a finite number. We show that this view is not accurate, and provide a new mathematical perspective under linear value function approximation that unifies these methods as a single iterative method solving same linear system, but using different matrix splitting schemes and preconditioners. We show that increasing the number of updates under the same target value function, i.e., the target network technique, is a transition from using a constant preconditioner to using a data-feature adaptive preconditioner. This elucidates, for the first time, why TD convergence does not necessarily imply FQI convergence, and establishes tight convergence connections among TD, PFQI, and FQI. Our framework enables sharper theoretical results than previous work and characterization of the convergence conditions for each algorithm, without relying on assumptions about the features (e.g., linear independence). We also provide an encoder-decoder perspective to better understand TD’s convergence conditions, and prove, for the first time, that when a large learning rate doesn’t work, trying a smaller one may help(for batch TD). Our framework also leads to the discovery of new crucial conditions on features for convergence, and shows how common assumptions about features influence convergence, e.g., the assumption of linearly independent features can be dropped without compromising the convergence guarantees of stochastic TD in the on-policy setting. This paper is also the first to introduce matrix splitting into the convergence analysis of these algorithms.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work analyzes the theoretical properties of policy evaluation algorithms in both the on- and off-policy settings. It characterizes the solutions of (expected) temporal difference update (TD), fitted Q iteration (FQI) and partial FQI (PFQI) using precise conditions (iff). The analysis is based on rank invariance (Condition 4.1). The paper then shows the convergence/stability properties of these methods based on rank invariance (Sec.5-7).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The theoretical analysis focuses on a key condition of rank invariance (Condition 4.1), which is complementary to the feature independence condition widely used in the literature. It unifies three algorithms’ updates from a pre-conditioning point of view of solving linear systems. It also theoretically characterizes the singularity and consistency of the LSTD system using the rank invariant condition.\n\n2. This new condition also offers new insights into the sufficient and necessary conditions for the convergence of the three algorithms with proper step sizes.\n\nWeaknesses\n\n1. A main concern is that the content in this work is very dense as a conference paper, and it seems that it would benefit from being a journal paper instead. This causes some problems such as notations not being clearly explained in the main text (e.g., $\\sigma$ and $\\mathfrak{R}$ in Corollary 6.3), and it has to constantly refer to the respective appendix sections for the complete discussion, which hurts the reading flow very much.\n\n2. Some writing can be more precise. For example, L147: \"Its solution is the same as the original linear system\" but this should depend on the choice of $M$ (at least not iff).\n\nMinor comments\n\n- The capitalization of section titles is inconsistent throughout the paper.\n- The space before the left brackets is inconsistent throughout the paper.\n- Matrix bolding is inconsistent throughout the paper (e.g., see Proposition 4.2). It seems that not bolding any matrices would be easier as there are many $A, \\Phi, \\Sigma$, etc.\n- $\\theta_{\\phi,r}$ not sure why we need to emphasize $\\phi$ here. Throughout the paper, it seems that all that matters is to show that this depends on $r$.\n- L106 misses transpose\n- L124 can refer to the equation below it instead of (23) in the appendix\n- L166 can refer to (8) instead of (67)\n- L238 of of, L240 $\\rho$ should be explained. Also, it seems that both $\\rho$ and $\\lambda$ are used in the paper for the same meaning.\n- Eq.(31) is empty, and it makes more sense to switch (30) and (29) since (30) is (28) in matrix form.\n- L736 uses $\\theta_t$, (33) uses $\\theta_k$ and (37) uses $\\theta_{k+1}$. Please make sure the indices are consistent (maybe $k+1$ is the most clear). Also in the main text (L137).\n\nQUESTIONS:\nFor the cases of TD convergence versus FQI convergence, Appendix G.3 shows two counterexamples that one does not imply the other. However, it is not clear how these examples came to light or how they were specifically constructed. Further insights would be helpful. In addition, how likely are we going to see such cases in practice?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides an analysis of the relationship between temporal different (TD), partially fitted Q iteration (PFQI) and fitted Q iteration (FQI) in the linear function approximation setting. The authors formulate the problem as solving a preconditioned linear system, with TD PFQI and FQI all  and uses this to analyse the stability and convergence\n\nSTRENGTHS AND WEAKNESSES:\nSTRENGTHS:\n\nThe paper is mathematically rigorous and well written. It clearly introduces notation and carefully analyses each algorithm systematically. One of the better parts of the paper is the correction of previous mistakes in SOME of the related work. A main contribution is the formulation of the problem as a preconditioned linear system. The paper then goes through analysing each of TD, PFQI and FQI, providing sufficient and necessary conditions for convergence of each. The discussion around each theoretical result is intuitive and offers nice insights. The paper also generalises analysis for when the Hessian is singular by analysing the Moore-Penrose pseudo-inverse. \n\nWEAKNESSES:\n\nThis paper seems very similar to [1] in terms of its goals and I'm not sure it offers a significant enough contribution beyond this. \n\nNOT AS GENERAL AS OTHER WORKS: The paper is limited to linear function approximation for finite state-action MDPs in the expected TD setting, whereas papers like [1] study general nonlinear smooth stochastic TD methods in general state-action MDPs. This is especially important in TD and related methods where nonlinearity is not only commonplace, but is a major source of instability itself, and so not accounting for this weakens the power of the analysis.\n\nMANY RESULTS ARE RE-FORMULATIONS OF KNOWN RESULTS: \n\nMany results can be recovered from [1] by noting that the following matrices are equivalent under the authors' notation for the special case of a linear function approximator: $ A_\\textrm{LSTD} = J_\\textrm{TD}(\\omega,\\omega^\\star )$,  $ \\Sigma_\\textrm{cov} = H(\\omega,\\omega^\\star;\\omega_l )$  and $\\gamma \\Sigma_\\textrm{cr} = J_\\delta(\\omega,\\omega^\\star;\\omega_l )$.\n\nIn particular: \n\nThe update equation on line 140 is the same as Lemma 3 in [1]\n\nEq. 6 is simply the TD fixed point equation, which has been characterised many times before. \n\nAnalysis of FQI\n\nTheorem  5.1 draws the same conclusions as Theorem 1 from [1] where convergence of FQI is governed by the convergence to zero of $\\prod_{i=0}^{l-1} H(\\omega_{i+1}^\\star,\\omega^\\star;\\omega_i^\\star )^{-1} J_\\delta(\\omega_i,\\omega^\\star;\\omega^\\star ) $, which reduces down EXACTLY to the convergence of  $(H_\\textrm{FQI})^l=(\\gamma \\Sigma_\\textrm{cov}^{-1}\\Sigma_\\textrm{cr})^l $ in the linear case, thereby recovering the same result. The only slight generalisation in the authors' work is the use of the Moore-Penrose Pseudo inverse, however beyond this the result is not as general as [1] as it does not allow for nonlinear function approximation and general state-action spaces. Indeed, this motivated Assumption 5 in [1], which guarantees stability of FQI using the max  2-norm of  $H(\\omega_{i+1}^\\star,\\omega^\\star;\\omega_i^\\star )^{-1} J_\\delta(\\omega_i,\\omega^\\star;\\omega^\\star )$. \n\nAnalysis of TD\n\nIn theorem 6.1, the conclusion that $H_\\textrm{TD}$ must be semiconvergent has been characterised in many ways before [1] [2]. When considering the conditions needed for the spectral norm to be less than 1 (as the authors identify), this recovers the famous deadly triad of TD [1].\n\nAnalysis of PFQI\n\nThe analysis of PFQI and connection to TD and FQI relies on the preconditioning formulation in Eq. 8. This has been characterised in [1] by the condition function in Eq. 7 $C(\\alpha,k)$. The reason for the condition function in [1] is that under the more general nonlinear analysis, an analytic matrix equation does not exist, and so it acts as an upper bound in terms of max eigenvalues to account for the nonlinearity, but plays the same role in analysis and leads the the same conclusions: \n\nTheorem 7.1 draws the conclusion that convergence relies on $H_\\textrm{PFQI}$ being semi-convergent. In the more general nonlinear  stochastic setting of [1], the same conclusions can be drawn from Theorem 3 if $C(\\alpha,k)<1$, showing convergence to a ball due to stochasticity. $C(\\alpha,k)<1$ is a slightly more strict assumption, however this is needed due to the upper bounding because of nonlinearity, but implies $H_\\textrm{PFQI}$ being semi-convergent in the linear case. \n\nResults in Theorem 8.1 have been shown in [1] under an aysmptotic analysis: the authors of [1] show that convergence of PFQI stems from convergence of TD under the diminishing stepsize regime in Theorem 2, implying there exist a sequence of stepsizes such that PFQI always converges if TD converges. \n\nIn Theorem 8.3, the learning rate conditions for convergence given by the authors are the same as those in [1]: observe that the condition function $C(\\alpha,k)$ converges for the same learning rate interval $(0,\\frac{2}{ \\lambda^\\star_H})$ giving convergence of PFQI from FQI (Property 3 of [1]). These results are completely analogous to the convergence of the precondition function converging, albeit for a nonlinear analysis. The proof that convergence of FQI implies convergence of PFQI for large enough $t$ recovers the same result of Theorem 4 of [1] for large enough $k$ using the preconditioning matrix instead of the condition function. \n\nFinally, the apparently novel conclusion that convergence of PFQI implies convergence of FQI, even when TD does not converge, is discussed extensively in Section 5.4 of [1] including a graphical representation of this property in Fig .1 \n\nSTRAWMANNING OF RELATED WORK:  \n\nIn my opinion, it is not acceptable that the authors claim novelty for the many of the main the results of the paper (Theorem 5.1, 6.1, 7.1 8.1 and 8.3) because, as I have shown above, they have been proved before. What is worse though is that the authors seem to suggest that ALL previous work is misleading or incorrect: \n\nLine 32: `The traditional perspective fails to fully capture the convergence connections between these algorithms and may lead to incorrect conclusions. For example, one might erroneously conclude that TD convergence necessarily implies FQI convergence.'\n\nI don't think any rational person would draw this conclusion from [1] as the connections that authors have made in this paper are also made clearly there. \n\nNOT A SIGNIFICANT ENOUGH CONTRIBUTION: \n\nIn light of this, the current paper and introduced unified view does not offer a substantial contribution. The contributions are a slight clarification of necessary and sufficient conditions, a compiling of existing results and a minor technical relaxation of some assumptions (i.e. no need for linearly independent features in the on policy case and relaxation of invertible Hessian) at the expense of introducing additional assumptions that are too restrictive to apply for most practical methods - in particular finite MDPs, linear function approximation and analysis only in the path mean case. \n\nLITTLE PRACTICAL BENEFIT:\n\nIn my opinion, the purported benefits of the introduced unified view and analysis have little practical benefit. Relaxing the assumption of an invertible Hessian makes little difference when a arbitrarily small amount of regularisation can easily satisfy the assumption. As mentioned before, the introduced stricter assumptions DO affect the ability to apply the analysis to real world algorithms.\n\nMinor weaknesses (These don't count towards my score)\n\nTypos: \n\nLine 7: solving same linear system, Line 18: help(for batch TD), Line 139: Target Network.When, Line 184: (Condition 4.1). it can Line 194: features(Condition 4.3)as, Line 196 features(Condition 4.3), Line 198 features(Condition 4.3) Line 236:  invariance(Condition 4.1), Line 238: he impact of of rank invariance, Line 242: the target linear system the solution of the FQI linear system Line 262: The fixed point it converge to Line 327: features(Condition 4.3)do Line 327: invariance(Condition 4.1),\n\nTitles seems to use a mixture of title case and non title case, sometimes in the same title\n\n[1] Fellows et al. Why Target Networks Stabilise Temporal Difference Methods, ICML 2023. https://arxiv.org/pdf/2302.12537v3\n\n[2] Tsitsiklis and Van Roy. An Analysis of Temporal-Difference Learning with Function Approximation, https://www.mit.edu/~jnt/Papers/J063-97-bvr-td.pdf\n\nQUESTIONS:\nBefore the paper is published ANYWHERE, the authors need to do a better job of CAREFULLY ACKNOWLEDGING related worked instead of presenting their theorems as corrections, clarifications or novelties and claiming existing work is misleading. \n\nEven with this essential correction, I'm not sure the paper contributes enough. A true unified view would have to account for nonlinearity, stochasticity and non-finite MDPs, as these have been tackled before and better represent the reality of where TD algorithms are actually used. I can't raise my score until this is the case. \n\nAnother approach would be to re-write the paper as a review and correction paper. Here previous results and conclusions from existing papers could be carefully collated (acknowledging contribution and similarities between works, corrected if necessary) and expanded upon for certain cases (like in the linear, expected finite case here). I'm not sure this would be a significant enough contribution on its own, but would leave that to the AC to decide.\n\nLIMITATIONS:\nThe authors present many analyses for which identical or similar results have been found before without proper acknowledgement and in a much more restrictive setting. This significantly limits the paper's contribution and claims of a unified framework for analysing off policy RL.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper offers a unified perspective on reinforcement-learning algorithms, Temporal Difference (TD), Fitted Q-Iteration (FQI), and Partially Fitted Q-Iteration (PFQI), by viewing them as matrix-splitting schemes equipped with preconditioners. In the off-policy evaluation setting, the authors derive a single iterative update based on the state–action covariance and cross-covariance matrices, with TD, FQI, and PFQI emerging as special cases. Within this framework they introduce rank invariance, a condition that is both necessary and sufficient for the consistency of the target linear system. Rank invariance ensures existence and uniqueness of the solution and, in turn, leads to characterization of convergence conditions for the three algorithms.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper’s main contribution is to clarify the role of rank invariance, a necessary and sufficient condition for the underlying linear system, and to distinguish it from simple linear independence condition. Leveraging this property, this work rigorously analyzes TD, FQI, and PFQI, yielding results that are both necessary and meaningful to the reinforcement learning theory literature.\n\nWeaknesses:\nThe theoretical guarantees do not extend to policy control settings (nonlinear problems), and the paper does not verify its claims with empirical experiments.\n\nQUESTIONS:\nBecause linear problems have been widely studied in many contexts, I’m somewhat concerned about the novelty of this paper related to rank invariance condition. In Appendix Section I, the authors discuss the convergence conditions used in earlier work. As I understand it, the linear-independence condition has already been addressed in prior studies, whereas the rank-invariance condition has not been clearly articulated. Is this interpretation correct? If not, could you clarify?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the off-policy policy evaluation (OPE) problem with linear function approximation. The authors study the TD, FQI, and Partial FQI (PFQI) algorithms and establish that they can be viewed as solving the same \"target\" linear system but with varying choices of splitting schemes and preconditioners. The authors study this \"target\" linear system and establish necessary and sufficient conditions for the existence and uniqueness of a solution. The authors study the convergence of TD, FQI, and PFQI and establish necessary and sufficient conditions for each of these, with connections to each other, and elucidate why convergence of TD and FQI do not necessarily imply convergence of each other.\n\nSTRENGTHS AND WEAKNESSES:\nThis is a very solid paper which in my view should clearly be accepted to the conference. The results are not extremely technical in hindsight (all the proofs boil down to linear algebra) but impactful for understanding these fundamental algorithms and providing definitive convergence conditions and relations between them. Despite their simplicity, many of these results have been previously documented as far as I am aware. \n\nI have some minor complaints:\n- Typos: there are many typos and spacing issues throughout (especially in the appendix but even in the main text). Please review. \n- Clarity: There are some undefined terminology and notation especially when it comes to linear algebra. E.g. in Section 5 and 6: what is a semiconvergent matrix and what is the notation (A)^D? What is a proper splitting, and what are the implications of it (why should I care about Lemma 5.2)? What is the drazin (group) inverse? Bringing some of these concepts which are necessary for interpreting the results out of the appendix and into the main text would improve interpretability.\n- Missing discussion of related work: the condition that $\\rho(\\gamma \\Sigma_{cov}^\\dagger \\Sigma_{cr}) < 1$ is necessary and sufficient for FQI convergence has been noted in Perdomo et al. (reference [30] of this paper), and the condition that nonsingularity of $A_{LSTD}$ is necessary and sufficient for OPE has been noted in Mou, Pananjady, Wainwright (https://arxiv.org/abs/2012.05299), Amortila, Jiang, Xie (https://arxiv.org/abs/2011.01075) and Amortila, Jiang, Szepesvari (https://arxiv.org/abs/2307.13332).\n\nQUESTIONS:\nThe convergence guarantees can be clarified a bit: e.g. how do the solutions $(A_{FQI})^D b_{FQI} + (I - A_{FQI}(A_{FQI})^D)\\theta_0$ and  $A_{LSTD}^Db_{LSTD} + (I - (A_{LSTD})(A_{LSTD})^D)\\theta_0$ of Theorem 6.1 relate to the desired solution (e.g. assuming realizability, how does it relate to $\\theta^\\star$?)\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}