Understanding and Leveraging Overparameterization in Recursive Value Estimation

The theory of function approximation in reinforcement learning ( RL ) typically considers low capacity representations that incur a tradeoff between approximation error , stability and generalization . Current deep architectures , however , operate in an overparameterized regime where approximation error is not necessarily a bottleneck . To better understand the utility of deep models in RL we present an analysis of recursive value estimation using overparameterized linear representations that provides useful , transferable findings . First , we show that classical updates such as temporal difference ( TD ) learning or fitted-value-iteration ( FVI ) converge to different fixed points than residual minimization ( RM ) in the overparameterized linear case . We then develop a unified interpretation of overparameterized linear value estimation as minimizing the Euclidean norm of the weights subject to alternative constraints . A practical consequence is that RM can be modified by a simple alteration of the backup targets to obtain the same fixed points as FVI and TD ( when they converge ) , while universally ensuring stability . Further , we provide an analysis of the generalization error of these methods , demonstrating per iterate bounds on the value prediction error of FVI , and fixed point bounds for TD and RM . Given this understanding , we then develop new algorithmic tools for improving recursive value estimation with deep models . In particular , we extract two regularizers that penalize out-of-span top-layer weights and co-linearity in top-layer features respectively . Empirically we find that these regularizers dramatically improve the stability of TD and FVI , while allowing RM to match and even sometimes surpass their generalization performance with assured stability . 1 INTRODUCTION . Model-free value estimation remains a core method of reinforcement learning ( RL ) , lying at the heart of some of the most prominent achievements in this area ( Mnih et al. , 2015 ; Bellemare et al. , 2020 ) . Such success appears paradoxical however , given that value estimation is subject to the deadly triad : any value update that combines off-policy estimation with Bellman-bootstrapping and function approximation diverges in the worst case ( Sutton and Barto , 2018 ) . Without additional assumptions it is impossible to ensure the viability of iterative value estimation schemes , yet this remains a dominant method in RL—its popularity supported by empirical success in many applications . Such a sizable gap between theory and practice reflects limited understanding of such methods , how they behave in practice , and what accounts for their empirical success ( van Hasselt et al. , 2018 ; Achiam et al. , 2019 ) . Decomposing the deadly triad indicates that off-policy estimation and bootstrapping are difficult to forego : off-policy estimation is supported by the empirical effectiveness of action value maximization and replay buffers , while Bellman-bootstrapping provides significant advantages over Monte Carlo estimation ( Sutton , 1988 ) . On the other hand , our understanding of the third factor , the relationship between function representation and generalization , has evolved dramatically in recent years . Although it was once thought that restrictive function approximation—representations that lack capacity to fit all data constraints—might be essential for generalization , we now know that this view is oversimplified ( Belkin et al. , 2019 ) . The empirical success of deep learning ( Krizhevsky et al. , 2012 ) , extremely large models ( Brown et al. , 2020 ) and associated theoretical advances ( Jacot et al. , 2018 ) have made it clear that gradient-based training of overparameterized models embodies implicit biases that encourage generalization even after all data constraints are fit exactly . This success suggests a new opportunity for breaking the deadly triad : by leveraging overparameterized value representations one can avoid some of the most difficult tradeoffs in value-based RL ( Lu et al. , 2018 ) . The use of overparameterized deep models in value-based RL , however , still exhibits mysteries in stability and performance . Although one might expect larger capacity models to improve the stability of Bellman-bootstrapping , in fact the opposite appears to occur ( van Hasselt et al. , 2018 ) . Our own empirical experience indicates that classical value estimation with deep models always diverges eventually in non-toy problems . It has also been shown that value updating leads to premature rankcollapse in deep models ( Kumar et al. , 2021 ) , coinciding with instability and degrading generalization . In practice , some form of early-stopping is usually necessary to obtaining successful results , a fact that is not often emphasized in the literature ( Agarwal et al. , 2021 ) . Meanwhile , there is a long history of convergent methods being proposed in the RL literature—starting from residual gradient ( Baird , 1995 ) , to gradient-TD ( Sutton et al. , 2008 ; Maei et al. , 2009 ) , prox gradient TD ( Liu et al. , 2015 ; 2016 ) , and emphatic TD ( Yu , 2015 ; Sutton et al. , 2016 ) —yet none of these has demonstrated sufficient generalization quality to supplant unstable methods . The current state of development leaves an awkward tradeoff between stability and generalization . A stable recursive value estimation method that ensures generalization quality with overparametrization remains elusive . In this paper we investigate whether overparameterized value representations might allow the stabilitygeneralization tradeoff to be better managed , enabling stable estimation methods that break the deadly triad and generalize well . To this end , we first consider policy evaluation with overparameterized linear value representations , a simplified setting that still imposes the deadly triad ( Zhang et al. , 2021 ) . Here we first show that alternative updates , such as temporal differencing ( TD ) , fitted value iteration ( FVI ) and residual minimization ( RM ) converge to different fixed points in the overparameterized case ( when they converge ) , even though these updates share a common fixed point when the approximation error is zero and there are no extra degrees of freedom ( Dann et al. , 2014 ) . That is , these algorithms embody certain implicit biases that only become distinguishable in the overparameterized case . From this result , we observe that the fixed points lie in different bases , which we use to develop a unified view of iterative value estimation as minimizing the Euclidean norm of the weights subject to alternative constraint sets . This unification allows us to formulate alternative updates that share common fixed points with TD and FVI but guarantee stability without requiring regularization or prox constraints ( Zhang et al. , 2021 ) . Next , we analyze the generalization performance of these algorithms and provide a per-iterate bound on the value estimation error of FVI , and fixed point bounds on the value estimation error of TD . From these results , we identify two novel regularizers , one that closes the gap between RM and TD and another that quantifies the effect of the feature representation on the generalization bound . We deploy these regularizers in a realistic study of deep model training for optimal value estimation and observe systematic stability and generalization improvements . We also observe that the performance gap between RM and TD/FVI can be closed and in some cases eliminated . 2 RELATED WORK . Value estimation has a lengthy history throughout RL research . Our main focus is on off-policy value estimation with parametric function representations and iterative ( i.e. , gradient based ) updates . We do not consider exploration nor full planning problems ( i.e. , approximately solving an entire Markov decision process ( MDP ) ) in the theoretical development , but instead focus on offline value estimation ; however , we do apply the findings to policy improvement experiments in the empirical investigation . Dann et al . ( 2014 ) provide a comprehensive survey of value estimation with parametric function representations . Significant attention has been focused on underparameterized representations where backed up values are not necessarily expressible in the function class , however we focus on the overparameterized case where any backed up values can be assumed to be exactly representable with respect to finite data . This change fundamentally alters the conclusions one can draw about algorithm behavior , as we see below . One of the key consequences is that classical distinctions ( Scherrer , 2010 ; Dann et al. , 2014 ) between objectives—e.g. , mean squared Bellman error ( MSBE ) , mean squared projected Bellman error ( MSPBE ) , mean squared temporal difference error ( MSTDE ) , and the norm of the expected TD update ( NEU ) —all collapse when the Bellman errors can all be driven to zero . Despite this collapse , we find that algorithms targetting the different objectives—TD and LSTD for MSPBE ( Sutton , 1988 ; Bradtke and Barto , 1996 ) and RM without double sampling ( DS ) for MSTDE ( Maei et al. , 2009 ; Dann et al. , 2014 ) —converge to different fixed points given overparameterization , even when they ultimately satisfy the same set of temporal consistency constraints . It is well known that classical value updates can diverge given off-policy data and parametric function representations ( Baird , 1995 ; Tsitsiklis and Van Roy , 1996 ; 1997 ) . The stability of these methods has therefore been studied extensively with many mitigations proposed , including restricting the function representation ( Gordon , 1995 ; Szepesvári and Smart , 2004 ) or adjusting the representation to ensure contraction ( Kolter , 2011 ; Ghosh and Bellemare , 2020 ; Wang et al. , 2021b ) , or modifying the updates to achieve convergent variations , such as LSTD ( Bradtke and Barto , 1996 ; Yu , 2010 ) , FVI ( Ernst et al. , 2005 ; Munos and Szepesvári , 2005 ; Szepesvári and Munos , 2008 ; Lizotte , 2011 ) or the introduction of target networks ( Mnih et al. , 2015 ; Lillicrap et al. , 2016 ; Zhang et al. , 2021 ; Carvalho et al. , 2020 ) . Others have considered modified the updates to combat various statistical inefficiencies ( van Hasselt , 2010 ; Weng et al. , 2020 ; Konidaris et al. , 2011 ) . Another long running trend has been to consider two time-scale algorithms and analyses , reflected in gradient-TD methods ( Sutton et al. , 2008 ; Maei et al. , 2009 ) , prox gradient TD ( Liu et al. , 2015 ; 2016 ) , primal-dual TD ( Dai et al. , 2017 ; Du et al. , 2017 ) , and emphatic TD ( Yu , 2015 ; Sutton et al. , 2016 ) . Beyond mere convergence , however , we discover a greater diversity in fixed points among algorithms in the overparameterized case , which play a critical but previously unacknowledged role in generalization quality . The fact that minimizing MSPBE via TD methods still dominates practice appears surprising given the theoretical superiority of other objectives . It has been argued , for example , that direct policy gradient methods ( Sutton et al. , 1999 ) dominate minimizing Bellman error objectives ( Geist et al. , 2017 ) . Even among Bellman based approaches , it is known that MSBE can upper bound the value estimation error ( MSE ) whereas MSPBE can not ( Kolter , 2011 ; Dann et al. , 2014 ) , yet MSPBE minimization ( via TD based methods ) empirically dominates minimizing MSBE ( via residual methods ) . This dominance has been thought to be due to the double sampling bias of residual methods ( Baird , 1995 ; Dann et al. , 2014 ) , but we uncover a more interesting finding that their fixed points lie in different bases in the overparameterized setting , and that reducing this difference closes the performance gap . We analyze the convergence of classical updates given offline data and provide associated generalization bounds , with the primary goal of understanding the discrepancy between previous theory and the empirical success of TD/FVI versus RM . Although this theory sheds new light in exploitable ways , it can not overcome theoretical limits on offline value estimation , such as lower bounds on worst case error that are exponential in horizon length ( Wang et al. , 2021a ; b ; Zanette , 2021 ; Xiao et al. , 2021 ) . We analyze the convergence of the expected updates , extendible to the stochastic case using known techniques ( Yu , 2010 ; Bhandari et al. , 2018 ; Dalal et al. , 2018 ; Prashanth et al. , 2021 ; Patil et al. , 2021 ) . We expand the coverage of these earlier works by including alternative updates and focusing on the overparameterized case , uncovering previously unobserved differences in the fixed points . There is a growing body of work on linear value estimation and planning that leverages the insight of ( Parr et al. , 2008 ; Taylor and Parr , 2009 ) that linear value estimation is equivalent to linear model approximation . A number of works have strived to obtain provably efficient algorithms for approximating the optimal policy values in this setting , but these generally rely on exploration or strong assumptions about data coverage ( Song et al. , 2016 ; Yang and Wang , 2019 ; Duan et al. , 2020 ; Agarwal et al. , 2020 ; Jin et al. , 2020 ; Yang et al. , 2020 ; Hao et al. , 2021 ) that we do not make . Instead we study linear value estimation to gain insight , but rather than focus on linear planning we leverage the findings to improve the empirical performance of value estimation with deep models .

Summary: This paper studies the convergence properties of three classical value estimation algorithms (TD, FVI and RM) under over-parameterized linear case. The difference among convergence results are interpreted unifiedly through different constraints in an optimization problem. It also proposes an generalization bound for FVI. Furthermore, based on the results mentioned before, it proposes two regularizers to help convergence in deep reinforcement learning and experimentally evaluates their performance.