Understanding and Leveraging Overparameterization in Recursive Value Estimation

The theory of function approximation in reinforcement learning ( RL ) typically considers low capacity representations that incur a tradeoff between approximation error , stability and generalization . Current deep architectures , however , operate in an overparameterized regime where approximation error is not necessarily a bottleneck . To better understand the utility of deep models in RL we present an analysis of recursive value estimation using overparameterized linear representations that provides useful , transferable findings . First , we show that classical updates such as temporal difference ( TD ) learning or fitted-value-iteration ( FVI ) converge to different fixed points than residual minimization ( RM ) in the overparameterized linear case . We then develop a unified interpretation of overparameterized linear value estimation as minimizing the Euclidean norm of the weights subject to alternative constraints . A practical consequence is that RM can be modified by a simple alteration of the backup targets to obtain the same fixed points as FVI and TD ( when they converge ) , while universally ensuring stability . Further , we provide an analysis of the generalization error of these methods , demonstrating per iterate bounds on the value prediction error of FVI , and fixed point bounds for TD and RM . Given this understanding , we then develop new algorithmic tools for improving recursive value estimation with deep models . In particular , we extract two regularizers that penalize out-of-span top-layer weights and co-linearity in top-layer features respectively . Empirically we find that these regularizers dramatically improve the stability of TD and FVI , while allowing RM to match and even sometimes surpass their generalization performance with assured stability . 1 INTRODUCTION . Model-free value estimation remains a core method of reinforcement learning ( RL ) , lying at the heart of some of the most prominent achievements in this area ( Mnih et al. , 2015 ; Bellemare et al. , 2020 ) . Such success appears paradoxical however , given that value estimation is subject to the deadly triad : any value update that combines off-policy estimation with Bellman-bootstrapping and function approximation diverges in the worst case ( Sutton and Barto , 2018 ) . Without additional assumptions it is impossible to ensure the viability of iterative value estimation schemes , yet this remains a dominant method in RL—its popularity supported by empirical success in many applications . Such a sizable gap between theory and practice reflects limited understanding of such methods , how they behave in practice , and what accounts for their empirical success ( van Hasselt et al. , 2018 ; Achiam et al. , 2019 ) . Decomposing the deadly triad indicates that off-policy estimation and bootstrapping are difficult to forego : off-policy estimation is supported by the empirical effectiveness of action value maximization and replay buffers , while Bellman-bootstrapping provides significant advantages over Monte Carlo estimation ( Sutton , 1988 ) . On the other hand , our understanding of the third factor , the relationship between function representation and generalization , has evolved dramatically in recent years . Although it was once thought that restrictive function approximation—representations that lack capacity to fit all data constraints—might be essential for generalization , we now know that this view is oversimplified ( Belkin et al. , 2019 ) . The empirical success of deep learning ( Krizhevsky et al. , 2012 ) , extremely large models ( Brown et al. , 2020 ) and associated theoretical advances ( Jacot et al. , 2018 ) have made it clear that gradient-based training of overparameterized models embodies implicit biases that encourage generalization even after all data constraints are fit exactly . This success suggests a new opportunity for breaking the deadly triad : by leveraging overparameterized value representations one can avoid some of the most difficult tradeoffs in value-based RL ( Lu et al. , 2018 ) . The use of overparameterized deep models in value-based RL , however , still exhibits mysteries in stability and performance . Although one might expect larger capacity models to improve the stability of Bellman-bootstrapping , in fact the opposite appears to occur ( van Hasselt et al. , 2018 ) . Our own empirical experience indicates that classical value estimation with deep models always diverges eventually in non-toy problems . It has also been shown that value updating leads to premature rankcollapse in deep models ( Kumar et al. , 2021 ) , coinciding with instability and degrading generalization . In practice , some form of early-stopping is usually necessary to obtaining successful results , a fact that is not often emphasized in the literature ( Agarwal et al. , 2021 ) . Meanwhile , there is a long history of convergent methods being proposed in the RL literature—starting from residual gradient ( Baird , 1995 ) , to gradient-TD ( Sutton et al. , 2008 ; Maei et al. , 2009 ) , prox gradient TD ( Liu et al. , 2015 ; 2016 ) , and emphatic TD ( Yu , 2015 ; Sutton et al. , 2016 ) —yet none of these has demonstrated sufficient generalization quality to supplant unstable methods . The current state of development leaves an awkward tradeoff between stability and generalization . A stable recursive value estimation method that ensures generalization quality with overparametrization remains elusive . In this paper we investigate whether overparameterized value representations might allow the stabilitygeneralization tradeoff to be better managed , enabling stable estimation methods that break the deadly triad and generalize well . To this end , we first consider policy evaluation with overparameterized linear value representations , a simplified setting that still imposes the deadly triad ( Zhang et al. , 2021 ) . Here we first show that alternative updates , such as temporal differencing ( TD ) , fitted value iteration ( FVI ) and residual minimization ( RM ) converge to different fixed points in the overparameterized case ( when they converge ) , even though these updates share a common fixed point when the approximation error is zero and there are no extra degrees of freedom ( Dann et al. , 2014 ) . That is , these algorithms embody certain implicit biases that only become distinguishable in the overparameterized case . From this result , we observe that the fixed points lie in different bases , which we use to develop a unified view of iterative value estimation as minimizing the Euclidean norm of the weights subject to alternative constraint sets . This unification allows us to formulate alternative updates that share common fixed points with TD and FVI but guarantee stability without requiring regularization or prox constraints ( Zhang et al. , 2021 ) . Next , we analyze the generalization performance of these algorithms and provide a per-iterate bound on the value estimation error of FVI , and fixed point bounds on the value estimation error of TD . From these results , we identify two novel regularizers , one that closes the gap between RM and TD and another that quantifies the effect of the feature representation on the generalization bound . We deploy these regularizers in a realistic study of deep model training for optimal value estimation and observe systematic stability and generalization improvements . We also observe that the performance gap between RM and TD/FVI can be closed and in some cases eliminated . 2 RELATED WORK . Value estimation has a lengthy history throughout RL research . Our main focus is on off-policy value estimation with parametric function representations and iterative ( i.e. , gradient based ) updates . We do not consider exploration nor full planning problems ( i.e. , approximately solving an entire Markov decision process ( MDP ) ) in the theoretical development , but instead focus on offline value estimation ; however , we do apply the findings to policy improvement experiments in the empirical investigation . Dann et al . ( 2014 ) provide a comprehensive survey of value estimation with parametric function representations . Significant attention has been focused on underparameterized representations where backed up values are not necessarily expressible in the function class , however we focus on the overparameterized case where any backed up values can be assumed to be exactly representable with respect to finite data . This change fundamentally alters the conclusions one can draw about algorithm behavior , as we see below . One of the key consequences is that classical distinctions ( Scherrer , 2010 ; Dann et al. , 2014 ) between objectives—e.g. , mean squared Bellman error ( MSBE ) , mean squared projected Bellman error ( MSPBE ) , mean squared temporal difference error ( MSTDE ) , and the norm of the expected TD update ( NEU ) —all collapse when the Bellman errors can all be driven to zero . Despite this collapse , we find that algorithms targetting the different objectives—TD and LSTD for MSPBE ( Sutton , 1988 ; Bradtke and Barto , 1996 ) and RM without double sampling ( DS ) for MSTDE ( Maei et al. , 2009 ; Dann et al. , 2014 ) —converge to different fixed points given overparameterization , even when they ultimately satisfy the same set of temporal consistency constraints . It is well known that classical value updates can diverge given off-policy data and parametric function representations ( Baird , 1995 ; Tsitsiklis and Van Roy , 1996 ; 1997 ) . The stability of these methods has therefore been studied extensively with many mitigations proposed , including restricting the function representation ( Gordon , 1995 ; Szepesvári and Smart , 2004 ) or adjusting the representation to ensure contraction ( Kolter , 2011 ; Ghosh and Bellemare , 2020 ; Wang et al. , 2021b ) , or modifying the updates to achieve convergent variations , such as LSTD ( Bradtke and Barto , 1996 ; Yu , 2010 ) , FVI ( Ernst et al. , 2005 ; Munos and Szepesvári , 2005 ; Szepesvári and Munos , 2008 ; Lizotte , 2011 ) or the introduction of target networks ( Mnih et al. , 2015 ; Lillicrap et al. , 2016 ; Zhang et al. , 2021 ; Carvalho et al. , 2020 ) . Others have considered modified the updates to combat various statistical inefficiencies ( van Hasselt , 2010 ; Weng et al. , 2020 ; Konidaris et al. , 2011 ) . Another long running trend has been to consider two time-scale algorithms and analyses , reflected in gradient-TD methods ( Sutton et al. , 2008 ; Maei et al. , 2009 ) , prox gradient TD ( Liu et al. , 2015 ; 2016 ) , primal-dual TD ( Dai et al. , 2017 ; Du et al. , 2017 ) , and emphatic TD ( Yu , 2015 ; Sutton et al. , 2016 ) . Beyond mere convergence , however , we discover a greater diversity in fixed points among algorithms in the overparameterized case , which play a critical but previously unacknowledged role in generalization quality . The fact that minimizing MSPBE via TD methods still dominates practice appears surprising given the theoretical superiority of other objectives . It has been argued , for example , that direct policy gradient methods ( Sutton et al. , 1999 ) dominate minimizing Bellman error objectives ( Geist et al. , 2017 ) . Even among Bellman based approaches , it is known that MSBE can upper bound the value estimation error ( MSE ) whereas MSPBE can not ( Kolter , 2011 ; Dann et al. , 2014 ) , yet MSPBE minimization ( via TD based methods ) empirically dominates minimizing MSBE ( via residual methods ) . This dominance has been thought to be due to the double sampling bias of residual methods ( Baird , 1995 ; Dann et al. , 2014 ) , but we uncover a more interesting finding that their fixed points lie in different bases in the overparameterized setting , and that reducing this difference closes the performance gap . We analyze the convergence of classical updates given offline data and provide associated generalization bounds , with the primary goal of understanding the discrepancy between previous theory and the empirical success of TD/FVI versus RM . Although this theory sheds new light in exploitable ways , it can not overcome theoretical limits on offline value estimation , such as lower bounds on worst case error that are exponential in horizon length ( Wang et al. , 2021a ; b ; Zanette , 2021 ; Xiao et al. , 2021 ) . We analyze the convergence of the expected updates , extendible to the stochastic case using known techniques ( Yu , 2010 ; Bhandari et al. , 2018 ; Dalal et al. , 2018 ; Prashanth et al. , 2021 ; Patil et al. , 2021 ) . We expand the coverage of these earlier works by including alternative updates and focusing on the overparameterized case , uncovering previously unobserved differences in the fixed points . There is a growing body of work on linear value estimation and planning that leverages the insight of ( Parr et al. , 2008 ; Taylor and Parr , 2009 ) that linear value estimation is equivalent to linear model approximation . A number of works have strived to obtain provably efficient algorithms for approximating the optimal policy values in this setting , but these generally rely on exploration or strong assumptions about data coverage ( Song et al. , 2016 ; Yang and Wang , 2019 ; Duan et al. , 2020 ; Agarwal et al. , 2020 ; Jin et al. , 2020 ; Yang et al. , 2020 ; Hao et al. , 2021 ) that we do not make . Instead we study linear value estimation to gain insight , but rather than focus on linear planning we leverage the findings to improve the empirical performance of value estimation with deep models .

Summary: In this paper, the authors consider the overparameterized linear representations of TD, FVI, and RM. A unified interpretation of these algorithms of minimizing the Euclidean norm of the weights subject to alternative constraints is proposed. The paper is also supported by the empirical results.