Boosted Curriculum Reinforcement Learning

1 INTRODUCTION . The combination of reinforcement learning ( RL ) ( Sutton & Barto , 2018 ) algorithms with powerful function approximators , i.e. , deep RL ( François-Lavet et al. , 2018 ; Mnih et al. , 2015 ) , is a major breakthrough towards solving complex decision making and control problems that were impractical for previous shallow RL methods . However , the outstanding performance of deep RL techniques is obtained at the cost of a massive amount of data needed from the interaction with the environment , hindering the practicality of deep RL methods . Curriculum RL is a biologically inspired approach that frames the problem of learning a complex target task into learning a sequence of simplified , increasingly difficult , versions of it ( Florensa et al. , 2017 ; Shao et al. , 2018 ; Ivanovic et al. , 2019 ; Narvekar et al. , 2020 ) . Commonly , establishing the complexity of a task in RL is not trivial , as an indisputable notion of complexity is still missing . However , from a practical point of view , we can consider a task complex if it requires targeted exploration in the environment or complex policies to be solved . In curriculum RL , the notion of task complexity is key , and it is assumed that tailored tasks in a sequence of increasing complexity are presented to the learning agent . Such an appropriate design is , however , a difficult problem , requiring either human expertise ( Narvekar et al. , 2020 ) or solutions to automatize the selection of tasks following the learning of the agent ( Jiang et al. , 2015 ; Svetlik et al. , 2017 ; Klink et al. , 2020 ) . Regardless of using a handcrafted or an automatically generated curriculum , a common approach adopted by most value-based curriculum RL methods is to use the same function approximator throughout all the tasks ( Narvekar et al. , 2020 ) : given a task , an approximation of the action-value function is learned , and used as an initialization for learning the action-value function of the next task in the curriculum . For the success of this procedure , the function approximator needs to be powerful enough to handle the complexity of the target task . However , this requirement raises the key issue of adequately designing the function approximator . A complex function approximator is prone to overfitting , and sensitive to hyper-parameters , being detrimental for learning simple tasks in the curriculum ; conversely , using an overly simple function approximator can hinder the learning of the target task . Moreover , choosing an adequate function approximator a-priori is not trivial , as the target task is only visited at the end , and it is commonly unsolvable by non-curriculum RL methods . In this paper , we introduce a novel approach that models the action-value function of a task as a sum of residuals learned on the previous tasks in the curriculum . Our method increases the representativeness of the function approximator as new tasks are presented to the agent , leading to a procedure that increases the size of the functional space as the curriculum proceeds . This results in a learning procedure that increases the power of the function approximator according to the complexity of the task at hand , using small models in simple problems to avoid overfitting , and using large models ( resulting from the sum of small ones ) to handle the dimensionality of complex problems . Notably , if the curriculum is sufficiently fine-grained , the residuals become smoother functions , close to zero , that are easy to learn , even if the respective tasks are harder than the previous ones . We call our method boosted curriculum reinforcement learning ( BCRL ) , for its resemblance with the boosting technique in supervised learning ( Freund , 1995 ) . We provide a theoretical analysis of our approach under the lens of the approximate value iteration ( AVI ) framework ( Farahmand , 2011 ) . In AVI , an estimate of the optimal action-value function is obtained as an iterative process that starts from an arbitrary estimate and applies the optimal Bellman operator until convergence . It is shown that two sources of error exist in AVI : ( i ) computation of the optimal Bellman operator , ( ii ) representation of the action-value function ( Farahmand , 2011 ) . While the former is due to the need of using samples to approximate the unknown optimal Bellman operator , the latter depends on the representativeness of the functional space chosen to approximate the action-value function . In this work , we formalize the curriculum RL problem in the AVI setting to show that the representation of the action-value function obtained by BCRL results in a tighter bound on the approximation error compared to regular curriculum RL , consequently reducing the performance loss . We complement this analysis with an empirical evaluation of BCRL in AVI and RL settings , resorting to the fitted Q-iteration algorithm ( Ernst et al. , 2005 ) , least-squares policy iteration ( Lagoudakis & Parr , 2003 ) and deep Q-networks ( Mnih et al. , 2015 ) , confirming the advantages of BCRL over regular curriculum RL . 2 RELATED WORK . Learning under a dynamic set of training experience has been investigated in different fields throughout decades ( Saul , 1941 ; Elman , 1993 ; Asada et al. , 1996 ; Krueger & Dayan , 2009 ) . A recent paper by Bengio et al . ( 2009 ) has established the term curriculum learning for the concept of organizing training data of a neural network in a beneficial way for performance and generalization . This term carried over to the domain of RL , where curricula over training tasks have been shown to stabilize the highly challenging optimization problem of RL ( Narvekar et al. , 2020 ) . While a large body of research in the domain of curriculum reinforcement learning investigates the interesting problem of optimally selecting and scheduling the training tasks for an RL agent to optimize performance on a set of target tasks ( Florensa et al. , 2017 ; Ivanovic et al. , 2019 ; Shao et al. , 2018 ; Klink et al. , 2020 ) , the architecture of the learning agent is often assumed fixed and the transfer of agent behavior between subsequent tasks in the curriculum is realized by this shared fixed architecture . However , aside from investigations into the benefit of an adequately chosen evolution of training experience , investigations of an optimal learning architecture ( Elsken et al. , 2019 ) and the evolution of this architecture during learning ( Aoki & Siekevitz , 1988 ; Rumelhart et al. , 1988 ; Ash , 1989 ; Fahlman , 1990 ; Lahnajärvi et al. , 2002 ; Nitanda & Suzuki , 2018 ) have been conducted in supervised learning scenarios . The adaptation of these methods to the setting of RL has been limited to on-policy temporal-difference learning ( Rivest & Precup , 2003 ; Vamplew & Ollington , 2005b ; a ) and it has not been further pursued in recent years . Our work aims to improve the information transfer between subsequent tasks in curriculum value-based reinforcement learning via a boosting procedure ( Freund , 1995 ) . The benefit of the boosting framework is that it is well understood from a theoretical perspective ( Tosatto et al. , 2017 ) , allowing for the derivation of error bounds , as also presented in this paper . Further , it is agnostic to any value-based RL algorithm , avoiding the need to adjust the RL algorithm of choice to be suited for the evolution of the function approximator ( Rivest & Precup , 2003 ; Vamplew & Ollington , 2005a ; b ) . Aside from improving knowledge transfer via the learning architecture of the agent , the field of transfer learning investigates various approaches for transferring knowledge , which we shortly review in Appendix A . 3 PRELIMINARIES . 3.1 DEFINITIONS AND NOTATION . We recall the notions about Markov Decision Processes ( MDPs ) ( Puterman , 1990 ) , and the notation used for approximate value iteration ( AVI ) ( Farahmand , 2011 ) , that are relevant to our work . For a space Σ , we defineM ( Σ ) the set of probability measures over the σ-algebra σΣ . We denote B ( Σ , B ) as the space of bounded measurable functions w.r.t . σΣ with bound 0 < B < ∞ . A finiteaction discounted MDP is defined as a tuple T = 〈S , A , P , R , γ〉 , where S is a measurable state space , A is a finite set of actions , P : S × A →M ( S ) is a transition kernel , R : S × A × S → R is a reward function , and γ ∈ [ 0 , 1 ) is a discount factor . A policy π is a mapping from the state space S to a probability distribution over the action spaceA . A policy induces an action-value function Qπ ( s , a ) , E [ Σ∞t=0γtRt|S0=s , A0=a ] , corresponding to the expected cumulative discounted reward obtained performing action a in state s and following the policy π thereafter . RL aims at solving an MDP by finding an optimal policy π∗ which induces an optimal action-value function Q∗ ( s , a ) = supπ Q π ( s , a ) . For immediate rewards uniformly bounded by Rmax , the action-value function is bounded by Qmax = Rmax/1−γ . For a probability measure µ ∈ M ( S × A ) , we define the Lp ( µ ) -norm of a measurable function Q ∈ B ( S × A ) as ‖Q‖p , µ , [ ∫ S×A |Q ( s , a ) | pdµ ( s , a ) ] 1/p . Given a sequence of values x1 : n= ( x1 , . . . , xn ) for some space X , the empirical Lp-norm of a function f : X → R is ‖f‖p , x1 : n , 1n ∑n i=1 |f ( xi ) |p . Conveniently , when Xi ∼ µ , we have E [ ‖f‖p , x1 : n ] = ‖f‖p , µ . From now on , we consider the L2-norm when the subscript p is omitted . Operators . We define the operator Pπ as ( PπQ ) ( s , a ) , ∫ S P ( ds ′|s , a ) ∑ a′∈A π ( a ′|s′ ) Q ( s′ , a′ ) , for any given policy π . Then , the Bellman operator Tπ : B ( S ×A ) → B ( S ×A ) is ( TπQ ) ( s , a ) , r ( s , a ) + γ ( PπQ ) ( s , a ) . The optimal Bellman operator T ∗ : B ( S × A ) → B ( S × A ) is ( T ∗Q ) ( s , a ) , r ( s , a ) + γ ∫ S maxa′ P ( ds ′|s , a ) Q ( s′ , a′ ) ) . Finally , the fixed point of the Bellman operator is ( TπQπ ) = Qπ , and the fixed point of the optimal Bellman operator is ( T ∗Q∗ ) = Q∗ . As in Györfi et al . ( 2006 ) , we define the truncation operator βB : B ( S × A ) → B ( S × A , B ) such that , given a function f ∈ B ( S ×A ) , we have βBf ( s , a ) ∈ [ −B , B ] , ∀ ( s , a ) ∈ S ×A .

Summary: This paper studies a fundamental issue in curriculum design for a complex reinforcement learning problem (target task). When the target task is complex to solve with direct training on it, the learner can be trained over a sequence of tasks with increasing difficulty. Here, the tasks in the sequence vary in either the transition dynamics or reward function. The central question of the paper: developing a principled method for transferring the knowledge (Q-value function or residuals) gained in the current task of the curricula to the next task. The authors propose to use the sum of residuals trained on each task for knowledge transfer and develop a novel curriculum RL method called Boosted Curriculum Reinforcement learning. By leveraging techniques from approximate value iteration literature, they theoretically justify their choice for knowledge transfer and their algorithmic proposal. The authors also conduct interesting empirical investigations on fleshing out the importance of boosting and the curriculum components of their algorithm.