Solving Compositional Reinforcement Learning Problems via Task Reduction

1 INTRODUCTION . A large part of everyday human activities involves sequential tasks that have compositional structure , i.e. , complex tasks which are built up in a systematic way from simpler tasks ( Singh , 1991 ) . Even in some very simple scenarios such as lifting an object , opening a door , sitting down , or driving a car , we usually decompose an activity into multiple steps and take multiple physical actions to finish each of the steps in our mind . Building an intelligent agent that can solve a wide range of compositional decision making problems as such remains a long-standing challenge in AI . Deep reinforcement learning ( RL ) has recently shown promising capabilities for solving complex decision making problems . But most approaches for these compositional challenges either rely on carefully designed reward function ( Warde-Farley et al. , 2019 ; Yu et al. , 2019 ; Li et al. , 2020 ; OpenAI et al. , 2019 ) , which is typically subtle to derive and requires strong domain knowledge , or utilize a hierarchical policy structure ( Kulkarni et al. , 2016 ; Bacon et al. , 2017 ; Nachum et al. , 2018 ; Lynch et al. , 2019 ; Bagaria & Konidaris , 2020 ) , which typically assumes a set of low-level policies for skills and a high-level policy for choosing the next skill to use . Although the policy hierarchy introduces structural inductive bias into RL , effective low-level skills can be non-trivial to obtain and the bi-level policy structure often causes additional optimization difficulties in practice . In this paper , we propose a novel RL paradigm , Self-Imitation via Reduction ( SIR ) , which ( 1 ) naturally works on sparse-reward problems with compositional structures , and ( 2 ) does not impose any structural requirement on policy representation . SIR has two critical components , task reduction and imitation learning . Task reduction leverages the compositionality in a parameterized task space and tackles an unsolved hard task by actively “ simplifying ” it to an easier one whose solution is known already . When a hard task is successfully accomplished via task reduction , we can further run imitation learning on the obtained solution trajectories to significantly accelerate training . Fig . 1 illustrates task reduction in a pushing scenario with an elongated box ( brown ) and a cubic box ( blue ) . Consider pushing the cubic box to the goal position ( red ) . The difficulty comes with the wall on the table which contains a small door in the middle . When the door is clear ( Fig . 1a ) , the task is easy and straightforward to solve . However , when the door is blocked by the elongated box ( Fig . 1b ) , the task becomes significantly harder : the robot needs to clear the door before it can start pushing the cubic box . Such a compositional strategy is non-trivial to discover by directly running standard RL . By contrast , this solution is much easier to derive via task reduction : starting from an initial state with the door blocked ( Fig . 1b ) , the agent can first imagine a simplified task that has the same goal but starts from a different initial state with a clear door ( Fig . 1a ) ; then the agent can convert to this simplified task by physically perturbing the elongate box to reach the desired initial state ( Fig . 1c ) ; finally , a straightforward push completes the solution to the original hard task ( Fig . 1b ) . Notably , by running imitation learning on the composite reduction trajectories , we effectively incorporate the inductive bias of compositionality into the learned policy without the need of explicitly specifying any low-level skills or options , which significantly simplifies policy optimization . Moreover , although task reduction only performs 1-step planning from a planning perspective , SIR still retains the capability of learning an arbitrarily complex policy by alternating between imitation and reduction : as more tasks are solved , these learned tasks can recursively further serve as new reduction targets for unsolved ones in the task space , while the policy gradually generalizes to increasingly harder tasks by imitating solution trajectories with growing complexity . We implement SIR by jointly learning a goal-conditioned policy and a universal value function ( Schaul et al. , 2015 ) , so that task reduction can be accomplished via state search over the value function and policy execution with different goals . Empirical results show that SIR can significantly accelerate and improve policy learning on challenging sparse-reward continuous-control problems , including robotics pushing , stacking and maze navigation , with both object-based and visual state space . 2 RELATED WORK . Hierarchical RL ( HRL ) ( Barto & Mahadevan , 2003 ) is perhaps the most popular approach for solving compositional RL problems , which assumes a low-level policy for primitive skills and a high-level policy for sequentially proposing subgoals . Such a bi-level policy structure reduces the planning horizon for each policy but raises optimization challenges . For example , many HRL works require pretraining low-level skills ( Singh , 1991 ; Kulkarni et al. , 2016 ; Florensa et al. , 2017 ; Riedmiller et al. , 2018 ; Haarnoja et al. , 2018a ) or non-trivial algorithmic modifications to learn both modules jointly ( Dayan & Hinton , 1993 ; Silver & Ciosek , 2012 ; Bacon et al. , 2017 ; Vezhnevets et al. , 2017 ; Nachum et al. , 2018 ) . On the contrary , our paradigm works with any policy architecture of enough representation power by imitating composite trajectories . Notably , such an idea of exploiting regularities on behavior instead of constraints on policy structures can be traced back to the “ Hierarchies of Machines ” model in 1998 ( Parr & Russell , 1998 ) . Our approach is also related to the options framework ( Sutton et al. , 1999 ; Precup , 2000 ) , which explicitly decomposes a complex task into a sequence of temporal abstractions , i.e. , options . The options framework also assumes a hierarchical representation with an inter-option controller and a discrete set of reusable intra-option policies , which are typically non-trivial to obtain ( Konidaris & Barto , 2009 ; Konidaris et al. , 2012 ; Bagaria & Konidaris , 2020 ; Wulfmeier et al. , 2020 ) . SIR implicitly reuses previously learned “ skills ” via task reduction and distills all the composite strategies into a single policy . Besides , there are other works tackling different problems with conceptually similar composition techniques . Probabilistic policy reuse ( Fernández-Rebollo & Veloso , 2006 ; FernándezRebollo et al. , 2010 ) executes a discrete set of previously learned policies for structural exploration . Fang et al . ( 2019 ) consider a dynamic goal setting without manipulable objects and compose an agent trajectory and a goal trajectory that collide as supervision . Dhiman et al . ( 2018 ) transfer previous knowledge to new goals in tabular cases by constraining value functions . Task reduction produces an expanding solved task space , which is conceptually related to life-long learning ( Tessler et al. , 2017 ) and open-ended learning ( Wang et al. , 2019 ) . Evidently , many recent advances have shown that with generic architectures , e.g. , convolution ( LeCun et al. , 1995 ) and self-attention ( Vaswani et al. , 2017 ) , simply training on massive data yields surprisingly intelligent agents for image generation ( Chen et al. , 2020 ) , language modeling ( Brown et al. , 2020 ) , dialogue ( Adiwardana et al. , 2020 ) and gaming ( Schrittwieser et al. , 2019 ; Baker et al. , 2020 ) . Hence , we suggest building intelligent RL agents via a similar principle , i.e. , design a diverse task spectrum and then perform supervised learning on complex strategies . The technical foundation of SIR is to use a universal value function and a goal conditioned policy ( Kaelbling , 1993 ; Schaul et al. , 2015 ) , which has been successfully applied in various domains , including tackling sparse rewards ( Andrychowicz et al. , 2017 ) , visual navigation ( Zhu et al. , 2017 ) , planning ( Srinivas et al. , 2018 ) , imitation learning ( Ding et al. , 2019 ) and continuous control ( Nair et al. , 2018 ) . Besides , task reduction requires state-space search/planning over an approximate value function . Similar ideas of approximate state search have been also explored ( Pathak et al. , 2018 ; Eysenbach et al. , 2019 ; Nasiriany et al. , 2019 ; Ren et al. , 2019 ; Nair & Finn , 2020 ) . SIR accelerates learning by imitating its own behavior , which is typically called self-imitation ( Oh et al. , 2018 ; Ding et al. , 2019 ; Lynch et al. , 2019 ) or self-supervision ( Nair et al. , 2018 ; 2019 ; Pong et al. , 2019 ; Ghosh et al. , 2019 ) . In this paper , we use techniques similar to Rajeswaran et al . ( 2018 ) and Oh et al . ( 2018 ) , but in general , SIR does not rely on any particular imitation learning algorithm . There are also related works on compositional problems of orthogonal interests , such as learning to smoothly compose skills ( Lee et al. , 2019 ) , composing skills via embedding arithmetic ( Sahni et al. , 2017 ; Devin et al. , 2019 ) , composing skills in real-time ( Peng et al. , 2019 ) , following instructions ( Andreas et al. , 2017 ; Jiang et al. , 2019 ) and manually constructing “ iterative-refinement ” strategies to address supervised learning tasks ( Nguyen et al. , 2017 ; Hjelm et al. , 2016 ; Chang et al. , 2019 ) . Conceptually , our method is also related to curriculum learning ( CL ) . Many automatic CL methods ( Graves et al. , 2017 ; Florensa et al. , 2018 ; Matiisen et al. , 2019 ; Portelas et al. , 2019 ; Racaniere et al. , 2020 ) also assume a task space and explicitly construct task curricula from easy cases to hard ones over the space . In our paradigm , task reduction reduces a hard task to a simpler solved task , so the reduction trace over the solved tasks can be viewed as an implicit task curriculum . Such a phenomenon of an implicitly ever-expanding solved subspace has been also reported in other works for tackling goal-conditioned RL problems ( Warde-Farley et al. , 2019 ; Lynch et al. , 2019 ) . Despite the conceptual similarity , there are major differences between SIR and CL . SIR assumes a given task sampler and primarily focuses on efficiently finding a policy to a given task via task decomposition . By contrast , CL focuses on task generation and typically treats policy learning as a black box . We also show in the experiment section that SIR is complementary to CL training and can solve a challenging sparse-reward stacking task ( Sec . 5.2 ) that was unsolved by curriculum learning alone . We remark that it is also possible to extend SIR to a full CL method and leave it as future work . 3 PRELIMINARY AND NOTATIONS . We consider a multi-task reinforcement learning setting over a parameterized task space T . Each task T ∈ T is represented as a goal-conditioned Markov Decision Process ( MDP ) , T = ( S , A , sT , gT , r ( s , a ; gT ) , P ( s′|s , a ; gT ) ) , where S is the state space , A is the action space , sT , gT ∈ S are the initial state and the goal to reach , r ( s , a ; gT ) is the reward function of task T , and P ( s′|s , a ; gT ) is the transition probability . Particularly , we consider a sparse reward setting , i.e. , r ( s , a ; gT ) = I [ d ( s , gT ) ≤ δ ] , where the agent only receives a non-zero reward when the goal is achieved w.r.t . some distance metric d ( · , · ) . We assume that the state space and the action space are shared across tasks , and different tasks share the same physical dynamics ( i.e. , gT only influences the termination condition of the transition probability P ) , so a task T is determined by its initial state sT and goal gT . We also assume a task sampler C , which samples training tasks from T . C can be simply a uniform sampler or something more intelligent . For evaluation , we are interested in the most challenging tasks in T . The learning agent is represented as a stochastic policy πθ ( a|s , gT ) conditioning on state s and goal gT and parameterized by θ . We aim to find the optimal parameter θ ? such that policy πθ ? maximizes the expected accumulative reward across every task T ∈ T with a discounted factor γ , i.e. , θ ? = argmaxθ E [ ∑ t γ tr ( st , at ; gT ) ] where at ∼ πθ ( a|st , gT ) . Correspondingly , we also learn a universal value function Vψ ( st , gT ) for state st and goal gT parameterized by ψ w.r.t . πθ . The subscript θ in πθ and ψ in Vψ will be omitted for simplicity in the following content .

Summary: This paper presents Self Imitation via Reduction (SIR) an approach to learning long-horizon tasks by successively reducing it to easy to solve tasks, generating solutions to these easier tasks and self-imitation on successful task solutions. This is done by training a goal-conditioned policy together with a universal value function. When given a task that cannot be solved via the current policy, SIR searches for an intermediate state that divides the task into two easily solvable sub-tasks; this search is carried out by maximising the (composed) value of the sub-trajectories under the current policy. The policy is then executed for these sub-tasks in sequence; success in both these sub-tasks means a solution for the full task is now found. This solution is used as a demonstration that the policy can use for self-imitation via an advantage weighted behavioural cloning objective. This is combined with the policy loss of standard actor-critic algorithms such as SAC and PPO in both the off-policy and on-policy settings respectively and shows significant performance improvements compared to baselines on several long-horizon tasks such as robotic pushing, stacking 3 blocks and a multi-room maze task.