Align-RUDDER: Learning From Few Demonstrations by Reward Redistribution

Reinforcement Learning algorithms require a large number of samples to solve complex tasks with sparse and delayed rewards . Complex tasks are often hierarchically composed of sub-tasks . Solving a sub-task increases the return expectation and leads to a step in the Q-function . RUDDER identifies these steps and then redistributes reward to them , thus immediately giving reward if sub-tasks are solved . Since the delay of rewards is reduced , learning is considerably sped up . However , for complex tasks , current exploration strategies struggle with discovering episodes with high rewards . Therefore , we assume that episodes with high rewards are given as demonstrations and do not have to be discovered by exploration . Unfortunately , the number of demonstrations is typically small and RUDDER ’ s LSTM as a deep learning model does not learn well on these few training samples . Hence , we introduce Align-RUDDER , which is RUDDER with two major modifications . First , Align-RUDDER assumes that episodes with high rewards are given as demonstrations , replacing RUDDER ’ s safe exploration and lessons replay buffer . Second , we substitute RUDDER ’ s LSTM model by a profile model that is obtained from multiple sequence alignment of demonstrations . Profile models can be constructed from as few as two demonstrations . Align-RUDDER uses reward redistribution to speed up learning by reducing the delay of rewards . Align-RUDDER outperforms competitors on complex artificial tasks with delayed rewards and few demonstrations . On the MineCraft ObtainDiamond task , Align-RUDDER is able to mine a diamond , though not frequently . 1 INTRODUCTION . Reinforcement learning algorithms struggle with learning complex tasks that have sparse and delayed rewards ( Sutton & Barto , 2018 ; Rahmandad et al. , 2009 ; Luoma et al. , 2017 ) . For delayed rewards , temporal difference ( TD ) suffers from vanishing information ( Arjona-Medina et al. , 2019 ) . On the other hand Monte Carlo ( MC ) has high variance since it must average over all possible futures ( ArjonaMedina et al. , 2019 ) . Monte-Carlo Tree Search ( MCTS ) , used for Go and chess , can handle delayed and rare rewards since it has a perfect environment model ( Silver et al. , 2016 ; 2017 ) . RUDDER ( Arjona-Medina et al. , 2019 ; 2018 ) has been shown to excel in model-free learning of policies when only sparse and delayed rewards are given . RUDDER requires episodes with high rewards to store them in its lessons replay buffer for learning a reward redistribution model like an LSTM network . However , for complex tasks , current exploration strategies find episodes with high rewards only after an incommensurate long time . Humans and animals obtain high reward episodes by teachers , role models , or prototypes . Along this line , we assume that episodes with high rewards are given as demonstrations . Since generating demonstrations is often tedious for humans and time-consuming for exploration strategies , typically , only a few demonstrations are available . However , RUDDER ’ s LSTM ( Hochreiter , 1991 ; Hochreiter & Schmidhuber , 1997a ) as a deep learning method requires many examples for learning . Therefore , we introduce Align-RUDDER , which replaces RUDDER ’ s LSTM with a profile model obtained from multiple sequence alignment ( MSA ) of the demonstrations . Profile models are well known in bioinformatics . They are used to score new sequences according to their sequence similarity to the aligned sequences . Like RUDDER also Align-RUDDER performs reward redistribution —using an alignment model— , which considerably speeds up learning even if only a few demonstrations are available . Our main contributions are : • We suggest a reinforcement algorithm that works well for sparse and delayed rewards , where standard exploration fails but few demonstrations with high rewards are available . • We adopt multiple sequence alignment from bioinformatics to construct a reward redistribution technique that works with few demonstrations . • We propose a method that uses alignment techniques and reward redistribution for identifying sub-goals and sub-tasks which in turn allow for hierarchical reinforcement learning . 2 REVIEW OF RUDDER . Basic insight : Q-functions for complex tasks are step functions . Complex tasks are typically composed of sub-tasks . Therefore the Q-function of an optimal policy resembles a step function . The Q-function is the expected future return and it increases ( i.e , makes a step ) when a sub-task is completed . Identifying large steps in the Q-function speeds up learning since it allows ( i ) to increase the return by performing actions that cause the step and ( ii ) to sample episodes with a larger return for learning . An approximation to the Q-function must predict the expected future return for every state-action pair . However , a Q-function that resembles a step-function is mostly constant . Therefore predictions are only necessary at the steps . We have to identify the relevant state-actions that cause the steps and then predict the size of the steps . An LSTM network ( Hochreiter , 1991 ; Hochreiter & Schmidhuber , 1995 ; 1997a ; b ) can identify relevant state-actions that open the input gate to store the size of the steps in the memory cells . Consequently , LSTM only updates its states and changes its return prediction when a new relevant state-action pair is observed . Therefore , both the change of the prediction and opening input gates indicate Q-function steps through an LSTM network that predicts the return of an episode . Reward Redistribution . We consider episodic Markov decision processes ( MDPs ) , i.e. , the reward is only given once at the end of the sequence . The Q-function is assumed to be a step function , that is , the task can be decomposed into sub-tasks ( see previous paragraph ) . Reward redistribution aims at giving the differences in the Q-function of an optimal policy as a new immediate reward . Since the Q-function of an optimal policy is not known , we approximate it by predicting the expected return by an LSTM network or by an alignment model in this work . The differences in predictions determine the reward redistribution . The prediction model will first identify the largest steps in the Q-function as they decrease the prediction error most . Fortunately , just identifying the largest steps even with poor predictions speeds up learning considerably . See Figure 1 for a description of the reward redistribution . Learning methods based on reward redistribution . The redistributed reward serves as reward for a subsequent learning method : ( A ) The Q-values can be directly estimated ( Arjona-Medina et al. , 2019 ) , which is used in the experiments for the artificial tasks and BC pre-training for MineCraft . ( B ) Redistributed rewards can serve for learning with policy gradients like Proximal Policy Optimization ( PPO ) ( Schulman et al. , 2018 ) , which is used in the MineCraft experiments . ( C ) Redistributed rewards can serve for temporal difference learning like Q-learning ( Watkins , 1989 ) . LSTM models for reward redistribution . RUDDER uses an LSTM model for predicting the future return . The reward redistribution is the difference between two subsequent predictions . If a stateaction pair increases the prediction of the return , then it is immediately rewarded . Using state-action sub-sequences ( s , a ) 0 : t = ( s0 , a0 , . . . , st , at ) , the redistributed reward is Rt+1 = g ( ( s , a ) 0 : t ) − g ( ( s , a ) 0 : t−1 ) , where g is an LSTM model that predicts the return of the episode . The LSTM model learns at first to approximate the largest steps of the Q-function since they reduce the prediction error the most . 3 ALIGN-RUDDER : RUDDER WITH FEW DEMONSTRATIONS . In bioinformatics , sequence alignment identifies similarities between biological sequences to determine their evolutionary relationship ( Needleman & Wunsch , 1970 ; Smith & Waterman , 1981 ) . The result of the alignment of multiple sequences is a profile model . The profile model is a consensus sequence , a frequency matrix , or a Position-Specific Scoring Matrix ( PSSM ) ( Stormo et al. , 1982 ) . New sequences can be aligned to a profile model and receive an alignment score that indicates how well the new sequences agree to the profile model . Align-RUDDER uses such alignment techniques to align two or more high return demonstrations . For the alignment , we assume that the demonstrations follow the same underlying strategy , therefore they are similar to each other analog to being evolutionary related . If the agent generates a state-action sequence ( s , a ) 0 : t−1 , then this sequence is aligned to the profile model g giving a score g ( ( s , a ) 0 : t−1 ) . The next action of the agent extends the state-action sequence by one state-action pair ( st , at ) . The extended sequence ( s , a ) 0 : t is also aligned to the profile model g giving another score g ( ( s , a ) 0 : t ) . The redistributed reward Rt+1 is the difference of these scores : Rt+1 = g ( ( s , a ) 0 : t ) − g ( ( s , a ) 0 : t−1 ) ( see Eq . ( 1 ) ) . This difference indicates how much of the return is gained or lost by a adding another sequence element . Align-RUDDER scores how close an agent follows an underlying strategy , which has been extracted by the profile model . Similar to the LSTM model , we identify the largest steps in the Q-function via relevant events determined by the profile model . Therefore , redistributing the reward by sequence alignment fits into the RUDDER framework with all its theoretical guarantees . RUDDER ’ s theory for reward redistribution is valid for LSTM , other recurrent networks , attention mechanisms , or sequence and profile models . Advantages of alignment compared to LSTM . Learning an LSTM model is severely limited when very few demonstrations are available . First , LSTM is known to require a large number of samples to generalize to new sequences . In contrast , sequence alignment requires only two examples to generalize well as known from bioinformatics . Second , expert demonstrations have high rewards . Therefore random demonstrations with very low rewards have to be generated . LSTM does not generalize well when only these extreme reward cases can be observed in the training set . In contrast , sequence alignment only uses examples that are closely related ; that is , they belong to the same category ( expert demonstrations ) . Reward Redistribution by Sequence Alignment . The new reward redistribution approach consists of five steps , see Fig . 3 : ( I ) Define events to turn episodes of state-action sequences into sequences of events . ( II ) Determine an alignment scoring scheme , so that relevant events are aligned to each other . ( III ) Perform a multiple sequence alignment ( MSA ) of the demonstrations . ( IV ) Compute the profile model like a PSSM . ( V ) Redistribute the reward : Each sub-sequence τt of a new episode τ is aligned to the profile . The redistributed reward Rt+1 is proportional to the difference of scores S based on the PSSM given in step ( IV ) , i.e . Rt+1 ∝ S ( τt ) − S ( τt−1 ) . In the following , the five steps of Align-RUDDER ’ s reward redistribution are outlined . For the interested reader , each step is detailed in Sec . A.3 in the appendix . Finally , in Sec . A.7.3 in the appendix , we illustrate these five steps on the example of Minecraft . ( I ) Defining Events . Instead of states , we consider differences of consecutive states to detect a change caused by an important event like achieving a sub-goal . An event is defined as a cluster of state differences . We use similarity-based clustering like affinity propagation ( AP ) ( Frey & Dueck , 2007 ) . If states are only enumerated , we suggest to use the “ successor representation ” ( Dayan , 1993 ) or “ successor features ” ( Barreto et al. , 2017 ) . We use the demonstrations combined with state-action sequences generated by a random policy to construct the successor representation . A sequence of events is obtained from a state-action sequence by mapping states s to its cluster identifier e ( the event ) and ignoring the actions . Alignment techniques from bioinformatics assume sequences composed of a few events , e.g . 20 events . If there are too many events , good fitting alignments can not be distinguished from random alignments . This effect is known in bioinformatics as “ Inconsistency of Maximum Parsimony ” ( Felsenstein , 1978 ) . ( II ) Determining the Alignment Scoring System . A scoring matrix S with entries si , j determines the score for aligning event i with j . A priori , we only know that a relevant event should be aligned to itself but not to other events . Therefore , we set si , j = 1/pi for i = j and si , j = α for i 6= j . Here , pi is the relative frequency of event i in the demonstrations . α is a hyper-parameter , which is typically a small negative number . This scoring scheme encourages alignment of rare events , for which pi is small . For more details see Appendix Sec . A.3 . ( III ) Multiple sequence alignment ( MSA ) . An MSA algorithm maximizes the sum of all pairwise scores SMSA = ∑ i , j , i < j ∑L t=0 si , j , ti , tj , t in an alignment , where si , j , ti , tj , t is the score at alignment column t for aligning the event at position ti in sequence i to the event at position tj in sequence j. L ≥ T is the alignment length , since gaps make the alignment longer than the length of each sequence . We use ClustalW ( Thompson et al. , 1994 ) for MSA . MSA constructs a guiding tree by agglomerative hierarchical clustering of pairwise alignments between all demonstrations . This guiding tree allows to identify multiple strategies . For more details see Appendix Sec . A.3 . ( IV ) Position-Specific Scoring Matrix ( PSSM ) and MSA profile model . From the alignment , we construct a profile model as a ) column-wise event probabilities and b ) a PSSM ( Stormo et al. , 1982 ) . The PSSM is a column-wise scoring matrix to align new sequences to the profile model . More details are given in Appendix Sec . A.3 . ( V ) Reward Redistribution . The reward redistribution is based on the profile model . A sequence τ = e0 : T ( et is event at position t ) is aligned to the profile , which gives the score S ( τ ) = ∑L l=0 sl , tl . Here , sl , tl is the alignment score for event etl at position l in the alignment . Alignment gaps are columns to which no event was aligned , which have tl = T + 1 with gap penalty sl , T+1 . If τt = e0 : t is the prefix sequence of τ of length t+ 1 , then the reward redistribution Rt+1 for 0 6 t 6 T is Rt+1 = ( S ( τt ) − S ( τt−1 ) ) C = g ( ( s , a ) 0 : t ) − g ( ( s , a ) 0 : t−1 ) , RT+2 = G̃0 − T∑ t=0 Rt+1 , ( 1 ) where C = Edemo [ G̃0 ] / Edemo [ ∑T t=0 S ( τt ) − S ( τt−1 ) ] with S ( τ−1 ) = 0 . The original return of the sequence τ is G̃0 = ∑T t=0 R̃t+1 and the expectation of the return over demonstrations is Edemo . The constant C scales Rt+1 to the range of G̃0 . RT+2 is the correction of the redistributed reward ( Arjona-Medina et al. , 2019 ) , with zero expectation for demonstrations : Edemo [ RT+2 ] = 0 . Since τt = e0 : t and et = f ( st , at ) , we can set g ( ( s , a ) 0 : t ) = S ( τt ) C. We ensure strict return equivalence ( Arjona-Medina et al. , 2019 ) by G0 = ∑T+1 t=0 Rt+1 = G̃0 . The redistributed reward depends only on the past : Rt+1 = h ( ( s , a ) 0 : t ) . Sub-tasks . The reward redistribution identifies sub-tasks as alignment positions with high redistributed rewards . These sub-tasks are indicated by high scores s in the PSSM . Reward redistribution also determines the terminal states of sub-tasks since it assigns rewards for solving the sub-tasks . However , reward redistribution and Align-RUDDER can not guarantee that the redistributed reward is Markov . For redistributed Markov reward , options ( Sutton et al. , 1999 ) , MAXQ ( Dietterich , 2000 ) , or recursive option composition ( Silver & Ciosek , 2012 ) can be used . Higher Order Markov Reward Redistributions . Align-RUDDER may lead to higher-order Markov redistribution . Corollary 1 in the appendix states that the optimality criterion from Theorem 2 in Arjona-Medina et al . ( 2019 ) also holds for higher-order Markov reward redistributions . If the expected redistributed higher-order Markov reward is the difference of Q-values . In that case the redistribution is optimal , and there is no delayed reward . Furthermore , the optimal policies are the same as for the original problem . This corollary is the motivation for redistributing the reward to the steps in the Q-function . In the Appendix , Corollary 2 states that under a condition , an optimal higher-order reward redistribution can be expressed as the difference of Q-values .

Summary: The paper uses Sequence Alignment technique to redistribute rewards, to a similar effect as with LSTM in RUDDER. The hierarchical agent is trained with behavioral cloning and fine-tuned with RL (tabular in Rooms environment / PPO in MineRL). Tasks are automatically divided into subtasks and specialized agents used for the subtasks. The main contributions seem to be: a) introduction of the Sequence Alignment technique which works well with few expert demonstrations; b) experimental demonstration in Rooms / MineRL.