Learning from Noisy Demonstration Sets via Meta-Learned Suitability Assessor

1 INTRODUCTION . Imagine that you intend to learn how to make a free throw in basketball , which requires you to throw the ball into the basket from a fixed position . Without the proper knowledge , one may observe professional players perform a free throw on YouTube by obtaining numerous exemplary videos . However , learning from every demonstration videos might lead to worse performance , as they may contain unsuitable or even irrelevant content . The challenge of learning from noisy demonstration sets is as well crucial in the robot imitation learning regime , as demonstrations which are not aligned with achieving the intended goal deteriorate the learning process . The assumptions of optimality ( or at least sub-optimality ) of the demonstrations are often made in state-of-the-art imitation learning algorithms ( Ross & Bagnell , 2010 ; Ross et al. , 2011 ; Ho & Ermon , 2016 ; Sermanet et al. , 2018 ) . As a result , they are vulnerable to demonstrations that are potentially detrimental to the learning outcomes in the given set . To address assumptions of optimality , in this paper , we aim for a generic framework capable of learning from a noisy demonstration set , via evaluating the suitability of imitated skills judged by task specific heuristics . Prior works have handled deteriorated imitated outcomes due to noisy demonstration sets by utilizing expected Q-values provided in the demonstrations to avoid learning from bad demonstrated actions ( Kim et al. , 2013 ; Nair et al. , 2017 ; Gao et al. , 2018 ) . However , these works require demonstrations to have a rich representation such as incorporating the aforementioned Q-values , which may neither be available in other cases nor directly applicable to the agent training environment . Our framework , on the other hand , does not require demonstrations to contain specific information , and hence is able to cope with any forms of expert demonstrations . To be specific , we propose to first assess which demonstrations in the given set might be more suitable by learning from them , and then train the agent imitating only a selected subset . In order to achieve selectively learning from suitable demonstrations , we examine if the learning outcomes are favorable after imitating each demonstration in a set . Typically in robot learning regime , we are able to receive designed feedbacks in the target training environment to evaluate how well the agent performs . Thus , in each task , we predefine specific heuristics for assessing if the agent exhibits imitation learning outcomes desirable for reaching the goals of the tasks . The key challenges for the feasibility of assessing learned outcomes from each demonstration are the efficiency and generalization ability . A framework should be capable of producing these assessable outcomes as quick as possible , and generalizing to unseen demonstrations . To this end , we propose a framework with the demonstration suitability assessor leveraging meta-learning , where we train adaptive parameters via meta-imitation-learning . The meta-imitation-learned parameters can thus : ( 1 ) produce assessable imitated outcomes at testing time quicker than both imitation learning from scratch and fine-tune a pretrained initialization ( Finn et al. , 2017a ) , and ( 2 ) adapt to imitating newly sampled unseen demonstrations . Overall , the imitated outcomes after adaptation will be judged by task heuristics to indicate the suitability of certain demonstrations . We then train agents using imitation learning from the selected suitable demonstration subsets for obtaining better policies . We demonstrate two empirical approaches to utilize the resulting suitability assessments . One composes new subsets of demonstrations from the top ranked , the other iteratively fine-tunes the meta-learned parameters by strengthening or weakening certain demonstrations in a set according to current suitability judgments , producing a selected suitable subset at convergence . In some cases , the distribution of demonstration sets can be imbalanced or multi-modal . To prevent over-fitting to certain subsets of such demonstration distributions , we augment the meta-imitationtraining with a regularization objective-maximization of mutual information between the demonstration and the induced behavioral differences from imitating it . This additional regularization term aims to make the meta-trained parameters more responsive to the demonstrations being imitated , and as a result , help differentiate better the adapted behaviors from a noisy and diverse set . We test our framework on four different simulation sports environments in MuJuCo . Our results , both qualitative and quantitative , show that the proposed method outperforms various baselines , including vanilla MAML and fine-tuning a pretrained initialization , on learning better policies from noisy demonstration sets . 2 RELATED WORKS . Imitation Learning and Learning Diverse Behaviors In imitation learning , an agent tries to acquire skills by learning from demonstrated expert behaviors ( Schaal , 1997 ; Argall et al. , 2009 ; Ziebart et al. , 2008 ; Kuniyoshi , 1989 ; Hussein et al. , 2017 ; Ross et al. , 2011 ) . Ho & Ermon ( 2016 ) discover that imitation learning can be formulated as an adversarial training ( Goodfellow et al. , 2014 ) between demonstrations and imitated behaviors of the agent ( GAIL ) , which is adopted in this work as the main imitation learning framework . Extension works of GAIL ( Wang et al. , 2017 ; Hausman et al. , 2017 ; Li et al. , 2017 ) augment the original algorithm with a latent code or embedding for imitating a diverse set of demonstrations , increasing diversity of the imitated outcomes . Duan et al . ( 2017 ) learn an encoder-decoder framework for digesting multiple demonstrations and achieving one-shot imitation learning during testing time . Hester et al . ( 2018 ) train a deep q-network by initializing the replay buffer with expert demonstrations . The demonstration assessing process of our work can also be viewed as leveraging the results of few-shot imitation learning from diverse demonstrations . However , the emphasis of our goal is centered on utilizing the assessments to acquire better skills learning from selected suitable demonstrations . Learning from Imperfect Demonstrations Despite remarkable performances in imitative learning of complex control behaviors from prior works , most of the algorithms assume the optimality ( or at least sub-optimality ) of the demonstrations . The given demonstration set is usually presumed to contain only helpful demonstrations : it is not noisy . Kaiser et al . ( 1995 ) presents several types of experts violating the optimality assumption and basic ways to cope with them . Tomasello ( 2016 ) provides psychological evidences that infants imitate rationally , and that they are more prone to certain demonstrations that do not only fundamentally make sense but effectively help the development of their skills . Grollman & Billard ( 2011 ) utilize failure demonstrations as negative constraint on the exploration to prevent learning agent from approaching behaviors likewise . Kim et al . ( 2013 ) casts learning from limited demonstrations as a relaxed large-margin constrained optimization problem . Several recent works ( Gao et al. , 2018 ; Nair et al. , 2017 ) also assume the expert representation contains expected Q-values and utilize them to filter bad demonstrated actions . However , these additional information may not be available in other cases such as learning from raw video demonstrations ( Sermanet et al. , 2016 ; 2018 ) . Moreover , specifically for Q-values utilized in aforementioned works , the assumption of these information being applicable to the agent training environment does not necessarily hold . We thus aim for a more general framework capable of coping with any forms of demonstrations , while this work mainly based off state-action pairs . Meta Learning in Imitation Learning Our work is also closely related to a series of recent advancements in meta learning , that we as well seek fast adaptive parameters which should learn how to learn from certain demonstrations . Finn et al . ( 2017a ) propose a gradient-based model-agnostic framework ( MAML ) for training a policy prior that can quickly adapt to newly sampled tasks during testing ( termed as meta-testing ) time by small number of gradient update steps . In another work ( Finn et al. , 2017b ) MAML is adopted to train a set of adaptive parameters achieving gradient-based one-shot imitation learning during meta-testing time . As an extension of MAML with regularization in an inner imitative learning objective , we are , to our best knowledge , the first to utilize the meta learning framework to evaluate a demonstration set for learning a better policy from selected demonstrations . 3 PRELIMINARIES AND PROBLEM SETUP . A finite-horizon Markov Decision Process ( MDP ) can be defined asM = ( S , A , P , R , γ , p0 , T ) , where S and A denotes state and action spaces , P : S × A× S → R+ denotes the state transition probability , R : S × A → R is a reward function , p0 an initial state distribution , γ is the reward discount factor , and T is the horizon of the MDP . Let τ = ( s0 , a0 , ... , sT , aT ) be a trajectory of state and action pairs , the associated reward can be written as R ( τ ) = ∑T t=0 γ tR ( st , at ) . Reinforcement learning seeks a policy πθ ( a|s ) parameterized by θ that maximizes the expected total return : Eat∼πθ , st∼S [ R ( τ ) ] . In imitation learning regime , demonstrated expert behaviors are given while the reward function is missing , and the agent acquire desired skills learning from these demonstrations . We now formulate our problem setup : given a noisy demonstration set V = { v1 , ... , vn } , where each vi ∈ V can be represented as state-action pairs : { ( si , t , ai , t ) } , and several environment variants E = { e1 , ... , ek } , where each ei denotes different goal in a multi-goal learning task . We require a suitability assessment score Svi |ej ∈ R of a demonstration vi under ej for judging and selecting suitable demonstrations . And hence , the assessment scores { Svi |ej } can be the reference for selecting the most suitable demonstration as arg maxi Svi |ej , vi ∈ V , or a subset { vopt } for imitation learning . 4 METHOD . Our goal is to learn a good policy from a noisy demonstration set by imitating the selected suitable demonstrations . In order to produce imitated outcomes efficiently for suitability evaluation , we utilize the adaptation process in meta-learning framework , Model-Agnostic Meta-Learning ( MAML ) ( Finn et al. , 2017a ) , to meta-imitation-learn a set of fast adaptive parameters from the given demonstration set . At meta-testing time , the parameters should adapt to each sampled demonstration to produce assessable learning outcomes , leading to selected suitable subsets for the agent to learn from . To alleviate insensitivity imitating a particular demonstration to an unbalanced set distribution , motivated by ( Chen et al. , 2016 ) , we regularize the inner imitation objective of MAML with maximization of mutual information between the demonstration and the induced behavioral differences from imitating it . Fig.1 depicts the overview of our framework , and the pseudo codes are provided in the appendix . 4.1 META-TRAINING FOR FAST ADAPTIVE PARAMETERS . Following the conventions as described in MAML ( Finn et al. , 2017a ; b ) , meta-imitation-learning has two objectives , where the inner objective tries to optimize for adapting to a sampled individual demonstration , while the outer objective ( meta objective ) tries to optimize for fast adaptation across multiple newly sampled demonstrations . Denote fθ as the policy model mapping observations to actions parameterized by θ , the desired adaptive meta-parameters . The inner objective in the MAMLbased meta-imitation-learning is an imitation objective , where θ is updated to θ ′ i after learning from demonstration vi by gradient descent with learning rate α , which can be written as : θ ′ i = θ − α∇θLvi ( fθ ) , vi ∈ V ( 1 ) Here Lvi denotes the imitation loss imitating the sampled demonstration vi . And hence the meta objective and the corresponding meta-imitation-update can be derived as ( with meta learning rate β ) : min θ ∑ vi∈V Lvi ( fθ′i ) , where θ ← θ − β∇θ ∑ vi∈V Lvi ( fθ′i ) ( 2 )

Summary: The paper makes its intent plainly clear, it wants to remove the assumption that demonstrations are optimal.  Thus it should show that in a case that some demonstrations are bad, it outperforms other methods which assume they are all good. The method proposed, while interesting, well-conceived and potentially novel, is not convincingly tested to this end.