Learning a Behavioral Repertoire from Demonstrations

1 INTRODUCTION . Deep Reinforcement learning has shown impressive results , especially for board games ( Silver et al. , 2017 ) and video games ( Mnih et al. , 2015 ) . However , reinforcement learning ( RL ) has critical shortcomings when reward signals are sparse or interactions with the environment are expensive . There are several attempts to mitigate these shortcomings , including curriculum learning ( Bengio et al. , 2009 ; Graves et al. , 2016 ; Matiisen et al. , 2017 ) , reward shaping ( Ng et al. , 1999 ) , curiositydriven exploration ( Pathak et al. , 2017 ) , diversification ( Conti et al. , 2018 ; Eysenbach et al. , 2018 ) , and Imitation Learning ( Bakker & Kuniyoshi , 1996 ) . In this paper , we focus on Imitation Learning ( IL ) , wherein the goal is to learn a policy from a dataset of demonstrations , possibly coming from a human , another artificial system , or a collection of different entities . IL can be combined with RL , either to kick-start the learning process with IL and then improving the policy further with RL ( Silver et al. , 2016 ) or by running both methods in parallel ( Harmer et al. , 2018 ) . Traditional IL techniques result in a single policy , which usually expresses an “ averaged ” behavior among all the behaviors present in the training data . We see this as a major limitation of IL . It would be more desirable to instead learn a diverse set of policies , expressing all the different types of behaviors present in the dataset . Additionally , having a repertoire of different behaviors allows a system to adapt to changes when it is deployed . Addressing the limitations of current IL methods , we present a new IL approach called Behavioral Repertoire Imitation Learning ( BRIL ) , which is inspired by Quality-Diversity ( QD ) algorithms ( Pugh et al. , 2016 ; Mouret & Clune , 2015 ) and RL methods that learn multiple different behaviors . In contrast to traditional optimization techniques , QD-algorithms attempt to find a diverse set of high-quality solutions rather than a single optimal solution . When QD-algorithms search in policy space , they typically discover hundreds or thousands of different policies controlled by different neural networks . BRIL instead learns a behavioral repertoire using a single model that can be manipulated to express multiple behaviors , similarly to RL algorithms that learn a single policy for multiple goals ( Schaul et al. , 2015 ; Andrychowicz et al. , 2017 ) . BRIL consists of a multi-step process ( see Figure 1 ) wherein the experimenter : ( 1 ) extracts state-action pairs ( similarly to many IL approaches ) , ( 2 ) designs a set of behavioral dimensions to form a behavioral space ( similarly to many QD algorithms ) and determines the behavioral description ( coordinates in the space ) for each demonstration , ( 3 ) merges the data to form a dataset of state-action-behavior triplets , and ( 4 ) trains a model to predict actions from state-behavior pairs through supervised learning . When deployed , the model can act as a policy and the behavior of the model can be manipulated by changing its behavioral input features . BRIL is tested on the build-order planning problem in StarCraft , in which a high-level policy controls the build-order decisions for a bot that has otherwise scripted modules for low-level tasks , similarly to Churchill & Buro ( 2011 ) ; Justesen & Risi ( 2017 ) . We show that the learned policy can be optimized online by modulating the behavioral features using the Upper Confidence Bounds ( UCB1 ) algorithm , such that it outperforms the traditional IL approach against a fixed opponent . We believe this approach can be particularly useful when modeling human players in a game by expressing the entire range of distinct behaviors instead of the average of all . We hypothesize that this property can allow a system to be more robust to exploitation , which is a concern for AI systems in many games . Furthermore , BRIL could be useful in applications beyond games , such as adaptive and resilient robotics . 2 BACKGROUND . 2.1 IMITATION LEARNING . While Reinforcement Learning ( RL ) deals with learning a mapping ( a policy ) between states and actions by interacting with an environment , in Imitation Learning ( IL ) a policy is learned from demonstrations . Methods based on IL , also known as Learning from Demonstration ( LfD ) , have shown promise in the field of robotics ( Bakker & Kuniyoshi , 1996 ; Atkeson & Schaal , 1997 ; Schaal , 1999 ; Argall et al. , 2009 ; Nair et al. , 2018 ) and games ( Silver et al. , 2016 ; Justesen & Risi , 2017 ; Gudmundsson et al. , 2018 ; Thurau et al. , 2004 ; Gorman & Humphrys , 2007 ; Vinyals et al. , 2019 ; Harmer et al. , 2018 ) . IL is a form of supervised learning , in which the goal is to learn a policy π ( s ) , mapping a state s ∈ S to a probability distribution over possible actions . In contrast to an RL task , the agent can not interact with the environment during training but is instead presented with a dataset D of demonstrations . A demonstration dj ∈ D consists of kj sequential state-action pairs , where the action was taken in the state by some policy . While not a general requirement , in this paper a demonstration corresponds to an episode , i.e . starting from an initial state and ending in a terminal state . Generative Adversarial Imitation Learning ( GAIL ) uses a Generative Adversarial Network ( GAN ) architecture wherein the generator is a policy that produces trajectories ( without access to rewards ) and the discriminator has to distinguish between the generated trajectories and trajectories from a set of demonstrations ( Ho & Ermon , 2016 ) . Two extensions of GAIL learn a latent space of the demonstrations ( Wang et al. , 2017 ; Li et al. , 2017 ) , which results in a conditioned policy similarly to our approach . While our approach requires manually designed behavioral dimensions , this can give the user more control over the learned policy ; different behavioral spaces can be beneficial for different purposes . A latent space also does not explicitly bare meaning , in contrast to a manually defined behavioral space . Additionally , our approach learns a low-dimensional behavioral space that is suitable for fast adaptation . 2.2 QUALITY DIVERSITY & BEHAVIORAL REPERTOIRES . Traditional optimization algorithms aim at finding the optimal solution to a problem . Quality Diversity ( QD ) algorithms , on the other hand , attempt to find a set of high-performing solutions that each behave as differently as possible ( Pugh et al. , 2016 ) . QD-algorithms usually rely on evolutionary algorithms , such as Novelty Search with Local Competition ( NSLC ) ( Lehman & Stanley , 2011b ) or MAP-Elites ( Mouret & Clune , 2015 ) . NSLC is a population-based multi-objective evolutionary algorithm with a novelty objective that encourages diversity and a local competition objective that measures an individual ’ s ability to outperform similar individuals in the population . Individuals are added to an archive throughout the optimization process if they are significantly more novel than previously explored behaviors . MAP-Elites does not maintain a population throughout the evolutionary run , only an archive divided into cells that reflect the concept of behavioral niches in a pre-defined behavioral space . For example , in Cully & Mouret ( 2016 ) different cells in the map correspond to different walking gaits for a hexapod robot . Both NSLC and MAP-Elites results in an archive of diverse and high-performing solutions . The pressure toward diversity in QD-algorithms can help the optimization process escape local optima ( Lehman & Stanley , 2011a ) , while the diverse set of solutions also allows for online adaption by switching intelligently between these ( Cully et al. , 2015 ) . We will describe variations of such an adaption procedure in the next section . QD is related to the general idea of learning behavioral repertoires . Where QD-algorithms optimize towards a single quality objective by simultaneously searching for diversity , a behavioral repertoire can consist of solutions optimized towards different objectives as in the Transferability-based Behavioral Repertoire Evolution algorithm ( TBR-Evolution ) by Cully & Mouret ( 2016 ) . 2.3 BANDIT ALGORITHMS & BAYESIAN OPTIMIZATION . Given either a discrete set or a continuous distribution of options , we can intelligently decide which options to select to maximize the expected total return over several trials . To do this , we consider the discrete case as a k-armed bandit problem and the continuous case as a Bayesian optimization problem . In the continuous case , the problem can also be simplified to a k-armed bandit problem , simply by picking k options from the continuous space of options . The goal of a k-armed bandit problem is to maximize the total expected return after some number of trials by iteratively selecting one of k arms/options , each representing a fixed distribution of returns ( Sutton & Barto , 2018 ) . To solve this problem , one must balance exploitation ( leveraging an option that has rendered high returns in the past ) and exploration ( trying options to gain a better estimation of their expected value ) . A bandit algorithm is a general solution to k-armed bandit problems . One of the most popular of these is the Upper Confidence Bound 1 ( UCB1 ) algorithm ( Auer et al. , 2002 ) that first tries each arm once and then always selects the option that at each step maximizes : Xj + C √ 2 ln t nj , ( 1 ) where Xj is the mean return when selecting option j after t steps , nj is the number of times option j has been selected , and C is a constant that determines the level of exploration . This k-armed bandit approach can be considered as the discrete case of the more general approach of Bayesian optimization ( BO ) , in which a continuous black-box objective function is optimized . BO starts with a prior distribution over objective functions , which is then updated based on queries to the black-box function using Bayes ’ theorem . The Intelligent Trial and Error algorithm ( IT & E ) uses BO for robot adaptation to deal with changes in the environment by intelligently searching in the continuous behavioral space of policies found by MAP-Elites ( Cully et al. , 2015 ) . In their approach , the fitness of all solutions in the behavioral space is used to construct a prior distribution of the fitness , which is also called a behavior-performance map . A Bayesian optimizer is then used to sample a point in the map , record the observed performance , and compute a posterior distribution of the fitness . This process is continued until a satisfying solution is found . IT & E could also be applied as an adaptation procedure to the policy found by BRIL , by creating a prior distribution based on some quality information of the demonstrations , e.g . player rating or win-rate . A Bayesian optimizer is then used to optimize the behavioral feature input . In this paper we will , however , simplify the adaption process to a discrete k-armed bandit problem with k manually selected behavioral features and leave the use of Bayesian optimization techniques for future work .

Summary: The paper proposes behavioral repertoire imitation learning (BRIL) which aims to learn a collection of policy from diverse demonstrations. BRIL learns such a collection by learning a context-dependent policy, where the context variable represents behavior of each demonstration. To obtain a context variable, BRIL rely on user’s knowledge, where the user manually defines a feature space that describes behavior. This feature space is then reduced by using a dimensionality reduction method such as t-SNE. Lastly, the policy is learned by supervised learning (behavior cloning) with a state-context input variable and an action output variable. The method is experimentally evaluated on the StarCraft environment. The results show that BRIL performs better than two baselines: behavior cloning on diverse demonstrations and behavior cloning on clustered demonstrations.