Learning a Behavioral Repertoire from Demonstrations

1 INTRODUCTION . Deep Reinforcement learning has shown impressive results , especially for board games ( Silver et al. , 2017 ) and video games ( Mnih et al. , 2015 ) . However , reinforcement learning ( RL ) has critical shortcomings when reward signals are sparse or interactions with the environment are expensive . There are several attempts to mitigate these shortcomings , including curriculum learning ( Bengio et al. , 2009 ; Graves et al. , 2016 ; Matiisen et al. , 2017 ) , reward shaping ( Ng et al. , 1999 ) , curiositydriven exploration ( Pathak et al. , 2017 ) , diversification ( Conti et al. , 2018 ; Eysenbach et al. , 2018 ) , and Imitation Learning ( Bakker & Kuniyoshi , 1996 ) . In this paper , we focus on Imitation Learning ( IL ) , wherein the goal is to learn a policy from a dataset of demonstrations , possibly coming from a human , another artificial system , or a collection of different entities . IL can be combined with RL , either to kick-start the learning process with IL and then improving the policy further with RL ( Silver et al. , 2016 ) or by running both methods in parallel ( Harmer et al. , 2018 ) . Traditional IL techniques result in a single policy , which usually expresses an “ averaged ” behavior among all the behaviors present in the training data . We see this as a major limitation of IL . It would be more desirable to instead learn a diverse set of policies , expressing all the different types of behaviors present in the dataset . Additionally , having a repertoire of different behaviors allows a system to adapt to changes when it is deployed . Addressing the limitations of current IL methods , we present a new IL approach called Behavioral Repertoire Imitation Learning ( BRIL ) , which is inspired by Quality-Diversity ( QD ) algorithms ( Pugh et al. , 2016 ; Mouret & Clune , 2015 ) and RL methods that learn multiple different behaviors . In contrast to traditional optimization techniques , QD-algorithms attempt to find a diverse set of high-quality solutions rather than a single optimal solution . When QD-algorithms search in policy space , they typically discover hundreds or thousands of different policies controlled by different neural networks . BRIL instead learns a behavioral repertoire using a single model that can be manipulated to express multiple behaviors , similarly to RL algorithms that learn a single policy for multiple goals ( Schaul et al. , 2015 ; Andrychowicz et al. , 2017 ) . BRIL consists of a multi-step process ( see Figure 1 ) wherein the experimenter : ( 1 ) extracts state-action pairs ( similarly to many IL approaches ) , ( 2 ) designs a set of behavioral dimensions to form a behavioral space ( similarly to many QD algorithms ) and determines the behavioral description ( coordinates in the space ) for each demonstration , ( 3 ) merges the data to form a dataset of state-action-behavior triplets , and ( 4 ) trains a model to predict actions from state-behavior pairs through supervised learning . When deployed , the model can act as a policy and the behavior of the model can be manipulated by changing its behavioral input features . BRIL is tested on the build-order planning problem in StarCraft , in which a high-level policy controls the build-order decisions for a bot that has otherwise scripted modules for low-level tasks , similarly to Churchill & Buro ( 2011 ) ; Justesen & Risi ( 2017 ) . We show that the learned policy can be optimized online by modulating the behavioral features using the Upper Confidence Bounds ( UCB1 ) algorithm , such that it outperforms the traditional IL approach against a fixed opponent . We believe this approach can be particularly useful when modeling human players in a game by expressing the entire range of distinct behaviors instead of the average of all . We hypothesize that this property can allow a system to be more robust to exploitation , which is a concern for AI systems in many games . Furthermore , BRIL could be useful in applications beyond games , such as adaptive and resilient robotics . 2 BACKGROUND . 2.1 IMITATION LEARNING . While Reinforcement Learning ( RL ) deals with learning a mapping ( a policy ) between states and actions by interacting with an environment , in Imitation Learning ( IL ) a policy is learned from demonstrations . Methods based on IL , also known as Learning from Demonstration ( LfD ) , have shown promise in the field of robotics ( Bakker & Kuniyoshi , 1996 ; Atkeson & Schaal , 1997 ; Schaal , 1999 ; Argall et al. , 2009 ; Nair et al. , 2018 ) and games ( Silver et al. , 2016 ; Justesen & Risi , 2017 ; Gudmundsson et al. , 2018 ; Thurau et al. , 2004 ; Gorman & Humphrys , 2007 ; Vinyals et al. , 2019 ; Harmer et al. , 2018 ) . IL is a form of supervised learning , in which the goal is to learn a policy π ( s ) , mapping a state s ∈ S to a probability distribution over possible actions . In contrast to an RL task , the agent can not interact with the environment during training but is instead presented with a dataset D of demonstrations . A demonstration dj ∈ D consists of kj sequential state-action pairs , where the action was taken in the state by some policy . While not a general requirement , in this paper a demonstration corresponds to an episode , i.e . starting from an initial state and ending in a terminal state . Generative Adversarial Imitation Learning ( GAIL ) uses a Generative Adversarial Network ( GAN ) architecture wherein the generator is a policy that produces trajectories ( without access to rewards ) and the discriminator has to distinguish between the generated trajectories and trajectories from a set of demonstrations ( Ho & Ermon , 2016 ) . Two extensions of GAIL learn a latent space of the demonstrations ( Wang et al. , 2017 ; Li et al. , 2017 ) , which results in a conditioned policy similarly to our approach . While our approach requires manually designed behavioral dimensions , this can give the user more control over the learned policy ; different behavioral spaces can be beneficial for different purposes . A latent space also does not explicitly bare meaning , in contrast to a manually defined behavioral space . Additionally , our approach learns a low-dimensional behavioral space that is suitable for fast adaptation . 2.2 QUALITY DIVERSITY & BEHAVIORAL REPERTOIRES . Traditional optimization algorithms aim at finding the optimal solution to a problem . Quality Diversity ( QD ) algorithms , on the other hand , attempt to find a set of high-performing solutions that each behave as differently as possible ( Pugh et al. , 2016 ) . QD-algorithms usually rely on evolutionary algorithms , such as Novelty Search with Local Competition ( NSLC ) ( Lehman & Stanley , 2011b ) or MAP-Elites ( Mouret & Clune , 2015 ) . NSLC is a population-based multi-objective evolutionary algorithm with a novelty objective that encourages diversity and a local competition objective that measures an individual ’ s ability to outperform similar individuals in the population . Individuals are added to an archive throughout the optimization process if they are significantly more novel than previously explored behaviors . MAP-Elites does not maintain a population throughout the evolutionary run , only an archive divided into cells that reflect the concept of behavioral niches in a pre-defined behavioral space . For example , in Cully & Mouret ( 2016 ) different cells in the map correspond to different walking gaits for a hexapod robot . Both NSLC and MAP-Elites results in an archive of diverse and high-performing solutions . The pressure toward diversity in QD-algorithms can help the optimization process escape local optima ( Lehman & Stanley , 2011a ) , while the diverse set of solutions also allows for online adaption by switching intelligently between these ( Cully et al. , 2015 ) . We will describe variations of such an adaption procedure in the next section . QD is related to the general idea of learning behavioral repertoires . Where QD-algorithms optimize towards a single quality objective by simultaneously searching for diversity , a behavioral repertoire can consist of solutions optimized towards different objectives as in the Transferability-based Behavioral Repertoire Evolution algorithm ( TBR-Evolution ) by Cully & Mouret ( 2016 ) . 2.3 BANDIT ALGORITHMS & BAYESIAN OPTIMIZATION . Given either a discrete set or a continuous distribution of options , we can intelligently decide which options to select to maximize the expected total return over several trials . To do this , we consider the discrete case as a k-armed bandit problem and the continuous case as a Bayesian optimization problem . In the continuous case , the problem can also be simplified to a k-armed bandit problem , simply by picking k options from the continuous space of options . The goal of a k-armed bandit problem is to maximize the total expected return after some number of trials by iteratively selecting one of k arms/options , each representing a fixed distribution of returns ( Sutton & Barto , 2018 ) . To solve this problem , one must balance exploitation ( leveraging an option that has rendered high returns in the past ) and exploration ( trying options to gain a better estimation of their expected value ) . A bandit algorithm is a general solution to k-armed bandit problems . One of the most popular of these is the Upper Confidence Bound 1 ( UCB1 ) algorithm ( Auer et al. , 2002 ) that first tries each arm once and then always selects the option that at each step maximizes : Xj + C √ 2 ln t nj , ( 1 ) where Xj is the mean return when selecting option j after t steps , nj is the number of times option j has been selected , and C is a constant that determines the level of exploration . This k-armed bandit approach can be considered as the discrete case of the more general approach of Bayesian optimization ( BO ) , in which a continuous black-box objective function is optimized . BO starts with a prior distribution over objective functions , which is then updated based on queries to the black-box function using Bayes ’ theorem . The Intelligent Trial and Error algorithm ( IT & E ) uses BO for robot adaptation to deal with changes in the environment by intelligently searching in the continuous behavioral space of policies found by MAP-Elites ( Cully et al. , 2015 ) . In their approach , the fitness of all solutions in the behavioral space is used to construct a prior distribution of the fitness , which is also called a behavior-performance map . A Bayesian optimizer is then used to sample a point in the map , record the observed performance , and compute a posterior distribution of the fitness . This process is continued until a satisfying solution is found . IT & E could also be applied as an adaptation procedure to the policy found by BRIL , by creating a prior distribution based on some quality information of the demonstrations , e.g . player rating or win-rate . A Bayesian optimizer is then used to optimize the behavioral feature input . In this paper we will , however , simplify the adaption process to a discrete k-armed bandit problem with k manually selected behavioral features and leave the use of Bayesian optimization techniques for future work .

Summary: This paper presents Behavioral Repertoire Imitation Learning (BRIL) which is a way to learn a policy via imitation learning that can be modulated with different behavior inputs that adjust the policy's behavior.  Demonstrations used in training are labeled with differences in behavior across dimensions (which are then reduced to two dimensions using t-SNE), and then these behavior labels are provided as additional input when training a NN from demonstrations using behavior cloning.  Experimental results are shown for learning a BRIL policy from 7000+ demonstrations of humans playing StarCraft, and are compared to that of learning a single behavior cloned policy trained on all demonstrations as well as behavior cloned policies trained on subsets of demonstrations clustered by their behavior.