Learning a Behavioral Repertoire from Demonstrations
Authors: Niels Justesen
Miguel González Duque
Daniel Cabarcas Jaramillo
Jean-Baptiste Mouret
Sebastian Risi
Venue:  (2020)
Keywords: Behavioral Repertoires
Imitation Learning
Deep Learning
Adaptation
StarCraft 2

Abstract: Imitation Learning (IL) is a machine learning approach to learn a policy from a set of demonstrations. IL can be useful to kick-start learning before applying reinforcement learning (RL) but it can also be useful on its own, e.g. to learn to imitate human players in video games. However, a major limitation of current IL approaches is that they learn only a single ``"average" policy based on a dataset that possibly contains demonstrations of numerous different types of behaviors. In this paper, we present a new approach called Behavioral Repertoire Imitation Learning (BRIL) that instead learns a repertoire of behaviors from a set of demonstrations by augmenting the state-action pairs with behavioral descriptions. The outcome of this approach is a single neural network policy conditioned on a behavior description that can be precisely modulated. We apply this approach to train a policy on 7,777 human demonstrations for the build-order planning task in StarCraft II. Dimensionality reduction techniques are applied to construct a low-dimensional behavioral space from the high-dimensional army unit composition of each demonstration. The results demonstrate that the learned policy can be effectively manipulated to express distinct behaviors. Additionally, by applying the UCB1 algorithm, the policy can adapt its behavior -in-between games- to reach a performance beyond that of the traditional IL baseline approach.

Under review as a conference paper at ICLR 2020
LEARNING A BEHAVIORAL REPERTOIRE FROM DEMONSTRATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Imitation Learning (IL) is a machine learning approach to learn a policy from a set of demonstrations. can be useful to kick-start learning before applying re- inforcement learning (RL) but it can also be useful on its own, e.g. to learn to imitate human players in video games. However, a major limitation of current approaches is that they learn only single "average" policy based on a dataset that possibly contains demonstrations of numerous different types of behaviors. In this paper, we present new approach called Behavioral Repertoire Imitation Learning (BRIL) that instead learns a repertoire of behaviors from a set of demon- strations by augmenting the aate-nction pairs with behavioral descriptions. The outcome this approach is a single neural network policy conditioned on a be- havior description that can be precisely modulated. We apply this approach train policy on 7,777 human demonstrations for the build-order planning task StarCraft II. Dimensionality reduction techniques are applied construct low- dimensional behavioral space from the high-dimensionn of each demonstration. The results demonstrate that the learned policy can be ef- fectively manipulated to express distinct behaviors. Additionally, by applying the UCB1 algorithm, the policy can adapt its behavior in -e-beweeen games reach performance beyond that of the traditional IL baseline approach.
army unit composition
INTRODUCTION
Deep Reinforcement learning has shown impressive results, especially for board games (Silver al., 2017] and video games (Mnih al. 2015). However, reinforcement learning (RI.) has critical shortcomings when reward signals are sparse or interactions with the environment are expensive. There are several attempts to mitigate these shortcomings, including curriculum learning (Bengio ct al, 2009 Graves et al. 2016; Matisen et al. 2017), reward shaping (Ng et al. 1999), curiosity- driven exploration (Pathak et al. 2017), diversification (Conti et al., 2018 Eysenbach et al. 2018 and Imitation Learning (Bakker & Kuniyoshi 1996).
In this paper, we focus on Imitation Learning (IL), wherein the goal is to learn a policy from a dataset of demonstrations, possibly coming from human, another artificial system, a collection of different entities. can be combined with RL, either to kick-start the learning process with II. and then improving the policy further with RL (Silver et (016) or by running both methods in parallel (Harmer et al. 2018). Traditional IL techniques result in a single policy, which usually expresses an "averaged" behavior among all the behaviors present in the training data. We see this major limitation of IL. It would be more desirable to instead learn a diverse set of policies, expressing all the different types of behaviors present in the dataset. Additionally, having a repertoire of different behaviors allows system to adapt to changes when it is deployed.
Addressing the limitations of current methods, we present new approach called Behav- ioral Repertoire Imitation Learning (BRIL), which is inspired by Quality-Diversity (QD) algorithms (Pugh et al.) 2016; Mouret & Clune 2015) and RL methods that learn multiple different behaviors. In contrast to traditional optimization techniques, QD-algorithms attempt to find diverse set of high-quality solutions rather than a single optimal solution. When QD-algorithms search in policy space, they typically discover hundreds or thousands of different policies controlled by different neural networks. BRIL instead learns a behavioral repertoire using a single model that can be ma- nipulated to express multiple behaviors, similarly to RI. algorithms that learn single policy for

---

Under review as a conference paper at ICLR 2020
(2.a.) Determine the behavior in R" for each demonstration
(2.b) Reduce the dimensionality to R
Demonstrations
(4) Train policy a b)
supervised
Exiruct state-action pairs each demonstration
(3) Merge into state- action- action-behaviin behavior triplets
Figure I: Behavioral Repertoire Imitation Learning (BRIL) trains a policy a(s, b) supervised on a data set of state-action pairs augmented with behavioral descriptors in R for each demonstration. When deployed, a system can adapt its behavior by modulating b. High-dimensional behavioral spaces can be reduced using dimensionality reduction, as low-dimensional behavioral descriptions allow for faster adaption.
multiple goals (Schaul et al. 2013: Andrychowicz et al. 2017). BRIL consists of a multi-step pro- cess (see Figure wherein the experimenter: (I) extracts state-action pairs (similarly to many IL approaches), (2) designs set behavioral dimensions to form a behavioral space (similarly to many QD algorithms) and determines the behavioral description (coordinates in the space) for each demonstration, (3) merges the data to form dataset of state-action-behavie triplets, and (4) trains model to predict actions from state-behavior pairs through supervised learning. When deployed, the model can act as policy and the behavior of the model can be manipulated by changing its behavioral input features.
BRIL is tested on the build-order planning problem in StarCraft, in which high-level policy con- trols the build-order decisions for a bot that otherwise scripted modules for low-level tasks, similarly to [Churchill & Buro (201T); Justesen & Risi (2017). We show that the learned policy can be optimized online by modulating (UCBI) algorithm, such that it outperforms the traditional IL approach against fixed opponent. We believe this approach can be particularly useful when modeling human players in a game by expressing the entire range of distinet behaviors instead of the average of all. We hypothesize that this property can allow a system to be more robust to exploitation, which is a concern for Al systems in many games. Furthermore, BRIL could be useful in applications beyond games, such adaptive and resilient robotics.
behavioral features using the Upper Confidence Bounds
2 BACKGROUND
2.1 IMITATION LEARNING
While Reinforcement Learning (RL) deals with learning mapping policy) between states and actions by interacting with an environment, in Imitation Learning (IL) policy is learned from demonstrations. Methods based on IL. also known as Learning from Demonstration (LfD). have shown promise in the field of robotics (Bakker & Kuniyoshi] 1996
| Atkeson & Schaal| 1997 Schaal
1999 Argall et al., 2009; Nair al., 2018) and games (Silver al. 2016 Justesen & Risi 2017
Gudmundsson et al. 2018 Thurau et al. 2004 Gorman & Humphrys 2007 Vinyals et al. 2019
Harmer et al. 2018
Is form of supervised learning, in which the goal is to learn policy T (s), mapping state E S to probability distribution over possible actions. In contrast to an RI task, the agent cannot interact with the environment during training but is instead presented with dataset D of demonstrations. A demonstration d, E D consists ky sequential state-action pairs, where the action was taken the state by some policy. While not general requirement, in this paper demonstration corresponds to an episode, i.e. starting from an initial state and ending in a terminal state.
Generative Adversarial Imitation Learning (GAIL) uses Generative Adversarial Network (GAN) architecture wherein the generator a policy that produces trajectories (without access to rewards) and the discriminator has to distinguish between the generated trajectories and trajectories from
2

---

Under review as a conference paper at ICLR 2020
set of demonstrations (Ho & Ermon 2016). Two extensions of GAIL learn latent space of the demonstrations (Wang et al., 2017; Li et al. 2017). which results in a conditioned policy similarly to our approach. While approach requires manually designed behavioral dimensions, this can give the user more control over the learned policy; different behavioral spaces can be beneficial for different purposes. A latent space also does not explicitly hare meaning, in contrast to manually defined behavioral space. Additionally, our approach learns a low-dimensional behavioral space that suitable for fast adaptation.
2.2 QUALITY DIVERSITY BEHAVIORAL REPERTOIRES
Traditional optimization algorithms aim at finding the optimal solution to a problem. Quality Diver- sity (QD) algorithms, on the other hand, attempt to find a set of high-perforning solutions that each behave as differently as possible (Pugh et al. 2016). QD-algorithms usually rely evolutionary algorithms, such as Novelty Search with Local Competition (NSLC) (Lehman & Stanley) (201 or MAP-Elites (Mouret & Clune 2015). NSLC is a population-based multi-objective evolutionary algorithm with a novelty objective that encourages diversity and a local competition objective that measures an individual's ability to outperform similar individuals in the population. Individuals are added to an archive throughout the optimization process if they are significantly more novel than pre- viously explored behaviors. MAP-Elites does not maintain a population throughout the evolutionary run, only an archive divided into cells that reflect the concept of behavioral niches in a pre-defined behavioral space. For example, in Cully & Mouret (2016) different cells in the map correspond to different walking gaits for hexapod robot. Both NSLC and MAP-Elites results in an archive of diverse and high-pes perforrrff solutions. The pressure toward diversity in QD-algorithms can help the optimization process escape local optima (Lehman & Stanley, 201 while the diverse set of so- lutions also allows for online adaption by switching intelligently between these (Cully et al. 2015). We will describe variations of such an adaption procedure in the next section.
QD is related to the general idea of learning behavioral repertoires. Where QD-algorithms optimize towards a single quality objective by simultaneously searching for diversity, a behavioral repertoire can consist of solutions optimized towards different objectives as in the Transferability--oe -baed Be- havioral Repertoire Evolution algorithm (TBR-Evolution) by byCllly & Mouret (2016).
2.3 BANDIT ALGORITTMS & BAYESIAN OPTIMIZATION
Given either a discrete set or continuous distribution of options, we can intelligently decide which options to select to maximize the expected total return over several trials. To do this, we consider the discrete case as a k-armed bandit problem and the continuous case Bayesian optimization problem. In the continuous case, the problem can also be simplified to a k-armed bandit problem, simply by picking l options from the continuous space of options.
The goal of a k-armed bandit problem is to maximize the total expected return after some number of trials by iteratively selecting one of k arms/options, each representing fixed distribution of returns Sutton & Barto 2018). To solve this problem, one must balance exploitation (leveraging an option that has rendered high returns in the past) and exploration (trying options to gain better estimation of their expected value). A bandit algorithm is a general solution to k-armed bandit problems. One of the most popular of these is the Upper Confidence Bound 1 (UCB1) algorithm (Auer et al., 2002) that first tries each arm once and then always selects the option that at each step maximizes:
2nt
X,
C
nj
where X, is the mean return when selecting option j after I steps, n, the number of times option j has been selected, and C' is constant that determines the level of exploration.
This k-armed bandit approach can be considered as the discrete case of the more general approach of Bayesian optimization (BO), in which continuous black-box objective function is optimized. BO starts with a prior distribution over objective functions, which is then updated based on queries to the black-box function using Bayes' theorem. The Intelligent Trial and Error algorithm (IT&E) uses BO for robot adaptation to deal with changes in the environment by intelligently searching in the continuous behavioral space of policies found by MAP-Elites (Cully et al. 2015). In their approach,
3

---

Under review as a conference paper at ICLR 2020
the fitness of all solutions in the behavioral space is used to construct prior distribution of the fitness, which is also called a sehavior-performen sample a point in the map, record the observed performance, and compute posterior distribution of the fitness. This process is continued until satisfying solution is found. IT&E could also be applied as adaptation procedure to the policy found by BRII. by creating a prior distribution based on some quality information of the demonstrations, e.g. player rating or win-rate. A Bayesian optimizer then used to optimize the behavioral feature input. In this paper we will, however, simplify the adaption process to a discrete k-armed bandit problem with k manually selected behavioral features and leave the use of Bayesian optimization techniques for future work.
map. A Bayesian optimizer is then used to
2.4 DIMENSIONALITY REDUCTION
In this paper, the dimensionality of the behavior space is reduced using the t-distributed Stochas- tic Neighbor Embedding (t-SNE) by Maaten & Hinton (2008). In Stochastic Neighbor Embedding (SNE, a precursor to t-SNE), a graph is embedded by minimizing the distance between two prob- ability distributions measured with the Kullback-Leibler divergence. The first of these probability distributions reflect the similarity between the high-dimensional points, and the second one measures the similarity between the embedded, low dimensional points. The high-dimensional probability is fixed, while the embedding is iteratively updated to minimize the distance between its probabil- ity distribution and the fixed one. t-SNE uses Student t-distribution kernel for the embedding's 's probability, solving previously known flukes of SNE such as the crowding problem.
Even though t-SNE is considered state-of-the-art in dimension reduction, several other techniques could be explored. For instance, certain datasets' structure can be recovered in low dimensional space using simpler algorithms such as Principal Component Analysis (PCA). Otherwise, other methods such as Isometric Feature Mapping (Isomap) by TTnenbaum et al. (2000), Locally Linear Embedding (LLE) by [Saul & Roweis (2001) and Uniform Manifold Approximation and Projection (UMAP) by McInnes et al. (2018) could be used.
2.5 UNIVERSAL. POLICIES
In value-based RL, one typically learns state value function V* or state-action value function Q=(s, a) for a policy æ. Universal Value Function Approximators (UVFA) instead learn a joint distribution (*,gg) g) Q=,ss, a. over all goals G (Schaul et al.) 2015). UVFA can be learned using supervised learning from a training set of optimal values such as (s) or (7(s,a), a), or it can be learned through RL by switching between goals both when generating trajectories and when computing gradients. Hindsight Experience Replay is an extension to UVFAs, which performs an additional gradient update with the goal being replaced by the terminal state; this modification can give further improvements when it is infeasible to reach the goals (Andrychowicz et al.) 2017). An extension to Generative Adversarial Imitation Learning (GAIL) augments each trajectory with a context (Merel et al. 2017), which specifies the agent's 's sub-goals that can be modulated at test-time.
In our approach, we are not considering goals, but rather behaviors, intending to learn a universal policy (s, b) over states E S and behaviors b E B in particular behavioral space. We are thus combining the QD approach of designing a behavioral space with the idea of learning a universal policy express behaviors in this space.
3 BEHAVIORAL REPERTOIRE IMITATION LEARNING (BRIL)
This section describes two approaches to learning behavioral repertoires using IL. We first describe how behavioral space can be formed from demonstrations. Then we introduce a naive IL approach that first clusters demonstrations based on their coordinates in the behavioral space, and then applies traditional II. on each cluster. Finally, BRIL is introduced, which learns a single policy augmented with behavioral feature input rather than learning multiple policies for each behavioral cluster.
3.1 BEHAVIORAL SPACES FROM DEMONSTRATIONS
A behavioral space consists of some behavioral dimensions that are typically determined by the ex- perimenter. For example, in StarCraft, behavioral dimensions correspond the ratio of each
4

---

Under review as a conference paper at ICLR 2020
army unit produced throughout the game to express the strategic characteristics of the player. A behavioral space can require numerous dimensions to be able to express meaningful behavioral re- lationships between interesting solutions for problem. Intuitively. if the problem is complex, more dimensions can give a finer granularity in the diversity of solutions. However, there is a trade-off between granularity and adaptation, low-dimensional spaces easier search in. We thus pro- pose the idea of first designing a high-dimensional behavioral space and then reducing the number of dimensions through dimensionality reduction techniques. In our preliminary experiments, has shown beneficial to reduce the space to two dimensions, as it allows for easy visualization of the data distribution and it also seems to be a good trade-off between granularity and adaptation speed. In preliminary experiments with one-dimensional behavioral spaces, we noticed that nearby solutions could be wildly different.
3.2 IMITATION LEARNING ON BEHAVIORAL CLUSTERS
The naive IL approach for learning behavioral repertories trains n policies on n behaviorally diverse subsets of the demonstrations. This idea is similar to the state-space clustering in Thurau et al. (2004), but here we cluster data points behavioral space instead. When a behavioral space is defined, each demonstration can be specified by a particular behavioral description (a coordinate in the g2n dimensional space), where afterward a clustering algorithm can split the dataset into several subsets. Hereafter, traditional IL. can be applied to each subset to learn one policy for each behavioral cluster. This approach creates a discrete set of policies similar to current QD algorithms. However, introduces dilemma: if the clusters are small, there risk of overfitting to these reduced training sets. On the other hand, if the clusters are large but few, the granularity of behaviors is lost.
3.3 LEARNING BEITAVIORAL REPERTOIRES
QD algorithms typically fill an archive with diverse and high-quaitty solutions, sometimes resulting in thousands of policies stored in a single run, which increases the storage requirements in train- ing as well as in deployment. To reduce the storage requirement, one can decrease the size of the archive, with the trade-off of losing granularity in the behavioral space. The main approach intro- duced in this paper, called Behavioral Repertoire Imitation Learning (BRIL), solves these issues and reduces overfitting by employing a universal policy instead, in which a single policy is conditioned on behavioral description. In contrast to QD algorithms, the goal of BRIL is neither to optimize quality nor diversity directly. Instead, BRIL attempts to imitate and express the diverse range of behaviors and the quality that exists in given set of demonstrations. Additionally, BRIL produces a continuous space of policies which potentially more expressive than a discrete set.
BRII. extends the traditional imitation learning setting through the following approach. First, the behavioral characteristics of each demonstration are determined. If the dimensionality of these descriptions is large, it can be useful to reduce the space described in the earlier section. A training set of suate-action-beaal behavior triplets is then constructed, such that the behavior is equal to the behavioral deseription of the corresponding demonstration. Then, policy T(s, b) is trained in supervised way on this dataset to map states and behaviors to actions. Following this approach, the training set is not reduced to small behavioral clusters.
When the trained policy is deployed, the behavioral feature input can be modulated to manipulate its behavior. The simplest approach is to fix the behavioral features throughout an episode, evaluate the episodic return, and then consider new behavioral features for the next episode. This approach should allow for episodic, or inter-game, adaptivity, which will be explored our experiments. One could also manipulate the behavioral features during an episode e.g. by learning a meta- veta-yolicy.
4 EXPERIMENTS
This section presents the experimental results of applying BRIL to the game of StarCraft. Policies are trained to control the build-order planning module of a relatively simple scripted StarCraft bo that plays the Terran race. While the policy trained off-line, our experiments attempt to optimize the playing strength of this bot online, in-between cpisodes/games, by manipulating its behavior.
(REDACTED for anonymity]
5

---

Under review as a conference paper at ICLR 2020
4.1 BEHAVIORAL FEATURE SPACE
The behavioral space for StarCraft build-order policy be designed in many ways. Inspired by the AlphaStar League Strategy Map [Vinyals et al. 2019), the behavioral features are constructed from the army composition, such that the dimensions represent the ratios of each unit type. We achieve this by traversing all demonstrations in the data set, counting all the army unit creation events, and computing the relative ratios. Each demonstration thus has an n-dimensional behavioral feature description, where 15 is the number of army unit types for Terran.
To form a 2D behavioral space. which allows for easier online search and analysis, we apply Distributed Stochastic Neighbor Embedding (t-SNE). Fig. visualizes the points of all the demon- strations in this 2D space and Fig. 2 shows four plots where the points are colored to show the ratios of Marines, Marauders, Hellions, and Siege Tanks that were produced during these games.
4.2 CLUSTERING
For the baseline approach that applies IL to behavioral clusters, use density-based spatial clus- tering of applications with noise (DBSCAN) with 0.02 and minimum number of samples per cluster of 30. We performed grid-search on these two parameters to find the most meaningful data separation; however, the clustering is not perfect due to the many outliers. The clusters are visualized Fig. 2b. with outliers shown black.
Marines
Simillriiies of our appronch with feature input correspanding to cluster centroids
Marusders
Hellises
Siego Tanks
Figure 2: Visualizations of the 2D behavioral space of Terran army unit combinations in 7,777
Terran versus Zerg replays. Each point represents a replay from the Terran player's 's perspective.
The space was reduced using t-SNE. (a) The data points are illuminated (black is low and yellow high) by the ratio of Marines, Marauders, Hellions, or Siege Tanks produced in each game. (b) 62 clusters found by DBSCAN. Cluster centroids are marked with circle and the cluster number and outliers are black. The noticeable cluster has no army units. (c) The similarity between the behaviors of the human players and our approach with four different feature inputs, corresponding to the coordinates of centroids cluster 10. 11, 30, and 32. The behavior of our approach is averaged over 100 games against the easy Zerg bot and its nearest human behavior is marked with star. The behavior of the learned policy can efficiently manipulated to change its behavior. Additionally, control the behavior such that it resembles the behavior of human demonstration.

---

Under review as a conference paper at ICLR 2020
4.3 PERFORMANCE IN STARCRAFT
We trained three groups of neural networks, all with three hidden layers and 256 hidden nodes per layer: (1) One baseline model trained on the whole dataset with no augmentation of behavioral features, (2) BRII. model on the whole dataset with two extra input nodes for the behavioral features (i.e. the coordinates in Fig. 2p). and (3) several cluster baseline models trained only on demonstrations from their respective clusters without the augmented behavioral features.
We applied these trained policy models build-order modules in the scripted StarCraft II Terran bot sc2bot. is important to note that this is a very simplistic bot with several flaws and limitations. Therefore the main goal in this paper is not to achieve human-level performance in StarCraft. but rather to test if BRII allows us to do manipulate its behavior and enables online adaptation. The build-order module, here controlled by one of our policies, is queried with a state description and returns a build-order action, i.c. which building, research, or unit to produce next. The worker and building modules of the bot perform these actions accordingly, while assault, scout, and army modules control the army units. Importantly, policies we test act in a system that consists of both the bot, the opponent bot, and the game world. When we want to utilize our method for adaptation, we are thus not only adapting to the opponent but also the peculiarities of the bot itself.
Distance cluster centroid
Combat units produced
Method
Wins
C10
C11
C30
C32
Marines
Maranders
Hellions
S. Tanks
Reapers
0.58
0.22
0.39
0.75
50.5
0.7 3.2
266 7.6
6.5
1.1
IL (CTO)
3/100
0,05
0.76
0.81
0.71
1.1 + 2.3
0.1 1 0.3
3.11 6.1
0,1 = 0.4
0.1 0.33
IL (CLI)
7/100
0.74
0.00
0.52
0.96
18.8
+ 38.4
0.0 + 0.0
0.0 0 0.0
0.0
0.0 0.0
IL (C30)
18/100
0.76
0.21
0.31
0.79
43.5
1 62.6
0,9 5.
0.2
0.2
0.8
IL (C32)
0/100
0.71
0.94
0.57
0.04
0.1 + 0.2
0.0 0.0
0.0 0.0
0.0
9,9 18.5
BRIL. (C10)
27/100
0.21
0.85
0.81
0.60
2.4 + 4.9
0.0 0.0
14.6 + 18.9
4.0 5.2
0.2 0.6
BRIL (C11)
76/100
0.70
0.05
0.53
0.95
81.4
50.1
0.0 1
0.2
244
0.3 0.6
BRII. (C30)
47/100
0.60
0.31
0.29
0.65
41.6
+ 36.4
2.4 +: 6.7
0.7 2.5
4.1 + 7.8
0,5 + 1.2
BRIL (C32)
16/100
0.42
0.72
0.53
0.36
7.1 # 11.4
1.7 + 6.5
3,2 8.3
6.7 9.7
0.8 : 1.5
Wins euch option
Combut units produced
Method
Win
C10
C11
C30
C32
Marines
Marauders
Hellions
Siege Tanks
Reapers 0.3 E 1.3
BRIL (LCBI)
61/100
5/14
47/59
8/18
179
52.1 1: 47.7
0.5 1: 3.2
4.3 : 12.5
2.5 = 6.2
Table 1: Results in StarCraft using Imitation Learning (IL) on the whole training set, IL on individual clusters (C10, CI1, C30, and C32), Behavioral Repertoire Imitation Learning (BRIL) with fixed behavioral features corresponding to centroids in C10. C11, C30, and C32. Additionally, results are shown in which UCBI selects between the four behavioral features in- be-between games. Each variant played 100 games against the easy Zerg bot. The nearest demonstration in the entire dataset was found based on the bot's mean behavior (normalized army unit combination) and the distance to each cluster centroid are shown. The results demonstrate that by using certain behavioral features, the BRIL policy outperforms the traditional IL approach as well as IL on behavioral clusters.
We will first focus on the results of the traditional IL approach. 100 games on the two-player map CatalystI.F well the corresponding average behaviors (i.e. the army unit ratios). Our bot played as Terran against the built-in Easy Zerg bot. The traditional IL approach won 41/100 games. IL on behavioral clusters showed very poor performance with maximum of 18/100 wins by the model trained on C30. Besides the number of wins, we compute the nearest demonstration in the entire data set from the average behavior and use it an estimate of the policy's position in the 2D behavioral space. From the estimated point, we calculate the distance to each of the four cluster centroids. This analysis revealed that the policies trained on behavioral clusters express behaviors close to the clusters they were trained on (see the distances to the cluster centroids in Table D]. We hypothesize that the poor win rates of this naive approach are due to their training sets being too small such that the policies do not generalize to many of the states explored in the test environment.
the number of wins in
Table also shows the results for BRIL with the coordinates of the four cluster centroids behav- ioral features. BRIL (C11) achieves a win rate of 76/100, thus outperforming traditional IL. These results demonstrate that, for some particular environment, the model can be tuned to achieve higher performance than traditional IL. Analyzing the behavior of the bot with the behavioral features of C11 reveals that it performs an all-in Marine push, similarly to the behavior of the demonstrations
7

---

Under review as a conference paper at ICLR 2020
in C11 (notice the position of C11 Fig. 2 and the illumination of Marines on Fig. a). With the behavioral features of C30, the approach reached a higher win rate than traditional IL; however, not significant. We also notice that for both BRIL and IL on behavioral clusters, the average expressed behavior is closest to the cluster centroid that it was modulated to hehave as, among the four clusters selected. The results show that the behavior of the learned BRII. policy can be successfully controlled. However, the distances are on average larger than for IL on behavioral clusters.
this difference
4.4 ONLINE ADAPTATION
The final test aims to verify that we can indeed use BRIL for online adaptation. We apply the UCBI algorithm to select behavioral features from the discrete set of four options: {C10, CI1, C30, C32) (i.e. the two-dimensional feature descriptions of these cluster centroids). This approach enables the algorithm to switch between behavioral features in-between games based on the return of the previous one, which is I for win and 0 otherwise. The adaptive approach achieves 61/100 wins by identifying the behavior of C11 I as the best option. Not surprisingly, the win rate is lower than when having the behavioral features of C11 fixed, while outperforms traditional IL.
5 DISCUSSION
We proposed two new IL methods in this paper, one which learns a policy that is trained on only one behavioral cluster of data points and one which learns a single modifiable policy on the whole dataset. Our results suggest that policies trained on small behavioral clusters overfit and are thus unable generalize beyond the states available in the cluster. This drawback might be solved with fewer and larger clusters at the cost of losing granularity in the repertoire of policies. If data is abundant, this approach may also work better while we still suspect the same overfitting would occur. BRIL, on the other hand, is simple to implement and results in continuous distribution of policies by adjusting the behavioral features. Additionally, the results suggest that BRIL generalizes better, most likely because it learns from the whole training set. However, that generality potentially comes with the cost of higher divergence between the expected behavior (corresponding to the behavioral input features) and the resulting behavior when tested. While important concern, a divergence is somewhat expected since the test environment is very different from that of the training set (different maps and opponents).
Previous work showed how IL. can kick-start learning before applying RL (Silver et al. 2016 Vinyals al., 2019). With BRIL, one can easily form population of diverse solutions instead just one, which may be a promising approach for domains with a plethora of strategic choices like StarCraft, Promising future work could thus combine BRIL with ideas from AlphaStar to automati- cally form the initial population of policies used in the AlphaStar League.
6 CONCLUSIONS
We introduced new method called Behavioral Repertoire Imitation Learning (BRIL). By label- ing each demonstration d D with behavior descriptor confined within pre-defined behavioral space, BRIL can learn policy (s,, b) over states s E S and behaviors E B. In our experiments, a low-dimensional representation of the behavioral space obtained through dimensionality reduc- tion. The results in this paper demonstrate that BRIL can learn a policy that, when deployed, can be manipulated by conditioning it with a behavioral feature input b, express wide variety of be- haviors. Additionally, the observed behavior of the policy resembles the behavior characterized by Furthermore, a BRIL trained policy can be optimized online by searching for optimal behavioral features in a given setting. In our experiments, policy trained with BRIL was optimized online beyond the performance reached by traditional IL, using UCB1 to select among a set of discrete behavioral features.
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abheel, and Wojeiech Zaremba. Hindsight experience
8

---

Under review as a conference paper at ICLR 2020
play. In Advances in Neural Information Processing Systems, pp. 5048 5058,5058,00 2017.
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469 483, 2009.
Christopher Atkeson and Stefan Schaal. Robot learning from demonstration. In ICML, volume 97, pp. 12-20. Citeseer, 1997.
Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47:235-256, 2002.
Paul Bakker and Yasuo Kuniyoshi. Robot see, robot do: An overview of robot imitation. A/SB96 Workshop on Learning in Robots and Animals, pp. 3-11, 1996.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48. ACM, 2009.
Michael Buro. Real-time strategy games: A new Al research challenge. pp. 1534-1535, 01 2003.
David Churchill and Michael Buro. Build order optimization in Stareraft. In Seventh Artificial Intelligence and Interactive Digital Entertainment Conference, 2011.
David Churchill, Mike Preuss, Florian Richoux, Gabriel Synnaeve. Alberto Uriarte, Santiago On- tanón, and Michal Certický. Stareraft bots and competitions. 2016.
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via a pop- ulation of novelty-secking agents. In Advances in Neural Information Processing Systems, pp- 5027-5038, 2018.
Antoine Cully and J-B Mouret. Evolving behavioral repertoire for walking robot. Evolutionary computation, 24(1):59-88 2016.
Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean- San-Bappistt Mouret. Robots that can adapt like animals. Nature, 521(755)) 2015.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint D0XXvv:882200070 2018.
Bernard Gorman and Mark Humphrys. Imitative learning of combat behaviours in first-person computer games. Proceedings of CGAMES, 2007.
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using neural network with dynamic external memory. Nature, 538 (7626):471, 2016.
Stefan Freyr Gudmundsson, Philipp Eisen, Erik Poromaa, Alex Nodet, Sami Purmonen, Bartlomiej Kozakowski, Richard Meurling, and Lele Cao. Human-like playtesting with deep learning. In 2018 IEEE Conference on Computational Intelligence and Games (CIG), pp. 1-8. IEEF, 2018.
Jack Harmer, Linus Gisslén, Jorge del Val, Henrik Holst, Joakim Bergdahl, Tom Olsson, Kristoffer Sjöö, and Magnus Nordin. Imitation learning with concurrent actions in 3D games. In 2018 IEEE Conference on Computational Intelligence and Games (CIG), pp. 1-8. IEEE, 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pp. 4565-4573, 2016.
Niels Justesen and Sebastian Risi. Learning macromanagement in StarCraft from replays using deep learning. In 2017 IEEE Conference Computational Intelligence and Games (CIG), pp. 162-169. IEEF, 2017.
Niels Justesen, Philip Bontrager, Julian Togelius, and Sebastian Risi. Deep learning for video game playing. IEEE Transactions on Games, 2019.

---

Under review as a conference paper at ICLR 2020
Joel Lehman and Kenneth O Stanley. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary computation, 19(2):189-22) 20.
Joel Lehman and Kenneth Stanley. Evolving diversity of virtual creatures through novelty search and local competition. In Proceedings of the 13th annual conference on Genetic and evolutionary computation, pp- 21 211-218. ACM, 201 1b.
Yunzhu Li, Jiaming Song, and Stefano Ermon. InfoGAIL: Interpretable imitation learning from visual demonstrations. In Advances in Neural Information Processing Systems, pp. 3812-3822, 2017.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9(Nov): 2605, 2008
Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. Teacher-student curriculum learn- ing. arXiv preprint arXiv:1777000133 2017.
Leland Mclnnes, John Healy, and James Melville. Umap: Uniform manifold approximation and
projection for dimension reduction. arXiv preprini
2018.
Josh Merel, Yuval Tassa, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nicolas Heess. Learning human behaviors from motion capture by adversarial imitation. arXiv preprini arXiv: 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04909 2015.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 6292-6299. IEEE, 2018.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, volume 99, pp. 278-287, 1999.
Santiago Ontanón, Gabriel Synnaeve, Alberto Uriarte, Florian Richoux, David Churchill, and Mike Preuss. A survey of real-time strategy game ai research and competition in Starcraft. IEEE Transactions on Computational Intelligence and Al in games, 5(4):293-31, 2013.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-drivee exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16 16-17, 2017.
Justin K Pugh, Lisa B Soros, and Kenneth O Stanley. Quality diversity: A new frontier for evolu- tionary computation. Frontiers Robotics and AI, 3:40, 2016.
Lawrence Saul and Sam Roweis. An introduction to locally linear embedding. Journal of Machine Learning Research, 7, 01 2001.
Stefan Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences, (6):233-24) 2222,22222222 1999.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International conference on machine learning, pp. 1312-1320, 2015.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. nature, 529(7587)4484, 2016.
10

---

Under review as a conference paper at ICLR 2020
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint arXiv:1712.01811 2017.
Peng Sun, Xinghai Sun, Lei Han, Jiechao Xiong, Qing Wang, Bo Li, Yang Zheng, Ji Liu, Yongsheng Liu, Han Liu, et TStarBots: Defeating the Cheating Level Built-in Al StarCraft II in the Full Game arXiv preprint 1809.07799 2018.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, 2nd.
second edition, 2018. URL http:://incompplteeeeeee hht.leas...tttoooheeeeeeeeeeeeee
html
Zhentao Tang, Dongbin Zhao, Yuanheng Zhu, and Ping Guo. Reinforcement learning for build- order production in StarCraft II. In 2018 Eighth International Conference on Information Science and Technology (ICIST), pp. 153-158. IEEE, 2018.
Joshua B. Tenenbaum, Vin De Silva, and John C. Langford. A Global Geometric Framework for Nonlinear Dimensionality Reduction. Science Reports, 1000December):15111155 2000. doi:
0-0007997833-442-2749---8
Christian Thurau, Christian Bauckhage, and Gerhard Sagerer. Imitation learning at all levels of game-AI. In Proceedings of the international conference on computer games, artificial intelli- gence, design and education, volume 5, 2004.
Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al. Starcraft II: new challenge for reinforcement learning. arXiv preprint arXiv: 04782, 2017.
Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathicu, Max Jaderberg, Woj- ciech M. Czamecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, Timo Ewalds, Dan Horgan, Manuel Kroiss, Ivo Danihelka, John Agapiou, Junhyuk Oh, Valentin Dal- ibard, David Choi, Laurent Sifre, Yury Sulsky, Sasha Vezhnevets, James Molloy, Trevor Cai, David Budden, Tom Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Toby Pohlen, Yuhuai Wu, Dani Yogatama, Julia Cohen, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Chris Apps. Koray Kavukcuoglu, Demis Hassabis, and David Silver. AlphaStar: Mastering the Real-Time Strategy Game StarCraft II. hattps://deepminnne com /b/g/////////o aaphastar-ma.coo
2019.
Ziyu Wang, Josh S Merel, Scott E Reed, Nando de Freitas, Gregory Wayne, and Nicolas Heess. Robust imitation of diverse behaviors. In Advances in Neural Information Processing Systems, pp. 5320-5329 2017.
Huikai Wu, Junge Zhang, and Kaiqi Huang. Msc: A dataset for macro-management in StarCraft II. arXiv preprint arXiv: 171003311, 2017.
A APPENDIX
A.1 PREDICTION ACCURACY
Data from StarCraft replays are extracted with sc2reaper tool built using the StarCraft II Learn- ing Environment inspired by the MSC Database (Wu et al. 2017). 7,777 replays of Terran vs. Zerg were processed, extracting state-action pairs every half a second resulting in dataset of 1,625,671 state-action pairs
These states contain an abstraction of the game state similar the one found Justesen & Risi (017).. This abstraction includes: (1) the agent's resources, supply. units and technologies, (2) tally of the enemy's units that have been observed, and (3) the agent's units and technologies in production, including how far they are from being completed.
PIREDACTED for anonymity]
11

---

Under review as a conference paper at ICLR 2020
Method
Mean accuracy
Mean test loss
#f replays
IL
48.090 + 0.080
1.775 0.003
7777
BRIL
48,167 + 0.083
1.768 = 0.003
7777
II. (C10)
32.6088 + 0.417
2.096 -0.03
189
(CI1)
71.326 = 0.316
1.088 = 0.008
IL (C30)
45.709 + 0.499
1.976 = 0.020
149
IL (C32)
45.855 + 0.681
1.770 = 0.006
97
Table 2: Test accuracy and loss for IL, BRIL, and IL trained on clusters 10. 11, 30 and 32. Results
show significant difference between the II. and BRIL in terms of prediction accuracy. BRIL however, able to express multiple behaviors based on the additional input (see Table
Once the replays were post- post-processed for clustering. the dataset was split into training/test/validation following 60% 30% / 10% split per cluster. With this splitting, the three groups of models that were discussed in subsection were trained.
These experiments were carried out ten times per model. Table 2 shows the mean test accuracy and mean test loss over these ten models for the baseline (II.) approach, the novel BRIL approach, and four different cluster baselines (Clusters 10, 11, 30 and 32). which were selected for their wildly different behaviors. The results show that augmenting by behavioral features has no significant effect on the test accuracy loss. However, the next section shows how our new approach is able to express different behaviors with single neural network.
A.2 STARCRAFT
Video games are popular testbeds for the development and testing of AI algorithms (Justesen al., 2019). Real-Time Strategy (RTS) games, such as StarCraft, are among the hardest games for algo- rithms to learn as they contain a myriad of problems, such dealing with imperfect information, adversarial real-time planning, sparse rewards, or huge state and action spaces (Buro (003). Sev- eral algorithms and bots have been built by the AI community (Ontanón et el. [013] Churchil [013]/hurchil al.) 2016) to compete in tournaments such the AIIDE StarCraft AT Competition the CIG StarCraft RTS AI Competitior and the Student StarCraft AI Competitior
This paper deals with the problem of learning policy for build-order planning in StarCraft. Simi- larly to the work by Justesen & Risi 2017). a neural network-
policy is trained using IL from state-action pairs, and the policy is then combined with a bot with scripted procedures for low-level tasks. However, the network in the work by Justesen & Risi (2017) learns an "average" policy, while the approach introduced in this paper is able to learn a behavioral repertoire from demonstrations. The build-order planning problem has also been approached with RL,, optimizing build-order pol- icy for specific bot (Tang et al. 2018 Sun et al. 2018).
The StarCraft II Learning Environment (SC2LE) is an API that allows scripted bots and RL algo- rithms to interact with the game (Vinyals et al. (017). SC2LE also allows data extraction thereby enabling IL from human demonstrations (Wu et al. 2017). AlphaStar is a recent approach that learns a human-level Al policy for StarCraft II using combination of IL and RL (Vinyals et al. 2019). First, a neural network was trained from supervised demonstrations. The IL model was then used seed tournament with several RL agents (called the AlphaStar League). in which the agents competed. This approach resulted in series of competitive and diverse behaviors. Despite the skill level of AlphaStar, does not offer a solution for inter-game adaptation and the diversity of the AlphaStar League has to be managed carefully.
A.3 ARMY COMPOSITIONS IN-GAME
Replays in each of the clusters we examined in the bandit problem (that is, C10, C11, C30 and C32) exhibit a particular army composition in the game. Cluster 11, for example, shows strategy com-
wwn. http://wwww......oooccccccccaaaa
http://cilab.ee http:///illla...oooo....rrr.coot
http://ssaattourraeeee
12

---

Under review as a conference paper at ICLR 2020
BRIL (C10)
BRIL (C11)
BRIL (C30)
BRIL (C32)
Figure 3: Screenshots typical army compositions produced by our trained BRIL policy with behavioral features corresponding to the centroids of cluster 10, 11, 30 and 32. BRIL (C10) executes early timing pushes with Hellions and Cyclones, BRIL (C11) is aggressive with Marines only, BRIL (C30) creates mixed armies with many Marines and Siege Tanks, and BRIL (C32) also creates mixed armies but with less Marines and more Widow Mines.
posed almost purely of Marines (see Fig. D.. Fig. 3 shows screenshots of typical army compositions produced by the BRIL policy with the four different behavioral features.
13


Reviews:
[{"review_id": "rJg75VLZ9B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents Behavioral Repertoire Imitation Learning (BRIL) which is a way to learn a policy via imitation learning that can be modulated with different behavior inputs that adjust the policy's behavior.  Demonstrations used in training are labeled with differences in behavior across dimensions (which are then reduced to two dimensions using t-SNE), and then these behavior labels are provided as additional input when training a NN from demonstrations using behavior cloning.  Experimental results are shown for learning a BRIL policy from 7000+ demonstrations of humans playing StarCraft, and are compared to that of learning a single behavior cloned policy trained on all demonstrations as well as behavior cloned policies trained on subsets of demonstrations clustered by their behavior. \n\nOne of the authors' claims is that BRIL is able to learn a policy that can have a wide variety of behaviors based on the behavior input given as input to the policy which is backed by their results.  They also claim that the BRIL model can be tuned to have higher performance than a policy learned by traditional IL which is shown by their results, although I think that may be somewhat of a function of the environment and set of demonstrations.\n\nAnother claim by the authors in section 4.3 is that behavior can be successfully controlled, and they say \"for both BRIL and IL on behavioral clusters, the average expressed behavior is closest to the cluster centroid that it was modulated to behave as, among the four clusters we selected.\"  I'm not sure this statement is true as IL (C30) is closer to C11 than C30 in table 1.\n\nThe biggest weakness in this paper -- and barrier to acceptance -- is in the really small sample size of the results where only 4 cluster behaviors out of 62 are evaluated.  I would like to see information about the aggregate performance and relative behaviors of BRIL policies compared to the policies trained on the clustered subset of demonstrations across all clusters (or certainly a lot more than just 4 which is less than 10%!) in order to have a better evaluation of the BRIL approach.  The claims of BRIL doing better than clustered IL and being close to a behavior cluster centroid are not that convincing with so few sampled data points.  \n\nIn section 4.2 how do you define or quantify \"the most meaningful data separation\" when doing a grid search on parameters for clustering?\n\nOutside of making things easier to visualize, I'm not sure why it is necessary to reduce the dimensionality of the behavior input for BRIL. Reducing the dimensionality also reduces the set of different policy behaviors that can be expressed.  Additionally, if one might want to adjust a behavior such as having more or less of a certain unit type in StarCraft, the reduced behavior space makes it harder and less intuitive to do so. \n\nShowing that UCB can end up selecting the best BRIL policy (out of 4) and thereby resulting in a better cumulative win record than the traditional IL baseline is kind of trivial and doesn't add much to the paper.  Using a Bayesian optimizer as suggested for future work to select the best behavior inputs for the BRIL policy would be more interesting and give more credence for BRIL being useful in discovering better policies.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1eOuKlwKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThe paper proposes behavioral repertoire imitation learning (BRIL) which aims to learn a collection of policy from diverse demonstrations. BRIL learns such a collection by learning a context-dependent policy, where the context variable represents behavior of each demonstration. To obtain a context variable, BRIL rely on user’s knowledge, where the user manually defines a feature space that describes behavior. This feature space is then reduced by using a dimensionality reduction method such as t-SNE. Lastly, the policy is learned by supervised learning (behavior cloning) with a state-context input variable and an action output variable. The method is experimentally evaluated on the StarCraft environment. The results show that BRIL performs better than two baselines: behavior cloning on diverse demonstrations and behavior cloning on clustered demonstrations. \n\nScore: \nThe weaknesses of the paper are novelty, clarity, and evaluation. Please see the detailed comments below. I vote for rejection. \n\nComments:\n- Novelty of the proposed idea. \nThe major issue of the paper is the lack of novelty. The idea of learning a context-dependent policy in BRIL closely resembles that of existing multi-modal IL methods (Wang et al., 2017, Li et al., 2017). The main difference is that, BRIL relies on a manually defined context variable (behavioral feature space). In contrast, the existing methods aim to learn the context variable from demonstrations. BRIL is too simple when compared to the existing methods. Moreover, using a manually specified feature space does not go well with the main principle of deep learning, which is to learn informative feature spaces from data end-to-end. I think that ICLR is not a suitable venue for this paper. \n\n- Clarity of the proposed method.\nThe second issue of the paper is clarity. Specifically, two important steps of BRIL is policy learning by supervised learning (behavior cloning) and dimensionality reduction by t-SNE. However, explanations of these two steps are vague and incomplete. For example, in Section 2.1, the paper describes IL as supervised learning, but does not mention the issue of covariate shift, which is well-known when treating IL as supervised learning (Ross et al., 2011). Also, it is incorrect to state that an IL agent cannot interact with the environment during training, since many IL methods such as GAIL require interactions with the environment during training. Meanwhile, in Section 2.4, it is unclear how probability distributions in t-SNE reflect similarity between data points. \n\n[1] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. AISTATS, 2011.\n\n- Evaluation of the proposed method is too narrow.\nThe paper lacks important baseline methods in the experiment. Specifically, the paper does not compare BRIL against multi-modal IL methods (Wang et al., 2017, Li et al., 2017) which also learn a context-dependent policy. Moreover, BRIL is evaluated only on the StarCraft environment with only one kind of manually specified feature. This raises a question of generality and sensitivity against the choice of feature of BRIL. To improve the paper, I suggest the authors to compared the proposed method against multi-modal IL methods on different environment, and evaluate BRIL with different choices of the behavioral features.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BJeFAQ3Ttr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work examines the problem of using training a policy which can emulate a variety of different strategies based on a set of demonstrations representing this space of strategies.  The proposed method, BRIL, computes a feature vector for each demonstration, and then employs a dimensionality reduction technique to map the demonstrations to a latent space of strategies.  BRIL then preforms behavioral cloning on these demonstrations, with the reduced representation of the current strategy as an additional input to the policy model.  Empirical evaluation of BRIL is conducted in StarCraft II, where the agent is tasked with scheduling the construction of different units (other aspects of play are controlled by built-in AI).  Results show that when conditioned on good strategies, the BRIL model is superior to a base imitation learning model trained without strategy information.\n\nThe best way to view this work is as a method for learning goal conditioned polices from demonstrations.  While the raw data does not distinguish between different strategies, a postprocessing phase generates task description vectors with specific semantics (the ratios of different unit types built) which are given as input to the BRIL model in addition to the current state.  Therefore, rather than identifying the latent space of strategies present in the demonstration data, BIRL learns a policy which is parameterized by an externally defined space of target behaviors.  The dimensionality reduction step in the strategy space does recover some latent structure, but this is still with respect to a task-specific space of strategy annotations, rather than the demonstrated behaviors themselves.  \n\nWhile this is not an issue with BRIL itself, it should be made clear in the paper that BRIL is learning policies conditioned on explicit strategy descriptions, so that the work can be properly positioned in the literature.  Neither the theoretical discussion nor the empirical results compare BRIL against existing work on learning goal-conditioned policies.\n\nThe empirical results comparing BRIL against a base IL agent are interesting, however, in that BRIL, when conditioned on the best strategy (building mostly marines), actually outperforms IL trained solely on episodes which followed this strategy.  This suggest the possibility that BRIL was able to transfer knowledge from episodes demonstrating other strategies to make up for the limited information available in the demonstrations of the target behavior.\n\nIn section 4.4 the paper briefly discusses the idea of evaluating different strategies executed by the BRIL model, and selecting those that perform best, as a means of learning strong policies for the StarCraft task.  The use of human demonstrations to improve the performance of learning algorithms on difficult control tasks is a widely studied problem, and optimization.  This idea is not explored in any great detail however, and no comparisons with existing methods combining IL with RL are conducted, so the value of BRIL in that context is unclear.  I would, however, recommend this as direction for future work with BRIL.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a way to lean context-dependent policies from demonstrations, where the context represents behavior labels obtained by annotating demonstrations with differences in behavior across dimensions and the reduced in 2 dimensions. Results are conducted in the domain of StarCraft. The main concerns from the reviewers related to the paper’s novelty (as pointed by R2) and experiments (particularly the lack of comparison with other methods and the evaluation of only 4 out of the 62 behaviour clusters, as pointed by R3). As such, I cannot recommend acceptance, as current results do not provide strong empirical evidence about the superiority of the method against other alternatives.
Decision: Reject