Title: LEARNING CALIBRATABLE POLICIES USING PRO-GRAMMATIC STYLE-CONSISTENCY

Abstract: We study the important and challenging problem of controllable generation of long-term sequential behaviors. Solutions to this problem would impact many applications, such as calibrating behaviors of AI agents in games or predicting player trajectories in sports. In contrast to the well-studied areas of controllable generation of images, text, and speech, there are significant challenges that are unique to or exacerbated by generating long-term behaviors: how should we specify the factors of variation to control, and how can we ensure that the generated temporal behavior faithfully demonstrates diverse styles? In this paper, we leverage large amounts of raw behavioral data to learn policies that can be calibrated to generate a diverse range of behavior styles (e.g., aggressive versus passive play in sports). Inspired by recent work on leveraging programmatic labeling functions, we present a novel framework that combines imitation learning with data programming to learn style-calibratable policies. Our primary technical contribution is a formal notion of style-consistency as a learning objective, and its integration with conventional imitation learning approaches. We evaluate our framework using demonstrations from professional basketball players and agents in the Mu-JoCo physics environment, and show that our learned policies can be accurately calibrated to generate interesting behavior styles in both domains.

1. INTRODUCTION:

The widespread availability of recorded tracking data is enabling the study of complex behaviors in many domains, including sports (Chen et al., 2016a; Le et al., 2017; Zhan et al., 2019) , video games (Kurin et al., 2017; Broll et al., 2019) , laboratory animals (Eyjolfsdottir et al., 2014; 2017; Johnson et al., 2016) , facial expressions (Suwajanakorn et al., 2017; Taylor et al., 2017) , commonplace activities such as cooking (Nishimura et al., 2019) , and driving (Bojarski et al., 2016; Chang et al., 2019) . The tracking data is often obtained from multiple experts and can exhibit very diverse styles (e.g., aggressive versus passive play in sports). Our work is motivated by the opportunity to maximally leverage these datasets by cleanly extracting such styles in addition to modeling the raw behaviors.

Our goal is to train policies that can be controlled, or calibrated, to produce different behavioral styles inherent in the demonstration data. For example, Figure 1a depicts demonstrations from real basketball players with variations of many types, including movement speed, desired destinations, tendencies for long versus short passes, and curvature of movement routes, amongst many others. A calibratable policy would be able to generate trajectories consistent with various styles, such as low movement speed as in Figure 1b , or approach the basket as in Figure 1c , or to both styles simultaneously as in Figure 1d . Importantly, we aim to train a single policy that can generate behaviors calibrated across multiple styles. Having such policies would empower many downstream tasks, including behavior discovery (Eyjolfsdottir et al., 2014 ), realistic simulations (Le et al., 2017) , virtual agent design (Broll et al., 2019) , and counterfactual behavioral reasoning (Zhan et al., 2019) .

We focus on three research questions. The first question is strategic: what systematic form of domain knowledge can we leverage to quickly and cleanly extract style information from raw behavioral data? The second question is formulaic: how can we formalize the learning objective to encourage learning style-calibratable policies? The third question is algorithmic: how do we design practical learning approaches that reliably optimize the learning objective? To address these challenges, we present a novel framework inspired by data programming (Ratner et al., 2016) , a paradigm in weak supervision that utilizes automated labeling procedures, called labeling functions, to learn without ground-truth labels. In our setting, labeling functions enable domain experts to quickly translate domain knowledge of diverse styles into programmatically generated style annotations. For instance, it is trivial to write programmatic labeling functions for the two styles-speed and destination-depicted in Figure 1 . Labeling functions also motivate a metric for learning, which we call programmatic style-consistency, to evaluate calibration of policies: rollouts generated for a specific style should return the same style label when fed to the labeling function. Finally, our framework is generic and is easily integrated into conventional imitation learning approaches. To summarize, our contributions are:

• We propose a novel framework for learning policies calibrated to diverse behavior styles.

• Our framework allows users to express styles as labeling functions, which can be quickly applied to programmatically produce a weak signal of style labels. • Our framework introduces style-consistency as a metric to evaluate calibration to styles.

• We present an algorithm to learn calibratable policies that maximize style-consistency of the generated behaviors, and validate it in basketball and simulated physics environments.

2. BACKGROUND: IMITATION LEARNING USING TRAJECTORY VAES:

Since our focus is on learning style-calibratable generative policies, for simplicity we develop our approach with the basic imitation learning paradigm of behavioral cloning using trajectory variational autoencoders, which we describe here. Interesting future directions include composing our approach with more advanced imitation learning approaches as well as with reinforcement learning.

Notation. Let S and A denote the environment state and action spaces. At each timestep t, an agent observes state s t ∈ S and executes action a t ∈ A using a policy π : S → A. The environment then transitions to the next state s t+1 according to a (typically unknown) dynamics function f : S × A → S. For the rest of this paper, we assume f is deterministic; a modification of our approach for stochastic f is included in Appendix B. A trajectory τ is a sequence of T state-action pairs and the last state: τ = {(s t , a t )} T t=1 ∪ {s T +1 }. Let D be a set of N trajectories collected from expert demonstrations. In our experiments, each trajectory in D has the same length T , but in general this does not need to be the case.

Learning objective. We begin with the basic imitation learning paradigm of behavioral cloning (Syed & Schapire, 2008) . The goal is to learn a policy that behaves like the pre-collected demonstrations:

EQUATION

where L imitation is a loss function that quantifies the mismatch between the actions chosen by π and those in the demonstrations. Since we are primarily interested in probabilistic or generative policies, we typically use (variations of) negative log-likelihood: L(τ, π) = T t=1 -log π(a t |s t ), where π(a t |s t ) is the probability of π choosing action a t in state s t .

Trajectory Variational Autoencoders. A common model choice for instantiating π is the trajectory variational autoencoder (TVAE), which is a sequential generative model built on top of variational autoencoders (Kingma & Welling, 2014) , and have been shown to work well in a range of generative policy learning settings (Wang et al., 2017; Ha & Eck, 2018; Co-Reyes et al., 2018) . In its simplest form, a TVAE introduces a latent variable z (also called a trajectory embedding) with prior distribution p, an encoder network q φ , and a policy decoder π θ . Its imitation learning objective is:

L tvae (τ, π θ ; q φ ) = E q φ (z|τ ) T t=1 -log π θ (a t |s t , z) + D KL q θ (z|τ )||p(z) .

(2)

The main shortcoming of TVAEs and related approaches, which we address in Sections 3 & 4, is that the resulting policies cannot be easily calibrated to generate specific styles of behavior. For instance, the goal of the trajectory embedding z is to capture all the styles that exist in the expert demonstrations, but there is no guarantee that the embeddings cleanly encode the desired styles in a calibrated way. Previous work has largely relied on unsupervised learning techniques that either require significant domain knowledge (Le et al., 2017) , or have trouble scaling to complex styles commonly found in real-world applications (Wang et al., 2017; Li et al., 2017) .

3. PROGRAMMATIC STYLE-CONSISTENCY:

Building upon the basic setup in Section 2, we focus on the setting where the demonstrations D contain diverse behavior styles. To start, let y ∈ Y denote a single style label (e.g., speed or destination, as shown in Figure 1 ). Our goal is to learn a policy π that can be explicitly calibrated to y, i.e., trajectories generated by π(•|y) should match the demonstrations in D that exhibit style y.

Obtaining style labels can be expensive using conventional annotation methods, and unreliable using unsupervised approaches. We instead utilize easily programmable labeling functions that automatically produce style labels, described next. We then formalize a notion of style-consistency as a learning objective, and in Section 4 describe a practical learning approach.

Labeling functions. Introduced in the data programming paradigm (Ratner et al., 2016) , labeling functions programmatically produce weak and noisy labels to learn models on otherwise unlabeled datasets. A significant benefit is that labeling functions are often simple scripts that can be quickly applied to the dataset, which is much cheaper than manual annotations and more reliable than unsupervised methods. In our framework, we study behavior styles that can be represented as labeling functions, which we denote λ, that map trajectories τ to style labels y. A simple example is:

EQUATION

which distinguishes between trajectories with large (greater than a threshold c) versus small total displacement. We experiment with a range of labeling functions, as described in Section 6. Multiple labeling functions can be provided at once, possibly from multiple users. Many behavior styles used in previous work can be represented as labeling functions, e.g., agent speed (Wang et al., 2017) . We use trajectory-level labels λ(τ ) in our experiments, but in general labeling functions can be applied on subsequences λ(τ t:t+h ) to obtain per-timestep labels. We can efficiently annotate datasets using labeling functions, which we denote as λ(D) = {(τ i , λ(τ i ))} N i=1 . Our goal can now be phrased as: given λ(D), train a policy π : S × Y → A such that π(•|y) is calibrated to styles y found in λ(D).

Style-consistency.:

A key insight in our work is that labeling functions naturally induce a metric for calibration. If a policy π(•|y) is calibrated to λ, we would expect the generated behaviors to be consistent with the label. So, we expect the following loss to be small:

EQUATION

where p(y) is a prior over the style labels, and τ is obtained by executing the style-conditioned policy in the environment. L style is thus a disagreement loss over labels that is minimized at λ(τ ) = y, e.g., L style λ(τ ), y = 1{λ(τ ) = y} for categorical labels. We refer to (4) as the style-consistency loss, and say that π(•|y) is maximally calibrated to λ when (4) is minimized. Our full learning objective incorporating ( 4) with ( 1) is:

EQUATION

The simplest choice for the prior distribution p(y) is the marginal distribution of styles in λ(D).

The first term in ( 5) is a standard imitation learning objective and can be tractably estimated using λ(D). To enforce style-consistency with the second term, conceptually we need to sample several y ∼ p(y), then several rollouts τ ∼ π(• | y) from the current policy, and query the labeling function for each of them. Furthermore, if λ is a non-differentiable function defined over the entire trajectory, as is the case in (3), then we cannot simply backpropagate the style-consistency loss. In Section 4, we introduce differentiable approximations to more easily optimize the challenging objective in (5).

Multiple styles. Our notion of style-consistency can be easily extended to simultaneously optimize for multiple styles. Suppose we have M labeling functions {λ i } M i=1 and corresponding label spaces {Y i } M i=1 . Let λ denote (λ 1 , . . . , λ M ) and y denote (y 1 , . . . , y M ). Then style-consistency becomes:

EQUATION

Note that style-consistency is optimized when the generated trajectory agrees with all labeling functions. Although this can be very challenging to achieve, it describes the most desirable outcome, i.e. π(•|y) is a policy that can be calibrated to all styles simultaneously.

4. LEARNING APPROACH:

Algorithm 1 Generic recipe for optimizing (5) 1: Input: demonstrations D, labeling functions λ 2: construct λ(D) by applying λ on trajectories in D 3: optimize (7) to convergence to learn C λ ψ * 4: optimize (8) to convergence to learn π * Optimizing ( 5) is challenging due to the long-time horizon and non-differentiability of the labeling functions λ.1 Given unlimited queries to the environment, one could naively employ model-free reinforcement learning, e.g., estimating (4) using rollouts and optimizing using policy gradient approaches. We instead take a model-based approach, described generically in Algorithm 1, that is more computationally-efficient and decomposable. The advantages of our approach are that it is compatible with batch or offline learning, and enables easier diagnosis of deficiencies in the algorithmic framework. To develop our approach, we first introduce a label approximator for λ, and then show how to optimize through the environmental dynamics using a differentiable model-based learning approach.

Approximating labeling functions. To deal with non-differentiability of λ, we approximate it with a differentiable function C λ ψ parameterized by ψ:

EQUATION

Here, L label is a differentiable loss that approximates L style , such as cross-entropy loss when L style is the 0/1 loss. In our experiments we use a recurrent neural net to represent C λ ψ . We then modify the style-consistency term in (5) with C λ ψ * and optimize:

EQUATION

Optimizing L style over trajectories. The next challenge to be addressed is one of credit assignment over time steps. For instance, consider the labeling function in (3) that computes the difference between the first and last states. Our label approximator C λ ψ * may converge to a solution that ignores all inputs except for s 1 and s T +1 . In this case, gradient descent through C λ ψ * provides no information about intermediate timesteps. In other words, effective optimization of style-consistency in (8) requires informative learning signals on all actions taken by the policy.

In general, there are two types of approaches to address this challenge: model-free and model-based. A model-free solution views this credit assignment challenge as analogous to that faced by RL, and repurposes generic reinforcement learning algorithms. We instead choose a model-based approach for two reasons: (a) we found it to be compositionally simpler and easier to debug; and (b) we can use the learned model to obtain hallucinated rollouts of the current policy efficiently during training.

Modeling dynamics for credit assignment. Our model-based approach utilizes a dynamics model M ϕ to approximate the environment's dynamics by predicting the change in state given the current state and action:

EQUATION

where L dynamics is often L 2 or squared-L 2 loss (Nagabandi et al., 2018; Luo et al., 2019) . This allows us to generate trajectories by rolling out: 

s t+1 = s t + M ϕ s t ,

5. RELATED WORK:

Our work combines ideas from imitation learning and data programming, developing a weakly supervised approach for more explicit and fine-grained calibration. This is related to learning disentangled representations and controllable generative modeling, reviewed below.

Imitation learning of diverse behaviors has focused on unsupervised approaches to infer latent variables/codes that capture behavior styles (Li et al., 2017; Hausman et al., 2017; Wang et al., 2017) . Similar approaches have also been studied for generating text conditioned on attributes such as sentiment or tense (Hu et al., 2017) . A typical strategy is to maximize the mutual information between the latent codes and trajectories, in contrast to our notion of programmatic style-consistency.

Disentangled representation learning aims to learn representations where each latent dimension corresponds to exactly one desired factor of variation (Bengio et al., 2012) . Recent studies (Locatello et al., 2019) have noted that popular techniques (Chen et al., 2016b; Higgins et al., 2017; Kim & Mnih, 2018; Chen et al., 2018) can be sensitive to hyperparameters and that evaluation metrics can be correlated with certain model classes and datasets, which suggests that unsupervised learning approaches may, in general, be unreliable for discovering cleanly calibratable representations.

Conditional generation for images has recently focused on attribute manipulation (Bao et al., 2017; Creswell et al., 2017; Klys et al., 2018) , which aims to enforce that changing a label affects only one aspect of the image while keeping everything else the same (similar to disentangled representation learning). We extend these models and compare with our approach in Section 6. Our experimental results suggest that these algorithms do not necessarily scale well into sequential domains.

Enforcing consistency in generative modeling, such as cycle-consistency in image generation (Zhu et al., 2017) , and self-consistency in hierarchical reinforcement learning (Co-Reyes et al., 2018) has proved beneficial. The former minimizes a discriminative disagreement, whereas the latter minimizes a distributional disagreement between two sets of generated behaviors (e.g., KLdivergence). From this perspective, our style-consistency notion is more similar to the former; however we also enforce consistency over multiple time-steps, which is more similar to the latter.

6. EXPERIMENTS:

We first briefly describe our experimental setup and choice of baselines, and then discuss our main experimental results. A full description of the experiments is available in Appendix C.

Data. We validate our framework on two datasets: 1) a collection of professional basketball player trajectories with the goal of learning a policy that generates realistic player-movement, and 2) a Cheetah agent running horizontally in MuJoCo (Todorov et al., 2012) with the goal of learning a policy with calibrated gaits. The former has a known dynamics function: f (s t , a t ) = s t + a t , where s t and a t are the player's position and velocity on the court respectively; we expect the dynamics model M ϕ to easily recover this function. The latter has an unknown dynamics function (which we learn a model of when approximating style-consistency). We obtain Cheetah demonstrations from a collection of policies trained using pytorch-a2c-ppo-acktr (Kostrikov, 2018) to interface with the DeepMind Control Suite's Cheetah domain (Tassa et al., 2018) -see Appendix C for details.

Labeling functions. Labeling functions for Basketball include: 1) average SPEED of the player, 2) DISPLACEMENT from initial to final position, 3) distance from final position to a fixed DESTINATION on the court (e.g. the basket), 4) mean DIRECTION of travel, and 5) CURVATURE of the trajectory, which measures the player's propensity to change directions. For Cheetah, we have labeling functions for the agent's 1) SPEED, 2) TORSO HEIGHT, 3) BACK-FOOT HEIGHT, and 4) FRONT-FOOT HEIGHT that can be trivially extracted from the environment.

We threshold the aforementioned labeling functions into categorical labels (leaving real-valued labels for future work) and use (4) for style-consistency with L style as the 0/1 loss. We use crossentropy for L label and list all other hyperparameters in Appendix C. Whenever we report styleconsistency results, we use 1 -L style in (4) so that all results are easily interpreted as accuracies.

Baselines. We compare our approach, CTVAE-style, with 3 baseline policy models:

1. CTVAE: The conditional version of TVAEs (Wang et al., 2017) . 2. CTVAE-info: CTVAE with information factorization (Creswell et al., 2017) that implicitly maximizes style-consistency by removing all information correlated with y from z. 3. CTVAE-mi: CTVAE with mutual information maximization between style labels and trajectories. This is a supervised variant of unsupervised models (Chen et al., 2016b; Li et al., 2017) , and also requires learning a dynamics model for sampling policy rollouts.

Detailed descriptions and model parameters of baselines are in Appendix A and C respectively. All models build upon TVAEs, which are also conditioned on a latent variable (see Section 2). We highlight that the underlying model choice is orthogonal to our contributions; our framework is compatible with any imitation learning algorithm (see Table 13 in Appendix).

6.1. HOW WELL CAN WE CALIBRATE POLICIES FOR INDIVIDUAL STYLES?:

We first threshold labeling functions into 3 classes for Basketball and 2 classes for Cheetah; the marginal distribution p(y) of styles in λ(D) is roughly uniform over these classes. Then we learn a policy π * calibrated to each of these styles. Finally, we generate rollouts from each of the learned policies to measure style-consistency. Table 1 compares the median style-consistency (over 5 seeds) of learned policies. For Basketball, CTVAE-style significantly outperforms baselines and achieves almost perfect style-consistency for 4 of the 5 styles (the best style-consistency over 5 seeds outperforms all baselines, shown in Tables 8a and 9a in Appendix C). For Cheetah, CTVAE-style outperforms all baselines, but the absolute performance is lower than for Basketball (mostly due to the more complex environment dynamics).

We visualize our CTVAE-style policy calibrated for DESTINATION(net) (with style-consistency of 0.97) in Figure 2 . The green boundaries divide the court into 3 regions, one for each label class. although the policy is calibrated for one style, rollouts still exhibit diverse behaviors (i.e. distribution of trajectories did not collapse into a single mode), which suggests that there are other styles being imitated. Section 6.2 examines this further by testing calibration to multiple styles simultaneously.

We also consider cases in which labeling functions can have several classes and non-uniform distributions (i.e. some styles are more/less common than others). We threshold DESTINATION(net) into 6 classes for Basketball and SPEED into 4 classes for Cheetah and compare the policies in Table 2 . In general, we observe degradation in overall style-consistency accuracies as the number of classes increase. However, CTVAE-style policies still consistently achieve better styleconsistency than baselines in this setting as well. In the appendix, we visualize all 6 classes of DESTINATION(net) in Figure 4 and include another experiment with up to 8 classes of DISPLACEMENT in Table 8c . These results suggest that incorporating programmatic styleconsistency while training via (8) can yield good qualitative and quantitative calibration results.

Basketball We now consider multiple style-consistency as in ( 6), which measures the total accuracy with all labeling functions simultaneously. For instance, in addition to terminating close to the net in Figure 2 , a user may also want to control the speed at which the agent moves towards the target destination.

Table 3 compares the style-consistency of policies calibrated for up to 5 styles for Basketball and 3 styles for Cheetah. Calibrating for multiple styles simultaneously is a very difficult task for baselines, as their style-consistency degrades significantly as the number of styles increases. On the other hand, CTVAE-style sees a modest decrease in style-consistency but is still significantly better calibrated (0.75 style-consistency for all 5 styles vs. only 0.30 for the best baseline in Basketball).

Under review as a conference paper at ICLR 2020 We visualize a CTVAE-style policy calibrated for two styles in Basketball with style-consistency 0.93 in Figure 3 . CTVAE-style outperforms baselines in Cheetah as well, but there is still room for improvement to reach maximal style-consistency in future work. 4 : KL-divergence and negative log-density per timestep for TVAE models (lower is better). CTVAE-style is comparable to baselines for Basketball, but is slightly worse for Cheetah.

6.3. WHAT:

In Table 4 , we investigate whether CTVAEstyle's superior style-consistency is attained at a significant cost to imitation quality, since we jointly optimize both in (5). For Basketball, high style-consistency is achieved without any degradation in imitation quality. For Cheetah, negative log-likelihood is slightly worse; a followup experiment in Table 12 of the appendix shows that we can improve imitation quality with further training, which can sometimes modestly decrease style-consistency.

7. CONCLUSION AND FUTURE WORK:

We propose a novel framework for imitating diverse behavior styles while also calibrating to desired styles. Our framework leverages labeling functions to tractably represent styles and introduces programmatic style-consistency, a metric that allows for fair comparison between calibrated policies. Our experiments demonstrate strong empirical calibration results.

We believe that our framework lays the foundation for many directions of future research. First, can one model more complex styles not easily captured with a single labeling function (e.g. aggressive vs. passive play in sports) by composing simpler labeling functions (e.g. max speed, distance to closest opponent, number of fouls committed, etc.), similar to (Ratner et al., 2016; Bach et al., 2017) ? Second, can we use these per-timestep labels to model transient styles, or simplify the credit assignment problem when learning to calibrate? Third, can we blend our programmatic supervision with unsupervised learning approaches to arrive at effective semi-supervised solutions? Fourth, can we use leverage model-free approaches to further optimize self-consistency, e.g., to fine-tune from our model-based approach? Finally, can we integrate our framework with reinforcement learning to also optimize for environmental rewards?

A BASELINE POLICY MODELS 1) Conditional-TVAE (CTVAE). The conditional version of TVAEs optimizes:

EQUATION

2) CTVAE with information factorization (CTVAE-info). (Creswell et al., 2017; Klys et al., 2018) augment conditional-VAE models with an auxiliary network A ψ (z) which is trained to predict the label y from z, while the encoder q φ is also trained to minimize the accuracy of A ψ . This model implicitly maximizes self-consistency by removing the information correlated with y from z, so that any information pertaining to y that the decoder needs for reconstruction must all come from y.

While this model was previously used for image generation, we extend it into the sequential domain:

EQUATION

)

Note that the encoder in ( 10) and ( 11) differ in that q φ (z|τ ) is no longer conditioned on the label y.

3) CTVAE with mutual information maximization (CTVAE-mi). In addition to (10), we can also maximize the mutual information between labels and trajectories I(y; τ ). This quantity is hard to maximize directly, so instead we maximize the variational lower bound:

EQUATION

where r ψ approximates the true posterior p(y|τ ). In our setting, the prior over labels is known, so H(y) is a constant. Thus, the learning objective is:

EQUATION

Optimizing ( 13) also requires collecting rollouts with the current policy, so similarly we also pretrain and fine-tune a dynamics model M ϕ . This baseline can be interpreted as a supervised analogue of unsupervised models that maximize mutual information in (Li et al., 2017; Hausman et al., 2017) .

B STOCHASTIC DYNAMICS FUNCTION:

If the dynamics function f of the environment is stochastic, we modify our approach in Algorithm 2 by changing the form of our dynamics model. We can model the change in state as a Gaussian distribution and minimize the negative log-likelihood:

EQUATION

where 

∆ t = s t+1 -s t , µ t = M ϕµ (s t , a t ), σ t = M ϕσ (s t ,

C EXPERIMENT DETAILS:

Dataset details. See Table 5 . Basketball trajectories are collected from tracking real players in the NBA. Figure 5 shows the distribution of basketball labeling functions applied on the training set. For Cheetah, we train 125 policies using PPO (Schulman et al., 2017) to run forwards at speeds ranging from 0 to 4 (m/s). We collect 25 trajectories per policy by sampling actions from the policy. We use (Kostrikov, 2018) to interface with (Tassa et al., 2018) . Figure 6 shows the distributions of Cheetah labeling functions applied on the training set.

Training hyperparameters. See Table 6 .

Model parameters. We model all trajectory embeddings z as a diagonal Gaussian with a standard normal prior. Encoder q φ and label approximators C λ ψ are bi-directional GRUs (Cho et al., 2014 ) followed by linear layers. Policy π θ is recurrent for basketball, but not for Cheetah. The Gaussian log sigma returned by π θ is state-dependent for basketball, but state-independent for Cheetah. For Cheetah, we made these choices based on prior work in Mujoco for training gait policies. For Basketball, we observed a lot more variation in the 500k demonstrations so we experimented with more flexible model classes. See Table 7 . Note that SPEED is the most diverse behavior because we pre-trained the policies to achieve various speeds when collecting demonstrations, similar to (Wang et al., 2017) . For more diversity with respect to other behaviors, we can also incorporate a target behavior as part of the reward when pre-training Cheetah policies.

:

This issue is not encountered in previous work on style-dependent imitation learning(Li et al.,

2017;Hausman et al., 2017), since they use purely unsupervised methods such as maximizing mutual information.

Figures and Tables:

Figure 1: Basketball trajectories from policies that are: (a) the expert; (b) calibrated to move at low speeds; (c) calibrated to terminate near the basket (within green boundary); and (d) calibrated for both (b) & (c) simultaneously. Diamonds ( ) and dots (•) indicate initial and final positions.

Figure 2: CTVAE-style rollouts calibrated for DESTINATION(net), 0.97 style-consistency. Diamonds ( ) and dots (•) indicate initial and final positions. Regions divided by green lines represent label classes.

Figure 3: CTVAE-style rollouts calibrated for 2 styles: label class 1 of DESTINATION(net) (see Figure 2) and each class for SPEED, with 0.93 style-consistency. Diamonds ( ) and dots (•) indicate initial and final positions.

Figure 4: Rollouts from our policy calibrated to DESTINATION(net) with 6 classes. The 5 green boundaries divide the court into 6 regions, each corresponding to a label class. The label indicates the target region of a trajectory's final position (•). This policy achieves a style-consistency of 0.93, as indicated in Table8b. Note that the initial position ( ) is the same as in Figures2 and 3for comparison, but in general we sample an initial position from the prior p(y) to compute styleconsistency.

Figure 5: Histogram of basketball labeling functions applied on the training set (before applying thresholds). Basketball trajectories are collected from tracking real players in the NBA.

Figure6: Histogram of Cheetah labeling functions applied on the training set (before applying thresholds). Note that SPEED is the most diverse behavior because we pre-trained the policies to achieve various speeds when collecting demonstrations, similar to(Wang et al., 2017). For more diversity with respect to other behaviors, we can also incorporate a target behavior as part of the reward when pre-training Cheetah policies.

π(s t ) . Then optimizing for styleconsistency in (8) would backpropagate through our dynamics model M ϕ and provide informative learning signals to the policy at every timestep.We outline our model-based approach in Algorithm 2. Lines 10-12 describe an optional step to finetune the dynamics model by querying the environment for trajectories of the current policy (similar toLuo et al. (2019)); we found that this can help improve style-consistency in some experiments.

Table content: <table><tr><td colspan="2">Algorithm 2 Model-based approach for optimizing style-consistency</td><td/></tr><tr><td/><td/><td>Train policy π</td></tr><tr><td>10:</td><td>for nenv iterations do</td><td/></tr><tr><td>11:</td><td>τenv ← collect 1 trajectory from environment with π</td><td/></tr><tr><td>12:</td><td>optimize (9) with τenv</td><td>Fine-tune dynamics model Mϕ</td></tr></table>

Individual Style Calibration: Style-consistency (×10 -2 , median over 5 seeds) of policies evaluated with 4,000 Basketball and 500 Cheetah rollouts. Trained separately for each style, CTVAE-style policies outperform baselines for all styles in Cheetah and 4/5 styles in Basketball.

Table content: <table><tr><td/><td/><td/><td>Basketball</td><td/><td/><td/><td colspan="2">Cheetah</td><td/></tr><tr><td>Model</td><td colspan="2">Speed Disp.</td><td>Dest.</td><td>Dir.</td><td colspan="5">Curve Speed Torso BFoot FFoot</td></tr><tr><td>CTVAE</td><td>83</td><td>72</td><td>82</td><td>77</td><td>61</td><td>59</td><td>63</td><td>68</td><td>68</td></tr><tr><td>CTVAE-info</td><td>84</td><td>71</td><td>79</td><td>72</td><td>60</td><td>57</td><td>63</td><td>65</td><td>66</td></tr><tr><td>CTVAE-mi</td><td>86</td><td>74</td><td>82</td><td>77</td><td>72</td><td>60</td><td>65</td><td>65</td><td>70</td></tr><tr><td>CTVAE-style</td><td>95</td><td>96</td><td>97</td><td>97</td><td>68</td><td>79</td><td>80</td><td>80</td><td>77</td></tr></table>

Fine-grained Style-consistency: (×10 -2 , median over 5 seeds) Training on labeling functions with more classes yields increasingly fine-grained calibration of behavior. Although CTVAEstyle degrades as the number of classes increases, it outperforms baselines for all styles.

Table content: <table><tr><td/><td/><td colspan="3">-DESTINATION(net)</td><td colspan="2">Cheetah -SPEED</td></tr><tr><td>Model</td><td>2 classes</td><td>3 classes</td><td>4 classes</td><td>6 classes</td><td>3 classes</td><td>4 classes</td></tr><tr><td>CTVAE</td><td>87</td><td>82</td><td>78</td><td>74</td><td>45</td><td>37</td></tr><tr><td>CTVAE-info</td><td>87</td><td>81</td><td>75</td><td>77</td><td>49</td><td>39</td></tr><tr><td>CTVAE-mi</td><td>88</td><td>81</td><td>74</td><td>76</td><td>48</td><td>37</td></tr><tr><td>CTVAE-style</td><td>98</td><td>97</td><td>89</td><td>84</td><td>59</td><td>51</td></tr><tr><td colspan="6">6.2 CAN WE CALIBRATE POLICIES FOR MULTIPLE STYLES SIMULTANEOUSLY?</td><td/></tr></table>

Multi Style-consistency: (10 -2 , median over 5 seeds) Simultaneously calibrated to multiple styles, CTVAE-style policies outperform baselines for all styles in Cheetah and in Basketball.

Table content: <table><tr><td/><td/><td colspan="2">Basketball</td><td/><td colspan="2">Cheetah</td></tr><tr><td>Model</td><td>2 styles</td><td>3 styles</td><td>4 styles</td><td>5 styles</td><td>2 styles</td><td>3 styles</td></tr><tr><td>CTVAE</td><td>71</td><td>58</td><td>50</td><td>37</td><td>41</td><td>28</td></tr><tr><td>CTVAE-info</td><td>69</td><td>58</td><td>51</td><td>32</td><td>41</td><td>27</td></tr><tr><td>CTVAE-mi</td><td>72</td><td>56</td><td>51</td><td>30</td><td>40</td><td>28</td></tr><tr><td>CTVAE-style</td><td>93</td><td>88</td><td>88</td><td>75</td><td>54</td><td>40</td></tr></table>

a t ), and M ϕµ , M ϕσ are neural networks that can share weights. We can sample a change in state during rollouts using the reparametrization trick(Kingma & Welling, 2014), which allows us to backpropagate through the dynamics model during training.

Table content: <table/>

for more model details. Dataset parameters for basketball and Cheetah environments. size # batch b n dynamics n label n policy n collect n env learning rate

Table content: <table><tr><td/><td/><td colspan="2">|S| |A|</td><td>T</td><td>N train</td><td>N test</td><td colspan="2">frequency (Hz)</td></tr><tr><td/><td>Basketball</td><td>2</td><td>2</td><td colspan="3">24 520,015 67,320</td><td>3</td></tr><tr><td/><td>Cheetah</td><td>18</td><td>6</td><td>200</td><td>2,500</td><td>625</td><td>40</td></tr><tr><td>Basketball</td><td>128</td><td>4,063</td><td/><td>10 • b</td><td colspan="2">20 • b 30 • b</td><td>128</td><td>0</td><td>2 • 10 -4</td></tr><tr><td>Cheetah</td><td>16</td><td>157</td><td/><td>50 • b</td><td colspan="2">20 • b 60 • b</td><td>16</td><td>1</td><td>10 -3</td></tr></table>

Hyperparameters for Algorithm 2. b is the number of batches to see all trajectories in the dataset once. We also use L 2 regularization of 10 -5 for training the dynamics model M ϕ .

Table content: <table><tr><td/><td colspan="4">z-dim q φ GRU C λ ψ GRU π θ GRU</td><td>π θ sizes</td><td>M ϕ sizes</td></tr><tr><td>Basketball</td><td>4</td><td>128</td><td>128</td><td>128</td><td colspan="2">(128,128) (128,128)</td></tr><tr><td>Cheetah</td><td>8</td><td>200</td><td>200</td><td>-</td><td colspan="2">(200,200) (500,500)</td></tr></table>

Model parameters for basketball and Cheetah environments. Style-consistency wrt. DISPLACEMENT of up to 8 classes (roughly uniform distributions).

Table content: <table><tr><td>Model</td><td/><td>Speed</td><td colspan="5">Displacement Destination</td><td colspan="2">Direction</td><td colspan="3">Curvature</td></tr><tr><td>CTVAE</td><td colspan="5">82 83 85 71 72 74</td><td colspan="7">81 82 82 76 77 80 60 61 62</td></tr><tr><td>CTVAE-info</td><td colspan="5">84 84 87 69 71 74</td><td colspan="7">78 79 83 71 72 74 60 60 62</td></tr><tr><td>CTVAE-mi</td><td colspan="5">84 86 87 71 74 74</td><td colspan="7">80 82 84 75 77 78 58 72 74</td></tr><tr><td colspan="6">CTVAE-style 34 95 97 89 96 97</td><td colspan="7">91 97 98 96 97 98 52 68 83</td></tr><tr><td colspan="11">(a) Style-consistency wrt. single styles of 3 classes (roughly uniform distributions).</td><td/><td/></tr><tr><td>Model</td><td/><td>2 classes</td><td/><td colspan="3">3 classes</td><td colspan="2">4 classes</td><td/><td colspan="2">6 classes</td><td/></tr><tr><td>CTVAE</td><td>86</td><td>87</td><td>87</td><td>80</td><td>82</td><td>83</td><td>76</td><td>78</td><td>79</td><td>70</td><td>74</td><td>77</td></tr><tr><td>CTVAE-info</td><td>83</td><td>87</td><td>88</td><td>79</td><td>81</td><td>83</td><td>73</td><td>75</td><td>78</td><td>71</td><td>77</td><td>78</td></tr><tr><td>CTVAE-mi</td><td>86</td><td>88</td><td>88</td><td>80</td><td>81</td><td>84</td><td>71</td><td>74</td><td>79</td><td>73</td><td>76</td><td>78</td></tr><tr><td colspan="2">CTVAE-style 97</td><td>98</td><td>99</td><td>68</td><td>97</td><td>98</td><td>35</td><td>89</td><td>95</td><td>67</td><td>84</td><td>93</td></tr><tr><td colspan="12">(b) Style-consistency wrt. DESTINATION(net) with up to 6 classes (non-uniform distributions).</td><td/></tr><tr><td>Model</td><td colspan="2">2 classes</td><td/><td>3 classes</td><td/><td colspan="2">4 classes</td><td colspan="2">6 classes</td><td colspan="3">8 classes</td></tr><tr><td>CTVAE</td><td colspan="12">91 92 93 79 83 84 76 79 79 68 70 72 64 66 69</td></tr><tr><td>CTVAE-info</td><td colspan="12">90 90 92 83 83 85 75 76 77 68 70 72 60 63 67</td></tr><tr><td>CTVAE-mi</td><td colspan="12">90 92 93 81 84 86 75 77 80 66 70 72 62 62 67</td></tr><tr><td colspan="13">CTVAE-style 98 99 99 15 98 99 15 96 96 02 92 94 80 90 93</td></tr><tr><td>(c) Model</td><td/><td>2 styles</td><td/><td colspan="2">3 styles</td><td/><td colspan="2">4 styles</td><td/><td colspan="2">5 styles</td><td/></tr><tr><td>CTVAE</td><td>67</td><td>71</td><td>73</td><td>58</td><td>58</td><td>62</td><td>49</td><td>50</td><td>52</td><td>27</td><td>37</td><td>35</td></tr><tr><td>CTVAE-info</td><td>68</td><td>69</td><td>70</td><td>54</td><td>58</td><td>59</td><td>48</td><td>51</td><td>54</td><td>28</td><td>32</td><td>35</td></tr><tr><td>CTVAE-mi</td><td>71</td><td>72</td><td>73</td><td>48</td><td>56</td><td>61</td><td>45</td><td>51</td><td>52</td><td>16</td><td>30</td><td>31</td></tr><tr><td colspan="2">CTVAE-style 92</td><td>93</td><td>94</td><td>86</td><td>88</td><td>90</td><td>62</td><td>88</td><td>88</td><td>66</td><td>75</td><td>80</td></tr></table>

[min, median, max]  style-consistency (×10 -2 , 5 seeds) of policies evaluated with 4,000 basketball rollouts each. CTVAE-style policies significantly outperform baselines in all experiments and are calibrated at almost maximal style-consistency for 4/5 labeling functions. We note some rare failure cases with our approach, which we leave as a direction for improvement for future work.

Table content: <table><tr><td/><td>Speed</td><td/><td colspan="2">Torso Height</td><td colspan="2">B-Foot Height</td><td colspan="2">F-Foot Height</td></tr><tr><td>Model</td><td>NLD</td><td>SC</td><td>NLD</td><td>SC</td><td>NLD</td><td>SC</td><td>NLD</td><td>SC</td></tr><tr><td>CTVAE-style</td><td>-0.28</td><td>79</td><td>-0.24</td><td>80</td><td>-0.16</td><td>80</td><td>-0.22</td><td>77</td></tr><tr><td>CTVAE-style+</td><td>-0.49</td><td>70</td><td>-0.42</td><td>83</td><td>-0.36</td><td>80</td><td>-0.42</td><td>74</td></tr></table>

We report the median negative log-density per timestep (lower is better) and styleconsistency (higher is better) of CTVAE-style policies for Cheetah (5 seeds). The first row corresponds to experiments in Tables1 and 9a, and the second row corresponds to the same experiments with 50% more training iterations. The KL-divergence in the two sets of experiments are roughly the same. Although imitation quality improves, style-consistency can sometimes degrade (e.g. SPEED, FRONT-FOOT HEIGHT), indicating a possible trade-off between imitation quality and style-consistency.

Table content: <table><tr><td/><td/><td/><td>Style-consistency ↑</td><td/><td/><td/></tr><tr><td>Model</td><td>Min</td><td>-</td><td>Median</td><td>-</td><td>Max</td><td>NLD ↓</td></tr><tr><td>RNN</td><td>79</td><td>79</td><td>80</td><td>81</td><td>81</td><td>-7.7</td></tr><tr><td>RNN-style</td><td>81</td><td>86</td><td>91</td><td>95</td><td>98</td><td>-7.6</td></tr><tr><td>CTVAE</td><td>81</td><td>82</td><td>82</td><td>82</td><td>82</td><td>-8.0</td></tr><tr><td>CTVAE-style</td><td>91</td><td>92</td><td>97</td><td>98</td><td>98</td><td>-7.8</td></tr></table>

Comparing style-consistency (×10 -2 ) between RNN and CTVAE policy models for DESTINATION in basketball. The style-consistency for 5 seeds are listed in increasing order. Our algorithm improves style-consistency for both policy models at the cost of a slight degradation in imitation quality. In general, CTVAE performs better than RNN in terms of both style-consistency and imitation quality.

Table content: <table><tr><td/><td>Speed</td><td>Displacement</td><td colspan="2">Destination</td><td>Direction</td><td>Curvature</td></tr><tr><td>L label</td><td>3.96 ± 0.33</td><td>4.58 ± 0.20</td><td colspan="2">1.61 ± 0.18</td><td>3.19 ± 0.25</td><td>28.31 ± 0.95</td></tr><tr><td/><td colspan="5">(a) Basketball labeling functions for experiments in section 6.1.</td></tr><tr><td/><td>Speed</td><td colspan="2">Torso Height</td><td colspan="2">B-Foot Height</td><td>F-Foot Height</td></tr><tr><td>L label</td><td>3.24 ± 0.83</td><td colspan="2">15.87 ± 1.78</td><td colspan="2">17.25 ± 0.73</td><td>14.75 ± 0.74</td></tr><tr><td/><td colspan="5">(b) Cheetah labeling functions for experiments in section 6.1.</td></tr></table>

Mean and standard deviation cross-entropy loss (L label , ×10 -2 ) over 5 seeds of learned label approximators C λ ψ * on test trajectories after n label training iterations for experiments in section 6.1. C λ ψ * is only used during training; when computing style-consistency for our quantitative results, we use original labeling functions λ. M ϕ test error Basketball 1.47 ± 0.59(×10 -7 ) Cheetah 1.93 ± 0.08(×10 -2 )

Table content: <table/>

Average mean-squared error of the dynamics model M ϕ per timestep per dimension on test trajectories after training for n dynamics iterations

Table content: <table/>

Peer review (iclr2020):
This paper proposes a method to train style-conditional policies. The method has three components, a dynamics model, a labeling procedure and its approximation, and a policy. The model is trained on real trajectories, while the policy is trained on both real and simulated data coming from the model. The policy is trained to both imitate and be sensitive to the style labelling of states. Such a method yields policies that can be executed with styles chosen externally, e.g. by a user. This paper proposes a novel method that is an interesting take on imitation learning, but it is hard to judge how relevant this method is, as the paper has several inconsistencies and weaknesses that need to be resolved before it is accepted. Inconsistencies: the reported NLL results do not match the trajectories shown in Figures. Some equations do not seem to reflect what is being optimized. Weaknesses: Many quantities that should be reported are not: quality of models, diversity of policies, etc. The source of datasets should be clarified. (see detailed comments) It*s interesting that this method can leverage these very sparse or poor quality annotations, but it would be helpful to get a sense of how good the annotations provided here are. What*s the threshold were annotation*s quality is too low to be helpful? In the general case I suspect this threshold to be higher than what is hinted at in this paper, especially given Table 2. What happens in more realistic settings where e.g. users may specify hundreds of different labels? Detailed comments: - how dependent on having a variety of policies is this method? It*s not clear how diverse the set of policies coming from the basketball dataset nor the Cheetah policies is, nor how this diversity affects learning. An experiment with explicitly different levels of diversity would strengthen understanding of this method. For Cheetah, it would be easy to report p(y) as a function of the target forward speed, that would give readers a sense of diversity for each label. - For something like cheetah, with the labels that you propose being a very simple function of the state space, this somewhat resembles DIAYN [1] but with some grounding. Would it make sense to compare to a similar baseline? - (Table 1) Why report (only) the median over 5 seeds? Papers usually report means. Plus, an indication of variance would be nice. - It*s not clear what the basketball dataset is. Where does it come from? Does it come with a simulator? If not, how do you evaluate style consistency or run step 11-12 of Alg. 2? Do you have a train/valid/test split to choose hyperparameters? (the appendix only suggests a train/test split) - Why not cite MuJoCo? [2] - For Figure 2 & 3, I suggest lowering the transparency/alpha value of the trajectories, as there is a lot of overlap. - Table 1 is somewhat confusing. In (4) and the paragraph thereafter, you define mathcal{L}^{style} as an error rate, i.e. when lambda(	au) 
eq y, but in Table 1 you seem to report instead accuracy? (i.e. when lambda(	au) = y) If so, then you are reporting percentages, so 	imes 10^2 rather than 10^-2. - You never report how well C_psi is doing, and it*s not clear to me why C_psi is needed at all. When optimizing (8) are you directly treating (8)*s inner terms as negative rewards which is differentiated wrt pi*s parameters? If so, you are doing a form of DDPG, but there is a problem: after (4) you mention that L^style is not differentiable, meaning C_psi doesn*t provide gradient information to pi. I assume that you acutally optimize (8) with L^label? Whether that*s what you*re doing or not, it should be clarified. If you are truly using L^style, then it*s not clear why C_psi is needed, as there is no differentiability anyways, and simply using lambda directly will provide more signal. - I*m somewhat perplexed by the values of Table 4. A log-likelihood of -190 represents a probability of 10^-83 (a _negative_ LL of -190 is, on the other hand, impossible by virtue of logs and probabilities, but I assume it is a *typo*). What is this the probability of? Entire trajectories? If so it would make more sense to report the _average_ log-likelihood, i.e. per timestep, because at this point in the paper, readers have no sense of how long trajectories are, and thus what these likelihoods represent. (for example, a NLL of 190 could be an average likelihood of 0.5 for 275 steps, or of 0.1 for 83 steps, or of .8 for 850 steps, which are all very different results! According to the appendix Table 5, the basketball trajectories are 25 steps long, which would mean that the imitation objective is not respected at all, exp(-190/25)=.0005, which would mean that pi(a|s) is .0005 on average.) - Again Table 4, if the *-* is indeed a typo, then CTVAE-style is actually performing better than the baselines, especially for Cheetah (normally one wants negative log-likelihood to be as close to 0 as possible). Same for Table 10, if the NLL column is truly actually log-likelihood, then the *style+* objective really degrades imitation quality rather than improves it. [1] Diversity is All You Need: Learning Skills without a Reward Function, Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, Sergey Levine [2] MuJoCo: A physics engine for model-based control, Emanuel Todorov, Tom Erez Yuval Tassa