Learning Calibratable Policies using Programmatic Style-Consistency
Authors: Eric Zhan
Albert Tseng
Yisong Yue
Adith Swaminathan
Matthew Hausknecht
Venue:  (2020)
Keywords: imitation learning
conditional generation
data programming

Abstract: We study the important and challenging problem of controllable generation of long-term sequential behaviors. Solutions to this problem would impact many applications, such as calibrating behaviors of AI agents in games or predicting player trajectories in sports. In contrast to the well-studied areas of controllable generation of images, text, and speech, there are significant challenges that are unique to or exacerbated by generating long-term behaviors: how should we specify the factors of variation to control, and how can we ensure that the generated temporal behavior faithfully demonstrates diverse styles? In this paper, we leverage large amounts of raw behavioral data to learn policies that can be calibrated to generate a diverse range of behavior styles (e.g., aggressive versus passive play in sports). Inspired by recent work on leveraging programmatic labeling functions, we present a novel framework that combines imitation learning with data programming to learn style-calibratable policies. Our primary technical contribution is a formal notion of style-consistency as a learning objective, and its integration with conventional imitation learning approaches. We evaluate our framework using demonstrations from professional basketball players and agents in the MuJoCo physics environment, and show that our learned policies can be accurately calibrated to generate interesting behavior styles in both domains.

Under review as a conference paper at ICLR 2020
LEARNING CALIBRATABLE POLICIES USING PRO- GRAMMATIC STYLE-CONSISTENCY
Anonymous authors Paper under double-blind review
ABSTRACT
We study the important and challenging problem of controllable generation of long-term sequential behaviors. Solutions to this problem would impact many applications, such as calibrating behaviors of Al agents in games or predicting player trajectories in sports. In contrast to the well-studied areas of controllable generation of images, text, and speech, there are significant challenges that are unique to or exacerbated by generating long-term behaviors: how should we spec- ify the factors of variation to control, and how can we ensure that the generated temporal behavior faithfully demonstrates diverse styles? this paper, we lever- age large amounts of raw behavioral data to learn policies that can be calibrated to generate diverse range of behavior styles (c.g., aggressive versus passive play in sports). Inspired by recent work on leveraging programmatic labeling functions, we present a novel framework that combines imitation learning with data pro- gramming learn style-calibrttable policies. Our primary technical contribution is a formal notion of style-consistency as a learning objective, and its integra- tion with conventional imitation learning approaches. We evaluate our framework using demonstrations from professional basketball players and agents in the Mu- JoCo physics environment, and show that our learned policies can be accurately calibrated to generate interesting behavior styles in both domains.
INTRODUCTION
The widespread availability of recorded tracking data is enabling the study of complex behaviors in many domains, including sports (Chen et al. [2016a] Le et al., 2017 et al.] (019). video games (Kurin et al. 2017 Broll et al. 2019), laboratory animals (Eyjolfsdottir et al. 2014 2017 Johnson let a 2016)), facial expressions (Suwajanakorn et al. 2017 Taylor et al. (2117) commonplace activ- ities such as cooking (Nishimura et al. 2019). and driving (Bojarski et al. 2016 Chang et al. 2019). The tracking data is often obtained from multiple experts and can exhibit very diverse style styles (e.g., aggressive versus passive play in sports). Our work is motivated by the opportunity to maximally leverage these datasets by cleanly extracting such styles in addition to modeling the raw behaviors.
Our goal is to train policies that can be controlled, or calibrated, to produce different behavioral styles inherent in the demonstration data. For example, Figure [ depicts demonstrations from real basketball players with variations of many types, including movement speed, desired destinations, tendencies for long versus short passes, and curvature of movement routes, amongst many others. A calibratable policy would be able to generate trajectories consistent with various styles, such as low movement speed in Fi, or approach the basket as in Figure T.. or both styles simul- taneously as in Figure [t. Importantly, we aim to train a single policy that can generate behaviors calibrated across multiple styles. Having such policies would empower many downstream tasks, in- cluding behavior discovery (Eyjolfsdottir et al. 2014), realistic simulations (Le et al. 2017). virtual agent design (Broll et al. 2019). and counterfactual behavioral reasoning (Zhan et al. 2019).
We focus three research questions. The first question is strategic: what systematic form of domain knowledge can we leverage to quickly and cleanly extract style information from raw behavioral data? The second question is formulaic: how we formalize the learning objective to encourage learning style-calibratable policies? The third question is algorithmic: how do we design practical learning approaches that reliably optimize the learning objective?

---

Under review as a conference paper at ICLR 2020
b
(a) Expert demonstrations
(b) Style: SPEED
(c) Style: DESTINATION
(d) Both styles
Figure I: Basketball trajectories from policies that are: (a) the expert; (b) calibrated to move at low speeds; (c) calibrated terminate near the basket (within green boundary); and (d) calibrated for both (b) & (c) simultaneously. Diamonds and dots (+) indicate initial and final positions.
To address these challenges, we present a novel framework inspired by data programming (Ratner ct al. 2016), paradigm in weak supervision that utilizes automated labeling procedures, called labeling functions, to learn without ground-truth labels. In our setting, labeling functions enable domain experts to quickly translate domain knowledge of diverse styles into programmatically gen- erated style annotations. For instance, it is trivial to write programmatic labeling functions for the two styles-speed and destination-depictee in Figure [ Labeling functions also motivate met- ric for learning, which we call progrummatic style-consistency. to evaluate calibration of policies: rollouts generated for a specific style should return the same style label when fed to the labeling func- tion. Finally, our framework is generic and is easily integrated into conventional imitation learning approaches. To summarize, our contributions are:
We propose novel framework for learning policies calibrated to diverse behavior styles.
Our framework allows users to express styles as labeling functions, which can be quickly applied programmatically produce a weak signal of style labels.
Our framework introduces style-consistency as metric evaluate calibration to styles.
We present an algorithm to learn calibratable policies that maximize style-consistency of the generated behaviors, and validate in basketball and simulated physics environments.
BACKGROUND: IMITATION LEARNING USING TRAJECTORY VAES
Since our focus is learning style-calibratable generative policies, for simplicity develop approach with the basic imitation learning paradigm of behavioral cloning using trajectory varia- tional autoencoders, which we describe here. Interesting future directions include composing our approach with more advanced imitation learning approaches as well as with reinforcement learning.
Notation.
Let S and A denote the environment state and action spaces. At each timestep 4, an agent observes state 8; E S and executes action a; E A using a policy ग S + A. The environment then transitions to the next state S++1 according to a (typically unknown) dynamics function f : S x A + S. For the rest of this paper, we assume f is deterministic; modification of our approach for stochastic the last state: т = {s,, a)) { Let 'D be set of N trajectories collected from expert demonstrations. our experiments, each trajectory in D has the same length T, but in general this does not need to be the case.
included in Appendix B A trajectory is a sequence of T state-action pairs and
Learning objective. We begin with the basic imitation learning paradigm of behavioral cloning (Syed & Schapire 2008). The goal is to learn policy that behaves like the pre-collected demon-
strations:
[imiiaation (T, (
π arg min E,ND
where Cimitation is loss function that quantifies the mismatch between the actions chosen by л and those in the demonstrations. Since we are primarily interested in probabilistic or generative policies, we typically use (variations of) negative log- likelihoohoo:: L(T, = Σ - -log 7T (a, (st), where
((a (S:) is the probability of choosing action a in state St-
2

---

Under review as a conference paper at ICLR 2020
Trajectory Variational Autoencoders. A common model choice for instantiating π is the trajectory variational autoencoder (TVAE). which is a sequential generative model built on top of variational autoencoders (Kingma & Welling 2014), and have been shown to work well in range of gener- ative policy learning settings (Wang et al,, 2017 Ha & Eck, 2018 Co-Reyes et al. 2018). In its simplest form, TVAE introduces a latent variable (also cal called trajectory emhedding) with prior distribution p, an encoder network qá, and a policy decoder sT. Its imitation learning objective is:
L (T, no: go) = (x
Σ log No (a /sc. z) + DEL (10
The main shortcoming of TVAEs and related approaches, which we address in Sections 3 & is that the resulting policies cannot be easily calibrated to generate specific styles of behavior. For instance, the goal of the trajectory embedding is to capture all the styles that exist in the expert demonstrations, but there is no guarantee that the embeddings cleanly encode the desired styles in a calibrated way. Previous work has largely relied on unsupervised learning techniques that either require significant domain knowledge {Le et al. 2017), or have trouble scaling to complex styles commonly found in real woaldworld applications (Wang et al. 2017 Li et al. 2017
PROGRAMMATIC STYLE-CONSISTENS
Building upon the basic setup in Section 2] we focus on the setting where the demonstrations D contain diverse behavior styles. To start, 1et E Y denote single style label (c.g.. speed or destination, as shown in Figure D. Our goal is to learn a policy ι that can be explicitly calibrated to y, i.e., trajectories generated by (-lyy) should match the demonstrations in 'D that exhibit style
Obtaining style labels can be expensive using conventional annotation methods, and unreliable using unsupervised approaches. We instead utilize easily programmable labeling functions that automat- ically produce style labels, described next. We then formalize a notion of style-consistency as a learning objective, and in Section describe a practical learning approach.
Labeling fuunctions. Introduced in the data programming paradigm (Ratner et al. 2016). labeling functions programmatically produce weak and noisy labels to learn models on otherwise unlabeled datasets. A significant benefit is that labeling functions are often simple scripts that can be quickly applied to the dataset, which is much cheaper than manual annotations and more reliable than unsu- pervised methods. In our framework, we study behavior styles that can be represented as labeling functions, which we denote X, that map trajectories 7 to style labels y. A simple example is:
X(7) 11 [1| -5112 > c),
which distinguishes between trajectories with large (greater than a threshold e) versus small total displacement. We experiment with range of labeling functions, as described in Section Multiple labeling functions can be provided once, possibly from multiple users. Many behavior styles used in previous work can be represented as labeling functions, c.g., agent speed (Wang et al.) 2017). We use trajectory-level labels X(7) our experiments. but in general labeling functions can be applied on subsequences A(TE++)) to obtain per-timestep labels. We can efficiently annotate datasets using labeling functions, which we denote as A(D) = {(T,, ((T.))) Our goal can now be phrased as: given A(D), train policy S Y + A such that r(-ly) is calibrated to styles y found in X(D).
Style-consistency. key insight our work is that labeling functions naturally induce metric for calibration. If policy T-lly) is calibrated to X, we would expect the generated behaviors to be consistent with the label. So, we expect the following loss to be small:
Eysply),r
Catyle (X(T).)).
(4)
where p(y) is prior over the style labels, and 7 is obtained by executing the style-conditioned policy in the environment. Latyee is thus disagreement loss over labels that is minimized at A(T) y, e.g., gstye (A(T).y) = 1 (((()) * y) for categorical labels. We refer to (4) as the style-consistency loss, and say that क( (-l)) maximally calibrated to À when ( is minimized. Our full learning objective incorporating (4) with !1 is:
cimitasion (T.((- [x(7))] + -y-p0)) [<th
= arg min (1,66))~((D)
(5)
3

---

Under review as a conference paper at ICLR 2020
The simplest choice for the prior distribution p(y) is the marginal distribution of styles in A(D). The first term in (5) is a standard imitation learning objective and can be tractably estimated using (DD). To enforce style-consistency with the second term, conceptually we need to sample several y ~ p(y), then several rollouts 7 ~ x( y) from the current policy, and query the labeling function for each of them. Furthermore, if À is non-differentiable function defined over the entire trajectory. as is the case in (3. then we cannot simply backpropagate the style-consistency loss. In Section 4 introduce differentiable approximations to more easily optimize the challenging objective in (5).
Multiple styles. Our notion of style-consistency can be easily extended to simultaneously optimize for multiple styles. Suppose we have M labeling functions {A, M and corresponding label spaces (Y)), Let X denote (A1,--.;A) Aw) and denote (y1-- - --M)- Then style-consistency becomes:
M
C tyle Σ
(MG)))))))))))))
(6)
Note that style-consistency is optimized when the generated trajectory agrees with all labeling func- tions. Although this can be very challenging to achieve, it describes the most desirable outcome, i.e. π (-ly) is a policy that can be calibrated to all styles simultaneously.
4 LEARNING APPROACH
Optimizing (5) is challenging due the long-time horizon and non-differentiabiliiy of the laheling functions X Given unlim-
Algorithm 1 Generic recipe for optimizing (5)
1: Input: demonstrations D, labeling functions À
ited queries to the environment, one could
2: construct A(D) by applying À on trajectories in 'D
naively employ model-free reinforcement
3: optimize 7 convergence to learn CAA
learning, e.g., estimating (4) using rollouts
4: optimize 8 convergence to learn
and optimizing using policy gradient ap- proaches. We instead take a model-based approach, described generically in Algorithm D that is more computationally-effiiees compatible with batch offline learning, and enables easier diagnosis of deficiencies in the algo- rithmic framework. To develop our approach, we first introduce label approximator for X, and then show how to optimize through the environmental dynamics using diffcrentiable model-based learning approach.
and decomposable. The advantages of our approach are that it is
Approximating labeling functions. To deal with non-differentiability of X, we approximate it with a differentiable function Cp parameterized by db:
arg min EE
((((7))~((D) Clabel (T)
Here, Clabel is differentiable loss that approximates Layle such as cross-entropy loss when Layle the 0/1 loss. In experiments style-consistency term in (5) with Co. and optimize:
recurrent neural net represent Co., We then modify the
x. arg min E (AAD))))A( clmitucion (T, ((a(-1((1))]] + Bysp())r (ct (T))
(8)
Optimizing Style over trajectories. The next challenge to be addressed is one of credit assignment over time steps. For instance, consider the labeling function in (3) that computes the difference between the first and last states. Our label approximator C... may converge a solution that ignores
all inputs except for $ and $1+1- In this case, gradient descent through e provides no information about intermediate timesteps. In other words, effective optimization of style-consistency in (8) requires informative learning signals on all actions taken by the policy.
In general, there are two types of approaches to address this challenge: model-free and madel- A model-free solution views this credit assignment challenge as analogous that faced by RL, and
'This issue is not encountered in previous work on style-dependent imitation learning (Li et al. 2017 Hausman al. 2017, since they use purely unsupervised methods such maximizing mutual information.
4

---

Under review as a conference paper at ICLR 2020
repurposes generic reinforcement learning algorithms. We instead choose model-based approach for two reasons: (a) we found it to be compositionally simpler and easier to debug; and (b) we can use the learned model to obtain hallucinated rollouts of the current policy efficiently during training.
Modeling dynamics for credit assignment. Our model-based approach utilizes a dynamics model Mp approximate the environment's dynamics by predicting the change in state given the current state and action:
50" arg min ETAD Σ dyyaamics (M, at), (5t+1 - 5t)):
Luo et al. 2019). This allows us to generate trajectories by rolling out: S++1 = S; Mp 1 (s,, ग (s,) Then optimizing for style- consistency in would backpropagate through our dynamics model M₁ and provide informative learning signals to the policy at every timestep.
where Cdynamics is often La squared-Lg loss (Nagabandi et al. 2018
We outline model-based approach in Algorithm Lines 10 describe optional step fine- tune the dynamics model by querying the environment for trajectories of the current policy (similar to Luo et al. (2099); ); we found that this can help improve style-consistency in some experiments.
Algorithm Model-based approach for optimizing style-consistency
1: Input: demonstrations D. labeling function X, label approximator Cp. dynamics model Myp
2: X(D) + (T,))
3: for Tddynamics iterations do
4: optimize ? with batch from D
Train dynamics model Mc
5: for Raahhs iterations do
optimize 7 with batch from X(D)
Train label approximator C
7: for Riollcy itcrations do
8: 15 4- collect Tealleer trajectories with My and current policy т 9: optimize (8) with batch from X(D) and B
D. Train policy
10:
for Tleny ilerations do
Tanv f- collect trajectory from environment with x
12:
optimize (9) with Tew
D Fine-tune dynamics model My
5
RELATED WORK
Our work combines ideas from imitation learning and data programming, developing a weakly su- pervised approach for more explicit and fine-grained calibration. This is related to learning disen- langled representations and controllable generative modeling, reviewed below.
Imitation learning of diverse behaviors has focused on unsupervised approaches infer latent variables/codes that capture behavior styles (Li et al. 2017 Hausman et al., 2017 Waag et al., 2017). Similar approaches have also been studied for generating text conditioned on attributes such as sentiment or tense (Hu et al. 2017). 7 A typical strategy is to maximize the mutual information be- tween the latent codes and trajectories, contrast our notion of programmatic style-consistency.
Disentangled representation learning aims to learn representations where each latent dimension corresponds to exactly one desired factor of variation (Bengio al. 2012). Recent studies (Locatello et al. 2019) have noted that popular techniques (Chen et al. 2016b] Higgins et al. [2017] Kim & Mnih 2018 Chen et al. 2018) can be sensitive to hyperparameters and that evaluation metrics can be correlated with certain model classes and datasets, which suggests that unsupervised learning approaches may, in general, be unreliable for discovering cleanly calibratable representations.
Conditional generation for images has recently focused on attribute manipulation (Bao et al. 2017 Creswell et al. 2017, Klys et al. 2018), which aims to enforce that changing a label affects only one aspect of the image while keeping everything else the same (similar to disentangled representation learning). We extend these models and compare with our approach in Section 6 Our experimental results suggest that these algorithms do not necessarily scale well into sequential domains.
Enforcing consistency generative modeling, such as cycle-consistency in image generation Zhu et al. 2017), and seff-consistency in hierarchical reinforcement learning (Co-Reyes et al.,
5

---

Under review as a conference paper at ICLR 2020
2018) has proved beneficial. The former minimizes a discriminative disagreement, whereas the latter minimizes a distributional disagreement between two sets of generated behaviors (e.g.. KL- divergence). From this perspective. our style-consistency notion is more similar to the former; however we also enforce consistency over multiple time-steps, which is more similar to the latter.
6 EXPERIMENTS
We first briefly describe our experimental setup and choice of basclines, and then discuss our main experimental results. A full description of the experiments is available in Appendix C
Data. We validate our framework on two datasets: 1) a collection of professional basketball player trajectories with the goal of learning a policy that generates realistic player-
and 2) Cheetah agent running horizontally in MuJoCo (Todorov et al. (012) with the goal of learning a policy with calibrated gaits. The former has known dynamics function: [[s,, a,) = S. + a,, where and a, are the player's position and velocity on the court respectively: we expect the dynamics model Mp to easily recover this function. The latter has an unknown dynamies function (which we learn model of when approximating style-consistency) We obtain Cheetah demonstrations from (Kostrikov 2018) to interface C for details.
a collection of policies trained with the DeepMind Control Suite's 's Cheetah domain (Tassa 2018
see Appendix
Labeling functions. Labeling functions for Basketball include: 1) average SPEED of the player, 2) DISPLACEMENT from initial to final position, 3) distance from final position to fixed DESTINATION on the court (e.g. the basket), 4) mean D DIRECTTON of travel, and 5) CURVATURE of the trajectory, which measures the player's propensity change directions. For Cheetah, we have labeling functions for the agent's 1) SPEED, 2) TORSO HEIGHT, 3) BACK-FOOT HEIGHT, and 4) F FRONT-FOOT HEIGHT that can be trivially extracted from the environment.
We threshold the aforementioned labeling functions into categorical labels (leaving real-valued la- bels for future work) and use 4 for style-consistency with L as the 0/1 loss. We use cross- entropy for glabel and list all other hyperparameters in Appendix C Whenever we report style- consistency results, we use 1 - Lstyle in 44 so that all results are easily interpreted accuracies.
Baselines. We compare our approach, CTVAE-style, with 3 baseline policy models:
1. CTVAE:
The conditional version of TVAEs (Wang et al. 2017).
2. CTVAE-info: CTVAE with information factorization (Creswell et al. 2017) that implicitly maximizes style-consisenncy by removing all information correlated with y from
3. CTVAE-mi: CTVAE with mutual information maximization between style labels and tra- jectories. This a supervised variant of unsupervised models (Chen al. 2016b Li al,, 2017), and also requires learning a dynamics model for sampling policy rollouts.
Detailed descriptions and model parameters of baselines are in Appendix A and respectively. All models build upon TVAEs, which are also conditioned on a latent variable (see Section 2). We highlight that the underlying model choice is orthogonal to our contributions; compatible with any imitation learning algorithm (see Table 13 in Appendix).
framework is
6.1 How WELL CAN WE CALIBRATE POLICIES FOR INDIVIDUAL STYLES?
We first threshold labeling functions into classes for Basketball and 2 classes for Cheetah; the marginal distribution p(y) of styles in X(D) is roughly uniform over these classes. Then we learn a policy nt" calibrated each of these styles. Finally, we generate rollouts from each of the learned policies to measure style-consistency. [Table[]comparee, the median style-consistency (over 5 seeds) of learned policies. For Basketball, CTVAA--ttye significantly outperforms baselines and achieves almost perfect style-consistency for 4 of the 5 styles (the best style-consistency over 5 seeds out- performs all baselines, shown in Tables 8a and 9a in Appendix C.. For Cheetah, CTVAE-style outperforms all baselines, but the absolute performance is lower than for Basketball (mostly due to the more complex environment dynamics).
We visualize CTVAE-style policy calibrated for DESTINATION (net) (with style-consistency of 0.97) in Figure 2 The green boundaries divide the court into regions, one for each label class. Policy rollouts almost always terminate in the corresponding region of the label class. Note that

---

Under review as a conference paper at ICLR 2020
Basketball
Cheetah
Model Speed
Disp.
Dest.
Dir.
Curve
Speed
Torso
BFoot
FFoot
CTVAE
83
72
82
77
61
59
63
68
CTVAE-info
84
71
79
72
60
57
63
65
66
CTVAE-mi
86
74
82
77
72
60
65
65
70
CTVAEsstyle
95
96
97
68
79
80
77
Table 1: Individual Style Calibration: Style-consistency (x10-?, median over seeds) of poli- cies evaluated with 4,000 Basketball and 500 Cheetah rollouts. Trained separately for each style, CTVAE-style policies outperform baselines for styles Cheetah and 4/5 styles Basketball.
Figure 2: CTVAE-style rollouts calibrated for DESTINATION (net), 0.97 style-consistency. Diamonds (+) and dots (.) indicate initial and final positions. Regions divided by green lines represent label classes.
D
D
(a) Label class 0 (close) (b) Label class (mid)
(c) Label class 2 (far)
although the policy is calibrated for one style, rollouts still exhibit diverse behaviors (i.e. distribution of trajectories did not collapse into single mode), which suggests that there are other styles being imitated. Section 6.2 examines this further by testing calibration to multiple styles simultaneously.
We also consider cases in which labeling functions can have several classes and non-uniform distri- butions (i.e. some styles are more/less common than others). We threshold DESTINATION (net) into classes for Basketball and SPEED into 4 classes for Cheetah and compare the policies in Table 2] In general, we observe degradation in overall style-comsistency accuracies as the num- ber of classes increase. However, CTVAE-style policies still consistently achieve better style- consistency than baselines in this setting as well. the appendix, we visualize all 6 classes of DESTINATION (net) in Figure 4 and include another experiment with up to 8 classes of DISPLACEMENT in Table Sc These results suggest that incorporating programmatic style- yield good qualitative and quantitative calibration results.
consistency while training via
Basketball
DEST INTTINATIONN (net)
Cheetah SPEED
Model
2 classes
classes
4 classes
6 classes
3 classes
classes
CTVAE
87
82
78
74
45
37
CTVAE-info
87
81
75
77
49
39
CTVAE-mi
88
81
74
76
48
37
CTVAE-style 98
89
84
59
51
Table Fine-grained Style-consistency: (x10-2, median seeds) Training on labeling func- tions with more classes yields increasingly fine-grained calibration of behavior. Although CTVAE- style degrades as the number of classes increases, it outperforms baselines for all styles.
6.2
CAN WE CALIBRATE POLICIES FOR MULTIPLE STYLES SIMULTANBOUSLY??
We now consider multiple style-consistency as in (6), which measures the total accuracy with all labeling functions simultancously. For instance, in addition to terminating close to the net in Figure 2. user may also want to control the speed at which the agent moves towards the target destination.
Table 3 compares the style-consistency of policies calibrated for up to 5 styles for Basketball and styles for Cheetah. Calibrating for multiple styles simultaneously is a very difficult task for base- lines, their style-consistency degrades significantly other hand, CTVAE-style sees a modest decrease in style-consistency but still significantly better calibrated (0.75 style-consiseency for all styles vs. only 0.30 for the best baseline in Baskethall).
the number of styles increases. On the

---

Under review as a conference paper at ICLR 2020
Basketball
Cheetah
Model
2 styles
styles
4 styles
5 styles
2 styles
3 styles
CTVAE
50
37
28
CTVAE-info
69
58
51
32
41
27
CTVAE-mi
72
56
51
30
40
28
CTVAE-style
93
88
75
54
Table Multi Style-consistency; (10-2, median over seeds) Simultaneously calibrated multi- ple styles, -tyeE- policies outperform baselines for all styles in Cheetah and in Basketball.
Figure CTVAE-style rollouts calibrated for styles: label class of DESTINATION (net) (see Figure 2 and each class for SPEED, with 0.93 style-consistency. Diamonds and dots indicate initial and final positions.
D
D
(a) Label class 0 (slow)
(b) Label class (mid)
(c) Label class 2 (fast)
We visualize a CTVAE-style policy calibrated for styles in Basketball with style-consistency 0.93 in Figure | CTVAE-style outperforms baselines in Cheetah as well, but there is still room for improvement reach maximal syleeconsistency future work.
6.3 WHAT IS THE TRADE-OFF BETWEEN COYESSCONSSST AND IMITATION QUALITY?
Basketball
Cheetah
In Table we investigate whether CTVAE- style's 's superior syyle-consistenyy is attained at significant cost to imitation quality, since we jointly optimize both in (5). For Basketball, high style-consistency is achieved without any degradation in imitation quality. For Chee- tah, negative log- likeliooi slightly worse; a followup experiment in Table [2 of the ap- pendix shows that we can improve imitation quality with further training, which can some- times modestly decrease style-consistency.
Model
DEL
NLD
DKL
NLD
TVAE
2.5
-7.9
29
-0.60
CTVAE
2.5
-8.0
29
-0.59
CTVAE-info
2.3
-7.9
29
-0.58
CTVAE-mi
2.6
-8.0
29
-0.57
CTVAE-style
2.3
-7.8
30
-0.28
Table 4: KL-divergence and negative log-density
per timestep for TVAE models (lower is better). CTVAE-style is comparable to baselines for Bas- ketball, but slightly worse for Cheetah.
CONCLUSION AND FUTURE WORK
We propose a novel framework for imitating diverse behavior styles while also calibrating to desired styles. Our framework leverages labeling functions to tractably represent styles and introduces pro- grammatic style-consistency, a metric that allows for fair comparison between calibrated policies. Our experiments demonstrate strong empirical calibration results.
believe that our framework lays the foundation for many directions future research. First, can one model more complex styles not easily captured with single labeling function (e.g. aggressive vs. passive play in sports) by composing simpler labeling functions (e.g. max speed, distance to closest opponent, number of fouls committed, etc.), similar (Ratner et al. 2016 [aach et al. 2017] Second, can we use these per-timestep labels to model transient styles, or simplity the credit assignment problem when learning calibrate? Third, can we blend programmatic supervision with unsupervised learning approaches to arrive at effective semi-supervised solutions? Fourth, can use leverage model-free approaches further optimize self-consistency, e.g., fine-tune from our model-based approach? Finally, can we integrate our framework with reinforcement learning to also optimize for environmental rewards?
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Stephen H. Bach, Bryan Dawei Hc, Alexander Ratner, and Christopher Ré. Learning the structure of generative models without labeled data. In International Conference on Machine Learning (ICML), 2017.
Jianmin Bao. Dong Chen, Fang Wen, Houqiang and Gang Hua. CVAE-GAN fine-grained image generation through asymmetric training. TEFE International Conference on Computer Vision (ICCV), 2017.
Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. Unsupervised feature learning and deep learning: review and new perspectives. arXiv preprint aroiv: 2012.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp. Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End end learning for self-driving cars. arXiv preprint arXiv: 1604.07316, 2016.
Brian Broll, Matthew Hausknecht, Dave Bignell, and Adith Swaminathan. Customizing scripted bots: Sam- ple efficient imitation learning for human like behavior in minecraft. AAMAS Workshop on Adaptive and Learning Agents, 2019.
Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagject Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, al. Argoverse: 3d tracking and forecasting with rich maps. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Jianhui Chen, Hoang M Le, Peter Carr, Yisong Yuc, and James J Little. Learning online smooth predictors for realtime camera planning using recurrent decision trees. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4688-4698 2016a.
Tian Qi Chen, Xuechen Li, Roger B. Grosse, and David Duvenaud. Isolating sources disentanglement variational autoencoders. In Neural Information Processing Systems (NeurIPS), 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Picter Abbecl. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Neural Information Pro- cessing Syxtems (NewrIPS), 2016b.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk, and Yoshua Ben- gio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv: /406. 1078, 2014.
John D Co Ceyes,, YuXuan Liu, Abhishek Gupta, Benjamin Eysenbach, Pieter Abbeel, and Sergey Levine. Self-consistent trajecctr autoencoder: Hierarchical reinforcement learning with trajectory embeddings. In International Conference on Machine Learning (ICMI),, 2018.
Antonia Creswell, Anil A. Bharath, and Biswa Sengupta. Conditional autoencoders with adversarial informa- tion factorization. arXiv preprint arXiv:1711.05 2017.
Eyrun Ryjolfsdottir, Steve Branson, Xavier P Bargos-Artizzu, Fric D Hoopfer, Jonathan Schor, David J Ander- son, and Pictro Perona. Detecting social actions of fruit flies. European Conference on Computer Vision, pp. 772-787. Springer, 2014.
Eyrun Eyjolfsdottir, Kristin Branson, Yisong Yue, and Pietro Perona. Learning recurrent representations for hierarchical behavior modeling. International Conference on Learning Representations (ICLR), 2017.
David Ha and Douglas Eck. neural representation of sketch drawings. International Conference on Learning Representations (ICLR), 2018
Karol Hausman, Yevgen Chebotar, Stefan Schaal, Gaurav Sukhatme, Joseph Lim. Multi-moda imitation learning from unstructured demonstrations using generative adversarial nets. In Neural Information Processing Systems (NeurIPS), 2017.
Irina Higgins, Loie Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mo- hamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR, 2017.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. Toward controlled genera- tion of text. In International Conference Machine Learning (ICML), 2017.
Matthew Johnson, David K Duvenaud, Alex Wiltschko, Ryan P Adams, and Sandeep R Datta. Composing graphical models with neural networks for structured representations and fast inference. In Advonces in neural information proessingg systems, 2016.

---

Under review as a conference paper at ICLR 2020
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference Machine Learn- ing (ICML), 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference Learn- ing Representations (ICLR). 2014.
Jack Klys, Jake Snell, and Richard S. Zemel. Learning latent subspaces in variational autoencoders. In Neural Information Processing Systems (NewrlPS), 2018.
Πlya Kostrikov. Pytorch implementatiions of reinforcement learning algorithms. httts:://itthu. com/ -lottriiee//yyttrrrc------------
gail_ 2018.
Vitaly Kurin, Sebastian Nowozin, Katja Hofmann, Lucas Beyer, and Bastian Leibe. The atari grand challenge dataset, arXiv preprint arXiv:1705. 2017.
Hoang M Le. Yisong Yue, Peter Carr, and Patrick Lucey. Coordinated multi-agent imitation learning. In International Conference on Machine Learning (ICML) 2017.
Yunzhu Li, Jiaming Song. and Stefano Ermon. Infogail: Interpretable imitation learning from visual demon- strations. Neural Information Processing Systems (NeurIPS), 2017.
Francesco Locatello, Stefan Bauer, Mario Lucie, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions the unsupervised learning of disentangled representations. In Interna- tional Conference on Machine Learning (ICML), 2019.
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. In International Conference on Learning Representations (ICLR), 2019.
Anusha Nagabandi, Gregory Kahn, Ronald S. Fearing, and Sergey Levine. Neural network dynamics for model- based deep reinforcement learning with model-free fine-tuning, In International Conference on Robotics and Automation (ICRA), 2018
Taichi Nishimura, Atsushi Hashimoto, Yoko Yamakata, and Shinsuke Mori. Frame selection for producing recipe with pictures from execution video of a recipe. In Proceedings of the 11th Workshop on Multimedia for Cooking and Eating Activities, pp. 9- 16. ACM, 2019.
Alexander Ratner, Christopher De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Cre- ating large training sets, quickly. Neural Information Processing Systems (NeurIPSS, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint 1rXiv:1707.06347, 2017.
Supasorn Suwajanakom, Steven M Seitz, and Ira Kemelnaachr--Shiizemman Synthesizing obama: learning lip syne from audio. ACM Transactions on Graphics (TOG), 36(4):95, 2017.
Umar Syed and Robert E Schapire. A game-theoretic approach apprenticeship learning. In Advances in neural information processing systems. pp. 1449-1456, 2008
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq. Timothy Lillicrap, and Martin A. Riedmiller. Deepmind control suite. arXiv preprint 1801:08801000600 2018.
Sarah Taylor, Taehwan Kim, Yisong Yue, Moshe Mahler, James Krahe, Anastasio Garcia Rodriguez, Jessica Hodgins, and lain Matthews. deep learning approach for generalized speech animation. ACM Transactions on Graphics (TOG), 36(4):93, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Inter- national Conference on Intelligent Robots and Systems (IROS), 2012.
Ziyu Wang, Josh Merel, Scott Reed, Greg Wayne, Nando de Freitas, and Nicolas Heess. Robust imitation of diverse behaviors. In Neural Information Processing Systems (NeurIPS), 2017.
Eric Zhan, Stephan Zheng, Yisong Yue, Long Sha, and Patrick Lucey. Generating multi-agent trajectories using programmatic weak supervision. In International Conference on Learning Representations (ICLR), 2019.
Jun-Yas Yan Zhu, Taesung Park, Phillip Isola, and Alexei Efros. Unpaired image-to-imase to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232. 2017.
10

---

Under review as a conference paper at ICLR 2020
A BASELINE POLICY MODELS
1) Conditional-TVAE (CTVAE).
The conditional version of TVAES optimizes:
no; Lowae To: qo) = (x(ra))
Σ log To (ar Sc, y. + (xL()0(417
(10)
2) CTVAE with information factorization (CTVAE-info). (Creswell et 2017 Klys et al, 2018) augment conditional- VAE models with an auxiliary network Ag which is trained to predict the label y from z, while the encoder dø also trained minimize the accuracy of Ag. This model implicitly maximizes self-consistency by removing the information correlated with from z, so that any information pertaining y that the decoder needs for reconstruction must all come from y. While this model was previously used for image generation, extend it into the sequential domain:
max 0,6
(el min L (A,()).))
log no (a (s. z, Dx (10()))))))
(11)
Note that the encoder in 10 and differ in that (z(T) is no longer conditioned on the label
3) CTVAE with mutual information maximization (CTVAE-mi). addition to (T0). we can
also maximize the mutual information between labels and trajectories T); This quantity is hard to maximize directly, so instead we maximize the variational lower bound:
I(y; By- [log (ylr)] + H(yy),
(12)
where re approximates the true posterior p(ylr). In setting, the prior over labels is known, so H(y) is a constant. Thus, the learning objective is:
L ctvvaemi
T, NT; (o) = Ltve (T, To) + Ey-ply))rmm
log (y[r)]-
(13)
Optimizing 13) also requires collecting rollouts with the current policy, so similarly we also pretrain and fine-tune dynamics model Mp- This baseline can be interpreted as supervised analogue of unsupervised models that maximize mutual information (Li Li al. 2017 Hausman et al. (017).
B STOCHASTIC DYNAMICS FUNCTION
the dynamics function f of the environment stochastic, we modify our approach in Algorithm 2 by changing the form of our dynamics model. We can model the change in state as a Gaussian distribution and minimize the negative log-likelihood:
Pu 4 = arg min ENAD
log p0gp(A ot),
(14)
where A, St+1 St, 14 MMp (Sc, ar). o, Mg. (Se, at), and M Mp. are neural networks that can share weights. We can sample a change in state during rollouts using the reparametrization trick (Kingma & Welling 2014), which allows us backpropagate through the dynamics model during training.
C EXPERIMENT DETAILS
Dataset details. See Table 5 Basketball trajectories are collected from tracking real players in the NBA. Figure 5 shows the distribution basketball labeling functions applied on training set. For Cheetah, we train 125 policies using PPO (Schulman et al.) 2017) to run forwards at speeds ranging from 0 to (m/s). We collect 25 trajectories per policy by sampling actions from the policy. We use [Kostrikov] 2018) to interface with (Tassa et al., 2018 Figure shows the distributions of Cheetah labeling functions applied on the training set.
Training hyperparameters. See Table

---

Under review as a conference paper at ICLR 2020
Model parameters. We model all trajectory embeddings diagonal Gaussian with standard normal prior. Encoder 96 and label approximators C. are bi-directional GRUs (Cho et al. 2014) followed by linear layers. Policy xy recurrent for baskethall, but not for Cheetah. The Gaussian log sigma returned by is state-dependent for basketball, but state-independent for Cheetah. For Cheetah, we made these choices based on prior work in Mujoco for training gait policies. For Basketball, observed lot more variation in the 500k demonstrations more flexible model classes. See Table 7 for more model details.
experimented with
[
[A]
T
Nurin
Nest
frequency (Hz)
Basketball
2
24
520,013
67,320
Cheetah
18
200
2,500
625
40
Table Dataset parameters for basketball and Cheetah environments.
batch size
batch
7ldynamics
Yilabel
Pppoliy
Ticollet
Teenv
learning rate
Basketball
128
4.063
10 b
20 b
30
128
0
2 10
Cheetah
16
157
50
20 b
60
16
10-3
Table 6: Hyperparameters for Algorithm 2
b is the number of batches all trajectories in the dataset once. We also use L2 regularization of 10-5 for training the dynamics model Ma.
z-dim
do GRU
C GRU
We GRU
Ty sizes
Mp sizes (128,1 (28,128) (500,500)
Baskethall
4
128
128
128
(128,128)
Cheetah
8
200
200
(200,200)
Table 7: Model parameters for basketball and Cheetah environments.
12

---

Under review as a conference paper at ICLR 2020
Model
Speed
Displacement
Destination
Direction
Curvature
CTVAE
82
83
85
71
72
74
81
82
82
76
77
80
60)
61
62
CTVAE-info
78
79
83
71
72
74
60
60
62
CTVAE-mi
84
86
87
71
74
74
80
82
84
75
77
78
58
72
74
CTVAE-style
34
95
97
89
96
97
91
97
98
96
97
98
52
68
83
(a) Style-consisenncy wrt. single styles of classes (roughly uniform distributions).
Model
classes
3 classes
classes
6 classes
CTVAE
86
87
87
80
82
83
76
78
79
70
74
77
CTVAE-info
83
87
88
79
81
83
73
75
78
71
77
78
CTVAE-mi
86
88
88
80
81
84
71
74
79
73
76
78
CTVAE-styke 97
98
99
68
97
98
35
89
95
67
84
93
(b) Style-consistency wrt. DEST INSTINATON (net) with up to classes (non-unifoon distributions).
Model
classes
3 classes
4 classes
classes
classes
CTVAE
91
92
93
79
83
84
76
79
79
68
70
72
64
66
69
CTVAE-info
90
90
92
83
83
85
75
76
77
68
70
72
60
63
67
CTVAE-mi
93
84
86
75
77
80
62
62
67
CTVAE-style
98
99
99
15
98
99
15
96
96
02
92
94
80
90
93
(c) Style- consisteey wrt. DISPLACEMEN of 8 classes (roughly uniform distributions).
Model
2 styles
3 styles
4 styles
5 styles
CTVAE
67
71
73
58
58
62
49
52
27
37
35
CTVAE-
68
69
70
54
58
59
48
51
54
28
32
35
CTVAE-m
71
72
73
48
56
45
52
16
30
31
CTVAB-style
92
93
94
86
88
90
62
88
88
66
75
80
(d) Style-comsistency wrt. multiple styles simultaneously.
Table 8: [min, median, max] style-consistency (x10-2, seeds) of policies evaluated with 4,000 basketball rollouts each. CTVAE-style policies significantly outperform baselines in all experiments and are calibrated at almost maximal style-consistency for 4/5 labeling functions. We note some rare failure cases with our approach, which we leave direction for improvement for future work.
Model
Speed
Torso Height
B-Foot Height
F-Foot Height
CTVAE
53
59
62
62
63
70
61
68
73
63
68
72
CTVAE-info
56
57
61
62
63
72
58
65
72
63
66
69
CTVAE-mi
53
60
62
62
65
70
60
65
70
66
70
73
CTVAE-style
68
79
81
79
80
84
80
88
74
77
80
(a) Style-consistency single styles of classes (roughly uniform distributions).
Model
3 classes
4 classes
Model
2 styles
3 styles
CTVAE
41
45
49
35
37
41
CTVAE
39
41
43
25
28
29
CTVAE-info
49
52
36
39
42
CTVAE-infa
39
41
46
25
27
30
CTVAE-mi
47
48
53
36
37
38
CTVAE-m
34
40
48
27
28
31
CTVAE-style
65
42
51
60
CTVAE-style
43
54
38
40
52
(b) Style-consise consistencstnyyye wrt. SPEED with varying # of (c) coyle-oonsittency wrt. multiple styles simultane- classes (non-uniform distributions).
ously.
Table 9: [min, median, max] style-consistency 2x 5 seeds) of policies evaluated with 500 Cheetah rollouts each. CTVAE-style policies consistently outperform all baselines, but we note that there is still room for improvement (to reach 100% kyyeecoonsitency).
13

---

Under review as a conference paper at ICLR 2020
Model
Speed
Displacement
Destination
Direction
Curvature
CTVAE
84 + 1.0
72 + 0.9
82 0.6
77 1.0
61 0.8
CTVAE-info
85 = 1.2
70 = 1.2
81 = 1.7
72 - 1.2
60 + 0.9
CTVAE-mi
86 = 1.5
73 = 1.5
82 1.1
77 1.1
71 + 3.4
CTVAE-style
81 + 31.4
94 = 3.4
94 + 3.8
97 + 0.5
67 + 12.6
(a) Style-consistency wrt. single styles of classes (roughly uniform distributions).
Model
2 classes
3 classes
classes
6 classes
CTVAE
87 + 0.7
82 1.3
77 + 1.7
75 + 1.8
CTVAE-info
86 + 1.9
81 + 1.6
75 +: 2.9
76 + 3.2
CTVAE-mi
88 + 0.4
82 1.8
75 + 3.4
75 2.0
CTVAE-style
98 = 0.8
86 + 14.4
74 + 26.8
82 = 13.0
(b) Style-consistency wit. DESTINATION (net) with to 6 classes (non-unifont distributions).
Model
classes
3 classes
4 classes
6 classes
8 classes
CTVAE
92 + 0.4
82 2.6
78 1.4
70 = 1.4
66 1.9
CTVAE-info
91 + 0.8
84 + 1.2
76 = 0.6
70 = 1.1
64 - 3.2
CTVAE-mi
92 + 1.4
83 2.3
77 = 2.5
68 = 2.2
64 = 2.5
CTVAE-style
+ 0.3
77 41.2
75 + 40.0
+ 42.9
88 5.8
(c) Styyescoonsi wrt. DISPLACEMENT of up 8 classes (roughly uniform distributions).
Model
stylees
3 styles
4 styles
5 styles
CTVAE
70 + 2.3
59 1.7
50 + 1.6
32 3.1
CTVAE-info
69 + 1.0
57 + 2.3
50 + 1.9
32 1.7
CTVAE-mi
72 + 0.8
52 + 5.1
51 + 0.8
26 4 7.1
CTVAE-style
93 + 1.2
88
1.6
87 2.5
76 3.3
(d) Style-consistency multiple styles simultaneously.
Table 10: Mean and standard deviation style-consistency (x10-2, 5 seeds) of policies evaluated with 4,000 basketball rollouts each. CTVAB-style policies generally outperform baselines. Lower mean style-consistency (and large standard deviation) for CTVAE-style is often due to failure cases, as can be seen from the minimum coyeiseonss values we report in Table g Understanding the causes of these failure cases and improving the algorithm's 's stability are possible directions for future work.
Model
Speed
Torso Height
B-Foot Height
F-Foot Height
CTVAE
57 + 3.9
64 3.1
67 + 4.2
69 3.7
CTVAE-info
58 + 2.1
65 + 4.2
64 + 5.4
66 + 2.7 70 2.6
CTVAE-mi
58 + 3.9
65 + 3.6
CTVAE-style
77
81 = 2.2
82 = 5,4
77 + 2.4
(a) Style-consisenncy wrt. single styles of 2 classes (roughly uniform distributions).
Model
3 classes 4 classes
Model
styles
3 styles
CTVAE
45 + 3.2
38 2.9
CTVAE
41 + 1.6
27 + 1.9
CTVAE-info
49 + 1.8
39 + 2.8
CTVAE-inf info
42 2.3
28 + 2.2
CTVAE-m mi
49 + 2.2
37 1.0
CTVAE- mi
41 + 4.9
29 + 1.6
CTVAE-
61 + 2.9
51 7.8
CTVAE-style
53 - 6.1
43 5.8
(b) Style-consee coall-consitencyy classes non-uniform distributions).
SPEED with varying # of (c) Style-consistency wrt. multiple styles simultanc- ously.
Table 11: Mean and standard deviation style-consistency (x10~2 5 seeds) of policies evaluated with 500 Cheetah rollouts each. CTVAE-style policies consistently outperform all baselines, but we note that there is still room for improvement (to reach 100% style-consistency..
14

---

Under review as a conference paper at ICLR 2020
Speed
Torso Height
B-Foot Height
F-Foot Height
Model NLDD
SC
NLD
SC
NLDD
SC
NLD
SC
CTVAE-style -0.28
79
-0.24
80
-0.16
80
-0.22
77
CTVAE-style+
-0.49
-0.42
83
-0.36
80
-0.42
74
We report the median negative log-density per timestep (lower is better) and style- consistency (higher is better) of CTVAE-style policies for Cheetah (5 seeds). The first row cor- responds to experiments
Table 12:
Tables [ and [aI and the second row corresponds ments with 50% more training iterations. The KI--divergence in the two sets of experiments are roughly the same. Although imitation quality improves, style-consistency can sometimes degrade (e.g. SPEED, FRONT-FOOT HEIGHT), indicating possible trade-off between imitation quality and style-consistency.
the same experi-
Style-consistency
Model
Min
Median
Max
NLD L
RNN
79
80
81
-7.7
RNN-style
81
86
91
95
98
-7.6
CTVAR
81
82
82
82
82
-8.0
CTVAE-style
91
92
97
98
98
-7.8
Table 13: Comparing style-consistency (x10-2) between RNN and CTVAE policy models for
DESTINATION in basketball. The style-consistency for 5 seeds are listed in increasing order. Our algorithm improves style-consistency for both policy models at the cost of slight degradation in imitation quality. In general, CTVAE performs better than RNN in terms of both style-consisten and imitation quality.
Speed
Displacement
Destination
Direction
Curvature 28.31 + 0.95
Clabe
3.96 + 0.33
4.58 + 0.20
1.61 + 0.18
3.19 + 0.25
(a) Basketball labeling functions for experiments in section 6.1
Speed
Torso Height
B-Foot Height
F-Foot Height
C laael
3.24 + 0.83
15.87 + 1.78
17.25 + 0.73
14.75 0.74
(b) Cheetah labeling functions for experiments in section 6.1
Table 14: Mean and standard deviation cross-entropy loss (Clabel x x10-3 over seeds of learned label approximators C.. on test trajectories after n lahel training iterations for experiments in section
6.1 CtA. is only used during training; when computing style-consistency for our quantitative results, we use original labeling functions X.
M3 test error
Basketball
1.47 + 0.59( x 10-T)
Cheetah
1.93 + 0.08 x1002 2)
Table 15: Average mean-squared error of the dynamics model Mpp per timestep per dimension on test trajectories after training for n dynamics iterations
15

---

Under review as a conference paper at ICLR 2020
(a) Label class 0 (closest)
(h) Label class
(c) Label class 2
(d) Label class
Label class 4
Label class 5 (farthest)
Figure 4: Rollouts from our policy calibrated to DESTINATION (net) with 6 classes. The 5 green boundaries divide the court into regions, each corresponding to label class. The label indicates the target region of trajectory's ss final position (+). This policy achieves style-consistency of 0.93, as indicated in Table 8b Note that the initial position (+) is the same as Figures 2 and for comparison, but in general we sample an initial position from the prior p(y) to compute style- consistency.
(a) Speed
(b) Displacement
Destination
(d) Direction
(e) Curvature
Figure 5: Histogram of basketball labeling functions applied on the training set (before applying
thresholds). Basketball trajectories collected from tracking real players in the NBA.
(a) Speed
(b) Torso Height
(c) Back-Foot Height
(d) Front-Foot Height
Figure 6: Histogram of Cheetah labeling functions applied the training (before applying thresholds). Note that SPEED is the most diverse behavior because we pre-trained the policies to achieve various speeds when collecting demonstrations, similar to (Wang et al., 2017). For more diversity with respect to other behaviors, we can also incorporate target behavior as part of the reward when pre-training Cheetah policies.
16


Reviews:
[{"review_id": "Skl6ZZOfKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose learning generative models for long-term sequences that can take a style argument and generate trajectories in that style. The motivating example used is reconstructing expert trajectories from basketball games - trajectories can be sampled based on whether we want fast movement (SPEED), or whether they end close to the basket (DESTINATION).\n\nIt follows a data programming paradigm. We do not directly have style labels, so to get around this, we define labeling functions, which take a trajectory and output some boolean value. (Real valued labels are allowed but are not considered in this work). We assume the desired style is defined by some combination of labels, and that we know this combination (i.e. a fast trajectory to the basket should have the \"speed above threshold c\" label and \"final location close to basket\" label, which we have labeling functions for.)\n\nOnce we have this labeling function, we learn a trajectory VAE with a few loss functions. Standard behavioral cloning loss, and a style consistency loss that encourages labels of the generated trajectory to match labels of the target style. To make the optimization fully differentiable, we approximate non-differentiable labels with a learned labeling function (i.e. learn a classifier and then use classifier probabilities as the label), and learn a model of the environment to allow backprops through the rolled-out dynamics model for the entire trajectory. It's argued that learning a model helps with credit assignment, from an RL perspective credit assignment seems like the wrong word. Nothing about learning the model makes it easier to assign credit, the main gain it gives is making the problem differentiable.\n\nOn the basketball dataset, and a dataset of episode collected from the HalfCheetah MuJoCo environment, they demonstrate better style consistency. I had trouble finding an exact definition of style-consistency here - I assume it's defined as \"given style c, how often does a trajectory sampled from pi(a|s,c) satisfy style c\". It would be good to define this.\n\nI would appreciate discussion on why style consistency classification is better than the mutual information baseline where MI between style labels and trajectory is maximized, it feels like they should be equivalent.\n\nOverall I think this is a reasonable paper. The domains considered are fairly simple, but the idea seems sounds and the results seem good. I am concerned at all the requirements though - the method assumes the dynamics of the environment are learnable, and that we can define labeling functions that cover the space of styles we want, both of which seem like strong requirements.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SJgzQP-2FB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a weak supervision method to obtain labels from functions that are easily programmable, and propose to use this for learning policies that can be \"calibrated\" for specific style. The paper demonstrates some experiments on a basketball environment and a halfcheetah environment, showing that the agent will perform according to corresponding styles.\n\nMy main concern here is the technical novelty of the proposed method: it seems that once we have the labels (which are limited to programmable functions), all we need to do is to learn a policy that conditions on the labels. In this case, we are not concerned with the latent variables whatsoever, therefore it seems that the CTVAE baselines are overkill for the task (learning latent variables that are not actually needed). Maybe more interesting baselines is to see how the two terms in (8) affect self-consistency performance, and not consider any methods that use unsupervised latent variables? \n\nMinor questions:\n\t- The method's name, CTVAE-style is a bit confusing, since the policy does not depend on any latent variable z? At least from how the policy is described pi(\\cdot |y) does not depend on unsupervised latent variables z.\n\t- Table 4, KL and NLL results do not seem to match? I wonder if the basketball kl should be multiplied by 10 and the cheetah ctval-style NLL is a typo?\n        - Is it possible to extend this to continuous labels? This seems technically viable but unclear empirically.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJekg7fLKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to train style-conditional policies. The method has three components, a dynamics model, a labeling procedure and its approximation, and a policy. The model is trained on real trajectories, while the policy is trained on both real and simulated data coming from the model. The policy is trained to both imitate and be sensitive to the style labelling of states. \nSuch a method yields policies that can be executed with styles chosen externally, e.g. by a user.\n\nThis paper proposes a novel method that is an interesting take on imitation learning, but it is hard to judge how relevant this method is, as the paper has several inconsistencies and weaknesses that need to be resolved before it is accepted.\n\nInconsistencies: the reported NLL results do not match the trajectories shown in Figures. Some equations do not seem to reflect what is being optimized.\nWeaknesses: Many quantities that should be reported are not: quality of models, diversity of policies, etc. The source of datasets should be clarified. (see detailed comments)\n\nIt's interesting that this method can leverage these very sparse or poor quality annotations, but it would be helpful to get a sense of how good the annotations provided here are. What's the threshold were annotation's quality is too low to be helpful? In the general case I suspect this threshold to be higher than what is hinted at in this paper, especially given Table 2. What happens in more realistic settings where e.g. users may specify hundreds of different labels?\n\nDetailed comments:\n- how dependent on having a variety of policies is this method? It's not clear how diverse the set of policies coming from the basketball dataset nor the Cheetah policies is, nor how this diversity affects learning. An experiment with explicitly different levels of diversity would strengthen understanding of this method. For Cheetah, it would be easy to report p(y) as a function of the target forward speed, that would give readers a sense of diversity for each label.\n- For something like cheetah, with the labels that you propose being a very simple function of the state space, this somewhat resembles DIAYN [1] but with some grounding. Would it make sense to compare to a similar baseline?\n- (Table 1) Why report (only) the median over 5 seeds? Papers usually report means. Plus, an indication of variance would be nice.\n- It's not clear what the basketball dataset is. Where does it come from? Does it come with a simulator? If not, how do you evaluate style consistency or run step 11-12 of Alg. 2? Do you have a train/valid/test split to choose hyperparameters? (the appendix only suggests a train/test split)\n- Why not cite MuJoCo? [2]\n- For Figure 2 & 3, I suggest lowering the transparency/alpha value of the trajectories, as there is a lot of overlap.\n- Table 1 is somewhat confusing. In (4) and the paragraph thereafter, you define \\mathcal{L}^{style} as an error rate, i.e. when \\lambda(\\tau) \\neq y, but in Table 1 you seem to report instead accuracy? (i.e. when \\lambda(\\tau) = y) If so, then you are reporting percentages, so \\times 10^2 rather than 10^-2.\n- You never report how well C_\\psi is doing, and it's not clear to me why C_\\psi is needed at all. When optimizing (8) are you directly treating (8)'s inner terms as negative rewards which is differentiated wrt \\pi's parameters? If so, you are doing a form of DDPG, but there is a problem: after (4) you mention that L^style is not differentiable, meaning C_\\psi doesn't provide gradient information to \\pi. I assume that you acutally optimize (8) with L^label? Whether that's what you're doing or not, it should be clarified. If you are truly using L^style, then it's not clear why C_\\psi is needed, as there is no differentiability anyways, and simply using \\lambda directly will provide more signal.\n- I'm somewhat perplexed by the values of Table 4. A log-likelihood of -190 represents a probability of 10^-83 (a _negative_ LL of -190 is, on the other hand, impossible by virtue of logs and probabilities, but I assume it is a \"typo\"). What is this the probability of? Entire trajectories? If so it would make more sense to report the _average_ log-likelihood, i.e. per timestep, because at this point in the paper, readers have no sense of how long trajectories are, and thus what these likelihoods represent. (for example, a NLL of 190 could be an average likelihood of 0.5 for 275 steps, or of 0.1 for 83 steps, or of .8 for 850 steps, which are all very different results! According to the appendix Table 5, the basketball trajectories are 25 steps long, which would mean that the imitation objective is not respected at all, exp(-190/25)=.0005, which would mean that pi(a|s) is .0005 on average.)\n- Again Table 4, if the \"-\" is indeed a typo, then CTVAE-style is actually performing better than the baselines, especially for Cheetah (normally one wants negative log-likelihood to be as close to 0 as possible). Same for Table 10, if the NLL column is truly actually log-likelihood, then the \"style+\" objective really degrades imitation quality rather than improves it.\n\n[1] Diversity is All You Need: Learning Skills without a Reward Function, Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, Sergey Levine\n[2] MuJoCo: A physics engine for model-based control, Emanuel Todorov, Tom Erez Yuval Tassa", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers generally reached a consensus that the work is not quite ready for acceptance in its current form. The central concerns were about the potentially limited novelty of the method, and the fact that it was not quite clear how good the annotations needed to be (or how robust the method would be to imperfect annotations). This, combined with an evaluation scenario that is non-standard and requires some guesswork to understand its difficulty, leaves one with the impression that it is not quite clear from the experiments whether the method really works well. I would recommend for the authors to improve the evaluation in the next submission.
Decision: Reject