Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper proposes a weak supervision method to obtain labels from functions that are easily programmable, and propose to use this for learning policies that can be "calibrated" for specific style. The paper demonstrates some experiments on a basketball environment and a halfcheetah environment, showing that the agent will perform according to corresponding styles.

My main concern here is the technical novelty of the proposed method: it seems that once we have the labels (which are limited to programmable functions), all we need to do is to learn a policy that conditions on the labels. In this case, we are not concerned with the latent variables whatsoever, therefore it seems that the CTVAE baselines are overkill for the task (learning latent variables that are not actually needed). Maybe more interesting baselines is to see how the two terms in (8) affect self-consistency performance, and not consider any methods that use unsupervised latent variables? 

Minor questions:
	- The method's name, CTVAE-style is a bit confusing, since the policy does not depend on any latent variable z? At least from how the policy is described pi(\cdot |y) does not depend on unsupervised latent variables z.
	- Table 4, KL and NLL results do not seem to match? I wonder if the basketball kl should be multiplied by 10 and the cheetah ctval-style NLL is a typo?
        - Is it possible to extend this to continuous labels? This seems technically viable but unclear empirically.

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes a method to train style-conditional policies. The method has three components, a dynamics model, a labeling procedure and its approximation, and a policy. The model is trained on real trajectories, while the policy is trained on both real and simulated data coming from the model. The policy is trained to both imitate and be sensitive to the style labelling of states. 
Such a method yields policies that can be executed with styles chosen externally, e.g. by a user.

This paper proposes a novel method that is an interesting take on imitation learning, but it is hard to judge how relevant this method is, as the paper has several inconsistencies and weaknesses that need to be resolved before it is accepted.

Inconsistencies: the reported NLL results do not match the trajectories shown in Figures. Some equations do not seem to reflect what is being optimized.
Weaknesses: Many quantities that should be reported are not: quality of models, diversity of policies, etc. The source of datasets should be clarified. (see detailed comments)

It's interesting that this method can leverage these very sparse or poor quality annotations, but it would be helpful to get a sense of how good the annotations provided here are. What's the threshold were annotation's quality is too low to be helpful? In the general case I suspect this threshold to be higher than what is hinted at in this paper, especially given Table 2. What happens in more realistic settings where e.g. users may specify hundreds of different labels?

Detailed comments:
- how dependent on having a variety of policies is this method? It's not clear how diverse the set of policies coming from the basketball dataset nor the Cheetah policies is, nor how this diversity affects learning. An experiment with explicitly different levels of diversity would strengthen understanding of this method. For Cheetah, it would be easy to report p(y) as a function of the target forward speed, that would give readers a sense of diversity for each label.
- For something like cheetah, with the labels that you propose being a very simple function of the state space, this somewhat resembles DIAYN [1] but with some grounding. Would it make sense to compare to a similar baseline?
- (Table 1) Why report (only) the median over 5 seeds? Papers usually report means. Plus, an indication of variance would be nice.
- It's not clear what the basketball dataset is. Where does it come from? Does it come with a simulator? If not, how do you evaluate style consistency or run step 11-12 of Alg. 2? Do you have a train/valid/test split to choose hyperparameters? (the appendix only suggests a train/test split)
- Why not cite MuJoCo? [2]
- For Figure 2 & 3, I suggest lowering the transparency/alpha value of the trajectories, as there is a lot of overlap.
- Table 1 is somewhat confusing. In (4) and the paragraph thereafter, you define \mathcal{L}^{style} as an error rate, i.e. when \lambda(\tau) \neq y, but in Table 1 you seem to report instead accuracy? (i.e. when \lambda(\tau) = y) If so, then you are reporting percentages, so \times 10^2 rather than 10^-2.
- You never report how well C_\psi is doing, and it's not clear to me why C_\psi is needed at all. When optimizing (8) are you directly treating (8)'s inner terms as negative rewards which is differentiated wrt \pi's parameters? If so, you are doing a form of DDPG, but there is a problem: after (4) you mention that L^style is not differentiable, meaning C_\psi doesn't provide gradient information to \pi. I assume that you acutally optimize (8) with L^label? Whether that's what you're doing or not, it should be clarified. If you are truly using L^style, then it's not clear why C_\psi is needed, as there is no differentiability anyways, and simply using \lambda directly will provide more signal.
- I'm somewhat perplexed by the values of Table 4. A log-likelihood of -190 represents a probability of 10^-83 (a _negative_ LL of -190 is, on the other hand, impossible by virtue of logs and probabilities, but I assume it is a "typo"). What is this the probability of? Entire trajectories? If so it would make more sense to report the _average_ log-likelihood, i.e. per timestep, because at this point in the paper, readers have no sense of how long trajectories are, and thus what these likelihoods represent. (for example, a NLL of 190 could be an average likelihood of 0.5 for 275 steps, or of 0.1 for 83 steps, or of .8 for 850 steps, which are all very different results! According to the appendix Table 5, the basketball trajectories are 25 steps long, which would mean that the imitation objective is not respected at all, exp(-190/25)=.0005, which would mean that pi(a|s) is .0005 on average.)
- Again Table 4, if the "-" is indeed a typo, then CTVAE-style is actually performing better than the baselines, especially for Cheetah (normally one wants negative log-likelihood to be as close to 0 as possible). Same for Table 10, if the NLL column is truly actually log-likelihood, then the "style+" objective really degrades imitation quality rather than improves it.

[1] Diversity is All You Need: Learning Skills without a Reward Function, Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, Sergey Levine
[2] MuJoCo: A physics engine for model-based control, Emanuel Todorov, Tom Erez Yuval Tassa
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The authors propose learning generative models for long-term sequences that can take a style argument and generate trajectories in that style. The motivating example used is reconstructing expert trajectories from basketball games - trajectories can be sampled based on whether we want fast movement (SPEED), or whether they end close to the basket (DESTINATION).

It follows a data programming paradigm. We do not directly have style labels, so to get around this, we define labeling functions, which take a trajectory and output some boolean value. (Real valued labels are allowed but are not considered in this work). We assume the desired style is defined by some combination of labels, and that we know this combination (i.e. a fast trajectory to the basket should have the "speed above threshold c" label and "final location close to basket" label, which we have labeling functions for.)

Once we have this labeling function, we learn a trajectory VAE with a few loss functions. Standard behavioral cloning loss, and a style consistency loss that encourages labels of the generated trajectory to match labels of the target style. To make the optimization fully differentiable, we approximate non-differentiable labels with a learned labeling function (i.e. learn a classifier and then use classifier probabilities as the label), and learn a model of the environment to allow backprops through the rolled-out dynamics model for the entire trajectory. It's argued that learning a model helps with credit assignment, from an RL perspective credit assignment seems like the wrong word. Nothing about learning the model makes it easier to assign credit, the main gain it gives is making the problem differentiable.

On the basketball dataset, and a dataset of episode collected from the HalfCheetah MuJoCo environment, they demonstrate better style consistency. I had trouble finding an exact definition of style-consistency here - I assume it's defined as "given style c, how often does a trajectory sampled from pi(a|s,c) satisfy style c". It would be good to define this.

I would appreciate discussion on why style consistency classification is better than the mutual information baseline where MI between style labels and trajectory is maximized, it feels like they should be equivalent.

Overall I think this is a reasonable paper. The domains considered are fairly simple, but the idea seems sounds and the results seem good. I am concerned at all the requirements though - the method assumes the dynamics of the environment are learnable, and that we can define labeling functions that cover the space of styles we want, both of which seem like strong requirements.
review_title: Official Blind Review #3

Meta-review: The reviewers generally reached a consensus that the work is not quite ready for acceptance in its current form. The central concerns were about the potentially limited novelty of the method, and the fact that it was not quite clear how good the annotations needed to be (or how robust the method would be to imperfect annotations). This, combined with an evaluation scenario that is non-standard and requires some guesswork to understand its difficulty, leaves one with the impression that it is not quite clear from the experiments whether the method really works well. I would recommend for the authors to improve the evaluation in the next submission.
Decision: Reject