Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: The problem of semi-supervised RL is a good one to formalize, although the approach taken by the paper seems to add perhaps excessive layers of complexity to the simplest possible approach (i.e., linearly solvable MDPs, entropic policies, ...). 

First, the terminology of "unlabeled MDP" seems to grate on the ear. A reward function is not  analogous to a "label"  in supervised learning. It does not tell the agent what action to do, since RL is not associative learning. It might hint as to whether an action taken is appropriate or not, but even that is often misleading, since whether the reward is positive, negative or zero might not reveal anything about whether the current action is desirable or not. The reward function is simply a highly noisy indicator of the true value function, and the correct "label" for the state is the return. The reward is just the first term in the return. 

It might be helpful to develop an SSRL model based on this idea, where "unlabeled MDPs" are simply those where the returns are identically zero (always), and "labeled" MDPs are those where the return is the exact value (this corresponds more to the regression setting in the paper). By expanding the return and progressively neutralizing successive terms, one is lead from a fully supervised MDP to a weakly supervised MDP (or even an MDP without  a reward function, which is technically not an MDP at all, but a controlled Markov process). 

The connection to bandit problems might be helpful to discuss as well. What is the equivalent "unlabeled" bandit problem, and how does the present approach relate to the huge literature on bandits (which are special cases of MDPs)?
review_title: Simpler framework for SSRL 

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper proposes to study the problem of semi-supervised RL where one has to distinguish between labelled MDPs that provide rewards, and unlabelled MDPs that are not associated with any reward signal. The underlying is very simple since it aims at simultaneously learning a policy based on the REINFORCE+entropy regularization technique, and also a model of the reward that will be used (as in inverse reinforcement learning) as a feedback over unlabelled MDPs. The experiments are made on different continous domains and show interesting results

The paper is well written, and easy to understand. It is based on a simple but efficient idea of simultaneously learning the policy and a model of the reward and the resulting algorithm exhibit interesting properties. The proposed idea is quite obvious, but the authors are the first ones to propose to test such a model. The experiments could be made stronger by mixing continuous and discrete problems but are convincing. 
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: In supervised learning, a significant advance occurred when the framework of semi-supervised learning was  adopted, which used the weaker approach of unsupervised learning to infer some property, such as a distance measure or a smoothness regularizer, which could then be used with a small number of labeled examples. The approach rested on the assumption of smoothness on the manifold, typically. 

This paper attempts to stretch this analogy to reinforcement learning, although the analogy is somewhat incoherent. Labels are not equivalent to reward functions, and positive or negative rewards do not mean the same as positive and negative labels. Still, the paper makes a worthwhile attempt to explore this notion of semi-supervised RL, which is clearly an important area that deserves more attention. The authors use the term "labeled MDP" to mean the typical MDP framework where the reward function is unknown. They use the confusing term "unlabeled MDP" to mean the situation where the reward is unknown, which is technically not an MDP (but a controlled Markov process). 

In the classical RL transfer learning setup, the agent is attempting to transfer learning from a source "labeled" MDP to a target "labeled" MDP (where both reward functions are known, but the learned policy is known only in the source MDP). In the semi-supervised RL setting, the target is an "unlabeled" CMP, and the source is both a "labeled" MDP and an "unlabeled" CMP. The basic approach is to use inverse RL to infer the unknown "labels" and then attempt to construct transfer. A further restriction is made to linearly solvable MDPs for technical reasons. Experiments are reported using three relatively complex domains using the Mujoco physics simulator. 

The work is interesting, but in the opinion of this reviewer, the work fails to provide a simple sufficiently general notion of semi-supervised RL that will be of sufficiently wide interest to the RL community. That remains to be done by a future paper, but in the interim, the work here is sufficiently interesting and the problem is certainly a worthwhile one to study. 
review_title: An approach to semi supervised RL using inverse RL 

final_score: rating: null
review_content: Dear authors, first of all, I find the paper really interesting. Here are some preliminary questions to help me to better understand what you propose:

Concerning the function R_\psi(t) which aims at evaluating the reward over a trajectory, could you please provide details about the shape of this function since it takes as an input a complete trajectory ?

Concerning the evaluation, it is not clear how it is done (evaluation made only on the unlabeled MDPs, or both the labeled and unlabeled ones ? )

It seems that the experimental tasks have a particular structure. For example, in the jumping experiment (the agent does not have information concerning the size of the wall) it seems that being able to solve the unsupervised MDPs (wall with a heigh of 0.5) is a good solution for the labeled MDPs (with a smaller wall). Could you please comment on this since it seems (but maybe I am wrong) to bias a little bit the interpretation of the experimental results ? Why not providing the height of the wall as an observation for example and to train with multiple heights ? 

The algorithm is used with MDGPS which supposes the reward to be differentiable. Do you have results on "more classic" tasks with classical rewards ? (and discrete actions spaces for example) ? 

 Thank you
review_title: Pre-review question

final_score: rating: 7: Good paper, accept
review_content: This paper formalizes the problem setting of having only a subset of available MDPs for which one has access to a reward. The authors name this setting "semi-supervised reinforcement learning" (SSRL), as a reference to semi-supervised learning (where one only has access to labels for a subset of the dataset). They provide an approach for solving SSRL named semi-supervised skill generalization (S3G), which builds on the framework of maximum entropy control. The whole approach is straightforward and amounts to an EM algorithm with partial labels (: they alternate iteratively between estimating a reward function (parametrized) and fitting a control policy using this reward function. They provide experiments on 4 tasks (obstacle, 2-link reacher, 2-link reacher with vision, half-cheetah) in MuJoCo.

The paper is well-written, and is overall clear. The appendix provides some more context, I think a few implementation details are missing to be able to fully reproduce the experiments from the paper, but they will provide the code.

The link to inverse reinforcement learning seems to be done correctly. However, there is no reference to off-policy policy learning, and, for instance, it seems to me that the \tau \in D_{samp} term of equation (3) could benefit from variance reduction as in e.g. TB(\lambda) [Precup et al. 2000] or Retrace(\lambda) [Munos et al. 2016].

The experimental section is convincing, but I would appreciate a precision (and small discussion) of this sentence "To extensively test the generalization capabilities of the policies learned with each method, we measure performance on a wide range of settings that is a superset of the unlabeled and labeled MDPs" with numbers for the different scenarios (or the replacement of superset by "union" if this is the case). It may explain better the poor results of "oracle" on "obstacle" and "2-link reacher", and reinforce* the further sentences "In the obstacle task, the true reward function is not sufficiently shaped for learning in the unlabeled MDPs. Hence, the reward regression and oracle methods perform poorly".

Correction on page 4: "5-tuple M_i = (S, A, T, R)" is a 4-tuple.

Overall, I think that this is a good and sound paper. I am personally unsure as to if all the parallels and/or references to previous work are complete, thus my confidence score of 3.

(* pun intended)
review_title: Review

Meta-review: This paper provides an interesting framework for handling semi-supervised RL problems, settings were one can interact with many MDPs drawn from some class, but where only a few have observable rewards; the agent then uses a policy derived from the labeled MDPs to estimate a reward function for the unlabeled MDPs. The approach is straightforward, and one reviewer raises the reasonable concern that this seems to be a fairly narrow definition and approach to the universe of things that could be thought of as semi-supervised reinforcement learning (elements like Laplacian value functions, e.g. proto value functions, for instance, present a very different take on could what also be considered semi-supervised RL).
 
 Overall, however, the main benefit of this paper is that the overall idea here is quite compelling. Even if it's a narrow view on semisupervised RL, it nonetheless is clearly thinking more broadly about skill and data efficiency than what is common in many current RL papers. Given this impressive scope, plus good performance (if only on a relatively small set of benchmarks), it seems like this paper is certainly above bar for acceptance.
 
 Pros:
 + Nice introduction of a new/modified semisupervised reinforce setting
 + Results on benchmarks look compelling (if still fairly small scale)
 
 Cons:
 - Rather limited view of the space of all possible semisupervised Rl
 - Results on hardest task (half-cheetah) aren't _that_ much better than much simpler approaches
Decision: Accept (Poster)