Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper extends softmax consistency by adding in a relative entropy term to the entropy regularization and applying trust region policy optimization instead of gradient descent.  I am not an expert in this area. It is hard to judge the significance of this extension.

The paper largely follows the work of Nachum et al 2017. The differences (i.e., the claimed novelty) from that work are the relative entropy and trust region method for training. However, the relative entropy term added seems like a marginal modification. Authors claimed that it satisfies the multi-step path consistency but the derivation is missing.

I am a bit confused about the way trust region method is used in the paper. Initially,  problem is written as a constrained optimization problem (12). It is then converted into a penalty form for softmax consistency. Finally, the Lagrange parameter is estimated from the trust region method. In addition, how do you get the Lagrange parameter from epsilon?

The pseudo code of the algorithm is missing. It would be much clearer if a detailed description of the algorithmic procedure is given.

How is the performance of Trust-PCL compared to PCL? 
review_title: It might be useful but looks like an incremental work. The technical presentation is not quite clear.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Clarity 
The paper is well-written and clear. 

Originality
The paper proposes a path consistency learning method with a new combination of entropy regularization and relative entropy. The paper leverages a novel method in determining the coefficient of relative entropy. 

Significance
- Trust-PCL achieves overall competitive with state-of-the-art external implementations.
- Trust-PCL (off-policy) significantly outperform TRPO in terms of data efficiency and final performance. 
- Even though the paper claims Trust-PCL (on-policy) is close to TRPO, the initial performance of TRPO looks better in HalfCheetah, Hopper, Walker2d and Ant. 
- Some ablation studies (e.g., on entropy regularization and relative entropy) and sensitivity analysis on parameters (e.g. \alpha and update frequency on \phi) would be helpful. 

Pros:
- The paper is well-written and clear. 
- Competitive with state-of-the-art external implementations
- Significant empirical advantage over TRPO.
-  Open source codes.

Cons:
- No ablation studies. 

review_title: Good paper

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a policy gradient method that employs entropy regularization and entropy constraint at the same time. The entropy regularization on action probability is to encourage the exploration of the policy, while the entropy constraint is to stabilize the gradient.

The major weakness of this paper is the unclear presentation. For example, the algorithm is never fully described, though a handful variants are discussed. How the off-policy version is implemented is missing.

In experiments, why the off-policy version of TRPO is not compared. Comparing the on-policy results, PCL does not show a significant advantage over TRPO. Moreover, the curves of TRPO is so unstable, which is a bit uncommon. 

What is the exploration strategy in the experiments? I guess it was softmax probability. However, in many cases, softmax does not perform a good exploration, even if the entropy regularization is added.

Another issue is the discussion of the entropy regularization in the objective function. This regularization, while helping exploration, do changes the original objective. When a policy is required to pass through a very narrow tunnel of states, the regularization that forces a wide action distribution could not have a good performance. Thus it would be more interesting to see experiments on more complex benchmark problems like humanoids.
review_title: Technique is not clear. Contribution is more like incremental. Experiments are insufficient.

Meta-review: This paper adapts (Nachum et al 2017) to continuous control via TRPO.   The work is incremental (not in the dirty sense of the word popular amongst researchers, but rather in the sense of "building atop a closely related work"), nontrivial,  and shows empirical promise.    The reviewers would like more exploration of the sensitivity of the hyper-parameters.
Decision: Accept (Poster)