Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: I was wondering if you think the results might depend significantly on the objective and reward functions? The task at hand is to follow some reference motion (Section 3.1) and it seems that controllers with an action space similar to the target space might have steeper initial learning curves. The reward function (Section 3.4) only takes into account position and velocity, which can be directly influenced by the PD and velocity controllers.

Assume that the objective is to implement an impedance controller instead of a trajectory controller and we evaluate the torque characteristics. Do you think the results will transfer to this or similar cases?
review_title: Influence of objective and reward functions?

final_score: rating: 6: Marginally above acceptance threshold
review_content: Paper studies deep reinforcement learning paradigm for controlling high dimensional characters. Experiments compare the effect different control parameterizations (torques, muscle-activations, PD control with target joint positions and target joint velocities) have on the performance of reinforcement learning and optimized control policies. Evaluated are different planer gate cycle trajectories. It is illustrated that more abstract parameterizations are in fact better and result in more robust and higher quality policies. 

> Significance & Originality:

The explored parameterizations are relatively standard in humanoid control. The real novelty is systematic evaluation of the various parameterizations. I think this type of study is important and insightful. However, the findings are very specific to the problem and specific tested architecture. Its not clear that findings will transferable to other networks on other control problems/domains. As such for the ICLR community, this may have limited breadth and perhaps would have broader appeal in robotics / graphics community. 

> Clarity:

The paper is well written and is pretty easy to understand for someone who has some background with constrained multi-body simulation and control. 

> Experiments:

Experimental validation is lacking somewhat in my opinion. Given that this is a fundamentally experimental paper, I would have liked to see more analysis of sensitivity to various parameters and analysis of variance of performance when policy is optimized multiple times.
review_title: Review

final_score: rating: null
review_content: I was wondering if you have experimented with sensitivity of the learning with respect to the manually specified parameters.  Particularly with respect to covariance matrix Sigma, mentioned in Section 2. 

Also, in Section 2 a [0,1] discount factor is introduced. However, later, in Results, it is mentioned that "No discounting is applied when calculating the NCR". Is discounting applied elsewhere? If so, how is it set?

Thank you. 


review_title: Sensitivity to the manually specified parameters

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper is straightforward, easy to read, and has clear results. 

Since all these parameterisations end up outputting torques, it seems like there shouldn't be much difference between them. There is a known function that convert from one representation to another (or at least to torques). Is it not possible that the only reason proportional control is a little better is that the tracking cost is a function of positions?

Would we get the same result if there was no reference-pose cost, only a locomotion cost?

Would we get the same result if the task was to spin a top? My guess is no. 

This work is interesting, but not likely to generalise to other scenarios, and in that sense is rather limited.

The video is nice.

review_title: Nice paper comparing control parameterisations in simulated locomotion domains.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper addresses a question that is often overlooked in reinforcement learning or locomotion experiment.

My biggest point of critique is that it's difficult to draw conclusions or reason beyond the results of the experiments. 
The authors only consider a single neural network architecture and a single reward function. For example, is the torque controller limited by the policy network?  
My suggestion is to vary the number of neurons or show that the same results hold for a different state representation (e.g. trained on pixel data). In the paper's current form, the term "DeepRL" seems arbitrary.

On the positive side, the paper is well-structured and easy to read. The experiments are sound, clear and easy to interpret. 
It's definitely an interesting line of work and beyond the extension to 3D, I would argue that considering more realistic physical constraints (e.g. actuator constraints, communication delays etc. on real robots) could greatly improve the impact of this work.

review_title: Interesting topic, sound experiments, hard to draw conclusions.

Meta-review: After reading the paper and the reviews, I believe that the paper presents a solid contribution and a detailed empirical exploration of the choice of action space representations for continuous control trajectory tracking tasks, but has limited relevance to the ICLR audience in its present form.
 
 The conclusion that PD controllers with learned gains provide improved learning speed and sometimes better final results than the "default" joint torque representation is intriguing and has some implications for future work on trajectory tracking for simulated articulated rigid body characters, but it's unclear from the current results what conclusion there is to take away for general algorithm or model design. Since the only evaluation is on trajectory tracking, it's also not even clear (as pointed out by other reviewers) to what degree these results will generalize to other tasks. In fact, a contrarian view might be that PD tracking is specifically a good fit for trajectory tracking with a known reference trajectory, but might be a poor fit for accomplishing a particular task, where more open-loop behaviors might be more optimal. The passive compliance of MTUs is also shown to often not be beneficial, but it's again not clear whether this might in fact be an artifact of the trajectory tracking task.
 
 But perhaps more problematically, the primary conclusions of the paper are of limited relevance when it comes to design of algorithms or models (or at least, the authors don't discuss this), but are more relevant for practitioners interested in applying deep RL for learning controllers for articulated rigid body systems such as robots or virtual characters. As such, it's not clear what fraction of the ICLR audience will find the paper relevant to their interests.
 
 I would strongly encourage the authors to continue their research: there is a lot to like about the present paper, including an elegant reinforcement learning methodology, rigorous empirical results for the specific case of trajectory following, and some very nice videos and examples.
Decision: Reject