Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The work investigates convergence guarantees of gradient-type policies for reinforcement learning and continuous control
problems, both in deterministic and randomized case, whiling coping with non-convexity of the objective. I found that the paper suffers many shortcomings that must be addressed:

1) The writing and organization is quite cumbersome and should be improved.
2) The authors state in the abstract (and elsewhere): "... showing that (model free) policy gradient methods globally converge to the optimal solution ...". This is misleading and NOT true. The authors show the convergence of the objective but not of the iterates sequence. This should be rephrased elsewhere.
3) An important literature on convergence of descent-type methods for semialgebraic objectives is available but not discussed.
review_title: GLOBAL CONVERGENCE OF POLICY GRADIENT METHODS FOR LINEARIZED CONTROL PROBLEMS

final_score: rating: 5: Marginally below acceptance threshold
review_content: I find this paper not suitable for ICLR. All the results are more or less direct applications of existing optimization techniques, and not provide fundamental new understandings of the learning REPRESENTATION.
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper studies the global convergence for policy gradient methods for linear control problems. 
(1) The topic of this paper seems to have minimal connection with ICRL. It might be more appropriate for this paper to be reviewed at a control/optimization conference, so that all the technical analysis can be evaluated carefully. 

(2) I am not convinced if the main results are novel. The convergence of policy gradient does not rely on the convexity of the loss function, which is known in the community of control and dynamic programming. The convergence of policy gradient is related to the convergence of actor-critic, which is essentially a form of policy iteration. I am not sure if it is a good idea to examine the convergence purely from an optimization perspective.

(3) The main results of this paper seem technical sound. However, the results seem a bit limited because it does not apply to neural-network function approximator. It does not apply to the more general control problem rather than quadratic cost function, which is quite restricted. I might have missed something here. I strongly suggest that these results be submitted to a more suitable venue.


review_title: Global Convergence of Policy Gradient Methods for Linearized Control Problems

Meta-review: The paper studies the global convergence for policy gradient methods for linear control problems.  Multiple reviewers point out strong concerns about the novelty of the results.
Decision: Reject