Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Pretext: Physics simulators are evolving and pose as a good approximation of the real world. Differentiable simulators are evolving and can provide analytic gradients.

Problem formulation: Can analytical gradients form differentiable simulators be leveraged to formulate better policy optimization algorithms?

Approach: Alternate between - optimization in trajectory space by explicitly taking gradients of the value function and - optimization in policy space by performing imitation learning over the improved actions. 

Strengths: Tractable and stable optimization procedure due to the decoupling of the trajectory and policy optimization. Seconds order monotonic efficient method requiring minimal hyper parameter searches. 

Discussions:
1. Inconsistencies between the positioning of the paper and experimental analysis
- Under the lens that the PODS primary contribution should be viewed as a new technique for policy optimization when environment-dynamics is differentiable, viewing SAC and POLO merely as a policy optimization algorithm (independent of training and test dynamics differences) is justified. Expectation under this viewpoint warrants comparison to similar (interleaving supervisedlearning with trajectory optimization) optimization paradigms like POLO(Plan online, learn offline: Efficient learning and exploration via model-based control
K Lowrey, A Rajeswaran, S Kakade, E Todorov), IGOR (Interactive control of diverse complex characters with neural networks. I Mordatch, K Lowrey, G Andrew, Z Popovic), GPS (Learning neural network policies with guided policy search under unknown dynamics S Levine, P Abbeel), etc. 
- Under the premise that PODS' primary contribution should be viewed as a paradigm for sim2real, we need to account for the training and test (true) dynamics being different. While SAC and PPO should train using samples from true dynamics for training and report performance on true dynamics as well. PODS' should train only using samples from train-dynamics but report performance using true dynamics. Sample complexity should be compared only wrt to the sample drawn from the true dynamics (i.e. training sample for model-free methods and sample required for system-ID for PODS and similar methods)

2. No results on commonly accepted benchmarks tasks are presented. I recognize that the authors claimed that these tasks can pose for future benchmark tasks. But in this case, the focus and rigor for an evaluation will be completely different. With both algorithms, as well as the tasks, being new, its hand to establish the strength and credibility of both using one another. My suggestion will be to implement one/ two commonly accepted benchmark tasks and compare against the reported performance on them. 

Suggestions: 
1. Inline with point1 above, an interesting idea will be to perform an experiment in the spirit of sim2real. Where few established benchmarks tasks can be implemented with the differential engine. This is used as training-dynamics for training policies, and the resulting policy is evaluated for performance on equivalent OpenAI-Gym-mujoco as true dynamics.
2. "a" and "pi" are interchangeably called policies. Introducing a more crisp definition separating the two will help with clarity.
2. Missing figure reference in the appendix A2
3. Information on the line search for the experiments reported will be helpful.
review_title: Paper needs better positioning, and appropriate supporting experimentation.

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary
-------------
The paper argues for using differentiable simulators for policy optimization. To avoid back propagation through time, the paper splits the policy optimization problem into two steps: i) find improved action sequence for a set of initial conditions, ii) fit a parametric policy to the set of improved action sequences. First and second order methods are presented and evaluated on a version of payload-on-crane stabilization problem.

Decision
-----------
I vote for rejecting this paper for the following reasons.

1. There is little novelty in this work. The method can be seen as a version of Guided Policy Search (GPS) [1] where the trajectory optimization part is done using analytical gradients.
2. The evaluations don't seem to be reliable. In Fig. 5, PPO and SAC show very bad performance. Given that the task is quite low-dimensional, it is surprising that they work so badly. If that is nevertheless the case, then it would be beneficial to add experiments at least on the Classic Control environments from OpenAI Gym, such that one can compare to other papers.
3. The choice of the step size alpha is questionable. The paper says that unlike PPO and SAC, the proposed method does not require any policy update regularization. However, setting a small value of alpha effectively limits the policy update step. So, the proposed algorithm is still a version of conservative policy update algorithms, such as [1].

References
---------------
[1] Montgomery, W., & Levine, S. (2016). Guided policy search as approximate mirror descent. arXiv preprint arXiv:1607.04614.
review_title: A version of Guided Policy Search with differentiable simulators; limited novelty, suspicious evaluations

final_score: rating: 6: Marginally above acceptance threshold
review_content: ## Summary
The paper presents a class of RL algorithm based on analytic gradients of the objective, 
coming directly from a fully differentiable simulator and differentiable rewards.
The analytic gradients are used to optimize trajectories.
The known states and actions from the optimized trajectories are then used to improve the deterministic policy.
Overall, this is an interesting paper that maximally leverages analytic gradient information
that come directly from a differentiable simulator, rather than a learned differentiable dynamics model.
The methods prove to be highly effective for the given class of problems they are tested on.

## Strengths
- This work provides strong insights into the benefits of differentiable simulators for
  trajectory-guided reinforcement learning.  The paper provides a solid discussion and evaluation of
  the use of first and second-order methods.
- It demonstrates effective and efficient results for a difficult class of problem that has direct application,
  i.e., cable-suspended loads

## Weaknesses
- The foundations of the trajectory-based optimization method exist in prior art [Zimmermann 2019]
- There is perhaps a restricted class of tasks can benefit from the approach:  contact collisions, infinite-horizons,
  and non-differentiable rewards may be off limits?

## Recommendation
The RL community can benefit from this work in several ways, as outlined with the various strengths given above.

## Questions
- Is the method restricted to fixed-length episodic tasks?  Fine if it is, but readers will want to know.
- Do local minima pose a problem, given that the method effectively does no true exploration?
- Can PODS be viewed as a form of fast analytic-derivative trajectory optimization followed by behavior cloning?
- Can hard constraints be handled, e.g., obstacles or joint limits that are to be avoided?
- What are the tradeoffs between working in handle-space vs robot joint space?

## Additional Feedback
What would be the impact of using many fewer rollouts per epoch?

Why not also compare the results for some tasks which are commonly used for RL, e.g., pendulum and acrobot swing-up?

Appendix A.2 only provides very minimal details on the environments.
What are the dimensions and masses of the various pendulum systems?

The following is an early paper (1998) that uses back-prop-through-time to directly optimize a control objective:
"NeuroAnimator:Fast Neural Network Emulation and Control of Physics-Based Models"

The proposed work has connections with Guided Policy Search (Levine and Koltun, IMCL 2013),
which also uses optimized trajectories as training data, only in their case for a stochastic policy
and using differential dynamic programming, instead of the deterministic policy setting used here
together with a Hessian-dependent Newton's method. Explaining these connections could be helpful
to many.

[ Rebuttal / question responses are acknowledged, and also the other reviews.
    I think the algorithm is sufficiently novel, and it does really well on some difficult new problems.
    I see the paper as being about a new policy optimization algorithm leveraging differential dynamics, and not about sim2real.
   However, the biggest limitation is pointed about by both R3 and R4, i.e., per R3: "With both algorithms, as well as the tasks, being new, its hand to establish the strength and credibility of both using one another. "  And so with this in mind, I am changing my score to a 6, i.e., marginally in favor of accept. ] 
review_title: contribution of interest to the RL community

Meta-review: The paper was evaluated by 3 knowledgeable reviewers, where 2 reviewers were leaning for acceptance and one reviewer argued for rejection, rendering the paper a borderline paper. The positive negative points that were raised about the paper during the discussion are summarized below:

Strength:
- The presented policy optimization method provides strong results
- It provided strong insights into the benefits of differentiable simulators for trajectory-guided reinforcement learning
- The direction of differentiable physics simulators is very promising and the provided benchmarks are interesting

Weak points:
- (i) The main contribution of the paper is a novel trajectory optimization method that uses analytical gradients. In a second step, a neural network is fitted to generalize the control from the single trajectories. The given approach is very much related to existing methods such as GPS or IGOR, just that the trajectory optimization is different. A comparison to these methods is needed. For example, how does the algorithm compare to using iLQG as trajectory optimization method ( we could also use analytical gradients for the linierazations used in iLQG)?
- (ii) While the presented tasks are very interesting, there is no benchmark on a more well known task. Hence, it is hard to evaluate the performance in comparison to other algorithms.

The paper defintely has interesting contributions in terms of the new trajectory optimization method and I could live with (ii) as the presented experiments are challenging and interesting, the contribution needs to be better evaluated as comparisions to other trajectory optimization methods are missing. I am sure that the paper will be accepted at another conference with this additional experiments, however, without it the paper is incomplete and I can unfortunately not recommend acceptance.


Decision: Reject