Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: The paper proposes learning physical derivatives, the derivative of the trajectory distribution with respect to policy parameters. The proposed method estimates changes in  trajectories at a particular theta by using finite differences,
then fitting Gaussian Processes per timestep to generalize to new dtheta's. The paper then proposes techniques to robustify the process against noise in the system.
To deal with temporal noise, where trajectories are approximately equal up to a time shift, they simply
estimate the optimal shift and use the shifted version to estimate. To address more complicated noise, they assume sensitivity of trajectories to noise is small relative 
to sensitivity to the parameters, and discretize the state space at a level such that trajectories that 
differ primarily due to inherent noise look the same at the discretized level, while perturbed policy 
parameters still lead to different trajectories. They then use the discretized trajectories to estimate 
the finite differences.

Experiments illustrate how the learned predictions compared to actual resulting perturbations and 
illustrate resulting trajectories on certain toy domains and a physical robotic finger. All experiments
are done with very low dimensional state and policy spaces (1-3 dimensions each). 

Without much more extensive experimental validation, this paper should be rejected. While I am not aware 
of any prior work on learning physical derivatives, the actual methods used are not novel in of themselves
beyond being applied towards learning derivatives with respect to the policy. As such, the method should be
of practical interest in order to be accepted. With limited experiments on a single very low dimensional 
domain and no comparisons against any alternative methods, there is little evidence demonstrating the actual 
effectiveness of the proposed method, especially on more complex domains and for downstream tasks.

Suggested Experiments:

- Stability in gradient estimation
It seems like it could require huge amounts of samples to be able to estimate gradients at parameters 
where for which the system is not very stable, as states in at later timesteps can easily change in 
hard to predict ways as the dynamics are propogated through time. This should be an especially big issue 
if we do not already have good stable controllers close to the desired solution and needed to actually 
conduct exploration in parameter space to solve a taask. I would appreciate more extensive evaluation 
across multiple different (simulated) domains and assessing the effectiveness of gradient estimation 
along random parameters.

- Dimensionality of policy parameters and state spaces
The current experiments only involve very small parameter spaces. It would be important to see how well
using finite differences and GP regression scales with a higher dimensional search space, which can be 
demonstrated on varying dimensionalities of an LQR system for example. It would also be important to see
how gradient estimation scales with high dimensional state spaces even with a small parameter space (like
in the PD controller experiment in the paper). 

- Direct Comparison against learning dynamics models
Using the same data, compare (with same metrics as in table 1) physical derivatives estimated with the 
proposed approach against learning GP dynamics models and rolling out the perturbed policy with the learned
model. Without a direct comparison against learning dynamics models and understanding what situations 
learning physical derivatives provides better estimates, it is unclear when or why one would prefer to learn 
physical derivatives in this way compared to a model based approach. 

- Quantitative results measuring costs of learned controllers
Despite the name of the paper and a description of how to compute a policy gradient via physical derivatives, 
there are no experiments involving such policy gradient updates as far as I can tell. While one advantage of the 
method (as well as model based approaches) is the ability to learn in unsupervised manner, it would be extremely 
helpful to validate how well the physical derivatives are estimated in terms of how useful they are for a downstream 
task, such as optimizing a controller for a cost function. Right now, experimental results lack any comparisons
to other methods or any other way to assess the effectiveness of estimating physical derivatives. 
A comparison against regular RL policy gradient methods (or other model free algorithms) and model based RL 
would give an idea as to whether the physical derivatives learned are actually useful.

Other questions and comments:
	- Table 1: is this evaluating the accuracy of the physical derivatives on the shaking data that it was
	  used for learning, or on a validation set? If on a validation set, would the validation perturbations be
	  drawn from the same distribution as the training set?
	- The zero shot planning experiment in section 4.4 seems very contrived. It does not seem like a useful task
	  to adjust the parameters of the PD controller in order to reach a state that isn't the target. The figures 
	  illustrating trajectories are also not very convincing and unclear. Two points are labelled source state and
	  target state, but it is not clear which is the intermediate state it is supposed to reach. In any case, most 
	  of the trajectories seem to vastly overshoot the target? final state, and it is hard to assess how close the
	  trajectories end up being to the intended states from a 2d representation of a 3d space. Quantitative results
	  would perhaps have been more useful in illustrating the effectiveness of using physical derivatives.
	- What is the purpose of figure 4? It does not appear to be referenced in the text and it is not clear what is
	  being shown.
	
Other notes not part of decision:
	Paper exceeds the 8 page recommended length
	Lots of small typos in the text

	
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper presents a method for control by estimating the gradient of trajectories w.r.t. the policy parameters by fitting a GP to a set of noisy trajectories executing the same controller. This is opposed to the majority of current RL methods that either learn a forward model or learn a policy. They argue that learning this gradient is a middle step between model-based and model-free RL. The method is shown to estimate gradients on simple policies (linear and nonlinear open-loop controllers, and a linear PD controller) for a free-space reaching robot, and update a controller to add a trajectory constraint to pass an intermediate state.

The paper does show that they can learn these derivatives on controllers from data, which is a cool proof of concept. The method to estimate gradients by “shaking” in a probabilistic way by fitting a GP to noisy trajectories is clever and interesting. But there are a few reasons why I believe this work is not ready for publication.

The paper only considers free-space reaching as a task, which is not a difficult problem as it does not have contacts. The policies considered are also very simple: an affine open-loop controller (U = Wt + B with 6 parameters), a simple nonlinear open-loop controller (U = Asin(wt) with 2 parameters) and a PD controller with 2 parameters. The motivation is not too convincing without showing some results on hard tasks: model-based RL methods work great in this setting, and are very likely to outperform the method proposed in the paper. The motivation for the proposed method avoids explicit model learning which is a similar motivation as model-free methods, so the paper should at least show that it works as a proof of concept in settings where model-free learning has some advantages, eg. environments with contacts. The paper should probably also compare to existing methods in those settings, although I understand that it might not outperform existing methods.

The results in section 4.4 which is the result of using the model to plan really shows that using the learned model to update the policy is probably not straightforward. The parameters of the PD controller that go from x_0 to x* are updated to pass a waypoint x*_t using the learned model. But in practice what this is basically doing is changing k_p to introduce a large, possibly inefficient deviation in the path from x_0 to x* that hits x*_t at time t. Directly planning for a path between x_0 to x*_t and then x*_t to x* would probably give a much cleaner path.

At a high level, the proposed method is likely to be difficult to apply on real problems because estimating the gradient of T w.r.t. pi is probably just much noisier than estimating the forward model directly, which is already a significant challenge. Perhaps one useful experiment is to somehow explicitly show how these two methods compare (eg. measure the variance of trajectory predictions of this method vs rolling out a learned forward model repeatedly).

Comments:

Equation 11 and 12 do not make sense/do not use standard notation. I suggest defining n (as a signal or a value, it is not clear at the moment) and defining a new output signal y_t instead of the x_t <- … notation. In particular, the way equation 11 is written seems to say the output x_t is a value-shifted version of the input x_t, NOT a time-shifted one.

The preliminaries section 1.1 does not discuss environment dynamics. This is significant because the paper seems to assume deterministic dynamics but this is never explicitly stated.

Voxelization as a solution to spatial noise is a bit surprising because discretizing the space throws away local gradient information, which seems valuable to the method. It would be good to understand the effect of this design decision better with an ablation.

Minor comments:
Page 9: 
- constrain -> constraint
- Assuem -> Assume
- such controller -> such a controller
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: *Summary of paper*
This paper investigates the use of random perturbations applied to a robotic policy to learn a local gradient useful for policy optimization. The method aims to learn a policy directly on a real physical robotic system, bypassing both simulation models and model-free RL. Training pairs are gathered by perturbations of a starting policy, and the "gradient" is captured in a probabilistic model learned from the training data. The paper includes experiments on a custom 3-DOF robotic platform.

*Decision*
I vote for rejecting this paper. While the idea is interesting, the paper lacks precision in key areas and the method is not placed in context among related work. Further, it fails to communicate key ideas (particularly in the experiments) to a non-robotics reader. Without sufficient clarity and background, it is not suited to a general machine learning conference.

- Lemma 3, which attempts to justify the use of voxelization, and its proof are both imprecise and inadequate. To improve precision, please define "error causes by voxelization" in mathematical terms, e.g. ||c_i - x_i||. Also, while the statement of the lemma un-intuitively implies that larger voxels introduce smaller errors, the proof seems to say that larger errors will result for smaller gradients if larger voxels are used.
- Related work: How does this work relate to random search/evolutionary computation? How does it compare to performing those methods or a model-free RL method directly on the robot? How does it compare to learning using an inaccurate model for robot dynamics? Presumably there are numerous methods that have been tried in this area, so further context is needed.
- The evaluation is unclear, at least to a non-expert in robotics. A lack of quantitative evaluation further exacerbates this issue: nearly all experiments, even those with associated plots, are characterized qualitatively and without reference the performance of related methods.

- In addition to addressing the limitations above, I would encourage the authors to consider the use of experiments in simulation to thoroughly and quantitatively investigate the convergence/bias/variance of the gradient model w.r.t. #DoF of the robot, length of the trajectory, voxelization, # sampled trajectories, perturbation sampling method, and robot reliability/reproducibility

*Additional feedback*
- spelling errors throughout; please check thoroughly
- the captions/labels/etc. in most figures is far too small to read in a printed copy of the paper
- What is the intuition for the "empirical distribution p_e(T|\pi) = ..." on page 2? Is it counting the exact matches between the trajectory T and the M observed trajectories? (This may be more clear in the context of voxelization introduced later.)
- Figure 3: what are the units for \gamma? what is the time step?
- many of the figures are out of order w.r.t. their introduction in the text
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: This paper addresses a very good question - can we do better in terms of model learning, so that we can find the much sought after middle ground between model free and model based RL. In particular, the authors ask, can we find a way to learn a model that is reward/task independent, so that a new task can be equally well handled. This is timely and the general thrust of the thinking, in terms of learning from perturbation around trajectories, is good but I am not sure the proposed methods are sufficiently well developed to merit publication. I am also concerned that the authors do not consider numerous issues with the setup that are fairly well understood as issues for system identification.

The main idea, as laid out in §1.1, is to observe that the parameter update depends mainly on the way a small perturbation in parameters is reflected as a variation in the optimal trajectory (by asking for the probability of a trajectory, this variation becomes the probability of a nearby trajectory). The authors then approach the approximation of this in terms of a discrete finite differences estimate. There are some extensions, such as using a local GP model instead of a local linear model and consideration of ways in which the system might not be exactly repeatable given initial states. These are all proper questions but there are many more important unanswered ones:

1. Starting with where the model setup begins, it is not clear why a complex nonlinear dynamical system, i.e., the typical multi-jointed robot taken as a dynamical system (so, not just kinematics and quasi-static movements), can be sufficiently well approximated using a discretised finite point set that is used at the start of §2 - how does one find the correct T, the correct step size, how does one change these for the local nature of the dynamics (some places might be smoother than others, in phase space), etc.? Even more importantly, are we assuming we know the proper state space ahead of time so that there is no history dependence due to unobserved variables?

2. As such, the authors are proposing to perform closed-loop system identification in a completely data-driven manner. It is well known that this is hard because in the absence of suitable excitation, not all necessary modes in the dynamics will be observed. The only controlled example considered, in §4.3, and subsequent discussion about 'zero-shot' generalisation is getting at this. However, neither at the conceptual level nor in terms of the detailed experiment do I see a good account of what allows this approach to learn all aspects of the dynamics of the system from just small perturbations around a closed loop trajectory.

3. In light of all this, I find the evaluation really weak. Some experiments I would have liked to have seen include - (i) a control experiment based on a standard multi-link arm to show how bad the issue of model mis-match is for the task being considered (I suspect, not much), (ii) experiments with local linearizations, and perhaps piecewise local linearizations, to show how much innovation is needed or is being achieved by the proposed advances, (iii) for us to be talking about 'zero shot' generalisation and the like, more sophisticated tasks beyond merely changing the reaching point (as I say before, it is not even clear that a good PID controller with a roughly plausible linearization is not sufficient to achieve similar effects, and certainly there is a plethora of more sophisticated baselines one could have drawn upon).

4. Some of the discussion comes across as a bit naive, e.g., we have a lemma 3 whose proof is simply a geometric argument about cubes without sufficient consideration of properties of dynamics. I don't doubt the result but in the way it is presented here, it seems shoddy.

Also, some smaller questions not properly explained:
a. How do you know which kernels for good for the GP in equations 9-10?
b.  Why should we expect the correlation procedure in §3.0.1 to always work without aliasing and what is the way to get at the suitable domain?



review_title: Official Blind Review #2

Meta-review: While the reviewers generally appreciated the idea behind the method in the paper, there was considerable concern about the experimental evaluation, which did not provide a convincing demonstration that the method works in interesting and relevant problem settings, and did not compare adequately to alternative approach. As such, I believe this paper is not quite ready for publication in its current form.
Decision: Reject