Learning by shaking: Computing policy gradients by physical forward-propagation
Authors: Arash Mehrjou
Ashkan Soleymani
Stefan Bauer
Bernhard Schölkopf
Venue:  (2020)
Keywords: Reinforcement Learning
Control Theory

Abstract: Model-free and model-based reinforcement learning are two ends of a spectrum. Learning a good policy without a dynamic model can be prohibitively expensive. Learning the dynamic model of a system can reduce the cost of learning the policy, but it can also introduce bias if it is not accurate. We propose a middle ground where instead of the transition model, the sensitivity of the trajectories with respect to the perturbation (shaking) of the parameters is learned. This allows us to predict the local behavior of the physical system around a set of nominal policies without knowing the actual model. We assay our method on a custom-built physical robot in extensive experiments and show the feasibility of the approach in practice. We investigate potential challenges when applying our method to physical systems and propose solutions to each of them.

Under review as a conference paper at ICLR 2020
LEARNING BY SHAKING: COMPUTING POLICY GRADI- ENTS BY PHYSICAL DORWARD-PROPAGATION
Anonymous authors Paper under double-blind review
ABSTRACT
Model-free and model-based reinforcement learning are two ends of spectrum. Learning a good policy without a dynamic model can be prohibitively expensive. Learning the dynamic model of system can reduce the cost of learning the pol- icy, but it can also introduce bias it is not accurate. We propose a middle ground where instead of the transition model, the sensitivity of the trajectories with re- spect to the perturbation (shaking) of the parameters is learned. This allows us predict the local behavior of the physical system around set of nominal poli- cies without knowing the actual model. We assay our method on a custom-built physical robot in extensive experiments and show the feasibility of the approach in practice. We investigate potential challenges when applying our method to physical systems and propose solutions each of them.
Figure I: Physical finger platform in action with different policies.
INTRODUCTION
Traditional reinforcement learning crucially relies (Sution & Barto 2018). However, re- ward binds the agent certain task for which the reward represents success. Aligned with the recent surge of interest in unsupervised methods in reinforcement learning (Baranes & Oudeyer, 2013 Bellemare et al. 2016 Gregor al. 20166: Hausman al., 20188] Houthooft et 2016) and previously proposed ideas (Schmidhuber 1991a 2010), we argue that there exist properties of dy- namical system which are not tied to any particular task, yet highly useful, and their knowledge can help solve other tasks more efficiently. This work focuses on the sensitivity of the produced trajecto- ries of the system with respect to the policy so called Physical Derivatives. The term physical comes from the fact that uses the physics of the system rather than any idealized model. We learn map from the directions in which policy parameters change to the directions in which every state of the trajectory changes. In general, our algorithm learns the Jacobian matrix of the system every time step through the trajectory. The training phase consists of physically calculating directional deriva- tives by the finite difference after applying perturbed versions of nominal policy (a.k.a. controller). Perturbing the parameters of the controller is the reason for naming our method shaking. The test phase uses these directional derivatives to compute derivatives along unseen directions. Due the difficulty of computing the Jacobian matrix by the finite difference in higher dimensions, we use random controllers joint with probabilistic learning methods to obtain robust estimate of the Jaco- bian matrix at each instant of time along trajectory. We are capable of this generalization unseen perturbations because the trajectories of physical systems live on an intrinsic -indennniional man- ifold and change slowly with the small changes in the parameters of the system (Koopman [931 This assumption holds as long as the system is not chaotic or close to bifurcation condition (Khlli] 2002).

---

Under review as a conference paper at ICLR 2020
1.1 PRELIMINARIES
A reward function describes how close the agent is to the solution of the target task. In the absence of the reward, the agent will be given means find its way towards the solution. Let X C R be a ddddienssionll state vector that fully describes the environment with which the agent interacts. At each state, the agent is allowed to take action u U S Rº from g-dimensional action space via a parameterised policy function = (x; 0). The agent will be rewarded r(x, u) by the function r X U + I& when it takes action u at state x. The goal of learning to update such that some desired target is achieved. The target can be anything as long as concrete reward function associated with it. In stochastic cases, return R. II(@) + IR is defined as a cumulative future discounted reward whose expectation is often of main interest. For parametric policies, the space of feasible parameters è has a one-to-one correspondence to the policy space II. The agent who takes the policy from state X produces the trajectory T ' where T is the space of possible trajectories. For a return function R: : T + R, the expected return becomes a function of the policy as J(To) ET R(T)) where the expectation is taken with respect the probability distribution To). There exist two major classes of approaches in reinforcement learning: value-based methods and value-free methods. In the first class, a surrogate function is defined to approximate the value of either a state (xx or a state-action pair Q(x, u). The policy is updated such that the agent tends towards states with higher values. The value-free methods update the policy directly without any need for an auxiliary function such as V or Q. This paper mainly concerns the second
class. The policy parameters are updated as
(.(TT)
O-1 = O2 a
00
0==:
and the gradient (J(Tr)/00 is written as
aJ ( 00
J Tp(TTTm Te)
R(T) dT
(2)
00
which is normally difficult compute in practice. As can be seen in eq. 2 the integrand of the r...s. consists of two terms. The second term R(T) is the return which is defined according to the target task. Hence, this term is task-dependent. The first term (0) though shows how the trajectories change with respect to a change in the policy. Notice that there is no notion of reward or any task-dependent quantities in this term. For an empirical distribution pe (7[
M E 8(T - ) the dependence of partial derivative of the distribtion of T on the partial derivative of T can be explicitely derived as
1
Ope (T(T)) 00
M Σ (T - T OT
0e
where iN is the unit doublet function (derivative of the Dirac delta function). This examplary dis- tribution makes clear that the change in the distribution of trajetories relates to the change of the trajectories themselves. As unsupervised object, 0T/00 is of main interest in this paper.
1.2 PHYSICAL DERIVATIVE
In this paper, we investigate the feasibility of learning a less explored unsupervised quantity, the so called Physical Derivative which is computed directly from the physical system. In abstract terms, we perturb the policy and learn the effect of its perturbation on the resulting trajectory. The difference from traditional RL whose algorithms are based on eq. (T) is the absence of specified reward function. Instead, we generate samples from 0p(T ( mo)/00 of eq. (2) that makes it possible to compute (J(o)) for an arbitrary return function R. If the exact model of the system is known, control theory has a full set of tools to intervene in the system with stability and performance guarantees. When the system is unknown, one could identify the system as a preliminary step followed by normal control synthesis process from control theory (Ljung 2001). Otherwise, the model and the policy can be learned together in a model-based RL (Sutton 1996) or in some cases adaptive control (Sastry& & Bodson, 2011). We argue that learning physical derivatives is middle ground. It is not model-based in the sense that it does not assume knowing the exact model of the system. Rather, it knows how the trajectories of the system change result of perturbing the policy
2

---

Under review as a conference paper at ICLR 2020
parameters. This differential information of the system has applications in many downstream tasks. This work focuses on the concept and introduction of physical derivatives and direct applications would go significantly beyond the scope of this work. Few potential applications are discussed with more details in appendix
Our contributions In summary, the key contributions of the current paper follows:
A method to generate training pairs to learn the map from the policy perturbations to the resulting changes the trajectories.
Learning the above map as a probabilistic function and showing that generalizes to unseen perturbations in the policy.
Use the inverse of the above map to perturb the policy in the desired direction achieve certain goals without conventional RL methods.
Use a physical custom- buill robotic platform to test the method and propose solutions to deal with the inherent issues of the physical system ensure the practicality of the method (see fig. forr images of the platform and and appendix [ for technical details).
The supplementary materials for the paper, including code and the videos of the robot in action can be found in https: lhttps: http://ssitss.gggll....s.. com/view/ physiccalder.vativee.
2
METHOD
In this section, describe pipeline to estimate the physical derivatives and proposed solu- tions to the inevitable challenges that are likely to occur while working with a real physical robot. We interested in 0T /30 which denotes how small change in the parameters O of the troller results a different trajectory produced by the system. We normally consider a finite period of time [0, TJ and the trajectory an ordered list of states T [xo. . XT] where the sub- script shows the time step. Therefore, having 00/00 is equivalent with having dx,000 for every t E {1,... (1,...,77 T). Notice that the initial state X, is chosen by us. Hence we can see it either constant or as changeable parameter in O. We kept it fixed in our experiments.
Assume X, Rd and RT Hence, Vex; = 0xx/00 Rdx where the th row of this matrix is V (8) E R. showing how the 8th dimension of the state vector changes in response to perturbation in 0. The directional derivative of vse in the direction d0 is defined as
50
V Tee
(4)
100
If (4) is available for linearly independent and orthonormal directions, (80(1),800) the directional derivative along an arbitrary 80 can be approximated by
(m)))
80(m)),
50 V D4
(V
(5)
where ej (50, 80()) is the coordinates of the desired direction in the coordinate system formed by the orthonormal bases.
In practice, m directions 806) can be randomly chosen or can be along some pre-defined axes of the coordinate system. To compute (Vori, 800)), the nominal policy parameters 0 are perturbed by 5003) as g(i) + 0 + 50(5) and the derivative is computed as
Vorr, 5001) lim Jzt ( - h00 Fat (0)
This quantity is often approximated by finite difference where h takes a small nonzero value. By perturbing the parameters along m orthonormal directions 80() and computing the approximate directional derivative by (6), V Zi can be computed along every arbitrary direction 50, meaning that, we can compute V Vor by evaluating along any direction which aim of this paper.
3

---

Under review as a conference paper at ICLR 2020
In the matrix form for R. compute V x / 21, Vag
$1...., Vi ra] in
single run by computing (6) for all d dimensions of the states, Let's define
X, V
X, 50lmm)
Ag
Rdxm and let A 50(m)) Therefore, if A x shows the directional derivative of along 60, we can write it as:
where Agx (80(1)
Figure 2: Gaussian (left) and uniform (right) shaking examples.
(0 60(2),
V Aox(A
which is only a vectoral representation of eq. Even though the linear formula of eq. ( requires only m directional derivatives, has two major downsides. First, it does not give clear way to incorporate more than m training directional physical derivatives. Second, the linear approximation remains valid only for very small 50. We propose Gaussian Process (GP) as a nonlinear probabilistic function approximator (Rasmussen 2003 to capture the maps 9: defined as
3t : θ + de (50) 6x
(9)
(10)
where subscript t shows the function that maps 50 to the change of the states 8x, at time step t. We considered distinct functions for every time step. Taking into account the commonality among the function approximators corresponding to different time steps is deferred to future research. Learn- ing this map requires training data that comes from initial data collection phase called shaking- Shaking refers to perturbing parameters of the controller to obtain the set of trajectories produced by the perturbed controllers.
The perturbation can be either regular or stochastic. Stochastic perturbations have the advantage over regular perturbations that the agent does not need be worried about perturbing the parameters in a particular direction. Besides, in some cases, perturbing the parameters of the policy in certain directions is infeasible. We propose two methods of shaking called Gaussian and Uniform shaking.
Gaussian shaking- Likely values of 0 create nominal policies encoded by (0(1), 0(2) ,.... 0(m) ).
We put Gaussian distributions centered at each of the nominal values resulting in a mixture of Gaus- sians. To reduce the hyper-parameters, we assume the variances of the Gaussians are themselves sampled from an exponential distribution making sure they all take positive values (See fig. left). Here, we manually choose reasonable value for the rate parameter of the exponential distribu- tion. Doing inference on the hyper- paremeersseterssscee of the sampling distributions can be topic for future research especially in active learning for a more clever less costly sampling stratgey.
Uniform shaking- In this setting, the state space of the changeable parameters of the policy is dis- cretized and uniform distribution assumed around each value of this grid with some overlapping with the neighboring cells (See fig. right).
We show the effect of each of these sampling methods later in section We observed that the results are less sensitive to the hyper-parameter, of the uniform sampling than Gaussian sampling. A carelessly chosen rate for the exponential distribution that generates the variances of the Gaussians in Gaussian sampling result in too local or global sampling that gives rise large variance or bias in the estimated gradients.
REAL WORLD CHALLENGES
In this section, present two major low-level challenges that are common when dealing with physical systems. There exist inherent noise and imperfection in the system that results in a change in the produced trajectories while the policy parameters are kept fixed. In our finger platform, observed two different major sources of noise which are likely to occur in other physical systems too. We call them temporal and spatial noise for the reasons that come in the following.
4

---

Under review as a conference paper at ICLR 2020
Temporal noise. The temporal noise represented by affects trajectories by shifting them in time
X X₁ n for / = 0, ,. T.
(11)
Notice that the absence of subscript / in shows that this noise is not time-dependent, i.e. the time shift does not change along the trajectory as time proceeds.
Spatial noise. The trajectories affected by spatial noise cannot be aligned with each other by shifting forward or backward in time. We can model this noise as a state-dependent influence on the state of the system at every time step. X2 f- X nx
(12)
The following definition makes the distinction more concrete.
Definition 1. Consider two trajectories (t) and (t) as two temporal signals. Assume St the shift-in-time operator defined as
St. TT(+
(13)
for an arbitrary function of time T(t). We say (t) is temporally noisy version of T() if
3/. E R. s.t. [[T(2) St, 11 S
(14)
where e a exper-parameter threshold that reflects our prior confidence about the accuracy of the motors, joints, physical and electrical elements (in general construction process) of the robot. On the other hand, T(2) is called a spatially noisy version of (1) if
Mo. E R s.l. (]
T Se
(15)
3.0.1 SOLUTION TO TEMPORAL NOISE
Fortunately, this type of noise is not state-dependent by definition. If we find out how much trajectory is shifted in time with respect to another trajectory, we can simply shift the trajectory for those many time steps and compensate for the delay. Hence, the problem becomes detecting the lagged trajectories with respect to a reference trajectory and also estimate the amount of the required time shift to compensate for the delay. We can either use physical landmarks the trajectories to align them or use the correlation between them as measure of alignment. The later gave better results, hence, we postpone the description of the former the appendix D.I
Correlation-based delay estimation In this method, we use the correlation between zero zero-meaned trajectories (+) and T to check if one the lagged version of the other one. The delay is found by
T-T x(())
7" = arginax Σ (S,x
(16)
where S. is shift-operator by T E Z time steps. In practice, we take one trajectory of (T(1), T ..... ((M)). e.g. T(r) as the reference and synchronize other trajectories with respect to it using eq. (66). The trajectories must be initially normalized to avoid trivial solutions where ev- ery trajectory pushed towards the larger parts of the reference trajectory. For illustrative purposes, the plots of fig. [1] show sample of the lagged trajectory from the finger platform and its correction by the above method.
3.1 SOLUTION TO SPATIAL NOISE
The spatial noise can be a stochastic function of the actuator, environmental change, and electronic drivers. perfect model of the transition dynamics X,+1 = fx,, uz), applying the same control sequence (uo, u,,..., -r } always results in the same sequence of states {x1, X2,..., xr when it starts from the same initial state Xg. This assumption is often violated in physical systems as different runs of the same policy may result in different trajectories as can be seen in fig. 10 in the Appendix. The noise in the dynamics can be any function of states, input, and time. Therefore, is difficult model this noise since it requires prohibitively large number of random experiments. The good news is that if the physical system is built properly, the effect of this noise is expectedly low. Based on our observations from the finger platform, can assume the following.
5

---

Under review as a conference paper at ICLR 2020
(a) Small voxels (y 0.01)
(b) Medium voxels(y (c) Large voxels(yy 0.16)
(d) Large voxels(y 0.2)
0.04)
Figure 3: The effect of voxels on supressing spatial noise of the physical system. The trajectories are produced by linear open-loop controllers as those in section 4.1 for the purpose of illustrating the effect of voxelization.
Assumption 2. Limit on the physical noise: Let's the control sequence U {uo, ui,..., UT -T be applied to the system M times resulting in multiple sequence of states ( T (d1)) There exists relatively small Ç such that
||T() - 1 SS for every i,j (1,2. ..... m]
(17)
The word relatively here means that the change of the trajectory due to the inherent physical noise of the system must be small compared the change of the trajectories when the parameters of the policy are perturbed.
To reduce the sensitivity of the estimated gradient this unwanted spatial noise, divide the state space of the physical system into regularly located adjacent cells called voxels. Each voxel vor(c) represented by its center and defined as
voz(c) = fx X
|x x cl |
(18)
where y is the parameter of the voxelization. The concept of the voxel roughly used as a super- state. Every state that ends up within vox(c) gives rise to the same superstate. After recording the trajectories form the robot, every state is mapped to the center of the voxel it belongs to as
f- for x E voz(c)
(19)
After voxelization, work with instead of For example, all the gradients of 7 are computed as Vec rather than Vox. To illustrate the positive effect of voxelization of the state space, can be seen in fig. 3 that increasing the voxel size improves the overlapping between two trajectories that deviate from each other due to the inherent spatial noise of the system not because of perturbing the parameters of the policy, but because of the inherent imperfection of the mechanical and elec- trical components of the system. This benefit comes with a cost which is the error introduced by voxelization. Fortunately, this error is bounded due the following lemma
Lemma 3.
The error caused by voxelization is bounded and inversely proportional to the size of each voxel (see appendix F. for brief proof).
After dealing with the challenge of inherent noise, we pursue the main goal of this paper which is estimating öT 000 directly from the physical system. In the following, we investigate the use of the different type of controllers to emphasize the extent of applicability the proposed method.
EXPERIMENTS
In this section, we show how physical derivatives can be estimated in practice through several ex- periments. Notice that our work is different from computing gradients around the working point of system by Emite-difference. We aim to collect samples from such gradients by perturbing grid of nominal values of the policy parameters and then generalize to unseen perturbations by Gaussian process as a probabilistic regression method. The experiments are designed to show each challenge separately and the efficacy of our proposed solution to it. Due to space constraints, details to the physical platform can be found in section A in the Appendix. Sed") for videos of the robot while collecting data for different experiments and more backup materials.
https://siit google..g..ooo.oooooooo e. wwt. wttpe/eooowweeeeeeeeeeeeecc 1derivateiiii

---

Under review as a conference paper at ICLR 2020
(a) Low noise
(b) Medium noise
(c) High noise
Figure 4: controller with various noise intensities on Kp parameter.
Figure 5: The time evolution of the GP approximated dt. for PD feedback controller at some exem- plary time instances
Figure 6:
The time evolution the GP approximated 9. for nonlinear sinusoidal open-loop con- troller at some exemplary time instances.
4.1 LINEAR OPEN-LOOP CONTROLLER
As simple yet general policy, in this section, we consider an open-loop controller which is linear function time. The policy u, [Mi,, U2t, Usze] constitutes the applied torques to the three motors {m1, m2, m of the system and is assigned
Uar wet for 1,
(20)
Notice that the torque consists of two terms. The first term wit grows with time and the second term remains constant. The controller has 6 parameters in total denoted by 0. The task is to predict Vox; for every I along the trajectory. In the training phase, the training data is obtained via shaking as described in section 2
fig. shows examples of nominal trajectories + trajectories produced by the perturbed controller and the computed derivatives. The arrows are plotted as they originate from the perturbed trajectories only for easier distinction. Each arrow corresponds to the change of the states at certain time step on the source trajectory as a result of perturbing the policy. Each figure corresponds to a pair of nominal values of {w, b} for the linear open-loop controller. See fig. 29 for examples.
4.2 NONLINEAR OPEN-LOOP CONTROLLER
Physical derivatives can naturally be computed for either linear or nonlinear controllers which makes it different from taking the gradient of models through time. In model-based methods, if the model's transition dynamics not differentiable, taking the derivative is theoretically challenging. However, method is taking advantage of the real physics of the system compute the gradients regardless of whether the approximating model is differentiable or not. To elaborate more on this, we test our method for simple but nonlinear policy, i.e., ut ssi(w.)). The sinusoidal torque is applied

---

Under review as a conference paper at ICLR 2020
The aggregate performance of our method to predict physical derivatives in unseen direc- Table 1:
tions of perturbations to the parameters. (-) shows time average. The first column is the normalized time averaged MSE. The second column is the time averaged GP score (closer is better. See ap- pendix E.4 for definition). The third column is the time averaged misalignment between derivatives. Every experiment is repeated for 10 voxel sizes and the values are chosen for the best voxel size. N: Gaussian sampling, U: uniform sampling.
Task
(MSE)
(Score)
(cos a)
PD controller (N)
0.00871
0.8991
0.9492
PD controller (U)
0.0018
0.9841
0.9723
Sine 2 joints (U)
0.0368
0.7992
0.9513
Sine 2 joints (N)
0.0796
0,6696
0,9553
(a)
(b)
(c)
(d)
Figure 7: Pysical gradients computed for various time steps along source trajetcory using two
perturbed trajectories of linear open-loop controller.
to either one or two motors of the system to investigate the performance of our method. We tested Gaussian and uniform shaking for 0 {AA, w) as parameters of this controller. The GP interpolation for the partial derivatives at some time instances along the trajectory can be seen in fig. 6 and more extensively in figs. 1o 18 in the Appendix. One might be interested in the direction of the predicted derivative instead of its exact size. To this end, we take several test perturbations for every time step and use cas(a:) measure of alignment between the predicted and ground-truth derivative vectors. The time evolution of the histogram of this measure along the trajectory shows better alignment as time proceeds. This effect can be seen in figs. 27 and 28 This confirms our observation of initial transient noise in the system that dies out gradually by the progression of time. The overall performance of our method in predicting physical derivatices unseen directions for two different shaking methods is shown in appendix
4.3 FREDBACK CONTROLLER
Often in practice, the policy incorporates some function of the states of the system. Some well- known examples which have been extensively used in control applications are P. PD, PI and PID controllers. Here, we consider two member of this family, i.e., and PD controllers. The policy becomes K,e for P controllers and the difference between the current state x and the desired state x". The parameters of the controller (Kp, are scalar values that are multiplied by the eπrоr vector element wise. This implies that the controller parameters are the same for three motors leaving the controller of the whole plat- form with two parameters that weights the value and the rate of the error. We applied the uniform and Gaussian shaking for the set of parameters interpolation for the physical derivatives at some time instances along the trajectory can be seen in fig. 6lnd more extensively in figs. to 24]in in the Appendix. The time evolution of the histogram of misalignment between predicted and ground- truth directional derivatives (see figs. 25 and 28 in the appendix) once again confirms the existence of the initial transient noise in the section 4.2] Similar to the sinusoidal experiment, the overall performance of our method is presented in appendix H
K,e K for PD controllers. The error shows
[Kp, Ka} with different scenarios. The GP
also observed
8

---

Under review as a conference paper at ICLR 2020
14
Sepaciti
(a) Short distance target
(b) Medium distance target
(c) Long distance target
Figure 8: shot planning with constraint satisfaction. The orange trajectory is the source pro- duced by the nominal controller. The green and blue are two sampled trajectoreis that are produced by perturbing hp to k. by eq. (2I
4.4 ZERO- SHRO-SHOTTAN PLANNING TASK
Our previous experiments in sections 4.2 and 4.3 showed that learning the physical derivative map is feasible for various types of controllers. In this section, we demonstrate example of constrain satisfaction task by means of the physical derivative map. In this experiment, the su- perscript (s) corresponds the nominal trajectory which is called source. Assuem the system is controlled by a PD controller to reach a target state x', i.c., the control torques are designed as = k((²) (x x*) k(*)x x. The controller does decent job reach the target state given rea- sonable values for kip and ka- However, such controller does not give us a clear way to shape the trajectory that starts from X. and ends x*. Assume it is desired that the nominal controlled trajectory T(s) passes through
intermediate state x at time t its way towards the target state x* (we can equally assume that the system must avoid some regions of the state space be- cause of safety reasons). The solution with physical derivatives is as follows Assume k(()) is fixed and only k(() is changeable. If the physical derivatives map is available, we have access to gr (kp k(()) = (x: / k())).
). By simple algebraic rearrangement, we have
x - x, (0) (k 1()1)
+ tx(.).
(21)
The new parameter of the policy is supposed to push the source trajectory (+)) towards target trajectory " that passes through the desired state x; at time t. The result of this experiment on our physical finger platform can be seen in fig. g
4.5 RELATED WORKS
A truly intelligent agent must develop some sort of general competence that allows it to combine primitive skills to master a range of tasks not only a single task associated with a specified re- ward function. The major part of such competence come from unsupervised experiences. Animals use similar competence to quickly adapt to new environments (Weng et al. (001). and function efficiently soon after birth before being exposed to massive supervised experience (Zador) 2019). Due to its generality, such basic skills can be inherited over generations rather than being learned from scratch (Gaier Ha 2019). Despite traditional RL that the learning is driven by an extrin- sic reward signal, intrinsically motivated RL concerns task-agnostic learning. Similar to animals' babies (Towwen et al., 1992), the agent may undergo developmental period in which it acquires reusable modular skills (Kaplan & Oudeyer 2003 Weng et al.) (001)) such as curiosity and confi- dence (Schmidhuber) 1991a Kompella et al., (017). Another aspect of such general competence is the ability of the agent remain safe during its learning and deployment period (Garcia & Fernández, 2015). In physical systems especially continuous control, stability is a major aspect of safety that implies states of the system converge to some invariant sets remain within a certain bound (Lyapunov) 1992). Control theory often assumes the model of the system known to guarantee stability (Khalil) 2002). In the absence of the model, model-based RI. learns the model along with

---

Under review as a conference paper at ICLR 2020
the policy. Hence, learning the transition model to predict the states in the future can be another intrinsic reward.
From technical point of view, work is relevant sensitivity analysis and how it is used train the parameters of models such as in [Chen et al. NeuralODE. The method seemed to be effective in many tasks including learning dynamics (Rudy et al. 2019). optimal control (Han et al.. 2018), and generative models (Grathwohl et al. 2018). Our method can seen as a mode-free sensitivity analysis in real-world systems. In NeuralODE, the gradient with respect to the parameters requires solving ODEs for both states and adjoint states that require a transition model. Since we are working directly on the physical system, we don't need to calculate the integrals forward in time. The systems itself acts as physical ODE solver. We refer to appendix F more detailed review of the related works.
CONCLUSION
In this paper, we present a method to learn the way that the trajectories of physical real world dynamical system changes with respect to change in the policy parameters. We tested our method on a custom-built platform called finger robot that allows testing a couple of controllers with various settings to show the applicability of our method for linear, nonlinear, open-loop, and feedback con- trollers. By estimating the physical derivative function, we showed that our method able to push controlled trajectory towards target intermediate state. We investigate the real-world challenges when doing a fine sensitive task such as estimating physical derivatives on a real robot and proposed solutions make algorithm robust to inherent imperfection and noise in physical systems. We focused mainly on low-level issues of physical derivative and showing the feasibility of estimating robustly. We expect that physical derivatives will contribute research areas such as safety, control with constrain satisfaction and trajectory planning, robust or safe control.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Adrien Baranes and Fierre-Yves Oudeyer. Active learning of inverse models with intrinsically mo- tivated goal exploration in robots. Robotics and Autonomous Systems, 31(1):49-75 2013.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pp. 1471-1479, 2016.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differ- ential equations. In Advances in neural information processing systems, pp. 6571 2018.
Adam Gaier and David Ha. Weight agnostic neural networks. CoRR, abs/1966.04355, 2019. URL httt:://arxis: wttp://////bb/////eeee 04358
Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1488) 2015.
Will Grathwohl, Ricky TQ Chen, Jesse Betterncourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint
180.0.7810 2018.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint 1r .77507, 2016.
Jiequn Han, Qianxiao Li, et al. A mean-field optimal control formulation of deep learning. arXiv preprint
2018.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. 2018.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Variational information maximizing exploration. 2016.
Frederic Kaplan and Pierre-Yvel Oudeyer. Motivational principles for visual know-how develop- ment. 2003.
Hassan K Khalil. Nonlinear systems. Upper Saddle River, 2002.
Varun Raj Kompella, Marijn Stollenga, Matthew Luciw, and Juergen Schmidhuber. Continual curiosity-driven skill acquisition from high-dimensional video inputs for humanoid robots. Arti- ficial Intelligence, 247:313-333 2017.
Bernard O Koopman. Hamiltonian systems and transformation in hilbert space. Proceedings of the National Academy of Sciences of the United States of America, 17(5):315, 1931.
Lennart Ljung. System identification. Wiley Encyclopedia of Electrical and Electronics Engineer-
ing. 2001.
Aleksandr Mikhailovich Lyapunov. The general problem of the stability of motion. International journal of control, 55(3):531-555 -534, 1992.
Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer School on Machine Learming, pp. 63-71. Springer, 2003.
Samuel H Rudy, J Nathan Kutz, and Steven I. Brunton. Deep learning of dynamics and signal-noise decomposition with time-stepping constraints. Journal of Computational Physics, 396:483-506. 2019.
Shankar Sastry and Marc Bodson. Adaptive control: stability, convergence and robustness. Courier Corporation, 201
Jürgen Schmidhuber. Curious bodel-building control systems. Proc. international joint confer- on neural networks, pp. 1458 1463, 1991a.

---

Under review as a conference paper at ICLR 2020
Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neu- ral controllers. In Proc. of the international conference on simulation of adaptive behavior: From animals to animats, pp. 222-222211111111 1991b.
Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE Transactions on Autonomous Mental Development, 2(3):230-247, 2010.
Leonid Kuvayev Rich Sutton. Model-based reinforcement learning with an approximate, learned model. In Proceedings of the ninth Yale workshop on adaptive and learning systems, pp. 101- 105, 1996.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
BCL Touwen, MS Hempel, and LCC Westra. The development of crawling between 18 months and four years. Developmental Medicine & Child Neurology, 34(5):410-416, 1992.
Juyang Weng, James McClelland, Alex Pentland, Olaf Sporns, Ida Stockman, Mriganka Sur, and Esther Thelen. Autonomous mental development by robots and animals. Science, 291(5504): 599-600, 2001.
Anthony M Zador. A critique of pure learning and what artificial neural networks learn from animal brains. Nature communications, 10(1):1-7, 2019.
Kemin Zhou and John Comstock Doyle. Essentials of robust control, volume 104. Prentice hall Upper Saddle River, NJ, 1998.
12

---

Under review as a conference paper at ICLR 2020
A PHYSICAL PLATFORM
In this section, we introduce the physical robot on which tested our method. The robot is called finger platform or simply finger throughout this paper. The range of movement for the motors are [0, n|, [0, n]. [0, 2nT respectively. The axes of the plots throughout the paper are radian. consists of three articulated arms with three degrees of freedom in total (see fig. [d]. The tors {m1, m2, m are depicted the figure. This naming remains consistent throughout this paper. Each swing in its own plane (see fig. 9ap. Each arm equipped with an encoder that measures its angle (see fig. 9b). The brushless motors controlled by electronic driver that receives torque val- ues applied to each motor from a computer terminal via a CAN bus and applies the torques to the motors(see fig. Dc Due the imperfections of the arms, motors, and drivers, did not any model for the system including the inertial matrix of the robot or the current-torque characteristic function of the motors. The low-cost and safe nature of this robot makes a suitable platform to test the idea of physical derivatives that requires applying many different controllers in the training phase.
is moved by separate brushless DC motor and has one degree of freedom to
(a) Motor
(b) Encoder
(c) Driver
(d) Finger
Figure 9: Components of the physical finger platform
B ADDITIONAL PLOTS ILLUSTRATING REAL WORLD CHALLENGES (SECTION 3
(a) 200
(b) 400
(c) 600
(d) 800
(e) 1000
Figure 10: Same controller applied for multiple runs. The trajectories are produced by the linear open-loop controller similar to those used in section 4.1 See the plot for a different set of nominal parameters of the controller in fig. 13 in the Appendix (Zooming recommended)
(a) t 200
(b) 400
(c) f = 600
(d) 800
(0) 1000
Figure 11: The same fig. 10 but for different setting.
13

---

Under review as a conference paper at ICLR 2020
(a) t 200
(b) 400
(c) 600
(d) 800
(e) 1000
Figure 12: Noisy linear open-loop controller
(a) / 200
(b) = 400
(c) 1 = 600
(d) = 800
(e) 1000
Figure 13: The as fig.
12 but for different nominal parameters of the policy.
m1
15
30
3
1.4
16
1.
3
32
4
34
(a) Trajectories affected by temporal noise
(b) Temporally aligned trajectories
Figure 14: The effect of temporal noise in delaying one trajectory versus the other one and its correction. The trajectories are produced by the linear open-loop controller similar to those used in section 4.1 Zooming is recommended)
C APPLICATIONS OF PHYSICAL DERIVATIVES
If we know how the states of trajectory change as result of change in the policy parameters, the policy can be easily updated to push the trajectory towards a desired one. example, assume we are interested in going from the current trajectory T(0) to the target trajectory T*. The distance between these trajectories can get minimized by perturbing the policy parameters in the direction T(0) T |/80. This direction is already available since have estimated ØT (0)/00 as a physical derivative. As an exemplary case, we show this application of our method in practice in section Other applications of physical derivatives are in robust control and safety. In both cases, the physical derivative allows us to predict the behaviour of the system if the policy changes in neighbourhood around nominal policy. Then, it is possible make sure that some performance or safety criteria will not be violated for the local perturbation in the policy. As a concrete example, for an autonomous driving system, there can be calibration phase during which physical derivatives of the car is estimated by perturbing the controller parameters around different nominal policies which are likely to occur in real roads. The calibration must done in a safe condition and before deploying the system. When deployed, the estimated physical derivatives can be used to predict the effect of a change the policy on the behaviour of the system and neutralize the change if it would move the car towards unsafe regions of its state space. The command that changes the policy can be issued by a high-level controller (e.g. guidance system), and the safety is confirmed by a low-level mechanisms through physical derivatives. This work focuses the concept and introduction of physical derivatives and direct applications would go significantly beyond the scope of this work. In the following more detailed description of the use of physical derivatives in robust and safe control.
14

---

Under review as a conference paper at ICLR 2020
Robust control In control theory, robust control relates to the design of controller whose per- formance is guaranteed for a range of systems and controllers belonging to a certain neighborhood around the nominal system (Zhou & Doyle 1998). It is desired have controller that keeps the performance of the system at a certain good level even if the parameters of the controller are not fixed to the theoretical values. Assume the performance of the system is associated with some func- tion of a trajectory E(T). Changing the parameters of the controller 0 results in a change in the trajectories. This allows us compute 0T/ /00 that consequently gives us (E( by the chain rule. Roughly speaking, between two sets of parameters 01 and 02, the set of parameters that gives the least 00/06 preferred. This means that by shaking the parameters of the controller and as- sessing the performance of the system, an estimate of the curvature of the landscape of ((T( is obtained. We prefer flatter regions of this space where small change in 0 does not cause drastic change in the performance metric E
Safety Safety refers to the situations in which the agent may hurt itself the environment and causes irreversible damages if it freely takes arbitrary actions (Garcia & Fernández 2013). For a safety-critical system whose full physical models are hard obtain, the physical gradients assist in avoiding restricting the parameters of the robot to avoid unsafe behavior. The physical derivatives are learned in the Lab environment before the robot is deployed into the wild. For example, a rover whose mission is to safely explore an unknown environment often enjoys learning loop that allows to adapt the new environment. Even though the learning in the new environment requires sufficient exploration, the physical derivatives can be used to give rough simulation of the robots next few states under given update to its parameters. The potential harmful updates might be detected by such simulation and be avoided.
D EXTENDED SET OF SOLUTIONS TO THE REAL WORLD CHALLENGES
D.1 DETECTING ZERO CROSSING
In this method, we take advantage special landmarks in the trajectories. The landmarks are typically caused by physical constraints of the system. For example, when robot's ss leg touches the ground, the velocity the leg becomes zero. Likewise, when a joint reaches its physical limit, the velocity of the connected arm to the joint becomes crossing occurs that can be used as a landmark to synchronize lagged trajectories with a reference trajectory. Even though this method will eliminate the temporal noise, it requires the presence of such landmarks along the trajectories. Notice that from mathematical point of view, there nothing special about zero. We can pick any value of states along a reference trajectory and synchronize all other trajectories with respect to it. However, in practice, physical landmarks are easier to detect and have less ambiguity that consequently gives a more accurate synchronization.
changes sign. In both cases,
E EXPERIMENTAL DETAILS
Starting position in all the experiments is ( 20 m). Task's overall details are as following:
Task
number of trajectories
timesteps
Linear (N)
640
1500
PD controller(N)
640
1500
PD controller(U)
1000
1500
Sine joint(N)
640
5000
Sine joint(U)
1000
5000
Sine 2 joints(U)
640
5000
Sine 2 jo)
1000
5000
In normal sampling cases, we ran 10 simulations for each set of À parameters which indicates noise level.
15

---

Under review as a conference paper at ICLR 2020
E.1 LINEAR
Mat = wil+bb for i 1,2,3
(22)
E.1.1 GAUSSIAN SAMPLING
04 W; + 6x0 for 1, 2,3 Euyi (N, e.p x ([Wll2)
ew exp(Aw) for Au 1, 5, 10, 50, 100, 500, 1000, 5000
b, = B, + Eb,i for = 1,2,3
Eb NO, x |BB:l2)
e exp(Ae) for b 1, 5, 10, 50, 100, 500, 1000, 5000
W = [0.00001, 0.0001, - 00.00001, B = [-0.28, 0.15, -
E.2 PD CONTROLLER
Final destination is 3
E.2. GAUSSIAN SAMPLING
kp KP
Ekp ~ N NO0, Ckp x ||KP])
Ckp exp(Akp) for
Akp = 1, 5, 10, 50, 100, 500, 1000, 5000
kd = KD +e
Ekd ((0, End KDI))
Ckd ~ expp(Axd (Akd) for Aua = 1, 5, 10, 50, 100, 500, 1000, 5000
E.2.2 UNIFORM SAMPLING
kp 1.)
kd = KD 0.01
E.3 SINE 1 JOINT
E.3.1 GAUSSIAN SAMPLING
W +
Em N(O,e, en x ||]]]
en ~ exp(>ww) for A. 1, 5, 10, 50, 100, 500, 1000, 5000
== A+e
6a N(0,c. ca ||^|])
ea ~ exp(Aa) for A, = 1,5, 10, 50, 100, 500, 1000,
W = 0.01, B 0.5
E.3.2 UNIFORM SAMPLING
(0.0055 0.015), A = 0.5
16

---

Under review as a conference paper at ICLR 2020
E.3.3
SINE 2 JOINTS
E.3.4
GAUSSIAN SAMPLING
W/ = W; + for i 1,2
cw, (0, Eur |
e exp(Aw) for Au 1, 10, 50, 100, 500, 1000, 5000
a = A + for i= 1,2
Ca5 (0, ea x |All))
ca ~ exp()a) for Aa = 1, 5, 10, 50, 100, 500, 5000
5000
W = 0.011 0.01], A = [-0.4,0.5
E.3.5 UNIFORM SAMPLING
W: ~ (0.005,, for i= 1,2, =
E.4 GP SCORE:
Definition of the GP score: The score is defined (1 uu/w))), where is the residual sum of squares ((Hrre "ypred and v is the total sum of squares (Verrue (eaan())ee))" The best possible score
1.0.
E.5 ZERO-SHOT PLANNING TASK:
For the task of section 4.4 Number of training trajectories; 100 each with 1500 time steps
Kd = 0.01
Kp = Uniformly sampled from [0.2, 0.6]
Initial point: X. = m/,
desired position [T/10,3 n/4,7 1/12]
F DETAILED LITERATURE REVIEW
There has been a recent surge of interest in unsupervised methods in reinforcement learning when a task-specific reward function is not the only driving force to train the agent (Baranes Oudeyer
2013 [Bellemare et al., 2016, Gregor et al.] 2016 Hausman et al.) 2018 Houthooft et 2016). A truly intelligent agent must behave intelligently in a range of tasks not only in a single task associ- ated with its reward function. This requires the agent to develop some sort of general competence that allows it to come up with solutions to new problems by combining some low-level primitive skills. This general competence is key factor in animals quickly and efficiently adapt new problem (Weng et al. 2001). By calling the traditional RL, extrinsicially motivated RL, the new framework is called intrinsically motivated RL. There have been many ideas in this line with various definitions for the terms motivation and intrinsic. Some researchers assume a developmental period which the agent acquires some reusable modular skills that can be easily combined tackle more sophisticated tasks (Kaplan & Oudeyer 20033 Weng et al. (200I). Curiosity and confidence are other unsupervised factors that can be used to drive the agent towards unexplored spaces to achieve new skills (Schmidhuber, 1991b] Kompella et al., (2017). Interestingly, there are observations in neuro- science that dopamine, a known substance that controls one's motivation for extrinsic rewards, is also associated with intrinsic properties of the agent such novelty and curiosity. A novel sensory stimulus activates the dopamine cells the same way they are activated by extrinsic reward. Children build collection of skills accumulatively while they engage in activities without specific goal, e.g., hitting a ball repeatedly without a long-term target such as scoring goal. The achieved skills contribute their stability while handling objects (Touwen al. 1992).
17

---

Under review as a conference paper at ICLR 2020
x
d(vō 1)
Aw
x
X(
d
Amr
x(
d
x
I 2)
Ame
Figure 15: The maximum potential error in the estimated gradients when the space is voxelized. As can be seen, the error is vanishing when the corresponding voxels to x(1) and x(5) are far from each other.
Another line of work concerns the fundamental constraints of the agent/environment and ensures those constraints are met while learning. For example, in many practical systems, learning episodes must halt if the system is likely to undergo irreversible change. For example, the training episodes of fragile robot must ensure the robot does not fall or will not be broken in any circumstance while acting under certain policy. The general name safe RL embodies ideas to tackle such issues in current interactive learning algorithms (Garcıa & Fernández 2015). One major aspect of safety is stability that loosely means that states of the system converge some invariant sets remain within a certain bound (Lyapunov) 1992). Control theory enjoys physical model of the system to guarantee stability (Khalil 20022. When the physical model is known in advance, the model is either learned along with the policy (model- based RL) or will be implicitly distilled in the value function (model-free RL) (Sutton & Barto 2018). Stability can be categorized as an intrinsic moti- vation for the agent. No matter what task the agent aims to solve, it must remain stable all the time. Learning the transition model which is the major concern of model-based RI can also be seen as trinsic motivation. The agent learns to predict the future step given the current state. The advantage of learning a model-even inaccurately-is twofold: the agent would know where to go and where not go. It knows which regions of the state space is unsafe to explore and must be avoided. It also knows which regions are unexplored and might be informative to improve the model. This brings us to another view intrinsic reward that encourages diversity.
Our work is also relevant to sensitivity analysis and its use in trainig the parameters of dynami- cal models. After Chen et al.]s NeuralODE training neural networks by sensitivity analysis of the network parameters, the method was successfully applied to various tasks such as learning dy- namies (Rudy al.] (209)) optimal control (Han al. 2018). and generative models (Grathwohi ett al. 2018). Our method can be seen as a mode-free sensitivity analysis in real-world systems. In NeuralODE, the gradient with respect to the parameters requries solving ODEs for both states and adjoint states that require transition model. Since we are working directly on the physical system, we don't 't need to calculate the integrals forward in time. The systems itself acts as physical ODE solver.
The importance of learning from unlabelled experiences is a known fact in animals. Many animals function efficiently soon after birth before being exposed to massive labeled experience. Part of it might be due to unsupervised learning but the major part of the story can be genetic heritage after years of evolution that Zador called genomic bottleneck. The same idea turned out to be valid in sta- tistical learning where an automatically discovered neural network architecture peforms surpsingly well with shared random weight (Gaier & Hap 2019). The embedded inductive bias in the neural network architectures could be analogous to the wiring of the brain of animal babies which transfers from generation to generation by genes.
18

---

Under review as a conference paper at ICLR 2020
F.1 PROOFS
Proof to the lemma on voxelization error.
Proof. The voxels become boxes in 3D as fig- 15 between two points in 3D coordinates. Hence the source of voxelization error is approximating the distance between two points 3D with the distance between the centers of the corresponding boxes to which those points belong. This error is written next to the boxes in fig. [5] The maximmum error is inversely proportional to the distance between voxels. Meaning that the voxels which are located far away will induce less voxelization enror This is intuitively clear. When two points too distant from each other, a slight change in their position would not change the distance between them considerably. The upper bound on the error, however, occurs for single voxel where the error is bounded by the size of the voxel.
The gradient is estimated as the distance
19

---

Under review as a conference paper at ICLR 2020
G MORE RESULTS
In this section, the results of the extra experiments that were eliminated from the main text due to the space limit are presented.
The following figures show GP models trained by a set of directional derivatives collected during
the shaking phase. The results are provided for the experiments of sections 4.2 and 4.3
Hol
Figure 16: The time evolution of the learned GP models from directional derivatives for dxs Okk
by Gaussian sampling
20

---

Under review as a conference paper at ICLR 2020
Figure 17: The time evolution of the learned GP models from directional derivatives for 05z/ Okp by uniform sampling
5"
MW
Figure 18: The time evolution of the learned GP models from directional derivatives for uz Ok by uniform sampling
21

---

Under review as a conference paper at ICLR 2020
8
0.4
Figure 19: The time evolution of the learned GP models from directional derivatives for dr / 0k by Gaussian sampling
Figure 20: The time evoltuion of the learned GP models from directional derivatives for dz Ok, by Gaussian sampling
22

---

Under review as a conference paper at ICLR 2020
6-
7-
Figure 21: The time evolution of the learned GP models from directional derivatives for 05z// Okp by Gaussian sampling
5
Figure 22: The time evolution of the learned GP models from directional derivatives for ox, /kkp by uniform sampling
23

---

Under review as a conference paper at ICLR 2020
Figure 23: The time evolution of the learned GP models from directional derivatives for 052 Okp by uniform sampling
5.44
Figure 24: The time evolution of the learned GP models from directional derivatives for uz, Ok, by uniform sampling
24

---

Under review as a conference paper at ICLR 2020
Figure 25: The time evoltuion of the histogram of cos(ar where the angle between the true and predicted directional derivative. The perturbations in the training phase are generated by Gaussian sampling.
Figure 26: The time evolution of the histogram of cos(a: where the angle between the true and predicted directional derivative. The perturbations in the training phase are generated by uniform sampling.
25

---

Under review as a conference paper at ICLR 2020
Figure 27: The time evolution of the histogram of cos(ar) where the angle between the true and predicted directional derivative. The perturbations in the training phase are generated by Gaussian sampling.
Figure 28: The time evolution of the histogram of cos(a) where the angle between the true and predicted directional derivative. The perturbations in the training phase are generated by uniform sampling.
26

---

Under review as a conference paper at ICLR 2020
(a)
(b)
(c)
(d)
(e)
(I)
(g)
(h)
ii)
i
(k)
Figure 29: Examples of trajectories produced by the perturbed controller and the computed deriva- tives along the trajectory. The arrows are plotted as they originate from the perturbed trajectories only for easier distinction. Each arrow corresponds to the change of the states at certain time step on the source trajectory as a result perturbing the policy. Each figure corresponds to a pair nom- inal values of {w. b} for the linear olen-loop controller of section 4.1 and the perturbed trajectories are produced by Gaussian sampling.
27


Reviews:
[{"review_id": "S1eCOIEOqS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes learning physical derivatives, the derivative of the trajectory distribution with respect to policy parameters. The proposed method estimates changes in  trajectories at a particular theta by using finite differences,\nthen fitting Gaussian Processes per timestep to generalize to new dtheta's. The paper then proposes techniques to robustify the process against noise in the system.\nTo deal with temporal noise, where trajectories are approximately equal up to a time shift, they simply\nestimate the optimal shift and use the shifted version to estimate. To address more complicated noise, they assume sensitivity of trajectories to noise is small relative \nto sensitivity to the parameters, and discretize the state space at a level such that trajectories that \ndiffer primarily due to inherent noise look the same at the discretized level, while perturbed policy \nparameters still lead to different trajectories. They then use the discretized trajectories to estimate \nthe finite differences.\n\nExperiments illustrate how the learned predictions compared to actual resulting perturbations and \nillustrate resulting trajectories on certain toy domains and a physical robotic finger. All experiments\nare done with very low dimensional state and policy spaces (1-3 dimensions each). \n\nWithout much more extensive experimental validation, this paper should be rejected. While I am not aware \nof any prior work on learning physical derivatives, the actual methods used are not novel in of themselves\nbeyond being applied towards learning derivatives with respect to the policy. As such, the method should be\nof practical interest in order to be accepted. With limited experiments on a single very low dimensional \ndomain and no comparisons against any alternative methods, there is little evidence demonstrating the actual \neffectiveness of the proposed method, especially on more complex domains and for downstream tasks.\n\nSuggested Experiments:\n\n- Stability in gradient estimation\nIt seems like it could require huge amounts of samples to be able to estimate gradients at parameters \nwhere for which the system is not very stable, as states in at later timesteps can easily change in \nhard to predict ways as the dynamics are propogated through time. This should be an especially big issue \nif we do not already have good stable controllers close to the desired solution and needed to actually \nconduct exploration in parameter space to solve a taask. I would appreciate more extensive evaluation \nacross multiple different (simulated) domains and assessing the effectiveness of gradient estimation \nalong random parameters.\n\n- Dimensionality of policy parameters and state spaces\nThe current experiments only involve very small parameter spaces. It would be important to see how well\nusing finite differences and GP regression scales with a higher dimensional search space, which can be \ndemonstrated on varying dimensionalities of an LQR system for example. It would also be important to see\nhow gradient estimation scales with high dimensional state spaces even with a small parameter space (like\nin the PD controller experiment in the paper). \n\n- Direct Comparison against learning dynamics models\nUsing the same data, compare (with same metrics as in table 1) physical derivatives estimated with the \nproposed approach against learning GP dynamics models and rolling out the perturbed policy with the learned\nmodel. Without a direct comparison against learning dynamics models and understanding what situations \nlearning physical derivatives provides better estimates, it is unclear when or why one would prefer to learn \nphysical derivatives in this way compared to a model based approach. \n\n- Quantitative results measuring costs of learned controllers\nDespite the name of the paper and a description of how to compute a policy gradient via physical derivatives, \nthere are no experiments involving such policy gradient updates as far as I can tell. While one advantage of the \nmethod (as well as model based approaches) is the ability to learn in unsupervised manner, it would be extremely \nhelpful to validate how well the physical derivatives are estimated in terms of how useful they are for a downstream \ntask, such as optimizing a controller for a cost function. Right now, experimental results lack any comparisons\nto other methods or any other way to assess the effectiveness of estimating physical derivatives. \nA comparison against regular RL policy gradient methods (or other model free algorithms) and model based RL \nwould give an idea as to whether the physical derivatives learned are actually useful.\n\nOther questions and comments:\n\t- Table 1: is this evaluating the accuracy of the physical derivatives on the shaking data that it was\n\t  used for learning, or on a validation set? If on a validation set, would the validation perturbations be\n\t  drawn from the same distribution as the training set?\n\t- The zero shot planning experiment in section 4.4 seems very contrived. It does not seem like a useful task\n\t  to adjust the parameters of the PD controller in order to reach a state that isn't the target. The figures \n\t  illustrating trajectories are also not very convincing and unclear. Two points are labelled source state and\n\t  target state, but it is not clear which is the intermediate state it is supposed to reach. In any case, most \n\t  of the trajectories seem to vastly overshoot the target? final state, and it is hard to assess how close the\n\t  trajectories end up being to the intended states from a 2d representation of a 3d space. Quantitative results\n\t  would perhaps have been more useful in illustrating the effectiveness of using physical derivatives.\n\t- What is the purpose of figure 4? It does not appear to be referenced in the text and it is not clear what is\n\t  being shown.\n\t\nOther notes not part of decision:\n\tPaper exceeds the 8 page recommended length\n\tLots of small typos in the text\n\n\t", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HJeEH7Lf9B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary of paper*\nThis paper investigates the use of random perturbations applied to a robotic policy to learn a local gradient useful for policy optimization. The method aims to learn a policy directly on a real physical robotic system, bypassing both simulation models and model-free RL. Training pairs are gathered by perturbations of a starting policy, and the \"gradient\" is captured in a probabilistic model learned from the training data. The paper includes experiments on a custom 3-DOF robotic platform.\n\n*Decision*\nI vote for rejecting this paper. While the idea is interesting, the paper lacks precision in key areas and the method is not placed in context among related work. Further, it fails to communicate key ideas (particularly in the experiments) to a non-robotics reader. Without sufficient clarity and background, it is not suited to a general machine learning conference.\n\n- Lemma 3, which attempts to justify the use of voxelization, and its proof are both imprecise and inadequate. To improve precision, please define \"error causes by voxelization\" in mathematical terms, e.g. ||c_i - x_i||. Also, while the statement of the lemma un-intuitively implies that larger voxels introduce smaller errors, the proof seems to say that larger errors will result for smaller gradients if larger voxels are used.\n- Related work: How does this work relate to random search/evolutionary computation? How does it compare to performing those methods or a model-free RL method directly on the robot? How does it compare to learning using an inaccurate model for robot dynamics? Presumably there are numerous methods that have been tried in this area, so further context is needed.\n- The evaluation is unclear, at least to a non-expert in robotics. A lack of quantitative evaluation further exacerbates this issue: nearly all experiments, even those with associated plots, are characterized qualitatively and without reference the performance of related methods.\n\n- In addition to addressing the limitations above, I would encourage the authors to consider the use of experiments in simulation to thoroughly and quantitatively investigate the convergence/bias/variance of the gradient model w.r.t. #DoF of the robot, length of the trajectory, voxelization, # sampled trajectories, perturbation sampling method, and robot reliability/reproducibility\n\n*Additional feedback*\n- spelling errors throughout; please check thoroughly\n- the captions/labels/etc. in most figures is far too small to read in a printed copy of the paper\n- What is the intuition for the \"empirical distribution p_e(T|\\pi) = ...\" on page 2? Is it counting the exact matches between the trajectory T and the M observed trajectories? (This may be more clear in the context of voxelization introduced later.)\n- Figure 3: what are the units for \\gamma? what is the time step?\n- many of the figures are out of order w.r.t. their introduction in the text", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1xKZJ6f9B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for control by estimating the gradient of trajectories w.r.t. the policy parameters by fitting a GP to a set of noisy trajectories executing the same controller. This is opposed to the majority of current RL methods that either learn a forward model or learn a policy. They argue that learning this gradient is a middle step between model-based and model-free RL. The method is shown to estimate gradients on simple policies (linear and nonlinear open-loop controllers, and a linear PD controller) for a free-space reaching robot, and update a controller to add a trajectory constraint to pass an intermediate state.\n\nThe paper does show that they can learn these derivatives on controllers from data, which is a cool proof of concept. The method to estimate gradients by “shaking” in a probabilistic way by fitting a GP to noisy trajectories is clever and interesting. But there are a few reasons why I believe this work is not ready for publication.\n\nThe paper only considers free-space reaching as a task, which is not a difficult problem as it does not have contacts. The policies considered are also very simple: an affine open-loop controller (U = Wt + B with 6 parameters), a simple nonlinear open-loop controller (U = Asin(wt) with 2 parameters) and a PD controller with 2 parameters. The motivation is not too convincing without showing some results on hard tasks: model-based RL methods work great in this setting, and are very likely to outperform the method proposed in the paper. The motivation for the proposed method avoids explicit model learning which is a similar motivation as model-free methods, so the paper should at least show that it works as a proof of concept in settings where model-free learning has some advantages, eg. environments with contacts. The paper should probably also compare to existing methods in those settings, although I understand that it might not outperform existing methods.\n\nThe results in section 4.4 which is the result of using the model to plan really shows that using the learned model to update the policy is probably not straightforward. The parameters of the PD controller that go from x_0 to x* are updated to pass a waypoint x*_t using the learned model. But in practice what this is basically doing is changing k_p to introduce a large, possibly inefficient deviation in the path from x_0 to x* that hits x*_t at time t. Directly planning for a path between x_0 to x*_t and then x*_t to x* would probably give a much cleaner path.\n\nAt a high level, the proposed method is likely to be difficult to apply on real problems because estimating the gradient of T w.r.t. pi is probably just much noisier than estimating the forward model directly, which is already a significant challenge. Perhaps one useful experiment is to somehow explicitly show how these two methods compare (eg. measure the variance of trajectory predictions of this method vs rolling out a learned forward model repeatedly).\n\nComments:\n\nEquation 11 and 12 do not make sense/do not use standard notation. I suggest defining n (as a signal or a value, it is not clear at the moment) and defining a new output signal y_t instead of the x_t <- … notation. In particular, the way equation 11 is written seems to say the output x_t is a value-shifted version of the input x_t, NOT a time-shifted one.\n\nThe preliminaries section 1.1 does not discuss environment dynamics. This is significant because the paper seems to assume deterministic dynamics but this is never explicitly stated.\n\nVoxelization as a solution to spatial noise is a bit surprising because discretizing the space throws away local gradient information, which seems valuable to the method. It would be good to understand the effect of this design decision better with an ablation.\n\nMinor comments:\nPage 9: \n- constrain -> constraint\n- Assuem -> Assume\n- such controller -> such a controller", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1ggQEC6tS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses a very good question - can we do better in terms of model learning, so that we can find the much sought after middle ground between model free and model based RL. In particular, the authors ask, can we find a way to learn a model that is reward/task independent, so that a new task can be equally well handled. This is timely and the general thrust of the thinking, in terms of learning from perturbation around trajectories, is good but I am not sure the proposed methods are sufficiently well developed to merit publication. I am also concerned that the authors do not consider numerous issues with the setup that are fairly well understood as issues for system identification.\n\nThe main idea, as laid out in §1.1, is to observe that the parameter update depends mainly on the way a small perturbation in parameters is reflected as a variation in the optimal trajectory (by asking for the probability of a trajectory, this variation becomes the probability of a nearby trajectory). The authors then approach the approximation of this in terms of a discrete finite differences estimate. There are some extensions, such as using a local GP model instead of a local linear model and consideration of ways in which the system might not be exactly repeatable given initial states. These are all proper questions but there are many more important unanswered ones:\n\n1. Starting with where the model setup begins, it is not clear why a complex nonlinear dynamical system, i.e., the typical multi-jointed robot taken as a dynamical system (so, not just kinematics and quasi-static movements), can be sufficiently well approximated using a discretised finite point set that is used at the start of §2 - how does one find the correct T, the correct step size, how does one change these for the local nature of the dynamics (some places might be smoother than others, in phase space), etc.? Even more importantly, are we assuming we know the proper state space ahead of time so that there is no history dependence due to unobserved variables?\n\n2. As such, the authors are proposing to perform closed-loop system identification in a completely data-driven manner. It is well known that this is hard because in the absence of suitable excitation, not all necessary modes in the dynamics will be observed. The only controlled example considered, in §4.3, and subsequent discussion about 'zero-shot' generalisation is getting at this. However, neither at the conceptual level nor in terms of the detailed experiment do I see a good account of what allows this approach to learn all aspects of the dynamics of the system from just small perturbations around a closed loop trajectory.\n\n3. In light of all this, I find the evaluation really weak. Some experiments I would have liked to have seen include - (i) a control experiment based on a standard multi-link arm to show how bad the issue of model mis-match is for the task being considered (I suspect, not much), (ii) experiments with local linearizations, and perhaps piecewise local linearizations, to show how much innovation is needed or is being achieved by the proposed advances, (iii) for us to be talking about 'zero shot' generalisation and the like, more sophisticated tasks beyond merely changing the reaching point (as I say before, it is not even clear that a good PID controller with a roughly plausible linearization is not sufficient to achieve similar effects, and certainly there is a plethora of more sophisticated baselines one could have drawn upon).\n\n4. Some of the discussion comes across as a bit naive, e.g., we have a lemma 3 whose proof is simply a geometric argument about cubes without sufficient consideration of properties of dynamics. I don't doubt the result but in the way it is presented here, it seems shoddy.\n\nAlso, some smaller questions not properly explained:\na. How do you know which kernels for good for the GP in equations 9-10?\nb.  Why should we expect the correlation procedure in §3.0.1 to always work without aliasing and what is the way to get at the suitable domain?\n\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the reviewers generally appreciated the idea behind the method in the paper, there was considerable concern about the experimental evaluation, which did not provide a convincing demonstration that the method works in interesting and relevant problem settings, and did not compare adequately to alternative approach. As such, I believe this paper is not quite ready for publication in its current form.
Decision: Reject