Learning by shaking: Computing policy gradients by physical forward-propagation

1 INTRODUCTION . Traditional reinforcement learning crucially relies on reward ( Sutton & Barto , 2018 ) . However , reward binds the agent to a certain task for which the reward represents success . Aligned with the recent surge of interest in unsupervised methods in reinforcement learning ( Baranes & Oudeyer , 2013 ; Bellemare et al. , 2016 ; Gregor et al. , 2016 ; Hausman et al. , 2018 ; Houthooft et al. , 2016 ) and previously proposed ideas ( Schmidhuber , 1991a ; 2010 ) , we argue that there exist properties of a dynamical system which are not tied to any particular task , yet highly useful , and their knowledge can help solve other tasks more efficiently . This work focuses on the sensitivity of the produced trajectories of the system with respect to the policy so called Physical Derivatives . The term physical comes from the fact that it uses the physics of the system rather than any idealized model . We learn a map from the directions in which policy parameters change to the directions in which every state of the trajectory changes . In general , our algorithm learns the Jacobian matrix of the system at every time step through the trajectory . The training phase consists of physically calculating directional derivatives by the finite difference after applying perturbed versions of a nominal policy ( a.k.a . controller ) . Perturbing the parameters of the controller is the reason for naming our method shaking . The test phase uses these directional derivatives to compute derivatives along unseen directions . Due to the difficulty of computing the Jacobian matrix by the finite difference in higher dimensions , we use random controllers joint with probabilistic learning methods to obtain a robust estimate of the Jacobian matrix at each instant of time along a trajectory . We are capable of this generalization to unseen perturbations because the trajectories of physical systems live on an intrinsic low-dimensional manifold and change slowly with the small changes in the parameters of the system ( Koopman , 1931 ) . This assumption holds as long as the system is not chaotic or close to a bifurcation condition ( Khalil , 2002 ) . 1.1 PRELIMINARIES . A reward function describes how close the agent is to the solution of the target task . In the absence of the reward , the agent will be given no means to find its way towards the solution . Let x ∈ X ⊆ Rd be a d-dimensional state vector that fully describes the environment with which the agent interacts . At each state , the agent is allowed to take action u ∈ U ⊆ Rq from a q-dimensional action space via a parameterised policy function u = π ( x ; θ ) . The agent will be rewarded r ( x , u ) by the function r : X × U → R when it takes action u at state x . The goal of learning is to update θ such that some desired target is achieved . The target can be anything as long as a concrete reward function is associated with it . In stochastic cases , return R : Π ( Θ ) → R is defined as a cumulative future discounted reward whose expectation is often of main interest . For parametric policies , the space of feasible parameters Θ has a one-to-one correspondence to the policy space Π . The agent who takes on the policy π from state x0 produces the trajectory T ∈ T where T is the space of possible trajectories . For a return function R : T→ R , the expected return becomes a function of the policy as J ( πθ ) = ET { R ( T ) } where the expectation is taken with respect to the probability distribution P ( T |πθ ) . There exist two major classes of approaches in reinforcement learning : value-based methods and value-free methods . In the first class , a surrogate function is defined to approximate the value of either a state V ( x ) or a state-action pair Q ( x , u ) . The policy is updated such that the agent tends towards states with higher values . The value-free methods update the policy directly without any need for an auxiliary function such as V or Q . This paper mainly concerns the second class . The policy parameters are updated as θt+1 = θt + α ∂J ( πθ ) ∂θ ∣∣∣∣ θ=θt ( 1 ) and the gradient ∂J ( πθ ) /∂θ is written as ∂J ( πθ ) ∂θ = ∫ T ∂p ( T |πθ ) ∂θ R ( T ) dT ( 2 ) which is normally difficult to compute in practice . As can be seen in eq . ( 2 ) , the integrand of the r.h.s . consists of two terms . The second term R ( T ) is the return which is defined according to the target task . Hence , this term is task-dependent . The first term ∂p ( T |πθ ) /∂θ though shows how the trajectories change with respect to a change in the policy . Notice that there is no notion of reward or any task-dependent quantities in this term . For an empirical distribution pe ( T |π ) = 1 M ∑M i=1 δ ( T − T ( i ) ) , the dependence of partial derivative of the distribtion of T on the partial derivative of T can be explicitely derived as ∂pe ( T |πθ ) ∂θ = 1 M M∑ i=1 u1 ( T − T ( i ) ) ∂T ∂θ ( 3 ) where u1 is the unit doublet function ( derivative of the Dirac delta function ) . This examplary distribution makes it clear that the change in the distribution of trajetories relates to the change of the trajectories themselves . As an unsupervised object , ∂T /∂θ is of main interest in this paper . 1.2 PHYSICAL DERIVATIVE . In this paper , we investigate the feasibility of learning a less explored unsupervised quantity , the so called Physical Derivative which is computed directly from the physical system . In abstract terms , we perturb the policy and learn the effect of its perturbation on the resulting trajectory . The difference from traditional RL whose algorithms are based on eq . ( 1 ) is the absence of a specified reward function . Instead , we generate samples from ∂p ( T |πθ ) /∂θ of eq . ( 2 ) that makes it possible to compute ∂J ( πθ ) /∂θ for an arbitrary return function R. If the exact model of the system is known , control theory has a full set of tools to intervene in the system with stability and performance guarantees . When the system is unknown , one could identify the system as a preliminary step followed by a normal control synthesis process from control theory ( Ljung , 2001 ) . Otherwise , the model and the policy can be learned together in a model-based RL ( Sutton , 1996 ) or in some cases adaptive control ( Sastry & Bodson , 2011 ) . We argue that learning physical derivatives is a middle ground . It is not model-based in the sense that it does not assume knowing the exact model of the system . Rather , it knows how the trajectories of the system change as a result of perturbing the policy parameters . This differential information of the system has applications in many downstream tasks . This work focuses on the concept and introduction of physical derivatives and direct applications would go significantly beyond the scope of this work . Few potential applications are discussed with more details in appendix C. Our contributions— In summary , the key contributions of the current paper are as follows : • A method to generate training pairs to learn the map from the policy perturbations to the resulting changes in the trajectories . • Learning the above map as a probabilistic function and showing that it generalizes to unseen perturbations in the policy . • Use the inverse of the above map to perturb the policy in the desired direction to achieve certain goals without conventional RL methods . • Use a physical custom-built robotic platform to test the method and propose solutions to deal with the inherent issues of the physical system to ensure the practicality of the method ( see fig . 1 for images of the platform and and appendix A for technical details ) . • The supplementary materials for the paper , including code and the videos of the robot in action can be found in https : //sites.google.com/view/ physicalderivatives/ 2 METHOD . In this section , we describe our pipeline to estimate the physical derivatives and our proposed solutions to the inevitable challenges that are likely to occur while working with a real physical robot . We are interested in ∂T /∂θ which denotes how a small change in the parameters θ of the controller results in a different trajectory produced by the system . We normally consider a finite period of time [ 0 , T ] and the trajectory is an ordered list of states T = [ x0 , x1 , . . . , xT ] where the subscript shows the time step . Therefore , having ∂T /∂θ is equivalent with having ∂xt/∂θ for every t ∈ { 1 , . . . , T } . Notice that the initial state x0 is chosen by us . Hence we can see it either as a constant or as a changeable parameter in θ . We kept it fixed in our experiments . Assume xt ∈ Rd and θ ∈ Rm . Hence , ∇θxt = ∂xt/∂θ ∈ Rd×m where the tth row of this matrix is ∇θxit = ( ∂xit/∂θ ) T ∈ Rm showing how the ith dimension of the state vector changes in response to a perturbation in θ . The directional derivative of xit in the direction δθ is defined as ∇δθθ xit = 〈∇θxit , δθ |δθ| 〉 . ( 4 ) If ( 4 ) is available form linearly independent and orthonormal directions , { δθ ( 1 ) , δθ ( 2 ) , . . . , δθ ( m ) } , the directional derivative along an arbitrary δθ can be approximated by ∇δθθ xit = m∑ j=1 cj〈∇θxit , δθ ( j ) 〉 ( 5 ) where cj = 〈δθ , δθ ( j ) 〉 is the coordinates of the desired direction in the coordinate system formed by the orthonormal bases . In practice , m directions δθ ( j ) can be randomly chosen or can be along some pre-defined axes of the coordinate system . To compute 〈∇θxit , δθ ( j ) 〉 , the nominal policy parameters θ are perturbed by δθ ( j ) as θ ( j ) ← θ + δθ ( j ) and the derivative is computed as 〈∇θxit , δθ ( j ) 〉 = lim h→0 xit ( θ + hδθ ( j ) ) − xit ( θ ) h . ( 6 ) This quantity is often approximated by finite difference where h takes a small nonzero value . By perturbing the parameters θ along m orthonormal directions δθ ( j ) and computing the approximate directional derivative by ( 6 ) , ∇δθθ xit can be computed along every arbitrary direction δθ , meaning that , we can compute∇θxit by evaluating it along any direction which is the aim of this paper . ✓1 < latexit sha1_base64= '' b2Ff/oUFJw0eznxXS1RygRK2bZk= '' > AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0mqoMeiF48V7Ae0oWy2m3bpZhN3J0IJ/RNePCji1b/jzX/jts1BWx8MPN6bYWZekEhh0HW/ncLa+sbmVnG7tLO7t39QPjxqmTjVjDdZLGPdCajhUijeRIGSdxLNaRRI3g7GtzO//cS1EbF6wEnC/YgOlQgFo2ilTg9HHGnf65crbtWdg6wSLycVyNHol796g5ilEVfIJDWm67kJ+hnVKJjk01IvNTyhbEyHvGupohE3fja/d0rOrDIgYaxtKSRz9fdERiNjJlFgOyOKI7PszcT/vG6K4bWfCZWkyBVbLApTSTAms+fJQGjOUE4soUwLeythI6opQxtRyYbgLb+8Slq1qndRrd1fVuo3eRxFOIFTOAcPrqAOd9CAJjCQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QPOtY/Q < /latexit > ✓2 < latexit sha1_base64= '' oe3DagNbCs6bjj10ybLfZH9d5SY= '' > AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0mqoMeiF48V7Ae0oWy2m3bpZhN3J0IJ/RNePCji1b/jzX/jts1BWx8MPN6bYWZekEhh0HW/ncLa+sbmVnG7tLO7t39QPjxqmTjVjDdZLGPdCajhUijeRIGSdxLNaRRI3g7GtzO//cS1EbF6wEnC/YgOlQgFo2ilTg9HHGm/1i9X3Ko7B1klXk4qkKPRL3/1BjFLI66QSWpM13MT9DOqUTDJp6VeanhC2ZgOeddSRSNu/Gx+75ScWWVAwljbUkjm6u+JjEbGTKLAdkYUR2bZm4n/ed0Uw2s/EypJkSu2WBSmkmBMZs+TgdCcoZxYQpkW9lbCRlRThjaikg3BW355lbRqVe+iWru/rNRv8jiKcAKncA4eXEEd7qABTWAg4Rle4c15dF6cd+dj0Vpw8plj+APn8wfQOY/R < /latexit > ✓1 < latexit sha1_base64= '' b2Ff/oUFJw0eznxXS1RygRK2bZk= '' > AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0mqoMeiF48V7Ae0oWy2m3bpZhN3J0IJ/RNePCji1b/jzX/jts1BWx8MPN6bYWZekEhh0HW/ncLa+sbmVnG7tLO7t39QPjxqmTjVjDdZLGPdCajhUijeRIGSdxLNaRRI3g7GtzO//cS1EbF6wEnC/YgOlQgFo2ilTg9HHGnf65crbtWdg6wSLycVyNHol796g5ilEVfIJDWm67kJ+hnVKJjk01IvNTyhbEyHvGupohE3fja/d0rOrDIgYaxtKSRz9fdERiNjJlFgOyOKI7PszcT/vG6K4bWfCZWkyBVbLApTSTAms+fJQGjOUE4soUwLeythI6opQxtRyYbgLb+8Slq1qndRrd1fVuo3eRxFOIFTOAcPrqAOd9CAJjCQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QPOtY/Q < /latexit > ✓2 < latexit sha1_base64= '' oe3DagNbCs6bjj10ybLfZH9d5SY= '' > AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0mqoMeiF48V7Ae0oWy2m3bpZhN3J0IJ/RNePCji1b/jzX/jts1BWx8MPN6bYWZekEhh0HW/ncLa+sbmVnG7tLO7t39QPjxqmTjVjDdZLGPdCajhUijeRIGSdxLNaRRI3g7GtzO//cS1EbF6wEnC/YgOlQgFo2ilTg9HHGm/1i9X3Ko7B1klXk4qkKPRL3/1BjFLI66QSWpM13MT9DOqUTDJp6VeanhC2ZgOeddSRSNu/Gx+75ScWWVAwljbUkjm6u+JjEbGTKLAdkYUR2bZm4n/ed0Uw2s/EypJkSu2WBSmkmBMZs+TgdCcoZxYQpkW9lbCRlRThjaikg3BW355lbRqVe+iWru/rNRv8jiKcAKncA4eXEEd7qABTWAg4Rle4c15dF6cd+dj0Vpw8plj+APn8wfQOY/R < /latexit > Figure 2 : Gaussian ( left ) and uniform ( right ) shaking examples . In the matrix form for x ∈ Rd , we can compute ∇δθ ( j ) θ x = [ ∇δθ ( j ) θ x1 , ∇δθ ( j ) θ x1 , . . . , ∇δθ ( j ) θ xd ] T in a single run by computing ( 6 ) for all d dimensions of the states . Let ’ s define ∆θx , [ ∇δθ ( 1 ) θ x , ∇δθ ( 2 ) θ x , . . . , ∇δθ ( m ) θ x ] ( 7 ) where ∆θx ∈ Rd×m and let Λ = [ δθ ( 1 ) , δθ ( 2 ) , . . . , δθ ( m ) ] . Therefore , if ∆δθθ x shows the directional derivative of x along δθ , we can write it as : ∇δθθ x = ∆θx ( ΛTδθ ) ( 8 ) which is only a vectoral representation of eq . ( 4 ) . Even though the linear formula of eq . ( 8 ) requires only m directional derivatives , it has two major downsides . First , it does not give a clear way to incorporate more than m training directional physical derivatives . Second , the linear approximation remains valid only for very small δθ . We propose Gaussian Process ( GP ) as a nonlinear probabilistic function approximator ( Rasmussen , 2003 ) to capture the maps ĝt defined as ĝt : Θ→ X ( 9 ) ĝt ( δθ ) = δx ( 10 ) where subscript t shows the function that maps δθ to the change of the states δxt at time step t. We considered distinct functions for every time step . Taking into account the commonality among the function approximators corresponding to different time steps is deferred to future research . Learning this map requires training data that comes from an initial data collection phase called shaking . Shaking refers to perturbing parameters of the controller to obtain the set of trajectories produced by the perturbed controllers . The perturbation can be either regular or stochastic . Stochastic perturbations have the advantage over regular perturbations that the agent does not need to be worried about perturbing the parameters in a particular direction . Besides , in some cases , perturbing the parameters of the policy in certain directions is infeasible . We propose two methods of shaking called Gaussian and Uniform shaking . Gaussian shaking— Likely values of θ create nominal policies encoded by { θ ( 1 ) , θ ( 2 ) , . . . , θ ( m ) } . We put Gaussian distributions centered at each of the nominal values resulting in a mixture of Gaussians . To reduce the hyper-parameters , we assume the variances of the Gaussians are themselves sampled from an exponential distribution making sure they all take positive values ( See fig . 2 left ) . Here , we manually choose a reasonable value for the rate parameter of the exponential distribution . Doing inference on the hyper-parameters of the sampling distributions can be a topic for future research especially in active learning for a more clever less costly sampling stratgey . Uniform shaking— In this setting , the state space of the changeable parameters of the policy is discretized and a uniform distribution is assumed around each value of this grid with some overlapping with the neighboring cells ( See fig . 2 right ) . We show the effect of each of these sampling methods later in section 4 . We observed that the results are less sensitive to the hyper-parameters of the uniform sampling than Gaussian sampling . A carelessly chosen rate for the exponential distribution that generates the variances of the Gaussians in Gaussian sampling can result in too local or global sampling that gives rise to a large variance or bias in the estimated gradients .

Summary: This paper presents a method for control by estimating the gradient of trajectories w.r.t. the policy parameters by fitting a GP to a set of noisy trajectories executing the same controller. This is opposed to the majority of current RL methods that either learn a forward model or learn a policy. They argue that learning this gradient is a middle step between model-based and model-free RL. The method is shown to estimate gradients on simple policies (linear and nonlinear open-loop controllers, and a linear PD controller) for a free-space reaching robot, and update a controller to add a trajectory constraint to pass an intermediate state.