# Reinforcement Learning with Dynamic Movement Primitives for Obstacle Avoidance
## 2 Obstacle Avoidance for Dynamic Movement Primitives  
Dynamic motion primitive is a trajectory learning method that can modify its ongoing control strategy with a reactive strategy, so it can be used for obstacle avoidance. Now, we briefly review the formulation of DMPS and how to accomplish obstacle avoidance with DMPs. The movement trajectory can be generated by using DMPs. The differential equations of DMPs are inspired from a modified linear spring-damper system with an external forcing term [21]:  
\[\tau\dot{\mathbf{v}}=\mathbf{K}\big{(}\mathbf{x}_{g}-\mathbf{x}\big{)}- \mathbf{D}\mathbf{v}+\big{(}\mathbf{x}_{g}-\mathbf{x}_{0}\big{)}\mathbf{f}(t) \boldsymbol{\theta}, \tag{1}\]  
\[\tau\dot{x}=\
u. \tag{2}\]  
where \(\mathbf{x}\) and \(\mathbf{v}\) are, respectively, the displacement and velocity; \(\mathbf{x}_{g}\) is the goal point of a movement while \(\mathbf{x}_{0}\) is the starting point; \(\mathbf{K}\) is the stiffness matrix corresponding to the stiffness of the system while \(\mathbf{D}\) is the damping matrix; \(\tau\) is a constant scaling factor to determine the movement period. The last term on the right-hand side of (1) is modulatable and parameter vector \(\boldsymbol{\theta}\) can be learned to generate arbitrarily movements. \(\mathbf{f}\) is Gaussian basis function, and it is defined as  
\[\left(\mathbf{f}(t)\right)_{j}=\frac{w_{j,t}s_{t}}{\sum_{k=1}^{p}w_{k,t}}. \tag{3}\]  
In the definition equation for \(\mathbf{f}(s)\), \(w_{j}=\exp\left(-0.5h_{j}\big{(}s_{j}-c_{j}\big{)}^{2}\right)\) is Gaussian function with parameters \(h_{j}\) and \(c_{j}\) corresponding to the center and width. \(\mathbf{f}(s)\) depends explicitly on a phase variable \(s\) instead of time. \(s\) is defined as  
\[\tau s=-\alpha s, \tag{4}\]  
where \(\alpha\) is a preset constant. (4) can be called a 'canonical system'. The state variable s with initial value 1 converges to 0 during the moving duration. To achieve the avoidance behaviors, a repellent acceleration term \(\boldsymbol{\varphi}(\mathbf{x},\mathbf{v})\) is added to the transformation system (1)  
\[\tau\dot{\mathbf{v}}=\mathbf{K}\big{(}\mathbf{x}_{g}-\mathbf{x}\big{)}- \mathbf{D}\mathbf{v}+\big{(}\mathbf{x}_{g}-\mathbf{x}_{0}\big{)}\mathbf{f}(s )\boldsymbol{\theta}+\boldsymbol{\varphi}(\mathbf{x},\mathbf{v}). \tag{5}\]  
For the additional term, one of the most commonly used forms is to model human obstacle avoidance behavior with a differential equation. Here, we will leave aside the concrete dimensions while only constructing a general form. It can be extended to high orlow dimensional space depending on the actual tasks. The differential equation is written as [5]  
\[\boldsymbol{\varphi}(\mathbf{x},\mathbf{v})=-\
abla_{\mathbf{x}}\mathbf{U}( \mathbf{x},\mathbf{v})=\lambda\mathbf{R}\mathbf{v}\alpha\exp(-\beta\alpha), \tag{6}\]  
where \(\mathbf{U}\) is a potential depending on position \(\mathbf{x}\) and velocity \(\mathbf{v}\); \(\beta\) and \(\lambda\) are constants, \(\lambda\) is the potential field intensity factor, and it can determine the strength of the repellent term in 2D or 3D space; \(\mathbf{R}\) is a rotation matrix of angle \(\frac{\pi}{2}\) with the axis \(\mathbf{r}=(\mathbf{o}-\mathbf{x})\times\mathbf{v}\), where \(\mathbf{o}\) denotes the obstacle position and \(\times\) denotes the cross product; \(\alpha\) is a steering angle, and it is defined as [6] (shown in Figure 1). \[\alpha=\arccos\left(\frac{(\mathbf{o}-\mathbf{x})^{\mathsf{T}}\mathbf{v}}{ \|\mathbf{o}-\mathbf{x}\|\|\mathbf{v}\|}\right), \tag{7}\]  
where \(\|\|\) is the Euclidean norm. The value of \(\alpha\) is always positive. In this work, we only focus on this formulation of the repulsive term. Compared with other approaches, the formulation can guarantee convergence to the goal position, even if the effects of obstacles persist during the whole movement duration. Because the variables \(\mathbf{x}\), \(\mathbf{v}\) and \(\mathbf{o}\) are all available, obstacle-avoidance movements are also easily obtained by using these extended DMPs.