# Reinforcement Learning with Dynamic Movement Primitives for Obstacle Avoidance
## 3 Reinforcement Learning for Obstacle Avoidance  
### Reinforcement Learning: The PI2 Algorithm  
Autonomous learning systems are generally used in the field of control, and reinforcement learning is one of their frameworks [22]. PI2 (policy improvement with path integrals) is a scalable RL method for high dimensional continuous state systems. Moreover, PI2 is a sampling-based and model-free learning method, and algorithm parameter adjustment is not required. Therefore, we can fully focus on the cost function design in the process of applying PI2. The most important challenge is the cost function design in the application of RL algorithms including PI2. For PI2, the cost function is generally defined as  
\[J(\mathbf{\tau}_{i})=\phi_{t_{N}}+\int_{t_{i}}^{t_{N}}{(r_{t}+\mathbf{u}_{t}^{T} \mathbf{R}\mathbf{u}_{t})d_{t}}, \tag{8}\]  
where \(J\) is the a finite horizon cost function for a trajectory \(\mathbf{\tau}_{i}\); \(\phi_{t_{N}}\) denotes the terminal cost at time \(t_{N}\); \(r_{t}+\mathbf{u}_{t}^{T}\mathbf{R}\mathbf{u}_{t}\) is the immediate cost at time \(t\), and here, \(r_{t}\) is an arbitrary state-dependent cost function, \(\mathbf{u}\) is a policy parameter vector determined by \(\mathbf{\theta}\) and \(\lambda\), and \(\mathbf{R}\) is a positive definite weight matrix. The goal of applying PI2 is to find the policy \(\mathbf{u}_{t}\) that minimizes the cost function  
\[V_{t}=\min_{\mathbf{u}_{t}:t_{N}}E_{\mathbf{\tau}_{i}}[J(\mathbf{\tau}_{i})], \tag{9}\]  
where the expectation \(E_{\mathbf{\tau}_{i}}[. ]\) is taken over all possible trajectories when the policy \(\mathbf{u}\) is used. In the process of applying the policy improvement method, we minimize the cost function through an iterative process of exploration and parameter updating. The principles of stochastic optimal control can be used to solve the PI2, and the details are discussed in [18]. The PI2 algorithm needs to be applied to the controlled dynamics systems with parameterized policy  
\[\dot{\mathbf{x}}_{t}=\mathbf{f}_{t}+\mathbf{G}_{t}(\mathbf{u}_{t}+\mathbf{\epsilon}_{ t}), \tag{10}\]  
where \(\mathbf{x}_{t}\) is the system state; \(\mathbf{f}_{t}\) is the passive dynamics, \(\mathbf{G}_{t}\) is the control matrix, and they are both related to the system state; \(\mathbf{u}_{t}\) is the control input with Gaussian noise \(\mathbf{\epsilon}_{t}\), The variance of the Gaussian noise is \(\mathbf{\Sigma}\). The transformation system of DMP falls into one kind of this control system [22]. Figure 2: Obstacle avoidance behavior with different values of \(\lambda\). shape. According to (5) and (6), the transformation system with obstacle avoidance can be written as  
\[\tau\dot{\mathbf{v}}=\mathbf{K}\big{(}\mathbf{x}_{g}-\mathbf{x}\big{)}-\mathbf{D} \mathbf{v}+\big{(}\mathbf{x}_{g}-\mathbf{x}_{0}\big{)}\mathbf{f}(s)\boldsymbol{ \theta}+\lambda\mathbf{R}\mathbf{v}\alpha\exp(-\beta\alpha). \tag{19}\]  
Since the PI2 algorithm is only a special case of optimal control solution, it can be applied to control systems with parameterized control policy [17]. By using the PI2 algorithm, the transformation system (19) can be parameterized  
\[\tau\dot{\mathbf{v}}=\mathbf{K}\big{(}\mathbf{x}_{g}-\mathbf{x}\big{)}- \mathbf{D}\mathbf{v}+\big{(}\mathbf{x}_{g}-\mathbf{x}_{0}\big{)}\mathbf{f}(s) \Big{(}\boldsymbol{\theta}+\boldsymbol{\varepsilon^{\theta}}\Big{)}+\Big{(} \lambda+\varepsilon^{\lambda}\Big{)}\mathbf{R}\mathbf{v}\alpha\exp(-\beta \alpha), \tag{20}\]  
where \(\varepsilon^{\theta}\) is the noise which is added to explore the movement shape and \(\varepsilon^{\lambda}\) is added to explore the strength of the potential. In addition, \(\boldsymbol{\varepsilon^{\theta}}\) and \(\varepsilon^{\lambda}\) are generated by sampling from Gaussian functions, and their variances are \(\Sigma^{\theta}\) and \(\Sigma^{\lambda}\), respectively. Simultaneous optimization of these two parameters belongs to the category of hierarchical reinforcement learning [20].