# Reinforcement Learning with Dynamic Movement Primitives for Obstacle Avoidance
## 3 Reinforcement Learning for Obstacle Avoidance
### Reinforcement Learning: The PI2 Algorithm
The parameters \(\theta\) and \(\lambda\) are learned simultaneously on different levels of abstraction. In the learning process, the exploration for the shape of DMP usually occurs in the fixed potential field. This means that the potential update should begin before updating the shape. Because the strength of potential \(\lambda\) remains unchanged during execution, it is not temporal-dependent on the cost. To this end, only the cost-to-go at \(t=0\) is employed to calculate the probability of \(\lambda\). The cost used is the total cost of the trajectory in the last roll-out. No negative interference exists between learning \(\boldsymbol{\theta}\) and \(\lambda\), as they are updated with the equally probability weights because their costs are exactly equal [19; 20]. The PI2 algorithm applied in learning both potential strength and DMP shape is summarized in Algorithm 1. ```
0: Initial potential strength \(\lambda_{0}\); initial DMP shape parameters \(\boldsymbol{\theta}_{0}\); constant \(\beta\); terminal cost \(\phi_{t_{N}}\); immediate cost term \(r_{t}+\boldsymbol{\theta}_{t}^{T}\mathbf{R}\boldsymbol{\theta}_{t}\); variance of noise \(\Sigma^{\theta}\) and \(\Sigma^{\lambda}\); Gaussian basis function from the system dynamics \(\mathbf{g}_{t}\); number of roll-outs per update \(K\)
0: Final potential strength \(\lambda\); final DMP shape parameters \(\boldsymbol{\theta}\)
1:while cost \(J\) not converged do
2:for k = 1 to K do
3:\(\boldsymbol{\tau}_{k,i=0:N}\), \(\boldsymbol{\theta}_{k,i=0:N}\), \(\lambda_{k}=\text{create\_trajectory}\big{(}k,\boldsymbol{\theta},\boldsymbol{ \Sigma^{\theta}},\lambda,\Sigma^{\lambda}\big{)}\)
4: calculate generalized trajectory cost \(s(\boldsymbol{\tau}_{i,k})\) with (16)
5: calculate probability of a trajectory \(p(\boldsymbol{\tau}_{i,k})\) with (15)
6:endfor
7:for i = 0 to N do
8:\(\delta\boldsymbol{\theta}_{t_{i}}=\sum_{k=1}^{K}\big{[}p(\boldsymbol{\tau}_{ i,k})\boldsymbol{M}_{t_{i}k}\boldsymbol{\varepsilon}_{t_{i,k}}^{\boldsymbol{ \theta}}\big{]}\), where \(\mathbf{M}_{t_{i}k}=\frac{\mathbf{R}^{-1}\mathbf{g}_{t_{i}k}\mathbf{g}_{t_{i}k }^{T}}{\mathbf{g}_{t_{i}^{T}}^{T}\mathbf{R}\mathbf{g}_{t_{i}k}}\)
9:\(\big{[}\delta\boldsymbol{\theta}_{i}\big{]}=\frac{\sum_{k=0}^{N-1}\big{(}N-1 \big{)}[\delta\boldsymbol{\theta}_{t_{i}}]_{w}\mu_{t_{i}}}{\sum_{k=0}^{N-1} (N-1)\mu_{t_{i}}}\)
10:\(\boldsymbol{\theta}=\boldsymbol{\theta}+\delta\boldsymbol{\theta}\)
11:endfor
12:\(\mathbf{p}(\boldsymbol{\tau}_{0,k})=\frac{\varepsilon^{-\frac{1}{\lambda}}( \boldsymbol{\tau}_{0,k})}{\sum_{k=1}^{K}\big{[}e^{-\frac{1}{\lambda}\xi^{k}( \boldsymbol{\tau}_{0})}\big{]}}\)
13:\(\delta\lambda=\sum\limits_{k=1}^{K}\big{[}\mathbf{p}(\boldsymbol{\tau}_{0,k}) \boldsymbol{\varepsilon}_{k}^{\lambda}\big{]}\)
14:\(\lambda=\lambda+\delta\lambda\)
15:endwhile
```  
**Algorithm 1** PI2 algorithm for learning potential strength \(\lambda\) and DMP shape parameters \(\boldsymbol{\theta}\). Since the state of a DMP system can be divided into the controlled part and the uncontrolled part, in the meantime, the control transition matrix depends on only one variable of the uncontrolled part [18; 21], and the trajectory cost of DMP can be given as  
\[s(\mathbf{\tau}_{i,k})=\phi_{t_{N},k}+\frac{1}{2}\sum_{j=i}^{N-1}r_{t_{j},k}+\frac{1 }{2}\sum_{j=i+1}^{N-1}\Big{(}\theta+\mathbf{M}_{t_{j},k}\mathbf{\epsilon}_{t_{j},k}^ {\mathbf{\theta}}\Big{)}^{T}\mathbf{R}\Big{(}\theta+\mathbf{M}_{t_{j},k}\mathbf{ \epsilon}_{t_{j},k}^{\mathbf{\theta}}\Big{)}, \tag{21}\]  
where \(\mathbf{M}_{t_{j},k}\) is computable (cf. Algorithm 1). In this way, only local sampling and an iterative procedure need to be considered in the process of updating \(\mathbf{\theta}\).