## 3 Quadcopter Control
### Deep Reinforcement Learning for Online Parameter Estimation and Tuning
\begin{array}{c}\frac{\partial Q(s,a)}{\partial\theta}=\frac{\partial Q(s,a )}{\partial a}\times\frac{\partial a}{\partial\theta}\\ \
abla_{\theta}J(\theta)=\mathbb{E}_{s\sim\rho_{\mu}}\Big{[}\left.\
abla_{ \theta}\mu_{\theta}(s)\times\
abla_{a}Q^{\mu}(s,a)\right|_{a=\mu_{\theta}(s)} \right]\end{array}  \tag{44}\]  
To obtain an unbiased estimate of the Q-value of any action and compute its gradient, it is possible to calculate a function approximator \(Q_{\varphi}(s,a)\), as long as it is compatible, and to minimize the quadratic error with the true Q-values:  
\[\
abla_{\theta}J(\theta)=\mathbb{E}_{s\sim\rho_{\mu}}\Big{[}\left.\
abla_{ \theta}\mu_{\theta}(s)\times\
abla_{a}Q_{\varphi}(s,a)\right|_{a=\mu_{\theta} (s)}\Big{]} \tag{45}\]  
\[J(\varphi)=\mathbb{E}_{s\sim\rho_{\mu}}\Big{[}\Big{(}Q^{\mu}(s,\mu_{\theta}( s))-Q_{\varphi}(s,\mu_{\theta}(s)))^{2}\Big{]} \tag{46}\]  
The goal of the DDPG algorithm was to extend the DPG to incorporate non-linear function approximators. The objective was fulfilled by combining DQN and DPG concepts to create an algorithm working over continuous space. Furthermore, the following elements were added to the base DPG algorithm. * An experience reply memory to store past transitions and learn off-policy. * Target networks to stabilize learning. DDPG updates the parameters of target networks after each update of the trained network using a sliding average (soft update) for both the actor and the critic:  
\[\theta^{Q_{target}} \leftarrow\tau\theta^{Q}+(1-\tau)\theta^{Q_{target}} \tag{47}\] \[\theta^{H_{target}} \leftarrow\tau\theta^{\mu}+(1-\tau)\theta^{H_{target}} \tag{48}\]  
where \(\tau\) is a hyperparameter between 0 and 1. The modified update rule guarantees that the target networks are always lagging behind the trained networks, providing more stability to the learning of Q-values. The key idea borrowed from DPG is the policy gradient for the actor. The critic is learned using regular Q-learning and target networks:  
\[J(\varphi)=\mathbb{E}_{s\sim\rho_{\mu}}\Big{[}\Big{(}r\big{(}s,a,s^{\prime} \big{)}+\gamma Q_{target}\big{(}s^{\prime},\mu_{\theta^{\prime}}\big{(}s^{ \prime}\big{)}\big{)}-Q_{\varphi}(s,a))^{2}\Big{]} \tag{49}\]  
where \(Q_{\varphi^{\prime}}(s^{\prime},\mu_{\theta^{\prime}}(s^{\prime}))\) is the value of the action that is estimated to return the largest total future reward, based on all possible actions that can be made in the next state. In the Equation (49), \(\gamma\) is the discount factor. Noise is added to improve exploration:  
\[a_{t}=\mu_{\theta}(s_{t})+\xi_{noise} \tag{50}\]  
The additive noise is an Ornstein-Uhlenbeck process that generate temporally correlated noise with zero mean. The target value can be computed using the target network:  
\[y_{RLi}=r_{i}+\gamma Q_{target}(s_{i+1},\mu_{target}(s_{i+1}|\theta^{H_{target }})\Big{|}\theta^{Q_{target}}) \tag{51}\]The following loss function is minimized, resulting in an update in the critic. A sampling policy gradient can also be applied to update the actor:  
\[L=\left(\frac{1}{N}\right)\sum\limits_{i}\left(y_{RLi}-Q\Big{(}s_{i},a_{i} \Big{|}\theta^{Q}\Big{)}\right)^{2} \tag{52}\]  
\[\
abla_{\theta^{H}}\int\approx\frac{1}{N}\sum\limits_{i}\
abla_{a}Q\Big{(}s,a \Big{|}\theta^{Q}\Big{)}\Bigg{|}_{s=s_{i}\,a=\mu(s_{i})}\
abla_{\theta^{H}}\mu(s|\theta^{H})\Bigg{|}_{s_{i}}\
abla_{a} \tag{53}\]  
Figure 4 depicts the aforementioned learning process in the DDPG algorithm. The architecture is comprised of two actor-critic networks. The target networks in the architecture stabilized the learning procedure. The noise improves the exploration of the reachable control states. The proposed RL-based control algorithm (Algorithm 1) is described in the following pseudocode. ```
1: Initial policy network \(\mu\) and critic network \(Q\) with weights \(\theta^{H}\)and \(\theta^{Q}\) respectively.