# Robust Quadrotor Control through Reinforcement Learning with Disturbance Compensation
## 2 Preliminary Information
### Reinforcement Learning  
The standard RL framework comprises a learning agent interacting with the environment according to a Markov decision process (MDP). A state transition has a Markov  
Figure 1: Body-fixed frame of the quadrotor and motor placement. property if the probability of this transition is independent of its history. The MDP involves solving decision problems with the Markov property, and RL theories are based on the MDP. The standard MDP is defined by the tuple \((\mathcal{S},\mathcal{A},P^{a}_{\text{sg}^{\prime}},r,\rho_{0},\gamma)\), where \(\mathcal{S}\) is the state space, \(\mathcal{A}\) is the action space, \(P^{a}_{\text{sg}^{\prime}}:\mathcal{S}\times\mathcal{A}\times\mathcal{S} \rightarrow\mathbb{R}+\) is the transition probability density of the environment, \(r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}\) is the reward function, \(\rho_{0}:\mathcal{S}\rightarrow\mathbb{R}^{+}\) is the distribution of the initial state \(s_{0}\), and \(\gamma\in(0,1)\) is the discount factor. In modern deep RL conducted using neural networks, the agent selects an action at each time step according to the policy \(\pi(a|s;\theta)=Pr(a|s;\theta)\), where \(\theta\in\mathbb{R}^{N_{0}}\) is the weight of the neural network. The goal of the MDP is to find a policy \(\pi(a\in\mathcal{A}|s)\) that can maximize the cumulative discounted reward. \[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t}). \tag{5}\]  
A state-dependent value function \(V^{\pi}\) that measures the expected discounted reward with respect to \(\pi\) can be defined as follows:  
\[V^{\pi}(s)=\mathbb{E}\left[\sum_{l\geq 0}\gamma^{l}r_{t+l}\bigg{|}s_{t}=s,\pi \right]. \tag{6}\]  
The state-action-dependent value function can be defined as follows:  
\[Q^{\pi}(s,a)=\mathbb{E}\left[\sum_{l\geq 0}\gamma^{l}r_{t+l}\bigg{|}s_{t}=s,a_{t }=a,\pi\right]. \tag{7}\]  
The advantage function can be defined as follows:  
\[A^{\pi}(s,a)=Q^{\pi}(s,a)-V^{\pi}(s), \tag{8}\]  
where \(A^{\pi}\) is the difference between the expected value when selecting some specific action \(a\). The advantage function can be used to determine whether the selected action is suitable with respect to policy \(\pi\). Many basic RL algorithms, such as the policy gradient method [25], off-policy actor-critic algorithm [26], and trust region policy optimization [27] can be used to optimize a policy. To maximize the expected reward function \(V^{\pi}(s)\), the neural network parameterized policy \(\pi=\pi(a|s;\theta)\) is adjusted as follows:  
\[\theta\leftarrow\theta+\alpha\sum_{s\in\mathcal{S},a\in\mathcal{A}}\rho^{\pi} (s)Q^{\pi}(s,a)\frac{\partial\pi(a|s)}{\partial\theta}, \tag{9}\]  
where  
\[\rho^{\pi}(s)=\sum_{t\geq 0}\gamma^{t}Pr(s_{t}=s|t,a\sim\pi) \tag{10}\]  
is the state occurrence probability and \(\alpha>0\) is the size of the learning step. Equation (9) is an expression for the policy gradient [28]. By using the state distribution \(\rho\) and state-action value function \(Q\), a policy can be improved without any environmental information. The state distribution \(\rho^{\pi}(s)\) depends on the policy \(\pi\), which indicates that it must be re-estimated when the policy is changed. In [26], the policy gradient was analyzed by replacing the original policy \(\pi\) with another policy \(\mu\); therefore, (9) had the following form in [26]. \[\sum_{s\in\mathcal{S},a\in\mathcal{A}}\rho^{\mu}(s)Q^{\pi}(s,a)\frac{\partial \pi(a|s)}{\partial\theta}. \tag{11}\]  
Equation (11) can still maximize \(V^{\pi}\) with distinct policy gradient strategies. To solve the nonlinear dynamic control problem for a quadrotor, we used the proximal policy optimization algorithm (PPO) [29] and off-policy training method [22] to train the actor and critic functions.