## 4 Model Predictive Control for Power Optimization and DDPG-Based Optimization Scheduling
### DDPG-Based Optimization Scheduling
#### 4.2.1 Exploration Noise and Markov Decision Process  
In order to better explore the unknown environment in the decision-making process, Ornstein-Uhlenbeck (_OUI_) noise, which is suitable for inertial systems, is added [22], with noise decay set to achieve a balance between exploration and exploitation. The specific mathematical expression is as follows:  
\[\begin{array}{l}d\mathcal{N}_{t,OU}=\theta(\mu-\mathcal{N}_{t,OU})dt+\sigma dW _{t}\\ a_{t}=P(s_{t}|\theta^{p})+\mathcal{N}_{t,OU}\end{array} \tag{34}\]  
where \(t\) denotes the time step, \(\mathcal{N}_{t,OU}\) is the \(OU\) noise value, \(\theta\) is the mean-reversion rate, \(\mu\) is the mean, \(\sigma\) is the volatility, and \(W_{t}\) is the Brownian motion. \(a_{t}\) is the action, \(P(s_{t}|\theta^{p})\) is the actor network in DDPG, \(\theta^{p}\) are the network parameters, and \(s_{t}\) is the system state. The temporal correlation of the \(OU\) process allows the system to explore multiple steps in the same direction to accumulate training experience, thereby effectively improving the exploration efficiency of the inertial system.  
Reinforcement learning tasks are typically formulated as a Markov decision process (MDP), defined as a five-tuple (\(\mathbf{S},\mathbf{A},\mathbf{P},\mathbf{R}\), \(\gamma\)), where \(\mathbf{S}\) is the state space, \(\mathbf{A}\) represents the action space, \(\mathbf{P}\) is the transition matrix, \(\mathbf{R}\) indicates the reward, and \(\gamma\) is the discount factor. In this paper, the model-free Deep Deterministic Policy Gradient (DDPG) algorithm is employed, eliminating the need for a transition matrix \(\mathbf{P}\). Based on the characteristics of the hydrogen-powered hybrid UAV energy system, the defined objective function, operating constraints, state space, action space, and reward function are as follows:  
(1) Objective function  
The objective function is primarily defined by the systemic operating cost and the state of the energy storage device, ensuring an optimal balance between energy efficiency and operational reliability. It is expressed as follows:  
\[\begin{array}{lcl}J&=&\min\sum\limits_{t=1}^{T}C_{t}^{OM}+\sum\limits_{i=1}k _{i}C_{T}^{S}\\ &=&\min k_{1}\sum\limits_{t=1}^{T}C_{1}(P_{E,t}^{FC})+k_{2}|SOC_{T}-SOC_{0}| \end{array} \tag{35}\]  
where \(C_{1}\) and \(P_{E,t}^{FC}\) are equivalent to hydrogen consumption of the systemic devices and the power at time \(t\), respectively, \(k_{i}\) and \(k_{2}\) are the weighting coefficients, respectively.  
Figure 5: The DDPG flowchart.  
2. Operating constraints The constraint of electrical balance can be described as follows: \[P_{E,t}^{FC}+P_{E,t}^{B}=P_{E,t}^{L}\] (36) The constraint of PEMFC power is shown as follows: \[P_{E,\text{min}}^{FC}\leq P_{E,t}^{FC}\leq P_{E,\text{max}}^{FC}\] (37) The upper and lower boundary of SOC are constrained as follows: \[SOC_{\text{min}}\leq SOC_{t}\leq SOC_{\text{max}}\] (38) The initial SOC value of battery can be expressed as follows: \[SOC_{0}=0.5\cdot(SOC_{\text{min}}+SOC_{\text{max}})\] (39) 3. State space The state space matrix \(S\) can be expressed in Equation (40): \[S=\left[\sin t,\cos t,P_{E,t}^{L},SOC_{t}\right]^{T}\] (40) where \(t\) indicates the time step, \(P_{E,t}^{L}\) is the electrical load (i.e., motor power) at time \(t\), and \(SOC_{t}\) is the battery state of charge at time \(t\). (4) Action space The state space matrix \(A\) can be expressed in Equation (41): \[\mathbf{A}=\left[P_{E,t}^{FC}\right]^{T}\] (41) where \(P_{E,t}^{FC}\) is the electrical power of the hydrogen fuel cell. (5) Reward function The reward function \(R\) is defined as follows: \[R=-\left(C_{t}^{OM}+\alpha C_{T}^{S}+l_{1}\middle|P_{out,t}^{E}\middle|\right)\] (42) where \(P_{out,t}^{E}\) is defined as the scheduling action based on the policy network at time \(t\), representing the surplus (greater than 0) or deficiency (less than 0) when the power balance cannot be satisfied. \(l_{1}\) is the weight adjusted based on training experience and results.