## 4 Design of Deep Reinforcement Learning Controller for Stabilized Platform
### Design of DDPG_DOB  
In the practical environment, the stabilized platform control system is affected by frictional disturbance, which results in nonlinear characteristics, such as dead-zone nonlinearity and saturation, which lead to steady-state error, oscillation, and hysteresis phenomena within the system. Therefore, this section proposes the integration of a disturbance observer to accurately estimate and compensate for frictional disturbances, thereby eliminating their adverse effects on a stabilized platform.  
The structural configuration of the stabilized platform control system enhanced by the incorporation of a disturbance observer is shown in Figure 5.  
The utilization of a disturbance observer plays a crucial role in estimating and mitigating disturbances. This approach enables the real-time adjustment of the controller's output based on the observed disturbances, subsequently altering the input to the stabilized platform-controlled object. As illustrated in Figure 5, the disturbance observer takes the tool face angle as input and generates an estimated value for external disturbances. Simultaneously, the DDPG controller undergoes training, taking into account the current tool face angle and the tool face angle error. The disparity between the controller's output and the disturbance observer's output serves as the input to the stabilized platform-controlled object. The operational principle of the disturbance observer is elucidated in Figure 6.  
Figure 4: (**a**) Actor network structure and (**b**) Critic network structure.  
Figure 5: Structure diagram of the stabilized platform control system enhanced with a disturbance observer.  
As depicted in Figure 6, the diagram elucidates the following components: \(u\) is the input to the controller; \(G(s)\) symbolizes the transfer function of the controlled object; \(y\) signifies the system's control output; \(G_{n}^{-1}(s)\) denotes the inverse of the nominal model; \(Q(s)\) is the low-pass filter; \(d\) signifies the disturbance; \(\hat{d}\) represents the estimated value of the disturbance \(d\); \(\xi\) accounts for measurement noise; and \(u\prime\) is the control input to the system.  
These components collectively contribute to the control inputs of the system:  
\[u\prime=u-\hat{d}+d, \tag{14}\]  
As delineated in Figure 6, when the controller output signal is denoted as \(U(s)\), the disturbance signal as \(D(s)\), and the measurement noise as \(N(s)\), their corresponding transfer functions are represented as follows:  
\[G_{UY}=\frac{Y(s)}{U(s)}=\frac{G(s)G_{n}(s)}{G_{n}(s)+Q(s)(G(s)-G_{n}(s))}, \tag{15}\]  
\[G_{DY}=\frac{Y(s)}{D(s)}=\frac{G(s)G_{n}(s)(1-Q(s))}{G_{n}(s)+Q(s)(G(s)-G_{n}(s ))}, \tag{16}\]  
\[G_{NY}=\frac{Y(s)}{N(s)}=\frac{G(s)Q(s)}{G_{n}(s)+Q(s)(G(s)-G_{n}(s))}, \tag{17}\]  
Leveraging Equation (14), Figure 6 is streamlined to Figure 7 through equivalent transformation.  
The mathematical expression of the low-pass filter is:  
\[Q(s)=\frac{1+\sum\limits_{k=1}^{N-2}a_{k}{(\tau s)}^{k}}{1+\sum\limits_{k=1}^{ N}a_{k}{(\tau s)}^{k}}, \tag{18}\]  
Figure 6: Principle block diagram of disturbance observer.  
Figure 7: Simplified block diagram of disturbance observer through equivalent transformation.  
where \(N\) signifies the order of the denominator; \(a_{k}=\frac{N!}{k!(N-k)!}\) represents the coefficient; and \(\tau\) denotes the time constant.  
The incorporation of a disturbance observer into the system enables the observation of equivalent disturbances that affect the system, enabling disturbance compensation within the control framework. Ultimately, this leads to effective disturbance suppression, mitigating the impact of disturbances on the control system.