## 1 Introduction  
In recent years, unmanned aerial vehicles (UAVs) have attracted significant research interest on account of their wide range of applications in both the civilian and military fields, such as agricultural production, logistics and distribution, urban management, border blockade prevention, reconnaissance, surveillance, etc. [1, 2, 3, 4, 5]. Compared with other flight vehicles, a quadrotor UAV has many significant advantages, including a simple structure, vertical take-off and landing, a low manufacturing cost, and so on [6]. However, the quadrotor UAV flight control system is an underactuated system, and its dynamic model possesses the characteristics of nonlinearity and a strong coupling of the translational and rotational dynamics. Moreover, it is very sensitive to disturbances, including external gust disturbances, aerodynamic effects, model uncertainties, and so on. Considering the trajectory tracking control of a quadrotor UAV as the basis for the previously mentioned applications, a series of extensive studies has been undertaken on this subject. For practical applications, linear control schemes have been adopted to linearize the complex dynamic model to ensure the basic operation of a quadrotor UAV, such as proportional-integral-derivative (PID) control [7] and linear quadratic regulator (LQR) control [8]. However, the linear model can cause performance degradations when performing complex tasks and when in unknown environments. In order to improve the tracking performance, various nonlinear control strategies have been further studied, including sliding mode control [9; 10], backstepping control [11], adaptive control [12], and so on. However, the control schemes mentioned above only focus on the stability of the system and do not consider the control cost. Since mobile vehicles can carry limited energy equipment, it is necessary to consider the control cost and energy optimization [13; 14]. Therefore, it is of great significance to design a tracking control scheme that can not only stabilize the quadrotor UAV flight control system, but also minimize its control cost. Adaptive dynamic programming (ADP) is an optimization-theory-based control method and achieves a balance between the control cost and control performance by adaptively adjusting the control policy. As an effective method for solving the optimal control problems of nonlinear systems, it has received much attention in its related fields. [PERSON] et al. [15] first proposed a heuristic dynamic programming algorithm based on reinforcement learning and designed a preliminary online learning control framework. [PERSON] et al. [16] proposed an online algorithm based on actor-critic neural networks (AC NNs) to learn the optimal control solutions for nonlinear systems of known dynamics with an infinite horizon cost. The AC NNs were used to approximate the control policy and cost function, respectively. [PERSON] et al. [17] investigated an ADP-based optimal tracking problem for nonlinear systems. [PERSON] et al. [18] solved the robust stabilization problem via adaptive-critic-based techniques. [PERSON] et al. [19] introduced a new optimized backstepping control method based on ADP and backstepping technologies, which solved the tracking control problem of strict feedback systems. Furthermore, ADP technology has also been extensively researched in fault-tolerant control [20], zero-sum games [21], multi-agent systems [22], etc. However, most of the existing weight update laws are required to satisfy persistent excitation (PE) conditions. It is worth mentioning that some scholars have performed successive studies to avoid the use of PE conditions and make ADP technology easier to implement in practical engineering. In [23; 24], the experience replay technique was utilized to overcome the PE condition for weight convergence. [PERSON] et al. [25] proposed a simplified online learning strategy, which removed the PE conditions via the concurrent learning method. [PERSON] et al.