## 1 Introduction  
With the rapid economic growth, the increase in electricity demand and the large-scale access of distributed energy sources have made the scale of power grids increasingly large and complex, thereby, introducing more power grid disturbance factors. Power grid disturbances (such as load change disturbances, etc.) may degrade the power quality of the frequency and threaten the security of power grids [1, 2, 3], and thus it is necessary to impose load-frequency control on power grids to maintain the power frequency on a fixed value (e.g., 50 and 60 Hz). Load frequency control, as a means to maintain the balance of supply and demand, is re-emphasized when a large number of distributed energy sources are connected. Aggregated distributed energy sources (ADES) are connected to power grids based on fast response power electronics and aggregated output techniques, which provides a new flexible implementation method for load-frequency control [4; 5]. ADES can quickly respond to the control instructions issued by digital computers [6; 7], eliminate frequency deviation, and stabilize the frequency in a relatively small range in the presence of interference. ADES consists of multiple small energy storage systems. Compared with a single energy storage system, ADES may have greater power and capacity ratings, and it can be regarded as an entity of system operation. In this process, the ADES controller typically distributes the sub-control tasks to a single node. The ADES controller assigns the realization task of control output to each distributed energy system for realization. However, in this paper, this research focuses on the intelligent frequency modulation algorithm, and the dispatch method of the aggregator has been investigated in many studies [8; 9; 10; 11]. Therefore, this paper does not give a detailed introduction to the dispatch algorithm. At present, some methods have been proposed for the frequency control of power systems [12; 13; 14; 15; 16]. As a high-dimensional nonlinear complex large-scale system, the control strategy design of modern power grids (the control output of ADES controllers) involves large-scale analytical modeling [17; 18; 19]. Data-driven methods are often used to solve the analytical modeling issues [20; 21]. Reinforcement-learning techniques as a type of data-driven method can effectively solve the frequency control issues of complex power grids [21]. Reinforcement learning aims to directly optimize the control strategy of power grids based on the measured frequency data without analyzing the influence of the accurate analytical model of power grids in the control process. With appropriate training and cost-return functions, data-driven frequency control methods based on reinforcement learning can achieve good control performance. Reinforcement learning can easily obtain a suitable power grid frequency control strategy under nonlinear working conditions [22]. [PERSON] et al. [23] proposed an adaptive nonlinear control technique that applied reinforcement learning to the frequency control issues of single-region power grids. [PERSON] et al. proposed a deep reinforcement-learning (DRL) model in continuous scope to improve the load frequency performance of single-region power grids [22]. However, the concurrent learning of multi-zone controllers causes the overall operating environment to be no longer stable. Such learning algorithms lack consistent gradient signals in multi-region power grids [24]; therefore, they cannot guarantee frequency control performance in a cooperative manner. A multi-agent reinforcement-learning (MARL)-based frequency control strategy was proposed to solve the frequency control issues in multi-region power grids with a co-current learning convergence guarantee [25; 26]. This strategy divides each sub-region of multi-region power grids into a single agent. Each agent coordinates to control the frequency of multi-region power grids.