A Two-Layer User Energy Management Strategy for Virtual Power Plants Based on HG-Multi-Agent Reinforcement Learning  
[PERSON]  
1 State Key Laboratory of Intelligent Power Distribution Equipment and System, Tianjin University, Tianjin 300072, China; [EMAIL_ADDRESS] (S.T. ); [EMAIL_ADDRESS] (Q.X)1 State Key Laboratory of Intelligent Power Distribution Equipment and System, Tianjin University, Tianjin 300072, China; ts_2022@itu. focuses on formulating VPP optimization strategies that meet the needs of multiple stakeholders and accommodate the diversity in the dynamics of user energy behavior. Many studies have been conducted on developing optimized operation strategies for VPPs. Considering the trend towards multi-agent participation in the VPP, the discussion below primarily focuses on two aspects: the energy behaviors of each user and the multi-agent game-solving method. Regarding the energy behaviors of each user, to explore the dispatch potential of users in particular, their energy consumption patterns should first be modeled. As a crucial means of energy regulation in VPPs, demand response (DR) has garnered significant attention. Some scholars have established operational models under DR mechanisms. In reference [8], a three-stage stochastic two-tier optimization model for VPP is proposed, which enhances the market competitiveness and economic efficiency of virtual power plants by incorporating demand response scenarios for distributed resources within the VPP into the model. In reference [9], a risk-constrained day-ahead scheduling strategy based on information gap decision theory is proposed, considering the uncertainties from the participation of concentrating solar power plants, responsive residential, and industrial loads in the demand response mechanism. In reference [10], a bidirectional flow of power and information between energy service providers and electricity consumers within a smart grid model is established. It applies a price-based dynamic DR mechanism, which utilizes clustering algorithms to identify user types for implementing DR. Various market-clearing models considering DR mechanisms are proposed in reference [11; 12; 13; 14]. Some scholars have placed VPP models within reinforcement learning frameworks for their research. In reference [15], dynamic demands within VPPs are considered, and a multi-agent reinforcement learning energy trading model that includes users and energy supply appliances is developed. In reference [16], a multi-stage reward mechanism for VPPs within a deep learning framework is introduced, quantifying the impact of multiple entities on the VPP during operation, thereby reducing the difficulty of model training. In reference [17; 18], an optimization model based on Q-learning to reduce energy costs and dissatisfaction costs caused by DR is proposed. Most of the literature establishes multi-agent models for VPPs, utilizes price-based DR regulation models for energy supply, or employs reinforcement learning frameworks to enhance model accuracy during operation. However, as the variety of electrical appliances in household users increases, the difficulty of load regulation also rises. The energy dispatch instructions issued by VPPs struggle to accommodate diverse user energy demands, thus limiting the dispatch potential on the user side. Reinforcement learning methods, with their ability to iteratively explore and experiment in dynamic environments and possessing learning memory, hold certain advantages in formulating complex multi-appliance instant adjustment schemes. Within reinforcement learning methods, Q-Learning has emerged as a preferred choice for device-level energy management in residential users due to several key advantages: it does not rely on an accurate environmental model, features a simple algorithmic structure, offers strong adaptability to changing conditions, and demonstrates good convergence properties.