These characteristics make Q-Learning particularly well-suited for managing the complex and dynamic energy consumption behaviors in household-level VPP applications. Regarding the multi-agent game-solving method, the diverse array of energy supply and consumption entities within VPPs possesses varying interests. Conventional centralized algorithms have limited capabilities in resolving conflicts among multiple stakeholders, resulting in poor solution speed and accuracy. Therefore, algorithms rooted in game theory have become a more suitable choice. In reference [19], a non-cooperative game model between VPPs is developed to achieve optimal pricing of photovoltaic feed-in with reduced load bias. In reference [20], a hierarchical energy management structure for VPPs is pro posed and a two-tier optimal scheduling model based on multi-intelligent agent systems and DR is established to realize the benefits of VPPs. In reference [21; 22], the authors consider the incompleteness of the information game of multiple stakeholders within a VPP and research the interaction and gaming behaviors among stakeholders, ultimately improving the overall benefit. In reference [23], the energy management of the VPP is modeled using the multi-leader-multi-follower Stackelberg game theory, which enhances both VPP profit and customer energy consumption levels. In reference [24], a profit-sharing model for multiple VPP operators in energy reserve trading is established, benefit-sharing is designed using asymmetric Nash bargaining, and the model's validity is confirmed through the alternating direction multiplicative decomposition technique. In reference [25], differences in the accumulated knowledge of game participants are considered, local incomplete game theory is introduced into the computational model of the multi-subject game for multi-agent systems, and the stability of the methodology is analyzed. In reference [26], the authors address the optimal scheduling problem faced by power retail companies in the context of Virtual Power Plants (VPPs), taking into account the uncertainties associated with photovoltaic generation, energy storage, and controllable loads. To this end, a leader-follower (Stackelberg) game-based optimization model is developed. In reference [27], a prospect theory-based Stackelberg game model is proposed to address the energy trading problem between prosumers and power companies under price uncertainty. By introducing reference points and loss aversion coefficients, the model captures the subjective decision-making behavior of prosumers and determines the optimal energy trading price. In reference [28], A multi-level energy management approach based on a multi-leader-follower game theory framework is proposed, enabling negotiation between power suppliers and building users as well as user demand scheduling. This approach effectively reduces the overall system cost. Although the above literature has established a more accurate multi-agent game model, there remains potential for enhancement in algorithmic speed and accuracy. Based on the above analysis, a novel two-layer user energy management strategy based on HG-multi-agent reinforcement learning is proposed. The primary contributions of this article can be outlined in the following three aspects:  
1. A two-layer optimization framework for the VPP has been proposed, in which the upper layer coordinates the scheduling and benefit allocation among various stakeholders and the lower layer executes intelligent decision-making for users. 2. A detailed household power management model is proposed to generate predicted power demands and allocate power within each household user in the lower layer. 3. A HG-multi-agent reinforcement-based method is proposed, which reconstructs the objective functions of each stakeholder with the hierarchical game algorithm to simplify the solving process, utilizing the multi-agent Q-learning algorithm combined with upper-layer solution information to accelerate training and solving.