## 3 Mathematical Model for the Proposed Framework
### Mathematical Model for the Lower Layer  
In the lower layer of the framework, a mathematical model under the reinforcement learning paradigm is established for household users. The model incorporates three types of appliance agents into HEMS: fixed appliance agent, curtailable appliance agent, and shiftable appliance agent. Under the reinforcement learning paradigm, a task is considered a Markov decision process (MDP) if the state and environment of the intelligence satisfy Markovian properties. An MDP typically consists of five tuples (\(S\), \(A\), \(P\) (, ), \(r\), \(u\)), where \(S\) denotes the state set, \(A\) denotes the action space, \(P\) (, ) denotes the action transition probability, \(r\) denotes the immediate reward, and \(u\) is the learning rate. The details of the MDP formulation are described as follows:  
1. State  
The environmental information required during the operation of appliances together forms the agent's state space. In this paper, the state \(S_{t}\) at the time slot \(t\) can be defined as follows:  
\[S_{t}=[\lambda_{t}^{H},\lambda_{t+1}^{H},\ldots,\lambda_{T}^{H},L_{t}^{H},L_{t +1}^{H},\ldots,L_{T}^{H}] \tag{13}\]  
where \([\lambda_{t}^{H},\lambda_{t+1}^{H},\ldots,\lambda_{T}^{H}]\) are the tariff under the current time slot \(t\) and the predicted tariff from the next time slot \(t+1\) to the end time slot \(T\); \([L_{t}^{H},L_{t+1}^{H},\ldots,L_{T}^{H}]\) are the adjusted load target under the current time slot \(t\) and the adjusted load target from the next time slot \(t+1\) to the end time slot \(T\). 1. Action  
The control actions of household appliance agents can be categorized based on the type of appliance as follows [29]:  
1. Fixed appliance agent: Fixed appliances, such as EPS, require stability. These appliances are essential for daily life, so their demand must be met. The action space for these appliances includes the following: \[A_{fixed}=[on]\] (14)
2. Curtailable appliance agent: Curtailable appliances, such as REF and AC, possess the characteristic of load reduction. Household users can adjust the demand for such appliances within a certain range, enabling a discretized selection of load reduction actions within the following action space: \[A_{curta}=[A_{REF,k},A_{AC,k}]\] (15) where \(k\) is the level, which is classified according to the operating power of the appliance; \(A_{REF,k}\) and \(A_{AC,k}\) represent the action values of REF and AC at the \(k\) level. (c) Shiftable appliance agent: Shiftable appliances, such as CP, WM, and WH, can flexibly adjust the timing of energy usage. These appliances operate at two distinct working points: on and off. The action space for these appliances is as follows:  
\[A_{shift}=[A_{CP},A_{WM},A_{WH}] \tag{16}\]  
(3) Reward  
Setting the reward function can constrain the action of the appliance agent to achieve the training needs. The reward function settings of various types of appliance agents are as follows:  
(a) Fixed appliance agent: The output of fixed appliances cannot be changed, so their reward value is only related to the tariff. The reward function is as follows:  
\[E_{fixed}=\max[(-\lambda_{Ag,t}\times P_{EPS,t}),0] \tag{17}\]  
where \(\lambda_{Ag,t}\) is the time-sharing tariff given by the VPP aggregator's upper-layer game at time slot \(t\); \(P_{EPS,t}\) is the case of the appliance output of the EPS at time slot \(t\). (b) Curtailable appliance agent: The incentive function for curtailable appliances includes the satisfaction costs resulting from the tariff and load adjustments. The reward function is as follows:  
\[E_{curta}=\max[(-\lambda_{Ag,t}\times P_{k,t}),0]-E_{j,t} \tag{18}\]  
where \(k\) is the set of curtailable appliances; \(E_{j,t}\) is the satisfaction cost, given by Equation (10). (c) Shiftable appliance agent: The incentive function for shiftable appliances includes the satisfaction costs resulting from the tariff and load adjustments.