## 2 Methods
### Setting of Indoor Temperature and Humidity  
The real-time predicted thermal dissatisfaction rate values were applied to the online learning of room temperature and humidity to obtain reasonable setting values for indoor environmental parameters. The DDPG algorithm was a deep reinforcement learning algorithm based on the Actor-Critic structure, using a convolutional neural network approximation to represent the policy function and the value Q function, corresponding to the Actor-network and the Critic-network, respectively [29]. In the intelligent setting model based on the DDPG algorithm [30], the controller adopted a deterministic strategy to meet the comfort and energy-saving requirements of the room occupants based on the current environmental state information, and the temperature and humidity setting value for the next moment was related to the current occupant thermal dissatisfaction rate and the adopted temperature and humidity setting value. The predicted thermal dissatisfaction rate value and the environmental state were used as inputs to the DDPG controller to output the indoor set values and achieve online regulation of indoor temperature and humidity. The controller learned to obtain the optimal indoor temperature and humidity settings to change the thermal environment state, which is shown in Figure 2.  
Based on the DDPG algorithm, an online learning model for indoor temperature and humidity was developed and the system state, action, and reward functions are as follows:  
1. State: A reasonable indoor temperature and humidity setting value can reduce system energy consumption while ensuring human thermal comfort. The system state was the relevant parameters affecting thermal comfort and energy-saving including indoor temperature, relative humidity, airflow rate, clothing thermal resistance, and human metabolic rate. Individual variability made some physical quantities more difficult to measure such as human metabolic rate and clothing thermal resistance, which could only be obtained by approximate values. Due to the small range of airflow rate variation in the thermal environment of the closed building and other reasons, the more difficult-to-measure parameters were fixed as the average value in the current thermal environment, and the state space at moment \(t\) is defined as shown in Equation (7). \[S_{t}=[T_{t},RH_{t},T_{mrt},BPD_{t}],\] (7) where \(T_{t},RH_{t}\), and \(T_{mrt}\) correspond to the indoor temperature, relative humidity, and mean radiant temperature of time series \(t\), respectively, and \(BPD_{t}\) is the predicted value of thermal dissatisfaction rate, and \(t\) is the time series, \(t=1,2,\dots\). 2. Action: The thermal environment state was changed by adjusting the room temperature and humidity settings. When the environment state is \(s_{t}\), the room temperature and humidity settings are used as action parameters, and when the environment state is \(s_{t+1}\), the action \(a_{t+1}\) is executed. All actions are selected in the action space, and the action space at time \(t\) is shown in Equation (8). \[A_{t}=[T_{set\_t},H_{set\_t}],\] (8) where \(A_{t}\) is the action of the Markov decision process at moment \(t\), \(T_{set\_t}\), and \(H_{set\_t}\) are the set values of indoor air temperature and relative humidity after the controller action, and the control action is determined by the control strategy \(\mu\), as shown in Equation (9). \[A_{t}=\mu(S_{t}).\] (9)  
Reward: In practical scenarios, thermal comfort and energy savings conflict with each other. Setting a reward function to balance the conflict between them was the core of online regulation of air conditioning systems. Designing a suitable reward function is beneficial to  
Figure 2: Intelligent setting of indoor parameters based on DDPG.