# Deep reinforcement learning for coordinated air-conditioner control in groups of buildings using smart meter data

Sharath Ram Kumar, Arvind Easwaran, Benoit Delinchant, Rémy Rigo-Mariani

<figure><img src="image_1.png" /></figure>

## ► To cite this version:

Sharath Ram Kumar, Arvind Easwaran, Benoit Delinchant, Rémy Rigo-Mariani. Deep reinforcement learning for coordinated air-conditioner control in groups of buildings using smart meter data. Engineering Applications of Artificial Intelligence, 2025, 166, pp.113536. 10.1016/j.engappai.2025.113536. hal-05441903

HAL Id: hal-05441903

https://hal.science/hal-05441903v1

Submitted on 5 Jan 2026

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_3.png" /></figure>

# Deep reinforcement learning for coordinated air-conditioner control in groups of buildings using smart meter data ☆

Sharath Ram Kumar<sup>a,b,c</sup>,<sup>*</sup> Arvind Easwaran<sup>b,c</sup>,<sup>†</sup> Benoit Delinchant<sup>a,c</sup>,<sup>‡</sup> Remy Rigo-Mariani<sup>a,c</sup>

## ARTICLE INFO

### Keywords:

Air-conditioner

Demand response

Reinforcement learning

Peak reduction

Hvac

Actor-critic

## ABSTRACT

Coordinated control of residential air-conditioning systems is a promising demand response scheme to reduce peak loads and lower energy bills at a district level. Existing schemes have achieved only limited real-world success due to infrastructural requirements and low consumer adoption. In this paper, a scalable method for the coordinated control of air-conditioners in a group of hundreds of buildings is proposed in which consumers are allowed to override control signals based on their local thermal comfort. The control is based on a novel Split-Input Actor-Critic Reinforcement Learning architecture with a neural network that suggests temperature setpoints for each consumer. It is trained on a grey-box model of the system developed using historical smart meter data. The approach notably does not require behind-the-meter inputs during deployment. In a test system, the controller is able to reduce peak loads by up to 20% without increasing the total energy consumption compared to the baseline case with local control only. The robustness of the controller to different reinforcement learning architecture, inputs, model accuracy and stochasticity is studied. Additionally, the learnt policies are visualized, providing insights about the decision-making of the controller.

## 1. Introduction and literature review

Peak loads on the power grid present a major challenge today in terms of equipment ageing (Shaghaghi et al., 2024) and an increase in carbon emissions (Van Orden and Simović, 2023). Indeed, to meet such loads, utility companies rely on peaking power plants, which are often fossil-fuel-based, outdated, and inefficient (Benetti et al., 2016). Despite the increasing use of cleaner energy sources like solar and wind, fossil fuels still account for approximately 75% of global electricity demand as of 2023 (IEA, 2023). Research has shown that targeting peak loads can be a more effective strategy for reducing emissions from buildings than simply reducing overall consumption (St-Jacques et al., 2024).

Demand response (DR) schemes are promising avenues for achieving grid-level targets such as peak load reduction (Stelytte et al., 2022) by encouraging end-users to alter their consumption patterns. There are broadly two types of DR — incentive-based (IBP) and price-based programs (PBP) (Mohajeryami et al., 2016). IBPs directly provide incentives to consumers in exchange for their participation in DR and are typically deployed in industrial contexts. PBPs utilize the price of electricity as a means to modulate the electricity demand

with consumers who wish to lower their bills (Kumar et al., 2024). While academic interest in the field has grown exponentially in recent years (Benetti et al., 2016), a recent International Energy Agency (IEA) report (IEA, 2021) estimates that the total installed demand response capacity from buildings needs to increase tenfold (from 25 GW to 250 GW between 2020 and 2030) in order to meet global net-zero emissions targets. In this context, DR programs aimed at groups of residential consumers are of significant interest, since the sector accounts for roughly 30% of electrical energy consumption (IEA, 2023). PBPs, such as real-time or time-of-use pricing, are by far the most commonly used approach today in residential DR. However, their effectiveness has been hindered by poor incentives, low participation rates and negligible response to price variations (Fabra et al., 2021; Silva et al., 2017; Kubli et al., 2018). Consequently, there is growing interest in IBPs for residential DR.

Heating, ventilation and air-conditioning (HVAC) loads account for nearly 40% of energy use in the residential sector (González-Torres et al., 2022) and contribute heavily to peak loads (Sigauke and Nemukula, 2020). As such, IBPs based on HVAC flexibility are promising

<footnote><sup>a</sup> Univ. Grenoble Alpes, CNRS, Grenoble INP, G2Elab, Grenoble, France</footnote>

<footnote><sup>b</sup> Nanyang Technological University, Singapore</footnote>

<footnote><sup>c</sup> CNRS@CREATE Ltd., Singapore</footnote>

<footnote>☆ This article is part of a Special issue entitled: 'AI-Driven Innovations in Cyber-Physical Systems' published in Engineering Applications of Artificial Intelligence.</footnote>

<footnote>* Corresponding author at: Univ. Grenoble Alpes, CNRS, Grenoble INP, G2Elab, Grenoble, France.</footnote>

<footnote>E-mail addresses: sharath.ramkumar@cnrsatcreate.sg (S. Ram Kumar), arvinde@ntu.edu.sg (A. Easwaran), benoit.delinchant@g2elab.grenoble-inp.fr (B. Delinchant), remy.rigo-mariani@g2elab.grenoble-inp.fr (R. Rigo-Mariani).</footnote>
List of Abbreviations

<table><tr><td>AC</td><td>Air-Conditioner</td></tr><tr><td>ACM</td><td>Approximate Consumer Model</td></tr><tr><td>AG</td><td>Aggregator</td></tr><tr><td>C-PPO</td><td>Centralized PPO</td></tr><tr><td>CB-Det</td><td>Comfort-Based Deterministic</td></tr><tr><td>CB-Sto</td><td>Comfort-Based Stochastic</td></tr><tr><td>COP</td><td>Coefficient of Performance</td></tr><tr><td>DNN</td><td>Deep Neural Network</td></tr><tr><td>DR</td><td>Demand Response</td></tr><tr><td>DRL</td><td>Deep Reinforcement Learning</td></tr><tr><td>DSI-PPO</td><td>Decentralized SI-PPO</td></tr><tr><td>EV</td><td>Explained Variance</td></tr><tr><td>HVAC</td><td>Heating, Ventilation and Air-Conditioning</td></tr><tr><td>IBP</td><td>Incentive-Based Program</td></tr><tr><td>IBP</td><td>Price-Based Program</td></tr><tr><td>MARL</td><td>Multi-Agent Reinforcement Learning</td></tr><tr><td>MF-Sto</td><td>Multifactor Stochastic</td></tr><tr><td>MSE</td><td>Mean-Squared Error</td></tr><tr><td>NILM</td><td>Non-Intrusive Load Monitoring</td></tr><tr><td>PC</td><td>Pearson Cross-Correlation Coefficient</td></tr><tr><td>PPO</td><td>Proximal Policy Optimization</td></tr><tr><td>RC</td><td>Resistive Capacitive</td></tr><tr><td>RL</td><td>Reinforcement Learning</td></tr><tr><td>SI-PPO</td><td>Split-Input PPO</td></tr><tr><td>t-SNE</td><td>T-distributed Stochastic Neighbour Embedding</td></tr><tr><td>TMY</td><td>Typical Meteorological Year</td></tr></table>

due to their scale and to the possibility of exploiting thermal inertia in buildings to shift loads (Hao et al., 2015). Coordinated management of HVAC resources at the district level is a challenging problem, and is the subject of much ongoing research (Celik et al., 2017). Compared to other sources of energy flexibility, such as battery storage and electric vehicles (EVs), HVAC loads have significantly more DR potential (Mahmoud et al., 2020). A wide range of solutions have been proposed, covering different architectures (centralized, decentralized and hybrid) and methodologies (optimization, data-driven learning, heuristics) (Waseem et al., 2021), but only a few successful trials have been reported involving small groups of consumers (Kohlhepp et al., 2019). A comparative analysis of different control schemes is provided in Biagioni et al. (2024), where hybrid architectures combining data with models were found to achieve the best performance in simulations. From a social perspective, surveys have shown that potential consumers are reluctant to participate due to poor incentive structures, perceived risks of giving up control, technology requirements, high effort to respond to signals and other personal factors (Parrish et al., 2020; Nilsson and Bartusch, 2024). Designing DR schemes that allow consumer interaction, such as allowing signals to be overridden during periods of high discomfort, could potentially improve participation rates (Kubli et al., 2018; Broberg and Persson, 2016). Thus, there is a pressing need for DR schemes that can address these practical concerns while providing grid-level benefits (Weck et al., 2017).

Table 1 provides a brief overview of the existing literature on the subject, while a more in-depth review is available in Waseem et al. (2021) and Rajasekhar et al. (2020). It should be noted that all the studies are based on computer simulations, and the reported scale need not translate to real-world applications. The limitations correspond to data, model and technology (denoted as “Tech”) requirements, which are detailed below.

A major barrier to the adoption of DR schemes is the need for data that may be considered private from the perspective of the consumer. Often termed “behind-the-meter” data, this may include measurements of the indoor temperatures, occupancy, appliance-level load monitoring or load schedules (Harper et al., 2022). This is especially problematic in centralized architectures, such as in Du et al. (2021), where a central Deep Reinforcement Learning (DRL) agent takes control decisions for a group of buildings, but requires zone temperatures and comfort bounds from consumers. The “Data” limitation in Table 1 applies to techniques which require data other than smart meter readings to be shared with an external entity such as a central aggregator.

The need for mathematical models at the controls stage represents a data availability issue, since models require significant engineering effort and are unique to individual buildings (Killian and Kozek, 2016). For example, in Pinto et al. (2021), high-fidelity thermal models (using EnergyPlus) of consumer buildings are used to generate training data for representative deep neural network (DNN) models, which are subsequently used in a DRL training environment. A closely related issue is that of not considering the impact of model inaccuracy when evaluating approaches. An example is (Cole et al., 2014), in which a linear model of the consumers’ cooling energy consumption is used to implement a Mixed-Integer Linear Programming (MILP) solution, and the results are reported based on the same model (there is no difference between the model and reality). These limitations are captured under the “Model” heading in Table 1.

A popular way to overcome these issues is to adopt a decentralized (and optionally, model-free) architecture such as in Charbonnier et al. (2022), where multiple flexibility sources are coordinated for a group of consumers using Q-learning. These techniques generally encounter issues with scalability — for example, the benefits in Charbonnier et al. (2022) were not achievable beyond a cluster size of 30 consumers. In addition to this, decentralized approaches require computational power at the consumer site, which may represent a technological barrier. Several techniques rely on the solution of a mathematical optimization problem, the output of a DNN, or heuristic-based calculations from data (Lankeshwara et al., 2022; Kou et al., 2021; Wu et al., 2018; Behboodi et al., 2018). Deploying devices at consumer sites increases the costs of implementation and maintenance, and may discourage consumers from participating in the scheme (Nilsson and Bartusch, 2024). The “Tech” limitation in Table 1 represents approaches that require dedicated control infrastructure at the consumer site for their implementation.

One last limitation identified for HVAC-based DR schemes is the need to maintain the thermal comfort of the consumer. A key issue is that this is subjective and context-dependent, and commonly used measurements to quantify thermal comfort might not accurately capture the level of discomfort faced in reality (Enescu, 2017). Most of the literature considered in this paper provide mechanisms to ensure that the indoor temperature stays within certain bounds, such as through constraints in an optimization problem (Erdinc et al., 2017; Cole et al., 2014) or by penalizing comfort violations in the objective function (Pinto et al., 2021; Du et al., 2021; Charbonnier et al., 2022). However, recent empirical studies have shown that consumers are sensitive both to the instantaneous and the cumulative thermal discomfort, and their behaviour may be better described through models that consider the history of deviation from an ideal/desired temperature (Ryu et al., 2020). Additionally, considering such comfort constraints at the controller stage requires the indoor temperature to be measured, which leads back to potential privacy issues.

## 1.1. Novelty of the proposed approach

A review of the literature shows that the practical deployment of HVAC-based DR solutions continues to be limited by both social and technological barriers. Despite longstanding evidence from the social sciences highlighting consumer concerns such as data privacy and
<table><caption>**Table 1**
Overview of existing literature.</caption><thead><tr><th rowspan="2">Ref</th><th rowspan="2">Objective</th><th rowspan="2">Architecture</th><th rowspan="2">Methodology</th><th rowspan="2">#Consumers</th><th colspan="3">Limitations</th></tr><tr><th>Data</th><th>Model</th><th>Tech</th></tr></thead><tbody><tr><td>Charbonnier et al. (2022)</td><td>Minimize costs</td><td>Decentralized</td><td>Q-Learning</td><td>30</td><td>N</td><td>N</td><td>Y</td></tr><tr><td>Lankeshwara et al. (2022)</td><td>Load tracking</td><td>Decentralized</td><td>ADMM</td><td>500</td><td>N</td><td>Y</td><td>Y</td></tr><tr><td>Tindemans et al. (2015)</td><td>Load tracking</td><td>Decentralized</td><td>Analytical</td><td>1000</td><td>Y</td><td>Y</td><td>Y</td></tr><tr><td>Behboodi et al. (2018)</td><td>Minimize costs</td><td>Decentralized</td><td>Bidding strategy</td><td>1000</td><td>N</td><td>N</td><td>Y</td></tr><tr><td>Homod et al. (2023)</td><td>Minimize costs</td><td>Decentralized</td><td>MARL</td><td>6</td><td>Y</td><td>Y</td><td>Y</td></tr><tr><td>Xue et al. (2025)</td><td>Minimize costs</td><td>Decentralized</td><td>GA-MADDPG</td><td>3</td><td>Y</td><td>N</td><td>N</td></tr><tr><td>Xia et al. (2025)</td><td>Minimize costs</td><td>Decentralized</td><td>MADDPG</td><td>32</td><td>Y</td><td>N</td><td>N</td></tr><tr><td>Mai et al. (2024)</td><td>Load tracking</td><td>Decentralized</td><td>MARL</td><td>50</td><td>Y</td><td>N</td><td>N</td></tr><tr><td>Sanandaji et al. (2014)</td><td>Load balancing</td><td>Hybrid</td><td>Heuristic</td><td>10 000</td><td>Y</td><td>Y</td><td>Y</td></tr><tr><td>Kou et al. (2021)</td><td>Minimize costs</td><td>Hybrid</td><td>ADMM, DDPG</td><td>100</td><td>Y</td><td>Y</td><td>Y</td></tr><tr><td>Wu et al. (2018)</td><td>Frequency regulation</td><td>Centralized</td><td>Aggregate droop curve</td><td>1000</td><td>Y</td><td>N</td><td>Y</td></tr><tr><td>Claessens et al. (2018)</td><td>Profit, Load flattening</td><td>Centralized</td><td>Fitted-Q iteration</td><td>100</td><td>Y</td><td>N</td><td>N</td></tr><tr><td>Bashash and Fathy (2013)</td><td>Track wind power</td><td>Centralized</td><td>Optimization</td><td>1000</td><td>N</td><td>Y</td><td>N</td></tr><tr><td>Pinto et al. (2021)</td><td>Peak reduction</td><td>Centralized</td><td>SAC</td><td>4</td><td>Y</td><td>Y</td><td>N</td></tr><tr><td>Du et al. (2021)</td><td>Consumption reduction</td><td>Centralized</td><td>DDPG</td><td>2</td><td>Y</td><td>N</td><td>N</td></tr><tr><td>Gokhale et al. (2023)</td><td>Load tracking</td><td>Centralized</td><td>Q-Learning</td><td>8</td><td>Y</td><td>N</td><td>N</td></tr><tr><td>Lu (2012)</td><td>Load tracking</td><td>Centralized</td><td>Heuristic</td><td>1000</td><td>Y</td><td>Y</td><td>N</td></tr><tr><td>Song et al. (2018)</td><td>Minimize costs</td><td>Centralized</td><td>Optimization</td><td>2000</td><td>Y</td><td>Y</td><td>N</td></tr><tr><td>Erdinc et al. (2017)</td><td>Track DR signals</td><td>Centralized</td><td>MILP</td><td>40</td><td>Y</td><td>Y</td><td>N</td></tr><tr><td>Cole et al. (2014)</td><td>Peak reduction</td><td>Centralized</td><td>Optimization</td><td>900</td><td>Y</td><td>Y</td><td>N</td></tr><tr><td>Yang et al. (2024)</td><td>Load tracking</td><td>Centralized</td><td>DDQN</td><td>20</td><td>Y</td><td>N</td><td>N</td></tr><tr><td>Ding et al. (2025)</td><td>Minimize consumption</td><td>Centralized</td><td>MPPI</td><td>5</td><td>Y</td><td>N</td><td>N</td></tr><tr><td>This work</td><td>Peak reduction</td><td>Centralized</td><td>PPO</td><td>100</td><td>N</td><td>N</td><td>N</td></tr></tbody></table>

loss of agency, research from the controls community has consistently prioritized solution optimality — an emphasis that persists even in recent work (Ding et al., 2025; Xue et al., 2025).

This paper proposes a method to control air-conditioners (ACs) in a group of 100 buildings to reduce aggregate peak loads while explicitly allowing consumers to override control signals based on their thermal comfort. A centralized architecture is adopted, with an aggregator that sends individual temperature setpoint signals to each consumer on an hourly basis. It is based on approximate grey-box models of consumers and a novel Split-Input Actor-Critic DRL architecture that enhances the learning performance while limiting online data requirements. With this in mind, the key novelties of the proposed approach are:

1. A hybrid solution combining grey-box building models with actor-critic deep reinforcement learning is presented, enabling model-based insights during training while remaining fully data-driven during deployment.

2. The method offers a scalable centralized architecture capable of coordinating control across hundreds of buildings without requiring behind-the-meter information or pre-existing models. It relies solely on smart meter and weather data, thereby protecting consumer privacy while explicitly accounting for model inaccuracy through domain randomization (Zhao et al., 2020).

3. The framework maintains user thermal comfort by allowing consumers to override control signals based on local conditions, ensuring practicality and promoting user engagement.

In summary, the main novelty of the methodology is that it is simultaneously scalable, easy to implement, and respects consumer privacy. It is motivated by the need to address several technology-centric and consumer-centric adoption barriers identified from real-world trials of such systems.

The rest of the paper is arranged as follows: Section 2 gives an overview of the system consisting of an aggregator and a set of consumers. Section 3 details the various steps in proposed method. Section 4 analyses the key results from the experiments performed. Finally, Section 5 concludes the discussion and provides a summary of the key findings.

<figure><img src="image_4.png" /><figcaption>**Fig. 1.** System illustration.</figcaption></figure>

## 2. System description

Fig. 1 shows a schematic of the system components and interactions. The case study is based on the control of a group of residential consumers ($n \in N$) over time ($t \in T$ at 1-hour resolution here) situated in a tropical region (only cooling loads are considered). Each consumer is equipped with a smart meter, and has a cooling load $P_{n,t}^{ac}$ (for the AC system - this is the controllable load) and an electrical load $P_{n,t}^{elec}$ (non-controllable appliance load). The overall consumption is metered and aggregated as $P_{n,t}^{grid}$. Each consumer also has a preferred
indoor temperature $T_n^{in,ideal}$. In the default case, the AC setpoint $T_{n,t}^{set} = T_n^{in,ideal} \forall t$.

The group is controlled by an aggregator (AG), such as a utility company, with the objective of reducing the overall peak load of the system. At every time step $t$, the AG accesses the latest smart meter readings ($P_{n,t-1}^{grid}$) and the ambient temperature ($T_t^{amb}$). It uses these inputs to generate dimensionless control signals ($T_{n,t+1}^{set,req} \in [-1, 1]$, indicating an indoor setpoint adjustment from lower to higher values relative to $T_n^{in,ideal}$. One of the key novelties in the proposed approach is that the consumer has the freedom to arbitrarily override $T_{n,t+1}^{set,req}$ depending on their local thermal comfort conditions. The AG's strategy must account for the possibility of overrides (without any feedback signal), since they can adversely affect its objective of peak reduction (Wildstein et al., 2023).

## 2.1. Quantifying thermal discomfort and the override mechanism

Recent empirical studies have shown that consumers' tolerance to thermal discomfort depends both on the duration and the intensity of the temperature deviation from an ideal value (Ryu et al., 2020; Jian et al., 2022). In addition to $T_n^{in,ideal}$, the thermal comfort preferences for each consumer are characterized by a discomfort capacity $D_n^{max}$ (in °Ch). The actual thermal discomfort at time $t$, $D_{n,t}$, is calculated as shown in Eq. (1) and measures the cumulative deviation from $T_n^{in,ideal}$ over a window of time.

$$D_{n,t} = \int_{k=1}^{K} \max(0, T_{n,t-k}^{in} - T_n^{in,ideal})dk \quad (1)$$

Here, $K$ is the window size (in hours) over which the discomfort is calculated and its value is set to $K = 2$ h,<fnref n="1" /> while $dk$ is the step size (here, 15 min). The consumer allows the AG to request a setpoint change of up to 3 °C from $T_n^{in,ideal}$. The linear mapping $f_n(T)$ in Eq. (2) maps $T_{n,t}^{set,req}$ to temperatures in °C.

$$f_n(T) = T_n^{in,ideal} + 3 \cdot T \quad (2)$$

The override mechanism, which determines the actual AC setpoint $T_{n,t}^{set}$, is shown in Eq. (3). No restrictions are placed on the override action ($o_{n,t} \in \{0, 1\}$) - however, it is assumed that the discomfort is the only factor influencing this decision. In our experiments, we study the cases with and without a feedback signal to the AG regarding the override status to quantify its impact on the effectiveness of the solution.

$$T_{n,t}^{set} = \begin{cases} f_n(T_{n,t}^{set,req}), & \text{if } (T_{n,t}^{set,req} < 0) \text{ OR } (T_{n,t}^{set,req} > 0 \text{ and } D_{n,t} < D_n^{max}) \\ T_n^{in,ideal}, & \text{otherwise (with } o_{n,t} = 1 \end{cases} \quad (3)$$

## 2.2. Calculation of cooling and appliance loads

Each building contains a thermostatically controlled air-conditioning (AC) unit to satisfy the cooling demand of its occupants. An AC has a rated maximum power denoted by $P_n^{ac,max}$, and follows the setpoint-driven operation shown in Eq. (4). Here, $T_{n,t}^{in}$ is the indoor temperature and $T^{db}$ is a deadband around the setpoint where $T_{n,t}^{in}$ is permitted to float. In the default case, the AC setpoint $T_{n,t}^{set} = T_n^{in,ideal} \forall t$.

$$P_{n,t}^{ac} = \begin{cases} P_n^{ac,max}, & T_{n,t}^{in} - T_{n,t}^{set} > T^{db} \\ 0, & T_{n,t}^{in} - T_{n,t}^{set} < T^{db} \\ P_{n,t-1}^{ac}, & \text{otherwise} \end{cases} \quad (4)$$

The evolution of the indoor temperature $T_{n,t}^{in}$ in response to the cooling power and ambient conditions is calculated using a 4R2C thermal model of the building envelope as suggested in Harb et al. (2016). The electrical load $P_{n,t}^{elec}$ is taken from time-series data generated using a load generation tool, “LoadProfileGenerator” (Pflugradt et al., 2022), and is not controllable. Detailed descriptions of the 4R2C model and the synthetic load generation process are given in Appendix. Fig. 2 shows $\sum_{n \in N} P_{n,t}^{grid}$ and $T_t^{amb}$ over a week in the default case.

# 3. Methodology

## 3.1. Actor-critic deep reinforcement learning

Reinforcement learning (RL) is a type of machine learning where an agent learns optimal actions by interacting with an environment and receiving feedback in the form of rewards (Sutton and Barto, 2020). The goal is to maximize the cumulative reward over time with actions taken from observations of system states. DRL extends RL by using DNNs to handle high-dimensional state and action spaces, enabling application in complex domains like robotics and energy systems (Mnih et al., 2015).

Actor-critic methods are a hybrid class of RL algorithms that combine value-based and policy-based learning (Konda and Tsitsiklis, 1999). The “actor” is responsible for choosing actions ($a_t$) based on the current state ($s_t$) - this is the control agent. The “critic” evaluates those actions by estimating the value function ($V_\pi(s_t)$), which gives the expected future reward from $s_t$ under the current policy, and guides the actor’s decisions by providing feedback on the error between the predicted and actual returns ($r_t$). Fig. 3 shows the agent-environment interaction in actor-critic RL.

This architecture allows flexibility in the information provided to the actor and critic. For example, the critic can receive a fuller set of inputs, such as those predicted by models, while the actor uses only directly observable data (Pinto et al., 2017). This allows the critic to improve the actor’s performance while avoiding the need to access or estimate non-observable inputs during deployment, since only the actor is used at that stage. This aspect of actor-critic algorithms is explored in this paper.

## 3.2. Implementation

The solution is based on actor-critic DRL algorithms and grey-box building models. At the aggregator level, historical smart-meter ($P_{n,t}^{grid}$) and weather ($T_t^{amb}$) data is first collected from the uncontrolled system (i.e., $T_{n,t}^{set} = T_t^{in,ideal}$). This data is used to create a set of approximate consumer models (ACMs) for every consumer, which are used inside training environments for an actor-critic DRL agent to learn a policy offline. The trained actor network is finally deployed in the reference system. An overview of the steps involved in the end-to-end methodology is shown in Fig. 4.

### 3.2.1. Assumptions

A major assumption made in the process of generating ACMs (described in the next section) is that the AG is able to disaggregate $P_{n,t-1}^{grid}$ into the historical cooling load $P_{n,t-1}^{ac}$ and electrical load $P_{n,t-1}^{elec}$, using the historical ambient temperature data $T_t^{amb}$ (Liang et al., 2019). $P_{n,t-1}^{ac}$ and $P_{n,t-1}^{elec}$ are not directly observable by the AG as they are behind-the-meter data. While this is a strong assumption, recent works have achieved good performance on this task (Liang et al., 2019; Liang and Ma, 2020; Qi et al., 2020). These works utilize DNNs and hidden Markov models, and take advantage of the correlation between $T_t^{amb}$ and $P_{n,t-1}^{ac}$. Additionally, as discussed in Cimen et al. (2021), the complexity of load disaggregation greatly increases with the number of components (such as appliances) required. Here, only the cooling load is to be separated; further, the decomposition is only performed on historical data, and is not needed in real-time. A second assumption is that all ACs follow the operation described in Eq. (4) with a fixed deadband temperature $T^{db} = 0.5$ °C. This model is used in many works discussed in Section 1, and is considered valid for residential settings (Rajasekhar et al., 2020).

<footnote n="1">¹ In practice, this means that a consumer's actions may be impacted by discomfort encountered up to 2 h before the current time step.</footnote>
<figure><img src="image_5.png" /><figcaption>Fig. 2. Aggregated load and ambient temperature over a typical week in the case study with N = 100 and $T_{n,t}^{set} = T_{n,t}^{ideal}$.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Fig. 3. Agent-environment interaction in actor-critic RL.</figcaption></figure>

### 3.2.2. Description and generation of Approximate Consumer Models (ACMs)

ACMs are grey-box models of consumers consisting of a thermal component (representing the consumer's building and AC system) and a behavioural component (representing their thermal comfort preferences). An overview of the components and working of an ACM is depicted in Fig. 5 - the model parameters are shown in blue, and hat notations are used to describe unknown quantities which need to be estimated. ACMs are employed in a recursive manner to generate time-series predictions of the indoor temperature ($\hat{T}_{n,t}^{in}$), AC power ($\hat{P}_{n,t}^{ac}$) and the override status ($\hat{o}_{n,t}$) given a sequence of control signals ($T_{n,t}^{set,req}$) and weather ($T_{n,t}^{amb}$).

The thermal comfort model is parameterized by estimates of the ideal indoor temperature $\hat{T}_{n,t}^{in,ideal}$ and discomfort capacity $\hat{D}_{n,t}^{max}$. These cannot be extracted from $P_{n,t}^{grid}$ since historical data was collected with fixed setpoints. As such, the AG assigns typical values for these parameters based on domain knowledge. For a given choice of $\hat{T}_{n,t}^{in,ideal}$ and $\hat{D}_{n,t}^{max}$, the behaviour described in Section 2.1 is incorporated in the model to calculate the estimated setpoint $\hat{T}_{n,t}^{set}$ and the override status $\hat{o}_{n,t}$.

In contrast, the AC parameters $P_{n,t}^{ac,max}$ and $T_{n,t}^{db}$ (the deadband temperature) are perfectly known to the AG. The former is inferred from $P_{n,t}^{ac}$ after disaggregation, while the latter is assumed to be a constant throughout the system. The AC model follows the operation in Eq. (4) and predicts the AC power $\hat{P}_{n,t}^{ac}$.

A first-order RC thermal network (Harb et al., 2016) is used to approximate the thermal behaviour of the buildings. This choice is suitable for data-limited contexts such as the present case, and is often found to outperform black-box approaches in comparative studies (Talib et al., 2023). The simplified thermal model uses 2 parameters - a resistive element $R_{n,t}^{in}$ and a capacitive element $C_{n,t}^{in}$. It predicts the evolution of the indoor temperature as a function of $\hat{T}_{n,t}^{in}$, $T_{n,t}^{amb}$ and $\hat{P}_{n,t}^{ac}$ as shown in Eq. (5). Here, COP is the coefficient-of-performance

of the AC unit and $\Delta t$ is the time step — in this work, COP = 3.5 and $\Delta t$ = 15 min.

$$ \hat{T}_{n,t}^{in} = \hat{T}_{n,t-1}^{in} + \frac{\Delta t}{C_{n,t}^{in}} \left( \frac{T_{t-1}^{amb} - \hat{T}_{n,t-1}^{in}}{R_{n,t}^{in}} - \text{COP} \cdot \hat{P}_{n,t-1}^{ac} \right) \quad (5) $$

The approach given in Melillo et al. (2020) and Gianniou et al. (2018) to create thermal models from meter data is used to find suitable parameter values, as follows. For a given $R_{n,t}^{in}$ and $C_{n,t}^{in}$, the ACM is used to estimate the AC power $\hat{P}_{n,t}^{ac}$ (this is a time-series prediction with the same length as $P_{n,t}^{ac}$) for the historical ambient temperature $T_{n,t}^{amb}$ and the estimated $\hat{T}_{n,t}^{in,ideal}$ (with $T_{n,t}^{set,req} = 0$).<fnref n="2" /> $R_{n,t}^{in}$ and $C_{n,t}^{in}$ are then fit by minimizing the mean-squared error (MSE) between $\hat{P}_{n,t}^{ac}$ and $P_{n,t}^{ac}$ as described in Eq. (6). The implementation of the differential evolution algorithm (Storn and Price, 1997), a meta-heuristic function minimization approach, in the scpy library<fnref n="3" /> was used in this work.

$$ R_{n,t}^{in}, C_{n,t}^{in} = \arg \min_{R_{n,t}^{in}, C_{n,t}^{in}} \text{MSE} \left( P_{n,t}^{ac}, \hat{P}_{n,t}^{ac} | R_{n,t}^{in}, C_{n,t}^{in}, T_{n,t}^{amb}, \hat{T}_{n,t}^{in,ideal} \right) \quad (6) $$

It is clear from Eq. (6) that a different choice for $\hat{T}_{n,t}^{in,ideal}$ would result in different thermal parameters for a given $P_{n,t}^{ac}$ and $T_{n,t}^{amb}$. In this work, the AG defines a set of M discrete values of $\hat{T}_{n,t}^{in,ideal}$, each representing a possible choice of the preferred indoor setpoint. A corresponding set of thermal parameters are generated for each choice. This gives rise to a set of M models for each consumer, and the entire space of approximate models is denoted as $b_{n,m} : \forall n \in N, m \in M$. $b_{n,m}$ and $b_{n,m'}$ have the same AC model but different thermal model parameters if $m \neq m'$. Also, $b_{n,m}$ and $b_{n',m'}$ have no common parameters if $n \neq n'$ irrespective of the values of m and $m'$. The discomfort capacity $D_{n,t}^{max}$ is sampled from a normal distribution reflective of typical values.

Since the $\hat{T}_{n,t}^{in,ideal}$ values used by the AG may be very different from the actual $T_{n,t}^{in,ideal}$, the ACM predictions may be highly inaccurate. However, since the thermal models are based on heat physics, they still capture the underlying system dynamics. The set $b_{n,m}$ is embedded into the training environment (described subsequently) for the RL agents with the goal of enhancing the robustness of the agent to differences in consumer behaviour. This process of incorporating uncertainties as random variables within the training environment of an RL agent is termed “Domain Randomization” (Zhao et al., 2020). It is often used in cases where agents are trained in a simulator before deployment in a real system, to handle the gap between simulation and reality.

### 3.2.3. Offline training environment for actor-critic RL agent

The RL agent is designed to individually monitor and control each building. The controller uses inputs over a window of length L steps from t-1 to t-L. For a consumer n, the states $s_{n,t}$ and actions $a_{n,t}$ at every time step t are as follows (bold notation indicates a vector

<footnote n="2">² The RC parameters are independent of $\hat{D}_{n,t}^{max}$ since no setpoint deviation is requested here.</footnote>

<footnote n="3">³ Version 1.13.1.</footnote>
<figure><img src="image_7.png" /><figcaption>Fig. 4. Sequence of operations in the proposed methodology.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Fig. 5. Block diagram of an ACM. Hat notations represent estimated/unknown quantities.</figcaption></figure>

containing L previous values and ~n denotes all consumers excluding n).

$$ \bullet s_{n,t} = \{t, \mathbf{T}^{\text{amb}}, \mathbf{P}_{n}^{\text{grid}}, \sum \mathbf{P}_{\sim n}^{\text{grid}}, \mathbf{T}_{n}^{\text{in}}, \mathbf{o}_{n}\} \in \mathbb{R}^{5 \cdot L+1} $$

$$ \bullet a_{n,t} = T_{n,t}^{\text{set,req}} \in [-1, 1] $$

Here, manual feature selection techniques such as in Lei et al. (2024) are not used — instead, $s_{n,t}$ is composed of all available information signals. $s_{n,t}$ contains behind-the-meter information (such as $T_{n,t}^{\text{in}}$ and $o_{n,t}$), which makes it impractical to learn a policy by direct interaction with the real system. Therefore, an offline training environment is created in which the ACMs are used to estimate these inputs. Within the training environment, $s_{n,t}$ is split into $s_{n,t}^{\text{measured}} = \{t, \mathbf{T}^{\text{amb}}, \hat{\mathbf{P}}_{n}^{\text{grid}}, \sum \mathbf{P}_{\sim n}^{\text{grid}}\}$

<figure><img src="image_9.png" /><figcaption>Fig. 6. Schematic of the proposed Split-Input architecture. Hats represent estimated quantities, bold notations are vectors over L time steps before t.</figcaption></figure>

and $s_{n,t}^{\text{measured}} = \{\hat{\mathbf{T}}_{n}^{\text{in}}, \hat{\mathbf{o}}_{n}\}$ (hat notations represent model estimates). Then, an actor-critic architecture is defined in which the actor only uses $s_{n,t}^{\text{measured}}$, while the critic uses the full input space (Pinto et al., 2017). This is called the “Split-Input” architecture in this work, and is shown in Fig. 6. The advantage is that the agent can learn a better policy with access to $s_{n,t}^{\text{measured}}$ in training while only using observable inputs in deployment.

It should be noted that only one ACM in $b_{n,m}$ is used in the training environment at any point of time, along with a “database” of the disaggregated $P_{n,t}^{\text{grid}}$ and $T_{t}^{\text{amb}}$. $\sum P_{\sim n,t}^{\text{grid}}$ is considered independent of $T_{n,t}^{\text{set,req}}$; hence, this is taken as the difference between the aggregate grid load $\sum_{n \in N} P_{n,t}^{\text{grid}}$ (from historical data) and $\hat{P}_{n,t}^{\text{grid}}$. The DRL agent is trained across a sequence of episodes, each of length 1 day. In each training episode, a random ACM $b_{n,m}$ is selected, and data from a random day is taken from the database. The agent interacts with this environment and uses the states, actions and rewards to update its parameters. By repeating this process over a large number of episodes,
<figure><img src="image_10.png" /><figcaption>Fig. 7. Visualization of reward function components.</figcaption></figure>

the agent learns a policy that generalizes to any element in $b_{n,m}$, and thus becomes robust to different consumer behaviours.

A single actor network is trained that sequentially accepts $s_{n,t}$ and outputs $a_{n,t} \forall n \in N$ during deployment. This is because RL-based approaches do not scale well if the dimensions of the state and action spaces depend on $N$ (due to the “curse of dimensionality” Charbonnier et al., 2022).<fnref n="4" /> Coordination across consumers is achieved through the reward function, which is described in detail in the next section.

### 3.2.4. Reward formulation

A sparse reward formulation is used to represent the objectives of the aggregator — i.e., the DRL agent only receives a reward at the end of the episode, after all control actions (i.e., $T_{n,t-1}^{\text{set,req}}$ to $T_{n,t-24}^{\text{set,req}}$) have been taken. The goal of the agent is to reduce the peaks in the aggregate power profile without increasing the total energy consumption compared to the default case (there is a tradeoff between these metrics in such systems, as discussed in Cole et al. (2014)). As such, the reward function consists of two terms - a peak reduction term $r^{\text{peak}}$, and a consumption term $r^{\text{cons}}$.

Aggregate peaks in the system occur when $P_{n,t}^{\text{elec}}$ and $P_{n,t}^{\text{ac}}$ for multiple consumers increase together, which depends on the weather and consumer demographics (Fan et al., 2017). We hypothesize that this effect

can be quantified by measuring the similarity between $P_{n,t-1:t-24}^{\text{grid}}$ and $\sum_{\sim n,t-1:t-24} P_{n,t}^{\text{grid}}$ (power profiles of the building and the aggregation over the past 24 h respectively). Consequently, $r^{\text{peak}}$ is designed to reduce the Pearson coefficient<fnref n="5" /> (PC) (used here as a measure of similarity) between the above terms as shown in Eq. (7). The tan function is used here to heavily penalize (reward) very high (low) values of the correlation, and $C_1$ is a positive scaling constant.

$$r^{\text{peak}} = C_1 \tan\left(\frac{-\pi \cdot \text{PC}(P_{n,t-1:t-24}^{\text{grid}}, P_{\sim n,t-1:t-24}^{\text{grid}})}{2}\right) \quad (7)$$

$r^{\text{cons}}$ is used to penalize agents that consume more energy than in the default case. Within each training episode, a baseline energy consumption ($E_n^b = \sum_t \hat{P}_{n,t}^{\text{grid}} \cdot \Delta t |T_{n,t}^{\text{set,req}} = 0|$) is first estimated. $r^{\text{cons}}$ is based on the ratio between the agent's consumption $E_n^{rl}(\sum_t \hat{P}_{n,t}^{\text{grid}} \cdot \Delta t |T_{n,t}^{\text{set,req}} = a_{n,t})$ and $E_n^b$ as shown in Eq. (8). Its value is negligible when this ratio is below a scaling constant $C_2$ and explodes to a high negative value above $C_2$.

$$r^{\text{cons}} = -\left[\frac{E_n^{rl}}{E_n^b} - C_2\right]^{10} \quad (8)$$

$$r_t = \begin{cases} r^{\text{peak}} + r^{\text{cons}}, & \text{end of training episode} \\ 0, & \text{otherwise} \end{cases} \quad (9)$$

The total reward $r_t$ at each time step is calculated as shown in Eq. (9). Fig. 7 plots the two terms in the reward function, as well as the heatmap of their summation. Since $r^{\text{peak}}$ and $r^{\text{cons}}$ are normalized, the same formulation and weights can be used for any $b_{n,m}$ that the agent encounters during the training process.

## 4. Results and discussion

To evaluate the performance of the agent, a test environment of $N = 100$ consumers was set up as described in the previous sections. The episode duration was fixed to 24 h (24 time steps), with a simulation time step of 15 min (each $T_{n,t}^{\text{set,req}}$ is applied for four simulation steps). The AG has access to two months of weather and smart meter data, covering the period from March–April, and the test period is 30 days in May. The data was first used to generate the set of ACMs $b_{n,m}$ through the process described in Section 3.2.2. The estimated ideal indoor temperature, $\hat{T}_{n,t}^{\text{ideal}}$, was discretized into 3 levels (i.e., $M = 3$) - {20 °C, 23 °C, 26 °C}, while the discomfort capacity $\hat{D}_n^{\text{max}}$ was sampled from a normal distribution $N(0.5, 0.25)$ °Ch. The duration of the input history window was set to $L = 8$. Then, the agents were trained offline for 2 million steps (equivalent to 21 000 simulated days or episodes), after which the trained actor network was deployed in the real system for the test phase. To control for the inherent randomness in RL-based solutions, every experiment described in this section was repeated for 3 random seeds, and the results were averaged. Typical Meteorological Year (TMY) weather data from Singapore was used in all experiments, accessed using the diypw Python library (Smith et al., 2021). In this work, a type of actor-critic algorithm called Proximal Policy Optimization (PPO) (Schulman et al., 2017), implemented via the stable-baselines3 library (Raffin et al., 2021),<fnref n="6" /> was used for its balance of performance and stability. The default hyperparameters are used in all cases, with one notable exception — the discount factor ($\gamma$) was set to 0.958 to align with the episode duration of 24 steps.<fnref n="7" /> In the reward function, a choice of $C_1 = 25$ and $C_2 = 0.3$ were empirically found to produce performant agents.

To study the impact of the Split-Input architecture shown in Fig. 6, a comparison was performed between DRL agents provided with different levels of training information in the input space as described

<footnote n="4">⁴ The collective state of the entire system has $N \cdot (5 \cdot L + 1)$ dimensions along with $N$ actions.</footnote>

<footnote n="5">⁵ zero-lag cross-correlation coefficient.</footnote>

<footnote n="6">⁶ Version 2.3.2.</footnote>

<footnote n="7">⁷ Here we use the heuristic $\frac{1}{1-\gamma} = 24$, and solve for $\gamma$.</footnote>
<caption>**Table 2**</caption>

Different DRL architectures considered in the study. Only actor inputs are required from the real system during deployment.

<table><thead><tr><th>Architecture</th><th>Actor inputs</th><th>Critic inputs</th></tr></thead><tbody><tr><td>Measured-Input</td><td>s<sup>measured</sup><sub>n,j</sub></td><td>s<sup>measured</sup><sub>n,j</sub></td></tr><tr><td>Split-Input</td><td>s<sup>measured</sup><sub>n,j</sub></td><td>s<sup>measured</sup><sub>n,j</sub> &cup; s<sup>predicted</sup><sub>n,j</sub></td></tr><tr><td>Full-Input</td><td>s<sup>measured</sup><sub>n,j</sub> &cup; s<sup>predicted</sup><sub>n,j</sub></td><td>s<sup>measured</sup><sub>n,j</sub> &cup; s<sup>predicted</sup><sub>n,j</sub></td></tr></tbody></table>

in Table 2. The default architecture proposed in this paper is the Split-Input architecture, in which only the critic is provided model-estimated inputs. In the Measured-Input case, intermediate model outputs are not used as agent inputs during the training process. Both the above architectures only require $s_{n,t}^{\text{measured}}$ during deployment. In contrast, the Full-Input architecture uses both $s_{n,t}^{\text{measured}}$ and $s_{n,t}^{\text{predicted}}$ both during training (model-estimated) and deployment. Since this requires $T_{n,t}^{\text{in}}$ and $o_{n,t}$ to be measured from the real system online, it is not realistic and is only used for comparison. In all three cases, a set of agents were also allowed to train directly on the reference system during the training period, bypassing the intermediate model fitting process. This is equivalent to using perfect models of the consumers for training — however, the consumption and weather data are still different across the training and test phases.

## 4.1. Baseline and alternate strategies for comparison

To compare the performance of the Split-Input approach, the following alternate strategies are considered:

1. **Baseline:** The fixed-setpoint strategy in which $T_{n,t}^{\text{set}} = T_n^{\text{in,ideal}} \forall n, t$ — it describes the situation where no flexibility is requested from the consumers.

2. **Precooling:** This strategy aims to shift and reduce the peaks by sending precooling signals ($T_{n,t}^{\text{set,req}} = -1 \forall n \in N$) 2 h before the expected peak loads, followed by high setpoint requests during the peak period ($T_{n,t}^{\text{set,req}} = 1 \forall n \in N$). From Fig. 2, it is visible that the peaks occur between 11 AM to 1 PM on weekends and 6 PM to 8 PM on weekdays — as such, precooling signals are sent at 9 AM on weekends and 4 PM on weekdays. The approach is analogous to time-of-use pricing approaches commonly used across the world (Nicolson et al., 2018).

3. **Decentralized Split-Input PPO (DSI-PPO):** A set of $N$ independent PPO agents, where each one is trained for a specific consumer $n \in N$. All agents use the Split-Input architecture, but each agent is trained only on the subset of $M$ ACMs corresponding to its assigned consumer. During deployment, $T_{n,t}^{\text{set}}$ is chosen independently for each consumer using their dedicated model. Compared to the nominal Split-Input case (i.e., centralized), this approach increases the parameter count and training time by a factor of $N$.

4. **Centralized PPO (C-PPO):** A single centralized PPO agent that simultaneously observes the entire state of the system and outputs an $N$-dimensional action, similar to the approach in Du et al. (2021). This agent takes only $s_{n,t}^{\text{measured}}$ as input, but is trained directly on the real system by looping the training data (i.e., ACMs are not used). It represents a hypothetical agent that learns an optimal policy through direct interaction with consumers, and is only presented for comparison purposes.<fnref n="8" /> The C-PPO agent uses a bigger DNN (with 4 times the number of parameters of the Split-Input network), and requires a significantly higher training time (~35 times higher than Split-Input).

## 4.2. Accuracy of ACM predictions

Fig. 8a and b provide a visualization of the prediction errors of the ACM over the test period, for the indoor temperature and AC power respectively. In both plots, the y-axes show the $M = 3$ values of $\hat{T}_n^{\text{in,ideal}}$ used to fit the ACM, while the x-axes are consumer indices $n$. The relatively high errors in Fig. 8a reflect the difference between $\hat{T}_n^{\text{in,ideal}}$ and $T_n^{\text{in,ideal}}$, since the AC keeps the indoor temperature close to the ideal value in the baseline policy. The error is systematically higher for $\hat{T}_n^{\text{in,ideal}} = 20$ °C than for 26 °C, suggesting that the actual $T_n^{\text{in,ideal}}$ is generally closer to the 26 °C. It should be noted that the AG has no way to know this, motivating the use of ACMs to train agents that are robust to any $T_n^{\text{in,ideal}}$. The MAE in the AC power is considerably lower, typically equivalent to around 15% of $P_n^{\text{ac,max}}$ for any $\hat{T}_n^{\text{in,ideal}}$. This is because the ACM parameters are fit by minimizing this error.

## 4.3. Peak reduction and effect on load profile

Fig. 9a compares the load duration curves achieved by different strategies over the test period, with annotations showing the fraction of energy consumed above an aggregated power of 200 kW. All the DRL agents, as well as the precooling strategy, are able to reduce the peak system load, with the Split-Input agent achieving a 20% reduction compared to the baseline. An additional benefit of the DRL control is that a larger fraction of the load is consumed at lower aggregate power. This is of importance since the duration of peak loads directly affects the period of time for which peaking power plants need to be activated, leading to higher costs and emissions (Van Orden and Simović, 2023).

The aggregate grid power ($\sum_n P_{n,t}^{\text{grid}}$) for an example weekend day is shown in Fig. 9b, while the corresponding control signals ($T_{n,t}^{\text{set,req}}$) are plotted in Fig. 9c (the DRL agents generate $N$ signals as every step $t$). Since the precooling signals are sent at the same time to all consumers, a new peak appears in that window due to the increased AC load. In contrast, by spreading out $T_{n,t}^{\text{set,req}}$ over time and across consumers, the DRL controllers are able to flatten the aggregated power profile more effectively. The C-PPO agent exhibits a bigger variance in its action selection, while the Split-Input and DSI-PPO agents learn a more direct strategy of precooling the buildings in the early hours of the morning to request high setpoints later in the day.

Table 3 displays the values of different objective metrics under each control strategy. Here, the Peak Energy Consumption is the energy consumed when $\sum_n P_{n,t}^{\text{grid}} > 200$ kW, and the Override Fraction shows the fraction of agent control signals that were overridden by consumers. A key observation is that the Split-Input agent achieves a similar level of performance to the C-PPO and DSI-PPO agents, despite having a significantly lower parameter count and training time.

The DRL-based strategies result in a 10%–20% higher override fraction compared to the baseline, which indicates that the peak reduction benefit comes at the cost of discomfort for the consumer. In comparison, the precooling strategy leads to much lower discomfort levels, which represents a major advantage. It must be noted that in all cases, the discomfort never exceeds the threshold value of $D_n^{\text{max}}$ as the consumer is allowed to override setpoint requests. Certain techniques, such as penalizing high setpoint requests in the reward function, may help to reduce the discomfort caused by the agent's actions.

Incentive schemes in real-world DR programs depend on factors such as the power mix, electricity markets and consumer tariff structures — as such, a detailed discussion is beyond the scope of this paper. Given the variation in discomfort encountered, any incentive offered to consumers must ensure that participants are compensated in proportion to their response to the control signals. The override feedback signal

<footnote n="8">⁸ Such an agent is not practical for real-world application due to the long training time (~4.5M steps, i.e., just over 500 years).</footnote>

<footnote n="9">⁹ Mean value of $P_n^{\text{ac,max}}$ across consumers in the real system is 1500 W.</footnote>
<figure><img src="image_11.png" /><figcaption>Fig. 8. (a) MAE of the indoor temperature (b) MAE of the AC power. Both are calculated over the test period under the baseline policy.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>Fig. 9. (a) Load duration curves (annotations are the fraction of energy consumed above the highlighted power) (b) Aggregated grid power for a typical day (c) Control signals under each strategy (mean and std deviation across all consumers).</figcaption></figure>

<table><caption>**Table 3**

Evaluation metrics for different strategies (Mean and Std for Split-Input).</caption><thead><tr><td>Strategy</td><td>Peak energy Cons (MWh)</td><td>Total energy Cons (MWh)</td><td>Override fraction</td></tr></thead><tbody><tr><td>Baseline</td><td>5.002</td><td>70.723</td><td>0.0</td></tr><tr><td>Precooling</td><td>4.843</td><td>70.640</td><td>0.019</td></tr><tr><td>C-PPO</td><td>2.139</td><td>72.252</td><td>0.104</td></tr><tr><td>DSI-PPO</td><td>2.303</td><td>70.837</td><td>0.199</td></tr><tr><td>Split-Input</td><td>2.051 (0.329)</td><td>70.915 (0.339)</td><td>0.189 (0.017)</td></tr></tbody></table>

$(o_{n,t})$ would serve as a suitable mechanism to measure this response — in its absence, the task of identifying the most responsive consumers is very difficult, and is the subject of open research. Using the weights in the reward function, the proposed approach may be adapted to meet the needs of the AG depending on the context.

### 4.4. Sensitivity to discomfort models

The results in 4.3 assume that the override model in the test system follows Eq. (3), where overrides occur through a deterministic process based only on thermal comfort. Recent studies suggest that real-world override behaviours are stochastic processes (Kaspar et al., 2024), and

may depend on several other factors (Sarran et al., 2021; Khorasani Zadeh et al., 2023). To evaluate the robustness of the approach to unseen override models, the Split-Input agent (trained using the comfort-based deterministic override model, denoted as CB-Det) was tested against the following alternate override models.

1. **No Overrides**: The consumers accept all setpoint requests from the agent, without any overrides.

2. **Comfort-Based Stochastic (CB-Sto)**: A stochastic reformulation of the thermal comfort-based override model in Eq. (3), with $p(o_{n,t} = 1) = \max(\frac{D_{n,t}}{D_{n,t}^{max}}, 1.0)$. This model is analogous to
<table><caption>**Table 4**
Performance comparison across override models.</caption><thead><tr><th>Override model</th><th>Peak energy Cons (MWh)</th><th>Total energy Cons (MWh)</th><th>Override fraction</th></tr></thead><tbody><tr><td>Baseline</td><td>5.002</td><td>70.723</td><td>0.0</td></tr><tr><td>No Overrides</td><td>0.503 (0.087)</td><td>61.664 (1.109)</td><td>0.0</td></tr><tr><td>CB-Det</td><td>2.051 (0.329)</td><td>70.915 (0.339)</td><td>0.189 (0.017)</td></tr><tr><td>CB-Sto</td><td>3.040 (0.526)</td><td>71.630 (0.245)</td><td>0.313 (0.016)</td></tr><tr><td>MF-Sto</td><td>1.350 (0.172)</td><td>68.583 (0.502)</td><td>0.447 (0.002)</td></tr></tbody></table>

<figure><img src="image_13.png" /><figcaption>**Fig. 10.** Consumer override patterns under different override models with the Split-Input agent. CB-Det represents the deterministic model used to train the agent.</figcaption></figure>

the indoor temperature-dependent probabilistic model presented in Kaspar et al. (2024).

3. **Multifactor Stochastic (MF-Sto):** A stochastic decision-tree model in which the leaf nodes represent override probabilities. It is based on the analysis presented in Sarran et al. (2021), where real-world DR overrides were found to depend on (in order of feature importance) a consumer's level of interaction with their thermostats,<fnref n="10" /> ambient and indoor temperatures, and the number and intensity of recent setpoint requests.

**Fig. 10** shows the fraction of overrides under the Split-Input policy over the test period for different override models. In the thermal comfort-based models, overrides are strongly concentrated during the late afternoon, with almost all the signals being overridden at around 6 PM. In contrast, in the multi-factor model, overrides are almost equally distributed throughout the day, which is a pattern completely unseen during the agent training. It should be noted that the total number of overrides are comparable in all cases.

**Table 4** shows the mean and standard deviation of evaluation metrics under each override model across multiple runs with different random seeds. In all cases, the DRL agent is able to reduce peak loads (between 20%–90%) without significantly changing the total energy consumption. This suggests that the learned policy is applicable to stochastic override behaviours that may occur in the real world even if it is trained on a deterministic model. The best metrics occur when overrides are absent in the system — in this study, the presence of overrides degrades the peak reduction performance of the agent by around 30% on average, which is consistent with the findings in Kaspar et al. (2024).

When overrides are present, the benefits of the methodology strongly depend on the nature of the override decision process. For

the comfort-based models used in this work, the stochastic reformulation reduces the agent's effectiveness and leads to higher energy consumption (both peak and total), as well as a bigger override fraction. Surprisingly, a higher fraction of overrides does not necessarily imply a degradation in the objective metrics. In the multifactor override model, the agent achieves significantly better metrics (50% lower peaks) compared to the comfort-based case, despite a higher fraction of overrides (10% higher). This can be attributed to the difference in override distributions resulting from different models — in the comfort-based override models, higher peak loads occur due to the concentrated nature of override signals, as seen in Fig. 10. In this sense, the multifactor model may be viewed as a stochastic extension of the case without overrides in which around 45% of the agent's actions are overridden (leading to a 20% increase in peak loads.).

### 4.5. Impact of actor-critic architecture and ACM accuracy

**Fig. 11** shows violin plots of $\sum_n P_{n,t}^{\text{grid}}$ for different control strategies over the test period. For DRL-based approaches, the results from using perfect models are plotted alongside the ones based on ACMs, denoted as “Perfect Models” and “ACM” respectively. The choice of architecture has a significant impact on the performance of the agent. Compared to the baseline, the peak reduction performance improves when using the Split-Input architecture, and approaches that of the Full-Input case (where behind-the-meter information is accessible from the real system). This suggests that the proposed architecture represents a good trade-off between performance and input data requirements. Notably, since all the DRL architectures share the same reward function, it is possible to use the technique even when there are different levels of data access for different consumers by deploying multiple agents.

Another observation is that under the Measured-Input case, the energy consumption is lower and a higher fraction of the load occurs at lower power levels (e.g., below 75 kW). While it may seem like a positive outcome at first glance, this actually happens when agents adopt a conservative strategy of sending high setpoints more often to exploit any available flexibility (discussed subsequently). This may reduce the energy consumption, but it causes a higher level of discomfort to consumers.

The peak reduction performance of the agent improves with the accuracy of the models, with a systematic improvement of around 3% in all cases compared to using approximate models (Fig. 11). This suggests that the agents are able to learn competent policies using the ACMs despite their inaccuracy, since the problem dynamics are well-captured. Improving the model accuracy can then help to fine-tune the policies to suit specific consumer groups. Since the relative improvement is nearly the same across all RL architectures, it can be inferred that the fine-tuning effect results from accuracy improvements in inputs that are shared among all architectures (in this case, only the load $P_{n,t}^{\text{grid}}$).

Another interesting observation is that the performance achieved with perfect models is nearly identical in the Split-Input and Full-Input cases. This implies that by utilizing the actor-critic architecture described in this paper and improving the model accuracy, the need to obtain real measurements of behind-the-meter data during the deployment phase can be avoided. In addition to saving on the economic costs of implementing extra sensing infrastructure, such an approach is also more desirable for the consumer, since their data privacy is respected.

<footnote n="10">¹⁰ Here, consumers were uniformly distributed into 3 interaction levels — low, medium and high.</footnote>
<figure><img src="image_14.png" /><figcaption>Fig. 11. Violin plots of the aggregate grid power for different strategies (annotations show total energy consumption over the test period).</figcaption></figure>

The results also highlight the effectiveness of the domain random-
ization techniques employed in this work. By training a single Split-
Input agent on an ensemble of approximate consumer models — despite
their limited accuracy, as shown in Fig. 8 — the agent is able to learn
a generalized policy that captures common behavioural patterns across
consumers. This shared training approach enables stronger generaliza-
tion than the DSI-PPO agent, which relies on separate, personalized
agents for each consumer but fails to outperform the Split-Input base-
line. Similar findings have been reported in prior work on multi-agent
RL (MARL) (Kazmi et al., 2019), where shared policy training has
proven beneficial.

While domain randomization in this work is used primarily to
account for variations in consumer behaviour, the same technique can
be extended to model a broader range of real-world uncertainties,
such as sensor noise, occupancy variation, and communication delays,
to enhance the robustness of the trained policies. Such application-
dependent extensions have been explored in other domains including
robotics (Zhao et al., 2020), leader-follower systems (Tan et al., 2025)
and electric vehicle charging (Lei et al., 2025a). The general framework
proposed here (offline training using informed critics on an ensemble
of approximate models) is agnostic to model fidelity and can be readily
adapted to other domains that require sim-to-real transfer.

4.6. Sensitivity to disaggregation errors

As discussed in 3.2.1, it is assumed here that it is possible to
perfectly disaggregate the building and AC loads from historical data.
This information is used at two key stages in the methodology — first,
during ACM thermal parameter fitting, and second, for the consumer
electrical loads used in the training environment. To analyse the im-
pact of imperfect disaggregation on the outcomes achieved using this
approach, we systematically introduce noise of different levels (Zhang
et al., 2024) with a parameter $\epsilon$, and train Split-Input agents using
the noisy data. Here, for the RC parameters, it is used multiplicatively
(for e.g., $\delta R_n^{in} = R_n^{in} \times \epsilon$) and for the time-series load profiles, it is
used additively (sampling from $N(0, \text{mean}(P_n^{grid}) \times \epsilon)$). The evaluation
metrics under different values of $\epsilon$ are given in Table 5.

All agents were able to learn strategies that reduce peak loads
compared to the baseline case, suggesting that the proposed approach is
not severely affected by disaggregation errors. Agents that were trained
on noisy data perform slightly worse than those trained under perfect
disaggregation with 10% higher peak loads and 5% more overrides on
average. The architecture and training approach used here, based on
grey-box models, encourage agents to learn generalizable policies based
on system-level dynamics.

These experiments demonstrate the robustness of the proposed ap-
proach to model errors arising from imperfect disaggregation. In prac-
tice, non-intrusive load monitoring (NILM) errors can originate from
multiple sources, including overlapping appliance signatures and lim-
ited sensor resolution (Batra et al., 2014). This work assumes the
existence of a disaggregation algorithm and focuses solely on down-
stream control performance. Additionally, the disaggregation is not
required during the deployment phase since the actor networks operate
directly on meter data. As such, a comprehensive evaluation of various
disaggregation methods and their impact on agent performance is
beyond the scope of this study.

4.7. Analysis of the split-input architecture

Fig. 11 shows that the Split-Input architecture improves evaluation
metrics compared to the Measured-Input case despite both using an ac-
tor with a reduced input space. To understand how providing additional
inputs to the critic enables the training of a better actor network, we
study the impact of this paradigm on the training process, the hidden
activations of the DNNs, and on the final learnt policy.

4.7.1. During agent training

Fig. 12 shows the mean rewards and the Explained Variance (EV)
score (of the critic network) for different architectures over the training
duration. The EV score, calculated here between the estimated and ac-
tual cumulative rewards obtained by the agent, is a numerical measure
of the accuracy of the critic network (EV∈ [-1, 1], higher values are
better). While both architectures have a similar training reward curve,
the EV score for the Split-Input architecture is considerably higher,
suggesting that the critic in this case is able to learn a better estimate
of the true reward function.

In actor-critic RL algorithms, the output of the critic is used to
calculate the gradients used to update the actor network (Sutton and
Barto, 2020). As such, an accurate critic reduces the variance of these
gradient steps, guiding the actor towards a more stable policy. It should
be noted that the training reward in this case is calculated from an
approximate model of the real system. It is possible for two policies to
achieve similar rewards in the training environment while performing
differently in the actual system.

4.7.2. Visualizing hidden activations of the DNNs

Both the actor and critic are parameterized by DNNs in this study
(with 2 hidden layers of 64 neurons), and the outputs of their hidden
layers encode the key features used by the networks during infer-
ence (Pomerleau, 1989). Fig. 13 shows a coloured t-SNE projection (van
<table><caption>**Table 5**
Effect of noise due to disaggregation errors.</caption><thead><tr><th>Noise level</th><th>Peak energy Cons (MWh)</th><th>Total energy Cons (MWh)</th><th>Override fraction</th></tr></thead><tbody><tr><td>Baseline</td><td>5.002</td><td>70.723</td><td>0.0</td></tr><tr><td>No noise (ε = 0.0)</td><td>2.051 (0.329)</td><td>70.915 (0.339)</td><td>0.189 (0.017)</td></tr><tr><td>5% (ε = 0.05)</td><td>2.441 (0.171)</td><td>70.256 (0.372)</td><td>0.222 (0.028)</td></tr><tr><td>27.5% (ε = 0.275)</td><td>2.433 (0.630)</td><td>70.100 (0.332)</td><td>0.237 (0.033)</td></tr><tr><td>50.0% (ε = 0.5)</td><td>2.636 (0.149)</td><td>70.261 (0.066)</td><td>0.236 (0.009)</td></tr></tbody></table>

<figure><img src="image_15.png" /><figcaption>**Fig. 12.** Mean training rewards and Explained Variance (EV) scores for different agents during training. EV is a measure of the accuracy of the critic output (higher values are better, maximum value is 1.0).</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>**Fig. 13.** t-SNE plots of hidden activations of each neural network for different states. Colours indicate whether an override occurred in the previous time step (Yes, No). s is the silhouette score, with higher values indicating a better separation between the labels.</figcaption></figure>

der Maaten and Hinton, 2008) of the 64-dimensional hidden layer activations for different DNNs for a sample of 2000 states, equally split between those with $o_{n,t-1} = 1$ (orange) and $o_{n,t-1} = 0$ (blue). If the states belonging to each label are well-separated in the projection space (measured by the silhouette score, $s \in [-1, 1]$, where higher values indicate better separation), it is a clear indication that the feature plays an important role in the DNN output. It should be noted that the actor and critic share a common embedding space in the Measured-Input case, since they have the same inputs, and are therefore represented by one panel in Fig. 13.

The Split-Input critic attaches a high importance ($s = 0.33$) to the presence of overrides when estimating the value function compared to the Measured-Input case ($s = 0.20$), as evidenced by the significantly higher silhouette score for this feature. This indicates that one of the reasons for the improved critic accuracy noted in Fig. 12 is that the critic is able to better approximate the long-term effects of overrides on the cumulative reward at the end of the training episode. Interestingly,

<figure><img src="image_17.png" /><figcaption>**Fig. 14.** Visualization of agent policies in Split-Input and Measured-Input architectures for different weather. Subplot x- and y-axes show mean values of $\hat{P}_{n}^{\text{grid}}$ and $\sum \hat{P}_{-n}^{\text{grid}}$ respectively. Annotations are the fraction of the decision space where $|T_{n,t}^{\text{set,req}}| < 0.5$, used here as a measure of policy entropy.</figcaption></figure>

this marginally *reduces* the importance attached by the Split-Input actor to this feature ($s = 0.15$) compared to the Measured-Input actor. This could reflect the fact that while overrides play a critical role in estimating the value function (i.e., the critic output), the optimal policy (i.e., the actor) should focus on other factors to maximize the achieved reward. The absence of additional inputs thus leads to two effects — the Measured-Input critic output has a higher variance as it is unable to infer override behaviours from limited data, while the actor slightly overestimates their potential impact.

### 4.7.3. Implications at the policy level

Under the Measured-Input and Split-Input paradigms, the agent's decision depends on the building load ($\hat{P}_{n}^{\text{grid}}$), the aggregate load of the other buildings ($\sum \hat{P}_{-n}^{\text{grid}}$), the weather ($\hat{T}_{\text{amb}}$) and the time of day ($t$). Fig. 14 visualizes the policy in each actor-critic architecture (columns) under different weather conditions (rows). Each subplot shows the value of $T_{n,t}^{\text{set,req}}$ as a function of the mean $\hat{P}_{n}^{\text{grid}}$ and $\sum \hat{P}_{-n}^{\text{grid}}$. Numerical annotations in the subplots show the fraction of the decision space where $|T_{n,t}^{\text{set,req}}| < 0.5$, used here as a measure of policy entropy.

It is clear that the agents learn a similar mapping in both cases, with higher setpoints being requested as the building load and the aggregate load increase. In the Measured-Input case, a larger fraction of the decision space is assigned to higher setpoints, and the decision boundary is less sharp compared to the other case (with about 15% higher entropy). This illustrates the conservative policy adopted by the agent in the absence of additional context provided by the model-estimated inputs. By requesting higher setpoints more often, the agent runs the risk of more overrides (and higher consumer discomfort), which affects its peak reduction performance. In contrast, the policy learnt with the Split-Input architecture has a better distribution between actions, as well as a sharper decision boundary. It is clear that the agent is able to use the additional inputs to obtain a clearer understanding of the system dynamics, and adjust its policy to obtain a better outcome.
<table><caption>**Table 6**
Runtime for different stages.</caption><thead><tr><td>Task</td><td>Cumulative</td><td>Per unit</td><td>#Units</td></tr></thead><tbody><tr><td>Generate ACMs</td><td>36 h55 m ± 12 m15 s</td><td>11 m4 s ± 40 s</td><td>100 ACMs</td></tr><tr><td>Train Split-Input agent</td><td>37 m52 s ± 55 s</td><td>37 m52 s ± 55 s</td><td>1 agent (11 650 parameters)</td></tr><tr><td>Train C-PPO agent</td><td>19 h57 m21 s</td><td>19 h57 m21 s</td><td>1 agent (43 521 parameters)</td></tr></tbody></table>

This is supported by the dichotomy in the agents' response to changing weather signals. In the Measured-Input case, as the ambient temperature increases (downwards in Fig. 14), the decision boundary moves leftwards, increasing the chance for higher setpoints to be requested. This follows the intuitive notion that cooling loads are proportional to the ambient temperature — as such, the agent tries to reduce this load using higher setpoints. Interestingly, the agent with the Split-Input architecture learns the opposite — hotter weather pushes the decision boundary rightwards, giving a higher chance for low setpoints to be requested. This may be because the agent learns that consumers face a higher level of discomfort during hot weather, and have a higher probability of overriding the requested signals. The policy thus allows for more aggressive precooling during hot weather, in the expectation that consumers will be more amenable to higher setpoints later in the day. This results in a trade-off between energy consumption and peak reduction, as visible in Fig. 11.

### 4.7.4. Theoretical justification

While the analysis in this section focuses on empirical metrics, we note that the findings align with the theoretical justification for asymmetric actor-critic architectures presented in Lambrechts et al. (2025). In that work, the authors prove that the addition of more information to the critic improves its accuracy by eliminating a specific source of error (termed “aliasing error”). They also prove that a more accurate critic is theoretically guaranteed to result in a more optimal policy, due to a stronger and more stable gradient signal to update the actor. However, this is only true if the actor has sufficient inputs to take advantage of this signal, failing which the optimality of the policy is limited by the error of projecting critic gradients (calculated using a higher-dimensional input space) into the actor space. Finally, while Lambrechts et al. (2025) assumes that the actor and critic use linear models, a separate proof is presented in Tian et al. (2023) for the convergence of actor-critic RL with deep neural networks, which are used in the present work.

### 4.8. Runtime, reproducibility and robustness

The typical runtime for each step in the solution is tabulated in Table 6 for experiments performed on a laptop with an AMD Ryzen 9 5900HS processor and 24.0 GB of RAM. It is evident that the model generation takes the longest cumulative time — however, the runtime may decrease significantly with parallelization across multiple CPUs, which was not done in this study. The negligible runtime of the deployment step (~500 µs), which requires just a forward pass through a deep neural network, demonstrates the scalability of the solution. Since modern deep learning libraries are optimized to process inputs in batches, the marginal cost of performing multiple inferences is very low.

Table 6 also highlights a key advantage of the architecture proposed in this paper — the time required to train an agent is 40 times lower than that required for a fully centralized architecture, and requires only a quarter of the parameter count. Additionally, since the input and output dimensions in the C-PPO case depend on N, the inclusion (or removal) of even a single participant from the scheme would necessitate the training of a new agent. The choice of sharing network weights while making decisions using each consumer's local state eliminates this problem, which greatly simplifies real-world deployment. In comparison, the decentralized DSI-PPO approach requires N times

<figure><img src="image_18.png" /><figcaption>**Fig. 15.** (a) Avg episode reward in training and (b) Violin plots of the aggregated load for 10 runs of the split-input architecture with different random seeds. $\Delta$ is the percentage peak reduction compared to the baseline.</figcaption></figure>

the number of parameters and training time compared to the Split-Input case, which may become prohibitively expensive as N increases. Additionally, this approach raises privacy concerns, since consumers might not be willing to allow their data to be used to train and deploy targeted agents.

An important concern with RL based solutions is the issue of reproducibility, due to the inherent variance in most algorithms and stochasticity in the environments. While several recommendations in Lynnerup et al. (2020) have been followed in this work to encourage reproducibility, the robustness of the method to random variations can only be quantified empirically. Fig. 15a and b show the average rewards obtained during training and violin plots of the aggregated load in the test period respectively for 10 repeated runs of the Split-Input architecture experiment, each with different random seeds.<fnref n="11" /> The learning process of the agent is relatively stable since the rewards converge to the same high value within the training period irrespective of the seed. The agents also consistently reduce peaks compared to the baseline, with only a 5% deviation between the best and worst results. The policies achieve a similar load distribution with a sharp decrease in the energy consumption above 200 kW. This suggests that the PPO-based training approach used in this paper is robust to algorithmic variance.

## 5. Conclusion and future scope

A novel demand response scheme based on the co-ordinated control of air-conditioner setpoints in a group of residential buildings is proposed, based on grey-box modelling and actor-critic reinforcement

<footnote n="11">¹¹ This changes the order in which models are sampled from $b_{n,m}$ during training, in addition to algorithmic randomness. All runs still use the same data.</footnote>
learning. A key feature is that consumers are allowed to freely override the setpoint signals sent by the DRL agent based on their thermal comfort, preserving their privacy and promoting interaction. Using a novel Split-Input architecture, the approach is able to achieve a significant peak reduction (of up to 20% from the baseline) in the aggregated electrical consumption, while eliminating the need for “behind-the-meter” information such as indoor temperature signals. The impact of this architecture was investigated by visualizing the agents’ policies. The sensitivity of the solution to the accuracy of the models and the quality of the training data was quantified, and the impact of the signals on the consumers’ thermal comfort was studied.

The main factors affecting the peak reduction performance are the actor-critic architecture and inputs, while the model accuracy has a fine-tuning effect. The use of interpretable consumer models that respect the temperature dynamics of the system resulted in agents that were robust to different consumer behaviours, and generalized well to the problem. The final policies were more stable and reflective of a deeper understanding of the system despite having limited access to data. The performance of the methodology also depends on external factors, specifically on the distribution of overrides over the day. Adversarial patterns, such as a high concentration of overrides during specific periods of the day, inherently reduce the ability of the agent to reduce peaks. For a given distribution, it was found that stochasticity in the override model further degrades the DRL agent performance by around 5%. However, the learnt policy remains applicable to different override models, and reduce peak loads by at least 20% in all cases. Finally, it was found that while the control signals are able to decorrelate the consumption profiles and reduce aggregate peaks, certain consumers faced a significant increase in their discomfort, although within their discomfort capacity.

The results of the study hinge on two key assumptions about the system — the AC operation model and the possibility of load disaggregation. While the same AC model is widely used in literature, and may be sufficient to describe residential cooling systems, the impact of different technologies (such as inverter-based ACs) needs to be studied in future work. Disaggregation of smart-meter data into HVAC and non-HVAC components is also not a trivial problem, and is an open research topic. However, for the particular case here, where historical data of both load and weather are available, along with a known operation model of the AC, recent work has achieved promising results (Liang et al., 2019; Liang and Ma, 2020; Qi et al., 2020). A noise-sensitivity analysis shows that the disaggregation errors may reduce the agent performance by around 10% - however, the approach still results in a significant peak load reduction compared to the baseline in all cases.

While the present study relies on the use of grey-box models to capture the system physics, other approaches may result in better performance. Future work should focus on alternate model creation strategies, such as Physics-Informed Neural Networks (Lei et al., 2025b), and provide a more detailed analysis of the most important inputs to enhance the performance of the RL agents. Another limitation of the current work is that a detailed comparison is not provided other advanced control algorithms such as Model Predictive Control. While the literature review highlights the difficulty of scaling such techniques to hundreds of consumers, future work should investigate alternate control approaches, and quantify the relative tradeoffs associated with each technique. Finally, while the present work highlights the importance of the override model on agent performance, future work should focus on online adaptation techniques to adapt the RL policy to real-world override signals in a data-driven manner.

## CRediT authorship contribution statement

**Sharath Ram Kumar:** Writing – review & editing, Writing – original draft, Software, Methodology, Investigation, Data curation, Conceptualization.
**Arvind Easwaran:** Writing – review & editing, Supervision, Investigation, Conceptualization.
**Benoit Delinchant:** Validation, Supervision, Methodology, Conceptualization.
**Remy Rigo-Mariani:** Writing – review & editing, Supervision, Methodology, Investigation.

## Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## Acknowledgements

This research is part of the programme DesCartes and is supported by the National Research Foundation, Prime Minister's Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme.

## Appendix. Description of reference system

### A.1. Generation of appliance load dataset

The appliance loads are pre-calculated using the synthetic load generation software 'LoadProfileGenerator' (Pflugradt et al., 2022). The tool performs bottom-up simulations of household activities based on a desire-driven behavioural simulation using a psychological model. It is capable of generating detailed time-series data based on inputs such as household composition, occupants, appliances and type of day. The load profiles in this study were generated using the templates provided in the tool, which are reflective of residents in Germany (it is assumed that residential appliance load profiles are similar in tropical countries such as Singapore). The synthetic profiles cover a period of 1 year at a resolution of 15 min, for a group of 100 households. An overview of the composition of the dataset is given in Table A.7.

The distribution of household types is representative of the population statistics released by the Singapore Department of Statistics.<fnref n="12" /> It should be noted that the loads within a given household template are not identical due to the stochastic nature of the simulation; instead, they may share consumption patterns.

### A.2. Calculation of cooling loads in the reference system

The cooling loads of a consumer are calculated through a combination of a thermal model of the building, an air conditioner model and a thermal comfort model. The AC is modelled as a thermostatically controlled device with a single speed compressor, that operates as described in Section 2.2. The thermal comfort of the consumer is represented using a discomfort capacity model as explained in Section 2.1. The maximum discomfort values were randomly set between 0.025 °C and 0.65 °C, with a mean and standard deviation of 0.24 ± 0.14 °C. This matches the values found empirically in Ryu et al. (2020) for a group of residents in Daegu, Korea. The preferred indoor temperatures were sampled from a normal distribution, i.e., $T_{n}^{in,ideal} \sim N(24, 2)°C$ which is reflective of typical preferences in tropical climates.

Since the study requires the dynamic simulation of the temperature evolution of a large group of buildings, existing tools such as Energy-Plus are not feasible due to time and resource constraints. As such, a 4R2C thermal equivalent circuit is used to calculate the heat dynamics of the building, with the structure shown in Fig. A.16. The model and its implementation follow the explanation in Harb et al. (2016), who evaluated different RC thermal models based on their ability to accurately capture the dynamics of a real building. Once the model parameters and $T_{n}^{in,ideal}$ are fixed, $P_{n}^{ac,max}$ is set sufficiently high to meet the cooling demand on the hottest day (highest average $T^{amb}$) in the weather data.

The model considers the impact of ambient temperature $T^{amb}$, solar irradiation $Q^{irr}$ and cooling power $\Phi^{ac}$ on the indoor temperature $T^{in}$.

<footnote n="12">¹² https://tablebuilder.singstat.gov.sg/table/TS/M810651.</footnote>
<table><caption>**Table A.7**
Composition of simulated households in LoadProfileGenerator.</caption><thead><tr><th rowspan="2">Household template</th><th rowspan="2">Count</th><th colspan="3">Load statistics (kW)</th><th colspan="3">Energy consumption distribution (%)</th></tr><tr><th>Avg</th><th>Std</th><th>Peak</th><th>12 AM-8 AM</th><th>8 AM-4 PM</th><th>4 PM-12 AM</th></tr></thead><tbody><tr><td>CHR05 2 working parents, 3 children</td><td>5</td><td>0.71</td><td>1.08</td><td>9.58</td><td>17</td><td>34</td><td>49</td></tr><tr><td>CHR27 2 working parents, 2 children</td><td>10</td><td>0.57</td><td>0.95</td><td>12.68</td><td>15</td><td>32</td><td>53</td></tr><tr><td>CHR03 2 working parents, 1 Child</td><td>30</td><td>0.54</td><td>0.84</td><td>8.94</td><td>19</td><td>30</td><td>51</td></tr><tr><td>CHR15 2 working parents, 2 children, 2 seniors</td><td>9</td><td>1.22</td><td>1.42</td><td>13.08</td><td>16</td><td>50</td><td>33</td></tr><tr><td>CHR01 2 working parents, No children</td><td>10</td><td>0.49</td><td>0.74</td><td>12.31</td><td>21</td><td>33</td><td>46</td></tr><tr><td>CHR02 2 working parents, No children</td><td>11</td><td>0.35</td><td>0.66</td><td>8.25</td><td>20</td><td>36</td><td>44</td></tr><tr><td>CHR22 1 working parent, 1 child</td><td>7</td><td>0.21</td><td>0.35</td><td>5.52</td><td>25</td><td>33</td><td>42</td></tr><tr><td>CHR06 1 unemployed adult</td><td>4</td><td>0.28</td><td>0.35</td><td>7.57</td><td>23</td><td>41</td><td>35</td></tr><tr><td>CHR29 1 working adult</td><td>7</td><td>0.22</td><td>0.4</td><td>5.65</td><td>32</td><td>25</td><td>43</td></tr><tr><td>CHR31 1 retired adult</td><td>7</td><td>0.29</td><td>0.39</td><td>4.88</td><td>31</td><td>31</td><td>37</td></tr></tbody><tfoot><tr><td>Total</td><td>100</td><td>51.24</td><td>40.00</td><td>264.23</td><td>19</td><td>36</td><td>45</td></tr></tfoot></table>

<table><caption>**Table A.8**
Parameter ranges for 4R2C models.</caption><thead><tr><th></th><th>C<sup>in</sup> (Wh/K)</th><th>C<sup>e</sup> (Wh/K)</th><th>R<sup>in,a</sup> (W/K)</th><th>R<sup>ia,a</sup> (W/K)</th><th>R<sup>ia,e</sup> (W/K)</th><th>R<sup>e,a</sup> (W/K)</th></tr></thead><tbody><tr><th>Mean</th><td>4000</td><td>4000</td><td>0.006</td><td>0.015</td><td>0.0009</td><td>0.001</td></tr><tr><th>Std</th><td>1000</td><td>1000</td><td>0.003</td><td>0.003</td><td>0.0005</td><td>0.0003</td></tr><tr><th>Min</th><td>1000</td><td>1000</td><td>0.0005</td><td>0.0005</td><td>0.0001</td><td>0.0005</td></tr><tr><th>Max</th><td>100 000</td><td>500 000</td><td>0.05</td><td>0.05</td><td>0.01</td><td>0.05</td></tr></tbody></table>

<figure><img src="image_19.png" /><figcaption>**Fig. A.16.** 4R2C model network (Harb et al., 2016).</figcaption></figure>

and the external wall temperature $T^e$. Notably, the layer of indoor air adjacent to the wall is treated as a separate node with a temperature $T^{ia}$, and the effect of $Q^{irr}$ on the external wall is captured by an equivalent temperature $T^{amb,eq}$. The physical characteristics of the wall are represented by $R^{e,a}$ and $C^e$ - in this work, concrete walls with a thickness of 150 mm and a height of 2.6 m are used to calculate typical values for these parameters. The effects of ventilation and poor insulation are captured by the shunt resistance $R^{in,a}$. The thermal capacitance of the indoor zone, $C^{in}$, captures the effects of objects inside the building, such as floors and furnishings. The heat transfer between $T^{ia}$ and $T^{in}$ is modelled as a convective process, moderated by the parameter $R^{in,ia}$. The parameter values used for the buildings are sampled from the ranges given in Table A.8, which reflect plausible ranges based on the interpretations mentioned above.

As a validation of the used parameters, the time constant $\tau_n$ of the final thermal networks were calculated using the methodology in John et al. (2018), in which a study of 10,000 real buildings in North America is performed. It was found that the distribution of $\tau_n$ obtained from the models closely match the values reported from the study, with a peak between $\tau^n = 20h$ and $\tau^n = 40h$. Fig. A.17 depicts the temperature evolution of a few example buildings (for the same setpoint and AC power), clearly showing the impact of the different thermal model parameters.

<figure><img src="image_20.png" /><figcaption>**Fig. A.17.** Cooling power and indoor temperatures for different buildings in the reference system.</figcaption></figure>

## Data availability

Data will be made available on request.

## References

Bashash, S., Fathy, H.K., 2013. Modeling and control of aggregate air conditioning loads for robust renewable power management. IEEE Trans. Control Syst. Technol. 21 (4), 1318-1327. http://dx.doi.org/10.1109/TCST.2012.2204261.

Batra, N., Parson, O., Berges, M., Singh, A., Rogers, A., 2014. A comparison of non-intrusive load monitoring methods for commercial and residential buildings. http://dx.doi.org/10.48550/arXiv.1408.6595, arXiv:1408.6595.
Behboodi, S., Chassin, D.P., Djilali, N., Crawford, C., 2018. Transactive control of fast-acting demand response based on thermostatic loads in real-time retail electricity markets. Appl. Energy 210, 1310–1320. http://dx.doi.org/10.1016/j.apenergy. 2017.07.058.

Benetti, G., Caprino, D., Della Vedova, M.L., Facchinetti, T., 2016. Electric load management approaches for peak load reduction: A systematic literature review and state of the art. Sustain. Cities Soc. 20, 124–141. http://dx.doi.org/10.1016/j. scs.2015.05.002.

Biagioni, D., Zhang, X., Adcock, C., Sinner, M., Graf, P., King, J., 2024. Comparative analysis of grid-interactive building control algorithms: From model-based to learning-based approaches. Eng. Appl. Artif. Intell. 133, 108498. http://dx.doi.org/ 10.1016/j.engappai.2024.108498.

Broberg, T., Persson, L., 2016. Is our everyday comfort for sale? preferences for demand management on the electricity market. Energy Econ. 54, 24–32. http://dx.doi.org/10.1016/j.eneco.2015.11.005.

Celik, B., Roche, R., Suryanarayanan, S., Bouquain, D., Miraoui, A., 2017. Electric energy management in residential areas through coordination of multiple smart homes. Renew. Sustain. Energy Rev. 80, 260–275. http://dx.doi.org/10.1016/j.rser. 2017.05.118.

Charbonnier, F., Morstyn, T., McCulloch, M.D., 2022. Scalable multi-agent reinforcement learning for distributed control of residential energy flexibility. Appl. Energy 314, http://dx.doi.org/10.1016/j.apenergy.2022.118825, arXiv:2203.03417.

Cimen, H., Cetinkaya, N., Vasquez, J.C., Guerrero, J.M., 2021. A microgrid energy management system based on non-intrusive load monitoring via multitask learning. IEEE Trans. Smart Grid 12 (2), 977–987. http://dx.doi.org/10.1109/TSG.2020. 3027491.

Claessens, B.J., Vanhoudt, D., Desmedt, J., Ruelens, F., 2018. Model-free control of thermostatically controlled loads connected to a district heating network. Energy Build. 159, 1–10. http://dx.doi.org/10.1016/j.enbuild.2017.08.052.

Cole, W.J., Rhodes, J.D., Gorman, W., Perez, K.X., Webber, M.E., Edgar, T.F., 2014. Community-scale residential air conditioning control for effective grid management. Appl. Energy 130, 428–436. http://dx.doi.org/10.1016/j.apenergy.2014.05.067.

Ding, X., Cerpa, A., Du, W., 2025. Multi-zone HVAC control with model-based deep reinforcement learning. IEEE Trans. Autom. Sci. Eng. 22, 4408–4426. http://dx. doi.org/10.1109/TASE.2024.3410951.

Du, Y., Zandi, H., Kotevska, O., Kurte, K., Munk, J., Amasyali, K., Mckee, E., Li, F., 2021. Intelligent multi-zone residential HVAC control strategy based on deep reinforcement learning. Appl. Energy 281, 116117. http://dx.doi.org/10.1016/j. apenergy.2020.116117.

Enescu, D., 2017. A review of thermal comfort models and indicators for indoor environments. Renew. Sustain. Energy Rev. 79, 1353–1379. http://dx.doi.org/10. 1016/j.rser.2017.05.175.

Erdinc, O., Tascikaraoglu, A., Paterakis, N.G., Eren, Y., Catalao, J., 2017. End-user comfort oriented day-ahead planning for responsive residential HVAC demand aggregation considering weather forecasts. In: 2017 IEEE Power & Energy Society General Meeting. p. 1. http://dx.doi.org/10.1109/PESGM.2017.8274516.

Fabra, N., Rapson, D., Reguant, M., Wang, J., 2021. Estimating the elasticity to real-time pricing: Evidence from the spanish electricity market. AEA Pap. Proc. 111, 425–429. http://dx.doi.org/10.1257/pandp.20211007.

Fan, H., MacGill, I.F., Sproul, A.B., 2017. Statistical analysis of drivers of residential peak electricity demand. Energy Build. 141, 205–217. http://dx.doi.org/10.1016/ j.enbuild.2017.02.030.

Gianniou, P., Reinhart, C., Hsu, D., Heller, A., Rode, C., 2018. Estimation of temperature setpoints and heat transfer coefficients among residential buildings in Denmark based on smart meter data. Build. Environ. 139, 125–133. http://dx.doi.org/10. 1016/j.buildenv.2018.05.016.

Gokhale, G., Tiben, N., Verwee, M.-S., Lahariya, M., Claessens, B., Develder, C., 2023. Real-world implementation of reinforcement learning based energy coordination for a cluster of households. In: Proceedings of the 10th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation. ACM, New York, NY, USA, pp. 347–351. http://dx.doi.org/10.1145/3600100.3625681.

González-Torres, M., Pérez-Lombard, L., Coronel, J.F., Maestre, I.R., Paolo, B., 2022. Activity and efficiency trends for the residential sector across countries. Energy Build. 273, 112428. http://dx.doi.org/10.1016/j.enbuild.2022.112428.

Hao, H., Sanandaji, B.M., Poolla, K., Vincent, T.L., 2015. Potentials and economics of residential thermal loads providing regulation reserve. Energy Policy 79, 115–126. http://dx.doi.org/10.1016/j.enpol.2015.01.013.

Harb, H., Boyanov, N., Hernandez, L., Streblow, R., Müller, D., 2016. Development and validation of grey-box models for forecasting the thermal response of occupied buildings. Energy Build. 117, 199–207. http://dx.doi.org/10.1016/j.enbuild.2016. 02.021.

Harper, S., Mehrnezhad, M., Mace, J., 2022. User privacy concerns in commercial smart buildings. J. Comput. Secur. 30 (3), 465–497. http://dx.doi.org/10.3233/JCS- 210035.

Homod, R.Z., Yaseen, Z.M., Hussein, A.K., Almusaed, A., Alawi, O.A., Falah, M.W., Abdelrazek, A.H., Ahmed, W., Eltaweel, M., 2023. Deep clustering of cooperative multi-agent reinforcement learning to optimize multi chiller HVAC systems for smart buildings energy management. J. Build. Eng. 65, 105689. http://dx.doi.org/ 10.1016/j.jobe.2022.105689.

IEA, 2021. Net zero by 2050 – Analysis. https://www.iea.org/reports/net-zero-by-2050.

IEA, 2023. Electricity market report 2023 – Analysis. https://www.iea.org/reports/ electricity-market-report-2023.

Jian, Y., Liu, J., Pei, Z., Chen, J., 2022. Occupants' tolerance of thermal discomfort before turning on air conditioning in summer and the effects of age and gender. J. Build. Eng. 50, 104099. http://dx.doi.org/10.1016/j.jobe.2022.104099.

John, C., Vallianos, C., Candesedo, J., Athienitis, A., 2018. Estimating time constants for over 10,000 residential buildings in north america: Towards a statistical characterization of thermal dynamics. In: Healthy, Intelligent and Resilient Buildings and Urban Environments. International Association of Building Physics (IABP), pp. 1383–1388. http://dx.doi.org/10.14305/ibpc.2018.ps17.

Kaspar, K., Nweye, K., Busemi, G., Capozzoli, A., Nagy, Z., Pinto, G., Eicker, U., Ouf, M.M., 2024. Effects of occupant thermostat preferences and override behavior on residential demand response in CityLearn. Energy Build. 114830. http://dx.doi. org/10.1016/j.enbuild.2024.114830.

Kazmi, H., Suykens, J., Balint, A., Driesen, J., 2019. Multi-agent reinforcement learning for modeling and control of thermostatically controlled loads. Appl. Energy 238, 1022–1035. http://dx.doi.org/10.1016/j.apenergy.2019.01.140.

Khorasani Zadeh, Z., Ouf, M., Gunay, B., Delcroix, B., Larochelle Martin, G., Daoud, A., 2023. Development of prediction models for thermostat override behavior in direct load control events. Energy Build. 301, 113707. http://dx.doi.org/10.1016/ j.enbuild.2023.113707.

Killian, M., Kozek, M., 2016. Ten questions concerning model predictive control for energy efficient buildings. Build. Environ. 105, 403–412. http://dx.doi.org/10. 1016/j.buildenv.2016.05.034.

Kohlhepp, P., Harb, H., Wolisz, H., Waczowicz, S., Müller, D., Hagenmeyer, V., 2019. Large-scale grid integration of residential thermal energy storages as demand-side flexibility resource: A review of international field studies. Renew. Sustain. Energy Rev. 101, 527–547. http://dx.doi.org/10.1016/j.rser.2018.09.045.

Konda, V., Tsitsiklis, J., 1999. Actor-critic algorithms. In: Advances in Neural Information Processing Systems. Vol. 12, MIT Press.

Kou, X., Du, Y., Li, F., Pulgar-Painemal, H., Zandi, H., Dong, J., Olama, M.M., 2021. Model-based and data-driven HVAC control strategies for residential demand response. IEEE Open Access J. Power Energy 8, 186–197. http://dx.doi.org/10. 1109/OAJPE.2021.3075426.

Kubli, M., Loock, M., Wüsthagen, R., 2018. The flexible presumer: Measuring the willingness to co-create distributed flexibility. Energy Policy 114, 540–548. http://dx.doi.org/10.1016/j.enpol.2017.12.044.

Kumar, S.R., Easwaran, A., Delinchant, B., Rigo-Mariani, R., 2024. Real-time retail electricity pricing using offline reinforcement learning. In: Proceedings of the 15th ACM International Conference on Future and Sustainable Energy Systems. Association for Computing Machinery, New York, NY, USA, pp. 454–458. http://dx.doi.org/10.1145/3632775.3661964.

Lambrechts, G., Ernst, D., Mahajan, A., 2025. A theoretical justification for asymmetric actor-critic algorithms. In: Forty-Second International Conference on Machine Learning.

Lankeshwara, G., Sharma, R., Yan, R., Saha, T.K., 2022. A hierarchical control scheme for residential air-conditioning loads to provide real-time market services under uncertainties. Energy 250, 123796. http://dx.doi.org/10.1016/j.energy.2022. 123796.

Lei, N., Zhang, H., Hu, J., Hu, Z., Wang, Z., 2025a. Sim-to-real design and development of reinforcement learning-based energy management strategies for fuel cell electric vehicles. Appl. Energy 393, 126030. http://dx.doi.org/10.1016/j.apenergy.2025. 126030.

Lei, N., Zhang, H., Li, R., Yu, J., Wang, H., Wang, Z., 2024. Physics-informed data-driven modeling approach for commuting-oriented hybrid powertrain optimization. Energy Convers. Manage. 299, 117814. http://dx.doi.org/10.1016/j.enconman. 2023.117814.

Lei, N., Zhang, H., Wang, H., Hu, Z., Chen, H., Hu, J., Wang, Z., 2025b. Theory-constrained neural network with modular interpretability for fuel cell vehicle modelling. IEEE Trans. Veh. Technol. 1–13. http://dx.doi.org/10.1109/TVT.2025. 3533486.

Liang, H., Ma, J., 2020. Separation of residential space cooling usage from smart meter data. IEEE Trans. Smart Grid 11 (4), 3107–3118. http://dx.doi.org/10.1109/TSG. 2020.2965958.

Liang, M., Meng, Y., Lu, N., Lubkeman, D., Kling, A., 2019. HVAC load disaggregation using low-resolution smart meter data. In: 2019 IEEE Power & Energy Society Innovative Smart Grid Technologies Conference. ISGT, IEEE, Washington, DC, USA, pp. 1–5. http://dx.doi.org/10.1109/ISGT.2019.8791578.

Lu, N., 2012. An evaluation of the HVAC load potential for providing load balancing service. IEEE Trans. Smart Grid 3 (3), 1263–1270. http://dx.doi.org/10.1109/TSG. 2012.2183649.

Lynnerup, N.A., Nolling, L., Hasle, R., Hallam, J., 2020. A survey on reproducibility by evaluating deep reinforcement learning algorithms on real-world robots. In: Proceedings of the Conference on Robot Learning. PMLR, pp. 466–489.

Mahmoud, M., Ramadan, M., Naher, S., Pullen, K., Baroutaji, A., Olabi, A.G., 2020. Recent advances in district energy systems: A review. Therm. Sci. Eng. Prog. 20, http://dx.doi.org/10.1016/j.tsep.2020.100678.

Mai, V., Maisonneuve, P., Zhang, T., Nekoei, H., Paull, L., Lesage-Landry, A., 2024. Multi-agent reinforcement learning for fast-timescale demand response of residential loads. Mach. Learn. 113 (8), 5203–5234. http://dx.doi.org/10.1007/s10994- 023-0646-0.
Melillo, A., Durrer, R., Worlitschek, J., Schütz, P., 2020. First results of remote building characterisation based on smart meter measurement data. Energy 200, 117525. http://dx.doi.org/10.1016/j.energy.2020.117525.

Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D., 2015. Human-level control through deep reinforcement learning. Nature 518 (7540), 529-533. http://dx.doi.org/10.1038/nature14236.

Mohajeryami, S., Moghaddam, I.N., Doostan, M., Vatani, B., Schwarz, P., 2016. A novel economic model for price-based demand response. Electr. Power Syst. Res. 135, 1-9. http://dx.doi.org/10.1016/j.epsr.2016.03.026.

Nicolson, M.L., Fell, M.J., Huebner, G.M., 2018. Consumer demand for time of use electricity tariffs: A systematized review of the empirical evidence. Renew. Sustain. Energy Rev. 97, 276-289. http://dx.doi.org/10.1016/j.rser.2018.08.040.

Nilsson, A., Bartusch, C., 2024. Empowered or enchained? exploring consumer perspectives on direct load control. Energy Policy 192, http://dx.doi.org/10.1016/j.enpol.2024.114248.

Parrish, B., Heptonstall, P., Gross, R., Sovacool, B.K., 2020. A systematic review of motivations, enablers and barriers for consumer engagement with residential demand response. Energy Policy 138, 111221. http://dx.doi.org/10.1016/j.enpol.2019.111221.

Pflugradt, N., Stenzel, P., Kotzur, L., Stolten, D., 2022. LoadProfileGenerator: An agent-based BehaviorSimulation for generating residential load profiles. J. Open Source Softw. 7 (71), 3574. http://dx.doi.org/10.21105/joss.03574.

Pinto, L., Andrychowicz, M., Welinder, P., Zaremba, W., Abbeel, P., 2017. Asymmetric actor critic for image-based robot learning. arXiv:1710.06542.

Pinto, G., Deltetto, D., Capozzoli, A., 2021. Data-driven district energy management with surrogate models and deep reinforcement learning. Appl. Energy 304, http://dx.doi.org/10.1016/j.apenergy.2021.117642.

Pomerleau, D.A., 1989. What's hidden in the hidden layers.

Qi, N., Cheng, L., Xu, H., Wu, K., Li, X., Wang, Y., Liu, R., 2020. Smart meter data-driven evaluation of operational demand response potential of residential air conditioning loads. Appl. Energy 279, 115708. http://dx.doi.org/10.1016/j.apenergy.2020.115708.

Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., Dormann, N., 2021. Stable-BASELINES3: Reliable reinforcement learning implementations. J. Mach. Learn. Res. 22 (268), 1-8.

Rajasekhar, B., Tushar, W., Lork, C., Zhou, Y., Yuen, C., Pindoriya, N.M., Wood, K.L., 2020. A survey of computational intelligence techniques for air-conditioners energy management. IEEE Trans. Emerg. Top. Comput. Intell. 4 (4), 555-570. http://dx.doi.org/10.1109/TETCI.2020.2991728.

Ryu, J., Kim, J., Hong, W., De Dear, R., 2020. Quantifying householder tolerance of thermal discomfort before turning on air-conditioner. Energy Build. 211, 109797. http://dx.doi.org/10.1016/j.enbuild.2020.109797.

Sanandaji, B.M., Hao, H., Poolla, K., 2014. Fast regulation service provision via aggregation of thermostatically controlled loads. In: 2014 47th Hawaii International Conference on System Sciences. pp. 2388-2397. http://dx.doi.org/10.1109/HICSS.2014.300.

Sarran, L., Gunay, H.B., O'Brien, W., Hviid, C.A., Rode, C., 2021. A data-driven study of thermostat overrides during demand response events. Energy Policy 153, 112290. http://dx.doi.org/10.1016/j.enpol.2021.112290.

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O., 2017. Proximal policy optimization algorithms. arXiv:1707.06347.

Shaghaghi, A., Taghitahooneh, M., Yahyaeifar, M., Dashti, R., 2024. Management of aging assets in power distribution systems: A comprehensive review. Energy Strat. Rev. 55, 101529. http://dx.doi.org/10.1016/j.esr.2024.101529.

Sigauke, C., Nemukula, M.M., 2020. Modelling extreme peak electricity demand during a heatwave period: A case study. Energy Syst. 11 (1), 139-161. http://dx.doi.org/10.1007/s12667-018-0311-y.

Silva, S., Soares, I., Pinho, C., 2017. Electricity demand response to price changes: The Portuguese case taking into account income differences. Energy Econ. 65, 335-342. http://dx.doi.org/10.1016/j.eneco.2017.05.018.

Smith, A., St&uuml;mler, Rmer, B., Thurber, T., Vernon, C., 2021. Diyepw: A Python package for do-it-yourself EnergyPlus weather file generation. J. Open Source Softw. 6 (64), 3313. http://dx.doi.org/10.21105/joss.03313.

Song, M., Gao, C., Yan, H., Yang, J., 2018. Thermal battery modeling of inverter air conditioning for demand response. IEEE Trans. Smart Grid 9 (6), 5522-5534. http://dx.doi.org/10.1109/TSG.2017.2689820.

St-Jacques, M., Buckingham, S., O'Brien, W., 2024. Do peak energy loads and peak GHG emissions correlate in buildings? insights from emission duration curves and emission event duration curves analysis. J. Clean. Prod. 459, 142438. http://dx.doi.org/10.1016/j.jclepro.2024.142438.

Statelyte, D., Radziukyniene, N., Radziukynas, V., 2022. Overview of demand-response services: A review. Energies 15 (5), 1659. http://dx.doi.org/10.3390/en15051659.

Storn, R., Price, K., 1997. Differential evolution – A simple and efficient heuristic for global optimization over continuous spaces. J. Global Optim. 11 (4), 341-359. http://dx.doi.org/10.1023/A:10082028212328.

Sutton, R.S., Barto, A., 2020. Reinforcement Learning: An Introduction, second ed. Adaptive Computation and Machine Learning, The MIT Press, Cambridge, Massachusetts London, England.

Talib, A., Park, S., Im, P., Joe, J., 2023. Grey-box and ANN-based building models for multistep-ahead prediction of indoor temperature to implement model predictive control. Eng. Appl. Artif. Intell. 126, 107115. http://dx.doi.org/10.1016/j.engappai.2023.107115.

Tan, J., Xue, S., Li, H., Guo, Z., Cao, H., Chen, B., 2025. Hierarchical safe reinforcement learning control for leader-follower systems with prescribed performance. IEEE Trans. Autom. Sci. Eng. 22, 19568-19581. http://dx.doi.org/10.1109/TASE.2025.3596912.

Tian, H., Olshovsky, A., Paschalidis, I., 2023. Convergence of actor-critic with multi-layer neural networks. In: Thirty-Seventh Conference on Neural Information Processing Systems.

Tindemans, S.H., Trovato, V., Strbac, G., 2015. Decentralized control of thermostatic loads for flexible demand response. IEEE Trans. Control Syst. Technol. 23 (5), 1685-1700. http://dx.doi.org/10.1109/TCST.2014.2381163.

van der Maaten, L., Hinton, G., 2008. Visualizing data using T-SNE. J. Mach. Learn. Res. 9 (86), 2579-2605.

Van Orden, E., Simović, U., 2023. Managing peak load in the past, present, and future: Challenges and opportunities for a flexible energy future. Clim. Energy 39 (12), 15-20. http://dx.doi.org/10.1002/gas.22354.

Waseem, M., Lin, Z., Ding, Y., Wen, F., Liu, S., Palu, I., 2021. Technologies and practical implementations of air-conditioner based demand response. J. Mod. Power Syst. Clean Energy 9 (6), 1395-1413. http://dx.doi.org/10.35833/MPCE.2019.000449.

Weck, M.H.J., van Hooff, J., van Sark, W.G.J.H.M., 2017. Review of barriers to the introduction of residential demand response: A case study in the Netherlands. Int. J. Energy Res. 41 (6), 790-816. http://dx.doi.org/10.1002/er.3683.

Wildstein, P.J., Craig, M.T., Vaishnav, P., 2023. Participant overrides can halve the reliability value of direct load control programs. Energy Build. 299, 113606. http://dx.doi.org/10.1016/j.enbuild.2023.113606.

Wu, X., He, J., Xu, Y., Lu, J., Lu, N., Wang, X., 2018. Hierarchical control of residential HVAC units for primary frequency regulation. IEEE Trans. Smart Grid 9 (4), 3844-3856. http://dx.doi.org/10.1109/TSG.2017.2766880.

Xia, Y., Wang, X., Yin, X., Bo, W., Wang, L., Li, S., Li, K., 2025. Federated accelerated deep reinforcement learning for multi-zone HVAC control in commercial buildings. IEEE Trans. Smart Grid 16 (3), 2599-2610. http://dx.doi.org/10.1109/TSG.2024.3524756.

Xue, W., Jia, N., Zhao, M., 2025. Multi-agent deep reinforcement learning based HVAC control for multi-zone buildings considering zone-energy-allocation optimization. Energy Build. 329, 115241. http://dx.doi.org/10.1016/j.enbuild.2024.115241.

Yang, H.-T., Tsai, H.-T., Liao, J.-T., Wu, Y.-S., 2024. Optimizing demand response in multi-HVAC systems by rl control with comfort consideration. In: 2024 IEEE 19th Conference on Industrial Electronics and Applications (ICIEA). pp. 1-6. http://dx.doi.org/10.1109/ICIEA61579.2024.10664992.

Zhang, H., Lei, N., Chen, B., Li, B., Li, R., Wang, Z., 2024. Modeling and control system optimization for electrified vehicles: A data-driven approach. Energy 310, 133196. http://dx.doi.org/10.1016/j.energy.2024.133196.

Zhao, W., Queralta, J.P., Westerlund, T., 2020. Sim-to-real transfer in deep reinforcement learning for robotics: A survey. http://dx.doi.org/10.1109/SSCI47803.2020.9308468, arXiv:2009.13303.