# Learning Efficient Dynamic Controller for HVAC System

Authors: Dongsheng Xu
Journal: Mobile Information Systems
 (2022)

---

## Abstract

People employ a variety of devices to achieve comfort in various aspects of their lives; for example, numerous types of air conditioners are used to maintain a pleasant indoor temperature. The wide applications of heating, ventilation, and air conditioning (HVAC) systems have significantly improved the comfort level of the indoor environment for commercial buildings. However, air conditioners have their own set of issues, the most significant of which is the increased energy usage. Besides the fact that HVAC is a recent move toward the indoor temperature adjustment, the energy cost that arose from the HVAC system is still high, which takes up over 40% of the total energy. How to develop an energy-eﬀicient HVAC system has become one of the challenging topics that need further investigation. This paper addresses this problem with the aid of deep learning (DL) algorithms. Furthermore, this study proposes an energy-eﬀicient controller and improvement schemes for HVAC systems based on deep reinforcement learning algorithms. To reduce the implementation cost of the proposed design, this study utilizes neural network pruning techniques to develop a low-complexity HVAC controller. The experimental results show that the algorithm proposed in this study not only retains a high energy-saving rate but also reduces the implementation cost of the HVAC control algorithm.


---

## Body

## 1. Introduction

With the development of society, heating, ventilation, and air conditioning (HVAC) system is widely deployed and quickly expanding, providing a comfortable indoor environment for commercial buildings. Building energy costs account for almost 40% of overall energy usage in the United States, according to a research study published in [1]. The consumption arises from several aspects, such as HVAC and lighting. However, the HVAC system consumes the majority of the total energy. Therefore, how to improve the HVAC controller to develop an energy-saving schedule has become one of the research topics.One of the alternatives that may be used in a building to lower the energy demand for HVAC systems is a radiant cooling system. In terms of delivering thermal comfort, radiant cooling is thought to be more energy-efficient than traditional HVAC systems [2] because water has a larger thermal capacity than air, and on the other hand, a pump, which is utilized in a radiant cooling system, also gives better efficiency as compared to a fan. This technology might be more effective in buildings with a lot of glass and a lot of solar radiation, but unfortunately, not every building meets this criterion. However, there are numerous serious challenges with this system, including condensate buildup on the cold radiant surface, which causes water damage, mildew, and other problems, complicated controls necessitating the use of trained maintenance personnel, and temperatures below 23°C are difficult to sustain. So, the improvement of the HVAC system in terms of energy efficiency with the integration of this technology is not that easy. As a result, in this article, we applied a machine learning approach to make the HVAC system more dynamic and responsive, which also leads to increased energy efficiency.ML and DL technologies are becoming more common in consumer products such as cameras and smartphones, and they are responsible for many facets of contemporary life, ranging from Internet searches to social media content filtering to e-commerce website suggestions. The problems that were faced by artificial intelligence for years are extensively solved with the advent and progression in deep learning. It becomes useful for a wide range of business, scientific, and government applications because it is very good at recognizing subtle structures in high-dimensional data [3]. It has outperformed machine learning in a number of areas, including speech recognition [4], object identification, and recognition [5]. Deep neural networks (DNN) have been brought into reinforcement learning in recent years, and they have had a lot of success with value function approximation. Reinforcement learning techniques are ideal for teaching an agent to govern itself by allowing it to interact with its surroundings [6]. In the last several years, reinforcement learning has also been applied in HVAC systems in order to make them more intelligent and environmentally responsive, as discussed in the Related Work section of this article.Although yielding significant improvements, the existing deep learning-based works mostly require a large number of computing resources to perform the control algorithms, which is not realistic for embedded devices and real implementation. In this research work, we use cutting-edge deep learning methods to solve the aforementioned issues. The contributions are as follows. First, we proposed an energy-eﬀicient controller for an HVAC system by utilizing deep reinforcement learning algorithms and improvement schemes. To reduce the implementation cost of the proposed design, we utilize neural network pruning techniques to develop a low-complexity HVAC controller. Finally, we present experimental data to demonstrate the efficacy of the suggested algorithms in the context of HVAC control.The remaining structure of the paper is organized as follows: the literature on HVAC-related work is covered in Section2. Section 3 proposed deep reinforcement learning-based adaptive controller for HVAC. This section further elaborates the HVAC system based on reinforcement learning. Section 4 is about neural network optimization. The evaluation and experiments are presented in Section 5. Finally, Section 6 concludes the whole paper.


## 2. Related Work

An intelligent and reliable controller plays an essential role in the automation of HVAC control. The existing HVAC systems of commercial buildings include several parts: physical condition sensing and intelligent control core. The environment sensing module of the HVAC system is sophisticated enough due to the well-developed sensors and other features. As for the control part, many researchers have made efforts to develop eﬀicient control strategies for building HVAC systems. For instance, in [7], the authors design an occupancy-predictive model-based HVAC control algorithm and implement it on a low-cost embedded system. They achieve significant energy savings goals.Prior research attempts to optimize the energy use of HVAC systems required either the creation of mathematical models of HVAC systems or the analysis of large amounts of historical operational data to learn optimal operational settings. However, Satrio et al. [8] presented a model-free control policy that improves HVAC operations by learning optimal settings without any prior historical data. They utilized a set of algorithms to achieve this goal, including machine learning (regression decision tree, k-NN) and recursive brute force search. According to the author’s claim, the proposed algorithm gives about 31.17% energy efficiency compared to other existing approaches.Conventional HVAC control relies on rule-based control strategies. This requires complex human-crafted rules and a tuning process, which is unrealistic for real implementation. To address the problem, a deep reinforcement learning-based control scheme is proposed in [9]. In comparison with traditional rule-based techniques, the agent is able to intelligently learn the most efficient strategy for controlling the building HVAC systems, saving more energy. But the unknown thermal dynamics of the indoor building are still challenging. The authors in [10] present a multiagent deep reinforcement learning that incorporates an attention mechanism, which does not need any prior information. The work in [11] exploits video images to realize thermal comfort inference and improves the quality of HVAC control.


## 3. Proposed Adaptive Control Scheme for Building the HVAC System

### 3.1. Overall Control and Design Flow

The HVAC control system inside the building is used to maintain desired temperature conditions for each building area. The temperature is adjusted based on the current room conditions and outside environments, like indoor temperatures and outdoor weather, bringing a comfortable indoor environment for people. According to the analysis of Section3.2, the HVAC control operation may be considered a Markov decision process. We formulate the HVAC control process as shown in Figure 1, which illustrates the basic control flow of the HVAC system in the commercial building.Figure 1
Basic control flow of HVAC system in the commercial building.In the case of HVAC control, the controller is thought of as an agent that inter-relates with the environment of the commercial. The controller sends a signal to the HVAC system to produce different levels of airflow. The environment arrives at various states after every time performing actions on the HVAC system. As the actions are taken and the HVAC system generates different effects on the environment, the agent may receive various rewards and state information.In order to develop eﬀicient and reliable HVAC controllers, we propose a novel adaptive controller by taking the advantages of deep reinforcement learning, optimizations for neural networks (NN), and building simulation techniques. The proposed HVAC control design makes a good tradeoff between control accuracy and implementation overhead. Furthermore, the suggested control algorithm may be implemented in an automated way with extremely low algorithm complexity thanks to the use of network optimization algorithms.To reach the goal of precise control for building temperature, the control algorithms should be intelligent and have some degree of adaptiveness to automatically adjust the temperature without much human intervention. These impose high requirements on the core algorithms of the control system. Hence, we exploit the deep reinforcement learning techniques that are at the cutting edge of technology to realize accurate temperature control and maintain high-quality comfort for people.Since DNN’s are computation-dominant, we need to consider two important considerations while constructing the HVAC control algorithm: low algorithm complexity and low response latency [12]. We utilized the existing optimization schemes to prune the neural network models of deep reinforcement learning, thereby reducing the inherent computation complexity as well as the implementation overhead.The general design of our proposed intelligent HVAC controller for commercial buildings based on deep reinforcement learning algorithms is depicted in Figure2. The depicted diagram involves two phases: the inference phase and the training phase. For the training phase of the proposed algorithm, we used the advanced simulation tool to emulate a particular building’s thermal dynamics due to the limited amount of real data and training cost. The learned model is optimised using a network pruning approach once the training phase has converged to lower the needed compute complexity. Finally, we implemented the trained control algorithm and send a control signal to the HVAC system. The experiment results are obtained and collected.Figure 2
Proposed deep reinforcement learning-based HVAC controller.


### 3.2. HVAC Control with Deep Reinforcement Learning

For simplicity of controller algorithms design, we considered the HVAC system that produces conditioned air which typically flows at a constant temperature. In this case, the airflow rate can be switched between multiple levels. The challenge of finding the best control sequence for the HVAC system may be viewed as the task of HVAC control for commercial buildings. However, due to the complexity of the building environment, developing precise control algorithms that spend the least amount of energy while maintaining a high degree of comfort is challenging. To solve this issue, identical to [9], we leverage the deep Q-learning model [13] and propose optimization tricks for the application case of HVAC control.We discretized the control signal of the HVAC system utilizing an infinite resolution of airflow rate by considering the true dependability of simulation that accommodates a real HVAC system. This is because the control algorithm is unable to accept and process sensor data with very high resolution, not only hindering the training convergence but also increasing hardship for the realistic system. Therefore, we assume that the discrete airflow rate is uniformly distributed between the minimum flow ratefmin and the maximum flow rate fmax. As a result, the control signal (or action sequence) is expressed as A=a1,…,at from time step 1 to time step t. It is clear that the size of the activity area would grow exponentially as the time steps and airflow rate levels increase.The temperature of the next time step after tuning can be regarded as that is only resolute by the current building states, indoor or outdoor environmental conditions, and the control signals of the HVAC system. It does not depend on the building of earlier conditions [9]. Hence, the Markov decision process may be used to describe HVAC control methods (MDP). Under the premise of MDP, the enormous search space for the action sequence is greatly minimized.Because the reward function directs the agent to take action and make decisions, the reward design is critical in deep reinforcement learning. It determines the performance and intelligence level of trained control algorithms. Based on the state and reward feedback of the environment, a reward should be carefully designed to reflect the goal of the constructed model. As mentioned before, the goal of an HVAC controller includes two aspects: keeping the interior of the building at a suitable temperature and reducing energy use over time. We integrate these two targets into a single reward function. The reward function at timet is given as follows:(1)rt=−energyat−1,st−1+λTti−Tmin+Tti−Tmax,where the reward function contains two parts, incorporating the HVAC system’s energy consumption and the metric that evaluates the comfort level of the indoor environment. The obtained reward from the environment is determined by the current state of the interior environment as well as the preceding time step’s energy use. The comfort level is evaluated on the basis of the variance between the real temperature and desired comfortable temperature. The reward is negative because the goal is to minimize energy consumption and retain minimum temperature fluctuations. By balancing the two factors, the agent gets the best reward function.For each time step, equation (1) calculates the reward. However, the suggested control algorithms would attempt to maximize the overall reward over a specified time period. The reward for a single time step is added together to form a total payout. The agent’s purpose is to maximize the total reward between the first and final states by doing the following:(2)Rtotal=∑t=1nγtrt,where Rtotal denotes the summation of the reward from the beginning to the end of the action sequence. γ∈θ represents the discount factor which measures the contribution of rewards in the short and long run. For instance, γ=1 indicates that the agent treats rewards from different time steps equally. The relevance of the reward acquired from prior time steps is diminished as the size of the reward decreases.According to [14], the state of the environment and action space is very huge; thus, it is not possible to store all of them in a standard reinforcement learning Q table. In the new deep Q-learning techniques, a DNN is employed to estimate the Q value procedure to solve this problem. On the basis of the Bellman equation, the updation of Q value is formulated as follows:(3)Qst,at←Qst,at+αRt+1+γmaxaQst+1,a−Qst,at,where α denotes the learning rate and Qst,at represents the Q value function corresponding to the current time. Qst+1,a represents the Q value associated with the next time step after taking action a.The weights of the NN’s are randomly initialized and defined asw. The weights of NN’s are iteratively updated using the minibatch SGD method. More precisely, the weights of NN’s w are updated using the gradient of the loss function during training and is formulated as follows:(4)wk+1=wk−α⋅ERt+1+γmaxaQst+1,a;wk−−Qst,at;wk×∇wkQst,at;wk,where the aim network weights at kth iteration are defined as wk−, which are constant during training and are only changed every training iteration C using the Q network weights wk. The mean squared error (MSE) between the neural network output and the desired Q value is the loss function employed.According to the analysis in [14], the weight updating in (4) overestimates the values of actions in the target network and Q network. This is due to the fact that the anticipated and target values are computed using the same network, resulting in divergence between the calculated results. The phenomenon of exaggerating may hinder the control algorithms from selecting the best course of action in a variety of situations, thus declining the overall performance of algorithms. Different deep Q network and target networks can be used to tackle this problem [14]. In this situation, the simple Q value function is(5)Qst,at←Qst,at+αRt+1+γQ′st+1,a−Qst,at,where functions Q′⋅ and a=maxaQst+1,a are utilized to calculate the predictable Q value using the action a that optimizes the value of the projected Q value network.According to [13], if we simply run the deep Q learning algorithms on the state-action pairs as they occur during simulation, the agent training process may become unstable. We offer the experience replay buffer to provide the agent with some memory capabilities by storing the transition between found data and experienced data. The experienced replaying is used to lower the association of observed samples and increase the resiliency of the training process. The agent’s proficiencies, comprising taken actions, states, and achievements, are stored in the experience replays buffer, as illustrated in Figure 2.The two most important principles in reinforcement learning are exploration and exploitation. Exploration allows an agent to enhance their existing knowledge of each action, whereas exploitation allows them to choose the best-rewarded action based on current action-value estimates. To strike a balance between exploration and exploitation, theϵ-greedy strategy is tested, which randomly chooses between exploration and exploitation. The following equation may be used to represent it:(6)at=maxaQs,a,with probability1−ε,random action,with probabilityε.With probability1−ε, the agent will choose the action that maximizes the current Q value. The agent, on the other hand, chooses a random action with a high probability. To put it another way, the ϵ-greedy policy will allow the agent to exploit the majority of the time.


## 3.1. Overall Control and Design Flow

The HVAC control system inside the building is used to maintain desired temperature conditions for each building area. The temperature is adjusted based on the current room conditions and outside environments, like indoor temperatures and outdoor weather, bringing a comfortable indoor environment for people. According to the analysis of Section3.2, the HVAC control operation may be considered a Markov decision process. We formulate the HVAC control process as shown in Figure 1, which illustrates the basic control flow of the HVAC system in the commercial building.Figure 1
Basic control flow of HVAC system in the commercial building.In the case of HVAC control, the controller is thought of as an agent that inter-relates with the environment of the commercial. The controller sends a signal to the HVAC system to produce different levels of airflow. The environment arrives at various states after every time performing actions on the HVAC system. As the actions are taken and the HVAC system generates different effects on the environment, the agent may receive various rewards and state information.In order to develop eﬀicient and reliable HVAC controllers, we propose a novel adaptive controller by taking the advantages of deep reinforcement learning, optimizations for neural networks (NN), and building simulation techniques. The proposed HVAC control design makes a good tradeoff between control accuracy and implementation overhead. Furthermore, the suggested control algorithm may be implemented in an automated way with extremely low algorithm complexity thanks to the use of network optimization algorithms.To reach the goal of precise control for building temperature, the control algorithms should be intelligent and have some degree of adaptiveness to automatically adjust the temperature without much human intervention. These impose high requirements on the core algorithms of the control system. Hence, we exploit the deep reinforcement learning techniques that are at the cutting edge of technology to realize accurate temperature control and maintain high-quality comfort for people.Since DNN’s are computation-dominant, we need to consider two important considerations while constructing the HVAC control algorithm: low algorithm complexity and low response latency [12]. We utilized the existing optimization schemes to prune the neural network models of deep reinforcement learning, thereby reducing the inherent computation complexity as well as the implementation overhead.The general design of our proposed intelligent HVAC controller for commercial buildings based on deep reinforcement learning algorithms is depicted in Figure2. The depicted diagram involves two phases: the inference phase and the training phase. For the training phase of the proposed algorithm, we used the advanced simulation tool to emulate a particular building’s thermal dynamics due to the limited amount of real data and training cost. The learned model is optimised using a network pruning approach once the training phase has converged to lower the needed compute complexity. Finally, we implemented the trained control algorithm and send a control signal to the HVAC system. The experiment results are obtained and collected.Figure 2
Proposed deep reinforcement learning-based HVAC controller.


## 3.2. HVAC Control with Deep Reinforcement Learning

For simplicity of controller algorithms design, we considered the HVAC system that produces conditioned air which typically flows at a constant temperature. In this case, the airflow rate can be switched between multiple levels. The challenge of finding the best control sequence for the HVAC system may be viewed as the task of HVAC control for commercial buildings. However, due to the complexity of the building environment, developing precise control algorithms that spend the least amount of energy while maintaining a high degree of comfort is challenging. To solve this issue, identical to [9], we leverage the deep Q-learning model [13] and propose optimization tricks for the application case of HVAC control.We discretized the control signal of the HVAC system utilizing an infinite resolution of airflow rate by considering the true dependability of simulation that accommodates a real HVAC system. This is because the control algorithm is unable to accept and process sensor data with very high resolution, not only hindering the training convergence but also increasing hardship for the realistic system. Therefore, we assume that the discrete airflow rate is uniformly distributed between the minimum flow ratefmin and the maximum flow rate fmax. As a result, the control signal (or action sequence) is expressed as A=a1,…,at from time step 1 to time step t. It is clear that the size of the activity area would grow exponentially as the time steps and airflow rate levels increase.The temperature of the next time step after tuning can be regarded as that is only resolute by the current building states, indoor or outdoor environmental conditions, and the control signals of the HVAC system. It does not depend on the building of earlier conditions [9]. Hence, the Markov decision process may be used to describe HVAC control methods (MDP). Under the premise of MDP, the enormous search space for the action sequence is greatly minimized.Because the reward function directs the agent to take action and make decisions, the reward design is critical in deep reinforcement learning. It determines the performance and intelligence level of trained control algorithms. Based on the state and reward feedback of the environment, a reward should be carefully designed to reflect the goal of the constructed model. As mentioned before, the goal of an HVAC controller includes two aspects: keeping the interior of the building at a suitable temperature and reducing energy use over time. We integrate these two targets into a single reward function. The reward function at timet is given as follows:(1)rt=−energyat−1,st−1+λTti−Tmin+Tti−Tmax,where the reward function contains two parts, incorporating the HVAC system’s energy consumption and the metric that evaluates the comfort level of the indoor environment. The obtained reward from the environment is determined by the current state of the interior environment as well as the preceding time step’s energy use. The comfort level is evaluated on the basis of the variance between the real temperature and desired comfortable temperature. The reward is negative because the goal is to minimize energy consumption and retain minimum temperature fluctuations. By balancing the two factors, the agent gets the best reward function.For each time step, equation (1) calculates the reward. However, the suggested control algorithms would attempt to maximize the overall reward over a specified time period. The reward for a single time step is added together to form a total payout. The agent’s purpose is to maximize the total reward between the first and final states by doing the following:(2)Rtotal=∑t=1nγtrt,where Rtotal denotes the summation of the reward from the beginning to the end of the action sequence. γ∈θ represents the discount factor which measures the contribution of rewards in the short and long run. For instance, γ=1 indicates that the agent treats rewards from different time steps equally. The relevance of the reward acquired from prior time steps is diminished as the size of the reward decreases.According to [14], the state of the environment and action space is very huge; thus, it is not possible to store all of them in a standard reinforcement learning Q table. In the new deep Q-learning techniques, a DNN is employed to estimate the Q value procedure to solve this problem. On the basis of the Bellman equation, the updation of Q value is formulated as follows:(3)Qst,at←Qst,at+αRt+1+γmaxaQst+1,a−Qst,at,where α denotes the learning rate and Qst,at represents the Q value function corresponding to the current time. Qst+1,a represents the Q value associated with the next time step after taking action a.The weights of the NN’s are randomly initialized and defined asw. The weights of NN’s are iteratively updated using the minibatch SGD method. More precisely, the weights of NN’s w are updated using the gradient of the loss function during training and is formulated as follows:(4)wk+1=wk−α⋅ERt+1+γmaxaQst+1,a;wk−−Qst,at;wk×∇wkQst,at;wk,where the aim network weights at kth iteration are defined as wk−, which are constant during training and are only changed every training iteration C using the Q network weights wk. The mean squared error (MSE) between the neural network output and the desired Q value is the loss function employed.According to the analysis in [14], the weight updating in (4) overestimates the values of actions in the target network and Q network. This is due to the fact that the anticipated and target values are computed using the same network, resulting in divergence between the calculated results. The phenomenon of exaggerating may hinder the control algorithms from selecting the best course of action in a variety of situations, thus declining the overall performance of algorithms. Different deep Q network and target networks can be used to tackle this problem [14]. In this situation, the simple Q value function is(5)Qst,at←Qst,at+αRt+1+γQ′st+1,a−Qst,at,where functions Q′⋅ and a=maxaQst+1,a are utilized to calculate the predictable Q value using the action a that optimizes the value of the projected Q value network.According to [13], if we simply run the deep Q learning algorithms on the state-action pairs as they occur during simulation, the agent training process may become unstable. We offer the experience replay buffer to provide the agent with some memory capabilities by storing the transition between found data and experienced data. The experienced replaying is used to lower the association of observed samples and increase the resiliency of the training process. The agent’s proficiencies, comprising taken actions, states, and achievements, are stored in the experience replays buffer, as illustrated in Figure 2.The two most important principles in reinforcement learning are exploration and exploitation. Exploration allows an agent to enhance their existing knowledge of each action, whereas exploitation allows them to choose the best-rewarded action based on current action-value estimates. To strike a balance between exploration and exploitation, theϵ-greedy strategy is tested, which randomly chooses between exploration and exploitation. The following equation may be used to represent it:(6)at=maxaQs,a,with probability1−ε,random action,with probabilityε.With probability1−ε, the agent will choose the action that maximizes the current Q value. The agent, on the other hand, chooses a random action with a high probability. To put it another way, the ϵ-greedy policy will allow the agent to exploit the majority of the time.


## 4. Network Optimization

Neural network optimization is a strategy for reducing losses by changing the properties of your neural network, such as weights and learning rate. Researchers use various optimizers in order to reduce loss and enhance accuracy such as stochastic gradient descent, gradient descent, and minibatch gradient descent. However, in this paper, we applied a new approach called neural network pruning.


### 4.1. Pruning of Neural Networks

The existing deep neural network models normally require a large number of computing resources and memory, which becomes the bottleneck when running low-latency models on embedded devices. Fully-connected network training and inference are computationally and memory-intensive tasks, requiring tremendous data movement and computation. Hence, real-time inference tasks are difficult to accomplish on embedded devices with low computational capability. Moreover, energy efficiency is another major issue when implementing current deep neural network models.These constraints not only raise implementation costs but also make it difficult to deploy suggested models in resource-constrained settings. One of the methods to solve these challenges is neural network pruning techniques [12, 15], which are used to shrink network topologies by eliminating superfluous connections between fully linked layers. To improve neural network architecture, a variety of network pruning strategies are available. One of the potential strategies for obtaining sparse neural network topologies is deep compression [15, 16]. The core concept behind deep compression is to gradually locate and remove superfluous weights with tiny magnitudes.The neuron connections of the original and trimmed fully-connected layers are shown in Figure3. Because the initial layer connection is thick, it necessitates a high number of weights. For each tier, network pruning looks for the most representative and significant links. Then, by setting the weights to zero, the trivial connections may be deleted, significantly lowering the weights and the model’s size as well as computing complexity. After implementing the modifications, the neural network’s inference may be sped up by many times without sacrificing performance. Meanwhile, when enormous quantities of weights are trimmed, the model’s size will drop dramatically.Figure 3
Illustration of original DNN (a) and its pruned variant (b).However, the criteria used to prune the networks in deep compression is simplistic, making it difficult to ensure that model performance is lost to the bare minimum. In this work, we adopt the other pruning techniques such as ThiNet [17] which pruned the model by an analytical metric. To minimize the performance degradation and prune the weights as many as possible, the authors in [17] use the following reconstruction error:(7)w^=argminw∑i=1my^i−wTx^i′2.where x^i′ are the randomly selected samples that are used to evaluate the construction error. Equation (7) can be understood as a typical linear regression issue. Thus, it has the close-form solution w^=XTX−1XTy.


## 4.1. Pruning of Neural Networks

The existing deep neural network models normally require a large number of computing resources and memory, which becomes the bottleneck when running low-latency models on embedded devices. Fully-connected network training and inference are computationally and memory-intensive tasks, requiring tremendous data movement and computation. Hence, real-time inference tasks are difficult to accomplish on embedded devices with low computational capability. Moreover, energy efficiency is another major issue when implementing current deep neural network models.These constraints not only raise implementation costs but also make it difficult to deploy suggested models in resource-constrained settings. One of the methods to solve these challenges is neural network pruning techniques [12, 15], which are used to shrink network topologies by eliminating superfluous connections between fully linked layers. To improve neural network architecture, a variety of network pruning strategies are available. One of the potential strategies for obtaining sparse neural network topologies is deep compression [15, 16]. The core concept behind deep compression is to gradually locate and remove superfluous weights with tiny magnitudes.The neuron connections of the original and trimmed fully-connected layers are shown in Figure3. Because the initial layer connection is thick, it necessitates a high number of weights. For each tier, network pruning looks for the most representative and significant links. Then, by setting the weights to zero, the trivial connections may be deleted, significantly lowering the weights and the model’s size as well as computing complexity. After implementing the modifications, the neural network’s inference may be sped up by many times without sacrificing performance. Meanwhile, when enormous quantities of weights are trimmed, the model’s size will drop dramatically.Figure 3
Illustration of original DNN (a) and its pruned variant (b).However, the criteria used to prune the networks in deep compression is simplistic, making it difficult to ensure that model performance is lost to the bare minimum. In this work, we adopt the other pruning techniques such as ThiNet [17] which pruned the model by an analytical metric. To minimize the performance degradation and prune the weights as many as possible, the authors in [17] use the following reconstruction error:(7)w^=argminw∑i=1my^i−wTx^i′2.where x^i′ are the randomly selected samples that are used to evaluate the construction error. Equation (7) can be understood as a typical linear regression issue. Thus, it has the close-form solution w^=XTX−1XTy.


## 5. Experiments Result and Analysis

In this section, we carry out detailed experiments to confirm the resulting performance of the proposed adaptive control scheme for the HVAC system. Moreover, the comparison and analysis with other counterparts are also presented.


### 5.1. Simulation Setup

We used PyTorch, an advanced deep learning framework, to implement the suggested deep reinforcement learning algorithms. The training and inference processes are sped up using the GPU. Similar to the simulation flow in [9], the thermal dynamics of the indoor environment are emulated using the simulation tool EnergyPlus. The comfortable temperature range is set as Tmin=20 and Tmax=26. Both the target network and the deep Q network have a five-layer basic structure, with each hidden layer containing 64, 128, 256, 192, and 96 neurons.The input of the trained controller is the concatenation of the outdoor temperature vector, indoor temperature vector, and current state of the HVAC system. The hyperparameters of training and network structures are given as follows. The minibatch size is set at 16 to provide a smooth convergence process and reliable weight updating. Suggested by [13], the ϵ-greedy policy will choose a random action with the probability of 0.1. We assign a small experience replay memory size of 50. Every six training cycles, the target network’s weights are replicated. The cumulative reward calculation’s discount factor is set to 0.97. For speedier training, the Adam optimizer [18] is utilized.


### 5.2. Experiment Results

This section represents the experimental results of the proposed model. Several experiments have been performed, and different results have been achieved, but the most prominent one is discussed here in this section.Figure4 illustrates the indoor temperature curve using different HVAC control algorithms. The baseline is the reinforcement learning-based HVAC controller. One commercial building in Nanjing, China, is selected as the indoor environment for conducting the experiment. The training is done using a simulation tool, and the test is done on a real structure.Figure 4
Results of indoor temperature under different HVAC control algorithms from 0: 00 to 16:00.It can be seen from the figure that the indoor temperature under the control of the proposed algorithms is more smooth and cause less temperature fluctuation in comparison with the baseline control algorithms. This demonstrates that the proposed algorithms are more effective in the application of HVAC control.I used the adopted pruning techniques as used in [17] and found out the relationship between the energy-saving rate and weight pruning percentage. The energy-saving rate and weight pruning percentage is shown in Figure 5. I did not use the average training reward to evaluate the performance of network pruning since the average reward is hard to reflect the final effects of network pruning. Hence, I used the more realistic metric of energy saving.Figure 5
Relationship between the energy-saving rate and weight pruning percentage.It is defined by the percentage of energy-saving using the proposed pruned control algorithm over the baseline algorithms in Figure5. From the figure, we can see that the unpruned network model delivers over 50% energy-saving compared to the baseline model. After setting different pruning thresholds, the degraded network performance causes a decline in the energy-saving rate but the energy-saving rate declines slowly. Furthermore, Figures 4 and 5 provide guidelines for us to decide which network structures to use according to different scenarios.


## 5.1. Simulation Setup

We used PyTorch, an advanced deep learning framework, to implement the suggested deep reinforcement learning algorithms. The training and inference processes are sped up using the GPU. Similar to the simulation flow in [9], the thermal dynamics of the indoor environment are emulated using the simulation tool EnergyPlus. The comfortable temperature range is set as Tmin=20 and Tmax=26. Both the target network and the deep Q network have a five-layer basic structure, with each hidden layer containing 64, 128, 256, 192, and 96 neurons.The input of the trained controller is the concatenation of the outdoor temperature vector, indoor temperature vector, and current state of the HVAC system. The hyperparameters of training and network structures are given as follows. The minibatch size is set at 16 to provide a smooth convergence process and reliable weight updating. Suggested by [13], the ϵ-greedy policy will choose a random action with the probability of 0.1. We assign a small experience replay memory size of 50. Every six training cycles, the target network’s weights are replicated. The cumulative reward calculation’s discount factor is set to 0.97. For speedier training, the Adam optimizer [18] is utilized.


## 5.2. Experiment Results

This section represents the experimental results of the proposed model. Several experiments have been performed, and different results have been achieved, but the most prominent one is discussed here in this section.Figure4 illustrates the indoor temperature curve using different HVAC control algorithms. The baseline is the reinforcement learning-based HVAC controller. One commercial building in Nanjing, China, is selected as the indoor environment for conducting the experiment. The training is done using a simulation tool, and the test is done on a real structure.Figure 4
Results of indoor temperature under different HVAC control algorithms from 0: 00 to 16:00.It can be seen from the figure that the indoor temperature under the control of the proposed algorithms is more smooth and cause less temperature fluctuation in comparison with the baseline control algorithms. This demonstrates that the proposed algorithms are more effective in the application of HVAC control.I used the adopted pruning techniques as used in [17] and found out the relationship between the energy-saving rate and weight pruning percentage. The energy-saving rate and weight pruning percentage is shown in Figure 5. I did not use the average training reward to evaluate the performance of network pruning since the average reward is hard to reflect the final effects of network pruning. Hence, I used the more realistic metric of energy saving.Figure 5
Relationship between the energy-saving rate and weight pruning percentage.It is defined by the percentage of energy-saving using the proposed pruned control algorithm over the baseline algorithms in Figure5. From the figure, we can see that the unpruned network model delivers over 50% energy-saving compared to the baseline model. After setting different pruning thresholds, the degraded network performance causes a decline in the energy-saving rate but the energy-saving rate declines slowly. Furthermore, Figures 4 and 5 provide guidelines for us to decide which network structures to use according to different scenarios.


## 6. Conclusion

A range of equipments are used by individuals to attain comfort in various aspects of their lives. The widespread use of heating, ventilation, and air conditioning (HVAC) systems has raised the degree of comfort in commercial buildings dramatically. Despite the fact that HVAC is a relatively new approach to interior temperature control, the energy cost associated with it remains substantial, accounting for more than 40% of overall energy use. Therefore, this paper proposes an efficient and effective controller for an HVAC system based on deep reinforcement learning algorithms. Furthermore, this paper transforms the HVAC control problem into a reinforcement learning format. It then utilizes the advanced optimization methods for DNN to develop a low-cost HVAC controller. The experimental results verify the effectiveness of the proposed algorithms with application to the HVAC control. In comparison with the baseline control algorithms, the indoor temperature under the management of the suggested algorithms is more smooth and causes less temperature fluctuation. On the other hand, the proposed model saves about 50% of the energy as compared to the baseline algorithm.



---

*Source: 4157511-2022-05-13.xml*