Data Center Cooling System Optimization Using Offline Reinforcement Learning
Authors: Xianyuan Zhan
Xiangyu Zhu
Peng Cheng
Xiao Hu
Ziteng He
Hanfei Geng
Jichao Leng
Huiwen Zheng
Chenhui Liu
Tianshun Hong
Yan Liang
Yunxin Liu
Feng Zhao
Venue: ICLR 2025 Poster (2025)
Keywords: Offline Reinforcement learning
data center optimization
cooling system
energy saving
Primary area: reinforcement learning
TL;DR: We developed and deployed a sample-efficient offline RL framework for energy efficiency optimization of data center's cooling system

Abstract: The recent advances in information technology and artificial intelligence have fueled a rapid expansion of the data center (DC) industry worldwide, accompanied by an immense appetite for electricity to power the DCs. In a typical DC, around 30-40% of the energy is spent on the cooling system rather than on computer servers, posing a pressing need for developing new energy-saving optimization technologies for DC cooling systems. However, optimizing such real-world industrial systems faces numerous challenges, including but not limited to a lack of reliable simulation environments, limited historical data, and stringent safety and control robustness requirements. In this work, we present a novel physics-informed offline reinforcement learning (RL) framework for energy efficiency optimization of DC cooling systems. The proposed framework models the complex dynamical patterns and physical dependencies inside a server room using a purposely designed graph neural network architecture that is compliant with the fundamental time-reversal symmetry. Because of its well-behaved and generalizable state-action representations, the model enables sample-efficient and robust latent space offline policy learning using limited real-world operational data. Our framework has been successfully deployed and verified in a large-scale production DC for closed-loop control of its air-cooling units (ACUs). We conducted a total of 2000 hours of short and long-term experiments in the production DC environment. The results show that our method achieves 14-21% energy savings in the DC cooling system, without any violation of the safety or operational constraints. We have also conducted a comprehensive evaluation of our approach in a real-world DC testbed environment. Our results have demonstrated the significant potential of offline RL in solving a broad range of data-limited, safety-critical real-world industrial control problems.

Published as a conference paper at ICLR 2025
DATA CENTER COOLING SYSTEM OPTIMIZATION USING OFFLINE REINFORCEMENT LEARNING
Xianyuan Zhan 1.2+ Xiangyu Zhu' Peng Cheng', Xiao Hu', Ziteng He', Hanfei Geng',
Jichao Leng', Huiwen Zheng', Chenhui Liu', Tianshun Hong", Yan Liang',
Yunxin Liu Feng Zhao
Institute for AT Industry Research, Tsinghua University
2 Shanghai Artificial Intelligence Laboratory
3 Global Data Solutions Co., Ltd. cn, Izhanxi anyuan, liuyunxin] B r. ts nigua edu. cn, fzu a I lum, mi t edu
ABSTRACT
The recent advances in information technology and artificial intelligence have fueled rapid expansion of the data center (DC) industry worldwide, accompanied by an immense appetite for electricity to power the DCs. typical DC, around 30~40% of the energy is spent on the cooling system rather than on computer servers, posing pressing need for developing new energy-saving optimization technologies for DC cooling systems. However, optimizing such real-world in- dustrial systems faces numerous challenges, including but not limited to lack reliable simulation environments, limited historical data, and stringent safety and control robustness requirements. In this work, we present a novel physics-informet offline reinforcement learning (RL) framework for energy efficiency optimization of DC cooling systems. The proposed framework models the complex dynam- ical patterns and physical dependencies inside a server room using purposely designed graph neural network architecture that is compliant with the fundamental time-reversal symmetry. Because of its well-behaved and generalizable state-action representations, the model enables sample-efficiees and robust latent space offline policy learning using limited real-world operational data. Our framework has been successfully deployed and verified in a large- scale production DC for closed-loop control of air-cooling units (ACUs). We conducted a total of 2000 hours short and long-term experiments in the production DC environment. The results show that our method achieves 14~21 % energy savings in the DC cooling system, with- out any violation of the safety operational constraints. We have also conducted comprehensive evaluation of our approach a real-world DC testbed environment. Our results have demonstrated the significant potential of offline RI in solving broad range of data-limited, safety-critical real-world industrial control problems.
INTRODUCTION
With the surge of demands in information technology (IT) and artificial intelligence (AI) in recent decades, data centers (DCs) have quickly emerged as crucial infrastructures modern society. Along with the rapid growth of the DC industry, comes immense energy and water consumption. In 2022, the global DC electricity consumption was estimated to be 240~340 TWh, accounting for around 1~~1.3 1.3% of global electricity demand (International Energy Agency, 2023). It is forecasted that by 2026, the DC energy consumption in the US will rise to approximately 6% of the country's total power usage (International Energy Agency, 2024). To deal with the considerable amount of heat generated from servers and achieve temperature regulation, cooling systems typically account for about 30~40% of total energy consumption in large-scale DCs (Van Heddeghem et al., 2014). Compared to server-side energy consumption that is primarily spent on computational tasks, reducing cooling energy consumption offers greater practical value for energy saving. How improve the energy efficiency of DC's cooling systems while ensuring thermal safety requirements has become a critical problem for the DC industry, which has great economic and environmental impacts.
"Equal contribution. Corresponding authors.

---

Published as a conference paper at ICLR 2025
a. Server room layout
Temperature fields under different server loads
Server
racks
33
Temperature ("C)
Temp. and
humiditt sensors
21
ACUs
Low server load
High server load
Figure I: Illustration of the DC floor-level cooling system and temperature fields under different server loads.
In typical DCs, cold water generated from chillers and evaporative cooling towers is sent to multiple air-cooling units (ACUs) in the server rooms to provide cold air for servers. Properly controlling these ACUs in the server room challenging industrial control task. The difficulties arise from several aspects. First, frequently changing sever loads and physical locations of servers produce complex and dynamic temperature fields inside the server room (see Figure an illustration). Reaching the maximum degree of energy saving requires joint control of multiple ACUs in a way that is fully load-aware and capable of capturing complex thermal dynamics. Second, commercial DCs have very strict thermal safety and operational requirements, making it quite challenging to strike the right balance between energy efficiency and thermal safety. Lastly, due to the complex thermal dynamics of the cooling system, it becomes exceptionally hard to build high-fidelity and scalable simulators. Although there are many efforts (Chen et al., 2019; Ran et al., 2022a;b; Mahbod et al., 2022; Wang al., 2022; Li et al., 2019; Chervonyi et al., 2022) that tried to build simulation environments based on techniques such as computational fluid dynamics (CFD) or multi-phy physics simulation, they suffer from nuanced system identification and calibration, while still having unavoidable large sim-to-real gaps. This makes the control policies learned using simulation-barn (RL) methods hardly deployable in real-world DCs. Until now, most DCs still use conventional semi-automatic control methods, such as local Proportional-Interr Integraa--Derivatii (PID) controllers for each ACU, which are manually tuned based on the expertise of human operators and operate conservatively to prevent overheating.
online reinforcement learning
The recently emerged offline RL approach (Fujimoto al., 2019; Zhan al., 2022a) has provided an attractive data-driven and simulator-free solution to overcome the above-mentioned drawbacks. It offers new possibility learn policies directly from the historical operational data of DC cooling systems, and leverage highly expressive deep neural networks to overcome the low expressiveness and scalability issues in conventional PID (Durand-Estee
et al., 2013) and model predictive control (MPC) (Lazic al., 2018; Mirhoseininejad et al., 2021; Ogawa et al., 2013) approaches. However, most existing offline RL algorithms require large amounts of training data with sufficient state- action space coverage to learn reasonable policies, otherwise will suffer from severe performance degradation (Li et al., 2022; Cheng et al., 2023). By contrast, although monitored by a large number of sensors, the historical operational data from real-world DC cooling systems are limited compared to control complexity, and the data coverage is also quite narrow as they are generated from existing conventional controllers. This reality poses stringent requirements on the -ito-f-disttibution (OOD) generalization and small-samplc learning capability for a deployable offline RL model.
In this paper, we develop physics-informed offline RL framework for energy-efficient DC cool- ing control. Specifically, we construct a special dynamics model to capture the complex thermal dynamics inside the server room, based on fundamental time-reversal symmetry (T-symmetry) compli- ance (Lamb & Roberts, 1998; Cheng et al., 2023) and graph neural network (GNN) (Kipf & Welling, 2017) architecture that embeds domain knowledge. Based on the well-behaved and generalizable latent representations provided by the model, develop sample-efficien offline RL algorithm, which learns and maximizes the value function the latent space, while regularizing the agreement of modicy-induced samples to both offline data distribution and T-symmetry consistency. The resulting algorithm enjoys great OOD generalization capability and is particularly effective given the limited real-world data availability.
Based on the proposed offline RL framework, we also developed a deployment- feriinnn system to facilitate real-world validation. Our system has been successfully deployed and verified in real-
2

---

Published as a conference paper at ICLR 2025
world large-scale commercial data center, achieving closed-loop control of its ACUs. Real-world validation experiments demonstrate that our system achieves 14-21 % energy savings in the DC cooling system without violating any safety operational constraints during total of 2000 hours of short and long-term experiments. As production DC facilities do not tolerate any safety violations, we also build real-world small-scale DC testhed environment (with 22 servers and ACU) fully evaluate and compare our approach against existing methods. Through comprchensive comparative experiments, our approach proves be safe, effective, and robust as compared to other baseline methods. Last but not least, our approach has values not restricted to the scope of data center cooling, but also broadly applicable to other lita-dimited, safety-critical industrial control scenarios.
2 BACKGROUND AND RELATED WORK
Data center cooling control optimization. The cooling loop of typical DCs consists of water-side and air-side sub-systems. The former cools water with chillers and evaporative cooling towers, while the latter circulates the cold water to ACUs on the server floors. Through air-water heat exchange, the cooled air is blown out from the ACUs, regulating the air temperature in the server room. The generated warm water is then sent back to the chillers and cooling towers for re-cooling. In this study, we focus on the air-side cooling in the server room (also called floor-level cooling (Lazic et al., 2018)), where the primary goal is to optimize the fan speed (control the airflow) and valve opening (control the amount of cold water supply) in multiple ACUs, in order to achieve energy saving while meeting the room temperature requirements and ensuring thermal safety.
Traditional air-side cooling control methods include the local PID control (Durand-Estebr et al., 2013), the two-stage method (Lazic et al., 2018; Mirhoseininejad et al., 2021; Ogawa et al., 2013: Garcia-Gabin et al., 2018), and expert-based control (Gao & Jamidar, 2014). Specifically, local PID control relies on local sensor feedback to regulate the fan speed and valve opening of each individual ACU based PID controllers, which is only applicable to small-scale control problems and unable to jointly optimize numerous ACUs. Two-stage methods first build a mechanism model and then apply optimization methods (such as MPC or linear quadratic control) to solve the cooling control problem based on the model. Both local PID and two-stage methods lack sufficient expressive power to capture complex state-action and dynamics patterns, and do not scale effectively with increasing problem size. Expert-based control leverages the experience and expertise of human operators to manage ACU cooling. requiring significant human labor and lacking transferability to different DC cooling systems. Recently, there have been many attempts to use online reinforcement learning (RI.) to solve the DC cooling optimization problem (Chen et al., 2019; Ran et al., 2022b;a; Mahbod et al., 2022; Wang et al., 2022; Li et al., 2019; Chervonyi et al., 2022; An et al., 2023). However, these studies are restricted to simulation- based policy learning and validation. For a safety-critical industrial control scenario like DC cooling, it is nearly impossible to interact with real systems during policy training, and building a high-fidelity simulator can be very costly and impractical. This makes the previous online RL methods hardly have any success in real-world deployment.
Offline reinforcement learning. Offline RL aims to solve a sequential decision- bocision-making problem formulated by Markov Decision Process (MDP), solely using fixed offline dataset D. The MDP is typically defined by a tuple (S, A,T,,, r)) (Sutton & Barto, 2018), 8), where S and .A denote the state and action spaces, respectively. T' (t, at) denotes the transition dynamics. r(se, oz) denotes the reward function. 7 is the discount factor. Our goal is to learn an optimized policy π 7'(s) based on dataset D to maximize the discounted cumulative return, i.e., R(T)) = E 7r dz)].
Under the offline setting, evaluating the RL value function in OOD regions can produce falsely optimistic values. Such exploitation errors can quickly build up during Bellman updates, eventually leading to severe value overestimation and misguiding policy learning. Hence most offline RL methods adopt various forms of data-related regularization schemes to stabilize policy learning. such as adding explicit or implicit behavioral constraints (Kumar et al., 2019; Fujimoto et al., 2019; Fujimoto & Gu, 2021; Li et al., 2022; 2023; Mao et al., 2024b), value regularization (Kumar et al., 2020; Xu et al., 2022c; Bai et al., 2021; Zhan et al., 2022b; Lyu et al., 2022; Niu et al., 2022), or adopt strict in ia-ma learning (Kostrikov et al., 2022; Xu et al., 2022a;b; Wang et al., 2024; Mao et al., 2024a). However, due to the exclusive use of strict data-related regularization, existing offline and poor OOD generalization performance (Li et al., 2022; Cheng et al., 2023), which greatly restricts their usability in most data-limited real-
RL methods often suffer from over-conservattee
3

---

Published as a conference paper at ICLR 2025
Server room layout
c. T-symmetry enforced thermal dynamncc model (TTDM)
State decoder (1(*))
x2
Latent represeniation
x2
1
NOB
Latent fonward dynamics
X2
x2
X2
Zi
%
00
Za
ALLP
Topological relationship
dynamics T-symmetry enforcement
State action encoder ((.00)
Action decoder (2g)
d. Sample-e efficient offline policy optimization
Policy evaluation
argmin ((aF)) (Q
(ctssan))
Sensor meaturements
Policy optimization
t argmax En ((ss.
(a(() a)²
External factors
Dependency among sensers
Value maximization
Behawioral
T-symmetry consistency
Dependency across ACUs
space
Figure Illustration of the physics-informed offline RL framework for energy-efficient DC cooling control.
world control scenarios. The recently proposed T-symmetry regularized offline RL (TSRL) (Cheng et al., 2023) relaxes the restrictive data-related constraints by leveraging the fundamental time- reversal symmetry (i.e., the underlying laws of physics should not change under the time-reversal transformation: - -t), which significantly outperforms existing offline RI. algorithms in terms of data efficiency and OOD generalization. Inspired by TSRL, we develop a new thysics-informed offline RL framework and a system tailored solving real-world complex, data-limited industrial control problems, such as DC cooling system optimization.
METHODOLOOGY
In this study, we develop a physics-informed offline RL framework and a system to solve the DC cooling control optimization problem. We first mathematically formulate the problem into standard MDP, with a specifically designed safety-aware reward function to ensure thermal-safe cooling control. The core component of our framework a T-symmetry enforced Thermal Dynamics Model (TTDM), with specifically designed GNN architecture to embed domain knowledge of spatial and control dependencies among sensors and ACUs. This model provides bell-behaved and generalizable representations, enabling data-efficient and robust offline policy learning in the latent space. Based on the proposed offline RL framework, we also built ACU control system that successfully deployed it in real-world DC environments. The overall framework of our method is illustrated in Figure 2.
3.1 PROBLEM FORMULATION AND REWARD DESIGN
As illustrated in Figure la, typical DC floor-level cooling systems involve several rows of server racks, flanked by two air handling rooms (AHRs) each side, each containing to 6 ACUs that blow cold air into the server rooms. The server racks are arranged with hot and cold aisles, utilizing hot (or cold) aisle containment. Temperature and humidity sensors are distributed throughout the aisles for overheating monitoring. The fan speed and valve opening can be controlled for each ACU to achieve desirable temperature regulation. However, commercial DCs have strict temperature requirements, improper control could cause cold aisle temperatures exceed the safety threshold and negatively impact server operations. To solve this problem, we formulate it into a MDP with states, actions, and reward function designed follows:
States.
The states of our problem s = {s., Sa, Se) contain three types of sensor inputs, including temperature and humidity sensor readings within the hot and cold aisles, and server rack temperature sensor readings, denoted as s,; the working states of ACUs, such as leaving water temperature (LWT), leaving air temperature (LAT),, and entering air temperature (EAT), denoted sa; and lastly, as
4

---

Published as a conference paper at ICLR 2025
the entering water temperature (EWT) of each ACU and the server power consumption cannot be manipulated or controlled by the floor-level cooling system, they are considered as external factors, denoted s.. For real-world DC server rooms, the state vector typically has around 80 dimensions after feature engineering, collected at 2 to 5-minute intervals.
Actions. The action consists of the controllable variables for all ACUs in the server room, specifically the fan speed fm (control the airflow) and valve opening Om (control the amount of water) for each ACU m. For a server room with 11 ACUs, the complete action vector has 22 dimensions.
Safety-aware reward function. We design reward function to balance energy saving and temperature regulation, taking into account both the operational parameters of the ACUs and the environmental factors within the cooling system. For an actual ACU m, high fan speed fm directly increases its power consumption, as the fan power consumption is proportional to the cube of the fan speed. On the other hand, a large valve opening Om could also marginally increase the energy consumption on the water-side sub-system. However, increasing fan speed and valve opening also improves the ACU's cooling effect, hence there is a complex trade-off. In terms of temperature safety constraints, our primary concern is whether the cold aisle temperature (CAT) T monitored by the corresponding temperature sensor violates the safety threshold pr. Additionally, for safety considerations and in consultation with an-site engineers' experience, we also regulate the LAT T," of ACU m below predefined threshold PL. The resulting reward function is thus designed as:
M
N (n + exp (T - pT)) Bs
M
M
r
91
m
32
34
Σ m==1
In (1 + exp (IT PL))
1===
==
(1) where ro is a bias constant to keep the reward positive, M is the number of ACUs, and N is the total number of temperature sensors in the cold aisles. Positive coefficients 31, 32, 33. 34 are used to weight the respective terms in the reward function, balancing energy saving optimization and temperature regulation within the cooling system.
3.2 T-SYMMETRY ENFORCED THERMAL DYNAMICS MODEL
To extract robust and generalizable representations conducive to sample-efficient offline policy learning, we construct a special T-symmetry enforced thermal dynamics model (TTDM) to model and explain the fundamental thermal dynamics patterns inside the server room. More specifically, we start by abstracting the cooling system into two coupled graph structures with corresponding adjacency matrices as illustrated in Figure 2b. In this graph, each green node represents the ACU features se and a, corresponding to its working states and controllable actions (fan speeds and valve openings), while each orange node represents sensor measurements 8,. As nearby sensor readings often have strong spatial correlation, while sensor readings themselves also have control dependencies nearby ACUS, hence we connect these two types of nodes with orange and green edges to reflect the spatial and control dependencies respectively based on domain knowledge.
The detailed encoder-decoder architecture of TTDM is shown in Figure 2c. We design a state-action encoder o(s, a) that contains a pair of GCN blocks (Kipf & Welling, 2017), to capture the spatial dependencies between sensor nodes (orange) and control dependencies across sensor nodes and ACU nodes (green). The external factors 8, are integrated through two-layer MIP derive the final embedded representations. From the encoder Φ(s, a), we can obtain the latent representations of the current state, action and next state 2s, Za, Za' from data. To further enhance the reliability and generalizability of the learned representations, we introduce a pair of ODE latent forward dynamics f(2,, 2a) = 2, and reverse dynamics g(2,,22 2a) = -2 to enforce the T-symmetry consistency (f(2,, 2 = -9(2,, ). To learn this model, we design the following loss terms:
Reconstruction loss. As mentioned above, the state-action encoder d(s, a) = (ze, za) takes state- action pairs as input and outputs their corresponding latent representations. We then use a pair of state and action decoders v). (2.) and (za) to ensure that the learned representations can be mapped back to the original data space:
(s)c(s, I|s (2.(2))11 + I|a (2
(2)
Latent ODE forward and reverse dynamics. We utilize the similar approach in Cheng et al. (2023) and Champion et al. (2019), embedding discrete-time first-order ODE system capture
5

---

Published as a conference paper at ICLR 2025
the latent forward dynamics f(2. 2a) &g. and the reverse dynamics g( [2) -2.. where 23 = 2z' Zg. The reason that we model the latent dynamics as ODE systems is to encourage learning parsimonious models (Champion et al., 2019), which enables the model to capture more fundamental properties from the data, thereby helping avoid severe over-fitting that commonly occurs in small-sample learning situations and maximally promotes generalization. Note that based on the chain-rule, write & dz, 0=. ds Vazg é. Hence to enforce the ODE property,
can use the following loss to train ff and g:
l lfwa(s, a. (V , 28 22 ab(s,a)
(3)
Üs
x - (-2.)l13 = (b(s'aa) (-8) (-
(res(s,
(4)
as'
Os'
Moreover, we also require the state decoder vb (=.) has the capability to decode & from és (i.e., ().()) = 8), to ensure is compatible with the ODE property. This implies the following loss:
Eas (s, a, 8¹) | v.. (
T-symmetry regularization. To obtaín a well-behaved latent representation derived from ø. we enforce adapted version of T-symmetry for the discrete- time MDP setting (Cheng al., 2023), by constraining the two latent ODE dynamics to satisfy f(z,, za) = - -2(zs' (a) This leads to the following T-symmetry consistency loss:
(6) Note that in above loss term, we leverage the fact 25¹ 2, + 2/3 = 2s + f(2,: za) and use g(2,
&r-sym (Ze: (Zg:
za) (2. 2n) + 9 + f(z, za), 2a)212
[" za), za) instead of g(zs,, =a) to further couple the learning process of f and g. We find this treatment can better regulate the learning process of the latent ODE forward and reverse dynamics in empirical experiments.
Final learning objective.
Finally, the complete loss function of TTDM is:
LITDM = Σ
1Erec + fidd + frve + ldo + (s.
(e,0,
3.3 HAMPLE-EFFICIEE OFFLINE POLICY OPTIMIZATION
We construct highly sample-efficient offline RL algorithm for energy-efficient DC cooling control by integrating the properties the learned TTDM. The most notable benefit of leveraging TTDM in offline policy learning lies in the well-behaved compact data representations produced by its state-action encoder ó(s, a), which are both information-rich (capturing fundamental dynamics information) and robust (well-regularized and T-symmetry preserving). This can greatly enhance offline policy learning and generalization on OOD areas, crucial for the small-sample learning setting. Consequently, instead of learning the action-value function in the original data space in typical RL algorithms, we learn our action-value function within the latent space (i.e., Q(ze, 2).). This provides more reliable value estimates even with limited offline data. Specifically, we update our Q-function using the following objective with the safety-awan reward function defined in Eq. (1):
Q = argmin F(,,,,)'>> (r(s.a) 2 (6 (s', (6))) - ()((s,a)))
(8)
For policy optimization, we adopt similar treatment as in TD3+BC (Fujimoto & Gu, 2021), where we maximize the value function Q but in the latent space, and constrain the policy output actions closer actions within the dataset. However, solely adding the regularization offline behavioral data is insufficient to ensure reasonable generalization performance. Hence we further regularize the T-symmetry consistency of policy-induced samples (s, (s)) using the T-symmetry consistency loss -ym as in Cheng et al. (2023). This enforces the policy to generate actions that are compliant with T-symmetry, even OOD areas, thereby greatly enhancing the generalization performance and sample efficiency of policy learning. The final policy optimization objective is presented as follows:
π argmax E(s,))< D ((o((((( (())) (TT() a)² ey (Ø(s, (s)))]
where we follow TD3+BC and use A, = a/IEs s (Ø(ss, (,, a) I/N] as the normalization term to balance the strength of value maximization and policy regularization (N is the number of samples in training batch). We tuned the scale parameter a in the range of [2.5, 10) during our experiments.

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of conventional PID control and our approach under comparable server load settings on two server rooms of the commercial DC. "AEP" and "EC" denote average electric power and energy consumption, respectively. We use the offline RL policy to control 4, 6, and all the ACUs in each room. ACLF is the air-side cooling load factor, calculated as the ratio of energy consumption of ACUs to servers, the lower the hetter.
PID
Ours (4 ACUs)
Ours (6 ACUs)
Ours (all ACUs)
Server Room
May 5th
May 6th
May 7th
May 8th 09:50- 17:20
May 9th 09:50 17:20
Sep 23 11:00
Nov 11 16:30-
11:00 - 17:30
09:50 17:20
11:00 - 17:30
Sep 29 10:30
Nov 12 16:30
Server AHP (KW)
555.31
552.17
548.61
549.
550.19
572.77 82199.92
577.63
Server EC (kWh)
3610.15
4141.34
3566.65
4120.42
4127.38
13864.55
ACU AEP (KW)
24.53
23.82
19.9
20.19
20.00
20.78
19.66
ACU (kWh)
159.42
178.73
129.24
151.44
149.97
2981.86
471.8
ACLF (%)
4.42
4.32
3.62 (L.8%)
3.68 (115%)
3.63 (116%)
3.63 (116%)
3.40 (121%)
PID
Our (4 ACUs)
Ours (6 ACUs)
Ours (all ACUs)
Server Room B
May 5th
May 6th
May 7th
May 8th
May 9th 09:50 17:20
Sep 23 11:00-
Oct 30 10:10 10
11:00 - 17:30
09:50 17:20
11:00 - 17:30
09:50- 17:20
Sep 29 10:30
Nov 1 17:30
Server AEP (kW)
617.18 18
602.04
593,28
610.57
611.34
576.52
619.55
Server EC (kWh)
4010.83
4520.42
3853.19
4579.69
4586.52
82746.24
34302.42
ACU AEP (kW)
37.2
36.38
30.58
31.66
31.76
29.15
30.06
ACU (kWh)
241.79
272.9
198.75
237.43
238.15
4183,44
1663
ACLF (S6)
6.03
6.04
5.16 (3.4%)
5.18 (1149)
5.19 (114%)
5.06 (1169))
4.85 (120%)
4 REAL-WORLD EXPERIMENTS
To validate our proposed physics-informed offline RL framework, we develop a deployment-friendly
software system to support the close-loop control of ACUs using the learned policy. We successfully deployed our system and conducted a series of experiments (from January to December 2024) in large-scale commercial data center in China, controlling up to 4, 6, and all (10 or 11) ACUs in two of its server rooms (referred as Room A and B in the later content). Our method has been operated effectively and safely for over 2000 hours in total. As conducting experiments in a production environment suffers lots of restrictions, to further validate our method, we also built a real-world small-scale DC testbed to conduct more comprehensive comparative experiments and model ablations. The testbed contains 22 servers and an ACU, and supports testing a wide range of server load settings. More information about the two real-world DC testing environments and the collected historical operational datasets can be found in Appendix B. Throughout this study, we train and validate our model on real-world data and environments, with completely simulation involved.
4.1 VALIDATION ON REAL- WORLD DATA CENTER
Comparison with conventional control. We first compare our DC cooling optimization method with the default ACU PID controllers on two server rooms the real-world commercial data center. As our experiments are conducted in the real production environment, we are only allowed by the DC operator to control 4 out of 11 ACUs in the room in the early stages of the experiment, once the effectiveness was validated, we proceeded with experiments controlling 6 ACUs and then all ACUs in a server room. To ensure a fair comparison, we select several time periods (lengths from 5.5 7.5 hours) that have similar server load patterns for comparison. Table I shows the results on energy consumption metrics, including the average electric power and total energy consumption of servers and the ACU cooling system. We use the Air-side Cooling Load Factor (ACLF) analyze the floor-level cooling system's 's energy efficiency, which is widely adopted by the DC industry. It is calculated as the ratio of the ACU system's 's energy consumption the servers' energy consumption during the test period. Lower ACLF indicates higher energy efficiency. In the tested two server rooms, our method improves the cooling system's 's energy efficiency by 14% to 21 % compared to the default PID controllers. Throughout our experiment, we observed no thermal safety violations and regulated the cold aisle temperature (CAT) well below the required operational threshold.
Control quality. We also conducted consecutive 48-hour experiments to compare the control behaviors of our method and the PID controllers in Server Room B with fluctuating server loads. The results presented in Figure 3, where compare the same 4 controlled ACUs and the temperature variation patterns of the directly impacted hot and cold aisles. As shown in Figure 3a, during the periods controlled by the PIDD controllers and our method, the total server load fluctuated
7

---

Published as a conference paper at ICLR 2025
PID Controller
Ours
5-13 12am
5 tam
5-13 12mm
5-13 Spmn
S- 2Jom
bam
5 12em
5-14 6pm
5-13 12am
6:29 1zom
6-20 6am
6:29 1am
6:29 spm
6:30 1aam
6:30 Bam
6 12pm
6:30 6pm
74 74
or Server load (le0) P
60c-
6500
ACLF (19
570-
Total server load
ACLF
5.5
540
75
(8)
60-
60-
45
Valve (80
21
70-
60-
60
50
50
Alsle temp, [n]]
30
30
25
HAT
25
20-
20:
Figure 3: Comparisons of key system metrics and the controllable actions of our method and the PID controller over 2-day testing periods in Server Room B. Figures on the left show results from the PID-controlled period (May 13-15, 2024), and figures on the right are the results controlled by our method (June 29 July 1, 2024).
at a similar level, but our method consistently achieved noticeably lower ACLF value than that of the PID controller, indicating higher energy efficiency. Figures 3b and 3c, we compare the controllable actions (fan speeds and valve openings) of the 4 controlled ACUs during the test period. For fan speeds, several ACUs controlled by the default PID controller remained almost constant for a long time, whereas the fan speeds of the 4 ACUs controlled by our method were dynamically adjusted throughout the testing period. Notably, during the PID control phase, there was a short period having drastic adjustments in fan speed and valve opening, while such abnormal control behavior was not observed in our method. In terms of overall control behavior, our method tends to lower the fan speeds while slightly increasing the cold water valve openings, which helps reduce ACU energy consumption while maintaining the same level of cooling capacity. Figure 3d shows hot and cold aisle temperature variations during the testing periods. The solid curve and shaded area represent the mean and the msd envelop of multiple temperature sensor readings. Our method slightly decreased the cold aisle temperature, even with less ACU energy consumption (lower ACLF). Moreover, we find that our method achieves significantly better temperature regulation for the hot aisle, which results in much more concentrated temperature distributions as compared to the PID controller, indicating a more uniform and stable temperature field inside the hot aisle. More results that showcase the superior adaptability of our method under drastic server load fluctuations can also be found in Appendix C.
Long-term fang-termae control performance. To verify the long-term robustness and energy-saving effectiveness of our method, we conducted two 14-day experiments by continuously running our offline RL policy and the PID controller on the 4 controllable ACUs in Server Room B. Our model was in operation from June 17 to July 1, 2024, while the PID controller was in operation from July 2-16, 2024. Figure presents the results of energy efficiency and temperature conditions of the directly influenced cold and hot aisles (see Appendix B. for details). In Figure 4a, each point represents the average total server load within an hour and the corresponding calculated ACLF value. The ACLF values of our model are consistently lower than those of the PID controller across all server load conditions, with even lower ACLF values observed under higher server loads. This again demonstrates the load-awareness of our approach, which enjoys a greater level of energy saving with the increase of server loads, forming sharp contrast to the almost constant ACLF level of the PID control. Figure 4b illustrates the temperature distribution in the most relevant hot aisle, where the PID controller resulted in a distribution clustering around C9 and 3.. By contrast, our method maintained a more concentrated temperature distribution around 3 leading to more uniform temperature field inside the hot aisle during the testing period. Figure 4c shows the temperature distribution of the two most relevant cold aisles during the 14-day experiments, both methods regulated the cold aisle temperature below the operational threshold of C5ºC These results demonstrates the potential of our method for safe and stable long-term deployment in weal-world data centers.
8

---

Published as a conference paper at ICLR 2025
6.5
Probability Density
0.6
PID Ours
Probability Density
1.5
Ours
PID
Target value (25%g)
ACLF (58)
6.0
0.4-
1.0
5.5
0.2
0.5
PID
5.0
500
550
600
650
0.0
30
24 Cold Aisle Temperature (*C)
Total Server Load (kW)
Hot Aisle Temperature
Figure 4: Results of the 14-day long-term experiments in Server Room B. ACLF values under different total server loads. b, c. Temperature distribution of the directly influenced hot and cold aisles.
Server Room A (Morning)
Server Room B (Morning)
Server Room B (Afternoon)
4.5
4.3%
4.1%
6
6.0%
5.6%
6.15
5.7%
5.5%
5.45
ACLF (88)
4.0
3.9%
5.2%
5.27
5.2%
5.0%
3.5
3,4%
3.0
PID
1
2
3
6
All (12)
PID
1
2
3
6
All [10)
PID
2
4
All (10)
Controlled
Controlled count
Controlled ACU
Figure 5: The energy-saving impact of controlling different numbers of ACUs through our approach.
Impact of the number of controlled ACUs. We also conducted additional experiments with our model controlling 1 to all ACUs to further investigate its energy-saving impact. The results are presented in Figure 5, which clearly show an increasing trend of energy efficiency with more ACUs controlled by our method. Figure 5a shows the experiment results conducted seven morning periods (10:30 13:30) in Server Room A; Figure 5b,c the right show the experiment results conducted seven morning (10:30 13:30) and afternoon (14:30 17:30) periods Server Room B. These promising results suggest that if more ACUs can be controlled by our method, it is very likely that we can achieve even higher energy efficiency.
4.2 EVALUATION AND ABLATION ON THE TESTBED
As testing in the production DC environment suffers lots of restrictions, to further validate our method, we conducted extensive exploratory experiments and model ablations in our testbed environment.
Comparative evaluation against baseline methods. We compare method with competing baseline methods including conventional industrial control methods PID and MPC (Lazic al., 2018), off off-policy RL-based DC cooling optimization method CCA (Li et al., 2019), mainstream offline RL algorithms IQL (Kostrikov et al., 2022) and CQL (Kumar et al., 2020), and the state-of-the- art safe offline RI. algorithm FISOR (Zheng al., 2024) (see Appendix D.2 for detailed descriptions). For the comparative experiments, we tested three server load conditions: low, medium, and high loads, with average electric power of 4.9kW. 7.4kW, and 8.0kW, respectively. Each method controlled the ACU in closed-loop mode for 6 hours under the same experimental conditions, and we recorded the energy efficiency and thermal safety metrics, i.e., ACLF and CAT violations (proportion of time steps during the experiment that the CAT exceeds the pre-defined threshold). To make the task more challenging, we set lower CAT threshold (22ºC) as compared to the one used in the commercial DC to test the capability of the algorithm in balancing energy saving and temperature regulation. The results are reported in Figure 6. Due to the smaller scale of the testbed and significantly lower server load as compared the real-world DC, the calculated ACLF values are higher than those observed in the real DC experiments. We observe some aggressive baseline methods (CCA and CQL) achieve lower energy consumption but perform poorly in terms of thermal safety, which is unacceptable. By contrast, our method achieved the highest energy efficiency under all load conditions, while ensuring no CAT violations throughout the experiments, outperforming all other baseline methods.
Ablation study, addition, we conducted ablation experiments to validate the effectiveness of key designs in method, including the GNN architecture and T-symmetry enforcement. Additional

---

Published as a conference paper at ICLR 2025
CAT viclation (1)
ACLF (80
40-
40
20
20
P10
MPC
CCA
IQL
COL
FISOR
Ours
FID
MPC
CCA
K0L
COL
FISOR
Ours
server
server
High server
Figure 6: Comparative evaluation of method against baseline methods on our real-world testbed.
0.04
0.03
w/o T-symmetry T-symmetry
45
Prediction error
Prediction error
0.03
(9)
30
0.02
GNN with GNN blocks
0.01
0.01
0.00
0
2
8
0.00
Low load
High load
Prediction steps
Prediction steps
W/o T-symmetry
With T-symmetry
Figure 7: Ablation experiments on the impact of GNN blocks and T-symmetry enforvement in our method.
ablation results the reward function design can be found in the Appendix C.2. In Figure 7a, b, we compare the multi-step prediction error of our proposed TTDM trained on the historical data of Server Room B with and without the GNN structure and T-symmetry enforcement. The prediction errors measured in terms of mean square error (MSE) on the predicted future states. The results show that incorporating domain knowledge (spatial and control dependencies among sensors and ACUs) using GNN blocks significantly reduces T'DD'ss prediction error, especially when the number of prediction steps increases. We also obtain similar results when incorporating T-symmetry enforcement in TTDM, which demonstrates that both the GNN architecture and T-symmetry design can substantially improve the capacity and generalization of our thermal dynamics model, thereby providing better modeling and representation of the offline dataset. In Figure 7c, we compare the offline policy optimization results of our method with and without T-symmetry under low and high server load conditions on the testbed. Each experiment ran continuously for 6 hours. The results show that, the version of our offline RL framework with T-symmetry achieves much better energy efficiency improvements in both load conditions compared to the version without T-symmetry. This indicates that T-symmetry plays a crucial role in enhancing the generalization during policy learning, therefore resulting in more performant policy given limited real-world data.
CONCLUSION
In this study, develop physics-informed offline RL framework and deployable system for energy-efficient DC cooling control. The core of our framework graph-structured and T-symmetry consistent thermal dynamics model, which provides well-behaved and generalizable representations, enabling highly sample-efficient offline policy learning in the latent space. Our system has been successfully deployed and validated in real-world large-scale commercial data center and achieved closed-loop control of its ACUs. Our empirical results show that our proposed method can achieve 14~21% energy savings in the real-world DC cooling system, and ran smoothly without any safety or operational constraints violation during long-term experiments. We also provide comprehensive comparative evaluations and ablations of our approach in real-world small-scale DC testbed environment that is constructed specifically for this research. Our work demonstrates the huge potential of offline RL in solving a broad range of complex real-world industrial control problems, especially for those having limited historical data and impossible to build high-fidelity simulators. Lastly, we also urge the RL community to move away from current toy simulation based RL benchmark environments and focus based RL benchmarks have many unrealistic and biased dataset/task settings, which often provide misleading insights that mismatch with observations in real-world practices.
real-world control problems. The current simulation-
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported by Carbon Neutrality and Energy System Transformation (CNEST) Program, and funding from Global Data Solutions Co., Ltd and Wuxi Research Institute of Applied Technolo- gies, Tsinghua University under Grant 20242001 120. We are especially grateful for all the support from Global Data Solutions Co., Ltd in our real-world DC experiments.
REFERENCES
Zhiyu An, Xianzhong Ding, Arya Rathee, and Wan Du Clue: Safe model-based hvac control using epistemic uncertainty estimation. In Proceedings of the 10th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation, pp. 149-158, 2023.
Kiam Heong Ang, Gregory Chong, and Yun Li. Pid control system analysis, design, and technology.
IEEE transactions on control systems technology, 13(4):559-577 2005.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhi-Hong Deng, Animesh Garg, Peng Liu, and Zhao- ran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. In International Conference on Learning Representations, 2021.
Kathleen Champion, Bethany Lusch, Nathan Kutz, and Steven L Brunton. Data-driven discovery of coordinates and governing equations. Proceedings of the National Academy of Sciences, 116(45): 22445-222 22451,, 2019.
Bingqing Chen, Zicheng Cai, and Mario Bergés. Gnu-rl: A precocial reinforcement learning solution for building hvac control using differentiable mpe policy. In Proceedings of the 6th ACM international conference on systems for energy-efficies buildings, cities, and transportation, pp. 316-325, 2019.
Peng Cheng, Xianyuan Zhan, Wenjia Zhang, Youfang Lin, Han Wang, Li Jiang, et al. Look beneath the surface: Exploiting fundamental symmetry for sample-efficient offline Advances Neural Information Processing Systems, 36, 2023.
Yuri Chervonyi, Praneet Dutta, Piotr Trochim, Octavian Voicu, Cosmin Paduraru, Crystal Qian, Emre Karagozler, Jared Quincy Davis, Richard Chippendale, Gautam Bajaj, et al. Semi-analytical industrial cooling system model for reinforcement learning. arXiv preprint arXiv:2207.13131, 2022.
Baptiste Durand-Esteber Cédric Le Bot, Jean Nicolas Mancos, and Eric Arquis. Data center optimiza- tion using pid regulation in cfd simulations. Energy and Buildings, 66: 154-1,4 2013.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072199 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052-2062. PMLR, 2019.
Jim Gao and Ratnesh Jamidar. Machine learning applications for data center optimization. Google
White Paper, 21, 2014.
Winston Gabiia-Gabin, Kateryna Mishchenko, and Erik Berglund. Cooling control of data cen- ters using linear quadratic regulators. In 2018 26th Mediterranean Conference on Control and Automation (MED), pp. 1-6. IEEE, 2018.
International
Energy Agency. works, 2023.
Data centres and data transmission
net-
URL https://www iea httppp/s//ww...aa.ocrree/eeeeeee http:///wwww.aaao//eeeeeeeeeeeee

---

Published as a conference paper at ICLR 2025
International Energy Agency. Electricity 2024 analysis and forecast to 2026, 2024. https: /tt////wwww... iea.
URL httpp://www http://www..aaa....prrootttttttt
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks.
In International Conference on Learning Representations, 2017.
Iya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, Jun 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 179-1191, 2020.
Jeroen S.W. Lamb and John A.G. Roberts. Tive-reversal symmetry in dynamical systems: A survey, Physica D: Nonlinear Phenomena, pp. 1-39, Jan 1998. doi: 10. 101
Nevena Lazic, Craig Boutilier, Tyler Lu, E. Wong, Binz Roy, M. Ryu, and Greg Imwalle. Data center cooling using model preeelppreiieive control. Advances Neural Information Processing Systems, Jan 2018.
Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang. When data geometry meets deep function: Generalizing offline reinforcement learning. The Eleventh International Conference on Learning Representations, 2022.
Jianxiong Li, Xiao Hu, Haoran Xu, Jingjing Liu, Xianyuan Zhan, Qing-Shan Jia, and Ya-Qin Zhang. Mind the gap: Offline policy optimization for imperfect rewards. In The Eleventh International Conference on Learning Representations, 2023.
Yuanlong Li, Yonggang Wen, Dacheng Tao, and Kyle Guan. Transforming cooling optimization for green data center via deep reinforcement learning. IEEE transactions on cybernetics. 50(5): 2002-2013, 2019.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Frez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 1=Xxvv:509.00977 2015.
Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:1711-1724, 2022.
Muhammad Haiqal Bin Mahbod, Chin Boon Chng, Poh Seng Lee, and Chee Kong Chui. Energy saving evaluation of an energy efficient data center using a model-free reinforcement learning approach. Applied Energy. 322:1 19392, 2022.
Liyuan Mao, Haoran Xu, Xianyuan Zhan, Weinan Zhang, and Amy Zhang. Diffusion-dice. In- sample diffusion guidance for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2024a.
Liyuan Mao, Haoran Xu, Weinan Zhang, and Xianyuan Zhan. Odice: Revealing the mystery of distribution correction estimation via orthogonal-gradiee update. The Twelfth International Conference on Learning Representations, 2024b.
SeyedMorteza Mirhoseininejad, Ghada Badawy, and Douglas G Down. A data-driven, multi-setpoint model predictive thermal control system for data centers. Journal of Network and Systems Management, 29:1-22, 2021.
Haoyi Niu, Shubham Sharma, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming Hu, and Xianyuan Zhan. When to trust your simulator: dynamics-aware hybrid aedine-and-onliin reinforcement learning. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pp. 36599-36612, 2022.
12

---

Published as a conference paper at ICLR 2025
Masatoshi Ogawa, Hiroshi Endo, Hiroyuki Fukuda, Hiroyoshi Kodama, Toshio Sugimoto, Takeshi Horie, Tsugito Maruyama, and Masao Kondo. Cooling control based on model predictive control using temperature information of it equipment for modular data center utilizing fresh-air. In 2013 13th International Conference on Control, Automation and Systems (ICCAS 2013), pp. 1815-1820. IEEE, 2013.
Yongyi Ran, Han Hu, Yonggang Wen, and Xin Zhou. Optimizing energy efficiency for data center via parameterized deep reinforcement learning. IEEE Transactions on Services Computing, 16(2):
1310-1323, 2022a.
Yongyi Ran, Xin Zhou, Han Hu, and Yonggang Wen. Optimizing data center energy efficiency via event-driven deep reinforcement learning. IEEE Transactions Services Computing, 16(2): 1296-1309, 2022b.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. 2018.
Ward Van Heddeghem, Sofie Lambert, Bart Lamnoo, Didier Colle, Mario Pickavet, and Piet Demeester. Trends in worldwide ict electricity consumption from 2007 to 2012. Computer communications, 50:64-76, 2014.
Ruihang Wang. Xinyi Zhang, Xin Zhou, Yonggang Wen, and Rui Tan. Toward physics-guided safe deep reinforcement learning for green data center cooling control. In 2022 ACM/IEEE 13th International Conference Cyber-Physical Systems (ICCPS), pp. 159-169. IEEE, 2022.
Xiangsen Wang, Haoran Xu, Yinan Zheng, and Xianyuan Zhan. Offline multi-agent reinforcement learning with implicit global-to-loce value regularization. Advances in Neural Information Processing Systems, 36, 2024.
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang. Victor Wai Kin Chan, and Xianyuan 7han. Offline rl with no ood actions: In-sample learning via implicit value regularization. In The Eleventh International Conference on Learning Representations, 2022a.
Haoran Xu, Jiang Li, Jianxiong Li, and Xianyuan Zhan. A policy- guiiee imitation approach for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2022b.
Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu, Constraints penalized q-learning for safe offiin reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022c.
Xianyuan Zhan, Haoran Xu, Yue Zhang, Xiangyu Zhu, Honglei Yin, and Yu Zheng. Deepthermal: Combustion optimization for thermal power generating units using offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 4680-4688, 2022a.
Xianyuan Zhan, Xiangyu Zhu, and Haoran Xu. Model-based offline planning with trajectory pruning. In International Joint Conference on Artificial Intelligence (ICCAI), 2022b.
Yinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang. Shengbo Eben Li. Xianyuan Zhan, and Jingjing Liu. Safe offline reinforcement learning with feasibility diffusion model. In The Twelfth International Conference on Learning Representations, 2024.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
A SYSTEM DEPLOYMENT
We have developed full-function software system to facilitate the deployment and validation of our proposed physics-informed offline RL framework. We successfully deployed our system in a large- scale commercial data center for production environment performance validation and the small-scale DC testbed for more comprchensive model evaluation and ablation. The overall deployed system architecture illustrated in Figure 8, which consists of two main phases: offline training and online deployment. In the offline training phase, the historical operational data of the floor-level cooling systems is exported from the DC log management system. The exported data undergoes automated data processing and feature engineering processes and stored in a historical dataset. Subsequently, based on the processed offline dataset, we train the T-symmetry enforced thermal dynamics model, followed by a sample-efficient offline policy learning module obtain the optimized floor-level cooling control policy. In the online deployment phase, the learned policy is deployed in local policy server within the data center to provide control services. Real-time data from the cooling systems is retrieved by the management system API, processed, and stored in real-time database. The system then forwards the real-time data to the policy server, which outputs optimized ACU control actions. These optimized control actions directly written into the ACUs via the Modbus protocol for closed-loop control.
Our developed system is deployment frienddl and broadly applicable various DC floor-level cooling systems with different configurations, exhibiting great flexibility and transferability. Moreover, environmental and server load conditions in the data center frequently change over time, the completely data-driven design of our system offers extra advantages. As it allows for re-collection of new historical data every few months, and uses the new data retrain and fine-tune the ACU control policy accordingly. This endows our system with high adaptability, providing an evolvable control optimization solution to a slowly changing industrial system.
Log
ACUs in the server room
log management system
Training
Historical data
Management system API
Modbus protocol
Control
Real-time states
T-symmetry enforced thermal dynamics model
Sample-efficient offline pollcy learning
Optimized actions
Policy
Real-time data
Deploy
server
Offline Training
Online Deployment
Figure Overall architecture the deployment system.
B REAL-WORLD TESTING ENVIRONMENTS AND EXPERIMENT SETUPS
B. PRODUCTION DATA CENTER ENVIRONMENT
Figure presents some photographs and the layout illustration of our real-world data center testing environment. In this large-scale commercial data center, we are granted permission conduct experiments in two designated server rooms. These server rooms host the real IT loads of a large video-sharing website in China. Specifically, in Server Room A, the average total server load around 550 kW, with an overall ACU power consumption of around 25 kW; in Server Room B, the average total server load is around 610 kW, and the overall ACU power consumption is about 37 kW. In the early stages of the experiment, we are only allowed to control 4 ACUs in each server room (ACU 1-6, 1-5, 2-5, and 2-4 on the left side in Server Room A; ACU 1-1, 1-2, 2-1, and 2-2 on the right side in Server Room B). These ACUs are arranged in pairs opposite sides of the room, directly influencing two cold aisles and one hot aisle. The remaining ACUs continue to operate under PID control. After verifying the effectiveness of the experiments, further used the model control
14

---

Published as a conference paper at ICLR 2025
a
ACU 2-5
ACU 2-4
ACU
ACU 2-2
ACU 2-1
ACU 1-6
ACU 1-5
ACU 1-4
ACU 1-2
ACU 1-1
Figure 9: The photographs and layout illustration of the real-world commercial data center. a. Photographs of the interior of server room, showeasing the hot aisle, cold aisle, and server racks from left to right. Overhead panoramic view of server room, illustrating the spatial arrangement of all pertinent equipment.
ACUs in each server room (ACU 1-2, ACU 1-4, ACU 1-6, ACU 2-1, ACU and ACU 2-4 in each server room). Finally, conducted experiments controlling all ACUs in both server rooms (In Server Room B, all 10 ACUs are shown in Figure 9b. In Server Room A, there are 11 ACUs, with an additional ACU (ACU 1-3) located in the position marked as 'UNUSED' at the bottom of Figure 9b). As we are testing on the safety-critical real production environment, it is not possible for us to fully evaluate and test other baseline methods, they may not have strong safety assurance. We leave these comparative experiments to our testbed environment, where we have full control.
We follow the DC industry's 's standard practice that specifies the target threshold of cold aisle temperature (CAT) as 25 For Server Room A, we collected about 20 months' historical operational data from the logging system, including approximately 180,000 data samples 5-minute intervals, involving 108 state and action features. Similarly, for Server Room B, we collected historical data over 15 months, amounting to approximately 140,000 data samples, also 5-minute intervals, and encompassing total of 101 state and action features. The amount of real-world data available to train our offline RL policy is significantly fewer than typical offline RL benchmark tasks like D4RL (Fu et al., 2020) (often using million data samples to learn simple tasks), especially considering the much larger scale of our problem. We run a series of offline policy evaluation tests and open-loop inspections select the best-performing models and deploy them in real systems for closed-loop control evaluation. During this phase, our control system takes the real-time data from the cooling system every five minutes as inputs, then computes the optimized actions and directly transmits these commands to the ACUs for modifying fan speeds and valve opening percentage. We conducted series of short and long-term experiments from January to December 2024. Our system has been operated safely for over 2000 hours. Through these comprehensive experiments, we verified that our proposed physics-informed offline RL framework and the resulting control system can operate both effectively and safely under the stringent safety and operational constraints of a real-world commercial data center.
B.2 REAL- WORLD TESTBED
To thoroughly assess the performance of our proposed method, we also constructed a real-world testbed environment, which contains 22 servers and an inter-column air conditioner the ACU
15

---

Published as a conference paper at ICLR 2025
Sense
Sensor
Sense
b
4'sensor
back Rack B02 front
front
Rack A02
back
sesensor
MXU
sensor outlet
ACU
inlet
4 *scnsor
4'sensor
sssnsor
back
Rack B01 front
front Rack A01
back
glass
PQU
sliding
glass
doors
doors
doors
Figure 10: The photographs and layout illustration of our constructed small-scale DC testbed. a, Illustration of the installed temperature and humidity sensors in our testbed. b. Layout illustration of the testbed.
(located between Rack I and Rack 2). This is compressor-based ACU, which is smaller than the typical ACUs in commercial data centers that the cold water from chillers and cooling towers as the cold source. Therefore the fans and the compressor inside the ACUs are the primary contributors the ACU's energy consumption. For the testbed environment, temperature regulation is achieved by adjusting the entering air temperature (EAT) setpoint of the ACU to ensure the CAT remains below the predetermined threshold. We installed sets of temperature and humidity sensors (24 in total) to monitor the internal temperature field inside our DC testhed environment. Moreover, we also have access to the interior temperature sensor readings from each server, which provides even finer-grainet grained monitoring of the thermal dynamics inside the testbed. Figure 10 provides detailed depiction the testbed environment configuration.
To support testing with wide variety of server loads, also developed software framework to assign servers with different load patterns that mimic real-world tasks. The software employs a Kubernetes (k8s) cluster architecture and is implemented under the CentOS Stream operating system. The ACU control is implemented through the Modbus protocol, which regulates the setpoint of the Entering Air Temperature (EAT) of the ACU, thereby indirectly adjusting the fan and compressor of the ACU. In our experiments, the control policies calculate and output the EAT setpoint every two minutes and control the ACU accordingly. The experimental server loads in our testbed range from approximately 5 to 8 kW, while the power consumption of ACU varies from 1.5 to 4 kW. We also built data collection and database management system using InfluxDB and Telegraf handle and store the real-time and historical data in our testbed. We collected the historical operational data over 61 days, comprising approximately 43,000 data samples 2-minute intervals, comprising 105 state and action features.
As we have complete control over our testbed, we can conduct extensive exploratory experiments with our proposed method and compare it with wide range of existing baseline methods without restriction. As temperature regulation in the DC testbed is comparably easier than in the large-scale production DC environment, employed stricter 229C CAT threshold to make the control tasks more challenging. Furthermore, we also conducted experiments on the impact of weight coefficients reward function and carried out ablation studies to further evaluate our method.
16

---

Published as a conference paper at ICLR 2025
b
Server (KVV
100
23.0
72
Valve opening (%)
52-
PID
Fan speed (9)
Ours
80
64-
PD
48-
CTT
Ours
Rack E
Rack C
56
44-
100
300
100
200 Time (min)
100
200 Time (min)
300
Time [min]
Time (min)
Figure 11: ACU control behaviors of our method and the PID controller under drastic server load fluctuation. Load variation pattern of three server racks (Rack C, D, E) during the selected time period, with one server rack having drastic load drop and increase. b, Temperature readings from the three most relevant cold aisle sensors. e, d, The variations in fan speed and valve opening for two ACUs during the time period, with one controlled by the PID controller (ACU 1-1) and the other by our method (ACU 1-2).
C ADDITIONAL RESULTS
C.I PERFORMANCE UNDER DRASTIC SERVER LOAD FLUCTUATION
To further evaluate the adaptability and load-awareness our method, we tested on a specific scenario with drastic server load fluctuations in Server Room B. We compare the control strategy of two ACUs with one controlled by the default PID controller and the other by our method. Experimental results presented in Figure 11. The PID controller demonstrates limited adaptabilityy in this scenario, with no adjustments to fan speeds and only marginal changes in valve opening percentage. In contrast, our offline RL approach was able to promptly adapt external changes, resulting in eneryy-efficient strategy. These results underscore the effectiveness and adaptability of our approach in highly dynamic DC service conditions.
optimal and
C.2 ADDITIONAL ABLATIONS REWARD FUNCTION DESIGN
We considered both the control parameters the ACUs and environmental factors within the cooling system to design reasonable reward function for RL policy learning. For the weight coefficient $1, 82, 83, 31 in the reward function Eq. (1), we set their values as the reciprocal of the mean of the corresponding reward term calculated based on the preprocessed dataset. This ensures each reward term has a similar scale. For the first constant term ro in the reward function, to keep the reward positive, calculate the of the other terms in the reward function for each record in the preprocessed dataset and take their maximum value plus as the value of ro-
To further investigate the robustness of our reward function design, we also conducted additional experiments on the testhed by varying the relative scale of the third term (52 >. In(1 + exp(TTT Pr))) in Eq. (1), which controls the strength of CAT violation penalty. Specifically, we test the default value of 32 as well as multiply by 5 and 10, to test the impact of prioritizing more on safety constraint satisfaction. We train three models with different 32 values and use the resulting models to control the ACU for 6 hours under low and high server load conditions on the testbed. In all these experiments, the CAT was controlled below the predefined threshold. Moreover, reported in Table 2, the energy-saving performances of the models under different /32 weight coefficients consistently achieve comparable and low ACLF values. This shows our designed reward function robust and does need much tuning to ensure good practical performance, which is particularly desirable for real-world deployments.
Table 2: Performance on the testbed using different scale of 3g in the reward function.
Default 32
5 /32
10 x $3
ACLF (%) under low server load
29.66 26.89
29.37 27.50
30.95
ACLF (%) under high server load
26.05
17

---

Published as a conference paper at ICLR 2025
12:10 Spon
H4-22 Ezen
13-22 dum
12.22 1imm
11-1 Spmn
at Zomn
1a-2 fama
13-12 1Jpmi
10-29 Spm
10:30 Laan
10:30 3tm
10 10-90 Sam
10:30:
t0 10 12pm
10-30 3pm
10 6pm
10:30 Spm
750-
PiD
Ours
Server load (XW)
750
PID
ACLF (96)
ACLE
Server
450
Total server
ACLF
450
Total server load
ACLF
Freq (H2))
80
CWP free (He)
EWIT ('C)
EVIT (C))
02.
CWP freq
EWT
CWP freq
EWT
In the full control experiment of ACUS in the commercial data center, the water-side related states are as follows. a, In Server Room A, before and after the use of our offline RI. policy for control, the overall server load remains stable, and both the chilled water pump frequency (CWP freq) and the ACUS' entering water temperature (EWT) also stay stable. After implementing our control policy, the ACLF value significantly decreases. b, In Server Room B, a similar comparison of the server load and wader-side indicators before and after the use of our offline RL policy for control, which shows consistent results with those in part a.
Figure 12:
C.3 ANALYSIS OF IMPACTS ON THE UPSTREAM WATER-SIDE COOLING SYSTEM
To evaluate the potential impact of optimizing the air-side cooling system using our method on the upstream water-side cooling system, we conducted additional analysis the water-side related states through two before-and-afeee tests. We select two time periods with relatively stable server loads in the server rooms (November 10-11 1 for Server Room A and October 29-30 for Server Room B) to compare the chilled water pump frequency (CWP freq) and the ACUs' entering water temperature (EWT) before and after using our offline RL policy for control. The CWP freq. and EWT are key states that reflect the working conditions of the water-side cooling system, which are external factors to the air-side cooling systems. Figure 12 shows the experimental results during the full control of all ACUs in Server Room A and Server Room B. The Figure 12a and 12b, the dashed vertical lines indicate the time points when our method took over the control. In both Server Room and Server Room B, before and after our method began controlling, the average EWT of the ACUs remained stable. Additionally, the chilled water pump frequency of the water-side cooling system also did not exhibit significant variations. However, comparing the results before and after adopting control method, the ACLF values in both rooms significantly decreased. These results demonstrate that although our method effectively reduces the air-side cooling system's 's energy consumption, it does not have a noticeable impact on the upstream water-side cooling system. Moreover, as also shown in Section 4.1, Figure 3 and 4, our offline RL policy enables much better temperature regulation and forms a more stable temperature field for the hot aisles, due to smartly coordinating the control of all ACUs based on the dynamic temperature patterns in the server rooms. This effectively decreases the oscillation in the conventional control approach, which often results in frequent overshoots during temperature control and causes higher ACU energy consumption. This partly explains why our method can have lower energy consumption but achieve the same or better cooling effect.
D IMPLEMENTATION DETAILS
D.1 PRACTICAL IMPLEMENTATIONS OF OUR PROPOSED METHOD
Data preprocessing. We preprocessed the DC raw data facilitate model training. Min-max normalization was applied to both states and actions using the following formulas: [SSma=
(s-Smin)
and ä [Amax- Amin) Smaz: Smin, Amazy Amin are maximum and minimum normalization bound- aries for state and action features. For actions, set Amin 0 and Amaz 100 both fan speed and valve openings are percentage values. For the states, as described in Section 3.1, there exist different types of sensor inputs: (ss: 8a, 3.), and each type of sensor reading has distinct scales.
(m-Am)n)
18

---

Published as a conference paper at ICLR 2025
Therefore, set different normalization scales for different types of sensors by consulting the do- main experts. Specifically, we denote all temperature-redee related sensor readings as Stemp (c.g. L))(s (s,(s,)). Lw)(s,,, and EWT(s,), humidity sensor readings Shumi- and power consumption of servers as ppower. Their corresponding normalization boundaries are presented in Table 3.
Table 3: Normalization boundaries for different state components.
Stemp
Shumi
8 Spowe
S Swin
1 40
0
S Smaz
100
150
Model architecture and hyperparameters. The architecture and algorithm hyperparameters in our proposed physics-informed offline RL framework are listed Table As discussed Section 3.3, the only hyperparameter that we tuned during our experiments is in the normalization term X (see Eq. (9)). We tuned a values in the range [2.5, 10] and deployed the best-performing model for long-term control in both the production DC and our testbed environments. This hyperparameter modulates the conservatism of the learned policy. We observe that reasonably increasing a can enhance the energy-saving performance certain degree.
Real-time data preprocessing and policy smoothing. In our deployed systems, we preprocess the real-time sensor data by filtering out problematic data samples and resample them into uniform time intervals (5 minutes for the large-scale commercial data center and 2 minutes for the small-scale DC testbed). To enhance the smoothness and robustness of the closed-loop ACU control commands generated by the policy, we apply temporal smoothing to the seeicy-geeerated actions in our practical implementation. Specifically, the final execution action the current time step is calculated as the average of policy output actions at the current time step and the previous 4 time steps, which provides a smoother control signal for ACUs.
Algorithm pseudocode. The pseudocode of our proposed physics- offline RL framework can be found in Algorithm
Algorithm 1
Require: Preprocessed historical dataset D, initialized value network Q, policy network æ, and the T-symmetry enforced thermal dynamics model (TTDM), which contains the state-action encoder ő(s, a), latent forward dynamics model f and latent reverse dynamics model g, state and action decoders (((2) and (=a)).
// Learning TTDM frDM offline dataset for 1 = 1,. ... ,T, training steps do
Sample mini-batch B of samples {(s, a, s(s. D and process through the state-action encoder ((s, a) to get the latent representations {[ Za, Zg', Ze. ).
Compute the forward and reverse dynamic losses based on Eq. (3) and Eq. (4)
Compute T-symmetry regularization loss over the two latent dynamics models based on Eq. (6)
Compute the reconstruction losses in Eq. (2) and Eq. (5)
Update TTDM network parameters by minimizing the overall learning objective in Eq. (7)
end for
/ Sample efficient offline policy optimization for t- 1,... .T training steps do
Sample mini-batch B of samples (s,,a,r,'')) D. where is calculated based Eq. (1)
Update the value network Q with the learned Φ(8, a) based on the objective in Eq. (8).
Update the policy based on the policy learning objective in Eq. (9).
end for
D.2 BASELINE ALGORITHMS
In our testbed experiments, compare method with the ACU's default PID controller, data- driven MPC method for DC cooling control developed by Google (Lazic et al., 2018), an off-policy RL--based DC cooling optimization method CCA (Li et al., 2019), mainstream offline RL methods
19

---

Published as a conference paper at ICLR 2025
Table 4: Hyperparameter details.
Hyperparameters
Value
Optimizer type
Adam
Learning rate
3e-4
Weight decay
le-5
Channel number
6
Common feature per node
4
GNN hidden layers
2
TTDM Architecture
GNN hidden units
256
Forward / reverse model hidden layers
2
Forward / reverse model hidden units
128
Fusion layers
2
Fusion layer units
128
Weight of fym and Eyec Weight lrus and ffwd
1
0.1
a
Tuned the range of [2.5.10]
Discount factor
0.99
Target update rate
0.005
Policy noise
0.2
Critic neural network layer width Actor neural network layer width
512
Offline RL Algorithm
512
Actor learning rate
3e-4
Optimizer type
Adam
Critic learning
3e-4
Policy noise clipping
0.5
Policy update frequency
2
Number of iterations
5e5
such as Implicit Q-Learning (IQL) (Kostrikov et al., 2022) and Conservative Q-Learning (CQL) (Kumar et al., 2020), and the state-of-the-art (SOTA) safe offline RL algorithm, FISOR (Zheng et al., 2024). We provide detailed descriptions of these baseline methods as follows.
Default PID controller. The ACU in experiments adopts a conventional PID controller (Ang et al., 2005) to adjust its fan speed and compressor to minimize the error between the target CAT setpoint and the system's 's actual CAT value. The controller consists of three components: the Proportional term, which responds to the current error; the Integral term, which accumulates past errors to correct steady-state offsets; and the Derivative term, which predicts future errors based the rate of change.
Data-driven MPC controller (Lazic al., 2018). This DC cooling control method is developed by Google, which learns a linear dynamics model of the floor-level cooling system for future state prediction, and optimizes the control action over a finite time horizon using MPC. At each time step, MPC solves a constrained optimization problem to minimize cost function while considering system constraints.
Cooling Control Algorithm (CCA) (Li et al., 2019). CCA is an actor-critic RL framework for DC cooling control. based on the classic off-policy RL algorithm deep deterministic policy gradient (DDPG) (Lillicrap et al., 2015). As DDPG is an online RI. method, CCA needs online interactions with a simulation environment to collect and store data in replay buffer, and sample training batches from the replay buffer for policy learning. In our offline learning setting, there is no reliable simulation environment available, we replace CCA's replay buffer to the offline dataset in implementation.
Implicit Q Learning (IQL) (Kostrikov et al., 2022). IQL is popular offline RL algorithm that uses expectile regression learn value functions from fixed datasets without explicit policy con- straints. avoids evaluating the potential OOD actions from the learned policy, therefore alleviating distributional shift, and typically enjoys stable offline policy learning.
20

---

Published as a conference paper at ICLR 2025
Conservative Q learning (CQL) (Kumar et al., 2020). CQL is another popular offiin RL algorithm that learns conservative estimates of Q-values on OOD actions to enforce offline behavioral data regularization and mitigate distribution shifts.
Feasibility- Safe Offline RL (FISOR) (Zheng et al., 2024). FISOR is the SOTA safety-centric offiine RL algorithm which enforces hard constraints by identifying the largest feasible region from (HJ) reachability analysis. It adopts a decoupled learning scheme that optimizes a diffusion model-based safe policy by maximizing reward within the feasible regions while minimizing safety violations within infeasible regions, thereby enjoying a strong safety performance and superior learning stability.
the offline dataset based on Hamilton-Jache
E REAL-W WORLD DATA ANALYSIS
In the real-world data center, due the use of PID group control for ACUs throughout the historical operation, and infrequent adjustments to the PID-related temperature setpoints, the action patterns of the ACUs system (fan speed and water valve opening) are narrowly distributed. Additionally, the distributions of other state features are mostly concentrated with single peak. All these factors pose significant challenges offline RL policy learning, requiring models with strong generalization capability to effectively learn and optimize control strategies. Figure 13 shows the historical dataset distributions collected from our real-world testbed, in which we collect system operational data from more diverse server load and control settings, resulting in relatively broader state-action space coverage. This actually makes the task more manageable for existing offline RL algorithms like CQL, IQL, and FISOR. However, baseline methods in the testhed experiments, and more importantly, achieves good performance in the much more challenging production DC environment.
have shown in Figure 6, our proposed method still outperforms the
F LIMITATIONS AND FUTURE WORKS
In this study, we only tested in single large-scale commercial DC facility and small-scale real- world testbed. For future works, we plan to further expand our experiments to multiple DC facilities with different air-side cooling system configurations. Also, our approach models the safety constraints by incorporating them as penalty terms inside the RL reward function, which adds complexity to reward design and may not be sufficient ensure safety under certain special conditions. Future investigations can be conducted to expand our method to safe offline RI. framework, with dedicated consideration of constraint satisfaction, which would provide more safety guarantees practice. Furthermore, it is also meaningful to explore the joint optimization of both cooling and server-side systems, which can fully maximize the potential for DC energy saving.
G LEARNING CURVES
Figure 14 reports the learning curves of the proposed TTDM and offline policy learning method. As not possible to directly interact with the real DC environment and evaluate the policy's performance during offline RL training, hence we report the Q-function learning loss and policy loss for different training steps. Both our proposed TTDM and the RL policy learning scheme enjoy stable model convergence during training.
21

---

Published as a conference paper at ICLR 2025
E379
0:50
E39
0:20
648
C34
0.10
604
0.5
0.00
0.30
099
0.00
0.000
EATO1 (9t)
EAT02
LATOI PC)
LAT02 (C)
LATOS
6:44
0.25
0.55
0:2
0.00
0.15
0.00
0.12
3023
Fru-
Fatespard 151
Frecopar ultaooppa (%)
(1C)
Settwor0304
0.16
5:3
0.7
0..0
000
E20
0.2
000
035
639
8:5
oxe
0.00
0.0
0.00
3.00
Ser Sersar0566 (NC)
7emorkC78 (C))
Sensor 0910 (C)
Sersor 010.011112222 ('C)
Senser 1314
0.15
0.39
612
0.55
0:5
3.05
0.09
Humidity
0.5
0.055
022
1.30
0.0000
0.04
038
E330
0301
02
0.8
0303
0.005
000
2330
6089
0.00
ong
Se12or2324.9 (C)
Rack powero1 (40
Rack poweroz
Rack www.r//ww03.mmm (W)
EAT setpoins (C)
Figure 13: Distributions of the state and action features in our historical dataset collected from the real-world DC testbed.
22

---

Published as a conference paper at ICLR 2025
CTTDM
Lyec
0.26
0.8
0.20
LO85
0.6
0.16
0.06
0.0
0.00
Training steps (xle3)
Training steps (xle3)
Ered
Erys
0.10
0.12
0.08
0.10
0.08
0.00
0.06
10
Training steps (xle3)
Training steps (xle3)
Eds
fT
0.35
0.030
0.25
0080
Lo59
0.15
0.015
0.010
0.05
0.005
0.000
10
Training steps (x1e3)
Training steps (x1e3)
Q-function loss
Policy loss
15.0
50
mm
12.5
40
w
10.0
Loss
90
7.5
5.0
20
2,5
0.0
Training steps (xle5)
Training steps (xle5)
Figure 14: a. Learning curves the overall loss function and each individual loss term TTDM. b. Learning curves for the offline RL policy learning.
23


Reviews:
[{"review_id": "NMkxhKshNW", "reviewer": "Reviewer_TQKL", "summary": "The authors present a physics-informed offline reinforcement learning framework for optimizing energy efficiency in data center cooling systems, addressing critical challenges like limited data and safety constraints. Using a graph neural network model that respects time-reversal symmetry, the framework enables efficient and robust policy learning from real-world operational data. The authors claimed that this method was successfully deployed in a large-scale commercial data center and achieved 14-18% energy savings over 1300 hours without violating safety constraints. The work demonstrates the potential of offline RL for complex, data-limited industrial applications and calls for a shift from simulation-based benchmarks to real-world problems for more practical and impactful RL research.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed solution integrates a physics-informed dynamics model to accurately capture the complex thermal behavior within the server room, paired with a graph neural network that embeds domain knowledge to reduce data requirements. \n2. The authors claim that this approach produces well-structured and generalizable latent representations, facilitating a sample-efficient offline RL algorithm that maximizes the value function in latent space with appropriate regularization.\n3. The implementation includes a safety-aware reward function to ensure operational reliability.\n4. The premise of this paper is that as offline RL enables efficient policy learning from pre-collected data, eliminating the risks and costs associated with continuous interaction in safety-critical or resource-constrained environments, it is more effective and practical than online RL.", "weaknesses": "1. Claimed but Not Established: The paper asserts strong out-of-distribution (OOD) generalization capabilities and effectiveness with limited real-world data, but these claims are insufficiently substantiated.\n2. Lack of Industry Baseline: The real-world validation experiments show 14-18% energy savings in DC cooling without safety violations, but the paper is unable to present any well-defined industry practice baseline with similar objectives for comparison under similar constraints. The comparison lacks fairness. Also, a thorough optimization metric for a data center operation should factor in elements other than safety violations. \n3. Model Generalizability & Insufficient Benchmark Comparison: The method heavily relies on modeling, but the generalizability and robustness of the modeling technique remain unverified. The method's performance is not evaluated against established benchmarks, limiting the validation of its general effectiveness.\n4. Data and Experiment Limitations: The method's performance evaluation is constrained by the definitions of the experimental setup and the data distributions used in this study, which is not standardized. \n5. Minimal Algorithmic Novelty: The approach offers little innovation compared to existing methods, limiting its algorithmic contribution.\n6. Comparison with other Physics-informed modeling: The paper does not convincingly demonstrate how the proposed approach is superior to well-established physics simulation models bootstrapped with collected data that use online RL to train. The claimed higher sample efficiency with the physics-informed model is not unique, as similar benefits are observed with both online and offline RL approaches.\n7. Unclear Baseline Performance: There is an inadequate explanation for why aggressive baseline methods like CCA and CQL achieve lower energy consumption but fail to maintain critical thermal safety.\n\nIn summary, this paper would be better suited for a domain conference centered around physics modeling and the application of standard AI techniques for optimization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J1jI0qRgSn", "reviewer": "Reviewer_JenY", "summary": "This paper proposes physics informed offline RL framework to control datacenter cooling systems. The proposed framework constructs a dynamics model based on T-symmetry and Graph Neural Networks to embed domain knowledge, TD3-BC to perform the policy optimization.  The framework is deployed on a real system and the authors develop a test bed to compare with existing approaches.", "questions": "1. Apart from the reward function, what are the other measures taken to prevent safety violations? \n2. Have the authors tried Offline-Online methods? wherein the policy trained using offline RL deployed is constantly improved using new data obtained? \n3. It seems like CQL does perform better than the proposed method in Figure 6, but at the cost of safety. What was the reward function used for CQL?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors tackle a problem of significant relevance, and develop a framework that can be implemented in the real world.\n2. The paper is generally well written, and the authors do a good job of testing the proposed framework under different conditions.", "weaknesses": "1. The algorithmic contribution is minimal. While effective, the proposed method is a combination of existing methods tailored to a specific use case.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GcrWteuI8C", "reviewer": "Reviewer_4H4V", "summary": "This paper presents a novel physics-informed offline reinforcement learning framework for optimizing energy efficiency in data center cooling systems. The core component is a T-symmetry enforced thermal dynamics model using graph neural networks to capture complex thermal patterns. This enables sample-efficient offline policy learning from limited historical data. The framework was successfully deployed in a large-scale commercial data center, controlling multiple air cooling units and achieving 14-18% energy savings without safety violations over 1300 hours of experiments. Comprehensive evaluations on a real-world testbed further demonstrated the method's effectiveness compared to baselines. The approach shows significant potential for solving data-limited, safety-critical industrial control problems beyond just data center cooling.", "questions": "1. Why does the reward need to be positive? \n2. How did you tune the hyper-parameters?\n3. T-symmetry is supposed to help with OOD generalization. Have you measured OOD in your dataset? \n4. The algorithm is based on TD3+BC. Why is TD3+BC not one of the baselines in your experiments? \n5. I'm surprised outside weather conditions is not considered in the modeling. Does the data center have perfect insulation? I would assume hotter conditions would lead to higher cooling demand. \n6. The ratio of ACU average electric power and average energy consumption seem to vary with each entry. Why would that be?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The methods proposed have been deployed in a real-world data center. This is a big, big accomplishment \n- The proposed methods go beyond simply applying an existing algorithm. They have applied state-of-the-art offline RL methods to the data center cooling problem. The idea to use both a GNN and a physics informed loss function makes sense. \n- The T-symmetry is an interesting idea that has been adopted from another paper, and they show it works well in their setting. \n- They have performed controlled experiments in a large scale data center as well as a small one where they have more control over the settings. \n- Their evaluation includes comparison against the state-of-the-art as well as ablation of their algorithm", "weaknesses": "- The key metric used in evaluation is ACLF -- Air-side cooling load factor. However, this metric is neither defined nor explained in detail. It appears to increase with server load based on the sentence -- \"Due to the smaller scale of the testbed and significantly lower server\nload as compared to the real-world DC, the calculated ACLF values are higher than those observed in the real DC experiments.\" \n- The Air Cooling Units (ACU) are being optimized in isolation, and it is unclear what is the impact on the upstream cooling system demand. Specifically, it is clear that the algorithm reduces the CAT (cold aisle temperature?) and therefore increases the demand on the chiller and cooling towers. Is the algorithm simply increasing the energy demand in the upstream system while reducing it in the ACUs? \n- There are many claims throughout the paper which are not substantiated. For example: it states: \"building a high-fidelity simulator can be very costly and impractical.\" How expensive is it, and why is it impractical? Another example: \"the fan power consumption is proportional to the cube of the fan speed\". Please cite the source of this information. \n- There are two externalities to the system -- entering water temperature and server load. While server load has been accounted for in the experiments, it is unclear if entering water temperature is constant throughout the experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qNZghvM6zA", "reviewer": "Reviewer_FhGh", "summary": "The paper introduces a physics-informed offline reinforcement learning framework for optimizing data center (DC) cooling systems, addressing their substantial energy demands. Utilizing a graph neural network (GNN) architecture with time-reversal symmetry (T-symmetry), the model effectively captures complex thermal dynamics and ensures robust policy learning from limited historical data. Deployed in a real-world DC, the framework demonstrated 14-18% energy savings over 1300 hours without safety violations. Validation in a testbed environment further confirmed its superior performance over conventional and baseline RL methods. This approach highlights offline RL's potential for data-limited, safety-critical industrial control applications beyond DCs.", "questions": "- Could the authors elaborate on how their model adapts to changing DC conditions over time and how often retraining is needed to maintain optimal performance?\n- What are the potential limitations or challenges when scaling the proposed approach to larger data centers or DCs with more complex cooling system architectures?\n- Can the authors provide more details on the computational cost of deploying the GNN and T-symmetry framework, particularly in comparison to simpler baseline models like PID controllers?\n- Was the impact of T-symmetry enforcement on training time and model convergence evaluated? It would be helpful to understand if this enhancement has significant trade-offs in terms of training efficiency.\n- Given that CLUE [1] achieved effective HVAC control with only seven days of training data, have the authors considered comparing their method's data efficiency to similar approaches? How would their framework perform with similarly limited training data?\n1. An, Zhiyu, Xianzhong Ding, Arya Rathee, and Wan Du. \"Clue: Safe model-based rl hvac control using epistemic uncertainty estimation.\" In Proceedings of the 10th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation, pp. 149-158. 2023.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper integrates graph neural networks (GNNs) with time-reversal symmetry (T-symmetry) to enhance offline reinforcement learning (RL), showcasing an innovative approach to optimizing DC cooling systems.\n- Comprehensive real-world validation includes 1300 hours of deployment in a production DC, proving the method’s effectiveness and robustness.\n- The methodology is clearly detailed with supporting figures and tables, making the framework’s architecture and results easy to follow.\n- Demonstrated 14-18% energy savings highlights significant practical impact, with potential for broader application to other data-limited, safety-critical industrial control scenarios.", "weaknesses": "- The paper relies solely on simulation-based testing in the small-scale DC testbed environment, limiting the generalizability of the results to more diverse real-world DCs. While the full-scale production test demonstrates feasibility, more varied test environments could strengthen the claims.\n- The deployment details and considerations for long-term adaptability, such as how frequently model retraining is needed or how it adapts to evolving DC conditions, are underexplored.\n- The scalability of the approach for larger DCs or environments with higher variability is not deeply discussed. Addressing the implications for larger-scale or more complex DC configurations would improve the scope of the work.\n- While T-symmetry and GNN integration are well-motivated, there is limited discussion on potential trade-offs, such as computational complexity or latency during model training and execution.\n- The paper could benefit from including a comparison with relevant existing methods like CLUE[1], which demonstrated data-efficient HVAC control with only seven days of training data and reduced comfort violations. Such a comparison would provide valuable context regarding the data efficiency and performance trade-offs of the proposed framework, particularly under conditions of limited training data.\n1. An, Zhiyu, Xianzhong Ding, Arya Rathee, and Wan Du. \"Clue: Safe model-based rl hvac control using epistemic uncertainty estimation.\" In Proceedings of the 10th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation, pp. 149-158. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)