## 4 A Case Study: DQN-Based Intelligent Thermal Management System
### Performance of DQN-Based Intelligent Decision Making
The training result presents convergence characteristics for the setting workload regime, demonstrating policy robustness under constrained combinatorial complexity. As demonstrated in Figure 11, the training loss converges at around 11,000 steps, having a convergent value of 0.0873. Figure 12 systematically compares the DQN agent's reward performance against three fixed-temperature baselines (15 \({}^{\circ}\)C, 20 \({}^{\circ}\)C, 24 \({}^{\circ}\)C) across four critical training stages. The first stage is the early training period, which is 0-2000 steps. Figure 12a represents the initial exploration phase where the untrained agent exhibits highly stochastic behavior. It indicates that total rewards marginally exceed those of a fixed 24 \({}^{\circ}\)C policy but underperform compared to a constant 20 \({}^{\circ}\)C baseline. While hotspot penalties remain absent due to stringent reward weighting (evidenced by suppressed green traces), the PUE reward stagnates near median values, indicating negligible energy savings. The mid-training period (2000-6000 steps) and advanced training period (6000-11,000 steps) are shown in Figure 12b,c. It reveals incremental total reward improvements, though PUE efficiency remains comparable to untrained levels. The critical transition occurs in the converged training period (11,000-16,000 steps), as illustrated in Figure 12d, where the trained agent achieves PUE parity with the 24 \({}^{\circ}\)C policy while simultaneously suppressing thermal anomalies, evidenced by total reward dominance over both fixed-temperature benchmarks. The DQN hotspot reward arises from the 24 \({}^{\circ}\)C baseline's excessive hotspot generation (right panels), penalized through the multi-objective reward structure. In the final period, the DQN achieves optimal trade-off and robust convergence, balancing all rewards and outperforming all baselines in total reward. In addition, we can conclude from Figure 13 that the DQN model generates more stable and robust cooling actions after being well trained (Figure 13b) compared with the early untrained agent (Figure 13a). The DQN successfully learns to elevate setpoints facing different workloads of the IT servers' power consumption, achieving optimal balance between PUE and hot-area reduction. This strategic setpoint elevation directly reduces cooling energy expenditure, validating the DQN's capacity to balance PUE minimization against thermal risk mitigation. The results quantitatively demonstrate that reward-weighted reinforcement learning successfully reconciles competing operational objectives inherent in data center thermal management. Figure 11: The 100-steps moving average loss curve of the DQN training process. Figure 12: Reward profiles of DQN decision and constant AC temperature 15, 20 and 24 at different time steps. As shown in Table 3, the total reward of DQN performs an average of \(-\)5.96, while 15 \({}^{\circ}\)C, 20 \({}^{\circ}\)C and 24 \({}^{\circ}\)C separately perform averages of \(-\)8.05, \(-\)8.20 and \(-\)9.64. The DQN decision-making agent performs at least a 26% increase in total reward. As for the PUE reward, which represents the energy-saving performance, the DQN agent averages at \(-\)4.52, while the three compare conditions each average at \(-\)7.53, \(-\)7.05 and \(-\)3.90. DQN performs a 40% and 36% improvement in energy saving compared with the constant 15 \({}^{\circ}\)C and 20 \({}^{\circ}\)C.  
Since real-time control is required for thermal management in our prototype, the DQN model needs to generate policy in second-level when applied in testbed. Though the training of DQN costs much time, the process of making optimal policy with a well-trained model can be conducted within 1.5 s. The detailed time consumption is summarized in Table 4.