## 4 A Case Study: DQN-Based Intelligent Thermal Management System
### Performance of DQN-Based Intelligent Decision Making  
In order to verify the ability of the DQN finding optimal cooling policy, a constant IT consumption circumstance is firstly set, where IT servers of constant heating power are given as input. Reward of DQN should finally converge to a particular optimal value. Since the state is steady and the IT consumption is a fixed value for each equipment, there is only one optimized steady state for each temperature set point. As demonstrated in Figure 9, the green line represents the PUE reward, the orange line represents the hotspot reward, the yellow line represents the temperature reward and the red line represents the total reward plus a threshold value. Due to that locations of temperature measuring points are random, which values are usually close to the ideal temperature, \(R_{temp}\) converges to zero during most of the training process. Figure 8: The scope of the IDC digital twin system. Figure 9: Reward curves of DQN decisions on constant server consumption. During the exploration phase (steps 200-350), periodic oscillations in the PUE reward and intermittent hotspot penalties indicate active policy iteration. After step 380, both rewards stabilize: the PUE reward converges to a constant value while the hotspot reward remains zero, demonstrating policy convergence to a steady-state configuration. Convergence within 380 steps highlights the algorithm's efficiency under simplified operational constraints. To further analyze the robustness of the decision-making performance, the DQN model is tested under a more complex situation where the servers' heating powers vary as a sinusoidal pattern. \[P_{i}=12\times\left(1+\sin\!\left(time\times\frac{2\pi}{step}\right)\right) \tag{11}\]  
The training results show that our DQN model generates optimal cooling policies with the highest reward. As demonstrated in Figure 10a, when the IT servers' consumption reduces, the DQN decided on a higher cooling system temperature in order to save energy. And when the IT consumption rises, the DQN agent preferred to decide on a relatively lower temperature to prevent the hotspot zone formation. The changing relationship of the reward curve and IT consumption curve has a strong correlation after being well trained, as illustrated in Figure 10b. It proves that our DQN model has strong adaptability and high robustness of decision-making performance when facing changing heating power of servers. However, the workload distribution in reality is actually to be random, and this fact brings difficulties to deal with the cooling optimization problem. Realistic workload variability introduces state-space complexity through combinatorial explosion. Six IT servers have four discrete power levels \(P=\{0,3,7,10\}\); the state space cardinality grows as \(4^{6}=4096\). To maintain tractability, experiments restrict IT servers from the physical testbed to empirically operate in the power range of \(P=\{0,3,7,10\}\). The detailed configuration information of the DQN architecture is provided in Table 2. \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameters** & **Value** \\ \hline Learning rate & \(lr=0.0001\) \\ Exponential decay & \(\lambda=0.95\) \\ Discount factor & \(\gamma=0.5\) \\ Exploration parameters & \(\epsilon_{0}=0.6\), \(\epsilon_{\min}=0.2\)\(\epsilon_{\text{decay}}=0.99\) \\ Reward weights & \(R_{temp}=0.2\), \(R_{PUE}=2.5\), \(R_{hotspot}=0.3\) \\ Thermal threshold & \(T_{drug}=50\)\({}^{\circ}\)C, \(T_{ideal}=25\)\({}^{\circ}\)C \\ Experience replay capacity & \(C=104\) \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Hyperparameter setting of DQN architecture for discrete heating power. Figure 10: **(a)** The optimal actions of DQN corresponding to IT consumption. **(b)** Reward profiles of DQN decision, constant AC temperature 16, 18, 20 and 22 \({}^{\circ}\)C after convergence. This configuration balances exploration-exploitation dynamics while mitigating training instability through learning rate annealing.