# Optimizing Container Placement in Data Centers by Deep Reinforcement Learning
## 4 Container Placement Optimization
### Reward Design  
A reward system plays a crucial role in training a DRL agent. For reinforcement learning agents, frequent rewards are needed. However, in most cases, the environment provides only sparse rewards [19]. The primary reward is to maximize the number of idle servers, which aligns with the goal of the objective function. Therefore, rewards should be assigned according to the number of servers left unused at the end of the process. However, providing rewards only at the final stage is insufficient for effective learning. Instead, intermediate rewards must be incorporated to facilitate continuous learning and optimization [20]. To achieve this, the minimum number of PSs required for container placement is first determined, and rewards are assigned to encourage placement on the fewest possible PSs. If multiple types of PSs are available, they must also be considered. To maximize the number of unused servers, containers should be placed starting from larger PSs, ensuring that more servers remain unused later. Therefore, rewards are assigned according to physical server size to promote this placement strategy. Since container placement aims to fit the maximum number of containers onto the fewest PSs, stacking containers efficiently is encouraged by awarding positive rewards when successful stacking occurs. However, if placement exceeds resource limits, a negative reward is given to prevent further stacking beyond capacity. These intermediate rewards must be carefully balanced to not interfere with the ultimate objective--maximizing the number of unused PSs. If intermediate rewards outweigh final rewards, the agent may optimize for them instead, preventing the objective function from being maximized. Therefore, a clear difference must exist between intermediate and final reward values. Based on these principles, the reward system was designed as follows. First, the ideal minimum number of PSs required to accommodate a given container set was calculated. This was determined by calculating the total CPU cores and memory size of all containers and identifying the minimum number of PSs needed. If a container is successfully deployed on a physical server within this ideal minimum range, a reward of 0.2 points is granted. \[r_{t}=0.2,\text{within }N_{ideal},\]  
\[N_{ideal}=\max[\text{t}1,\text{t}2],\]  
\[\text{t}1=\min[\frac{\sum_{j=1}^{N_{ct}}CPU_{ct_{j}}}{\sum_{i=1}^{t1}CPU_{ps_ {i}}}<1,(1\leq\text{t}1\leq N_{ps})],\]  
\[\text{t}2=\min[\frac{\sum_{j=1}^{N_{ct}}MEM_{ct_{j}}}{\sum_{i=1}^{t2}MEM_{ps _{i}}},(1\leq\text{t}2\leq N_{ps})].\]  
In other cases, the reward value is adjusted based on the type of physical server used. A higher reward is assigned for larger servers, while smaller rewards are given for smaller servers, with reward values ranging between 0 and \(-0.05\). \[r_{t}=\begin{cases} \begin{array}{cc}0,&\text{$TPS_{3}$}\\ -0.01,&\text{$TPS_{2}$}\\ -0.03,&\text{$TPS_{1}$}\\ -0.05,&\text{$TPS_{0}$}\\ \end{array} \\ \end{array}\]  
Because maximizing container stacking on a single physical server is the key objective, exponential rewards are granted when containers are placed on an already occupied server. in descending order from largest to smallest. This ordering tends to improve placement efficiency. For comparison, we selected four bin-packing algorithms: next fit (NF), first fit (FF), declining next fit (DNF), and declining first fit (DFF). Figure 3 presents the learning curve for the DRL agent, showing the improvements in the maximum number of unused PSs achieved as training progresses over episodes for the Data7 and Data8 cases. Additionally, the final performance levels achieved by the DFF and DNF algorithms are also displayed as horizontal lines. The results indicate that the DRL agent typically reaches peak performance after 200 episodes.