# Optimizing Container Placement in Data Centers by Deep Reinforcement Learning
## 2 Related Work
### Deep Reinforcement Learning (DRL) Methods  
DRL is a powerful AI technique that combines reinforcement learning and deep learning. It trains agents to make optimal decisions by interacting with an environmentto maximize rewards using deep neural networks to approximate complex functions. It is used to optimally allocate items or cloud resources in various studies.  
DRL algorithm for general bin-packing problems was proposed by [PERSON] et al. This self-adjusting training mechanism ensures that the agent consistently learns at an appropriate level, ultimately leading to an optimized placement strategy. Unlike previous heuristic methods, which require manual adaptation to specific problems, this approach leverages reinforcement learning to generalize across different optimization scenarios. The proposed algorithm has been shown to outperform Monte Carlo tree searches, heuristic algorithms, and integer-programming solvers [13].  
Another DRL algorithm for container placement was proposed by [PERSON] et al., who introduced an improved Q-Learning strategy that incorporates eligibility traces and dynamic \(\epsilon\)-greedy exploration. Eligibility traces accelerate learning by updating Q-values for both the current state-action pair and related past pairs. The dynamic \(\epsilon\)-greedy exploration strategy begins with a high exploration rate, gradually shifting toward exploitation while retaining some exploration to mitigate the risk of converging to local optima. The proposed method outperforms baseline algorithms, including First-Fit Host Selection, Least-Full Host Selection, and traditional Q-Learning. The results demonstrated that the resulting container allocation consumed less energy and featured a reduced service level agreement violation rate, confirming the effectiveness of the algorithm [14].  
A novel approach named CARL, which means Cost-optimized placement using Adversarial Reinforcement Learning, was proposed by [PERSON] et al. CARL employs Adversarial Reinforcement Learning (ARL) to solve the online container placement problem. ARL learns policies through the interaction between reinforcement learning agents and adversarial agents and thus presents placement strategies for various scenarios. In evaluation using workloads traces from Google Cloud and Alibaba Cloud, it demonstrated high throughput, cost reduction, and adaptability. It processed approximately 1900 container placement requests per second across 8900 candidate VMs and achieved up to 16% lower VM costs compared to existing heuristic methods. It also maintained performance despite changes in the workload [15].  
Although the heuristic algorithms above are simpler to implement and have faster convergence, they are prone to fall into local optimal solutions and are difficult to approach the global optimum. The meta-heuristic algorithms are highly flexible and easy to obtain the optimal solution, but it has more parameters, which are more difficult to adjust and may not perform well in complex problems. And although the deep learning-based allocation strategies are more suitable for dealing with complex problems, they require a large amount of data for training and are difficult to adapt to the real cluster container allocation environment [14].