# Optimizing Container Placement in Data Centers by Deep Reinforcement Learning
## 4 Container Placement Optimization  
### Container Placement Optimization Using DRL  
The symbols used in this algorithm are described in Table 1.  
Container placement optimization focuses on placing containers on the minimum number of PSs to maximize the number of unused servers. The objective function is defined as follows: max(\(N_{PS_{unsued}}\)), \(0\leq N_{PS_{unsued}}\leq N_{ps}\). To achieve this optimal arrangement, the actor-critic method is employed. Actor-critic is a prominent reinforcement-learning algorithm where an agent learns to select optimal actions within a given environment. This powerful method can solve complex problems across various environments and enables learning at each step. This agent consists of two key components: an actor, which learns policies, and a critic, which learns a value function. The actor is responsible for learning the policy function that determines the agent's actions. The policy function generates a probability distribution over possible actions in a given state and selects an action accordingly. The critic learns the value function, which estimates the expected value or average reward for a given state. It evaluates the action chosen by the agent and updates its value accordingly, ensuring improved decision-making over time [17,18]. \begin{table}
\begin{tabular}{c l l l} \hline \hline
**Symbol** & **Description** & **Symbol** & **Description** \\ \hline \(N_{ps}\) & Number of PSs & \(N_{batch}\) & Number of a batch \\ \(N_{ps_{unsued}}\) & Number of unused physical servers & \(B\) & Buffer to store a batch \\ \(p_{s_{i}}\) & Physical server \(i\) & \(CPU_{ps_{i}}\) & CPU size of physical server \(i\) \\ \(N_{ct}\) & Number of containers & \(CPU_{ctj}\) & CPU size of container \(j\) \\ \(ct_{j}\) & Container \(j\) & \(MEM_{ctj}\) & Memory size of container \(j\) \\ \(ID_{ctj}\) & Identification of container \(j\) & & Number of stacked containers on \\ \(s_{t}\) & State s of time t & \(NSC_{ps_{i}}\) & physical server \(i\) \\ \(a_{t}\) & Action a of time t & & \\ \(r_{t}\) & Reward r of time t & \(CPU_{SC_{ij}}\) & CPU size of stacked container \(j\) on \\ _Done_ & Whether to terminate & & physical server \(i\) \\ \(\gamma\) & Discount rate \(\gamma\in[0,1]\) & \(MEM_{SC_{ij}}\) & Memory size of stacked container \(j\) \\ \(\pi(s,a)\) & Probability of actions & & on physical server \(i\) \\ \(V(s_{t})\) & Value of state \(s_{t}\) & \(TPS_{k}\) & Type of PSs \(k\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Symbols and descriptions. Figure 2: **(a)** A traditional bin packing example with seven items of different sizes, **(b)** a container placement example with different sizes: red (8 Core, 2 GB), magenta (4 Core, 8 GB) and blue (2 Core, 2 GB). The state at time \(t\), denoted as \(s_{t}\), represents information about the container currently being considered for placement. It is defined as a vector comprising four parameters: total number of containers (\(N_{ct}\)), identifier of container j to be placed (\(ID_{ct_{j}}\)), its CPU size, and its memory size of container j:  
\[s_{t}=[N_{ct},\ ID_{ct_{j}},\ CPU\ \text{size of}\ ct_{i},\ \text{Memory size of}\ ct_{i}].\]  
This state vector value is fed to a neural network with a linear network with fixed-size input values. The rectified linear unit (ReLU) activation function is employed, and the Adam optimizer is used, with a learning rate of 0.01. The output of the actor network passes through a Softmax function to produce a probability distribution over the available PSs. The critic network outputs an estimate of the state value. Algorithm 1 illustrates the proposed algorithm, which is implemented as follows: First, the state value is fed into the actor network to generate the probability distribution (\(\pi(s_{t},a_{t})\)) corresponding to the number of available PSs. An action (\(a_{t}\)) is probabilistically selected from this distribution (line 7 in Algorithm 1).