# Adaptive BBU Migration Based on Deep Q-Learning for Cloud Radio Access Network
## 1 Introduction
On the one hand, overutilization of real servers hosting virtualized BBUs can result in a loss in service quality. On the other hand, underutilization of assets keeps power consumption under control but wastes physical assets. With virtualization and cloud computing, the Cloud-RAN promotes base station agility and scalability [22]. The use of resources dynamically relates to the current demand for the use of resources, with shared use of physical resources. With such an installation, the use of processing in terms of virtualization enables a single installation of BBUs in a server cluster, with a pool of BBUs supporting the virtual use of resources in a cluster is optimized, overall dependability of operations in a pool of BBUs is maximized, and use of power is reduced. In this work, an adaptive dynamic migration scheme with a dual-threshold model is proposed, specifying both upper and lower bounds for utilizing resources in physical nodes in a BBU pool. In the following, key contributions of this work are discussed:1. Create a BBU pool and virtualize it using the iCanCloud platform, a tool designed for modeling cloud environments. 2. Implement a two-threshold migration mechanism dynamically. Start with an adaptive migration trigger scheme with a deep reinforcement learning (DQN) algorithm that can detect when migration will best occur, skipping unnecessary migration in case of transient spikes in server resource consumption, high or low. Next, make a smart decision about selecting a virtual machine for migration through consideration of both migration count and duration. 3. The iCanCloud platform runs a simulation in an attempt to assess the proposed migration scheme, and it can enable efficiency in terms of energy and considerable migration count and migration time savings. Unlike a conventional manual scheme, such a scheme utilizes artificial intelligence for accuracy, evasion of invalid migration, and achievement of load balancing and adaptive dynamic migration. One of the traditional RL algorithms is Q-learning, which is a model-free learning technique [23]. In Q-learning, the environmental context is covered at the start. The action taken for each stage in the section should be determined by the policy and the present circumstances. The following state, s\({}_{t+1}\), and the associated reward, r\({}_{t}\), can then be acquired. The Bellman equation should then be used to update the action value Q(t) = Q(s\({}_{t}\), a\({}_{t}\)):  
\[Q(t)=Q(t)+\beta(r_{t}+\gamma_{max,t+1}Q(t+1)-Q(t)) \tag{1}\]  
In our instance, the environment is very complicated, thus a Q-table of this size would be unsuitable. Consequently, a neural network (NN)-based Q-learning technique is employed in its place. The training network and the target network are the two neural networks used in DQL's learning process. Despite having differing weights, these networks share the same architecture. When the parameters of the training DQN are backpropagated using the Adam [24] optimizer algorithm, the parameters of the target DQN are updated in accordance with the evaluation of the trained DQN at a specific rate. It is crucial to stress that the target network's parameters are periodically synchronized with the main Q-network's parameters rather than being trained. Training the main Q-network with the target network's Q-values improves training stability. Q-values representing multiple actions will be produced as output for a state value supplied as input to the NN. The action that maximizes the Q-value is chosen from these outputs, and the target action is defined as follows:  
\[\gamma_{i}=r_{i}+\gamma_{max,t+1}Q(t+1) \tag{2}\]  
\[L=\frac{1}{N}\sum_{i=0}^{N-1}(Q_{i}-y_{i})^{2} \tag{3}\]  
The proposed DQN-based migration strategy outperforms the state-of-the-art models: Kalman filter, IQR, MAD, LR, SES, and ARIMA in resource utilization, energy efficiency, migration time, and adaptability.