# Collaborative Beamforming with DQN for Interference Mitigation in 5G and Beyond Networks
## 3 Proposed DQN-Based 5G Beamforming Architecture
The DQN phase shifter's objective is to steer the interference angle, which is associated with reducing the interference power and enhancing the SINR. In our work, the state, \(s_{t}\), is organized with different information such as coordinates of users, coordinates of the base station, angle of the signal, serving power, and interference power from a particular base station for the user. The proposed DQN phase shifter can choose only one action \(\alpha_{t}\) among two possible actions, such as left shift angle and right shift angle. According to the concept of time delay in beam steering (if the time delay increases the phase moves to the left and, otherwise, the phase shifts to the right), we assume that if the served user is in the right direction from the interference signal the phase shift goes in the right direction; otherwise, it goes in the left direction. Therefore, the DQN phase shifter can either decrease the time delay or increase the time delay. We select the value of shift at 0.001 s, similar to [1] for either decreasing or increasing time delay. For each period of phase shifting, the interference signal loses some power, and the server signal's strength can be increased. Initially, the agent explores the environment by taking random actions because the agent does not know anything about the environment. After exploring, the agent exploits the environment by taking action from the neural network model. This method is called the epsilon-greedy technique. After taking action \(\alpha_{t}\), the DQN agent takes reward as the difference between served power \(P_{s}\) and interference's power \(l_{p}\). If our proposed DQN predicts the action properly, then the served power will be stronger than the interference power; otherwise, the interference power will be higher. Having this feedback from the environment to our proposed DQN agent can improve the predicted action. In this term, the reward is calculated in (5). \[r_{t}=P_{s}-I_{p} \tag{5}\]  
In the proposed method, our objective is to fulfill the SINR requirement for voice downlink transmission. The requirement is that our system must reach \(\text{SINR}\geq 3\) dB. If the DQN does not fulfill this requirement, the system fails to achieve the goal. During the training session, an essential technique is introduced, called experience reply memory, to reduce the correlated data between the action value and the target value,  
Figure 3: Radiation pattern of different phase angles. \(Y=r_{t}+df.argmax(Q(nextstate,nextaction))\), where \(df\) is discount factor that is between 0 and 1. This technique acts as the trajectory of the current state \(s_{t}\), action \(a_{t}\), reward \(r_{t}\), and next observation \(s_{t+1}\) and stores the experience \((s_{t},a_{t},r_{t},s_{t+1})\) into a replay memory (\(M\)) in each time step to reuse the past experience for better and stable performance after sampling is completed uniformly from the replay memory. Moreover, DQN uses the mini-batch technique to sample data randomly from the memory to train the neural network model via stochastic gradient descent so that the data are independent of each other. Thus, experience replay memory reduces the correlation between datasets. The main goal of the DQN is to optimize the loss function \(L(\theta)\) of the neural network, as in (6). \[L(\theta)=(Y-\theta)^{2}. \tag{6}\]  
In our DQN, we use a long short-term memory (LSTM) recurrent neural network model that estimates the output based on time series data. Figure 4 shows the flowchart of our proposed DQN system, and Figure 5 presents the proposed LSTM model summary with a fully connected neural network, where our first layer contains LSTM with eight units and the remaining layers built with a Dense layer with four and two units, respectively. Note that the final Dense layer's unit is two because our proposed DQN agent predicts binary action (i.e., left shift or right shift).