# Collaborative Beamforming with DQN for Interference Mitigation in 5G and Beyond Networks
## 3 Proposed DQN-Based 5G Beamforming Architecture  
In Figure 2, the proposed 5G beamforming architecture is presented where multiple base stations BS1, BS2......BSn are randomly allocated independently. In this work, we consider the orthogonal frequency-division multiplexing (OFDM) cellular network as similar to [1]. In the network, the green and red signal indicate the served beamforming and interference to a particular user. The users are randomly distributed in the associated area. Each \(BS\) is supposed to have two antennas, which are used to serve as voice bearers at sub-6 Ghz frequency bands with the association of one DQN phase shifter. The objective of the DQN phase shifter is to control the angle of the interference. In Figure 2a we can observe that BS1, BS2, and BS3 serve independently to users U1, U2, and U3, respectively, and in the meantime, the interference attacks with angle \(\theta_{it}\) from BS2 and BS3 to U1 and U2, respectively. In Figure 2b, our proposed DQN controls the steer with angle \(\theta_{it+1}\) of the interference from BS2 and BS3 in such a way that the power of the interference can be reduced as well as the directivity of the serving beamforming being almost unchanged. Figure 3 shows the effects of changing the angle in terms of radiation pattern, where the x-axis and y-axis represent the degree and power of the beam. As we can observe from Figure 3, the interference first arrives at angle \(\theta_{it}\). Then, after changing the phase of \(\theta_{it}\), it goes to \(\theta_{it+1}\) and reduces the directivity approximately \(-2\) dB. However, the directivity of the main beam remains the same even when we steer the angle of the interference signal. Basically, the beam-steering array factor (BSF) produces the steering angle of the main beam to serve a particular user. Similar to [1], we determine the beam steering array factor, \(BSF(\theta_{k})\), as in (1). \[BSF(\theta_{k})=\frac{1}{\sqrt{N}}\sum_{k=1}^{N}e^{jN\frac{\pi}{\omega}d\cos( \theta_{k})} \tag{1}\]where N = number of antenna, \(\omega\) = wavelength, \(\theta\) = angle of arriving signal, and \(d\) = antenna space. As we have mentioned earlier, our proposed DQN controls the angle \(\theta_{t}\) of the particular interference signal so that interference power can be reduced. Therefore, the steer of interference signal is followed as in (1). After calculating the BSF(\(\theta_{k}\)), we can derive the received power and interference power in terms of downlink transmission from \(BS\) to a particular user by (2) and (3) as per [1]:  
\[R_{p} =P_{s}\times h_{s}\times\text{BSF}(\theta_{k}), \tag{2}\] \[I_{p} =P_{i}\times h_{i}\times\text{BSF}(\theta_{k}), \tag{3}\]  
where \(R_{p}\) is the received power, \(I_{p}\) is the interference power, \(P_{s}\) is the serving power of a particular base station (i.e. ; BS1), \(I_{p}\) is the interference power of a particular base station (i.e., BS2), and \(h_{s}\) and \(h_{s}i\) are the channel gain of a particular \(BS\) (i.e., BS1, BS2 respectively). Now, we can calculate the SINR [1] for the served user at a particular time \(t\) in (4). \[\gamma(t)=10\log_{10}\biggl{(}\frac{R_{p}}{I_{p}+N_{p}}\biggr{)}, \tag{4}\]  
where \(N_{p}\) = noise power. In this paper, we apply our DQN to control the phase shifter to get the better SINR, \(\gamma(t)\) in terms of downlink voice bearer transmission. Figure 2: The proposed DQN-based 5G beamforming architecture: (**a**) Beamforming and interference in a traditional 5G network without DQN control, (**b**) Proposed DQN-based 5G beamforming with interference angle steering. _Reinforcement Learning-Based 5G Beamforming Algorithm_  
In this paper, we introduce our DQN-based deep reinforcement learning, which can control the phase shifter to reduce the slide lobe interference. In our case, the phase shifter is called a DQN agent, which is responsible for taking the best action \(\alpha_{t}\) from a state \(s_{t}\) in our cellular OFDM-based environment.