# Intelligent Transmission Control Scheme for 5G mmWave Networks Employing Hybrid Beamforming
## 4 Proposed Work
### Hybrid Beamforming for Multi-User mmWave Networks
#### 4.3.2 Multi-Agent Deep Reinforcement Learning for Beamforming Optimization  
A cooperative \"multi-agent deep reinforcement learning (MADRL)\" procedure, named \"multi-agent twin delayed deep deterministic policy gradient (MATD3)\", is created for \"NAFD cell-free mmWave networks\". In situations where exact \"channel state information (CSI)\" is understood, MADRL algorithms are used. Numerical replications are conducted to confirm the effectiveness of the recommended channel estimation techniques and to assess the MATD3 technique's progress. This section outlines the implementation of the MATD3 algorithm presented in this study, with a focus on treating users as performers in both uplink and downlink. It also covers the key components of the algorithm, such as the state information agents observe, their actions, and the rewards they receive. This technique builds on the MATD3 algorithm to dynamically adapt hybrid beamforming in multi-user systems. The method applies cooperative multi-agent deep reinforcement learning (MADRL), where each agent (downlink and uplink users) decides based on estimated channel state information (CSI) and interference. The strategy is to maximize communication performance regardless of user number fluctuations.  
Actions are determined by the optimization for each agent issue with variables. In particular, the \(jth\) transmission strength coefficient is involved in the uplink user's activity, \(P_{U,j}\), whereas the \(kth\) downlink user's action pertains to the strength coefficient \(\eta_{k}\).  
Each agent's observed value is the same as its estimated CSI. The \(jth\) uplink user agent specifically observes \(\hat{g}_{j,z}\), the projected corresponding CSI to all R-APs from all connection users, for all \(j\) and \(z\). It also captures interference data between uplink and downlink users, denoted as \(t_{k,j}\) for all \(k\) and \(j\). As for the downlink user device, it records the estimated equivalent CSI from each T-AP to every downlink user, represented as \(\hat{h}_{knn}\) for every \(k\) and \(m\), along with IUI information, denoted as \(t_{k,j}\), for every \(k\) and \(j\).  
Rewards are given when every agent interacts with their environment after ten minutes of training to receive their immediate reward. The reaction provided by \(R^{LB}_{D,k}(t)\) and \(R^{LB}_{U,j}(t)\) guides the agent in making the subsequent decision. We generate immediate reinforcement for two-way agents during the \(th\) training step, as outlined in the following:  
\[R^{LB}_{D,k}(t)=\omega_{D}R^{LB}_{D,k}+(P_{D}-P_{D,m},-1,1), \tag{26}\]  
\[R^{LB}_{U,j}(t)=\omega_{U}R^{LB}_{U,j}, \tag{27}\]  
where the term following the first one in the mentioned equation suggests that, in case the power constraint is met, positive feedback is implemented; otherwise, the opposite applies, with \(\beta\) being the appropriate penalty factor determined through multiple simulations. Thus, the sum of the immediate rewards at time slot \(t\) is calculated as follows:  
\[R(t)=\sum_{k=1}^{K}R^{LB}_{D,k}(t)+\sum_{j=1}^{J}R^{LB}_{U,j}(t) \tag{28}\]  
By analyzing the recreation outcomes, we confirm the efficiency of the suggested network estimate methods and the union of the MATD3 algorithm. This is further extended by the innovative method known as multi-user scenario-based MATD3 (Mu-MATD3), which adjusts the hybrid beamforming to constantly change user scenarios to maintain efficient communication regardless of changes in the user count. We have reached optimal spectral efficiency at a lower hardware cost process, and further, we use the process with the optimized beam training.