## 3 Proposed Beam Learning with MARL
### Proposed Algorithms
#### 3.4.3 Proposed MARL-Based Codebook Learning Function  
Proposed MARL-based beam codebook learning for \(N\) agents is given in Algorithm 2. The input to each actor network is the corresponding state. The state is the concatenation of the 24 length channel vector given as \(N_{CH}\) and 32 length phase vector which equals \(N_{BS}\). Thus, the length of state vectors are \(N_{CH}+N_{BS}\), which is 56 in this case. The output of the actor networks also comprises the predicted phase update vectors which are of length \(N_{BS}\), i.e., 32. The actor network includes a pair of hidden layers, each containing \(10\times(N_{CH}+N_{BS})\) neurons equating to 560. These layers are subsequently activated using rectified linear units (ReLU). The outcome of the actor network stands as the anticipated action. This outcome is then passed through hyperbolic tangent (tanh) activations, which are scaled by \(\pi\).  
Thus, the length of the input of each critic network for a four agents can be given as \((4\times(N_{CH}+N_{BS})+4\times N_{BS})\), i.e., 336 in this case. The output of the critic network is the predicted Q value, which is a real valued scalar. Hence, output dimension of critic network is 1. The critic network is composed of two hidden layers, each layer containing \(5\times(4\times(N_{CH}+N_{BS})+4\times N_{BS})\), i.e., 1680 neurons. Following this, ReLU activations are applied to these layers.  
Hyper parameter for the MARL is given in Table 2.  
\begin{table}
\begin{tabular}{l l} \hline \hline
**Optimizer** & **ADAM** \\ \hline Learning rate & 0.01 \\ \hline Target soft update parameter & 0.95 \\ \hline Replay buffer size & 12,288 \\ \hline Batch size & 1024 \\ \hline No of samples added to replay buffer before each network update & 100 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Hyper parameter table for MARL.  
```
1:Initialize actor networks, critic networks with random weights
2:Initialize target networks and with the weights of actor and critic networks
3:Initialize the replay memory D, minibatch size B, discount factor \(\gamma\)
4:Initialize reward processing algorithm
5:Initialize N beams with beam-steering codebook in procedure 2 of Section 3 for N clusters and N agents.
6:Initialize a random process \(N\) for action exploration
7:For each agent, initialize random initial beamforming vector as state, x.
8:for\(t=1\) to max-episode-lengthdo
9:For each agent i, select proto action \(a_{i}=\mu_{\theta_{i}}+N_{t}\) w.r.t. the current policy and exploration.
10:for each agent i, quantize proto action to valid beamforming vector with KNN for k = 1.
11: Execute action \(a=(a_{1},a_{2},...a_{N})\) and observe reward r (with Algorithm 1) and new state \(x^{\prime}\)
12: Store \((x,a,r,x^{\prime})\) in replay buffer \(D\)
13:\(x\gets x^{\prime}\)
14:foragent\(i=1\) to \(N\)do
15: Sample a random minibatch of S samples \((x^{j},a^{j},r^{j},x^{\prime\prime})\) from \(D\)
16: Set \(y_{j}=r_{i}^{j}+\gamma Q_{i}^{\mu^{\prime}}(x^{\prime j},a_{1}^{\prime},...,a_ {N}^{\prime})\big{|}_{a_{i}^{\prime}=\mu_{\xi}^{\prime}\big{(}a_{i}^{j}\big{)}}\)
17: Update critic by minimizing the loss \(L(\theta_{i})=\frac{1}{5}\sum_{j}\left(y^{i}-Q_{i}^{\mu}\left(x^{j},a_{1}^{j},...,a_{N}^{j}\right)\right)^{2}\)
18: Update actor using the sampled policy gradient:
19:endfor
20: Update target network parameter for each agent i:
21:\(\theta_{i}^{\prime}\gets\tau\theta_{i}+(1-\tau)\theta_{i}^{\prime}\)
22:endfor
```  
**Algorithm 2** Proposed MARL-based beam learning