## 4 Proposed Solution (Intelligent Agent Project)
The higher the adaptation step value, the higher the convergence speed. However, the excess error also becomes larger, which is undesirable. The \(\alpha\), \(\gamma\), and \(\varepsilon\) values were obtained after several simulations. The states of the environment were represented by discrete SINR values between \(-0.8\) and \(5\), which corresponded to index values in the range 1-18. Each action was identified with the label 1 or 2, indicative of beamforming and power control, respectively. An \(\varepsilon\)-greedy policy was adopted, with 80% possibility of selecting the better action. \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameters** & **Value** \\ \hline Number of elements in the array (\(K\)) & 8 \\ Number of signals (\(M\)) & 2 \\ Initial transmit power (P\({}_{0}\)) & 1 W \\ SINR threshold (\(\delta\)) & 2 dB \\ Step Adaptation (\(\mu\) ) & 0.001 \\ Noise Variance (\(\sigma^{2}\) ) & 0.1 \\ Distance between each element of the array (\(d\)) & \(\lambda/2\) \\ Learning Rate (\(\alpha\) ) & 0.1 \\ Discount Factor (\(\gamma\) ) & 0.9 \\ Greedy Rule (\(\varepsilon\)) & 0.2 \\ Attenuation of the \(i\)-th source (\(\rho_{1}\)) & 1 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Simulation Parameters. Two simulations were performed using the parameters shown in Table 1, but with the variation of the noise (\(\sigma^{2}\)) changed to 0.3 in the second simulation. In each simulation, the training was configured to execute 10, 50 and 250 episodes. The results of the first simulation are shown in Table 2, indicating the policies obtained after each different episode. Each line corresponds to a SINR value, and the columns correspond to the beamforming (BF) or power control (PC) processes. The values given in Table 2 correspond to the probability of selecting each technique, for the range of discretized SINR values. The optimum policy was obtained after 250 episodes, and was adopted for testing the agent. Figure 3 present (on the ordinate axis) the states (SINR), and the curves indicate the evolution of the SINR until reaching the target value (\(\delta=2\)). The switching sequence using an initial SINR of \(-0.8\) is illustrated in Figure 4, with the actions executed (beamforming or power control) indicated on the ordinate axis, and the curve showing the order of execution in each processing cycle. Index 1 indicates execution of beamforming, and Index 2 indicates power control. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**Policy 10**} & \multicolumn{3}{c}{**Policy 50**} & \multicolumn{3}{c}{**Policy 250**} \\ \hline
**INDEX** & **SINR(dB)** & **BF** & **PC** & **BF** & **PC** & **BF** & **PC** \\ \hline
1 & \(-0.8\) & 0.5 & 0.5 & 1 & 0 & 0 & 1 \\
2 & \(-0.6\) & 0.5 & 0.5 & 1 & 0 & 1 & 0 \\
3 & \(-0.4\) & 0.5 & 0.5 & 1 & 0 & 0 & 1 \\
4 & \(-0.2\) & 0 & 1 & 1 & 0 & 1 & 0 \\
5 & 0 & 0 & 1 & 0 & 1 & 0 & 1 \\
6 & 0.2 & 0 & 1 & 0 & 1 & 0 & 1 \\
7 & 0.4 & 0 & 1 & 0 & 1 & 0 & 1 \\
8 & 0.6 & 0 & 1 & 0 & 1 & 0 & 1 \\
9 & 0.8 & 1 & 0 & 1 & 0 & 0 & 1 \\
10 & 1 & 0 & 1 & 0 & 1 & 0 & 1 \\
11 & 1.2 & 0 & 1 & 1 & 0 & 0 & 1 \\
12 & 1.4 & 0 & 1 & 0 & 1 & 0 & 1 \\
13 & 1.6 & 0 & 1 & 0 & 1 & 0 & 1 \\
14 & 1.8 & 0 & 1 & 0 & 1 & 0 & 1 \\ \hline
15 & 2 & **Destination** & **Destination** & **Destination** & **Destination** \\ \hline
16 & 3 & 1 & 0 & 0 & 1 & 1 & 0 \\
17 & 4 & 1 & 0 & 1 & 0 & 1 & 0 \\
18 & 5 & 1 & 0 & 1 & 0 & 1 & 0 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Policy Improvement of the agent. Figure 3: (**a**) System response. Agent started with \(\mathrm{SINR}=-0.8\) dB; (**b**) System response. Agent started with \(\mathrm{SINR}=0\) dB; (**c**) System response. Agent started with \(\mathrm{SINR}=5\) dB. Table 3 presents the simulation results obtained using the same parameters, but with variance of the noise equal to 0.3.