## 4 Proposed Solution (Intelligent Agent Project)
In executing the available actions for \(s\), the reward (\(r\)) (line 19) and a new state ( s') are reached, according to the equation in line 20, and the matrix Q is updated (line 21). The process (lines 6-22) repeats itself until a final state is found, which in this work was defined by the SINR threshold. After attaining the final state, the algorithm is executed (from line 4) until the defined number of repetitions is completed. The stages of the Q-learning algorithm applied to the sensor array in the agent training step are shown in Algorithm 2. At the end of the training, a \(Q\) matrix is constructed, which is utilized in the functioning of the agent. It is important to note that for each BF action, there is storage of a matrix \(\boldsymbol{W}(s,\,1)\), containing all the optimum weight values, together with a matrix \(\boldsymbol{P}(s,\,2)\), for each PC action, corresponding to all the transmission power updates. Whenever the environment is changed, a new execution of Algorithm 2 is required. ```
1:Require: (\(a\), \(\varepsilon\), \(\gamma\), \(\xi\), \(M\))
2:Initialize \(Q\) (\(s\), \(a\))
3:While maximum episodes are not reached Do
4: Initialize \(s\) % initial state
5:While final state is not reached Do
6: Choose \(a\) according to \(\varepsilon\)-greedy rule; % \(a\) =1(BF) or \(a\)=2 (PC)
7:If a==1 % Beamforming
8:While\(|\varepsilon_{t}{}^{2}(n)-\varepsilon_{t}{}^{2}(n-1)|<\xi\) Do\(\forall\) threshold less than \(10^{-4}\)
9:For i=1, 2,..., \(M\)
10:\(w_{i}(n+1)=w_{i}(n)+\mu\mathbf{x}(n)\varepsilon_{i}(n)\)
11:End-For
12:End-While
13:End - If
14:If a==2 % Power Control
15:For i=1, 2,..., \(M\)
16:\(P_{i}=\delta P_{i}\frac{min\_mse}{1-min\_mse}\)%Updated powers
17:End-For
18:End - If
19:Watch \(r\) % Reward
20:\(s^{\prime}=(1-min\_mse)/min\_mse\)% New State
21:\(Q(s,a)=Q(s,a)+\alpha[r_{t+1}+\gamma.\max_{a}Q(s_{t+1},a)-Q(s,a)]\) % Updating Q-table
22:\(s\)\(s\)\(s\)'
23:End-While
24:End-While
25:return Q(s,a) % Q-values Matrix
```  
**Algorithm 2** Sensor array Q-learning algorithm. Algorithm 3 shows the functioning of the agent, where \(j\) is the processing cycle, corresponding to selection of an initial state, execution of an action, and attainment of a new state. ```
2:If (environment changes)
3:training % Algorithm 2
4:Else
5:Choose an initial state \(s\)
6:For j from 1 until max_value Do
7:Choose \(a_{max}\) for \(s\) and run
8:\(s\)\(s\)'
9:End-For

## 5 Simulation and Results

The functioning of the algorithm was demonstrated for two sources (\(M=2\)) using simulation of a situation with angles of 90\({}^{\circ}\) and 30\({}^{\circ}\) for the desired and interferent signals, respectively. The target SINR was 2 dB, and the initial powers were set at 1 W for both sources. All sources were modeled with polar binary signals (\(\
u_{i}(n)\in\{-1,1\}\)) random uniform distribution. The noise at each antenna element was modeled as Additive white Gaussian noise (AWGN) with variance \(\sigma^{2}\). The parameters used in the simulation are listed in Table 1.

In this paper, a linear array of sensors with eight elements and two signal sources was used for a desired source and an interfering source. A linear array with \(K\) elements can create up to \(K-1\) nulls in the direction of the interfering source. When the number of unwanted (interfering) sources is close to such a limit, the attenuation of unwanted signals is reduced and there are excess gains (greater than the gain attributed to the desired signal) in the proximity of the desired and undesired angles. Thus, the use of two sources is in the range between the limits established for the performance of the system. The distance between the array elements (\(d\)) is limited by the value \(\lambda/2\). This limitation avoids the production and overlapping of side lobes. The unit transmission powers were used to initialize the power control algorithm and to facilitate the calculations. The choice of \(\mu\) in the adaptation step is experimentally determined in order to provide stability of the algorithm.