## 4 Proposed Solution (Intelligent Agent Project)  
In this problem, the aim of the reinforcement learning modeling is to find an optimum policy able to indicate the most suitable techniques that the agent should select among the actions available, considering beamforming and power control. From the point of view of reinforcement learning, the problem can be modeled as follows: the state of the environment is represented by a discrete set of SINR values, so that S = {SINR1, SINR2,..., SINR=}, where SINR\({}_{m}\) represents the maximum value defined for the SINR. In each state \(s\in S\), the deciding agent must select an action \(a\) from a set of actions available in the state \(s\), denoted by _A(s)_. The possible actions available for each state are beamforming (BF) and power control (PC). The Q-learning algorithm governs the decision to explore or take advantage, using the policy known as \(\varepsilon\)-greedy. This policy is defined in the algorithm for selection of the action that possesses the highest value for the utility of the state (greedy criterion), with probability (\(1-\varepsilon\)), and for random action, with probability \(\varepsilon\). As a consequence of selection of an action \(a\in A\)_(s)_, starting from state \(s\) and at instant of decision \(t\), the deciding agent receives a reward _r(s,a)_. The selection of action \(a\) alters the perception of the agent in relation to the environment, leading to a new state \(s_{t+l}\) that conducts it to a new instant of decision \(t+l\). When the reward is positive, it is seen as a profit or prize, and when it is negative, it is seen as a cost or punishment. In definition of the return function (reward), an indication should be provided of the objective to be achieved by the algorithm. In this problem, an attempt is made to minimize the total transmission power, maintaining constant the gain of the desired signal, and keeping the SINR above a pre-established threshold \(\delta\). The reward was therefore defined as being positive when the SINR approached the threshold, and negative when the opposite was true. Once the states, actions, and return function had been defined, the next step was the process of training the agent, as shown in Algorithm 2. At the start of the learning process, the agent has no knowledge of the result obtained by choosing a particular action, so it performs various actions and observes the results. For a while, the agent explores many actions that result in increasingly greater rewards, and gradually tends to repeat them (exploration); after this, it acquires knowledge from the actions, and can sometimes learn to repeat those that result in greater rewards (exploitation). Algorithm 2 requires information for the parameters (line 1): learning coefficient (\(\alpha\)), parameter for regulation of the greedy criterion (\(\varepsilon\)), discount rate (\(\gamma\)), and LMS stop criterion (\(\xi\)). Subsequently, the matrix \(Q\) is initiated using random values (line 2), and the initial state is determined randomly within the values defined in \(s\) (line 4). An available action is chosen for \(s\) (line 6), and according to the action selected, the corresponding technique (BF or PC) is executed. If the action selected is BF, the algorithm will adjust the weights while the determined condition (line 8) remains satisfied, and if not, will update the transmission power in accordance with the equation presented in Algorithm 2 (line 16). Assuming that the initial action PC is chosen, a random value was given for the _min_mse_ variable equal to 2 (which is the minimum value of the error) and then the powers were updated. Otherwise, _i.e.,_ if the chosen initial action is BF, the _min_mse_ used to update the powers is based on the last iteration of the error vector \(\varepsilon_{i}\) in BF operation. The power is a vector of positions, where \(M\) is the number of signals that are addressing the array of sensors.