## 3 ML for SatCom RRM: Architecture and Techniques
### Reinforcement Learning for RRM  
Reinforcement learning is an area of ML focused on identifying actions that must be performed to maximize the reward signal; in other words, it is concerned with mapping situations to actions focused on seeking that reward. The agent in RL represents the hardware or software that must learn to perform a specific action; in that sense, the agent interacts with an \"environment\", which may be an actual decision process or a simulation. The agent works by observing the environment and making a decision and checking what effects it produces. Suppose that the result of that decision is beneficial. In that case, the agent automatically learns to repeat that decision in the future, while if the effect is detrimental, it will avoid making the same decision again [45]. The Markov Decision Process (MDP), on the other hand, provides a mathematical framework on how to model the interaction between the agent and the environment. A stochastic discrete-time model for which its evolution can be controlled over time is the key objective of MDP. The control policy is associated with a stochastic process and a value function. Our goal is to find the \"best\" policy that solves the described problem. MDP contains a set of states \(s\in S\), a set of actions \(a\in A\), a reward function \(r\in R\), and a set of transition probabilities \(p(s_{t+1}|s_{t},a_{t})\) to move from the current state \(s_{t}\) to the next state \(s_{t+1}\) given an action \(a_{t}\). The objective of an MDP is to find a policy that maximizes the expected cumulative rewards \(R=\sum_{i=0}^{\infty}\gamma^{i}r_{t+i}\), where \(r_{t+i}\) is the immediate reward at time \(t+i\), and \(\gamma\in[0,1]\) is the discount factor. According to the current state-of-the-art, reinforcement learning algorithms have been proposed to manage the resources of a multibeam satellite, even though it is still at an infancy stage. However, these algorithms can only manage a single resource in a multibeam system [27]. Regardless, a fully flexible payload must manage at least three resources (i.e.,  
Figure 6: Convolutional Neural Network (CNN) to manage power, bandwidth and beamwidth allocation. power, bandwidth and beamwidth). In that sense, we assume that as the available resources to operate across all satellite beams increase, it becomes a very challenging problem for a single agent (SA). Therefore, we proposed a Cooperative Multi-Agent (CMA) approach with better performance than SA [34]. The proposed system manages communication resources in response to changes in traffic demand. Resource management training is assumed to be performed online, i.e., whenever traffic requirements change, and the values are updated to retrain the RRM agents given the new conditions. For this reason, the processing time will play a critical role in the performance of RRM that must manage the available resources to minimize the error between the capacity offered at each beam and the required traffic demand and, at the same time, optimize the resources used over time. The RRM cost function can be reformulated as a multi-agent MDP that works cooperatively to achieve the maximum reward, as seen in Figure 7. All agents share the same reward, but each agent must meet some minimum conditions, which guarantees that, despite working cooperatively, each agent will seek its benefit. In the proposed cooperative environment, there is a global reward function, and each agent will know the states and actions of all agents. Each agent must meet the minimum requirements to achieve equilibrium in the system. The illustration of the CMA is shown in Figure 7a. Considering a multi-agent environment involving \(B\) agents, the \(b\)th agent observes the state of the globally shared environment and independently selects an action to perform. Then, the current state is transformed into a new state.