## 2 Methodology
### Algorithm  
The AI service applied in these tests is based on real-time optimization. Information for the optimizer is obtained from both the probe and the transcoder, which completes the set of states of the environment. This development at the code level allows us to represent the environment and to be able to run actions on the components of the service.  
Generally, AI algorithms require a dataset to create a model. The service we present is created in real time with the environment data, from the quality measurements offered by the probe to the transmission values of the transcoder. The techniques capable of applying intelligence are those consistent with RL, through which we expand thanks to the use of DL. This combination of techniques is called DRL.  
Optimization of the service is performed on the quality measured in the probe, exerting actions on the transcoder. Based on the environment values, the optimizer analyzes the policies and makes the most optimal decision, even seeking to solve possible bottlenecks in the network.  
DRL algorithms combine reward maximization techniques through action policies, with the addition of NN (Actor and Critic) as agents that interact with the environment to improve training efficiency. Based on the effects of the actions, the environment returns a reward. The algorithm developed and trained for this case is called A2C [19]. A high-level diagram is presented in Figure 2.  
Figure 1: Simulated general optimization workflow.  
In recent years, new RL algorithms have proliferated, but among them, A2C has been one of the most prominent due to the use of NNs with RL techniques. We emphasize that its choice for this work was due to the possibility of training agents in parallel, giving the option as future work of expanding the number of parallel sites for more efficient training, as well as the final staging of the trained model.  
Going into detail, the Advantage Function informs the algorithm how well (or poorly) an agent has behaved when performing a certain action compared to previous actions. This also helps inform the model that training is appropriate and that no further steps are required. That is, find a balance between a possible training bias while maintaining a data variance.  
The Actor-Critic is a set of DRL algorithms that combine the ideas of algorithms focused on obtaining the value function (and the action-value function) and those of maximizing the optimal policy function (check Equations (1) and (2) below). These equations are called Bellman equations [20]. They help agents assess the current environment situation and not wait for long-term results. These equations introduced the state value function (\(V_{\pi}\)) as the expected return over all possible actions for a specific state, and the state-action value function (\(q_{\pi}\)) as the expected return over all possible actions in some state, both following the policy \(\pi\). Other important variables from these equations are the expected value over the policy distribution (\(\mathbb{E}_{\pi}\)), the set of states and the current one (S and s), the set of actions and the current one (A and a), the accumulated reward (\(R_{t}\)), the returned reward for a specific trajectory (\(G_{t}\)), and the discount factor (\(\gamma\)).  
\[\begin{split} V_{\pi}(s)&=\mathbb{E}_{\pi}[R_{t}+ \gamma G_{t+1}|S_{t}=s]\\ &=\sum_{a}\pi(a|s)\sum_{s^{\prime},\tau}p(s^{\prime},r|s,a)[\tau +\gamma v_{\pi}(s^{\prime})]\\ q_{\pi}(s,a)&=\mathbb{E}_{\pi}[R_{t}+\gamma G_{t+1 }|S_{t}=s,A_{t}=a]\\ &=\sum_{s^{\prime},\tau}p(s^{\prime},r|s,a)[\tau+\gamma\sum_{a} \pi(a|s)q_{\pi}(s^{\prime},a^{\prime})]\\ &\quad\quad\quad\quad\quad v_{*}(s)=\max_{\pi}v_{\pi}(s)\\ q_{*}(s,a)&=\max_{\pi}q_{\pi}(s,a)\end{split} \tag{1}\]