## 3 Methodology
### Rlfoula Framework Description
#### 3.2.3 Employing Functional Agents in RL
Equations (1)-(7) describe the other metrics used in this paper specifically for IM, including recall, precision, F1 score, FPR, FNR, and sensitivity. \[\text{recall(sensitivity)}=\frac{\text{TP}}{\text{TP}+\text{FN}} \tag{1}\]  
\[\text{precision}=\frac{\text{TP}}{\text{TP}+\text{FP}} \tag{2}\]  
\[\text{F1 score}=2\times\frac{\text{recall}\times\text{precision}}{\text{recall} +\text{precision}} \tag{3}\]  
\[\text{FPR}=\frac{\text{FP}}{\text{FP}+\text{TN}} \tag{4}\]  
\[\text{FNR}=\frac{\text{FN}}{\text{TP}+\text{FN}} \tag{5}\]  
\[\text{specificity}=\frac{\text{TP}}{\text{TP}+\text{FN}} \tag{6}\]  
\[\text{accuracy}=\frac{\text{TP}+\text{TN}}{\text{TP}+\text{FN}+\text{TN}+ \text{FP}} \tag{7}\]  
\[\text{IM}=\frac{\sqrt{\left(\text{F1 score}^{2}\times\text{accuracy}^{2}\times \text{sensitivity}^{2}\times\text{specificity}^{2}\right)}}{\sqrt{\text{FPR}^{2} \times\text{FNR}^{2}}} \tag{8}\]  
The theory behind this equation is based on the practical meaning of each variable. A higher F1 score, accuracy, sensitivity, and specificity indicate better performance of the classification algorithm. Conversely, a lower FPR and FNR indicate better performance of the proposed algorithm. Since all these parameters are crucial in the RLPOUA framework, we combine them into a single equation, with F1 score, accuracy, sensitivity, and specificity in the numerator and FPR and FNR in the denominator. The IM is utilized by the first agent in the proposed RLFOUA algorithm to decide whether to retain or discard an algorithm combination. Following the computation of the IM by Agent 1, a comparison is made between the IM obtained from the current pipeline and the best IM derived from any of the previous iterations. In the event that enhancements are observed in the IM stemming from the current pipeline compared to the highest IM achieved in prior iterations, the model, algorithm details, metrics, and classification report are preserved. Additionally, if the IM value for the present pipeline registers an increase in comparison to the same pipeline's IM value from the previous iteration, corresponding rewards are escalated. Conversely, in cases where the IM value fails to exhibit an increase, rewards are diminished. Following this reward adjustment step, algorithm combinations possessing a reward falling below the predefined threshold (defaulted to zero in this algorithm) are excluded from the pool of algorithms available for further processing by Agent 1. This orchestrated sequence ensures the continuous refinement of the algorithm pool and model selection, based on the observed improvements in IM and rewards, in order to achieve heightened algorithmic efficacy. If we take out an algorithm combination from the pool, we keep track of its performance measures, like IM, along with its model. We perform this so we can compare it with the next combinations of algorithms. If the removed combination performs better, it is considered the top choice, and its model is saved. After running RLFOUA, we choose the model that performed the best between all iterations for testing. In the RLFOUA framework, Agent 1 and Agent 2 employ Q-learning, a value-based reinforcement learning algorithm, to guide its decision-making process. Q-learning involves learning a value function (Q-function) that calculates the return for each state-action pair in Equation (9). At each time step, Agent 1 updates its Q-values from Equation (10) based on the immediate rewards obtained which will be used for Agent 2 as well for resampling. In these equations, the variable \(V_{s}\) represents the expected cumulative reward at state s, \(R_{s}\) represents the immediate reward (IM metric in Equation (8)) upon transitioning to state s, \(P_{(s^{\prime}|s,a)}\) is the probability of transitioning to state s' from state s when taking action a, \(\gamma\) denotes the discount factor, which falls within the range of [0, 1], \(Q_{(s,a)}\) represents the estimated value (Q-value) of taking action a in state s, and s' is the next state.