## 3 A Contextual Bandit-Based Reinforcement Learning Approach
### Action Selection  
As discussed earlier, the classification threshold is defined by two boundaries: a lower boundary \(Li\) and an upper boundary \(Ui\). Since each category (benign or malicious) is associated with distinct levels of prediction confidence, each class naturally follows its own probability distribution. We initially set the central decision boundary at \(Z=0.5\), corresponding to the midpoint between the high-confidence regions of both classes. Here, prediction scores closer to zero indicate strong negative (benign) certainty, whereas those nearer to one signify robust positive (malicious) certainty. In a continuously learning system, each newly processed data instance \(\mathrm{Xt}\) may necessitate an adjustment of these boundaries based on prediction accuracy. Let \(yt\in\{0,1\}\) denote the true label of the observation \(Xt\) at time \(t\), where \(yt=0y\) denotes a benign instance and \(yt=1y\) a malicious one. Given a particular classifier \(ci\) producing a prediction \(yi,ty\_{i},t\) at time \(t\), we define an action \(a\) that responds accordingly. If \(yt=0\), the action affects the lower boundary \(\mathrm{Li}\); if \(yt=1y\), it influences the upper boundary \(Ui\). In this work, three well-established MAB algorithms are evaluated for selecting the appropriate action: Softmax, Upper Confidence Bound (UCB), and Thompson sampling, as introduced in Section 2.1. Furthermore, we propose an extended variant of UCB, denoted as UCBe, which employs the update rule in Equation (1) instead of a running average for its value estimates. Each algorithm leverages historical data collected through interactions with the environment to determine which action to choose. We conducted a preliminary parameter-tuning experiment on a small scale (50 iterations) before performing a more comprehensive assessment over 10,000 iterations. These evaluations were carried out under multiple proposed reward policies. Since all three algorithms depend on a defined reward policy, we assign both a reward value and a corresponding value estimate to each action. Let \(Rt(a)\) denote the reward obtained by selecting action \(a\) at time \(t\), as determined by the reward function discussed in Section 4.3. This function measures the impact of each action on the decision environment, guiding the classification boundaries toward an optimal setting that enhances cost-effectiveness. The action selection strategy utilizes expected values derived from historical rewards to establish the decision boundary. To represent the anticipated outcome of choosing a particular action \(a\) for a given classifier \(ci\), we define the value estimate \(Va\). Initially, \(Va\) is updated using performance feedback from the boundary's behavior at time \(t\). As new observations accumulate, these value estimates adjust to better reflect environmental responses, thereby improving future action selections. More accurate estimates lead to the more effective triage of incoming observations. To address this, we compute the expected value for each action using the following recursive update:  
\[V_{a,c_{i},t+1}=V_{a,c_{i},t}+\gamma[R_{t}(a,c_{i})-V_{a,c_{i},t}] \tag{1}\]  
where \(\gamma\in(0,1]\) is a decay factor determining the relative emphasis placed on recent rewards. A higher \(\gamma\) value grants greater importance to immediate feedback, while a lower value places more emphasis on historical averages. Empirical methods are employed to select this parameter. We measure the performance of the selection algorithms using two metrics: cumulative regret, and the total number of observations that undergo subsequent inference (triaged observations). Cumulative regret quantifies the cost of not always choosing the best possible action, while the frequency of triaged observations reveals how often boundaries are shifted away from regions where correct classifications are already stable.