## 3 A Contextual Bandit-Based Reinforcement Learning Approach
### Reward Policy  
We examined our contextual bandit-based reinforcement learning approach across several reward configuration strategies to determine which approach would yield the highest performance under the previously outlined decision environment and selection functions. In reinforcement learning contexts, a reward provides either favorable or unfavorable environmental feedback, thereby guiding the selection mechanism toward optimal actions. In our approach, assigning a reward at each time step t involves two distinct steps: (a) selecting the reward's magnitude, \(R\), and (b) determining its polarity (positive or negative). We evaluated four categories of reward policies: (i) static cost, (ii) imbalanced cost, (iii) contextual index, and (iv) distance. Static Cost Reward Function: Under the static cost framework, the reward is consistently assigned as either \(1\) or \(-1\). While this simplicity is appealing, it does not distinguish between scenarios that narrowly miss a positive outcome and those that are decidedly negative. Thus, all unfavorable results receive an equivalent penalty, regardless of proximity to a more favorable outcome. Implementing a static cost scheme requires the careful calibration of the cost value. In this study, we explored several fixed reward signals, i.e., 0.9, 0.8, 0.7, and 0.6, to determine their effectiveness. Contextual Index: As indicated earlier, relying solely on binary rewards (\(1\) or \(-1\)) treats every instance uniformly, which may not be ideal. To address this issue, we introduce a contextual index mechanism. Let \(ak\) represent the \(k-th\) action from the set of candidate actions \(A\), with \(1\leq k\leq v\). We arrange these actions in ascending order based on their values, making ak's position within the set a contextual factor. Actions with \(k\) closer to \(v\) correspond to higher-confidence predictions. The primary objective is to establish a boundary selection strategy that minimizes reliance on actions near regions of high-confidence predictions, since additional inference might be unnecessarily costly. To achieve this, we employ a logarithmic transformation, defining \(Rt(ak,\ ci)=log(k)\). This approach introduces reward scaling based on relative boundary certainty, reducing the risk of overfitting by avoiding overly aggressive reward gradients in highly confident zones. Imbalanced Cost: Training classifiers for cyber-attack detection often involves datasets with inherent class imbalances, generally with fewer malicious instances than benign ones. This imbalance can skew the reward function, making it disproportionately responsive to the majority class. To mitigate this effect, we draw on a reward function component proposed by [PERSON] et al. [56], which incorporates class distribution ratios. Specifically, we define an imbalance ratio \(\rho=\mid\!\!DP\!\mid/\mid\!\!DN\!\mid\), where \(DP\) represents the minority class subset and \(DN\) the majority class subset. Adjusting rewards according to \(\rho\) can improve the detection performance by ensuring that both true positives and true negatives are appropriately weighted. By integrating class imbalance into reward estimation, the model avoids skewed updates that could lead to overfitting on the dominant class. Distance: An alternative contextual strategy involves tying the reward to the distance between the predicted probability and the selected action. For each classifier \(ci\) and chosen action \(a\) at time \(t\), we define \(Rt(a,\ ci)=dP,a,t\), where \(dP,a,t=\mid\!\!Pi-a\!\mid\) denotes the magnitude of the discrepancy between the prediction \(Pi\) and action \(a\). This difference can be viewed as an error metric, restricted to the range \(0.5\leq dP,a,t\leq 1\), reflecting that the initial decision threshold \(Z=0.5\), and that ideal predictions approach 1 for benign and 0 for malicious classes. Since this method focuses solely on \(Pi\) and \(a\), it constrains the reward assessment to the local context rather than the broader operating environment.