## 3 A Contextual Bandit-Based Reinforcement Learning Approach
### Model Formulation  
This subsection models the pursuit of an optimal decision boundary for attack detection as a multi-armed bandit (MAB) optimization task. The overarching objectives of an attack detection system are twofold: to correctly recognize malicious behavior based on observable traits, and to remain practical and cost-conscious in real-world operational settings, for instance, by minimizing network delays. To achieve these goals, a collection of detection models must be thoroughly trained on high-quality datasets that capture recent and plausible attack scenarios.  
Relying on a single detection model is often insufficient due to difficulties in generalizing across multifaceted attack patterns. Thus, we adopt an ensemble detection paradigm. By leveraging multiple specialized detection models simultaneously, ensemble methods have shown improved odds of achieving high performance [49; 50; 51; 52]. However, this work does not address multi-classifying detectors intended to pinpoint specific attack categories. Instead, we define the detection mechanism as a sequence of binary classifiers \((c_{i})_{i=2}^{n}\) for \(i\) ranging from 2 to \(n\), where n denotes the total number of distinct classifiers designed for a given feature set and classification algorithm. In this context, the detection process partitions the attack space into various types using distinct subsets of the global feature set \(F\). Let \(Fi\ \subset\ F\) represent the subset of features applicable to the classifier \(ci\), and let \(yi,t\) denote \(ci\)'s predicted class at time \(t\).  
Feature extraction costs can differ substantially. Some feature sets require continuous real-time capture, such as those relying on network traffic data, while others, such as standardized log files, demand fewer resources. As a result, the solution can be conceptualized by grouping features into subsets ordered according to their computational expense. At each time step t, the system receives an observation \(Xt\) and utilizes one or more classifiers to infer \(Xt\)'s class. This inference produces a probability vector \(Ot=(P1,\ P2,\ \dots,\ Pn)\), where \(Pi\) is \(ci\)'s probability estimate at time \(t\).  
To conserve computational effort, higher-level (and more resource-intensive) classifiers \(ci+1,\ \dots,\ cn\) are only triggered if \(Pi\) resides within an uncertainty region defined by the lower and upper thresholds \(Li\) and \(Ui\), respectively. This conditional activation process, referred to as \"triaging\", helps avoid unnecessary evaluation. Specifically, if \(Pi\) falls between \(Li\) and \(Ui\) (i.e., \(Li<Pi<Ui\)), \(Xt\) is escalated to the next classifier \(ci+1\) for further scrutiny; otherwise, _ci_'s decision stands as the final output. Although reducing the number of subsequent classifications yields cost savings, it may also diminish the overall detection effectiveness, given that some nuanced attacks might require deeper analysis.  
To enhance flexibility in deciding when to trigger the next classifier, the traditional single, symmetric decision boundary is replaced with a dual-threshold mechanism. Because each classifier's uncertainty distribution is modeled independently with respect to \(Pi\), the class-specific probability distributions of _ci_ guide the selection of boundary values \(Li\) and \(Ui\) for each \(i\), where \(i<n-1\). This approach enables the dynamic adjustment of the decision environment and aims to balance computational cost against detection accuracy.