# Process-Oriented Stream Classification Pipeline: A Literature Review
## 6 Classifier Maintenance
### Evaluation
#### 6.4.2 Evaluation Metrics  
After the (delayed) labels arrive, they can be compared to the model predictions. Metrics can be calculated to assess the performance of the stream classifier. Generally, many works differentiate between three different types of metrics: indicators based on the confusion matrix, kappa statistics, and other criteria related to speed and memory usage [7; 36; 37; 84; 88]. As can be observed in the comparative work of [107], metrics based on the confusion matrix, displayed in Table 7, are the most often-used performance measures. As such, commonly used metrics for benchmarking are Accuracy \(Acc=(TP+TN)/(TP+FP+FN+TN)\), Recall \(Rec=TP/(TP+FN)\), and Precision \(Pre=TP/(TP+FP)\)[88; 112; 131]. Further, the \(F1\)-score \(F1=2\cdot\left(Pre\cdot Rec\right)/\left(Pre+Rec\right)\) aggregates two of the previous metrics in one score, which is particularly useful in the case of imbalanced data. Next to numerical performance indicators, refs. [36; 86] list the Receiver Operating Characteristic (ROC) curve as a standard graphical representation of the trade-off between class assignments. The points spanning the ROC curve depend on the threshold the classifier uses for assigning observations to the positive or negative class. By calculating (and displaying) a convex hull of the points along the ROC curve, the best classifier or threshold can easily be identified as the one with the largest convex hull and thus covered area. Similarly, the trade-off between recall and precision can be displayed in the Precision-Recall curve [86]. ROC and Precision-Recall curves can be summarized in a single value, the Area Under the curve (AUC), called AU-ROC or PR-AUC. To adapt these methods to the streaming context, they are calculated incrementally after the arrival of new observations. For example, ref. [171] presents the Prequential AU-ROC. In contrast to the conventional AU-ROC, it is invariant to changes in the class distribution, as it is incrementally updated. Although the metrics based on the confusion matrix are often used in stream settings, they are also criticized in the literature. For instance, ref. [169] mentions that the classes are frequently imbalanced in the stream setting. A potential approach for processing imbalanced data is aggregating as many fields as possible from the confusion matrix, as it provides a more accurate assessment of the actual classification abilities. Still, ref. [169] introduces the Kappa statistic \(\kappa\), ranging in the interval \([0,\,1]\). In its basic form, the Kappa statistic relates the accuracy of a trained classifier with the accuracy of a random classifier. A variant of this metric with a dedicated focus on the stream setting is \(\kappa_{per}\), which can handle data with temporal dependence [83]. In such scenarios--e.g., when investigating the recordings of surveillance data--the chronological order of observations is of particular relevance. In contrast to its classical counterpart, \(\kappa_{per}\) compares the trained classifier's accuracy with that of a persistent classifier, i.e., a classifier that always predicts according to the label of the most recently observed instance. Thus, this metric helps to identify misleading classifier performance over time once these two performances diverge. Related to this line of thought, ref. [84] presents the generalized or combined Kappa statistic by computing the geometric mean of \(\kappa\) and \(\kappa_{per}\). Since stream classification deals with a huge amount of data that must be processed and kept in memory, other criteria, such as algorithm speed, memory usage, and CPU capacity utilization, are also relevant for evaluation. Here, especially the RAM-hours are discussed [4; 36]. RAM hours indicate the number of gigabytes of RAM deployed per hour, which originally stems from the cost of renting cloud computing services [172]. Ref. [40] presents the running time, which is the time an algorithm needs to run on a dataset, i.e., a predefined number of data records, and is generally measured in seconds. Ref.