## 2 Materials and Methods
### Experimental Setup
#### 2.3.3 Quality Measures  
The AUC ROC is a measure of the ability of a binary classifier to separate two classes and can be seen as a single-number summary of a ROC plot [48]. In a ROC plot, the true-positive rate is plotted against the false-positive rate at increasing threshold levels for thresholding the output of the classifier. The higher the AUC ROC, the more easily the classifier can separate the two classes. A perfect classifier achieves a score of one by ranking all examples of the positive class higher than all examples of the negative class. Therefore, we use the AUC ROC as a measure of the quality of the produced anomaly scores: a high score indicates good separability between normal and anomalous points or subsequences. We are aware that the AUC ROC is not a suitable measure for unbalanced problems such as anomaly detection, where the anomalous class is small by definition, but we report it due to its widespread use in the literature.  
F1 Score  
The F1-Score is the harmonic mean of precision and recall and is defined as  
\[F1=2\cdot\frac{precision\cdot recall}{precision+recall}=\frac{TP}{TP+\frac{1}{2}( FP+FN)}\]  
where TP, FP and FN are the true-positive, false-positive and false-negative detections, and precision and recall are defined as:  
\[precision=\frac{TP}{TP+FP},recall=\frac{TP}{TP+FN}\.\]  
Since the F1 Score is calculated based on the result of the binary classification, it provides evidence about the quality of the threshold used. If a method has a high AUC ROC but a low F1 Score, this would indicate a poor threshold.  
The UCR score is the recommended metric provided with the UCR Anomaly Archive and is a binary score indicating whether or not a method was able to find the single anomaly in a time series. It is defined as:  
\[UCR_{\text{score}}:=\mathds{1}\left(min(a-L_{S},a-100)<t^{*}<max(b+L_{S},b+100)\right) \tag{3}\]  
where \(a\) and \(b\) are the beginning and end of the true anomaly with length \(L_{S}\), \(t^{*}\) is the timestamp of the point with the highest anomaly score, and \(\mathds{1}\left(\cdot\right)\) is the indicator function. For subsequences \(S_{i,j}\), we use the middle point \(t^{*}=i+\lfloor\frac{i-j}{2}\rfloor\). The tolerance of 100 time steps is added to account for very short anomalies [42].  
Being a binary measure, the UCR score tells us whether or not the single anomaly in a certain time series was detected by having the highest anomaly score. A UCR score of zero does not convey any information about whether the anomaly was found, but a false-positive result has a higher anomaly score or was not detected at all. As this might not be necessary for a situation such as a challenge, where only positive results matter, it is essential to consider other metrics such as the F1 score alongside visual inspection to correctly interpret the results. If a method shows a UCR score of one but a low F1 score at the same time, this indicates the detection of the true anomaly with the highest anomaly score; otherwise, the UCR score would be zero. A low F1 score, however, can be either caused by the presence of false-positive or false-negative results. It may also be subject to the detection of a short anomaly within the 100 time steps tolerance interval considered in Equation (3). Therefore, evaluating the F1 score alone is not sufficient. On the other hand, if a method shows a UCR score of 0 but a high F1 score, this indicates that the anomaly was identified without many false positives or false negatives, but that the anomaly score for the true anomaly ranked lower than for false detections. The reasons for false positives or false negatives are not sufficient for false positives or false negatives.  
false negatives can be manifold and will be discussed in more detail at the beginning of Section 4. When interpreting aggregated UCR scores, an averaged UCR score of 0.5 means that the true anomaly was successfully identified as having the highest anomaly score in half of the analyzed time series.