# Anomaly Detection in Financial Time Series by Principal Component Analysis and Neural Networks +
## 6 Model Evaluation: Additional Results
### Cutoff Value Robustness  
In most anomaly detection models, the cutoff value is a hand-set parameter. In our approach, the cutoff value is a model parameter and as such, it is calibrated through \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Method** & **Ano** & **PCA** & **LI** & **BF** & **Clean** \\ \hline Mean & 0.004208 & 0.001575 & 0.001554 & 0.001554 & 0.001552 \\ Standard deviation & 0.000272 & 0.000165 & 0.000170 & 0.000172 & 0.000173 \\ \hline \hline \end{tabular}
\end{table} 
Table 13: Mean and standard deviation errors on covariance matrix after imputation of anomalies with baseline methods. Figure 12: Original, contaminated, and reconstructed stock price paths. Figure 13: Original, contaminated, and imputed stock price paths in black, red, and orange. The two additional graphs represent the region around the anomalies, where the crosses and circles, respectively, show the true value of the stock and its value after imputation of the anomalies using the reconstructed value suggested by the PCA NN approach. the learning. Testing the robustness of the cutoff given by the model is therefore a must. The identification step is a unique step in the approach and concerns the robustness since it is the only step involving the cutoff calibration. The robustness was checked by shocking the suggested cutoff with different levels of noise and observing the impact of these shocks on the model's performance. We considered several shock amplitudes \(\gamma\in\pm\{10^{-4},10^{-3},10^{-2},10^{-1},1,2\}\). Table 14 reports the mean and standard deviations of the scores of the model. The cutoff calibrated on the synthetic training data sample was shocked. Scores for both the training and test sets were computed using the shocked cutoff values. Excluding the extreme cases where the shock amplitude was \(\pm\{1,2\}\), one can see that the accuracy was barely impacted by the shocked cutoff values, for both the training and test sets. The interpretation of the remaining results is split into two and is applicable for both the training and test sets. When negative shocks were applied, the model predicted more anomalies and fewer normal observations (compared to the predicted numbers with the calibrated cutoff value). Therefore, the model was able to identify the anomalies that were initially missed. Hence, applying negative shocks increased the recall. As for the precision, i.e., the rate of correctly identified contaminated time series, since on the left hand side of the cutoff value we were in the density region where contaminated time series represented the dominant class, the  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline \(\gamma\) & **Accuracy** & **Precision** & **Recall** & \(\mathbf{F_{1}}\) \\ \hline \(10^{-4}\) & 0.9097 & 0.9736 & 0.8421 & 0.9031 \\ \(-10^{-4}\) & 0.9097 & 0.9736 & 0.8421 & 0.9031 \\ \hline \(10^{-3}\  
new position of the cutoff value led to a misidentification of this type of observation. This entailed a deterioration in the precision. Opposite behaviours of the precision and recall were observed when positive shocks were applied since some abnormal time series were missed by the model. Figure 14 shows the precision and recall when cutoff values other than the one we calibrated were used in our model on the training and test sets. These scores were also compared with the performance of the no-skill model (the no-skill model assigns a positive label to all observations). We noted that our approach edged out the no-skill model for both sets. The area under the curve (AUC) scores (AUC score ranged from 0 to 1, with 1 being the score associated with a perfect model) for the training and test sets were, respectively, 0.97 and 0.87. This shows that we performed better on the training set, which was expected, but the performance on the test set was just as satisfactory. These results reinforce our conclusions regarding the robustness of our approach.