## 7 Observations and Recommendations
### Discussion on Performance/Issues  
As already mentioned at the beginning of the previous section the two main concerns when designing anomaly detection schemes for IWSNs are their resource constraints and lack training data. From the previous discussion it is clear that both of these primary concerns have been overlooked in the literature. Less than a third of the discussed non-parametric schemes considered the practical implications of the proposed schemes. Studies have however shown that many deployed IWSNs are error-prone and exhibit poor performances [71]. The resource constraints and distributed nature of IWSNs makes the design of these systems particularly challenging. Anomaly detection schemes run as monitoring agents on these already complex systems thus adding more complexity and furtherstraining these already resource constrained systems. It is thus important for proposed schemes consider how they would function as part of these complex environments. IWSN testbeds [72; 73; 74; 75; 76] are convenient tools to evaluate the feasibility of designed schemes in a practical environment. These testbeds are however costly and may not always be sufficient to evaluate complex machine learning algorithms but they provide a much clearer picture of how the schemes would fair in deployed full-scale systems than their simulation counterparts. None of the discussed schemes were evaluated using these testbeds, they instead made use of IWSN simulators such as TOSSIM [77] and NS-2 [78]. While these simulators do not accurately reflect the real world deployment environment, they are convenient tools in evaluating the feasibility and performance of proposed schemes and protocols. When considering the second important consideration, just over a third of the schemes did not require a substantial amount of prior knowledge. This is not ideal in IWSNs not only because labelled training data is not readily available in large quantities, but also because the algorithms have to be retrained if normal system behaviour changes. Considering the fact the IWSNs are deployed in dynamic environments and expected to last long periods of time, maintaining these schemes in a practical setting may not always be feasible. Another important consideration is the evaluation metrics of the schemes. Given the vast range of IWSN applications, there is no universal benchmark that can be applied across the board. As discussed in the previous section, the detection accuracy in anomaly detection schemes is also not a good measure of how well the algorithm is actually working. This means that in practice these evaluation metrics should be evaluated based on the specific application scenario. While many of the papers do mention the accuracy and false positive rate, the false negative rate is rarely mentioned. In fact only three of the surveyed papers considered the impact of the false negative rate in their results. The precision and recall of the proposed schemes were rarely taken into consideration in the surveyed literature. Given the fact that anomalies are rare obscure events in practice, this factor should not be overlooked when considering how well the scheme classifies anomalies. The receiver operating characteristic (ROC) curve [79] is becoming an increasingly popular tool for evaluating machine learning algorithms. The curve is the true positive rate (\(y\)-axis) of the classifier plotted against the false positive rate (\(x\)-axis) obtained from the confusion matrix. Using these ratios instead of the standard numerical accuracies is a better performance measure because it includes all the data samples, even those that were erroneously missed by the classifier. In this way designers are able to thoroughly evaluate how well their classifiers are able to distinguish between normal and anomalous data. The area under the curve (AUC) has become a more effective performance measure than the standard numerical accuracy and many of the schemes found in the literature were evaluated using this approach.