## 2 Research Methodology
### Model Performance Evaluation  
Evaluation metrics are used to understand and evaluate how effective the model is based on some score, performed on the test dataset. Different evaluation metrics emphasise different aspects of performance of the classifier algorithm. It is essential to define the objectives of evaluation in order to enable suitable selection of metrics. The classification approach for this study is a multiclass classification model and the evaluation of these models can be understood in the context of a binary classification model. The major evaluation metrics for binary classification model are derived from four categories. * True positives (TP) where the actual label is positive and correctly predicted class is positive;
* False positives (FP) where the actual label is negative but incorrectly predicted as positive;
* True negatives (TN) where the actual label is negative and correctly predicted as negative;
* False negatives (FN) where the actual label is positive but incorrectly predicted as negative. A multiclass classification problem can be viewed as a set of many binary classification problem. The commonly used evaluation metrics for multiclass classification based on these elements are described in Table 1 (in Appendix A). Standard metrics such as average accuracy and error rate can provide significant insight regarding the overall performance of the model. However, the data set used for this study consists of instances with unbalanced dataset, with the major class being the healthy system. Accuracy and error rate can become unreliable measures of model performance when the skew in class distribution is severe. Accuracy and error rate also have limitations with respect to different misclassification costs [42]. For railway application, the cost of misclassifying faulty systems as healthy systems is higher than misclassifying healthy systems as faulty systems. The former instance may lead to less intervention that can lead to hazardous deterioration of track system integrity and increase the risk of derailment. The later scenario of misclassifying healthy systems as faulty systems will not pose a threat to the safety and technical integrity of the system. However, the number of site visits might increase due to false alarms leading to increase in cost of maintenance. For the above-mentioned reasons accuracy and error were not considered for evaluating the models in this study. The limitations of accuracy and error can be tackled using precision and recall or a combination of both. Precision quantifies the number of true positives of all actual positive predictions, however precision alone cannot be used to evaluate the performance of the model as it does not provide insight on the missed positive predictions. Recall is used to complement precision as it quantifies the number of correct positive predictions made from all possible positive predictions that could have been made. Precision addresses the question \"of all the positive predictions, what is the probability of them being correct?\" and recall addresses the question of \"among all the items that belong to positive class, how many does the classifier actually detect as positive?\" Precision and recall aim at minimising false positives and false negatives respectively. Often increase in precision leads to the reduction of recall. F-score or F1-score, which is the harmonic mean of the fractions of precision and recall, can be used to combine the properties of both precision and recall in to one single metric. F1-score can be used as a strong metric for model evaluation when both precision and recall are important. For multiclass classification precision, recall and F1-score are represented by microaveraging or macroaveraging. A microaveraging method aggregates the contribution of all classes to calculate the average performance, whereas macroaveraging will calculate the performance individually for each class and then performs an average on it. Macroaverage considers each class equally.