# Assessment of Thermal Comfort in an Electric Bus Based on Machine Learning Classification
## 2 Materials and Methods
### Machine Learning Model Design
#### 2.5.2 Implementation and Accuracy Metrics  
To evaluate the classifiers mentioned in Section 2.5.1, the method called repeated random sub-sampling was used, where 70% of the data set was randomly selected as the train set and the other 30% was used as the test set while keeping stratification. Whereas the train set is used to build the models, the test set is employed to evaluate its accuracy by testing it with unseen data. For each classifier, the train set was used to fit a Min-MaxScaler [44], which was then employed to normalize all parameters on the same scale for both the train and the test set. Repeated random sub-sampling carries this random test split procedure, in our case for 100 times, which provides 100 scores for each model, which will then be used to measure the overall performance of the model. This is particularly useful for handling a small dataset (such as ours) as it also provides an insight on the variability of the scores, it reduces bias results as a single high or low score, which might come from lucky sampling, is not strongly reflected in the overall performance metric. Additionally, random seeds are used to ensure that the same train-test splits are employed for all the classifiers.  
The _ANN_ was implemented using the Python library PyTorch, all the other classifiers were trained using the Python library scikit-learn using its default hyperparameters. All of the methods used balanced class weights to help counteract the effect that the imbalance of the data set might have on the performance, with the exception of _AdaBoost_ and _kNN_, whose optimizations are not compatible with the balanced class weight strategy.  
To evaluate the accuracy of the results, the \(F_{1}\) score [45] is employed. \(F_{1}\) score was chosen over traditional accuracy (percentage of correctly classified data points) because it is a better metric for imbalanced data sets, such as the present one. Its main advantage is that it considers both _Precision_ and _Recall_, which is important when not only correctly classified but also miss-classifications are of interest. In the supplementary material precision-recall curves and receiver operating characteristic curves results can be found.  
For the given data set, positively classified data points are understood as data points that were classified in group 2 from the ML Model, while negative classification refers to the data points classified in group 1. Accordingly, True and False classification refers to the data points that were correctly/incorrectly classified. As an example, a False Positive Classification describes a data point that is wrongly (false) classified as belonging to group 2 (positive).  
_Precision_ is defined as the percentage of positive classifications that are actually correct, compared to all the other positive classifications (regardless of true or false). This is computed as  
\[Precision=\frac{TruePositives}{TruePositives+FalsePositives} \tag{4}\]  
_Recall_ (also known as _Sensitivity_) is the percentage of true positives compared to all the actual data points belonging to group 2 (positive) and is computed as  
\[Recall=\frac{TruePositives}{TruePositives+FalseNegatives} \tag{5}\]  
The \(F_{1}\) score is calculated as the harmonic mean of the _Precision_ and _Recall_ as shown in Equation (6):  
\[F_{1}=2\cdot\frac{Precision\cdot Recall}{Precision+Recall} \tag{6}\]