# Thermal Runaway Diagnosis of Lithium-Ion Cells Using Data-Driven Method
## 4 Performance Evaluation
### Comparison between Different Machine Learning Models  
The strictly limited amount of samples \(N=364\) may incur serious generalization problems such as overfitting or data selection bias for different ML models. To quantify the uncertainty of limited data, we evaluate the average and standard deviation of classification accuracies of multiple trained models for each ML technique. We train multiple SVM, NB, DTE, and MLP models based on the randomly shuffled dataset with different 10 random seeds. Table 5 summarizes the average and standard deviation of classification accuracies of 10 trained models of each SVM, NB, DTE, and MLP. The standard deviation of classification accuracy is not significant for each model. Due to the effective dropout layer, the MLP model shows a standard deviation range similar to the other ML models. This result proves that the Bayesian optimization algorithm and the cross-validation efficiently generalize each ML model by reducing the overfitting or data selection bias problems. Now, we compare the performance of the representative SVM, NB, DTE, and MLP models trained using the same training dataset. Figure 3 shows the confusion matrices of SVM, NB, DTE, and MLP models to predict three causes, including Heat, ISC, and Nail. A confusion matrix presents how well a classification model performs for the testing dataset [29]. Overall, the classification accuracies of SVM, NB, DTE, and MLP are 0.80, 0.71, 0.90, and 0.81, respectively. DTE predicts the abuse method of the thermal runaway well compared to other models of SVM, NB, and MLP due to the strength of the model aggregation of multiple decision trees. SVM and MLP show a comparable prediction accuracy of around 0.8, while NB has a higher classification error than other models. We \begin{table}
\begin{tabular}{c c c} \hline \hline  & **Range** & **Selected** \\ \hline Drop probability & \([0.1,0.8]\) & 0.5 \\ \hline Initial learning rate & \([10^{-3},1]\) & 0.001 \\ \hline Gradient decay factor & \([0.1,1]\) & 0.9 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Hyperparameter of MLP. Figure 2: MLP network architecture consisting of the input layer, FC layer, batch normalization layer, ReLu function, dropout layer, and softmax layer for the classification. note that the accuracy of DTE is comparable to the average one while other models of SVM, NB, and MLP slightly perform better than the corresponding ones of Table 5.  
We conduct the mid-p McNemar test of DTE since it allows us to evaluate the statistical significance of whether one classification model performs better than another [30]. The mid-p values of DTE compared to SVM, NB, and MLP are \(0.049,7.2861\times 10^{-4}\), and \(0.022\), respectively. All tests have a mid-p value of nearly zero, which indicates that DTE guarantees better prediction than other models. Both Heat and ISC classes are generally hard to distinguish for different classification models, as shown in Figure 3. Although the ISC abuse may occur at a considerably lower temperature than the one of Heat, there are no measured features that explicitly provide the detailed heat and mass changes during the thermal runaway experiments using FTRC. Furthermore, the real-time measurements are also hard to obtain in practice. The classification accuracies of DTE are around 0.94 and 0.75 for Heat and ISC classes, respectively, greater than SVM, NB, and MLP. SVM, NB, and MLP models show considerably low classification accuracy for the ISC class around 0.6. While SVM, DTE, and MLP perfectly predict nail abuse, the classification accuracy of NB is only around 0.84 for the Nail class. The NB model has classification errors between all abuse methods of the thermal runaway. Figure 4 presents the ROC curves using SVM, NB, DTE, and MLP models for different ISC, Heat, and Nail abuses. It shows the true positive ratio (TPR) over the false positive ratio (FPR) of each abuse throughout threshold intervals of [0, 1].