# Thermal Runaway Diagnosis of Lithium-Ion Cells Using Data-Driven Method
## 4 Performance Evaluation
### Comparison between Different Machine Learning Models
In each figure, as the curve approaches the upper left corner and the diagonal line, the classifier improves and worsens, respectively. Note that the upper left corner is the optimal operating point (i.e., TPR = 100%, FPR = 0%) of the classifier. The DTE curve moves toward the upper left corner, especially for Heat and Nail, because of the low FPR of DTE. DTE significantly outperforms other classifiers for all causes. To provide a more detailed analysis, Figure 4 also summarizes the AUCs of all models for different abuse methods. The DTE model consistently provides higher AUCs than those using other models for the Heat and Nail classes. For the ISC abuse, the AUC value of DTE remains above 0.92, while other models, including SVM, NB, and MLP, show relatively low AUC values below 0.82. Hence, the comparison between different models proves the strengths of the ensemble aggregation methods of diverse decision trees to capture the complex relationship between various features and abuse methods. Figure 3: Confusion matrix of SVM, NB, DTE, and MLP models for different Heat, ISC, and Nail abuses. Next, we discuss the performance sensitivity of ML models for various batteries of cell formats and manufacturers. Table 6 summarizes the classification accuracies of trained ML models for various cell formats, including 18650 and 21700, and manufacturers including KULR, LG, Molicel, Saft, Samsung, Sanyo, Sony, and Soteria. DTE still performs better than any other models for 18650 and 21700 cells. The prediction accuracy of the 21700 cell is worse than that of the 18650 cell for all models. The main reason is the imbalanced data between different cell formats of the open failure dataset [18]. The dataset ratio between 18650 and 21700 cell formats is 0.77:0.23. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & & **SVM** & **NB** & **DTE** & **MLP** \\ \hline \multirow{2}{*}{Cell format} & 18650 & 0.85 & 0.72 & 0.93 & 0.85 \\ \cline{2-6}  & 21700 & 0.46 & 0.64 & 0.82 & 0.56 \\ \hline \multirow{6}{*}{Manufacturer} & KULR & 0.58 & 0.42 & 0.59 & 0.33 \\ \cline{2-6}  & LG & 0.85 & 0.81 & 1 & 0.96 \\ \cline{2-6}  & Molicel & 0.9 & 1 & 1 & 1 \\ \cline{2-6}  & Saft & 0 & 0 & 1 & 0 \\ \cline{2-6}  & Samsung & 1 & 1 & 1 & 1 \\ \cline{2-6}  & Sanyo & 1 & 0.75 & 1 & 1 \\ \cline{2-6}  & Sony & 0.75 & 0.75 & 1 & 1 \\ \cline{2-6}  & Soteria & 0.79 & 0.57 & 0.86 & 0.71 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Classification accuracies of SVM, NB, DTE, and MLP models for various manufacturers and cell formats. Figure 4: ROC curves of SVM, NB, DTE, and MLP models for different Heat, ISC, and Nail abuses. Similarly, DTE shows better accuracy than other models, even for all different manufacturers. We note that the prediction accuracy of Saft, Samsung, Sanyo, and Sony does not heavily affect the overall performance since the number of testing samples for these products is around 3-4. In Table 6, KULR and Soteria batteries show relatively low accuracy for all models. Even though 21,700 cells from both KULR and LG are used for abuse tests, the prediction accuracies are considerably different. While the DTE model perfectly classifies the abuse classes for all 18,650 and 21,700 cells of LG, both cell formats of KULR return a significantly low prediction performance. The description of the KULR battery does not provide any distinct factors compared to other batteries from different manufacturers. On the other hand, one of the specific Soteria 18,650 cells has lower prediction accuracy when it includes the standard current collectors and a Dreamweaver gold separator, which is different from other battery products. Thus, we might need more experimental data and additional features to better predict the abuse classes of KULR and Soteria battery failure.