# Thermal Runaway Diagnosis of Lithium-Ion Cells Using Data-Driven Method
## 4 Performance Evaluation
### Sensitivity and Feature Analysis of DTE  
Since the DTE model considerably outperforms other models, we analyze more details of DTE, including the sensitivity of dataset size and feature importance. Even though the relationship between the dataset size and model heavily depends on the specific problems, ML models generally improve their performance with dataset size. However, the experimental battery failure data are costly to obtain in practice, especially for the thermal runaway of the battery. Thus, we conduct a sensitivity analysis to quantify the relationship between dataset size and model performance. Figure 5 presents the classification accuracy of DTE as a function of different ratios \(r=0.1,\ldots,1\) of the dataset size with respect to the total number of available samples \(N=364\). The error bar shows the standard deviation around the average accuracy. We observe a nearly logarithmic relationship of the average accuracy with the dataset size. The uncertainty shown as the error bar dramatically increases for the small dataset \(r\leq 0.4\) and stabilizes for the modest dataset \(r\geq 0.5\). We could achieve a comparable average accuracy of DTE for the dataset \(r\geq 0.8\) even though the uncertainty is still high for the dataset \(r=0.8\). Figure 6 compares the feature rank obtained by feature independence analysis using chi-square tests with the feature importance score of DTE. We first evaluate the independence of each feature variable to the abuse classes by using chi-square tests. Figure 6a shows the feature scores, defined as the negative logarithm of the \(p\)-values. The seven most important features of the feature independence analysis are HeatLossRate, NegEnP, CellEn, CellCap, BodyEnP, CorrLossEn, and ConfPos. The ratio of these seven scores over the total score sum is around 0.38 in Figure 6a. We further analyze the feature importance score of DTE, which is defined as the total changes in the risk due to splits on each feature divided by the number of branch nodes of the decision tree, as shown in Figure 6b [31]. Figure 5: Average and standard deviation of prediction accuracy of DTE with different dataset ratios (\(r\)) compared to the maximum available dataset \(N=364\). Note that the minimum importance score is 0. The weighted average is then applied to the feature importance score of the overall weak learners of the ensemble. HeatLossRate is the most crucial feature, followed by PreCellM and NegEnP. The ratio of the score sum of the three most important features of HeatLossRate, PreCellM, and NegEnP over the total score sum is 0.44. We observe that HeatLossRate and NegEnP are still critical factors even for the trained DTE model. Figure 7 presents three abuse classes, Heat, ISC, and Nail, against the two most critical features of DTE, namely, HeatLossRate and PreCellM, obtained from Figure 6b. The Nail class is relatively easy to separate from other classes of Heat and ISC. The Nail class has the lowest HeatLossRate, while the boundary with other classes increases as the PreCellM value increases. On the other hand, the Heat class shows the highest variation in HeatLossRate, particularly for PreCellM\(\simeq\) 50. The classification boundaries between Heat and ISC classes are apparent when HeatLossRate is in the range between 0.03 and 0.08 for PreCellM\(\simeq\) 50. Thus, this explains the dominant classification errors between Heat and ISC classes of different models, as shown in the confusion matrix of Figure 3.  
Partial dependence represents the relationships between feature variables and predicted scores (posterior probabilities) of all abuse methods obtained by the trained DTE model [31]. Figure 8 shows the estimated partial dependence of DTE for all abuse classes against the most critical features, HeatLossRate. According to DTE, the probability of the Heat class significantly changes around HeatLossRate\(\simeq\) 0.02, then stays almost flat. The probability of Nail is greater than 0.97 for HeatLossRate\(<\) 0.02, while it drops fast to almost 0.65 for HeatLossRate\(\geq\) 0.02.