## 4 Simulations
### Verification of Validity of Factor Combination Search Algorithm  
Having demonstrated that the trained NC-RBF-DNN is able to derive reasonable importance rankings for all input fields, we next assessed the benefits of the proposed factor combination algorithm in the construction of ensemble effective models based on this ranking. The verification process was implemented into three stages: (1) an examination of the number of combinations that the heap needs to check to find the optimal solution and a comparison, (2) a comparison of the results obtained by the heap and those obtained by the NC-RBF-DNN, and (3) a discussion of the factor combinations used in the optimal solution obtained by the heap. We use Figure 14 to explore the first part. The \(x\) axis in this figure indicates the number of factors in a combination; the solid curve in the figure shows the number of times the heap needs to find a near-optimal solution with combinations containing \(n\) factors, and the dashed curve shows the number of combinations with \(n\) factors to be found with the 16 factors in our target problem. Due to the massive differences between the values of these two curves, we used a logarithmic scale for the \(y\) axis. In Figure 14, we can see that our heap only needs fewer than 10 times to find a near-optimal solution. This is clearly far fewer than the possible number of combinations, thereby demonstrating that the proposed approach can achieve the goal with very little computation. We next use Figure 15 to look at the prediction errors resulting from the top several combinations in the heap with different number of factors in a combination. The values on the \(x\) axis represent the number of factors in a combination, and the multiple dots  
Figure 14: Number of combinations that need to be checked for near-optimal solution: (**a**) modeling with lightweight NC-RBF-DNN; (**b**) modeling with random forest. Figure 13: Errors resulting from modeling with top \(n\) factors. above each value on the \(x\) axis indicate the errors resulting from the top 10 combinations that popped out in the heap with \(n\) combinations. Finally, the solid and dashed curves in Figure 15 show the errors generated by the minimum value derived using the heap combination algorithm and the top \(n\) combinations obtained using the NC-RBF-DNN. We can observe two phenomena in this figure. The first is that whether we use the lightweight NC-RBF-DNN or the random forest, the minimum errors resulting from the combinations identified using the heap search algorithm were all less than those of modeling with the top several factors identified by the NC-RBF-DNN. This demonstrates the validity of the proposed algorithm. The second phenomenon is that the errors in the results from the top several combinations that popped out in the heap were all lower than those from the ranking derived by the NC-RBF-DNN. This also shows that if we uses the top several combinations identified using the factor combination search algorithm to achieve the ensemble computation for robust prediction, the computation errors of the ensemble model would almost always be lower than or close to those of modeling using the ranking obtained by the NC-RBF-DNN, which also demonstrates that our ensembling results are no poorer than only using a single solution. Finally, we use Figure 16 to look at the factors that the heap method chose with different numbers of factors in the combinations. The vertical values represent the number of factors in the combinations, and the horizontal values indicate which of the four groups ranked by the NC-RBF-DNN the factors selected by the heap fall in. A deeper color in the grid indicates a higher number of factors selected. From the two sub-graphs in Figure 16, we found that whether the lightweight NC-RBF-DNN or the random forest was used, the heap method chose very few factors from the two first groups; instead, it chose the most factors from the third group and the second most from the fourth group.