## 4 Model Evaluation  
### On-Track Validation and Comparison of ML Based Models  
Five models (LR, KNN, RF, FFNN, and CNN) are trained with the training data based on the aforementioned four different sets of input data structures (0D-allvar, 0D-seectvar, 2D-allvar, and 2D-selectvar). The training results are tested on the testing data (i.e., collocated CALIOP data). The overall accuracies of each model-variable combination after being aggregated over land, ocean, or the whole globe are compared in Figure 3.  
A comparison between Figure 3a,b clearly reveals that all models perform better over the ocean compared to land. This is expected as the land surfaces make complicated lower boundary conditions compared to the ocean. Reflections by ocean surface are generally very small and thus the proportion of satellite received radiance that is from the atmospheric particles are larger. Another reason is that the radiative characteristics of the dust can be similar to those from semi-arid land surfaces, which makes it difficult to distinguish dust as the surface signals dominate the atmospheric signals. For both 0D and 2D cases, the allVar case tended to outperform or showed similar performance with selectVar in FFNN. This shows that the advantage of using ANN is that it can handle the large predictor variables itself without feature selection. This is similar to RF and CNN as well. FFNN outperformed CNN for about 1.7% accuracy. This shows that utilizing spatial characteristics of \(5\times 5\) VIIIRS pixel window did not contribute to the model, compared to simply flattening the \(5\times 5\) window. Generally, CNN is best used for the image data, which usually consists hundreds or thousands of pixels. In that context, in future studies, a bigger window size could be used to improve the performance of CNN, since \(5\times 5\) window might be too small for CNN to perform efficiently. For each method, the best performing feature cases are selected and compared for a statistical significance of performance. With a \(t\)-test, FFNN outperformed other methods at the 95% confidence level. CNN and RF did not show significant performance difference between them at the 95% confidence level; however, both outperformed KNN and LR. KNN also significantly outperformed LR. As seen in the figure, the best model-predictor data combination is FFNN using the 2D-allvar, with overall accuracy of 84.99%. Thus, hereafter, we only analyze results from FFNN with a 2D-allvar predictor variable. Figure 4 shows the global map of dust fraction derived from CALIOP (Figure 4a) and the collocated on-track VIIRS observations using the FFNN method (Figure 4b). As expected, FFNN is fairly good at resembling the spatial pattern of dust in CALIOP. CALIOP and FFNN present a 30% to 50% dust occurring fraction in central Asia, North America, Australia, South America, and at the tip of South Africa, which are the regions that are  
Figure 3: Accuracy of five different ML based models with different structures of predictor variables. Accuracy is calculated as total number of correct predictions (both clear sky and dust) divided by total number of test dataset. Accuracy plotted on top of each bar represents the highest accuracy among the different structures of predictor variables, for each of the methods. (**a**) Land; (**b**) Ocean; (**c**) Land + Ocean. known as dust emission sources. However, CALIOP reports almost a 100% dust fraction in the Antarctic and adjacent Southern Ocean. As explained in Section 2, this is mostly misidentification due to retrieval artifacts. Inevitably, the FFNN method inherited this flaw from CALIOP and also reports high dust fractions in these regions. Nevertheless, it is evident from the figure that FFNN is able to closely reproduce the dust fraction map from CALIOP, with up to a 15% difference in dust distribution (Figure 4c), depending on the location.