## 2 Materials and Methods
### Experiment
#### 2.3.1 Feature Selection and Aggregation for Dual-Phase FBB  
In our experiment, we used ranking-based feature selection that individually evaluated each feature and sorted those features based on the evaluated scores [16]. We adopted a one-way \(F\)-test [17] and the Gini score [18] estimated by random forest to measure the quality of the feature subset. We scored each numerical regional SUVR with the \(p\)-Value from the one-way \(F\)-test and Gini scores in the training set. Subsequently, we evaluated the performance of the feature subset created by cumulatively adding individual features sorted according to their scores using five-fold cross validation in the training set. Finally, the top feature combinations selected from each phase of the FBB were aggregated as follows:  
\[C_{i,j}=F_{e,i}\ \cup\ F_{d,j}\ (i,j=1,2,\ldots,10). \tag{1}\]  
Here, \(F_{e,i}\) and \(F_{d,j}\) are the top feature combinations for each phase of the FBB; \(C\) is the feature set aggregated from a dual FBB and is used to extract the input variables for the predictive models. That is, \(C\) includes \(i+j\) feature combinations, and these combinations are used to create predictive models. In the comparison model with only a single-phase FBB (single FBB), without aggregating \(C\), either \(F_{e,i}\) or \(F_{d,j}\) is used to make the respective predictive models.  
Figure 4: Overview of the proposed framework to make the feasibility of AD classification with dual FBB. eFBB: early-phase FBB, dFBB: delay-phase FBB, dual FBB: dual-phase FBB, CV: cross validation, \(F_{e,i}\): best feature combination of eFBB, \(F_{d,j}\): best feature combination of dFBB. \(F_{e,i}\) and \(F_{d,j}\) are used to filter raw data, and the filtered data are used as independent variables for predictive models in the model selection and evaluation.  
2.3.2. Evaluation for Classification Model and Selected Feature Distribution  
As shown in Figure 4, in this experiment, we attempted to show improved patient classification performance of dual FBB compared to single FBB and the brain regions contributing to the classification task through our proposed framework. Therefore, we observed three types of validation items using only the test set.  
First, the validation of the patient classification performance of the trained predictive model was considered as follows: representative metrics were used to measure the classification performance. A weighted F1 score was used to overcome data imbalance issues that make a classifier biased toward a majority class [15]. We used the weighted F1 score to validate the quality of the selected feature combinations and evaluated the generalization performance of the predictive model for classifying the AD patient group and the control group.  
\[Accuracy=\frac{TP+TN}{TP+FP+TN+FN} \tag{2}\]  
\[\text{Weighted F1 score}=\frac{1}{\sum_{l\in L}|y_{l}|}\sum_{l\in L}\Bigl{(}1+ \beta^{2}\Bigr{)}\frac{Precision*Recall}{\beta^{2}Precision+Recall} \tag{3}\]  
In the model evaluation, we calculated the area under the curve (AUC) score as well. The receiver operating characteristic (ROC) curve is a graph showing the performance of the classification model at all classification thresholds. The area under the ROC curve, AUC, is used when the distribution for each class is different [19].  
Second, we considered the validation of the best feature set submitted by the proposed framework from the training set. Because we iteratively experimented with different seeds, we can simply present the frequencies of the regions suggested by the framework with a histogram. The histogram represents the importance of cortical brain regions in patient classification. We can observe the importance of each cortical region in eFBB imaging, which has rarely been covered in previous studies, or in dFBB imaging.