# A Dilemma-Based Learning-to-Rank Approach for Generative Design in Urban Architectural Regeneration
## 3 Experimental Setup
### Experimental Results
#### 3.2.3 Image Ranking Analysis  
The learning-to-rank algorithm was applied to evaluate the AIGI-generated images by solving the dilemma-based learning problem. For this case study, we used RankingSVM, SVC, NuSVC, and XGBoost models to assess performance. The dataset for training and validation was created by asking an architect to choose the best image from all possible pairwise combinations of the generated images from the first test case. By considering the selection in reverse order as identical, the final dataset for training the semi-supervised learning-to-rank algorithms consisted of 2352 image pairs. The test accuracy scores for each model at different training set ratios are shown in Figure 9. RankingSVM had the lowest performance, with an average accuracy of \(0.63\pm 0.02\), achieving its maximum accuracy at a 60% train set ratio. The SVC model performed slightly better with an average accuracy of \(0.67\pm 0.01\), with its highest accuracy at a 70% train set ratio. NuSVC outperformed the other SVM-based models with an average accuracy of \(0.71\pm 0.02\), peaking at a 50% train set ratio. XGBoost achieved the best results, with an average accuracy of \(0.75\pm 0.05\), reaching its highest accuracy at an 80% train set ratio. Table 4 presents the precision scores for each method across different training set ratios for the test sample. Comparing the average scores, RankingSVM showed the weakest performance, while XGBoost achieved the best results. Specifically, RankingSVM recorded its highest precision at the 60% training set ratio, and SVC performed best at the 70% ratio. Both NuSVC and XGBoost attained their highest precision scores at the 80% training set ratio. The recall scores are shown in Table 5. Consistent with accuracy and precision, RankingSVM yielded the lowest average recall score of 0.65, with its highest recall reaching 0.70 at the 60% training set ratio. Both SVC and NuSVC achieved similar average recall scores of 0.72. However, SVC performed better at lower training set ratios, with its highest score occurring at the 20% ratio, while NuSVC excelled at higher training set ratios, reaching its peak recall at 80%. XGBoost achieved the best performance overall, with an average recall score of 0.78, and its highest score at the 80% training set ratio. Finally, the F1-scores are summarized in Table 6. RankingSVM had an average F1-score of 0.63, with its highest score at the 6% training set ratio. SVC achieved an average F1-score of 0.69, peaking at the 70% training set ratio. Both NuSVC and XGBoost recorded their highest F1-scores at the 80% training set ratio, with averages of 0.71 and 0.77, respectively. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Test Case** & **Veget. Cov.** & **Water El.** & **Park Amen.** & **Feasib.** & **Spatial Align.** \\ \hline
1 & 47/49 (96\%) & 9/49 (18\%) & 34/49 (69\%) & 16/49 (33\%) & 17/49 (35\%) \\ \hline
2 & 46/49 (94\%) & 16/49 (33\%) & 24/49 (49\%) & 23/49 (47\%) & 29/49 (59\%) \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Visual analysis of the data based on the architectural criteria. Given the nature of the data and the task of applying these algorithms to rank the dataset, the evaluation metrics demonstrate that all the selected learning-to-rank algorithms can be effectively used for sorting the image solutions. It is also important to note that achieving a perfect ranking is not necessarily critical. The evaluation of the best image solution is subjective, with multiple images potentially being rated as equally optimal for different reasons. The proposed ranking system aimed to assist the architect by accelerating the manual evaluation process, highlighting the N-Best images based on their selections during the dilemma-based learning phase. Figure 10 illustrates the five best images per method for each data sample. It is worth noting that most of the five best images appear in all algorithms either in the same or different ordering positions.