# Multi-Objective Evolutionary Neural Architecture Search with Weight-Sharing Supernet
## 5 Experiments
### Search Performance Comparison on CIFAR-10 Dataset
#### 5.1.2 Experimental Results
It can be observed that \(A_{1}\) tends to search for small models (with model size \(<\) 0.24 M), while \(A_{2}\) is devoted to exploring large models (with model size \(>\) 0.24 M). However, the two sub-populations are not entirely isolated; both sub-populations are able to explore the middle region of the search space. Thus, it can be inferred that, through the communication stage, the exchanged individuals may be optimized towards the new reference point in the other sub-population, which enables the search in the unexplored area between the original sub-populations. Figure 8 gives the training and validation metrics obtained during the whole search phase. For weight optimization, it can be seen from Figure 7(a) that, during the entire process, the training accuracy gradually increases while the loss decreases. Both of them stabilize at around 500 epochs. For the architecture optimization, Figure 7(b) visualizes the information of the architecture population in each period. It can be observed that, as the update periods progress, the average validation accuracy increases, and the fluctuation of the average model size gradually decreases. Specifically, the average model sizes in the later periods remain close to the median model size in the entire search space, rather than prematurely converging to a small and limited region as shown in Figure 6(c). Figure 8: The training and validation metrics during the search phase of bi-population MOEA/D algorithm (with communication). To further demonstrate the effectiveness of our proposed algorithm, we also analyze and compare some Quality Indicators (QIs) between the bi-population MOEA/D and CARS algorithms. First, we focus on the diversity of the final population by computing the average distance to the two closest neighbors of each individual. Then, the average distance to the ideal point (which is set as (0, 0)) of each individual is calculated to further measure the quality of the Pareto fronts. In these experiments, our algorithm achieves two metric values of 0.3868 and 0.1195, while CARS achieves 0.1482 and 0.1046. It can be concluded that the two methods obtain Pareto fronts with comparable quality, and ours is capable of searching individuals with higher diversity. Finally, we compare the metrics in the training process between the CARS algorithm and the bi-population MOEA/D algorithm. As for the search time, the main costs for both algorithms lie in two parts. The first part is the weight optimization, with a total of 500 epochs throughout the entire search phase. The second part is the architecture optimization, with the population to be evaluated in each period. For both the improved NSGA-II algorithm used in CARS and the bi-population MOEA/D algorithm designed in this study, 2 \(\times\)\(N_{\text{pop}}\) individuals are required to be evaluated in each period. Under the settings in Table 1, the average search time of both algorithms falls within the range of 46,000-47,000 s, approximately 13 GPU hours (\(\approx\)0.54 GPU days). Through the comparison of the search performance on CIFAR-10 between the multi-objective evolutionary NAS methods, it can be concluded that the small model trap indeed exists in the DARTS search space. Using traditional NSGA-II or MOEA/D algorithms will lead to premature convergence, reducing the diversity of the population and neglecting the large models in the search space. Further experiments validate the effectiveness of the bi-population MOEA/D algorithm. By setting different sub-populations and different optimization directions, the small model trap can be effectively addressed. The introduced communication mechanism allows the algorithm to fully explore and develop in a complex and large search space, which is capable of obtaining well-distributed and high-performing network architectures.