CoMO-NAS: Core-Structures-Guided Multi-Objective Neural Architecture Search for Multi-Modal Classification

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 16:21:16

Abstract: Most existing NAS-based multi-modal classification (MMC-NAS) methods are optimized using the classification accuracy. They can not simultaneously provide multiple models with diverse perferences such as model complex and classification performance for meeting different users' demands.Combining NAS-MMC with multi-objective optimization is a nature way for this issue. However, the challenge problem of this solution is the high computation cost. For multi-objective optimization, the computing bottleneck is pareto front search. Some higher-quality MMC models (namely core structures, CSs) consisting of high-quality features and fusion operators are easier to identify. We find that CSs have a close relation with the pareto front (PF), i.e., the individuals lying in PF contain the CSs. Based on the finding, we propose an efficient multi-objective neural architecture search for multi-modal classification by applying CSs to guide the PF search (CoMO-NAS). In conclusion, experimental results thoroughly demonstrate the effectiveness of our CoMO-NAS. Compared to state-of-the-art competitors on benchmark multi-modal tasks, we achieve comparable performance with lower model complexity in shorter search time.

Main paper: 
 introduction: 
 The success of multi-modal fusion architectures owes much to their design [7, 16, 32] . Recently, with the advantages of escaping labor-intensive and challenging architectural design processes, neural architecture search (NAS) [18] has experienced unprecedented interest. NAS has achieved significant success in discovering optimized multi-modal feature fusion strategies, surpassing manually designed methods [14, 15, 24, 39] . However, these approaches focus solely on the need for high-accuracy fusion architectures. In addition to accurate predictions, practical applications also require NAS-MMC methods to find computationally efficient network architectures, such as low power consumption in mobile applications, low latency in autonomous driving applications, and deployability on edge devices like smartphones. It is widely observed that as network architecture complexity increases, predictive performance continues to improve. This has sparked competition between maximizing predictive performance while minimizing complexity, leading to the natural idea of introducing multi-objective optimization.Among the many diverse MMC-NAS methods, evolutionary algorithms (EAs) have garnered widespread attention due to their population-based nature and flexibility [44] . They offer a viable alternative to traditional ML-oriented approaches, especially within the scope of multi-objective NAS. Generally, EAs involve an iterative process where improvements are gradually made to individuals within the population by applying variations to selected individuals and recombining parts of multiple individuals. Despite being easily scalable to handle multiple objectives, most existing EAbased NAS methods are still single-objective driven. Additionally, a computational bottleneck for utilizing evolutionary algorithms in multi-objective optimization lies in the search for the Pareto frontier, requiring significant computational resources.To address the aforementioned issues, we propose a method called core structures-guided multi-objective neural architecture search (CoMO-NAS). As illustrated in Figure 1 , core structures are substructures composed of high-performing features and fusion operators, which often play a decisive role in the architecture's performance. These core structures align with the optimization objectives of multi-objectives, representing lower-complexity and high-performance substructures themselves. Additionally, through observations of existing advanced MMC-NAS methods, we find that the optimal fusion architectures identified by MMC-NAS typically include some excellent core structures. Building upon these findings, we further explore and discover that core structures frequently appear in the final Pareto frontier, and even more complex solutions within the Pareto frontier also contain core structures. Therefore, we propose utilizing core structures to guide multi-objective neural architecture search. To obtain core structures, we first divide the features extracted by the backbone network and predefined fusion operators into two parts: one part contains individually high-performing features and fusion operators constituting the core structure search space, while the other part contains lowerperforming features and fusion operators constituting the non-core structure search space. In the first stage, we utilize multi-objective evolutionary algorithms to search for core structures within the core structure search space, obtaining a portion of the Pareto frontier. In the second stage, we use the core structures along with the non-core structure search space to search for architectures with higher complexity and better performance, forming a complete Pareto frontier.Our research has been validated on multiple multi-modal datasets, showcasing optimal performance in terms of efficiency, complexity, and accuracy. Specifically, our contributions include:• We find that the core structures (CSs) that consist of features and fusion operators with higher performance in NAS-MMC can be used to guide the Pareto front (PF) search in multiobjective optimization because the individuals in PF often contain the CSs. This strategy is able to significantly improve search efficiency and solution quality. • With above the finding, we propose a method called core structure-guided multi-objective neural architecture search (CoMO-NAS). To the best our knowledge, CoMO-NAS introduces the concept of multi-objective algorithms in the field of MMC-NAS for the first time. It can efficiently provide multiple optimization solutions for different scenarios. • We conducted extensive experimental comparisons on multiple multi-modal tasks, and the results show that compared to state-of-the-art multi-modal feature fusion methods, CoMO-NAS has significant advantages in terms of search time and the number of model parameters. 
related work 2.1 multi-modal fusion: 
 Multi-modal fusion networks have demonstrated clear advantages over single-modal networks in various applications such as action recognition and sentiment analysis [9, 17, 33, 35, 43] . However, effectively combining multi-modal features to better utilize information remains a significant challenge [36] . multi-modal fusion techniques are typically divided into two main categories: the first category involves handcrafted fusion based on domain knowledge, such as early fusion, fusion of low-level features, late fusion, and fusion of decision-level features [16] . Some approaches also involve feature fusion at intermediate layers to facilitate later fusion and improve performance, for example, CentralNet [31] and MMTM [30] , which connect potential representations from each layer and pass them as auxiliary information to deeper layers. However, this approach significantly increases the parameter count of multi-modal fusion models. Additionally, there are recent works proposing dynamic multi-modal fusion, a novel method that adaptively fuses multi-modal data and generates data-relevant forward paths during inference [7, 19, 23, 34, 46] . The second category involves applying neural architecture search to automatically find the optimal fusion architecture [15] . Compared to the first category, this approach eliminates the laborious traditional handcrafted design and generally achieves better fusion results, albeit at the expense of requiring substantial computational resources and time.Neural Architecture Search (NAS) [18] has been introduced to automate the design of neural models, aiming to discover efficient architectures with competitive performance. This trend has sparked researchers' interest in migrating NAS to the field of multi-modal fusion, leading to the proposal of a series of multi-modal neural architecture search methods to automatically design optimal fusion network architectures. Perez-Rua et al. first explored and validated the feasibility of using NAS methods to address this issue. They proposed a search framework called MFAS [25] , which automatically selects single-modal features from all candidate features as inputs to the fusion module. However, due to MFAS's adoption of the black-box optimization algorithm SMBO, each update step requires training a set of DNNs, resulting in low efficiency. Additionally, MFAS only utilizes concatenation and fully connected (FC) layers for single-modal feature fusion, where the stack of FC layers poses a significant computational burden. Yin et al [39] . introduced a two-layer gradient-based search scheme named BM-NAS, allowing simultaneous search of input features and fusion operations for each multi-modal fusion module. However, it forces cells to have different predecessors, leading to a narrowed search space and potentially suboptimal results. EDF [15] utilizes evolutionary neural architecture search to find multi-modal fusion architectures for chemical structures, achieving superior results. However, due to the inherent limitations of evolutionary algorithms, the time cost is high. To address the time cost issue brought by evolutionary NAS, DC-NAS [14] proposes a divide-and-conquer evolutionary neural architecture method, greatly improving search efficiency.While the aforementioned methods have achieved significant success in the field of multimodal fusion, they are all driven by the goal of performance optimization. This may lead to a tendency to solely pursue performance during the search process, consequently increasing the complexity of the models. In addition to the high demand for accuracy, practical applications also require NAS-MMC methods to discover network architectures that are computationally efficient, catering to scenarios with low power consumption and memory constraints. To address this issue, we propose a multiobjective multimodal neural architecture search framework guided by core structures. This framework utilizes evolutionary algorithms to compensate for the shortcomings of existing methods. 
definition and motivation: 
 To avoid confusion, we provide precise definitions for certain terms here. A population consists of individuals, where each individual corresponds to a multi-modal classification (MMC) model encoded in the form of a tree, typically represented as a vector in post-order traversal for ease of displaying experimental results in this paper. All representations extracted from different modalities are collectively referred to as features. The space composed of high-quality features Feature set:Fusion operator set: 3 86.71% 6 [8, 7, 4, 3, 1, 5, -3, -0, -0, -1, -1] 11 88.57% and fusion operators is termed as the core structure search space, while the remaining constitutes the non-core structure search space.⋮ ... 2 = -1 2 = -Our work involves two key concepts: the Pareto frontier and core structures, which are closely intertwined. The core structures is a submodule that plays a central role in multi-modal fusion architectures. The performance of most fusion architectures stems from those core structures, typically composed of high-performing modality features and fusion operations, as depicted in Figure 1 . Core structures are usually architectures with good performance and low complexity, thus they often appear on the Pareto frontier and both of them exits close relationship. By leveraging core structures, we can identify partial solutions within the Pareto frontier. However, relying solely on core structures may not be sufficient when considering both good performance and complexity. Nevertheless, since these high-performing and complex architectures often contain core structures, identifying core structures first can guide the search for high-performing and complex architectures, forming a complete Pareto frontier to meet various scenario demands. Additionally, utilizing core structures to guide the search process can significantly accelerate the search for the optimal Pareto frontier by drastically reducing the evaluation of numerous ineffective architectures, thus saving a considerable amount of time.To provide a more intuitive understanding of the relationship between the two concepts, we conducted experiments on the CB dataset. First, we employed traditional multi-objective evolutionary algorithms. Next, we utilized core structures to guide the search across the entire PF. The specific algorithm steps are outlined in the following sections. The results are presented in Table 1 . From the experimental results, we can draw the following conclusions: (1) in terms of the HV [4] indicator, CoMO-NAS discovers a better Pareto frontier compared to traditional multi-objective evolutionary algorithms. Here, a higher HV indicator indicates better performance;(2) We observe that the Pareto frontier of CoMO-NAS includes core structures with IDs 4 and 5, and most other solutions are guided by core structures. For example, IDs 1 and 6.In this paper, we propose a core structure-guided multi-objective neural architecture search for finding the optimal Pareto front. CoMO-NAS consists of three steps: (1) unimodal feature extraction, (2) multi-objective core structures search (MOCSS) and (3) core structures-guided optimal Pareto frontier search. The main framework of CoMO-NAS is shown in Figure 2 .Feature Extraction. This study follows previous work on multi-modal fusion, such as MFAS, MMTM, and BM-NAS, using pre-trained single-model neural network models as feature extractors. We extract raw features from the intermediate layers of these models because neural network architectures typically have a layered or block-like structure, which naturally suits this extraction method. As the feature extractors used for different modalities vary, resulting in significant differences in the dimensions of the raw features-for example, text modalities may yield one-dimensional features, images two-dimensional, and videos three-dimensional-we employ global average pooling to uniformly convert them into feature vectors. This facilitates feature alignment, promotes subsequent feature fusion, and simultaneously reduces computational complexity for ease of processing.Multi-Objective Core Structures Search. To ensure the efficiency and convenience of vectorized feature fusion within the entire architecture, we employ five basic fusion operators for feature fusion. Here, we define 𝑥 𝑖 as the vector feature, 𝑛 denotes the number of vector features to be fused, and the superscript indicates the index of the fused vector feature. In this context, the fusion operator set 𝐹 encompasses the following operations:(1) Concatenation: The information from vector features is fused as follows:EQUATIONwhere [•, •] is the concatenation operator. Element-wise fusion operators require that the dimensions of input vectors are the same, hence different vector features need to be mapped into the same dimension space by a linear function before fusing. This can be achieved using a fully-connected layer (FC) without any activation function.(2) Addition: The information from vector features is fused as follows:EQUATION(3) Multiplication: The information from vector features is fused as follows:EQUATIONwhere • denotes Hadamard product, namely element-wise multiplication.(4) Max: The information from vector features is fused as follows:EQUATIONwhere max is element-wise max, also called max-pooling.(5) Average: The information from vector features is fused as follows:EQUATIONwhere + denotes element-wise addition, also called average-pooling.To obtain core structures, we can reduce the entire search space to the core structure search space by evaluating the performance of each feature and fusion operator at relatively low cost. Specifically, given 𝑛 features represented as 𝑀 1 , 𝑀 2 , ..., 𝑀 𝑛 , and a single-modal classifier 𝑓 , we pass each feature 𝑀 𝑖 to 𝑓 and select the top 𝑘 1 features with higher performance to form a high-quality feature set 𝑀 1 . Given 𝑚 fusion operators represented as 𝐹 1 , 𝐹 2 , ..., 𝐹 𝑚 , and a multi-modal classifier ℎ, we obtain the classification performance of each fusion operator 𝐹 𝑖 by replacing the fusion manner of ℎ with 𝐹 𝑖 and select the top 𝑘 2 fusion operators with higher performance to form a high-quality fusion operator set 𝐹 1 . The space composed of 𝑀 1 and 𝐹 1 is referred to as the core structure search space. Next, in the core structure search space, we use a multi-objective evolutionary algorithm for searching. After 𝑁 1 iterations of MOENAS, we obtain the core structure population and partially obtain the Pareto frontier solutions during the iteration process. Please refer to Section 3.3 for specific details about the MOENAS.Core structures-guided optimal Pareto frontier search. The core structure have a close relationship with the final Pareto frontier obtained through multi-objective search. This is because the core structures are characterized by low complexity and high precision. The final Pareto frontier typically includes some core structures, but to obtain a comprehensive optimal Pareto frontier, we need to integrate the remaining features and fusion operators. This is because the remaining features and fusion operators often provide complementary information, enhancing the overall performance of the fusion architecture. In the previous stage, we already obtained the core structures, and some more precise multimodal fusion architectures usually include the core structures as well. Therefore, we can quickly determine a high-quality search subspace by leveraging these core structures, which consists of the neighborhood surrounding the core structure. Here, the neighborhood refers to the continuous addition of substructures composed of the remaining features 𝑀 2 and the fusion operator set 𝐹 2 to the core structure, forming a multi-modal fusion architecture. To obtain the final Pareto frontier, we utilize the MOENAS algorithm, taking the core structure along with the remaining features 𝑀 2 and 𝐹 2 as new inputs. Through evolutionary algorithms, we can adaptively evaluate the fusion architectures around each core structure, achieving the goal of searching the neighborhood and ultimately obtaining the complete Pareto frontier.For both stages, we employ the NSGA-II algorithm to search for the core structure and the final Pareto frontier. The overall algorithm is presented in Algorithm 1.(1) Individual Encoding and Decoding: To cover various fusion strategies more flexibly, each individual in the population is encoded as a binary tree, capable of encompassing any fusion strategy. In this representation, leaf nodes represent modality features, while internal nodes represent fusion operators. Due to the inherent nature of binary trees, when there are k features, there must be k-1 fusion operators. For the decoding process, each individual corresponds to a multi-modal classification model. The binary tree can be decoded into a multi-modal classification model through the following steps: 1) Channeling modality features represented by the leaf nodes of the individual encoding tree into fully connected layers (FC) for feature alignment, facilitating feature fusion, and adding batch normalization (BN) layers to enhance model convergence speed and improve generalization; 2) Conducting feature fusion based on the fusion operators represented by the internal nodes; 3) Directing the fused features through FC and Softmax layers for the final prediction output.(2) Population Initialization: For the CoMO-NAS framework, there are two search stages, each with a different initialization approach. In the first stage, 𝐾 non-repeating individuals are generated in a random distribution within the core structure search subspace. 𝑂 𝑗 = 𝑃 ′ 𝑗 ∪ 𝑄 𝑗 ; 16:𝑃 ′ 𝑗+1 ← Get the next generation population from 𝑂 𝑗 using the non-dominated sorting method in NSGA-II; 17: end for 18: Return The optimal Pareto front population Specifically, 𝐾 unique features are randomly sampled from the feature pool, along with 𝑘 -1 fusion operators, and randomly combined into binary tree structures. This process is repeated 𝐾 times to create the initial population 𝑃 0 . In the second stage, the 𝐾 core structures obtained from the first stage are utilized as the initialization population 𝑃 𝑁 1 .(3) Individual Evaluation: In CoMO-NAS, the network architecture and weight parameters corresponding to each individual encoding are first obtained on the training data. Subsequently, the accuracy and complexity of the network are calculated based on the validation data, serving as the two objectives for individual evaluation to facilitate subsequent multi-objective algorithms.Objective1-Accuracy or Weighted F1 score: The first objective shown in Equations 6 is to maximize accuracy or weighted F1 score. The choice between accuracy and weighted F1 score as the first optimization objective depends on the characteristics of the dataset. When dealing with highly imbalanced data, we use the weighted F1 score for better evaluation, which is consistent with previous methods.EQUATIONWhere 𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 and 𝐹 1 𝑤𝑒𝑖𝑔ℎ𝑡𝑒𝑑 represent the functions computed for the individual 𝑝. Ojective2-Complexity: The second objective, as shown in Equation 7, is to minimize the complexity of each individual. Here, complexity refers to the length of the architecture, which is the sum of the number of features and fusion operators. Our goal is to achieve higher performance by using fewer features and fusion operators, while also effectively reducing the parameters and redundancy in the architecture. Existing methods often face an issue of maximizing performance by incorporating as many features and fusion operators as possible, leading to redundancy. The final fusion architectures they search for are often extensive, with a portion of them potentially yielding only marginal benefits. MoCo-NAS partially mitigates this problem by reducing the complexity of the model. min 𝑓 L (𝑝) = Features(𝑝) + Fusion operators(𝑝) (7) (4) Evolutionary Strategy: Using a reproduction method based on crossover and mutation, offspring populations are generated, as outlined in Algorithm 2. Initially, two pairing individuals are selected from the current population 𝑃 𝑔 through binary tournament selection. Subsequently, reproduction involves duplication, crossover, and mutation operations to yield offspring individuals. The entire algorithm comprises two stages: the first stage involves searching for core structures and partial Pareto frontiers, while the second stage utilizes core structures to explore the neighborhoods around them, aiming to achieve the complete optimal Pareto front. Different crossover and mutation rates are applied for distinct objectives. During the core structure search, following traditional evolutionary algorithm principles, the probability of crossover operation significantly surpasses that of mutation. The objective here is to explore core structures, where mutation generates a substructure from the core structure search space, substituting a specific substructure within the individual. Conversely, for exploring the neighborhoods around core structures to attain the entire Pareto front, opposite crossover and mutation rates are adopted. Utilizing an extremely low crossover rate and very high mutation rate aims at exploring the neighborhoods around core structures, resulting in a high probability of mutation. Mutation generates a substructure from the non-core structure space and incorporates it into the individual to explore the neighborhoods around core structures. The low crossover rate aims to prevent disruption of the core structures themselves. 
experiments 4.1 experimental settings: 
 In our experiments, all methods are implemented using TensorFlow 2.0.3. Our computational environment consistes of Ubuntu 16.04.4 with 16GB GPU memory, 512GB DDR4 RDIMM, 2X 40-Core Intel Xeon CPU E5-2698 v4 @ 2.20GHz, and NVIDIA Tesla P100. It is worth noting that the GPU configuration used in this paper is the same as the MFAS, EDF, and DC-NAS architectures.(1) Parameter settings: a) Training of DNNs: All deep neural network models are trained using the Adam algorithm. The learning rate is set to 0.001, with a first-moment exponential decay rate of 0.9 and a second-moment exponential decay rate of 0.999. Each network undergoes training for 100 epochs. To prevent overfitting, if the performance of a multi-modal neural network model does not improve after 10 epochs, the training process will be halted. b) CoMO-NAS: To efficiently utilize GPU resources, the population size is set to a multiple of the number of GPUs. We employed seven NVIDIA Tesla P100 GPUs for the CB dataset, NUS dataset, NTU RGB-D dataset, and EgoGesture dataset , with a population size of 28. The number of iterations is set to 10, with 6 iterations for core structure search and 4 iterations for searching the local region of core structures. During the core structure search phase, the crossover rate is 0.9, and the mutation rate is 0.2. During the search for the local region of core structures, the crossover rate is 0.1, and the mutation rate is 0.8. Considering that the MM-IMDB dataset is relatively simpler compared to the first two tasks, we used four NVIDIA Tesla P100 GPUs with a population size of 20. The rest of the settings are consistent with the above datasets.(2) Evaluation metrics: We utilize accuracy as the evaluation metric on CB, NUS, NTU RGB-D, and EgoGesture datasets, where higher values indicate better performance. On the MM-IMDB dataset, we employ F1-W as the evaluation metric, also aiming for higher values. Additionally, we use the Hypervolume (HV) to measure the quality of the final Pareto front obtained by the CoMO-NAS algorithm, validating the effectiveness of the core-guided search method compared to traditional approaches. It is important to note that larger HV values correspond to better algorithm performance.We validated five popular multi-modal datasets: (1) ChemBook-10k (CB) [15] dataset, designed for chemical structure image recognition in patent retrieval studies, which contains 100,000 chemical structure images distributed into 10,000 categories. (2) NUS-WIDE-128 (NUS) [29] dataset, which contains 43,800 images divided into 128 categories. We chose a subset of 10 categories totalling 23,438 images from this dataset. (3) MM-IMDB [1] dataset for the multi-label film genre classification task, which contains a total of 23 categories. To validate the effectiveness and efficiency of the proposed algorithm, we selected several state-of-the-art algorithms and compared them with CoMO-NAS. These peer competitors can be broadly categorized based on whether the architecture is manually designed. The first category is MMC whose fusion architectures are designed by human experts, including MBL [11] , MFB [40] , TFN [42] , LMF [21] , PTP [8] , TMC [7] , TMOA [20] , AWDR [37] , RAMC [10] , Maxout MLP [5] , VGG Transfer [28] , Two-stream [27] , GMU [1] , Cen-tralNet [31] , Inflated ResNet-50 [2] , Co-occurrence [13] , MMTM [30] , VGG-16 + LSTM [38] , C3D + LSTM + RSTTM [22] , I3D [3] , ResNext-101 [12] , and MTUT [6] . The second category is NASbased MMC methods including EDF [15] , MFAS [25] , BM-NAS [39] , 3D-CDC-NAS2 [41] , and DC-NAS [14] .Results on CB and NUS. To reduce the influence of variability stemming from data partitioning and network initialization, we partitioned each dataset uniformly into training and testing subsets. More precisely, instances from every category were randomly distributed, allocating 80% for training and 20% for testing. All methodologies were evaluated under identical data partitioning conditions. The experiments were iterated five times for each approach using consistent configurations, and the average performance, alongside standard deviation, was reported.According to the aforementioned algorithm, we first select highquality features and fusion operations to form the core structure . Subsequently, we search for core structures and utilize local algorithms based on these core structures to explore the entire Pareto frontier. To comprehensively showcase the advancements of MMC-NAS, we followed the experimental settings of EDF. We compared MMC-NAS with some advanced multi-modal fusion operators and existing sophisticated multi-modal fusion methods. From the results in Table 2 , it's evident that, compared to advanced fusion operators, we achieved a significant lead by employing basic fusion operators along with our search strategy. Among multi-modal methods, except for EDF and DC-NAS [14] , all others are non-NAS methods.Clearly, the performance of MMC-NAS surpasses manual selection. By utilizing core structures to guide the search across the entire Pareto frontier, even when balancing model complexity and accuracy, we can achieve performance comparable to state-of-the-art single-objective methods like EDF and DC-NAS [14] .Results on MM-IMDB. To ensure fair comparison with other explicitly multimodal fusion approaches, we adopted the same neural network backbone models as BM-NAS and DC-NAS to extract various modality features, using weighted F1 score as the evaluation metric. The specific parameter settings are as follows: population Results on NTU. To ensure the fairness of the experimental results, we followed the data preprocessing pipelines of BM-NAS and DC-NAS. Specifically, we used Inflated ResNet-50 [2] and Cooccurrence [13] as feature extractors for the skeleton and video modalities. For the CoMO-NAS evolutionary algorithm parameters, due to our approach of searching the Pareto frontier from the perspective of core structures, which narrows down the search space, the required population size and number of iterations are smaller than those of traditional evolutionary algorithms. For instance, while state-of-the-art DC-NAS may require 15 generations of population, we only need 10 generations to discover a Pareto frontier solution that matches the performance of DC-NAS. We set the population size to 28, the number of iterations to 10, the fusion modality dimension to 64, and allowed for the reuse of modality features. In Table 4 , our method exceeds most baseline methods while achieving comparable performance with the state-of-the-art DC-NAS, ensuring both model complexity and performance objectives.Results on Ego. We followed the methods of BM-NAS and DC-NAS, using ResNeXt-101 [12] as the backbone network for RGB and depth video modalities. CoMO-NAS was compared with various single-modal and multi-modal methods in terms of performance. The experimental settings for CoMO-NAS included a population size of 28, 15 iterations, reusable modality features, and a fusion dimension of 32. The experimental results on the EgoGesture dataset are presented in Table 5 . Compared to other unimodal/multimodal methods, CoMO-NAS achieved fusion performance comparable to the state-of-the-art method DC-NAS. This section aims to compare CoMO-NAS with several powerful MMC baseline methods, including MFAS, EDF, BM-NAS, DC-NAS, and MMTM [30] , focusing on search efficiency, complexity, model size, and performance to demonstrate its advanced capabilities. The research results have been comprehensively summarized in Table 6 . From the table, it can be observed that on five complex datasets, CoMO-NAS is able to find architectures with comparable performance but lower complexity and smaller model size compared to other methods, while leading in efficiency. For example, on the NUS and CB datasets, our efficiency is nearly four times that of the EDF method and twice that of DC-NAS, while the model complexity is halved. On the NTU RGB-D and EgoGesture datasets, while achieving comparable performance, we gain a significant advantage in model complexity. The time consumption for searching the optimal fusion model is reduced by almost six times compared to the latest method of BM-NAS, and by half compared to DC-NAS. This is attributed to our core structure-guided multi-objective neural architecture search framework, which significantly narrows down the search space, effectively avoids evaluating a large number of poorly performing models, and imposes multi-objective constraints on model complexity, thereby greatly reducing model redundancy and accelerating search speed. To provide a more in-depth analysis of the proposed CoMO-NAS, we conducted a detailed examination of each component and hyperparameter of CoMO-NAS via the ablation experiments on the CB dataset which is the most complex among the five datasets.Analysis of the Impact of MOCSS and CSG-OPFS Stages on CoMO-NAS: To further investigate the impact of MOCSS and CSG-OPFS on CoMO-NAS, we conducted a comprehensive analysis of three scenarios of CoMO-NAS. According to the results in Table 7 , we draw the following conclusions: compared to searching the entire space, utilizing core structures to guide multi-objective neural architecture search can lead to finding architectures with lower complexity but comparable performance in a shorter time. For example, the time was reduced from 62.70 hours to 38.15 hours, shortening the duration by 24.55 hours. From the results of CoMO-NAS 2 compared to CoMO-NAS, when only using the core structure search space without expanding the surrounding local space, the overall performance of the searched architectures declined. This is because low-quality features and fusion operators within the local space can also complement each other, thereby enhancing the overall performance of the architecture.Analysis of Core Structure Selection Strategies: To investigate the impact of core structure selection on subsequent Pareto frontier exploration, four experiments were conducted. In the first experiment, high-quality features and fusion operations were employed to search for core structures, while the subsequent four experiments involved the random selection of features and fusion operations for core structure exploration. The experimental results presented in the Table 8 unequivocally demonstrate that employing high-quality features and fusion operations for core structure search leads to significantly superior outcomes compared to randomly selecting features and fusion operations. 
conclusion: 
 In this paper, we has proposed a multi-objective neural architecture search method guided by core structures to address the limitations of existing MMC-NAS methods, which had focused solely on achieving high performance while ignoring the varying demands of different applications for classification performance and model size. Furthermore, it has resolved the issue of model redundancy that has arisen from pursuing high performance in existing MMC-NAS methods. By establishing the relationship between core structures and the Pareto frontier and utilizing core structures to guide the search across the entire Pareto frontier, the method has avoided evaluating numerous ineffective architectures, thereby significantly improving search efficiency. Extensive experiments has validated the advantages of CoMO-NAS. 
 
 Appendix:  


Decision: The paper aims to tackle the problem that most existing NAS-based multi-modal classification (MMC-NAS) methods are optimized using the classification accuracy so it can not meet different users' demands. Based on the finding that Core Structures have a close relation with the pareto front (PF), they propose an efficient multi-objective neural architecture search for multi-modal classification by applying CSs to guide the PF search (CoMO-NAS). Experimental results thoroughly demonstrate the effectiveness of our CoMO-NAS. Compared to SOTA benchmarks, the proposed method achieve comparable performance with lower model complexity in shorter search time.

The paper is well-written and easy to understand. The motivation is clear and the proposed method contains signifiant novelty. The reviewers post some questions on the paper writing and numeric problem. The rebuttal has addressed almost all the problem.

One reviewer g2n4 concerns the significant of the novelty while the reviewer address the novelty at the initial reviewer comments. After checking the paper itself, I suggest that the paper is based on interesting observation and the method performs well in both efficient and metrics. And the method will be public which will contribute to the field a lot.

Above all, I suggest to accept this paper.

Reviews: review: paper_summary: paper_summary: This paper proposes a core structure-guided multi-objective neural architecture search method based on a interesting finding to efficiently provide multiple optimization solutions for different scenarios. Sufficient experimental evaluations are provided.
strength_weakness: strength_weakness: 1.The paper summarized a comprehensive review of the current progress of related work and pointed out the limitations. 
2.The method has achieved significant improvements in experiments. 
3.This paper is well organized and written.
limitations: limitations: Some concerns and suggestions to improve this work are as follows:
1.While the methodology is interesting, the description of the framework lacks clarity and Figure 2 would benefit from explanatory symbols such as arrows. Please revise to improve reader understanding and ensure clear presentation of methodological processes.
2.The author conducted extensive experimental comparisons on multiple multi-modal tasks. Could you please give some qualitative analysis to explain this? 
3.It is recommended that the authors check any potential citation errors and enhance the manuscript's overall quality.
paper_summary: paper_summary: The paper titled "CoMO-NAS: Core-Structures-Guided Multi-Objective Neural Architecture Search for Multi-Modal Classification" presents a novel approach to Neural Architecture Search (NAS) that is specifically tailored for multi-modal classification tasks. The authors introduce a method that combines NAS with multi-objective optimization to address the limitations of existing NAS methods, which often focus solely on classification accuracy while neglecting other important factors such as model complexity and computational efficiency. The key innovation of CoMO-NAS is the use of high-quality substructures, termed "core structures" (CSs), to guide the search for the Pareto front, which represents a trade-off between model performance and complexity. The authors claim that their method achieves comparable performance with lower model complexity and in a shorter search time compared to state-of-the-art methods.
strength_weakness: strength_weakness: Novelty: The paper introduces a new concept of using core structures to guide the search in multi-objective optimization for NAS, which is an innovative approach to balancing the trade-off between model performance and complexity.

Theoretical Approach: The theoretical foundation for using core structures to approximate the Pareto front is well-reasoned and aligns with the optimization objectives of multi-objective problems.

Technical Correctness: The methodology is technically sound, with a clear explanation of how core structures are identified and used to guide the search process.

Adequate Evaluation: The authors have conducted extensive experiments on multiple multi-modal datasets, which demonstrates the effectiveness of CoMO-NAS in terms of efficiency, complexity, and accuracy.

Clarity: The paper is well-written and clearly explains the motivation, methodology, and results. The use of figures and tables enhances the reader's understanding of the proposed method.

Applications: The paper discusses the practical applications of NAS in various domains such as mobile applications, autonomous driving, and edge devices, which highlights the relevance of the research.
limitations: limitations: Generalization of Core Structures: The paper does not sufficiently address how the identified core structures generalize across different datasets or modalities. It is crucial to understand if these core structures are dataset-specific or if they have broader applicability.

Exploration of Search Space: While the core structure-guided search is presented as efficient, it may potentially limit the exploration of the entire search space. There is a risk that the method might miss out on other optimal or near-optimal architectures that do not contain the identified core structures.

Sensitivity Analysis: The paper lacks a sensitivity analysis of the hyperparameters used in the CoMO-NAS algorithm. Understanding how changes in these parameters affect the outcome is essential for users to effectively apply the method.

Documentation and Code Availability: The paper does not mention whether the code will be made publicly available. Open-source code would greatly enhance the paper's impact by allowing the community to build upon and further refine the proposed method.

Specificity of Methodology to Multi-Modal Classification: While the paper focuses on multi-modal classification, it is not clear how well the proposed method could be adapted to other types of tasks, such as regression or clustering.
paper_summary: paper_summary: The article discusses the limitations of existing neural architecture search (NAS)-based multi-modal classification (MMC-NAS) methods, pointing out that they typically optimize only for classification accuracy and cannot simultaneously provide diverse models that meet different user demands, such as model complexity and classification performance. Combining NAS-MMC with multi-objective optimization is a natural solution to this issue, but the main challenge lies in the high computational cost, particularly in the Pareto front (PF) search process.
The study finds that some high-quality MMC models, known as core structures (CSs), which consist of high-quality features and fusion operators, are easier to identify. These core structures have a close relationship with the Pareto front, meaning that individuals on the Pareto front contain the core structures. Based on this finding, the authors propose an efficient multi-objective neural architecture search method for multi-modal classification (CoMO-NAS), which uses core structures to guide the Pareto front search. Experimental results thoroughly demonstrate the effectiveness of CoMO-NAS. Compared to state-of-the-art competitors on benchmark multi-modal tasks, this method achieves comparable performance with lower model complexity and shorter search time.
strength_weakness: strength_weakness: I believe this article demonstrates significant novelty, as it introduces the concept of multi-objective optimization algorithms into the field of Multimodal Classification Neural Architecture Search (MMC-NAS) for the first time. By guiding Pareto Front (PF) searches through Core Structures (CSs), a new multi-objective neural architecture search method (CoMO-NAS) is proposed. In terms of technical correctness, the article, through observation and experimentation, reveals the important role of core structures in high-performance multimodal fusion architectures and elucidates the relationship between these structures and the Pareto Front. The CoMO-NAS method improves search efficiency and solution quality by incorporating core structures of high-performance features and fusion operators into the multi-objective multimodal neural architecture search process. To validate the correctness of the CoMO-NAS method, the article presents extensive experimental comparisons across multiple multimodal tasks, demonstrating CoMO-NAS's superior performance in terms of efficiency, complexity, and accuracy. The experimental results indicate that compared to state-of-the-art multimodal feature fusion methods, CoMO-NAS offers significant advantages in search time and the number of model parameters.
limitations: limitations: The article does not discuss the limitations of the proposed method or suggest directions for future work. Should we consider extending the CoMO-NAS architecture presented in this paper to other fields beyond multimodal classification tasks in the future?

There are some minor issues in the paper, such as in Table 2 where "CoMO-NAS" is missing "(ours)" and should be consistent with the rest of the table.
paper_summary: paper_summary: Combining NAS-MMC with multi-objective optimization faces the challenge of high computation cost. The authors propose a multi-objective neural architecture search for multi-modal classification by applying constraint satisfaction (CS) to guide the Pareto front (PF) search (CoMO-NAS). The authors claim that the proposed model has lower complexity and shorter search time.
strength_weakness: strength_weakness: The problem addressed in this paper is of significant value, and the authors' approach is quite innovative.
limitations: limitations: 1. The abstract and contribution sections of the paper need further enhancement. The current presentation makes it difficult to highlight the main contributions of the paper. The authors should clearly and explicitly state the key differences between the proposed method and the existing state-of-the-art methods to emphasize the paper's contributions.
2. The METHODS section needs reorganization, as the current presentation is somewhat difficult to follow. The authors introduce a new architecture, but they need to clearly explain why this architecture can reduce the existing computational cost. This explanation should include theoretical analysis as well as experimental validation and analysis.
3. The steps of the proposed method are numerous, but many steps are not clearly explained. For instance, in section 3.2.2, the feature fusion process involves five basic fusion operators. The authors should explain the purpose and rationale behind using these operators. Similarly, many other steps need to be elucidated to make the paper more coherent, centered around the main theme, and more readable.