Sequence Alignment-based Similarity Metric in Evolutionary Neural Architecture Search

Authors: mateo ávila pava
rené  groh
andreas m kist
Venue: automl_cc_AutoML_2024_Conference
Published: 2024-02-28 11:19:19

Abstract: Neural Architecture Search (NAS) has emerged as a powerful method for automating the design of deep neural networks across diverse applications, with evolutionary optimization showing particular promise in addressing its intricate demands. However, the effectiveness of this approach highly depends on balancing exploration and exploitation, ensuring that the search does not prematurely converge to suboptimal solutions while still achieving near-optimal outcomes. This paper addresses this challenge by proposing a novel similarity metric inspired by global sequence alignment from biology. Unlike most of the existing methods that require pre-trained models for comparison, our metric operates directly on neural network architectures within the defined search space, eliminating the need for model training. We outline the computation of the normalized similarity metric and demonstrate its application in quantifying diversity within populations in evolutionary NAS. Experimental results conducted on popular datasets for image classification, such as CIFAR-10, CIFAR-100, and ImageNet16-120, show the effectiveness of our approach in guiding diversity based on our suggested control function. Additionally, we highlight the usefulness of our similarity metric in comparing individuals to identify advantageous or disadvantageous architectural design choices. The code is available at \url{https://anon-github.automl.cc/r/evonas_similarity_metric-1BD3/}.

Main paper: 
 introduction: 
 The field of artificial intelligence (AI) has recently made significant progress, primarily due to the increasing sophistication of neural networks. With the rapid emergence of diverse architectural design choices (Szegedy et al., 2015; He et al., 2016; Chollet, 2017) , the process of crafting neural network architectures is becoming more and more complex. This complexity has made manual design approaches less practical, emphasizing the need for automated methods, commonly referred to as Neural Architecture Search (NAS). NAS involves exploring a vast space of possible network structures and optimizing for performance metrics such as accuracy, efficiency, and computational cost to ultimately select the most effective architecture for a given task or dataset (Elsken et al., 2019) . Besides reinforcement learning (Tan et al., 2019; Zoph and Le, 2016) , bayesian optimization (Jin et al., 2019; Kandasamy et al., 2018; White et al., 2021) or gradient-based optimization (Liu et al., 2018) , genetic optimization through evolutionary algorithms (EAs) is also commonly used for NAS.This meta-heuristic is inspired by natural selection, uses principles like survival of the fittest, and iteratively improves solutions to complex problems, such as NAS (Simon, 2013; Liu et al., 2021) . However, one huge limitation of evolutionary NAS algorithms is the constant trade-off between exploration and exploitation when optimizing a neural architecture for a given problem (Tan et al., 2009) . In the following, we refer to an individual as an individual in a population in terms of genetic optimization, which reflects in this case a single neural network architecture.In this work, we present a novel neural architecture similarity measure inspired by the sequence alignment of biological sequences, which we use to control diversity within evolutionary NAS (see Figure 1 ). Our main contributions are:• A novel similarity measure facilitating the comparison of candidate neural network architectures sampled from a search space in the context of NAS.• Demonstrating the ability of the similarity measure to balance between exploitation and exploration.• We show that our similarity measure can effectively identify unfavorable design choices by comparing pairs of individuals with high similarity but substantial differences in fitness. This allows gaining knowledge of neural network architectural design or refining the search space. Diversity 
similarity of neural network models: 
 Understanding and quantifying the similarity of neural networks presents a complex challenge due to the diverse perspectives on their comparability. A recent survey categorizes nearly 50 measures (Klabunde et al., 2023) into Functional Similarity Measures (FSM) and Representational Similarity Measures (RSM). FSM compares networks based on output behaviors, such as ModelDiff (Shah et al., 2023) and Disagreement (Madani et al., 2004) . RSM, including PWCCA (Morcos et al., 2018) and CKA (Kornblith et al., 2019) , analyzes activations across layers. Additionally, it includes metrics like Aligned Cosine Similarity (Hamilton et al., 2016) and Geometry Score (Khrulkov and Oseledets, 2018) . Both FSM and RSM are essential, as similar outputs may stem from different representations, and significant dissimilarity in output should be reflected in RSM. Choosing the most appropriate similarity measure, one that aligns closely with the specific requirements of your problem is crucial for accurate assessment.The effectiveness of evolutionary algorithms (EAs) can be affected by two main factors: (i) the loss of diversity in early generations, which results in limited exploration, and (ii) high mutation rates, which can undermine the effectiveness of the recombination operator by introducing excessive randomness and hindering the convergence of solutions (Bhattacharya, 2014) . To address these challenges by directly or indirectly controlling diversity, various strategies have been proposed. These include multinational evolutionary algorithms (Ursem, 1999) , crowding techniques (Mengshoel and Goldberg, 2008) , mass extinction models (Krink and Thomsen, 2001) , and Dynamic convergence-diversity guided EA (Li et al., 2018) , among others. By providing guidance and exerting greater control over diversity, EAs can balance exploration and exploitation, improving their ability to navigate complex problems and achieve reliable outcomes.DNA, RNA, and protein sequences encode specific downstream functionalities (Wilson et al., 2018) . For example, a natural protein sequence consists of a combination of the 20 canonical amino acids (e.g., glycine, serine, ...). The main assumption is that proteins with similar sequences have similar functions. This means that by comparing and aligning two protein sequences, one can infer evolutionary conserved regions or functional motifs. Similarly, two or more DNA and RNA sequences can be aligned with each other, which is important for the arrangement of nextgeneration sequencing data (Li and Homer, 2010) . A well-known algorithmic solution is the Clustal family (Chenna et al., 2003; Sievers et al., 2011) , which also serves as the inspiration for this work. 
proposed similarity metric based on sequence alignment: 
 Global sequence alignment, exemplified by the Needleman-Wunsch (NW) algorithm (Needleman and Wunsch, 1970) , systematically compares entire sequences like DNA, RNA, or proteins to reveal evolutionary relationships and conserved regions. This dynamic programming technique evaluates all possible alignments, assigning scores to matches, mismatches, and gaps, ultimately identifying the alignment with maximum similarity.In this light, we propose employing sequence alignment to compare individual sequences representing neural networks within the EA's population. By applying this approach globally to entire populations, we aim to identify and uncover effective design patterns, while also gaining insights into the evolution of population diversity across generations. To the best of our knowledge, this is the first time that sequence alignment is used to compare the similarity between neural networks.While distinct from Cell-based approaches like NASNet (Zoph et al., 2018) or DARTS (Liu et al., 2018) , our search space falls within the Chain-Structured category (White et al., 2023) . It offers flexibility for easy modifications due to its sequential architectural topology. Comprising primitive layers and hand-crafted blocks such as BottleneckBlock and ResidualBlock from ResNets (He et al., 2016) , our search space includes feature extraction (F) and fully connected dense (D) layers. Each neural network sample consists of 1-20 F layers chosen from 9 predefined options, and 0-2 D layers, including Dropout and Dense layers. A randomly selected Global pooling layer, chosen among Flatten, Global Average Pooling, and Global Max Pooling, facilitates feature aggregation between the F and D blocks. The macrostructure of individuals and available layers for each block, along with their parameters, is depicted in Figure 2 .To employ sequence alignment, our models must first be represented as sequences, with each character corresponding to a distinct layer or block in our search space. Each block or layer is assigned a unique character, as detailed in Table 1 . See Figure 3 for the conversion of network model architecture to a sequence. Neural networks, being more complex than simple sequences, require consideration of layerspecific properties in similarity measures. Unlike traditional sequence alignment methods, we propose a customized matching function that accounts for differences in layer types and associated parameters.In our matching-mismatching function, we evaluate layer similarity based on parameters. For example, comparing a Conv2D layer (C) with a ReLU layer (L) results in a mismatch value of -2, indicating incomparability. However, comparing a Conv2D layer (C) with a DepthwiseConv2D layer (D) allows for assigning a value reflecting their operational similarity, ranging from 0 to 2. We establish comparison criteria for each layer type, weighting parameters accordingly. This approach facilitates penalization or prioritization of alignment, accommodating specific block alignments (e.g., ResBlock, BottlenetBlock) over others (e.g., Dropout, InstanceNormalization). The penalty for introducing gaps or extending a sequence remains constant at -0.5.Custom matching functions were defined for the layers in our gene pool, enabling us to determine the absolute global alignment score between pairs of individuals. However, direct comparisons between individuals with different chromosome sizes are hindered by the absolute nature of the alignment scores, which are influenced by the number of genes in the sequences. To address this, we normalize the global alignment to obtain a relative alignment score between sequences 𝑆 1 and 𝑆 2 . A score of 1 signifies identical sequences, while a score of 0 indicates complete dissimilarity or no possible alignment. The normalization process is based on the maximum possible alignment value, which is determined by aligning 𝑆 1 with itself and 𝑆 2 with itself. The lowest alignment score occurs when no elements align and gaps equal to the sequence length are introduced. With our gap penalty of 0.5 being lower than the mismatch penalty of 2, the lowest alignment score is given by the number of introduced gaps multiplied by -0.5. Equation 1shows the computation of this similarity, and illustrative calculations are provided in Table 3 . The penalty for introducing gaps or extending a sequence is consistently set at -0.5. This relatively low value, compared to the mismatch penalty, allows for increased flexibility in aligning larger models with smaller ones, facilitating the discovery of potential local alignments. MatchScore(𝑋 , 𝑌 , 𝛼, 𝛽 ) = 𝛼 𝑋 𝛽 -𝑌 𝛽 𝑚𝑎𝑥 (𝛽 ) -𝑚𝑖𝑛 (𝛽 ) if Gene 𝐺 1 == 𝐶 && Gene 𝐺 2 == 𝐶 Weight Max match score: 3 𝑚 = MatchScore(𝐺 1 , 𝐺 2 , 𝛼 𝑓 , 𝑓 ) 𝛼 𝑓 = 0.𝑆 1 = CDBgO, 𝑆 2 = CADBGO S1 C - D B g O S2 C A D B G O alignment: Φ(𝑆 1 , 𝑆 2 ) = 8.8 sim(𝑆 1 , 𝑆 2 , 𝜌 ) = 8.8-(-3) 14.5-(-3) sim(𝑆 1 , 𝑆 2 , 𝜌 ) = 0.674Φ max (𝑆 1 , 𝑆 2 ) = max (Φ(𝑆 1 , 𝑆 1 ), Φ(𝑆 2 , 𝑆 2 )) Φ min (𝑆 1 , 𝑆 2 ) = 𝜌 • max (len(𝑆 1 ), len(𝑆 2 )) sim(𝑆 1 , 𝑆 2 , 𝜌) = Φ -Φ min Φ max -Φ min (1)Leveraging the relative similarity formula, we can numerically compare all individuals within the population even before they are trained. This process generates a square similarity matrix of size 𝑁 × 𝑁 , where N represents the population size. The diagonal of this matrix corresponds to the self-alignment of each element, resulting in values equal to 1. This similarity matrix serves as a valuable tool for monitoring and understanding the evolution of the population over generations. Additionally, it enables the identification of design patterns, clusters of similar individuals, and potential novel architectures. This information aids in diagnosing errors that may arise during the algorithm's execution, such as premature convergence or gene convergence. Figure 5 illustrates similarity measure matrices for an exemplary EA run on the ImageNet16-120 dataset. Individuals are sorted by model size in MB.  
diversity control in evolutionary nas: 
 The similarity matrix allows assessing the diversity within a population and tracking its evolution across generations. A straightforward approach involves calculating the median of all off-diagonal elements, providing a global measure of population diversity (𝑠 𝑔𝑒𝑛 ) for each generation 𝑔𝑒𝑛, Where a value of 1 denotes no diversity and a value of 0 signifies maximum diversity. The evolution of diversity is influenced by factors beyond the search space, including the composition of the initial population, the choice of the crossover operator, and the mutation rate. We hypothesize that incorporating diversity guidance boosts EA achieved optimal fitness and accelerates convergence, eliminating the necessity for extensive tuning of all operators and parameters while minimizing resource consumption or working with limited population sizes.To achieve a balanced Exploration-Exploitation equilibrium, we present a population diversity control strategy utilizing our similarity metric. This strategy iteratively generates populations using standard operators until the median similarity difference (Δ𝑠) between generations falls within the limits set by our control function. This mechanism seamlessly guides the breeding process within the evolutionary algorithm, ensuring convergence without requiring extensive parameter tuning (Figure 1 ). The proposed "Rigid" control function for the allowed Δ𝑠 is defined mathematically (Eq. 2). Our aim is to moderate diversity reduction while avoiding stagnation or slow growth. Upon reaching half of the total generations (𝑁 𝑔𝑒𝑛2 ), we restrict abrupt fluctuations in diversity to facilitate fine-tuning of local optima and ensure algorithm convergence.EQUATIONInitially, we conducted testing and implemented rigid control, which can be changed to accommodate other user-defined functions to guide diversity. The outlined process is encapsulated within the Breeding algorithm, as depicted in the pseudo-code presented in Algorithm 1. 
experiments and results: 
 This paper aims to demonstrate the versatility of our similarity metric, highlighting its effectiveness in evolutionary algorithms for diversity control and in identifying patterns and clusters within datasets. We conducted an evolutionary neural architecture search across three image classification datasets: CIFAR-10, CIFAR-100, and ImageNet16-120. ImageNet16-120, a downscaled variant of ImageNet, contains images resized to 16×16 pixels and labels ranging from 1 to 120, as defined in NAS-BENCH201 (Dong and Yang, 2020) . This downscaled version reduces computational costs while serving as a valuable benchmark (Chrabaszcz et al., 2017) . Our experiments involved ten different seeds and utilized separate training and evaluation sets for each dataset. CIFAR-10 and CIFAR-100 comprised 50k training images and 10k test images, while ImageNet16-120 included 151k training images and 6k test images. For training each individual neural network architecture, we utilized Stochastic Gradient Descent with a Nesterov momentum rate set to 0.9. The learning rate followed a cosine decay schedule, starting from 0.1 and tapering to 0, with a weight decay parameter of 0.0005. We used a batch size of 256 and trained for 12 epochs. In our evolutionary NAS framework, we progressed through 20 generations with a population size of 25. The top 5 models from each generation were selected based on performance. We applied a mutation rate of 20% and imposed architectural constraints limiting the number of feature layers to 20 and dense layers to a maximum of 2. Furthermore, we followed established methodologies from benchmark NAS-BENCH-201 (Dong and Yang, 2020) and its extension NATS-BENCH (Dong et al., 2020) for selecting data augmentation techniques and configuring hyperparameters. All searches were conducted using an NVIDIA A100 GPU, and due to resource constraints inherent in NAS algorithms, we limited our search to 25 individuals per generation.Two sets of experiments were conducted for each dataset to assess the impact of our diversity control strategy: one without any control (referred to as "Naive") and another incorporating our rigid control function ("Diversity Control"). The combined results for ImageNet16-120 across all seeds are depicted in Figure 6 , presenting maximum fitness scores and diversity values across generations.Additional results for CIFAR-10 and CIFAR-100 can be found in Appendix B, while results for individual seeds are detailed in Appendix C. For CIFAR-100, diversity control tests achieved a 0.6% higher fitness score compared to the Naive tests. Notably, diversity control led to higher fitness scores in most generations, except for generations 8 and 17. Similarly, for ImageNet16-120, a 1% improvement was observed with diversity control, and the best fitness scores occurred between generations 10 and 20. For CIFAR-10, only a 0.4% improvement was achieved with diversity control compared to the Naive method.Our diversity control function impacted all datasets, facilitating algorithm convergence and focusing on the best individuals near local optima during exploitation. For ImageNet16-120, diversity peaked at 0.75, while for CIFAR-100 and CIFAR-10, it reached 0.68 and 0.69, respectively. In contrast, without diversity control, diversity fluctuated between 0.55 and 0.64 for both datasets from the fifth generation onwards. Notably, with diversity control, diversity exceeded 0.8 for some seeds in ImageNet16-120 and CIFAR-10, indicating excessively small differences among individuals and negatively impacting algorithm performance. Thus, our diversity function could be enhanced by considering both the changes in diversity between consecutive generations (Δ𝑠) and the absolute diversity value for each generation (𝑠 𝑔𝑒𝑛 ).Particularly significant is the comparison of models with the highest fitness for each seed, visualized as unaligned sequences in Figure 7 . The Alignment chart reveals insights into gene (layer) distribution and prevailing consensus for each test. Notably, the BottleneckBlock with skip connection (red) and the DepthwiseConv (green) are prevalent among the best solutions, contributing to model efficiency and lightweight design. For CIFAR-100, the Naive test shows only three out of ten individuals with more than 13 genes, contrasting with eight out of ten individuals in the Diversity control test. In ImageNet16-120, both Naive and Diversity control scenarios predominantly yield individuals with fewer than 13 genes. However, Diversity control generates individuals with a higher occurrence of R (ResidualBlock) and T (BottleneckBlock) blocks, suggesting a tendency toward exploring more complex solutions. This trend is reflected in the sizes of their models, as detailed in Appendix D. The clustergram in Figure 8 further organizes these individuals hierarchically into similarity clusters, revealing insights into convergence patterns. Notably, for CIFAR-100 and CIFAR-10 datasets, solutions from both Naive and Diversity control methods exhibit significant disparities. Most regions show similarities below 0.5, with few instances reaching 0.6 to 0.7, suggesting diverse local optima discovery. For ImageNet16-120, Naive testing reveals a clustered solution set with similarities of 0.55 to 0.65, while Diversity control identifies a larger cluster (five individuals) with similarities of 0.6 to 0.82. Models are more similar due to repeated BottleneckBlocks occurrence. Figure 9 represents the frequency of occurrence of each gene or block of 2 or more genes and was created based on the sequence representation depicted in Figure 7 . Notably, the six genes with the highest number of repetitions are consistent across all datasets. In terms of blocks comprising 2 or more genes, the top three most frequently repeated blocks are TD, DB, and TT, observed across both datasets. Interestingly, CIFAR-100 exhibits a greater diversity of 2-gene combinations compared to ImageNet16-120. Additionally, the primary 3-gene blocks observed are TDB, TTD, and TDT. These findings underscore the significance of these layers in efficient image classification models such as MobileNet, ResNet, and DenseNet, among others. An essential application of our similarity metric, particularly evident through our similarity matrix, lies in identifying outliers and instances where there exists significant similarity between individuals but considerable disparity in fitness scores. This allows us to pinpoint which genes and their respective positions contribute positively or negatively to achieving a desirable fitness score. In Appendix F, we present some intriguing design choices, showcasing all three tested datasets. 
conclusion and future work: 
 In this study, we introduced a novel similarity metric based on bioinformatics sequence alignment techniques to compare neural network architectures within a defined search space. This metric effectively measured population diversity and was integrated into an evolutionary NAS framework to balance exploration and exploitation. Our proposed rigid control function efficiently reduced diversity and exploited different local optima across datasets. The analysis of the best individuals highlighted the significant impact of specific genes, such as DepthwiseConvolution (D) and Bot-tleneckBlock (T), on fitness scores. Moreover, we identified design cases with high similarity but notable differences in fitness scores, often involving the insertion or removal of BatchNormalization layers. Our findings are in line with recent works, where the use and selection of normalization layers were found substantial in model performance (Neubig and Kist, 2023) .However, it's crucial to acknowledge that due to computational constraints, we conducted limited experiments with 10 initial populations and the rigid control function on three datasets. Additionally, our search space doesn't provide insights into the global optimum, and evaluating all possible individuals within it is computationally expensive. Future work could involve testing with different population sizes and further exploration of different control functions as well as testing our metric in other chain-based search spaces. One limitation of our method is its limited applicability to cell-based architectures. It is better suited for chain-based or sequential architectures since our primary objective was to develop an intuitive and efficient method that facilitates straightforward comparisons among simple and sequential architectures. In future studies, the search space should also consider non-trainable layers (such as activation and pooling) to have a fixed order, as these are typically sequential but freely interchangeable, which could otherwise potentially reduce the performance of our proposed method.As a next step, it would be valuable to extend our method to accommodate tree-like or graphlike architectures by adopting a more general approach, such as utilizing the Graph Edit Distance. Instead of solely defining matching functions as we did, we would incorporate insertion, deletion, and substitution functions. However, this would inevitably increase the complexity of the problem. Consequently, combining sequence alignment for the macrostructure or simple genes (operations) with Graph Edit Distance for the cell-based blocks commonly employed in NAS benchmarks would enhance the versatility of our method in the future. 
broader impact statement: 
 Our research aims to enhance Neural Architecture Search (NAS) efficiency by improving convergence speed and optimization quality. While our method represents an advancement in the field, it requires intensive computational resources due to its exhaustive search approach. To address this, integrating our proposed similarity metric with more resource-efficient methodologies, like few-shot or single-shot NAS approaches, is a promising future direction. This aligns with our approach of comparing neural architectures without model training, offering potential sustainability benefits by reducing the environmental footprint associated with high computational demands.Zoph, B. and Le, Q. V. ( 2016). Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578.Zoph, B., Vasudevan, V., Shlens, J., and Le, Q. V. ( 2018). Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8697-8710. 
 
 Appendix: 1. For all authors. . . We confined our main work to a single specific control function due to the significant computational resources required to test various seeds with each function. However, different functions can be proposed depending on the desired approach to diversity control. An alternative function to prevent excessively restricting population diversity increase is to implement a linear functions-based on the total number of generations (𝑁 𝑔𝑒𝑛 ), ensuring that the lower bound function 𝑔(𝑔𝑒𝑛) does not remain constant at zero throughout all generations. This method represents a softer or more flexible control strategy and is defined in equation 3.EQUATIONBoth rigid and soft control functions are depicted in Figure 10 . This analysis was conducted exclusively with CIFAR10, revealing no significant difference in fitness scores between the naive and soft control strategies. Notably, rigid control demonstrated a slight performance advantage, as illustrated in Figure 11 . To ensure the robustness of our findings and the reliability of our metric across diverse initial populations, we conducted Evolutionary NAS using 10 distinct initial seeds. The fitness accuracy and similarity, both with and without rigid diversity control, are depicted for all three datasets in Figures 14, 15 and 16. We calculate the Pearson correlation coefficient between the maximum accuracy achieved per generation and the median diversity per generation across all three tested datasets and for each of the 10 seeds. Additionally, we compute the average correlation for each case and observe that the correlation is higher when utilizing our diversity control method compared to the baseline method. Alongside the fitness score, we recorded the size and inference time per image of the best solutions across generations for the three datasets, as shown in Figures 17 and 18 . We also wanted to test how similar the solutions were between the Naive strategy and the one with Diversity control. Therefore, we represented this in the clustergram shown in Figure 19 . F Examples of cases with significant similarity but considerable disparity in fitness scores between individualsFrom the similarity matrices of each test, we selected 12 noteworthy cases and presented in Figure 20 . The most frequent modifications between individuals include the deletion or insertion of a BatchNormalization(B) gene, as well as the substitution of BatchNormalization (B) with InstanceNormalization(I). In other cases, such as examples 1 and 6, where the similarity reaches values exceeding 0.97, the changes no longer occur between different types of layers or architectural structures, as they remain the same. Instead, the variations occur in the parameters associated with certain genes. In instances with more significant differences in similarity, we observe substitutions involving two or more genes, such as in example 2, where the BAB block is replaced by an InstanceNormalization (I) layer, or example 7, where the order of the D and C layers is changed. Therefore, leveraging Figure 9 and examples from Figure 20 , we can potentially narrow down or eliminate certain gene options within the search space or introduce additional rules to mitigate the generation of design patterns that consistently yield suboptimal results. AutoML 2024© 2024 the authors, released under CC BY 4.0 


Decision: The paper adapts sequence-alignment algorithms from biology and uses them as a similarity metric to compare neural architectures. The metric is useful for controlling the diversity of the architectures, and post hoc analysis was done to identify interesting patterns. 

I am dismissing the entire review from NV25 since evidence from many parts of the review suggests that it was generated with an LLM. 

The other reviewers lean towards accepting the paper. I agree with the reviewers and recommend acceptance of the paper.

Reviewer DuGc identified some limitations of the work, and the author's rebuttal addressed some of those through additional experiments. I strongly recommend that the authors incorporate the discussion of these limitations and the additional supporting experiments into the paper. This will significantly enhance the paper's quality.

Reviews: review: paper_summary: paper_summary: To better balance exploration and exploitation in NAS, this paper proposed a sequence representation of neural architectures associated with a sequence machining function to measure the similarity of neural architectures. Based on the sequence representation and matching functions, a genetic algorithm is proposed to control the diversity of the population.
main_review: main_review: This paper proposed a sequence presentation of neural architectures and a sequence-matching function to measure their similarities. Based on it, a diversity control evolutionary algorithm is proposed to improve exploration and exploitation. This paper should be improved from the following aspects. 

1. To represent a neural architecture as a sequence is not a good choice. Such architecture coding is not representative. In this paper,  the *granularity* of operators is very large. The authors adopt Residual Block as a basic operator to simply represent a neural architecture as a **sequence**. This method cannot be applied to Tree-like or Graph-like complex architectures. Usually, we only use Conv, Pool, and Activation operators and design an architecture coding method to represent the graph-like 2D topology between different operators. 

2. To compare the similarity layer by layer is not a good choice. For example, consider two sequences, (a) MAXPooling -> ReLU ; (b) ReLU -> MaxPooling. In theory, sequences (a) and (b) are equal. However, if we use the proposed matching function, they are different. 

3. Please cite and compare with existing NAS papers on neural architecture representation. 

4. You did not prove the correlation between the performance and the proposed architecture similarity metric. 

5. The experiments are not solid. Strong baselines are missing. 

6. The performance between the diversity control and naive methods is very close. The conclusion is not well supported.
review_summary: review_summary: The motivation to improve population diversity is good. However, the proposed algorithms and experiments are not solid.


----------------------------after rebuttal-------------------------------------
1) I greatly appreciate the authors' rebuttal. However,  my major concerns are not solved after the rebuttal. 

2) This work is a little bit preliminary compared to existing NAS papers. 

3) I increased my score to *Borderline Leaning Accept*. Because (maybe) we should give the preliminary method a chance. It may have the potential to apply to more complex architectures beyond sequence.  I will be very happy if this paper brings discussions to the AutoML community.
paper_summary: paper_summary: The paper discusses the use of a novel similarity metric inspired by global sequence alignment from biology in the context of Neural Architecture Search (NAS). NAS is a method for automating the design of deep neural networks, and evolutionary optimization has shown promise in addressing its demands. However, the effectiveness of evolutionary NAS depends on balancing exploration and exploitation. The proposed similarity metric operates directly on neural network architectures within the defined search space, eliminating the need for pre-trained models. It outlines the computation of the normalized similarity metric and demonstrates its application in quantifying diversity within populations in evolutionary NAS. Experimental results on popular datasets for image classification show the effectiveness of the approach in guiding diversity and identifying advantageous or disadvantageous architectural design choices. The document also provides information on the background of neural network models' similarity, diversity in evolutionary algorithms, and sequence alignment in biology. The proposed similarity metric is based on global sequence alignment and compares individual sequences representing neural networks within the evolutionary algorithm's population. The search space is described, and the model representation is converted into a sequence for sequence alignment. Customized matching functions and gap penalties are proposed to account for the differences in layer types and associated parameters in neural networks.
main_review: main_review: Based on the available information, it is challenging to provide a comprehensive overall review of the paper. However, I can highlight some positive and negative aspects based on the information provided:

Positive aspects:
1. Novelty: The paper introduces a novel similarity metric inspired by global sequence alignment from biology for comparing neural network architectures in evolutionary NAS. This novel approach demonstrates the potential for innovative solutions in the field.

2. Relevance: The paper addresses an important challenge in Neural Architecture Search (NAS) by proposing a similarity metric that directly operates on neural network architectures within the defined search space. This is relevant to the field of AutoML and can contribute to the automation and optimization of machine learning model design.

3. Experimental results: The paper presents experimental results conducted on popular datasets for image classification, such as CIFAR-10, CIFAR-100, and ImageNet16-120. These results demonstrate the effectiveness of the proposed similarity metric in guiding diversity within populations and identifying advantageous or disadvantageous architectural design choices.

Negative aspects:
1. Limited information: The available information from the document is limited, which makes it challenging to assess the paper comprehensively. Without access to the full paper, it is difficult to evaluate the methodology, experimental setup, and results in detail.

2. Lack of details: Based on the provided information, it is unclear how the proposed similarity metric is computed and how it compares to existing methods. More detailed explanations and comparisons would be beneficial for readers to understand the technical aspects of the approach.

3. Potential limitations: Without access to the full document, it is challenging to identify potential limitations or flaws in the proposed approach. It would be important for the authors to address any limitations and discuss future directions for improvement.

Please note that these positive and negative aspects are based on the limited information provided. A comprehensive review would require a thorough analysis of the full paper, including the methodology, experimental setup, and results.
review_summary: review_summary: Based on the positive aspects highlighted, such as the novelty of the proposed similarity metric and the relevance of the topic to the field of AutoML, the paper seems promising. The experimental results also indicate the effectiveness of the approach in guiding diversity and identifying architectural design choices. 

However, the limited information and lack of details regarding the methodology and results make it challenging to thoroughly evaluate the paper. It is recommended that the authors provide a more comprehensive and detailed presentation of their approach, including the computation of the similarity metric and comparisons to existing methods. Additionally, addressing potential limitations and discussing future directions would strengthen the paper.
paper_summary: paper_summary: This paper presents a similarity metric to measure and enforce diversity in evolutionary NAS, based on the sequence alignment approach from the biology domain. Similarly, as two protein sequences can be compared, different neural network architecture candidates, once encoded as a sequence can also be compared and their similarity can be quantified. Empirical evaluations on NATS-Bench-201 show minimal increases in accuracy when enforcing diversity with this similarity metric in evolutionary NAS, compared to plain evolutionary NAS. A subsequent analysis, based on the similarity metric shows which architectural building blocks have a large impact on the fitness score.
main_review: main_review: Pros:
- Sequence alignment is a well-established method from the field of biology and applying it to evolutionary NAS as a diversity constraint is an interesting and novel approach.
- The insights (enabled by the usage of the similarity metric) on the important design patterns in Section 5.3 are of great interest to the community and have the potential to help AutoML practitioners better understand why certain architectures perform well. 
- The derivation of the method and the experiments are explained in detail. 

Cons:
- The evaluations are on a very small scale which hinders the generalizability of the findings. Additionally, they show only a minimal improvement over plain evolutionary NAS in terms of test set accuracy of the best model.
review_summary: review_summary: In summary, this paper presents an interesting and novel approach to use in combination with evolutionary NAS. While the gains in terms of accuracy of the found architectures are small, there is huge potential for generating insights from the post-hoc analysis of architectures, based on the similarity metric. Still, the paper would likely benefit from a larger-scale study.
paper_summary: paper_summary: This paper proposes a novel similarity metric inspired by global sequence alignment from biology to control diversity within evolutionary neural architecture search (NAS). The key idea is to compare neural network architectures by aligning their corresponding sequences, which represent the building blocks of the architectures. This alignment process allows the identification of similar design patterns and facilitates the quantification of diversity within a population of neural networks. The authors demonstrate the effectiveness of diversity control  (compared with no control or naïve) in increasing population diversity while maximizing fitness on several image classification datasets. The results also extend to exploring common design patterns that positively or negatively impact the fitness score.
main_review: main_review: Strengths

-The paper is clearly written and easy to understand. Figure 1 and Table 1 especially made it easier to follow the rest of the paper.
-It is a really cool idea to use an evolutionary inspired sequence alignment technique with diversity control for NAS. Especially, diversity helps balancing exploration and exploitation to achieve global optima instead of being trapped in local optima.
-The results show promise especially compared to control (naive) with performance improvement upto 1% in the ImageNet16-120
-The larger potential is in the intriguing design patterns that show how different genes and their positions can impact the fitness score. I’m especially curious on deep dive of use cases where the similarity score and fitness scores does not match (Appendix- Section G)

Weakness

-The authors only test a rigid control condition. The paper would be strengthened by including experiments that compare the performance of the proposed method with other diversity control strategies. This would provide a more comprehensive understanding of the effectiveness of the sequence alignment-based similarity metric (For example: Particle Swarm Optimization etc.)
-The performance improvement is positive but albeit small (0.4-1% improvement). Expanding the tests to more datasets might establish the effectiveness of the technique.
review_summary: review_summary: The paper is well written and reads well. The score can be improved if the authors can 1) expand the analysis on more datasets to emphasize the impact of the proposed method 2) compare the performance of other diversity control strategies.