Analyzing Few-Shot Neural Architecture Search in a Metric-Driven Framework

Authors: timotée  ly-manson
mathieu  léonardon
abdeldjalil  aissa
el  bey
ghouthi boukli hacene
lukas  mauch
Venue: automl_cc_AutoML_2024_Conference
Published: 2024-02-26 21:56:39

Abstract: While Neural Architecture Search (NAS) methods help find optimal neural network architectures for diverse tasks, they often come with unreasonable costs. To tackle such a drawback, the one-shot NAS setting was introduced, where a supernet is used as a superposition of all architectures in the space and performs the search in a single training phase. While this method significantly reduces the cost of running NAS, the joint optimization of every architecture degrades the performance of the search. The few-shot NAS line of work tackles this issue by splitting the supernet into sub-supernets trained separately, each with a reduced level of weight-sharing, which gives rise to the new challenge of finding the best way to split the supernet. In particular, GM-NAS utilizes a gradient matching score to group operations in a splitting schema. We extend and generalize this method by building a framework with compatibility for any arbitrary architecture evaluation metric, enabling the generation of numerous and diverse splits. We leverage this new framework in conjunction with various metrics from the zero-shot NAS literature and investigate the benefits of splitting across algorithms and metrics. We find that architectures are distributed in disadvantageous ways inside splits, and that proposed supernet selection methods are flawed.

Main paper: 
 introduction: 
 Neural Architecture Search (NAS) [15, 19, 26] , has gained traction in recent times as it holds the promise to find the optimal neural network architecture for any given task. Early methods relied on reinforcement learning [26] or evolutionary algorithms [17] and incurred unaffordable training times when scaling up. Recent works use one-shot NAS [19] to reduce the computation requirements down to a single training. In the traditional one-shot NAS setting, the search space is reduced to a single directed acyclic graph (DAG) with searchable operations on all of its edges. Training is conducted on the supernet, whose edges in the DAG constitute a superposition of all possible operations. Namely, the weights of all architectures in the space are shared for every operation they have in common.Despite the significant improvements to search costs, one-shot NAS is criticized [2, 23] for its reduced efficacy as a proxy to candidate architecture performance and difficulties discovering better architectures in the search space. These flaws are linked to inherent properties of the joint training, namely co-adaptation. Indeed, some groups of operations may descend to undesirable directions when trained together. Few-shot NAS [8, 25] is introduced as a middle ground to reduce one-shot disadvantages while still maintaining reasonable computational costs. By partitioning the search space into smaller sets, the number of sub-supernets to train increases, but the quantity of weight-sharing in each of them is decreased. In practice, the accuracy of the sub-supernets is used as a proxy to choose a single sub-supernet to train, reducing the cost of few-shot NAS to be equivalent to that of a single one-shot NAS search.The authors of GM-NAS [8] argue that splitting exhaustively between all operations, as proposed in [25] , is wasteful as some operation groups could be less affected by co-adaptation. They propose On the selected edge, using any metric, operations are individually scored and compared.Partitions are formed via min-cut optimization.Figure 1 : Full supernet splitting framework to build sub-supernets from a splitting tree created with the help of the gradient matching score, which groups together operations whose gradients point to similar directions. Taking inspiration from the splitting schema described in GM-NAS, we introduce a supernet splitting framework which is, to the best of our knowledge, the most general framework in the few-shot NAS field to this date. Importantly, this extended supernet splitting framework allows to use any arbitrary metric in the decision-making of the splits. We make use of metrics first seen in the zero-shot NAS [1, 3, 16] literature, which are designed with the intent of scoring architectures, and implement them as splitting metrics in our framework.We take full advantage of the increased quantity and variety of sub-spaces and splits generated through this procedure to investigate several important assumptions for the few-shot NAS paradigm. Firstly, we verify that supernet splitting is indeed beneficial to helping one-shot NAS algorithms reach better-performing architectures. We also assess the impact of various metrics. Secondly, we observe that splitting is unable to isolate the best architectures in their own subspace, which undermines synergies with search algorithms. Finally, we show that commonly used proxies to subspace selection, such as the accuracy of the supernet, are flawed and cannot recognize a good supernet, much less indicate which contains the best architecture in the space.Our code is available at https://github.com/brain-bzh/metric_driven_few_shot_nas.2 Related Work 
one-shot nas: 
 Traditional NAS methods rely on reinforcement learning-based predictors [26] , evolutionary algorithms [17] or Bayesian kernels [21] , all of which are computationally heavy as they require the training of many candidate networks to act as labels in the search.Early on, weight-sharing is proposed as an alternative way to conduct NAS in a single training [19] , leading to the parallel research branch known as one-shot NAS. In the one-shot NAS paradigm, it is sufficient to train the supernet, which represents any candidate network in the search space. This change in representation enables the joint training of all candidate architectures with a single training phase.Many methods build on top of the weight-sharing mechanism to discover new candidate architectures. A popular setup [4, 7] is to utilize the trained supernet as a proxy for the search. The weights of the supernet are copied onto a sampled candidate architecture to estimate its performance without training and guide the search. Other lines of work aim to discover an architecture at the same time as training. DARTS [15] equips the supernet with architecture parameters learned in a differentiable manner. While this approach is simple and effective, it fails to find highly performing architectures as it hyperfocuses on non-parametric operations such as an identity [24] . Differently, SNAS [22] uses the distribution of architectures as the main objective of the supernet's training DSNAS, and optimizes the likelihood of sampling good architectures. DSNAS [9] further discretizes SNAS, while adopting progressive early stopping. In this work, we seek to evaluate the impact of supernet splitting on these various one-shot NAS algorithms. While one-shot NAS has made it possible to reduce search costs from several days down to a few hours, its performance is lackluster. It is exposed [23] that the architecture rankings produced by using a supernet proxy are highly degraded. This is largely attributed to co-adaptation [2] . Intuitively, the weights of each operation in the supernet after training converges are optimal in the context of joint training with every other operation in the space, but translate poorly to candidate architectures at evaluation time. This results in the mismatch between true and predicted architecture rankings.The search space introduced in DARTS [15] remains the standard for dedicated one-shot NAS search spaces. In this search space, cells are the base unit of candidate architectures, and are represented as a directed acyclic graph, where the nodes represent intermediate features while the edges are selectable operations. Two types of cells are included, depending on whether the features are downsampled in the same layer. Replicas of the searched cells are then stacked on top of each other to form the full network (Figure 2 ). This structure puts high emphasis on searching the right operations, with little concern for the size of the architecture. The NAS-Bench-201 [6] space is a reduction of the DARTS space with a single type of searchable cell and a reduced number of selectable operations. Due to its simplicity, it has been possible to exhaustively evaluate all of its 15625 candidate architectures on three datasets : CIFAR-10, CIFAR-100 [13] and ImageNet16-120, which is a downscaled subset of ImageNet [5] . As such it has become an important benchmark for one-shot NAS methods, which we leverage to assess key aspects of the few-shot NAS framework.Few-shot NAS [25] techniques are introduced to reduce co-adaptation in an orthogonal way to the one-shot search algorithm itself. Prior to the search, the supernet is splitted into several sub-supernets along an edge of the graph (Figure 1a ), such that operations located on that edge are no longer trained jointly. This operation shrinks the search space and reduces the effects of co-adaptation. By applying it several times, a splitting tree is created, with sub-supernets as the resulting leaves. The final splits are affected by the branching factor, which corresponds to the number of groups created at each branch, and the depth of the tree, which is the number of edges where splitting has been applied. The incurred time costs scale with the number of splits, unless an effective sub-supernet selection strategy is adopted. For this reason, the few-shot NAS paradigm can be seen as a middle-ground between the single-training procedure of one-shot NAS and the architecture sampling of traditional NAS. Using a naive policy where the supernet is split exhaustively along an edge, few-shot NAS reports an important gain to the correlation between supernet proxy and architecture ranking.GM-NAS [8] expands on few-shot NAS with the proposal to group together operations whose standalone gradient is similar. They remark that different group of operations may suffer varying degrees of co-adaptation, depending on whether the gradients of the operations are pointing to similar directions. In order to measure whether this is the case, the gradient matching metric is introduced :EQUATIONwhere S 𝑐𝑜𝑠 is the cosine similarity function, L is a loss function and 𝑚 𝑘 𝑜 is a sub-supernet with only operation 𝑜 left enabled on edge 𝑘.After evaluating gradient matching for every pair of operations, groups can be formed via bruteforced min-cut optimization. Due to the lesser number of generated sub-supernets, deeper levels of the splitting tree can be considered. When combined with classic one-shot NAS algorithms, GM-NAS reports significant improvement, notably reaching near-optimality on all three dataset of NAS-Bench-201 and state-of-the-art performance in the DARTS space when used in conjunction with SNAS.While GM-NAS introduces the question of searching for the best way to split a supernet, the underlying effects of splitting itself remain rather unexplored. Specifically, the results of the method could be attributed to the effectiveness of gradient matching as a metric, or to the positive influence of the splitting scheme. By generalizing the splitting framework further, we intend to give deeper insights into these techniques.In contrast with traditional NAS techniques and one-shot NAS, zero-shot NAS proposes to search for candidate architectures without training, relying instead on hand-crafted metrics to evaluate models at initialization within a single backward pass. This novel approach allows for low-cost, fast NAS that can be used in contexts such as edge computing [20] . Differently from their intended usage, we use metrics originated from this line of work to help with supernet splitting.NASWOT [16] introduced this paradigm by greedily searching the space with a metric constructed from the covariance of the jacobian, which we hereafter refer to as jacobcov. Further zero-cost proxies were introduced in [1] , which utilized the norm of the gradients gradnorm and several saliency metrics introduced in the neural network litterature : snip, grasp and synflow. These were used in conjunction with reinforcement learning or evolutionary algorithms.Differently, TE-NAS [3] uses a mixture of metrics to prune a supernet down to a single path. The condition number of the Neural Tangent Kernel [10] and the number of linear regions are chosen to strike a balance of trainability and expressivity. In the following paragraphs, we use these metrics in a standalone way and refer to them as ntk and #lr.The effectiveness of zero-shot NAS methods has been criticized, with subsequent work observing that naively counting the number of parameters in the model is a better proxy to model performance than most metrics employed in zero-shot NAS [12] . We include this metric in our study and refer to it as #params.Further details about the zero-shot metrics can be found in Appendix B. 
methods: 
 In summary, one-shot NAS methods train a supernet as a superposition of all candidate architectures in the space to guide architecture selection. Though, the joint training of all candidate architectures degrades the performance of the search. Few-shot NAS is introduced to mitigate this drawback by splitting the supernet into several sub-supernets, which are trained separately. The challenge of few-shot NAS is as follows : what is the best way to split the supernet? Motivated by the recent GM-NAS [8] which groups together architectures whose gradients are similar, we aim to extend the concept of metric-driven splitting to other promising metrics. In the following paragraphs, we introduce a further generalization of the GM-NAS framework to arbitrary network scoring metrics.The effectiveness of GM-NAS [8] suggests that it is possible to learn how to split the original search space into subsets such that some of the sub-spaces will be more favorable to one-shot search. However, a single metric that does so has been introduced, with no guarantees to be the best metric for this task. Furthermore, the effects of splitting itself on the search are not well-enough known to ascertain that the choice of the metric is essential. We seek to investigate the inner workings of supernet splitting. To do so, we introduce a supernet splitting framework inspired by GM-NAS. We generalize GM-NAS by using any metric that scores neural networks attributes to produce splits through pairwise comparison of each operation's impact on the metric. Intuitively, because these metrics are related to various aspects of a neural network (e.g. performance, trainability, expressivity), they can also be used to evaluate differences between several networks in terms of these aspects. Thus the relationship matrix of the edge can be constructed from :EQUATIONwhere 𝑓 is a given metric and 𝑚 𝑘 𝑜 is a sub-supernet with only operation 𝑜 left enabled on edge 𝑘.In practice, metrics compute a score within a single forward and backward pass on a randomly initialized model with all but the target operation removed at the target edge. The similarity between each pair of operations with respect to the metric is the difference of the scores. Assuming a small enough number of operations in the space, this optimization is performed exhaustively by assessing the cost of every permutation, as in GM-NAS. Thus the optimization objective is similar :EQUATIONwhere {U 𝑘 , O\U 𝑘 } are the obtained operation partitions for edge 𝑘.We implement the following metrics as the basis for supernet splitting: gradientmatching, gradnorm, jacobcov, snip, grasp, synflow, ntk, #lr, #params.We generate sub-supernets for every metric in the NAS-Bench-201 space, based on the same supernet checkpoint. Training details for the supernet can be found in Appendix A.2. The splitting tree goes to a depth of 3, with operations being split into two groups at each branch. In the end, we obtain 8 sub-supernet for each metric.Remark that the generation of a splitting tree supports other hyperparameters than the metric alone. The number of groups at each branch, also referred to as branching factor [8] , is an extra hyperparameter, which can degenerate the method to vanilla few-shot NAS [25] when set to its upper limit. This parameter could also be controlled more finely at each level of the tree with a dedicated policy. For the sake of simplicity, we fix the branching factor in subsequent experiments and only split operations into two groups. However, we do not apply the strict condition to keep splits balanced as in GM-NAS. We argue that some operations may behave so differently from others with respect to a specific metric on a given edge that it is justified to separate them from the others, even if this results in an uneven split. Differently from GM-NAS, we do not select an edge to split at run time, instead opting to always split the first three edges of the supernet in that order. In a framework with edge selection, various metrics may result in different edges being selected, thus keeping splitted edges fixed ensures full comparability between the metrics and keeps the focus on their operation selection capabilities. Furthermore, we do not apply warmup to sub-supernets in between splits, as the weights and gradients of the supernet, which all metrics are based on, have already converged. These weights are instead restarted once all splitting has concluded to avoid bias to the performance of associated one-shot NAS algorithms. choice of the metric. In this section, we evaluate the splits generated with 9 different metrics on 4 different one-shot NAS algorithms. In order to assess the impact of splitting by itself against the metrics, we also compare them against the random baseline, where splits are generated by sorting operations randomly into 2 groups. For a fairer comparison, we report SNAS and DSNAS runs with the same 3 seeds. In the case of DARTS algorithms, previous work [6] has shown that these algorithms are fully deterministic in the NAS-Bench-201 search space, therefore conducting repeat experiments is less relevant. We conduct these experiments on all 3 datasets of NAS-Bench-201 and report the results in Table 1 , Table 2 , and Table 3 . Note that we report only the performance of the best architecture found collectively by all sub-supernets, which is indicative of the maximum potential of each split-algorithm pair. The search cost required to reach this potential is the cost of running the one-shot NAS on all sub-supernets. Reducing this cost via e.g. supernet selection strategies may yield lower performance. Our experimental setup for each algorithm can be found in appendix A. Our results confirm the positive impact of splitting on the performance of various one-shot NAS algorithms. We observe that for most one-shot NAS algorithms, a better architecture is found when splitting regardless of the metric used. Note that even for the SNAS [22] algorithms that converge to a good architecture without splitting, performance is seldom degraded after splitting. Therefore, obtained results reinforce the core assumption of the few-shot NAS framework [8, 25] , that partitioning the search space is beneficial to the performance of conjoined NAS algorithms.As for the impact of the metrics used to generate sub-supernet networks, results reported in the different tables show that partitioning operations randomly yields similar or better results compared with other metrics. Furthermore, we find that the optimal metric varies across different datasets and different algorithms. In many cases, this happens with low variance, indicating that the splits generated by the metric are especially well suited for this combination of dataset and algorithm. Intuitively, the accuracy of architectures in the space varies relative to each other as tasks become more complex, while inherent properties of the supernet picked up by the selected metrics may not change as much. This indicates that the choice of an appropriate metric is not as important as the splitting operation itself. However, the fact that an algorithm searches more effectively when the space is split in specific ways can be linked with how the architectures are distributed within the splits. Therefore, we take a deeper look at sub-space distributions in the following section. 
what is in a split?: 
 The contents of each split remain unexplored in previous works. Indeed, both few-shot NAS and GM-NAS select a single sub-supernet based on its validation accuracy to cut costs down to similar levels to regular one-shot NAS. However, this disregards the fact that good architectures may be located in more than one split. Moreover, the distribution of architectures in the splits could explain why some metrics are more advantageous for particular algorithms. Specifically, the ideal split would group architectures based on their performance, with all best-performing architectures located in a single split.For each metric and dataset, we use the same 8 splits described in 3.2 and plot the distribution of architectures in each split with respect to their test accuracy from the tabulated NAS-Bench-201 [6] benchmark. Due to the sub-spaces being disjointed, the union of all 8 distributions amounts to the full search space. An example is shown in Fig 3 (top) . However, the vast majority of architectures in the search space have mediocre performance, while NAS algorithms will mostly converge towards the best architectures. Therefore, looking at the full distribution gives no insight on whether a good architecture is likely to be found in that sub-space. Thus, we display a second view of the distribution, restricted to the top 1% of architectures in the search space. An example is represented in Fig 3 (bottom) . We refer to Appendix D for distribution plots with other metrics and datasets.We observe that metrics create splits following two types of behaviors. Splitting with gradnorm, jacobcov, snip, grasp or ntk creates balanced sub-spaces following closely the distribution of the original space. Meanwhile, splitting with gradientmatching, synflow, #lr or #params groups a small number of architectures in some spaces, and a number roughly equal to half the size of the full space in another space. Importantly, we observe that across all metrics, the top 1% of architectures in the space is located in several sub-spaces, which is suboptimal. Unsurprisingly, when a larger proportion of the space's architecture is located in a single sub-space, a larger number of top 1% architectures is also present. Therefore, while some metrics may separate architectures in more advantageous ways, the performance gain could simply be attributed to the search taking place in a smaller search space, rather than inherent co-adaptation reduction inside the sub-supernets. Moreover, top architectures being spread out across multiple sub-spaces makes supernet selection impractical, as one may simply discard the global optimum while doing so.While the performance ceiling of NAS with splitting can be reached by training all sub-supernets exhaustively, past works strive to reduce the search down to similar costs as regular one-shot NAS. Few-shot NAS [25] selects the top 𝑘 sub-supernets with the highest validation accuracy after copying weights over from the source supernet. GM-NAS [8] propose a successive halving approach inspired by bandit optimization [11, 14] and discard the bottom half of architectures based on their validation accuracy following a set schedule, with a 2.5x cost increase only for 
conclusion: 
 One-shot NAS methods have been proposed to reduce the cost of running NAS, but they often degrade the performance of the search. Few-shot NAS methods tackle this issue by splitting the supernet into sub-supernets trained separately, each with a reduced level of weight-sharing. The challenge is to find the best way to split the supernet. A significant contribution in this direction is GM-NAS, which utilizes a gradient matching score to group operations in a splitting schema rather than an exhaustive approach. We extend this splitting framework to support any given splitting metric. By combining this generalization with several architecture scoring metrics from the zero-shot NAS literature, we can generate a greater variety of sub-spaces. We observe that partitioning the search space invariably increases the performance of one-shot NAS algorithms, while the best metric may change unpredictably depending on the task and the conjoined algorithm. Therefore, splitting itself is more important than the choice of a splitting metric. Moreover, we show that top architectures in the space are spread out across multiple sub-spaces, making it difficult to determine which sub-space is the best. Finally, we evaluate supernet selection techniques and determine that sub-supernet accuracy is an inadequate proxy for finding a good sub-space and might only degrade the performances. Overall, our work shows that, while few-shot NAS is a promising line of work to boost NAS performances in the one-shot setting, it requires careful tuning to reach its maximum performance with reduced costs. 
broader impact statement: 
 After careful reflection, the authors have determined that this work presents no notable negative impacts to society or the environment.  
 
 Appendix: In this section, we document the definition of each metric used.Gradients of the model are computed over a single batch of training data, then concatenated. This metric is equal to the norm of the obtained tensor :EQUATION)where 𝑚 is the evaluated model with parameters 𝑤, L is the cross-entropy loss and ∥ • ∥ F is the Frobenius norm.As with gradnorm, gradients are computed over a single batch of training data then concatenated. We first define the covariance matrix of obtained Jacobian tensor:EQUATION)where 𝑚 is the evaluated model with parameters 𝑤 and L is the cross-entropy loss. Let 𝜆 𝑚 1 . . . 𝜆 𝑚 𝑛 be the eigenvalues of 𝐶 (𝑚). The jacobcov metric is defined as follows:EQUATIONThe snip metric is computed over single batches of training data and is defined as follows:EQUATIONwhere 𝑖 are the layers of the model, with 𝑁 the total number of layers. 𝜃 𝑖 are the parameters at layer 𝑖 and L is the cross-entropy loss.The grasp metric is computed over single batches of training data and is defined as follows:EQUATION)where 𝑖 are the layers of the model, with 𝑁 the total number of layers. 𝜃 𝑖 are the parameters at layer 𝑖 and L is the cross-entropy loss. 𝐻 is the Hessian, which is estimated following [18] .The synflow metric is similar to snip:synflow(𝑚) = 𝑁 ∑︁ 𝑖 | 𝜕L 𝜕𝜃 𝑖 ⊙ 𝜃 𝑖 | (9)However, it is not computed over batches of training data. Instead, a batch of synthetic data with full matrices of 1s is passed to the model.The ntk metric is the condition number of the Neural Tangent Kernel (NTK) [10] .Gradients are computed over a single batch of training data then concatenated. Then, following TE-NAS [3] , the NTK is approximated as:EQUATION)where 𝑚 is the evaluated model with parameters 𝑤 and L is the cross-entropy loss. Let 𝜆 Θ(𝑚) 0 . . . 𝜆 Θ(𝑚) 𝑛 the eigenvalues of Θ(𝑚) sorted in ascending order of magnitude, the condition number of the NTK is defined as:ntk(𝑚) = 𝜆 Θ(𝑚) 𝑛 /𝜆 Θ(𝑚) 0 (11)The #lr metric is defined as the number of linear regions after the activations of the model. In practice, the number of linear regions is estimated from a batch of training data, through careful analysis of the signs of the post-activations, following TE-NAS [3] .The #params metric is simply defined as the number of parameters in the model, which is obtained at initialization with no training data needed.In this section, we report the results of the experiment from section 4.3 when splitting using various other metrics. All other parameters of the experiment remain the same, and we use SNAS [22] on the CIFAR10 task for the search. Out of 5 reported metrics including gradientmatching, gradnorm, jacobcov, ntk and params, only the supernet selection phase of jacobcov yields satisfactory results, where the sub-supernet that finds the best performing architecture is also clearly the one selected in the end, while not being cut prematurely if applying successive halving.We observe in the case of params that supernet selection remains very unstable up to the end of the training, with no sub-supernet coming out as the clear winner, while the best performing sub-supernet would be cut by applying successive halving.Finally, the cases of gradnorm and ntk show that best-performing sub-supernets can experience collapse towards the end of the training, resulting in the selection of a sub-optimal supernet compared to the global peak over the search. This could indicate a mismatch of the search algorithm with supernet selection.Overall, supernet selection using validation accuracy as a proxy often exhibits undesirable behaviors for finding the best performing sub-supernet. sub-supernets, compared to one-shot NAS. Interestingly, both approaches use sub-supernet accuracy as a proxy to being a good supernet, but few guarantees of this relationship have been given. In order to assess whether accuracy is a good proxy for supernet selection, we use splits obtained with gradientmatching and run SNAS[22] on each of 8 sub-supernets, using the same experimental setup described in (appendix), only extended to 600 epochs.We report the results for CIFAR-10 in Fig 4.Firstly, we find that, by the end of 600 epochs and for most of the training, the sub-supernet with highest validation accuracy is not the one that finds the highest performing architecture with SNAS, which shows that validation accuracy is not a suitable proxy to selecting a sub-supernet. Furthermore, by applying a successive halving scheme similar to GM-NAS, where half of the sub-spaces are trimmed at epochs (30, 100, 600), we observe that the best sub-space is discarded early on. This indicates that supernet accuracy is especially inaccurate as a proxy in early stages of the training. As such, supernet selection with validation accuracy is highly likely to degrade the final performance of few-shot NAS compared to its maximum possible performance. 


Decision: The authors proposed an extension of GM-NAS to be able to leverage arbitrary architecture evaluation metric. The method is clearly stated and evaluated with a diverse variety of metrics. The method is well motivated and looks reasonable. One critical question is about the practicality of the proposal. Looking at the results, the gradient matching in GM-NAS is already robust enough from the table. Then in practical, why not just using GM-NAS? If we choose this new approach, what is the best practice to efficiently secure a reliable and robust result? It would be great for the authors to discuss these in depth. Despite the practicality issue, considering the method contribution and analysis, I would recommend "accept".

General Suggestions:
- Reproducibility reviewer has difficulty to fully execute (some work and some don't). It would be great to continue improve the code quality and documentation.
- Ablation studies are critical to the paper, it would be great to move it to the main part and further enhance it.
- As an extension to GM-NAS, it is great to make it more general, while practicality concern is not addressed. If it can not be addressed in this paper, it would be great to have a in-depth discussion for this aspect.

Reviews: review: paper_summary: paper_summary: - Verifying that splitting the supernet into sub-spaces does help one-shot NAS algorithms find better-performing architectures compared to searching the full supernet space.
- Assessing the impact of using different metrics/objectives in the splitting procedure.
- Observing that despite splitting, the sub-spaces created are unable to isolate/contain the globally best architectures separately, which limits the synergies with search algorithms.
- Showing that commonly used proxies/heuristics for selecting promising sub-spaces, like the accuracy of the supernet itself, are flawed and cannot reliably identify good supernets or indicate which ones contain the best architectures.
main_review: main_review: One-short and Few-shot NAS are significant areas of research for AutoML. This study focuses on analyzing the impact of using validation accuracy for subnet selection as well as investigating the benefits of splitting across 17 algorithms and metrics concluding supernet selection methods are flawed. However, the study does not provide any ablation study(if conducted) and merely claims based on tunning epochs. It's important to provide a theoretical explanation as to why authors observe this phenomenon to further prove the efficacy of their experimental design. If the output of their insight is based on building a novel framework that can generalize based on GM-NAS and other existing techniques, authors must describe this framework in detail and their unique contributions.
review_summary: review_summary: One-short and Few-shot NAS are significant areas of research for AutoML. This study focuses on analyzing the impact of using validation accuracy for subnet selection as well as investigating the benefits of splitting across several algorithms and metrics concluding supernet selection methods are flawed. However, the study does not provide any ablation study(if conducted) and merely claims based on tunning epochs. It's important to provide a theoretical explanation as to why authors observe this phenomenon to further prove the efficacy of their experimental design. If the output of their insight is based on building a novel framework that can be generalized based on GM-NAS, authors must describe this framework in detail, highlight their unique contributions and how some of the changes made in Section 3.2 affect the outcome. However, the insights mentioned in the paper are very interesting and could benefit not only AutoML group of researchers but non-AutoML researchers as well.
paper_summary: paper_summary: The paper delves into the realm of few-shot Neural Architecture Search (NAS) methods, focusing specifically on the technique of splitting the one-shot NAS supernetwork into sub-supernetworks. This subdivision aims to simplify the search for optimal architectures, particularly facilitating gradient-based NAS optimization. By evaluating the methodologies proposed in two seminal works [1] and [2], the paper scrutinizes the efficacy of sub-supernetwork splits in mitigating co-adaptation during the search process. Additionally, it explores the utilization of zero-cost proxy metrics, shedding light on their potential in the few-shot NAS paradigm. Notably, the study unearths empirical evidence suggesting that the success of few-shot NAS methods may not solely stem from co-adaptation alleviation via sub-supernetwork splits, but rather from the reduced search space itself. Furthermore, it uncovers shortcomings in the evaluation phase of certain sub-supernetwork techniques, thereby enriching the understanding of the intricacies involved in few-shot NAS methodologies.

**-- References --**

[1] Y. Zhao, L. Wang, Y. Tian, R. Fonseca, and T. Guo. Few-shot neural architecture search. In ICML 2021.

[2] S. Hu, R. Wang, L. Hong, Z. Li, C.-J. Hsieh, and J. Feng. Generalizing few-shot NAS with gradient matching. In ICLR 2022.
main_review: main_review: This paper shows some interesting findings regarding few-shot NAS approaches. Below I list some of the positive and negative points of the paper:

Pros:

1. **Clear Motivation and Objectives:** The paper effectively articulates the rationale behind investigating few-shot NAS methods and delineates clear research objectives. This clarity enhances the reader's understanding and underscores the significance of the study.

2. **Insightful Results and Analysis:** The findings presented in the paper offer valuable insights into the underlying mechanisms of few-shot NAS approaches. By dissecting the efficacy of sub-supernetwork splits and zero-cost proxy metrics, the study contributes to a nuanced understanding of these methodologies.

3. **Robust Experimental Protocol:** The experimental setup is well-designed and executed, bolstering the reliability and reproducibility of the results.

4. **First to evaluate Zero-cost Proxies in few-shot NAS:** The evaluation of zero-cost proxy metrics within the context of few-shot NAS is new. This novel exploration expands the methodological toolkit available for assessing NAS methodologies and opens avenues for further research in this domain.

5. **Clarity of Presentation:** The writing style employed in the paper is lucid and accessible, ensuring that complex concepts are conveyed effectively to the reader. This clarity enhances the readability and comprehensibility of the manuscript.

6. **Availability of Code:** The provision of code accompanying the paper fosters transparency and facilitates the replication of experiments by other researchers. This openness enhances the reproducibility of the study's findings.

Cons:

1. **Limited Scope of Search Spaces:** A notable limitation of the study is the restricted scope of search spaces examined. While the experiments provide valuable insights, extending the analysis to encompass diverse NAS benchmarks such as the ones in NAS-Bench-Suite (https://arxiv.org/abs/2201.13396) or NAS-Bench-Suite-Zero (https://arxiv.org/abs/2210.03230) would bolster the generalizability and robustness of the conclusions. This way, their claims such as "while some metrics may separate architectures in more advantageous ways, the performance gain could simply be attributed to the search taking place in a smaller search space, rather than inherent co-adaptation reduction inside the sub-supernets", will be better suported.
review_summary: review_summary: Overall, I think this paper provides a useful empirical study of few-shot NAS methods, with results demonstrating that some of the claims from previous work might not be correct. It also investigates the applicability of zero-cost proxies inside the few-shot NAS realm. The clear articulation of research objectives, insightful findings, experimental protocol, and accessibility, render it a valuable addition to the existing literature. I lean towards acceptance, even though the authors need to provide another search space where their empirical findings are the same.
paper_summary: paper_summary: To resolve the unreasonable cost issue of typical NAS methods, this paper introduces a few-shot NAS setting. They verify several assumptions in the few-show / one-shot NAS setting: 1. supernet splitting is indeed beneficial; 2. splitting is unable to isolate the best architectures; 3. commonly used proxies to subspace selection are flawed and cannot recognize a good supernet.
main_review: main_review: This paper does a good experimental analysis for few-shot NAS. However, the motivation and contributions should be addressed more clearly and the writing also needs to be improved. Existing questions are listed in the above.
review_summary: review_summary: This paper focuses on an interesting topic for NAS. However, the authors didn't present the motivation, contributions and details of the method clearly. I believe it would be improved after a major revision. Unfortunately, I have to give a score of weak reject at this time.
paper_summary: paper_summary: The main contributions of the paper "Analyzing Few-Shot Neural Architecture Search in a Metric-Driven Framework" are as follows:

Introduction of Few-Shot NAS: The paper introduces the concept of Few-Shot Neural Architecture Search (NAS) as a method to reduce the costs associated with traditional NAS techniques by splitting the supernet into sub-supernets trained separately with reduced weight-sharing.

Generalization of Splitting Framework: The authors extend and generalize the Few-Shot NAS method by developing a framework that supports any arbitrary architecture evaluation metric. This framework allows for the generation of diverse splits, enhancing the efficiency of architecture evaluation in NAS.

Integration of Zero-Shot NAS Metrics: The paper incorporates various metrics from the zero-shot NAS literature into the splitting framework, providing insights into the benefits of splitting across different algorithms and metrics. Metrics such as jacobcov, gradnorm, snip, grasp, synflow, ntk, #lr, and #params are utilized to improve the performance of NAS algorithms.

Evaluation of Supernet Selection Techniques: The authors evaluate supernet selection methods and highlight the inadequacy of sub-supernet accuracy as a proxy for finding optimal sub-spaces. The study reveals that selecting sub-supernets based on validation accuracy may degrade the final performance of few-shot NAS.

Overall, the paper contributes to the advancement of NAS techniques by proposing a metric-driven framework for analyzing few-shot NAS methods, exploring the effectiveness of splitting strategies, and evaluating the impact of different metrics on architecture search performance.
main_review: main_review: The paper "Analyzing Few-Shot Neural Architecture Search in a Metric-Driven Framework" presents a novel metric-driven framework for analyzing few-shot NAS methods, focusing on the splitting of the supernet into sub-supernets for improved architecture search. Here is an overall review highlighting the positive and negative aspects of the paper:

Positive Aspects:

Novel Framework: The paper introduces a novel metric-driven framework for analyzing few-shot NAS methods, which addresses the challenge of splitting the supernet into sub-supernets. This framework provides a structured approach to evaluating the impact of different metrics on architecture search performance.

Integration of Zero-Shot NAS Metrics: By integrating various metrics from the zero-shot NAS literature, the paper enriches the analysis and provides a comprehensive evaluation of the splitting strategies. This integration adds depth to the study and offers insights into the effectiveness of different metrics in the context of few-shot NAS.

Comprehensive Experiments: The paper conducts comprehensive experiments to evaluate the impact of splitting on the performance of one-shot NAS algorithms using different metrics. The comparison against a random baseline and the reporting of results across multiple datasets demonstrate a thorough experimental design.

Clear Presentation: The paper effectively communicates the contributions, methodology, experiments, and conclusions in a clear and structured manner. The logical flow of the paper aids in understanding the proposed framework and the experimental results.

Negative Aspects:

Lack of Code Availability: While the code used for generating the splits is provided, the absence of open-sourced implementations of the NAS algorithms used in the study limits the reproducibility of the experiments. Providing access to all code and data is essential for ensuring reproducibility and transparency in research.

Limited Robustness Analysis: The paper could benefit from a more detailed analysis of the robustness of the proposed framework and metrics across different datasets or experimental settings. A more extensive evaluation of the framework's generalizability would strengthen the validity of the findings.

Comparison with State-of-the-Art: While the paper introduces a novel framework and integrates zero-shot NAS metrics, a direct comparison with existing state-of-the-art methods in the field of AutoML could further highlight the contributions and novelty of the proposed approach.

In conclusion, the paper makes significant contributions to the field of AutoML through its novel metric-driven framework for analyzing few-shot NAS methods. Addressing the limitations related to code availability, robustness analysis, and comparison with existing methods could further enhance the impact and credibility of the research.
review_summary: review_summary: The paper "Analyzing Few-Shot Neural Architecture Search in a Metric-Driven Framework" presents a novel metric-driven approach for analyzing few-shot NAS methods, focusing on supernet splitting and metric integration. The paper demonstrates clear contributions, comprehensive experiments, and a well-structured presentation. However, limitations in code availability, robustness analysis, and comparison with state-of-the-art methods were noted.

Recommendation: I recommend acceptance with minor revisions. The paper's contributions and experimental design are strong, but addressing the limitations related to code availability, robustness analysis, and comparison with existing methods would further enhance the research's impact and reproducibility. With these minor revisions, the paper can make a valuable contribution to the field of AutoML.