Weight-Entanglement Meets Gradient-Based Neural Architecture Search

Venue: automl_cc_AutoML_2024_Conference
Published: 2024-02-08 21:26:52

Abstract: Weight sharing is a fundamental concept in neural architecture search (NAS), enabling gradient-based methods to explore cell-based architectural spaces significantly faster than traditional blackbox approaches. In parallel, weight entanglement has emerged as a technique for more intricate parameter sharing amongst macro-architectural spaces. Since weight-entanglement is not directly compatible with gradient-based NAS methods, these  two paradigms have largely developed independently in parallel sub-communities. This paper aims to bridge the gap between these sub-communities by proposing a novel scheme to adapt gradient-based methods for weight-entangled spaces. This enables us to conduct an in-depth comparative assessment and analysis of the performance of gradient-based NAS in weight-entangled search spaces. Our findings reveal that this integration of weight-entanglement and gradient-based NAS brings forth the various benefits of gradient-based methods, while preserving the memory efficiency of weight-entangled spaces. The code for our work is openly accessible at https://anon-github.automl.cc/r/TangleNAS-5BA5.

Main paper: 
 introduction: 
 The concept of weight-sharing in Neural Architecture Search (NAS) arose from the need to improve the efficiency of conventional blackbox NAS algorithms, which demand significant computational resources to evaluate individual architectures. Here, weight-sharing (WS) refers to the paradigm by which we represent the search space with a single large supernet, also known as the one-shot model, that subsumes all the candidate architectures in that space. Every edge of this supernet holds all the possible operations that can be assigned to that edge. Importantly, architectures that share a particular operation also share its corresponding operation weights, allowing simultaneous training of an exponential number of subnetworks, unlike the sequential approach of blackbox NAS.Gradient-based NAS algorithms (or optimizers), such as DARTS (Liu et al., 2019) , GDAS (Dong and Yang, 2019) and DrNAS (Chen et al., 2021b) , assign an architectural parameter to every choice of operation on a given edge of the supernet. The output feature maps of these edges are thus an aggregation of the outputs of the individual operations on that edge, weighted by their architectural parameters. These architectural parameters are learned using gradient updates by differentiating through the validation loss. Supernet weights and architecture parameters are therefore trained simultaneously in a bi-level fashion. Once this training phase is complete, the final architecture can be identified quickly, e.g., by selecting operations with the highest architectural weights on each edge as depicted in Figure 1 (b). However, more sophisticated methods have also been explored (Wang et al., 2021) for this selection.While gradient-based NAS methods have primarily been studied for cell-based NAS search spaces, a different class of search spaces focuses on macro-level structures (parameterizing kernel size, number of channels, etc.) for which all architectures in the space are subnetworks of the architecture with the largest architectural choices, which is identical to the supernet in this case. These search spaces share weights via the more intricate form of weight-entanglement (WE) between similar operations on the same edge; e.g., the nine weights of a 3 × 3 convolution are a subset of the 25 weights of a 5 × 5 convolution. This paradigm reduces the memory requirements of the supernet to the size of the largest architecture in the search space.In order to efficiently search over such weight-entanglement spaces, two-stage methods have been introduced that first pre-train the supernet, and then perform blackbox search on it to obtain the final architecture. OFA (Cai et al., 2020) , SPOS (Guo et al., 2020) , AutoFormer (Chen et al., 2021a) and HAT (Wang et al., 2020) are prominent examples of methods that fall into this category. Note that these methods do not employ additional architectural parameters for supernet training or search. They typically train the supernet by randomly sampling subnetworks and training them as depicted in Figure 1 (a). The post-hoc blackbox search relies on using the performance of subnetworks sampled from the trained supernet as a proxy for true performance on the unseen test set. To contrast with this two-stage approach, we refer to traditional gradient-based NAS approaches as single stage methods.While to date, weight-entangled spaces have only been explored with two-stage methods, and cellbased spaces have only been optimized with single-stage approaches, in this paper we bridge the gap between these parallel sub-communities. We do so by addressing the challenges associated with integrating off-the-shelf single-stage NAS methods with weight-entangled search spaces.After a discussion of related work (Section 2), we make the following main contributions:• We propose a generalized scheme to apply single-stage methods to weight-entangled spaces, while maintaining search efficiency and efficacy at larger scales (Section 3, with visualizations in Figure 1 (c) and Figure 2 ). We refer to this method as TangleNAS.• We propose a fair playground for the comparative evaluation of single and two-stage methods (Section 4.1) and study the effect of weight-entanglement in conventional cell-based search spaces (i.e., NASBench201 and the DARTS search space) (Section 4.2).• We evaluate our proposed generalized scheme for single-stage methods across a diverse set of weight-entangled macro search spaces and tasks, from image classification (Section 4.3.1, Section 4.3.2) to language modeling (Section 4.3.3).• We conduct a comprehensive evaluation of the properties of single and two-stage approaches including any-time-performance, memory consumption, robustness to training fraction and effect of fine-tuning (Section 5), demonstrating that our generalized gradient-based NAS method achieves the best of single and two-stage methods: the enhanced performance, improved supernet fine-tuning properties, superior any-time performance of single-stage methods, and the low memory consumption of two-stage methods .To facilitate reproducibility, our code is openly accessible here. 
related work: 
 Weight-sharing was first introduced in ENAS (Pham et al., 2018) , which reduced the computational cost of NAS by 1000× compared to previous methods. However, since this method used reinforcement learning its performance was quite brittle. Bender et al. (2018) simplified the technique, showing that searching for good architectures is possible by training the supernet directly with stochastic gradient descent. This was followed by DARTS (Liu et al., 2019) , which set the cornerstone for efficient and effective gradient-based, single-stage, NAS approaches.DARTS, however, had prominent failure modes, such as its discretization gap and convergence towards parameter-free operations, as outlined in Robust-DARTS (Zela et al., 2020) . Numerous gradient-based one-shot optimization techniques were developed since then (Cai et al., 2019; Nayman et al., 2019; Xu et al., 2019; Dong and Yang, 2019; Hu et al., 2020; Li et al., 2021; Chen et al., 2021b; Zhang et al., 2021) . Among these, we highlight DrNAS (Chen et al., 2021b) , which we will use in our experiments as a representative of gradient-based NAS methods. DrNAS treats one-shot search as a distribution learning problem, where the parameters of a Dirichlet distribution over architectural parameters are learned to identify promising regions of the search space.Despite the remarkable performance of single-stage methods, they are not directly applicable to some real-world architectural domains, such as transformers, because of the macro-level structure of these search spaces. DASH (Shen et al., 2022) employs a DARTS-like methodology to optimize CNN topologies (i.e. kernel size, dilation) for a diverse set of tasks, reducing computational complexity by appropriately padding and summing kernels with different sizes and dilations. FBNet-v2 (Wan et al., 2020) makes an attempt along these lines for CNN topologies, but its methodology is not easily extendable to search spaces like transformers with multiple interacting modalities, such as embedding dimension, number of heads, expansion ratio, and depth.𝞪 1 𝞪 2 𝞪 3 𝜷 1 𝜷 2 𝜷 3 𝞪 1 𝞪 2 𝞪 3 𝜷 1 𝜷 2 𝜷 3Weight-entanglement, on the other hand, provides a more effective way of weight-sharing exclusive to macro-level architectural spaces. The concept of weight-entanglement was developed in slimmable networks (Yu et al., 2018; Yu and Huang, 2019) , OFA (Cai et al., 2020) and BigNAS (Yu et al., 2020) in the context of convolutional networks (see also AtomNAS (Mei et al., 2019) ) and later spelled out in AutoFormer (Chen et al., 2021a) , where it was applied to the transformer architecture.Single-path-one-shot (SPOS) methods (Guo et al., 2020) have shown a lot of promise in searching weight-entangled spaces. SPOS trains a supernet by uniformly sampling single paths (one at a time to limit memory consumption) and then training the weights along that path. The supernet training is followed by a black-box search that uses the performance of the models sampled from the trained supernet as a proxy.OFA used a similar idea to optimize different dimensions of CNN architectures, such as its depth, width, kernel size and resolution. Additionally, it enforced training of larger to smaller sub-networks sequentially to prevent interference between sub-networks. Subsequently, AutoFormer adopted the SPOS method to optimize a weight-entangled space of transformer architectures.In this work, we demonstrate how single-stage methods can be applied to macro-level search spaces with weight-entanglement, where one can benefit from the time efficiency and remarkable effectiveness of modern differentiable NAS optimizers, while concurrently maintaining memory efficiency of the weight-entangled space. While we chose to adopt DrNAS as the primary approach for our exploration of the weight-entangled space in this study, it is worth highlighting that our methodology is generally applicable to other gradient-based NAS methods. 
methodology: single-stage nas with weight-superposition: 
 Our primary goal in this work is to effectively apply single-stage NAS to search spaces with macrolevel architectural choices. To reduce the memory consumption and preserve computational efficiency, we propose two major modifications to single-stage methods. In our framework, the operators on any given edge inherit the weights of the largest operator on that edge. This reduces the size of the supernet to the size of the largest individual architecture in the search space.+ 𝜶 3 + 𝜶 2 𝜶 1 𝜶 1 + 𝜶 2 + 𝜶 3 = 1 𝜶 1 + 𝜶 2 𝜶 1Adapted weights Algorithm 1 TangleNAS = 𝜶 1 𝛽 1 + 𝜶 2 𝛽 1 + 𝜶 1 𝛽 2 + 𝜶 2 𝛽 2 𝜶 1 𝛽 1 +𝜶 1 𝛽 2 +𝜶 2 𝛽 1 +𝜶 2 𝛽 2 = 1 𝜶 1 𝛽 1 +𝜶 1 𝛽 2 =𝜶 1 𝜶 1 𝛽 1 (b) = Adapted weights Saved supernet weights 𝜶 1 𝛽 1 +𝜶 2 𝛽 1 =𝛽 11: Input: M ← number of cells, N ← number of operations O ← [o 1 , o 2 , o 3 , ...o N ] Wmax ← ∪ N i-1 w i A ← [α 1 , α 2 , α 3 , ...α N ] γ = learning rate of A, η = learning rate of Wmax f is a function or distribution s.t. N i=1 f (α i ) = 1 2: Cell j ← DAG(O j , Wmax j ) / * defined for j=1...M * / 3: Supernet ← ∪ M j Cell j ∪ A 4: / *8: o j (x, Wmax) = o j,i (x, N i=1 f (α i ) Wmax[: i])9: end for 10: / * weights and architecture update * / 11:A = A -γ∇ A L val (Wmax * , A) 12: Wmax = Wmax -η∇ Wmax L train (Wmax, A) 13: / * Architecture Selection * / 14: selected_arch ← arg max(A)Then, instead of weighting the outputs of operations on a given edge by architectural parameters, as singlestage NAS methods do (see Figure 1 (b)), we simply take each operation choice, zero-pad that to match the dimensions of the largest operation and sum these terms, each weighted by the corresponding architectural parameter. Figure 2 (a) provides an overview of the idea for a single architectural choice such as the kernel size. This is equivalent to taking the largest operation and re-scaling the weights of each sub-operation by the sum of architectural parameters of operations it is embedded in (see right-most weight matrix in Figure 2(a) . This process results in a superposition of operation weights whose structural property is similar to the largest operation. Consequently, a single forward pass suffices to capture the effect of all operational choices, thus making our approach computationally efficient. Furthermore, to accommodate for operations depending on two or more architectural dimension choices we introduce the combi-superposition outlined in Figure 2 (b) and Algorithm 2. This allows us to apply any arbitrary gradient-based NAS method, such as DARTS, GDAS or DrNAS, to macro-level search spaces that leverage weight-entanglement, without additional memory consumption and computational cost during forward propagation. We refer to our memory and compute optimal single-stage architecture search method as TangleNAS and provide an overview of the ap-  
experiments: 
 We evaluate TangleNAS across a broad range of search spaces, ranging from cell-based spaces, which serve as the foundation for single-stage methods, to weight-entangled convolutional and transformer spaces, which are central to two-stage methods. We initiate our studies by exploring three simple toy search spaces, which include a collection of cell-based and weight-entangled spaces. Later, we scale our experiments to larger analogs of these spaces. In all our experiments, we use WE to refer to the supernet type with entangled weights between operation choices and WS to refer to standard weight-sharing proposed in cell-based spaces. For details on our experimental setup please refer to Appendix E. Furthermore, in all our experiments the focus is on unconstrained search, i.e., a scenario where the user is interested in obtaining the architecture with the best performance on their metric of choice. The two-stage baselines we mainly compare against are SPOS (Guo et al., 2020) with Random Search (SPOS+RS) and SPOS with Evolutionary Search (SPOS+ES). For MobileNetV3 (Section 4.3.2) and ViT (Section 4.3.1) we use the original training scheme from OFA (Cai et al., 2020) and Autoformer (Chen et al., 2021a) , respectively. Both of these works use SPOS (Guo et al., 2020) as their foundation.We begin the evaluation of TangleNAS on two relatively compact toy search spaces that we designed as a contribution to the community to allow faster iterations of algorithm development:• Toy cell space: a small version of the DARTS space; architectures are evaluated on the Fashion-MNIST dataset.• Toy conv-macro space: a small space inspired by MobileNet, including kernel sizes and the number of channels in each convolution layer; architectures are evaluated on CIFAR-10.We describe these spaces in Appendix C, including links to code for these open source toy benchmarks. The results of these experiments are summarized in Tables 1 and 2 . Across both of these search spaces, TangleNAS outperforms its two-stage counterparts. Also, DrNAS without weight entanglement performs very poorly on the macro level search space.We now start our comparative analysis of single and two-stage approaches by applying them to cell-based spaces that serve as focal points in the single-stage NAS literature to evaluate TangleNAS against DrNAS and SPOS. Here, we use the popular NAS-Bench-201 (NB201) (Dong and Yang, 2020) and DARTS (Liu et al., 2019) search spaces. We refer the reader to Appendix C for details about these spaces and Appendix E.4 for the experimental setup.Our contribution on these spaces is two-fold. First, we study the effects of weight-entanglement on cell-based spaces in conjunction with single-stage methods. To this end, we entangle the weights of similar operations with different kernel sizes on both search spaces. For NB201, the weights of the 1x1 and 3x3 convolutions are entangled, and in the DARTS search space, the weights of dilated and separable convolutions with kernel sizes 3x3 and 5x5 are entangled. Second, we study SPOS on the NB201 and DARTS search spaces. To the best of our knowledge, we are the first to study a two-stage method like SPOS in such cell search spaces.Tables 3 and 4 show the results. For both search spaces, TangleNAS yields the best results, outperforming the single-stage baseline DrNAS with WS, as well as both SPOS variants. TangleNAS also significantly lowers the memory requirements and runtime compared to its weight-sharing counterpart. We note that overall, the SPOS methods are ineffective in these cell search spaces. Given the promising results of TangleNAS on the toy and cell-based spaces, we now extend our evaluation to the home base of two-stage methods. We study TangleNAS on a vision transformer space (AutoFormer-T) and a convolutional space (MobileNetV3), which have been proposed and explored using two-stage methods by Chen et al. (2021a) and Cai et al. (2020) , respectively, as well as a language model transformer search space built around GPT-2 Radford et al. (2019) . We evaluate TangleNAS on the AutoFormer-T space introduced by Chen et al. (2021a), which is based on vision transformers. The search space consists of the choices of embedding dimensions and number of layers, and for each layer, its MLP expansion ratio and the number of heads it uses to compute attention. More details can be found in Table 17 in the appendix. The embedding dimension choice is held constant throughout the network while the number of heads and the MLP expansion ratio varies for every layer. This amounts to a total search space size of about 10 13 architectures. We train our supernet using the same training hyperparameters as used in AutoFormer. Table 6 : Evaluation on the AutoFormer-T space on downstream tasks. ImageNet accuracies are obtained through inheritance, whereas accuracies for the other datasets are achieved through finetuning the imagenet-pretrained model.Table 5 and Table 6 evaluates TangleNAS against AutoFormer on the AutoFormer-T space. Interestingly, we observe that although AutoFormer sometimes outperforms TangleNAS upon inheritance from the supernet, the TangleNAS architectures are always better upon fine-tuning and much better upon retraining. For ImageNet-1k we obtain an accuracy of 78.842% by inheriting the weights of the optimal architecture in contrast to 75.474% by unconstrained evolutionary search on the on AutoFormer-T space (Chen et al., 2021a) , i.e. a net improvement of 3.368% (see Table 6 ). Next, we study a convolutional search space based on the MobileNetV3 architecture. The search space is defined in Table 19 in the appendix and contains about 2 × 10 19 architectures. This follows from the search space designed by OFA (Cai et al., 2020) , which searches for kernel-size, number of blocks and channel-expansion factor. Unlike OFA, we do not use the progressive shrinking scheme for the supernet training but activate all choices in our supernet at all times.Table 7 shows that on this OFA search space, TangleNAS still outperforms OFA (based on both unconstrained evolutionary and random search on the pre-trained OFA supernet). Notably, TangleNAS even yields an architecture with higher accuracy than the largest architecture in the space (while OFA yields worse architectures).Finally, given the growing interest in efficiently training large language models and recent developments in scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) , we study our efficient and scalable TangleNAS method on a language-model space.We create our language model space around a smaller version of Andrej Karpathy's nanoGPT model1 .In this transformer search space, we search for the embedding dimension, number of heads, number of layers and MLP ratios as defined in Table 18 (in the appendix). For each of these transformer search dimensions we consider 3 choices. We evaluate our small language model on the TinyStories (Eldan and Li, 2023) dataset. We improve over the SPOS+ES and SPOS+RS baselines as presented in Table 8 . This improvement is statistically very significant with a two-tailed p-value of 0.0064 for ours v/s SPOS+ES and p-value of 0.0127 for ours v/s SPOS+RS. 
results and discussion: 
 For a more in-depth evaluation, we now compare different properties of single-stage and two-stage methods, starting from their any-time performance, impact of the portion of train-val split on their performance and centered-kernel alignment of the supernet feature maps. Moreover, we also study the effect of pretraining, fine-tuning and retraining on the AutoFormer space for CIFAR10 and CIFAR100 and downstream performance of imagenet pre-trained best model on several classification datasets. Finally, we discuss the insights derived from our single-stage approach in designing architectures on real world tasks. Any-Time performance An essential characteristic of any neural architecture search (NAS) method is the attainment of strong any-time performance. End users of NAS techniques frequently prioritize the expedited identification of competitive architectures within limited time constraints. This becomes particularly crucial in light of the escalating computational expenses associated with training sophisticated and large neural networks, such as Transformers. Strong any-time performance is a pivotal criterion for the effective applicability of NAS in such resource-intensive domains. We thus compare the any-time performance of TangleNAS. Figure 3 demonstrates that TangleNAS (DrNAS+WE) is faster than its baseline method DrNAS+WS. Similarly, Figure 4 shows that TangleNAS has better anytime performance than AutoFormer. Architecture design insights In transformer spaces reducing the mlp-ratio in the initial layers has a relatively low impact on performance (Figure 5 ) and can often work competitively or outperform handcrafted architectures. This observation is consistent across ViT and Language Model spaces. Conversely, number of heads and embedding dimension have a significant impact on the quantitative metric of choice. Pruning a few of the final layers also has a relatively low impact on performance. In the MobileNetV3 space, we find a strong preference for larger number of channels and larger network depth. In contrast, we discover that scaling laws for transformers may not necessary apply in convolutional spaces especially for kernel sizes.We observe that earlier layers favor 5x5 kernel sizes while later ones prefer 7x7 kernel sizes (3 being the smallest and 7 the largest).Effect of pretraining, fine-tuning and retraining We study the impact of inheriting, fine-tuning, and retraining on the CIFAR10 and CIFAR100 datasets on the AutoFormer space. We observe that retraining almost always outperforms fine-tuning and inheriting. This brings to question the rank correlation between the inherited accuracy and retraining accuracy of two-stage methods and the training interference (Xu et al., 2022) in two-stage methods. Strong rank-correlation is desirable especially in two-stage methods as these methods rely on the performance proxy upon inheritance for the black-box search. Furthermore, we observe that while the SPOS+RS and SPOS+ES methods are performant upon inheritance, TangleNAS outperforms these models after fine-tuning and training from scratch. Improved supernet training properties for single-stage methods is further supported by Centered Kernel Alignment (CKA) analysis (Kornblith et al., 2019) (Table 25 ).ImageNet-1k pre-trained architecture on downstream tasks Finally, we study the impact on fine-tuning the best model obtained from the search on downstream datasets. We follow the finetuning pipeline proposed in AutoFormer and fine-tune on different fine and coarse-grained datasets. We observe from Table 6 that the architecture discovered by TangleNAS on ImageNet is much more performant in fine-tuning to various datasets (CIFAR-10, CIFAR-100, Flowers, Pets and Cars) than the architecture discovered by SPOS. 
conclusion: 
 In this paper, we investigate two of the most widely used NAS methods: single-stage and two-stage, each traditionally applied to different types of search spaces. We extend the applicability of singlestage NAS methods to weight-entangled spaces, a domain primarily explored by two-stage methods.By doing so, we aim to bridge the gap between these two distinct sub-fields. We empirically evaluate our proposed single-stage method, TangleNAS, on a diverse set of weight-entangled search spaces and tasks, showcasing its ability to outperform conventional two-stage NAS methods while enhancing search efficiency. Furthermore, we highlight the advantages of introducing weight-entanglement into traditional cell-based spaces, whenever feasible. Doing so not only improves the speed of single-stage methods but also reduces memory consumption. The modular nature of TangleNAS enables it to accommodate any differentiable optimizer and hence take advantage of the continually-improving set of methods for gradient-based NAS. Our results on various macro-level search spaces are already very encouraging and we therefore hope that this line of work will enable the NAS community to further improve modern architectures like Transformers. 
 
 Appendix: https://github.com/karpathy/nanoGPT 


Decision: All reviewers agree with the contribution of this work. Authors have also carefully prepared the rebuttal and addressed most of the issues mentioned in the review.

Reviews: review: paper_summary: paper_summary: This paper proposes a novel scheme to adapt gradient-based methods for weight-entangled spaces.The findings highlight that this approach combines the advantages of gradient-based methods with the memory efficiency of weight-entangled spaces, bridging the gap between these two sub-communities in NAS research.
main_review: main_review: # Strengths:
- Paper is well-written and addresses important research questions. 
- TangleNAS is evaluated on a diverse set of weight-entangled search spaces and tasks including language models.
review_summary: review_summary: Minor modifications are required for the paper to be considered finalized.
paper_summary: paper_summary: Conventionally, neural architecture search over macro search spaces is conducted using a Blackbox optimization algorithm such as evolutionary algorithm combined with weight entanglement. This is conducted in two stages. On the other hand, NAS on micro search spaces is conducted with weight sharing approach over the super-net and using a gradient based optimization algorithm. This is conducted in a single stage. The authors present an approach to combine the gradient-based weight sharing approach with the weight-entanglement approach. The authors propose their algorithm as Tangle-NAS and have conducted comprehensive survey to present its credibility.
main_review: main_review: The authors address a topic in traditional NAS which has not been covered upon for a long time, i.e. the combination of both micro and macro search spaces with a suitable strategy. While the area is niche, the approach taken by authors is obvious. The paper is well-written except for the formulation section. More details in the Formulation section can benefit the readers. 

I also believe that huge emphasis was laid on experiments rather than developing the formulation. It is my opinion that the proposed method is an obvious combination of two traditional NAS methods and I do not see a significant novel contribution in terms of search space design or search strategy. Having said this, I appreciate the authors for the exhaustive and detailed experiments conducted that provide a contemporary review of Weight Sharing and Weight Entanglement in NAS. The authors have included LLM in their study which is a positive sign, however, a discussion on the adverse effect of high computational resources required for NAS on Generative AI would have benefitted the readers. Perhaps a discussion on how the current method be modified to avoid the (re)training of massive (pre-trained) Generative AI might have provided a broader scope for the method. 

While the research community and AI/ML practitioners can still perform NAS on traditional AI, the same cannot be said for Generative AI. Thus, the addition of Generative AI as a case study in the work appears to be ornamental. Instead of this, if the authors would have added works with NAS on complex Vision tasks such as Image Enhancement (rather than classification alone), it would have shed light on some of the topics in NAS that remain less explored. 

The conclusions are ambitious. While the authors claim that with their proposed approach, consumes less energy, they don't show any evidence for the same. The fact that high computational resources are required by black box methods such as evolutionary algorithms for performing NAS is established over a long time. Comparing the proposed approach with such methods (in their vanilla form) and claiming a large amount of energy saving may not be beneficial and does not signify anything new to the readers working in this domain. Further a marginal improvement in resource-requirement compared to existing gradient based NAS methods will not help researchers and practitioners to eliminate manual tuning completely. 

Nevertheless, the work presented will provide a significant starting point for researchers in the domain of NAS and I appreciate for conducting a thorough study with the given idea.
review_summary: review_summary: The contribution by the authors is a very thorough study on the proposed idea of combining Weight Entanglement with Gradient based NAS. While the approach to do the same is not entire novel, the experiments conducted will provide several insights to the researchers working in the domain. The only minor flaw that I see is the lack of any new search strategy or method for search space design and a combination of existing methods.
paper_summary: paper_summary: This paper extends weight sharing common in NAS with weight entanglement to allow more intricate weight sharing. Specifically, the authors propose two major modifications to weight-sharing by 1) sharing the weights on every edge with the largest operator rather than the mixed ops to reduce the supernet size and 2) propose weight-superposition by zero-padding the operations to the largest operation. The authors then empirically show their method across a comprehensive set of search spaces to show their effectiveness over previous weight-sharing-based NAS approaches.
main_review: main_review: Overall, I think the paper presents a reasonable and straightforward modification to the weight-sharing NAS setup. Extensive experimental validation was also performed across many search spaces, ranging from the classical CNN search spaces in early NAS papers to emerging paradigms such as Autoformer and language modelling. While the idea may be viewed a straightforward adaptation of ideas from another community to NAS, I do think the work is solid and merit acceptance. My only concern is that while comprehensive and I commend the authors for their effort, the extent of gain seems marginal across many search spaces, possibly also due to the fact that some of the search spaces are already rather saturated, but I'd appreciate if the authors could provide some comment on this.
review_summary: review_summary: The paper is well-written and presents solid research ideas that are of interest to at least some members of the AutoML community. The experiments are very thorough, and I recommend acceptance of this paper.
paper_summary: paper_summary: Weight entanglement is not directly compatible with gradient-based NAS methods, so the authors would like to bridge these two parts. The authors claimed their findings reveal that this integration of weight-entanglement and gradient-based NAS brings forth the various benefits of gradient-based methods while preserving the memory efficiency of weight-entangled spaces.
main_review: main_review: I couldn't ignore the improved performance of the paper. But the presentation of the paper should be improved, as I said in the "Clarity."

Meanwhile, I didn't see many novelties in this paper. The paper proposed an algorithm incorporating gradient methods and weight entanglement techniques for neural architecture search problems. For both gradient methods and weight entanglement techniques, the authors just use the existing ones without many novelties. The paper probably is cited only in the NAS domain and might have a huge impact on the whole AutoML domain due to their hybrids of existing techniques.
review_summary: review_summary: Please refer to the "Overall Review".