Venue: AutoML 2022 Track Main

Reviews:

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: The paper introduces a new one-shot neural architecture search method that addresses the issue of scale variance in computer vision problems by allowing more flexibility in the multi-scale aggregation and network depth. To ensure an efficient exploration, the authors propose grouped sampling to help the supernet training and multi-scale topology evolution to help the architecture selection. The method is evaluated on semantic segmentation and pose estimation and the authors report  improvement over existing related methods.
overall_review: strengths:
The paper introduces a new method that allows using one-shot supernet for typically resource intensive computer vision tasks like semantic segmentation. It shows the importance of multi-scale fusion not only at the last layer but also for the intermediate layers and extends the auto-deeplab design to support this property  as well as flexible bloc depth. Furthermore, it enable multi-path search for better space exploration.

weaknesses:
Minor formatting issues and typos (missing bold font for table 3, Cityscape on line 250) 
review_summary: This paper focuses on an interesting topic and proposes two  components that may facilitate the research area. The proposed methods are reasonable and well evaluated. I think this work is a good submission and should be accepted.
review_title: null

final_score: rating: 6: Strong accept, should be highlighted
review_content: summary_of_contributions: The main contribution of ScaleNAS is that it presents a novel way of utilising multiple representations of different scales by employing grouped sampling and evolutionary search. Doing so, they significantly boost the robustness of the segmentation/pose estimation models they discover since they make the representations scale-invariant, which is crucial for real-world applications. The authors also design a flexible search space to explore different depths of network layers for different scales. 

Arguably, the most important contribution of the paper is its novel multi-path one-shot search method. They address the caveats of current one-shot methods (namely the limitation for training on a single path) by proposing a multi-scale aggregation search space, which in turn unveils new challenges like the difficulty of exploring the new colossal search space. To overcome this they use elite architectures, evolutionary approaches and group sampling.
overall_review: The paper introduces a novel way to substantially reduce the needed computational power needed for NAS and also presents an application of NAS to pose estimation, which is also a niche area. It combines multiple state-of-the-art methodologies to achieve what seems to be a similar way to Miao, Caijing, et al.'s "Sixray: A large-scale security inspection x-ray benchmark for prohibited item discovery in overlapping images.", but instead of helping a model just to have different auxiliary scales for the sake of addressing item crowding, they introduce a flexible search space containing a multitude of models and manage to beat the state-of-the-art in two separate common datasets with a strong performance increase. 

I believe the findings of the paper are of tremendous importance for the field, yet I would like to point out that using one-shot learning with SuperNets can lead to networks of immense size, which makes training such networks not only harder and slower but also hinders the general interpretability of networks. The approach can be benefitted from the introduction of a pruning method to discard paths that are not promising, which can also help with the reduction of the search space, which even though reduced is quite big (which is normal for any NAS approach). 
review_summary: A novel method of incorporating clustered sampling and evolutionary computation with scale-aware representation search is presented. The approach is well explained with numerous different experiments proving its efficiency and also ablation studies for some of the components. 

The code is of good quality and although some comments/tests and other qualitative components are missing or neglected, they are compensated with a well structured, clear and elaborate README file, which makes it easy for practitioners with the right hardware to rerun and even build upon the results of the authors. 

Overall the paper is written of a professional standard and reads like a Google or Facebook research paper.
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: The paper presents ScaleNAS, a Neural Architecture Search method, to find an optimized architecture for visual recognition tasks. 
The search space of the method is *multiscale*: each branch can have different depths, as opposed to searching the overall depth for the entire network, and each feature maps can be randomly connected to each of the other scales of feature maps.
To sample a novel architecture, the authors propose a *grouped sampling* method. The search space is divided in several subspaces (here 9) to cover the full space, and sampling is done from each of the subspace.
At the beginning of training, the model and a teacher model are initialized by the largest architecture from the search space pretrained on ImageNet.
At each iteration of training, a novel architecture is sampled following the grouped sampling approach, and trained using the true labels of the given task and soft labels from the teacher network. The full trained model is called *SuperScaleNet*.
After training, the best subnetworks from *SuperScaleNet* are searched using an evolutionary method, that allows for crossover of modules between subnetworks and mutation of connections. 
The experiments are done on the Semantic Segmentation and Human Pose Estimation tasks, where subnetworks found by ScaleNAS outperforms state-of-the-art methods on each task.
overall_review: ### Positive

- The training of a single super network (one-shot training) and sampling subnetwork greatly reduces the total computation cost of the method. 
- The architectures found outperforms state-of-the-art methods on each task (semantic segmentation and human pose estimation), with lower number of parameters and GFLOPs.
- Both Group Sampling and Topology Evolution are more effective searching strategies and improve over other used strategies.

### Negative

- The method is not really *multitask*, as it trains a super network and find subnetworks tailored for a given task. Even though the search space and the method is the same for each task, the architectures found are not the same between tasks.
review_summary: The method presented in the paper incorporates several techniques that improve the searching and sampling strategies for Neural Architecture Search. The paper also show that one-shot training for multi-path and high resolution tasks can achieve state-of-the-art results for a lower total computation cost.
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: The authors propose a one-shot NAS method to learn multi-branch architectures with high-resolution representations that can be applied to different visual recognition tasks. To this aim they design a search space based on HRNet with learnable depth and branching pattern. The search is done in two-stages consisting of supernet training and architecture search. For supernet training, they propose grouped sampling to enhance exploration during the sampling of architectures from the supernet. For the architecture search, a multi-scale topology evolution is proposed with a crossover step that acts between stage modules of the network. Due to the two-stage procedure, this also allows to extract all pareto optimal architectures instead of a single architecture as in previously proposed methods.
overall_review: Strengths:

1. Search space: The design of the search space around HRNet allows it to be applied to many visual recognition tasks, in particular also those that require high-resolution features. In addition, the search space has a very flexible macro-architecture with learnable branching pattern.

2. Experimental results show good performance improvements over other NAS algorithms and baselines on several datasets while still maintaining a relatively low search cost which is further amortized in the case of many deployment scenarios.

Weaknesses:

1. The search space is very flexible regarding the macro-architecture (i.e., the structure within the different stages) but in its current form does not consider the micro architecture (e.g., operations or width of hidden representations). In addition, scale variance and high-resolution representation learning based on HRNet has e.g., been explored in HR-NAS (https://openaccess.thecvf.com/content/CVPR2021/html/Ding_HR-NAS_Searching_Efficient_High-Resolution_Neural_Architectures_With_Lightweight_Transformers_CVPR_2021_paper.html) which has a less flexible branching pattern in each stage but can e.g., learn the number of feature maps in the different resolution branches. This work is unfortunately not mentioned at all and a detailed comparison with advantages and disadvantages of ScaleNAS compared to HR-NAS would be important. In addition, the main strength, namely the high flexibility in the macro-architecture is not really discussed in the experiments in the main paper. How many branches are e.g., used in the top-performing architectures from the experiments?

2. Grouped sampling and the sandwich rule: Grouped sampling is phrased as an alternative to the sandwich rule. However, isn’t it rather an alternative to the random sampling of an architecture which is done as part of the sandwich rule? I.e., couldn’t the largest and smallest architecture still be sampled in addition to the architecture sampled with grouped sampling? Is there something like the “smallest” architecture in the case of the proposed search space?

3. Grouped sampling procedure: Grouped sampling is motivated by stating that the architecture distribution is approximately normal according to the central limit theorem. However, the architecture distribution is a complex compound distribution, and it is not directly clear to me why the central limit theorem applies. E.g., for SPOS, this is not the case as all architectures are equally likely. The description of the grouped sampling for branch depth is also not quite clear to me (see also “Clarity” section). Does the grouped sampling cover the whole search space?

4. Relation to MixPath: One results of MixPath (https://arxiv.org/abs/2001.05887) is that if you sum the feature vectors from multiple path those are close to multiples of the feature vectors from different paths which is causing training instability. Is this an issue in your search space as well as the number of inputs to a feature map is not fixed and why? If not, wouldn’t it be possible to directly compare to MixPath as you already have the trained teacher network where you could add the architecture parameters?

5. Discussion and comparison of some relevant related work seems to be missing, e.g., a.)	HR-NAS (see point 1) b.)	The grouped sampling strategy that divides the search space into smaller sub-search seems to be similar to Few-shot NAS (https://arxiv.org/abs/2006.06863) and there are also other approaches such as AttentiveNAS (https://openaccess.thecvf.com/content/CVPR2021/papers/Wang_AttentiveNAS_Improving_Neural_Architecture_Search_via_Attentive_Sampling_CVPR_2021_paper.pdf) that address the sampling in the sandwich rule. Again, the paper would benefit from these a more detailed discussion and comparison with these approaches. c.)	FasterSeg (https://openreview.net/forum?id=BJgqQ6NYvB) could be added as an additional multi-branch method

Minor:

1. Reading the formulation “ScaleNAS solves multiple tasks at a time” (line 10) could be mistaken to think that the method is about multi-task networks and not of the fact the method can be applied to multiple tasks

2. Are 40 deployment scenarios (line 259) realistic? Even for much smaller numbers the proposed approach already needs less training time.

Spelling:

•	line 239: “accuraices” -> accuracies

•	line 259: “s” -> is
review_summary: The authors tackle the problem of applying one-shot NAS to learn high-resolution representations for different visual recognition tasks and demonstrate improved performance on several datasets compared to other NAS methods. The main weakness in my opinion is the missing comparison (or at least discussion) to the very similar HR-NAS paper, the limited micro architecture of the search space and the clarity of the grouped sampling approach. This leads me to reject the paper in its current form.
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: The paper describes a NAS solution for dense prediction tasks, such as semantic segmentation and human pose estimation. The primary intellectual difference is the search space, where the authors considered a multi-stage, cross-scale, generic search space. Experiments are performed on benchmark datasets in semantic segmentation and human pose estimation. 
overall_review: I do believe that NAS-for-dense-prediction is relatively overlooked compared against NAS-for-image-classification. And the best way to do multi-scale is the primary difference. This paper considers a new search space that is more general and challenging than before. Though the proposed approach has flavors from many existing works such as OFA, BigNAS etc, I think the overall it is a good contribution to the AutoML community. Ablation studies are performed to show the impact of both proposed components (grouped sampling and topology evolution), and the end result performs competitively, especially on human pose estimation. 
review_summary: I recommend acceptance, though I strongly suggest the authors add illustration and discussion of the found ScaleNAS nets for reproducibility and insights.
review_title: null

Meta-review: metareview: This work focuses on neural architecture search (NAS) for dense prediction. The proposed method, called ScaleNAS, considers a multi-stage, cross-scale search space and consists of two phases: supernet training with grouped sampling and multi-scale topology evolution. Results provided for semantic segmentation and human pose estimation show improved performance relative to previous NAS methods.

Reviewers highlighted the significance of NAS for high-resolution representation learning and how the proposed ScaleNAS considers a general and more challenging search space than previous work. Experimental quality is another strength as ScaleNAS is shown to improve performance at a low search cost. Ablations effectively demonstrate the impact of the proposed grouped sampling and topology evolution procedures. Reviewers agreed that the work is generally reproducible.

In terms of weaknesses, the description of the training procedure could be improved.

Overall, this paper is of high quality and likely to be of interest and significance to AutoML community. Therefore I recommend acceptance.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept