Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: This paper focuses on downsampling layers, such as pooling and strided convolutions, in convolutional neural networks (CNNs). The authors analyze the performance of independently trained models with different pooling configurations on CIFAR10 using a ResNet20 network. They find that the position of downsampling layers significantly impacts network performance and that predefined downsampling configurations are not optimal. They also show that traditional one-shot Network Architecture Search (NAS) based on a single SuperNet does not work for optimizing pooling configurations due to parameter sharing issues. To address this, they propose a balanced mixture of SuperNets that associate pooling configurations with different weight models, reducing weight-sharing and inter-influence of pooling configurations on the SuperNet parameters. Their proposed approach outperforms other methods and improves over default pooling configurations on CIFAR10, CIFAR100, and Food101 datasets.
overall_review: Paper strength: 
1.	The motivation of the paper is interesting and clear.  

Paper weakness: 
1.	The presentation of this paper is not good and some details are not explained clearly. 
2.	The experiments are not convincing. 
3.	More in-depth analysis is expected. It is not enough to show comparable / better results than existing methods.

review_summary: Overall, this research contributes to the understanding of the impact of downsampling layers in CNNs and presents a novel approach for optimizing pooling configurations, which could have practical implications in the field of deep learning.
actions_required_to_increase_overall_recommendation: 1.	The work has studied the search for the optimal network under a fixed number of parameters. How about when you fixed the FLOPs?
2.	The transferability of the found network should be studied. At least, networks found over CIFAR10 should be evaluated on other datasets.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak impact, and/or weak evaluation.
review_content: summary_of_contributions: This paper found the existing one-shot NAS algorithms cannot be natively applied to find optimal pooling configuration. This paper hypothesized that the existing weight sharing results in this problem. So, a mixture of supernets is proposed. They use multiple supernets rather than a single supernet to estimate the accuracy of a pooling configuration. The proposed algorithm is demonstrated on CIFAR datasets and Food101. 
overall_review: This paper noted the configuration of pooling layers will affect performance. A new one-shot search algorithm is proposed to search for pooling layer configuration. Multiple supernets are used to estimate the performance of sub-net. 

As I mentioned, this paper has problems with experiments and hypotheses. I agree the configuration of pooling will affect the final performance to some extend. I am not sure 1) the proposed algorithm can effectively outperform baselines and 2) that similar experimental result will be observed on large-scale datasets and harder tasks. 
review_summary: The hypothesis and experimental results are not solid and convincing. The algorithm contribution is weak. 
actions_required_to_increase_overall_recommendation: Please try to address my major concerns about experiments.
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: The authors explore the impact of downsampling layers on the performance of CNNs and highlight the limitations of current NAS methods for optimizing downsampling configurations. The authors propose a balanced mixture of SuperNets to address these limitations and show that this approach outperforms other approaches and default pooling configurations on CIFAR10, CIFAR100, and Food101 datasets. The authors emphasize the importance of appropriate search space design and present a solution for learning the optimal scale of CNN feature maps through pooling/stride configurations. The proposed method reduces weight-sharing and inter-influence issues associated with pooling configurations, which leads to better ranking correlations. Finally, the authors also compare their proposed method with non-NAS-based approaches.
overall_review: The authors should clarify if this is the first kind of work to use NAS for the position of downsampling layers. The authors should probably take a look at the following paper and consider explaining the difference between their work: https://ieeexplore.ieee.org/document/9506777 

It may be a good idea to clarify in the search space subsection how the authors obtained 36 configurations for Resnet20 instead of placing it in Section 3.1. 

The GPU search time is not reported, and hence, the authors should also compare the computation time. The time required to train 36 architectures vs the search time of the proposed algorithm would be good to present in the paper. This can signify the importance of the search method. 

The experiments are limited. The results on large-scale datasets and resolution are missing. At least, the authors should have evaluated the mixture of supernets on the Imgenet-16-120 dataset. The authors experimented only on the Image Classification task and CNN-based network. Currently, Vision transformers are being utilized on various tasks and slowly replacing CNNs. Hence, the proposed search method should be tested on different kinds of architecture and tasks to truly show its effectiveness. 


review_summary: The presentation of the proposed approach is good. However, more evaluation is required to justify the correctness of the paper. The reviewer is fairly confident of the review after having gone through the search space and search algorithm. The review covered essential aspects of the paper, including the novelty and evaluation of the proposed approach.  
actions_required_to_increase_overall_recommendation: The authors should enhance the paper by adding more experimental results in terms of datasets and architectures. 

The accuracy aspect is fine, but the authors should also include FLOPs or the number of multiplication and accumulation (MAC) operations. Late pooling has better accuracy compared to early pooling, as shown in Figure 1. However, networks with early pooling have lower FLOPs and MACs due to a reduction in spatial resolution early in the network. The subsequent layers operate on lower spatial resolution feature maps. From Table 3, the baseline methods seem to work better than the proposed method in this paper. Therefore, FLOPs and MACs can be good for enhancing the paper. 

Minor:
Line 135: r^L, L (number of layers) is not defined before. It should be clarified. 

Line 136-137: therefore it is reasonable to exclude upsampling operations from the search space.: This statement seems to be obvious and redundant. The authors established already that they are searching for optimal downsampling operations. 

The reference on line 656 is missing.
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: This paper studies the placement of layers that reduce spatial dimensionality in convolution-based neural architecture search (NAS) search spaces for the vision domain. These decisions are often treated as fixed in such search spaces, often with a predefined number of blocks before each spatial pooling location. The authors first observe that standard weight-sharing-based NAS techniques fail to recover this optimal placement, and use a ResNet-20 backbone on three vision datasets as a testbed for this problem – this setup admits a small search space of only 36 architectures and allows for convenient evaluation of the de facto choices made by standard vision backbones, which turn out to be suboptimal. The authors propose a new weight-sharing NAS method that targets this issue–balanced mixtures of supernets, which reduces the amount of weight-sharing between the different supernets. The authors evaluate their method on three vision datasets and show that their method improves performance by searching over different pooling placement choices alone. 
overall_review: Strengths:
- The authors have identified an important problem in the NAS literature that appears to have been mostly understudied by prior work. 
- The balanced mixture of supernets idea appears to work for this problem – it is also interesting that prior weight-sharing methods fail. 
- A ResNet-based search space for studying this problem is extremely simple and seems to be exactly the right place to start for this problem. 

Weaknesses:
- It would be great to include an evaluation of other search spaces besides the ResNet-based search space in order to show that the results hold up when using other blocks. However, I do not view this as a deal breaker because the current results are interesting on their own. 
- One thing that seems missing is that since the proposed method is no longer single-shot in the usual weight-sharing sense, it could make sense to report training times and parameter counts compared to weight-sharing methods. While this is understandably not the point of the work, this is useful information. 
- Additional datasets would be nice to have, since the evaluation comprises only three well-known vision datasets. 

Other comments:
- Another interesting point that this work points out is the limitations of weight-sharing-based NAS. It would be interesting to study these limitations further. 
- Lines 340-341: “One could argue that NAS in general are a waste of computation, however, they help to avoid an even more expensive validation search for the optimal hyper-parameters of the model” – I’m not sure I agree with this, can the authors comment on this view? 

review_summary: The problem that the authors have identified is important, and their experimental design seems very appropriate. The resulting method also seems to work, while prior work fails on this simple search space, which is interesting. The evaluation could be strengthened by extending to additional backbones beyond ResNet, a comparison of search timings to standard weight-sharing methods, and additional datasets. 
actions_required_to_increase_overall_recommendation: - Additional backbone evaluations
- Timing comparisons
- Additional datasets (lower priority)
review_title: null

final_score: rating: 9: Strong Accept: Technically flawless paper with major impact and strong evaluation, with no obvious flaws. Should be highlighted in the program.
review_content: summary_of_contributions: This paper discusses the importance of downsampling layers in CNNs and their impact on network performance. The study shows that predefined downsampling configurations are not optimal and proposes a new approach of a balanced mixture of SuperNets to optimize pooling configurations. The proposed method outperforms other methods and improves the default pooling configurations in CIFAR10, CIFAR100, and Food101 datasets.
overall_review: 1. The method only achieves marginal improvement compared to the previous method, which is a very important baseline, DiffStride. 
2. The paper lacks a report on the efficiency of the proposed method, such as the number of GPU hours required for the experiments.
3. The experiments in the paper were limited to CIFAR-level and food-101 datasets, which are relatively small. It is recommended that the author conduct experiments on larger datasets, such as ImageNet, to demonstrate the potential of the proposed method.
review_summary: Please refer to the comments above. Considering the objective limitations, I am inclined to recommend a borderline leaning reject.




actions_required_to_increase_overall_recommendation: Please see the comments above.
review_title: null

Decision: Accept(Main Track)