Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper studied the adversarial robustness from the neural architecture perspective. In particular, the authors introduced a dilation network to increase the overall capacity of the neural network, so that it can simultaneously handle the clean examples and adversarial examples very well. A performance constraint and a FLOPs constraint are developed to guarantee the introduced dilation network can indeed benefit the accuracy and robustness with the minimal FLOPs involved. The authors have done extensive experiments to evaluate the proposed algorithm. 
main_review: In Line 121, the authors explained that the dilation network has been searched within a NASNet-like search space. In fact, there are quite a few different search spaces in the literature. Why the authors adopt such a search space? Further, is that necessary to customize a certain search space for the problem studied in this paper?

The authors suggested that the backbone network has a fixed architecture. But why not also do some modifications to the backbone network?

I agree it is necessary to investigate the FLOPs constraint when designing the dilation network. But why this FLOPs constraint is implemented in the formate like Eq. (11). How about directly include a separated item about FLOPs in the objective function?

limitations_and_societal_impact: Nothing particular
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper focuses on improving the adversarial robustness of the backbone CNNs with good natural accuracy. Different from those
adversarial training methods, this paper employs the dilation architecture to increase the capacity of the backbone network with minimal computational overhead. This paper provides both theoretical analysis and empirical results.
main_review: Originality: The authors leverage neural architecture search approach to dilate the backbone network, in order to improve adversarial robustness while maintaining natural accuracy. The dilation has interesting motivation to provide adversarial robustness with minimal computational overhead.

Quality: 
1) My main concern is that the robustness of the proposed method achieved is not significantly better than prior works. I list the detailed suggestions in the Limitations setction.
     a) I suggest the authors include results under AutoAttack for all tables, since AA is stronger and can provide a more general prospective view. Besides, AA would play the biggest role after combining the four attackers, so the authors are suggested to report the standard version of AA instead of splitting it into four attacks and reporting the performance under each respectively. 
     b) It seems that no improvement is achieved on top of TRADES-6. For example, in Table 1, the average between nat and pgd-20 is 70.675 while the average of NADAR-B is 69.83; in Table 2&3, it is lack of TRADES result on CIFAR-100 and Tiny-ImageNet. 
     c) The authors are suggested to compare the same group of competitors on all datasets and under different scenarios to demonstrate the effectiveness of their proposed method.

2) Another major concern is the computational cost during training.  After reading the experiment part, an intuitive question comes into my mind: what is the benefit of dilating an existing graph? Should the training process be faster than standard adversarial training? The authors are suggested to provide more evidence, either on saving training cost or improving performance, to demonstrate the motivation of their proposed method.

Clarity: This paper is well organized and clearly written, which allows an easy read. The abstract and introduction directly point
out the focused research direction, the main contributions, and the overview of the paper structure.
limitations_and_societal_impact: Yes, the authors addressed the limitations.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper proposes a neural architecture search that learns a residual (called dilation in the paper) to an existing network in order to improve the robustness to adversarial attacks. The loss of the search is based on improving the robustness to the attack and retaining the original network accuracy. They show improvement for wideresnet for various datasets. 
main_review: The idea of using NAS for improving robustness is very interesting. Yet, as the authors mention, it was already done before. 
While this does not eliminate the contribution of the paper, there are several problems in the current form of the paper that should be addressed before it fits for publication:

1. Theory: The authors presents some results that are supposed to support the proposed approach. Yet, these results are very straight forward to achieve so they are not really novel. But even if they are novel, they don't really analyze the proposed approach. First, They give only an upper bound to an upper bound of the loss. Second, they analyze a different architecture than the one used, which is basically a summation of two networks, which is not the case in this work. So, therefore, the theory part is not really relevant or explains the success. So right now it does not really support the paper results but just present some inequalities that I am not sure if they are helpful to something else or not. This should be shown.

2. The empirical results are limited due to the following two reasons:
a. In any architecture search there should be comparison to the best random architecture selected from the search space. This comparison is missing. 
b. There is no comparison to adversarial training of other searched architectures. The table in the paper just show their results without adversarial training. This is not a fair comparison.
c. Is the improvement just because of  the search or because of the adversarial training? Did the authors tried to use TRADES also for the other NAS based approaches for adversarial training?
d. Why not all results are reported for DARTS?

3. Ablation should be done for searching for the whole architecture with the current loss with the same number of parameters?

4. What if you use a larger resnet that has the same number of params like the resnet+search and train it with DARTS?

5. Finally, in the paper Intriguing properties of adversarial training at scale (ICLR 2020) it is shown that using larger networks improve adversarial training. So basically, is not this just explain the reason why the adversarial training works better here as the network is bigger? All the experiments I suggested above should be done to prove that the success achieved is not merely due to using a larger network but that the specific used search is really the source of improvement 



 
limitations_and_societal_impact: not related
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a method termed Neural Architecture Dilation for Adversarial Robustness  (NADAR) which adversarially searches for a dilation architecture for the backbone network. Beside the dilation, a standard performance constraint is also introduced to protect the standard accuracy on clear samples form dropping too much, and a FLOPs-awareness mechanism is utilized to lower computational overhead. Theoretical analyses and emperical results validate that NADAR can improve adversarial robustness at almost no cost of generalization.
main_review: Pros:
1.	The paper is well organized. Studying adversarial robustness from the perspective of neural architecture dilation is novel to solve the well-known accuracy-robustness trade-off, and is orthogonal to the adversarial training methods studied from the view of objective and optimizer.
2.	Comprehensive experimental results show that the dilation can effectively improve adversarial robustness, and the generalization performance constraint can guarantee a competitive standard accuracy. Those experimental results can support their main claims. 

Cons:
1.	In Table 1, please explain the meaning of “Arch.” and “Back.”. It is better to clearly show the percentage of capacity has been increased by NADAR compared to the backbone network. 
2.	It is better to report robust accuracy against stronger attacks, e.g., CW attacks[1] and AutoAttack, in Table 1-3. 
3.	The paper only discusses how to apply NADAR on ResNet-like backbones. It seems to be a limitation of this paper. Please provide some discussions about utilizing the proposed method based on the backbones those do not fit the ResNet-like architecture.

[1] Towards Evaluating the Robustness of Neural Networks. SP 2017
limitations_and_societal_impact: Limitations: Please carefully refer to Main Review.

Potential negative societal impact: This paper does not discuss the potential negative societal impact. However, I think the proposed method raises computational cost and requires larger storage sources, thus posing potential risks to environmental protection.
review_title: null

Meta-review: The paper proposes a neural architecture search method to improve the adversarial robustness of the backbone networks while maintaining good natural accuracy. The key idea is to introduce a dilation architecture to increase the capacity of the backbone network with minimal computational overhead. The paper is well motivated and organized. The new solution is novel and technically sound.

However, there are several concerns on the technical details. 1) The theoretical analysis is not specifically related to the proposed method on dilation of neural networks. While analysis of a general function has some implications, neural networks have their special properties which are studied extensively in the subarea of deep learning theory. The analysis would be strengthened if the analysis could be more specific to the neural network architectures used in the proposed method. 2) The benefits of using bigger networks have been pointed out in the literature, it is not clear whether the performance improvement is due to larger networks or the search procedure or both. More analysis is needed to really show that search architecture is very important even with larger spaces, otherwise, the main contribution would not be strong enough because the idea of using larger networks is not originally proposed in this paper. 3) The proposed method is based on PGD, which is not a very recent adversarial training method. It is unclear whether the proposed method can still maintain the advantage when using more advanced adversarial training methods, especially those that have already considered the accuracy on natural data.

Overall, the paper is novel and interesting. However, given the above concerns, the novelty and significance of the paper will degenerate. Addressing the concerns needs a significant amount of work. Although we think the paper is not ready for NeurIPS in this round, we believe that the paper would be a strong one if the concerns can be well addressed.

Decision: Accept (Poster)