Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper proposes a network model named EVPNet, inspired by the idea scale-space extreme value from SIFT, to improve network robustness to adversarial pertubations over textures. To achieve better robustness, EVPNet separates outliers (non-robust) from robust examples by extenting DoG to parametric DoG, utilising truncated ReLU, and then applying a projected normalisation layer to mimic PCA-SIFT like feature normalisation, which are the three novelties that the authors claim in this paper. In the experiments, FGSM and PGD are used to provide adversarial attacks, and experiments conducted on CIFAR-10 and SVHN reveal that EVPNet enhances network robustness.
Overall, this paper contributes to network robustness from an architecture perspective; in the contrast, most prior works focus more on robust feature extraction and loss function design. The ablation study of EVPNet demonstrates the effectiveness of its each novel component. A further investigation presents that EVPNet reduces the error ampplification effects. The example that the authors show in this paper demonstrates the improvement of EVPNet to image textures.

The reviewer has some main concerns regarding the claimed novelty:
1. pDoG computes the difference between outputs of two depth-convolution layers, but there is no evidence that the distribution of feature maps is gaussian or gaussian-like. There is no clarification for this point.

2. Truncated ReLU is a modified ReLU. Does the learnable truncated parameter \theta limit its applicability to different datasets?

3. The Projected Normalisation Layer (PNL) seems a reasonable implementation, but essentially it is not very different from batch normalisation. The authors state only its difference to global average pooling but not to batch normalisation which should be a better comparison.

For the experiments, the following should be addressed:

4. Experiments were conducted only for the SE-ResNet architecture via replacing its CNN kernel by the proposed EVPNet. Although SE-ResNet shows good performance on some common data, but the squeeze-excitation block might bring in non-robustness. Hence, the reviewer thinks that it is risky to claim: the replacement of EVPNet in CNN layers is robust to adverserial attacks based only on this implementation. Try EVPNet for a more basic network architecture (VGG) would be suggested.

5. The \epsilon, which represents the adversarial attack tolerance, is always 8. There is no explaination in this paper, why not other values.

Minor comments:
6. The authors did not clarify why the first novel component is called "parametric" DoG. There is a more "non-parametric" block.

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper presents a SIFT-feature inspired modification to the standard convolutional neural network (CNN). Specifically the authors propose three innovations: (1) a differences of Gaussians (DoG) convolutional filter; (2) a symmetric ReLU activation function (referred to as a truncated ReLU; and (3) a projected normalization layer. The paper makes the claim that the proposed CNN variant (referred to as the EVPNet) demonstrates superior performance as well as improved robustness to adversarial attacks.

Clarity: Overall, the paper is not particularly well written. There are multiple missing articles and other grammatical errors that make it a bit arduous to read, though I do not believe they have obstructed my ability to understand the contributions. The section describing the projected normalization layer (second half of page 5) is a bit confusing. Figure 2(c) is not helpful in shedding light on the details, though I think a more detailed figure could be quite helpful. Beyond these issues, the paper is relatively clear in the presentation of the material. 

Novelty: Over the last few years there have been many, many proposals for how to vary the basic CNN architecture to improve performance. Some of these have lead to genuine performance gains and have become part of the standard CNN specification. ReLUs, ResNets and Batch Normalization are particularly prominent examples of contributions that have been shown to lead to improvements in performance. Yet the vast majority of these sorts of proposals ultimately make little or no impact on the field. In light of this, I would rate the novelty of the basic goal of this paper as relatively low, though the specific proposal is novel to me and seems reasonable. 

Impact: The impact potential for this paper lies with the performance offered by the proposed innovations. With respect to overall performance improvement the proposed method has not been shown to perform quite at a state-of-the-art level, as given by these resource:
SVHN: https://paperswithcode.com/sota/image-classification-on-svhn
CIFAR-10: https://paperswithcode.com/sota/image-classification-on-cifar-10

The authors compare the performance of their proposed EVPNet against a fair baseline - a squeeze-and-excite ResNet model. These sorts of controlled experiments are useful, but the actual reported performance for both models are somewhat off of the state-of-the-art and it's not clear that the relatively small benefit the authors show over their baselines are maintained for higher performing architectural configurations. Can this architecture be competitive with the state-of-the-art? The current paper in it's current 

Most of the results relate to the claim that the proposed model is robust to adversarial examples. Unfortunately, this is not a particular area of expertise for me, so it's difficult for me to provide a confident assessment of the contribution here, though I will say two things: (i) the method seems to provide a significant increase in adversarial robustness across the
baseline architectures investigated. (ii) the authors demonstrate that the benefit provided by the proposed architecture seems to persist even when training for adversarial defence is introduced. 

I would have liked to see more datasets explored in Experiments section. I  especially would have liked to see results on ImageNet. 

My current rating is weak reject based on the weakness of the writing and the lack of strong empirical evidence in support of the effectiveness of the proposed contributions.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: In this paper, a new network architecture called EVPNet was proposed to improve robustness of CNNs to adversarial perturbations. To this end, EVPNet employs three methods to leverage scale invariant properties of SIFT features in CNNs.

The proposed network and the methods are interesting, and provide promising results in the experiments. However, there are several issues with the paper:

- The authors claim that Gaussian kernels are replaced by convolution kernels to mimic DoGs. However, it is not clear (1) how this replacement, or employment of convolution kernels can mimic DoGs, or (2) more precisely, how the corresponding learned convolution kernels approximate Gaussian kernels. In order to verify and justify this claim, please provide detailed theoretical and experimental analyses.

- It is also claimed that “a 1 × 1 conv-layer, can be viewed as a PCA with learnable projection matrix”. However, this statements is not clear. How do you assure that a 1x1 conv layer  employs a PCA operation or the corresponding projection?

- What does \| \|_p denote? Does it denote \ell_p norm?

- What does x denote in d = w x h? Previously, it was used to denote matrix size.

- Why do you compute \ell_2 norm for row vectors instead of column vectors? How do the results change when they are calculated for column vectors?

- According to the notation, s_0 and s_1 are vectors. Then, what does max denote in (14)? That is, how do you compute max(s_0, s_1), more precisely?

- In the statement “PNL produces a hyper-ball in the manifold space”, what do you mean by the “manifold space”? What are the structures (e.g. geometry, metrics etc.) and members of this space?

- Please conceptually and theoretically compare the proposed method with state-of-the-art methods following similar motivation, such as the following:

Weng et al., Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach, ICLR 2018.

review_title: Official Blind Review #3

Meta-review: This manuscript proposed biologically-inspired modifications to convolutional neural networks including differences of Gaussians convolutional filter, a truncated ReLU, and a modified projected normalization layer. The authors' results indicate that the modifications improve performance as well as improved robustness to adversarial attacks.

The reviewers and AC agree that the problem studied is timely and interesting, and closely related to a variety of recent work on robust model architectures. However, this manuscript also received quite divergent reviews, resulting from differences in opinion about the novelty and importance of the results. In reviews and discussion, the reviewers noted issues with clarity of the presentation and sufficient justification of the approach and results. In the opinion of the AC,  the manuscript in its current state is borderline and could be improved with more convincing empirical justification.
Decision: Reject