Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper improves and aggregates techniques for 1-Lipschitz CNNs with respect to the $l_2$ norm. First, the authors approximate the gradient for Skew Orthogonal Convolutions (SOC) so the throughput is increased. Second, the paper cooperates Curvature-based Robustness Certificate (CRC) into CNN to replace the last linear layer of the model. Third, it introduces a novel 1-Lipschitz pooling layer which is learnable. Combining these three techniques achieves better standard/robust accuracy on CIFAR-10/CIFAR-100 with a shorter training time.
strengths_and_weaknesses: - This paper provides a good summary of previous techniques and some non-trivial improvements to them. 
- The majority of the empirical gain comes from applying CRC to the last layer, which is a decent empirical contribution but not completely novel.  
- The new pooling layer is pretty novel as far as I know, but the performance gain is not consistent. Especially for CIFAR-100, the performance drops except for LipConvnet-5. Therefore, I am not sure whether it is a good contribution or not.
- Faster gradient computation for SOC is novel and the speedup brought is notable. However, I am not sure if it is useful in practice because the decrease of provable robust accuracy is not negligible.
- Some writing is not clear enough, especially for readers who are not familiar with previous works. See my questions below. I suggest thorough proofreading.
- Figure 1(a) is very similar to Figure 3 of [Singla and Feizi, 2021]. Please reference.
questions: - Can you also report the state-of-the-art by other methods, e.g., randomized smoothing?
- The best 36/255 provable robust accuracy on CIFAR-10 is achieved by LipConvnet-5, which is very different from previous results where the best is LipConvnet-20. Do you have any insights into the change?
- In which part of the network do you apply the ``identity map'' in Line 217? Or do you mean that the gradient of the objective is computed as the gradient of $\ell(h_{\Omega}(g_{\Phi}(x) + \epsilon, l)) + \cdots$ where $\epsilon$ is the adversarial perturbation?
- Does "+CRC" rows in tables include faster gradient computation? If not, what is the performance change for "+CRC +Fast"?

Minor:
- Line 49, there is no LipConvnet-$n$ in Table 1.
- Eq (7), there is a missing bracket.
- Line 212, why is $h$ a parameter of $g$?
limitations: I do not see the negative societal impact of this work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The work proposes 3 methods that could be used to improve the training of the 1-Lipschitz networks from different aspects: (1) an approximation approach for the calculation of gradients in SOC layers which makes the method more time-efficient; (2) changing the final linear layer of a network to an MLP and specifically apply adversarial training on the MLP yields superior performances; (3) a more expressive pooling method compared with max-pooling. The work then empirically demonstrates that using (1) results in time-efficient models with no significant sacrifice in performance and using (2)(3) together boost both natural accuracy and robustness of trained models.
strengths_and_weaknesses: Strengths:
The overall presentation is clear and easy to follow. The relationships with prior arts are detailly discussed. Also, the empirical results show that the proposed methods are indeed helpful. 

Weaknesses:
The reviewer did not find any major flaws in the paper. 

questions: In the experiment section, it is not entirely clear how the individual components of the proposed methods contribute to the final results. Currently, the message is (1) the fast computation of gradients mostly makes the performances drop; (2) using all 3 proposed methods increases the performances, and (3) from Table 5 and 6 in the Appendix, it seems like using the projection pooling alone also sometimes sacrifices the performances. So does this mean that all the gain is from the usage of MLP? A more detailed ablation study could be done here to better investigate the effect of each individual component.
limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents techniques for accelerating and improving the training of 1-Lipschitz CNNs for certified $\ell_2$ robustness. Specifically, the paper proposes an approximate gradient computation procedure for skew orthogonal convolutions (SOC) which reduces the training time of SOC-based CNNs. The paper also presents a trainable 1-Lipschitz projection-based pooling layer and proposes to replace the last linear layer of 1-Lipschitz CNNs with a certifiable 1-hidden layer MLP for improved robustness. The combination of these techniques improves the state-of-the-art provable $\ell_2$ robustness on CIFAR-10 and CIFAR-100.
strengths_and_weaknesses: Strengths:
- The paper is well motivated and well written. The paper is clearly organized, and the proposed methods are explained pretty well. The theoretical results are clearly stated and all the proofs are provided in the Appendix
- The experimental results seem to back the author's claims, in terms of demonstrating the improvement in certified robustness, however I have some questions regarding the efficacy of the pooling layer (see below)

Weaknesses:
- Going deeper with LipConvnet (as n increases) decreases the natural and robust accuracies consistently, which is counter-intuitive. I understand this is not due to the paper's proposed method, but it makes me wonder why go deeper with these networks in the first place?
- The claim 1-Lipschitz is not technically correct, due to the exponential approximation performed in SOC, is this why going deeper hurts performance?
- The method is restricted to $\ell_2$ norm robustness
- The pooling layer seems to be counter-productive, as it decreases the robustness (see Tables 5 & 6 in the Appendix) in a fairly consistent manner. 
- This is a bit unusual, but I have a found a lot of text similarities between this paper and that of the cited work [1]. A decent portion of the text seems to be copied or poorly paraphrased (e.g. Intro, parts of background). I understand the paper seems to be inspired by [1], as the titles are also similar and the methodology (improved last linear layer, improved non-linearity) are aligned. But, in my opinion, this does not justify such 'lazy' writing.


[1] Singla, S., Singla, S., & Feizi, S. (2021, September). Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100. In International Conference on Learning Representations.
questions: 1) How exactly is the certified robust accuracy computed? Is it just the fraction of test inputs whose certificate is smaller than the perturbation strength? It would be better to include this in the main manuscript. 

2) Continuing on the pooling layer, it seems to me from Tables 6 and 7 that adding the proposed pooling layers decreases performance. Why is that? and if it is the case then why is it needed? Does it work well in conjunction with the MLP layer? (i.e., is +MLP + pool better than +MLP with standard pool?)

3) Is there a reason why the HouseHolder activation not being used in this work? The authors are aware that it provides improvements in robustness. Does it not stack well with the proposed methods?

4) How long does to take to certify using the proposed methods? and how does it compare to the baselines used in the paper?

limitations: Paper does not properly address the limitations, e.g. what is the overhead of certification?
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes an 1-Lipschitz CNN with improved standard and provable robust accuracies on CIFAR-10 and CIFAR-100 with 1) faster gradient copmutation of SOC layers; 2) 1-hidden layer MLP instaed of last linear layer; 3) 1-Lipschitz pooling layer called projection pooling. 
strengths_and_weaknesses: Strengths: 
* Originality: The proposed methods for improving 1-Lipschitz CNN on standard and provable robust accuracies is original. 
* Quality: The introduced methods for faster gradient computation of SOC layers using approximation, as well as using projection pooling with 1-hidden layer MLP instead of last linear layer is well motivated and justified.  
* Clarity: The paper is clearly written with necessary details. 

Weakness: 
* Significance: The authors mentioned by themsleves that the proposed CRC-Lip for certifying robustness is computationally expensive even for CIFAR-100, this may limit its applicability to broader problems (e.g. ImageNet and etc.) 
questions: * This paper ends abruptly without a conclusion section. Is this intended? 
* CRC-Lip looks like an effective way to boost performance on 1-Lipschitz CNN, can it scale to larger problems ? 
* Comparison only includes LipConvnet, are there any other relavent works not mentioned? 
limitations: Assumptions and Limitations are mentioned in the paper, would be hepful to have a dedicated summarization.
review_title: null

Meta-review: metareview: This paper introduces several new techniques to improve the $\ell_2$ adversarial robustness of CNNs, including approximating the gradient for Skew Orthogonal Convolutions (SOC), replacing the final linear layer with a 1-hidden-layer MLP, and introducing a new class of pooling layers. These techniques lead to improved efficiency and robust accuracies on CIFAR-10 and 100.

All reviewers agree that this paper has made interesting and solid contributions, and find the paper well-written. Therefore, I recommend it to be accepted to the conference.
confidence: Certain
Decision: Accept