Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a training framework named Layer-wise Orthogonal Training (LOT) that aims to train 1-Lipschitz convolution layers effectively and improves the certified robustness of Lipschitz-bounded models. The paper also proves that semi-supervised learning can benefit the robustness of Lipschitz-bounded models. The key idea of LOT is to utilize unconstrained matrices to approximate an orthogonal matrix and to speed up the inverse computation by mapping the matrices to the FFT domain. The experiments are done on CIFAR-10/100, taking LipConvnet with a different number of layers as the backbone, showing LOT outperforms its directly competing framework called SOC. The main contributions are:
1.	LOT parametrizes the orthogonal weight matrix W with an unconstrained matrix V, and uses FFT to calculate the inverse square root of matrices faster in the frequency domain.
2.	This paper derives the certified robustness of the Lipschitz constrained model under the semi-supervised setting.
3.	LOT improves certified L2 robust accuracy against SOC.

strengths_and_weaknesses: Strengths: 
1. The scale-invariant convolution kernel parametrization can improve robustness further than SOC.
2. The improvement in the Lipschitz is backed by analytical derivations.

Weaknesses: 
1. The derivation of this paper is specific to l2 attacks only, while attacks are conventionally taking l1, l2 and l_\infty norms. In fact, the benchmarked SOC paper also shows empirical robustness in other norms.
2. Only CIFAR10/100 test results are provided, which limits the practicality of the framework.
3. In Sect. 3.2, the semi-orthogonality in the linear operation context is fine, but then its convolution counterpart sounds hand-waving. Is the latter mathematically equivalent to its linearized counterpart? Can you use a toy example to show how to bridge the former to the latter? If they are not directly corresponding, then a proof that conv(W, W^T)=I_{conv} is needed.
4. The definition of W in terms of V requires multiple steps of convolution plus inversion in the FFT domain (and thus complex numbers). I wonder, once V is trained and fixed, why can’t W be computed exactly in terms of V, and then the convolution be done in one step using W? Is that due to non-exactness in the orthogonality or what?

questions: Further comments:
1. Sect. 5 on the theory is quite involved in terms of math, while its conclusion/discussion is rather trivial. It sounds a bit like “if it’s good then it’s good”.
2. The same issue happens for Fig. 1 and its discussion in Sect. 6.3, too. Why are the features from LOT “more meaningful” than those from SOC? Simply because the patterns have higher amplitudes and fluctuations?
3. Upper half of p.4 shows the FFT-domain operations. I suppose the matrices are holding complex numbers? How does that affect the complexity and stability of the Newton iterations? And how do you guarantee real matrices when mapped back to the spatial domain?
4. I wonder if your Lipschitz constraint can be imposed on the doubly block Toeplitz convolution operator in the OCNN paper [24] and alike? So that your convolution doesn’t need to be in multiple steps with high complexity and low numerical stability?
5. One very key issue I have spotted from the paper is the lurking instability in training/inference, as the authors need to deliberately jet up the accuracy to FP64 during training/inference and then trim back to FP32. This is likely due to the inverse operation which can become ill-conditioned. This may also be why only CIFAR10/100 examples are used, without moving on to the larger sizes like 224^2 ImageNet examples? In fact, do you have larger examples beyond 32^2 input sizes?
6. Following from above, it sounds impossible to quantize your network to INT8, INT4 or even binary as in other CNN compression works?
7. Last but actually the most important, the experiments are only conducted on LipConvnet which is NOT conventionally used in practice. People in the field know very well that robustness is very much architecture-dependent, where transformers exhibit both high accuracy and robustness. The authors may want to study the leaderboard at http://robust.art/ , to see how real-world networks perform under field tests on ImageNet-class inputs. This in turn renders the LipConvnet characterization of this work purely theoretical yet, frankly, with little practical value as the generalization of the results to the more common modern architectures is doubtful (ConvNeXt, Transformers, etc.). 

Editorial:
1. It is inappropriate to write V=\alpha V in several places throughout the paper.
2. line 65. “[23] first certify …”, certifies
3. line 70, “Recently, [2] show that …”, shows

limitations: See my questions above.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposed a method that utilizes the convolution theorem and Newton iteration to achieve 1-Lipschitz in convolutional layers of CNN and theoretically justified the effectiveness of semi-supervised training on improving empirical robustness.
strengths_and_weaknesses: Strengths: The overall presentation of the work is clear and easy to follow. Although the reviewer did not check all the proofs, the theoretical side of the paper is complete and carefully organized. The paper also talks about the actual implementation (e.g., dealing with padding/stride, etc) and the limitations of the method in great detail.

Weaknesses: 
1. The novelty of the method is somewhat limited. For example, calculating things in the Fourier domain is not new, at least the Cayley transform paper [1] cited in the work and [2] utilize similar idea; also, using Newton iteration to approximate matrix inversion is also explored in previous works [3] of the same domain. 

2. The experimental results are not super convincing. The paper mainly conducts experiments to compare with SOC and claims that "SOC has been shown to outperform other 1-Lipschitz models". However, the SOC paper mostly only compares with BCOP. I believe a more comprehensive comparison here would make the paper more convincing (at least compare with [1] and [3] which inspired this work?)
questions: My major concerns of the paper are stated in the weaknesses section. I would appreciate it and am willing to raise my score if the authors could illustrate more the comparisons between at least [1][2][3] and the proposed method, both theoretically (could be intuitive) and empirically. 

[1] Asher Trockman and J Zico Kolter. Orthogonalizing convolutional layers with the Cayley transform. In International Conference on Learning Representations, 2021.

[2] Liu, et al. Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training. In Advances in Neural Information Processing Systems, 2021.

[3] Huang, et al. Controllable orthogonalization in training DNNs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a method to train a 1-Lipschitz neural network to achieve improved certified L2 robustness by ensuring that the weight matrix in each layer is orthogonal. The weight matrix W is parameterized using an unconstrained matrix V as W = (V V^T)^(-1/2) V to ensure orthogonality. This parameterization is invariant to rescaling of the unconstrained matrix V, which the authors show leads to better robustness and expressiveness of the model. The paper also shows how to make convolution operations orthogonal. Lipschitz constraints have been shown to improve the empirical robustness of a model. The aim of this work is to show that such models can achieve better certified robustness as well under the semi-supervised learning setting.
strengths_and_weaknesses: Originality: The methods used in the paper such as parameterizing orthogonal weight matrices with unconstrained matrices have been considered before. However, the paper presents a new kind of parameterization that has desirable properties such as scale invariance.

Quality: The paper considers several different convolution operations such as circular padding, zero padding, larger strides, etc and shows ways to orthogonalize them. The paper performs a theoretical analysis to demonstrate the benefits of semi-supervised learning for certified robustness. The method has been evaluated and compared with a previous paper in this area for different model architectures.

Clarity: The paper is clearly written and easy to read.

Significance: The method achieves improved performance over the existing approach of Skew Orthogonal Convolutions (SOC).
questions: 1. The matrix V does not really seem to be unrestricted since it needs to satisfy the condition: || I - VV^T|| < 1. Also, it is not clear to me how scaling V can always make it satisfy the condition as stated in line 154, e.g., consider V = zero matrix in a fully connected layer.
2. How do we initialize V? Do we set it to a particular type of matrix such as an orthogonal matrix?
3. How do we ensure the condition on V is satisfied after each training step?

limitations: There does not seem to be any negative societal impact of this work.
review_title: null

Meta-review: metareview: This paper proposes a training framework named Layer-wise Orthogonal Training (LOT) that aims to train 1-Lipschitz convolution layers effectively and improves the certified robustness of Lipschitz-bounded models. The paper also proves that semi-supervised learning can benefit the robustness of Lipschitz-bounded models. 

All reviewers agree that this work is new, and the empirical improvement is significant. I follow the majority to recommend acceptance.
confidence: Less certain
Decision: Accept