Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a method for enforcing strict orthogonality of convolutional layers, by means of a factorization in the spectral domain. It shows that the technique can be extended to dealing with practical convolutions with strides, dilations and groups. The experiments demonstrated the superior performance in terms of adversarial robustness.
main_review: Orthogonality is an important problem in the design of neural network architecture that relates to many fundamental properties of the network such as trainability, generalizability and robustness. While the study of orthogonality in fully connected layers (or convolution layers with 4D kernels treated as 2D matrices) has a long history, it is only until very recent (in the past 2-3 years) that work on orthogonality of *convolution* layers emerges. This paper provides a solid study in this area by providing a method of enforcing orthogonality in convolutions, revealing its technical connections with previous methods, designing a deep residual Lipschitz network architectures and conducting solid experiments. I find the presentation to be mostly clear and easy to follow, though I feel that there is a tendency of overclaiming the contribution in the abstract & intro, see below.

- Complete parameterization of orthogonal convolution. The paper claims that it offers a complete parameterization of orthogonal convolution, but this is not really the case. As stated in Sec. 2, it only offers complete design for *separable* orthogonal 2D convolutions. This puts the technique in an unfavorable position compared to previous methods that does not require separability (e.g. Trockman & Kolter). 

- "Orthogonal Networks". The paper frequently use the phrase "orthogonal networks", but it is not clear what that term entails. For example, it is claimed that "Our versatile framework, for the first time, enables the study of architecture designs for deep orthogonal networks", which seems an overclaim since orthogonality in neural networks have already been extensively studied before. In addition, if "orthogonal network" means that the entire network is an orthogonal transformation, then this is kind of a useless network since orthogonality implies linearity (as long as it is surjective). If it means approximately orthogonal then it should consider, in addition to the convolutional layers, the effect of the nonlinear layers - right now there is no discussion of whether the GroupSort that is used as the nonlinear layer is approximately orthogonal or not. 

summary_of_the_review: Solid work on orthogonality of convolution, though there seem to be some overclaiming / imprecise statements in the intro/abstract that may be misleading.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a theoretical framework for orthogonal convolutional layers based on the equivalency of orthogonal convolution in the spatial domain and the paraunitary systems in the spectral domain. The proposed method parametrizes the orthogonal convolution layer as compositions of multiple convolutions in the spatial domain, resulting in exact orthogonality. The layers are also more memory and computationally efficient than most previous methods. 
main_review: **Strengths**
- The proposed method is theoretically grounded and relatively efficient in computations.
- The analysis of strided, dilated convolution layers is inspiring. 
- Numerical evidence on orthogonality evaluation of different designs for standard convolution shows that exact orthogonality is achieved.

**Weaknesses**
- It is nice to see that exact orthogonality is achieved however it remains unclear exact orthogonality is actually helpful or needed. For example, from Table 2, the proposed SC-Fac achieves the most "accurate" orthogonality result in worse performance in both certified and practical robustness. Even though the author claims that the method achieves "comparable" results with other baseline methods, the results are consistently worse than the baselines.  

- The authors can compare their core idea with related work that is more heuristic, such as [1] which also considers achieving orthogonality in the spectral domain, as well as [2],[3].

[1] Liu et al. Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training

[2] Wang et al. Orthogonal Convolutional Neural Networks

[3] Bansai et al. Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?

- Even though the method is more computationally efficient, it is only compared with methods such as Cayley which is known to be computationally heavy. The method is still much computationally heavier than the original networks. It would be nice to have an extra line in Figure 4 showing the train time of the ordinary network. 
summary_of_the_review: The proposed method achieves exact orthogonal convolutional layers through re-parametrization. The method is theoretically grounded and easy to understand. Numerical proofs are provided to show exact orthogonality is achieved by composing a sequence of learnable convolutions.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper suggests a framework for designing orthogonal convolutions using paraunitary systems. The proposed framework studies orthogonalization of different convolution operations. The proposed methods were examined on several datasets in comparison with state-of-the-art orthogonalization methods.
main_review: The paper proposes a nice framework which covers different orthonormalization methods for convolutional kernels. In the analyses, the proposed framework performs on par with the state-of-the-art.

However, there are several issues with the paper. In general, first, some of the claims should be revised since they are not verified in the analyses. Second, experimental analyses should be improved with additional analyses on additional datasets and tasks in comparison with the other state-of-the-art methods.

More detailed comments are given as follows:

1. The paper states that “However, many previous approaches are heuristic, and the orthogonality of convolutional layers is not systematically studied.” However, there is a nice literature on orthogonal CNNs, which explores these CNNs from various aspects including generalization and convergence properties of models, in various tasks including image recognition, speech processing and NLP, including adversarial robustness studied in this paper. Then, could you please describe the systematic study referred to and proposed in this paper, which is not covered in the literature?

2. Please explain the statement “There are mature methods that represent orthogonal matrices via unconstrained parameters.” How does this enable to optimize parameters of orthogonal convolutions using standard optimizers instead of optimizers designed for optimization on orthogonal convolutions?

3. There are some issues with the definition and interpretation of orthogonal layers. First, an orthogonal layer is defined according to preservation of norm of input and output. This can be achieved using different types of parameters of convolutions, even with scaled Gaussian parameters. In addition, orthogonal convolutions proposed in the literature satisfy some particular orthogonality properties of matrices of parameters. Second, the orthogonal convolution proposed in this paper is associated with paraunitary property of the transfer matrix.

4. In the experimental analyses, in most of the results, state-of-the-art outperforms the proposed method, while in some results, the proposed method outperforms them. To show a more clear benefit of the proposed method over the state-of-the-art, could you please perform analyses on additional larger scale datasets such as Imagenet? Could you please also compare the proposed method with the methods which employ orthogonal matrices?

5. It is proposed that “related works proposing orthogonalization methods do not guarantee Lipschitzness after training”. However, the proposed orth. conv. employ deep Lipschitz networks to guarantee Lipschitzness. If the proposed orth. conv. does not employ deep Lipschitz networks, then does it  guarantee Lipschitzness? 

6. How do you optimize “learnable (column)-orthogonal matrices”?

7. While training models, how do you estimate and optimize h[z], H[z], ortho. Factors, and model params? In the code, Adam is used to optimize parameters. However, it is not clear how orth. factors are also optimized. Do you also optimize them using Adam?

8. How do you apply z-transform on input, kernel and output. For instance, if an input NxN image x is convolved with a 7x7 kernel h, then how do you apply z transform on x and h? That is do you apply path wise or holistically? Also, if x’ is a feature map of size CxWxH, where C is the number of channels, W and H are weight and height of the map, then how do you apply the z-transform on the map?

9. How do you compute ortho. factors efficiently?

10. How do you calculate model parameters A?

11. In the experiments, the proposed methods perform similar to the state-of-the-art. To show superiority of the methods in comparison with the state-of-the-art, additional analyses on larger scale datasets and models should be provided.

summary_of_the_review: The proposed framework is nice, and the initial results are promising. However, there are various unclear parts in the paper. In addition, some of the claims are not verified and experimental analyses are limited. Therefore, the paper should be improved with additional analyses and in detail revision for clear acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a new method for orthogonalizing the convolutional layers by exploring the equivalence between spatial orthogonal and spectral paraunitary. The work then empirically demonstrates the effectiveness of the proposed methods by comparing (1) the Lipschitzness (2) the results of adversarial robustness and (3) the time and memory cost among different methods. 

main_review: This work proposes a new method for orthogonalizing the convolutional layers by exploring the equivalence between spatial orthogonal and spectral paraunitary. The experiments are conducted on various networks including the shallow KW-large networks and slightly deeper WideResNet22. Although the reviewer does not check the submitted code in detail, the code is well-written and clearly commented. 
The major concerns are (1) the work seems to have made some overstatement of the contributions, claiming that all the previous work are heuristic, and the proposed approach is systematic with theoretical justification. The reviewer does not quite buy this point, and better explanation on this is needed; (2) the experimental results are not consistently showing the advantages of the proposed method, also the improvement in terms of computational efficiency seems to be marginal.

Below are some more detailed comments.

1. The reviewer found the implementation of the proposed method somewhat hard to follow, it could be better to incorporate an algorithm view of the method to clearly present it. 

2. In the paper, the Q matrix is defined as an orthogonal matrix that is randomly initialized and fixed during training. But the reviewer didn’t find the associated implementation in the code (correct me if I missed anything), so the reviewer is wondering how the Q matrix is constructed in the experiments.

3. When demonstrating the results of adversarial robustness, the paper devotes to $\ell_2$ norm based attacks. The reviewer is curious about the results of  $\ell_\infty$ based attacks. 

4. The reviewer notices that in the code, when considering striding, the authors include 2 use cases [stride_wide, stride_slim], the reviewer is curious about the actual definition of the different use cases. Besides, the code of the proposed method mentions that the kernel size should be a multiple of stride (in the stride_wide case, this constraint is bypassed by letting kernel_size = kernel_size * stride), the reviewer would appreciate it if this part is presented in more detail. (An algorithm for handling different cases would be nice). 

5. This paper is missing quite a few citations on related work:

https://arxiv.org/abs/1810.09102

https://arxiv.org/abs/2103.00673

https://arxiv.org/abs/1911.12207

https://arxiv.org/abs/1905.11926

Please refer and discuss the relationship
summary_of_the_review: The paper is well-presented overall. However, better positioning of the work, and more convincing experimental results are needed.
review_title: null

Meta-review: This paper proposes a method for parameterizing orthogonal convolutional layers that derives from paraunitary systems in the spectral domain and performs a comparison with other state-of-the-art orthogonalization methods. The paper argues that the approach is more computationally efficient than most previous methods and that the exact orthogonality is important to ensure robustness in some applications.

The reviewers had diverging opinions about the paper, with most reviewers appreciating the theoretical grounding and empirical analysis, but with some reviewers finding weakness in the clarity, reproducibility, and discussion of prior work. The revisions addressed many, but not all, of the reviewers' criticisms.

One point that was highlighted in the discussion is that the method is restricted to separable convolutions. The authors acknowledged this limitation, justifying the expressivity of the method with a comparison to CayleyConv (Trockman & Kolter) and a suggestion that more expressive parameterizations are not necessarily available in 2D. I am not sure this is entirely accurate. In the discussion of related work, the paper briefly mentions dynamical isometry and the prior work of Xiao et al. 2018, who develop a method for initializing orthogonal convolutional layers. What the current paper fails to recognize is that Algorithm 1 of Xiao et al. 2018 actually provides a method for parameterizing non-separable 2D convolutions: simply represent every orthogonal matrix in that algorithm in a standard way, e.g. via the exponential map. While I think there is certainly value in the connection to paraunitarity systems, it seems to me that the above approach would yield a simpler and more expressive representation, and is at minimum worth discussing.

Overall, between the mixed reviewer opinions and their lingering concerns and the existence of relevant prior art that was not discussed in sufficient depth, I believe this paper is not quite suitable for publication at this time.
Decision: Reject