Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a novel normalization technique for controlling the channel-wise Lipschitz of the convolutional layers. The normalization technique operates on the Fourier domain which could reduce the burden of doing full SVD and at the same time be more accurate compared to using approximate methods. The paper also empirically demonstrates the effectiveness of such normalization by showing experimental results on robustness under different sources of noises, easier training, and extended usage in GAN.

main_review: I'm quite positive in general. The paper is clearly motivated, well written, and smoothly presented. The method proposed is reasonable and well established. Also, the implementation is straightforward and since there is no need to tune hyperparameters, the method could easily serve as a plug-in module for most deep convolutional networks. The experimental part is comprehensive, the authors tried various sets of tasks to empirically test the performances of the proposed method and other methods. 

One remaining concern is that most experiments in this paper are primarily done on CIFAR-10, it would be more interesting to see more experiments conducted on more complex datasets like Imagenet.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper exploits the convolutional structure to maintain orthogonality on the convolutional kernel for guarantee the energy preservation in CNN, rather than the naively used methods using flattened convolutional kernels. It orthogonalizes the channel-wise convolutional kernel via preconditioning its circulant matrix transferred by FFT. The proposed ConvNorm also reduces the layer-wise spectral norm of the weight matrices. Experiments on small network (e.g, VGG 16 and 18-layer Resnet) using CIFAR-10 ImageNet verify the effectiveness of ConvNorm in model robustness and training GANs. 
main_review: **Pros:**

+The idea to orthogonalize the channel-wise convolutional kernel via preconditioning its circulant matrix is novel to me.   
+The related works are comprehensive discussed, as far as I am concerned.  
+The presentation is well organized  
+The effectiveness of ConvNorm in model robustness and training GANs are well validated by the experiments. 



**Cons:**

1.One of my main concern is the computational or memory cost of the proposed methods. This paper spends much of contents to show the computational problems of the previous methods (Lines 58~61, Line 76) and claims the efficiency of the proposed ConvNorm. However, I did not see analysis about the computational complexity or the time cost in experiments. What is the wall-clock time cost of the proposed method, compared to the baselines? I would like to see the time cost& memory cost in the described experiments, especially on ImageNet.   
2. I am not convinced with the claims“Easier training and better generalization” by the experiments: 1) Clearly, ConvNorm do not work well without BN in such a small network (VGG16-ResNet18); 2) the experiments should be conduct on larger network, e.g, 56-layer or 110-layer ResNet on CIFAR (1GPU is enough to run the experiments) or better using ResNet-50 on ImageNet. I wonder whether ConvNorm can work on 110-layer ResNet without BN. 

**Other comments:**

What is the y-axis of the Figure 2 (Left), the log-scale? It should be described in here. By the way, I believe it is more important to show the layer-wise condition number of the proposed ConvNorm, rather than the channel-wise condition number, because the motivation of orthogonal matrix W is for preserving the magnitude between the output and input, with y=Wx, or $\frac{\partial{L}}{\partial{x}} = W^T \frac{\partial{L}}{\partial{y}}$.

It is not clear why should stop the gradient shown in Line 229-234.  I believe this can work with feature activation (e.g. BN) in a network, but I conjecture it will suffer training instability without feature activations in deeper networks (e.g., 56-layer/110-layer ResNet for CIFAR classification). It is better to provide the results in deeper networks without feature activation. 

**update**:
I have read authors’ responses and other reviewers’ comments. I acknowledge the authors’ feedback which addresses my concerns. I hope the code (including the additional experiments for comparing layer-wise condition number) will be released if this paper gets accepted. I keep my score as 7. 

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduces a new way to enforce orthogonality in convolutional layers. The main idea is to make the computations in the frequency domain where convolutions (or cross-correlations) become products. Hence, exact orthogonality is achieved, while keeping the computational costs low using FFT.
main_review: Major revision

The paper is well written, although some of the computations are not easy to follow. Section 3.1 could be enlarged and better explained. If the authors need more space to do that, the previous section regarding neural networks in general could be deleted. It is only useful because it provides some notation.

The forward pass of the new layer is not very clear in the paper. It could be useful to explain it, because it might seem that you provide a formula for $Q(a)$ and then multiply the $x$ by that matrix, which would require a large computation and which you do not do. It gets more clear in the Appendix, but it is worth mentioning it in the paper. Besides, experiments of the computation time of the forward pass and a comparison with other methods would be valuable in the paper, since you claim that your method is faster and show that analytically.

Minor revisions:

Line 157: "The basic idea is to multiply a preconditioning matrix which approximates the following form": the sentence is not very clear in a first reading, now I understand what you mean, but please rephrase.
Line 177: "However, generalizing our method from a single channel to multiple channels is not obvious, that we discuss below.": again, please rephrase this sentence.

Overall Evaluation:
Apart from simple changes that can be made by the authors, the idea of the authors is clever and the implementation efficient. Besides, numerical experiments prove that their approach is valuable. At the moment, I think the paper is a little above the acceptance threshold and could be further improved with experiments regarding inference time.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes to orthogonalize the convolutional kernel in a channel-wise manner. To do the computation more efficiently, they calculate the matrix inversion via FFT in the frequency domain. They compare with the baselines on the tasks of adversarial attack, image classification and GAN.
main_review: Originality: I think that this paper is the first work proposed to orthogonalize output channels in the frequency domain via FFT. But this is not the first work that orthogonalizes weight in the frequency domain [1]. 

Clarity: 

Pros: The paper is well-written and easy to follow overall. 

Cons: Some typos: 
1. F* is not defined in Eq. 3, though defined in Appendix. 
2. 'Learnable affine transform' in line 235.

Quality: 

Pros: The paper provides the comparison with multiple baselines on multiple tasks, and shows empirical advantages.

Cons: 
1. I think the paper misses the empirical comparison with [1], at least in the task of adversarial attack. The proposed method shares a lot in common. E.g. focus on circular convolutions and Fourier domain, and [1] also shows the advantage in the task of adversarial attack.
2. It not clear how well the proposed method controls the spectral norm and condition number of the convolutional layer, and how well the input norm gets preserved, compared with baselines. Can you provide the spectral norm, condition number and ||Conv(x)||_2/||x||_2 (i.e. the ratio of the output norm and the input norm) for each layer, for ConvNorm and other baselines? (just in adversarial attack task is good enough)
3. I feel the run time is also not clear. Can you provide the run time comparison on ConvNorm and baselines under the different kernel settings (e.g. # channel_in, # channel_out, kernel_size, input_size)? I believe it can provide the community a better view of the advantages and limitations of each method.
4. The paper doesn't compare with orthogonalization-related baselines. And it seems the baselines have a better performance the ConvNorm. For instance, ONI [2] has 70.43 test acc on ImageNet with ResNet18, and OCNN [3] has 78.1 test acc on CIFAR-100 on ResNet18. 
5. What's more, [2, 3] conducts the experiments with more complex architecture (e.g. ResNet 50, ResNet 101), while ConvNorm only shows the results on ResNet 18. So the scalability is my another concern.
6. I am confused about affine transform. It seems gamma_k is a freely learned kernel. I wonder if the orthogonality gets changed after passing affine transformation? How's the spectral norm, condition number gonna change?

Significance: I cannot determine the significance at this moment, since the paper doesn't justify itself completely to me. But I believe I will have a better and more overall understanding if more statistics and comparison can be provided. And I will consider raising my score then. 


[1] Orthogonalizing Convolutional Layers with the Cayley Transform

[2] Controllable Orthogonalization in Training DNNs

[3] Orthogonal Convolutional Neural Networks

===============
post rebuttal: I have read the rebuttal; and raise my score accordingly.
limitations_and_societal_impact: The paper presents the limitations (e.g. theoretical explanation) and leaves them to future work.
review_title: null

Meta-review: In this paper, the authors present a simple and efficient ConvNorm method that can fully exploit the convolutional structure in the Fourier domain. The paper is clearly written and well motivated. Extensive experiments empirically demonstrate the effectiveness of such normalization. As all reviewers achieve consensus that the paper is well written and can be accepted, I vote for acceptance. The authors are expected to make a thorough revision by considering the the reviews.  

Decision: Accept (Poster)