Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The main contribution of this paper is to propose Adaptive Channel Dropout (ACD), which performs channel dropout before the covariance computation to obtain good trade-off between representation decorrelation and information preservation. ACD is more efficient and more effective than existing  post-normalization techniques. Since the core of ACD is dropout, it is only used during training, and is not needed in inference. There are other minor contributions, such as Adaptive Power Normalization (APN), which is a slight improvement over matrix power normalization (MPN) [34,42]. The paper also performs more analysis of MPN.
strengths_and_weaknesses: Strengths
 - ACD is more effective and efficient than existing post-normalization techniques.
 - more extensive analysis of alpha for MPN

Weaknesses
 - One of the main contribution is equation (5), but the intuition/explanation is not very clear. Some more theoretical discussion would also be helpful.
 - The current presentation makes it hard to understand what are the main contributions. There are many things going on: analysis of MPN, LogM. APN. ACD... My take is ACD is the main contribution, but would be helpful to make it clear in the paper
 - The main advantages of ACD seems to come from the fact that it is a dropout-based technique. The fact that its faster and is not required in inference are because it is based on dropout, which is quite different from post-norm/pre-norm. It would be helpful to compare to more regularization techniques in training.
questions:  - more (theoretical) discussion of equation (5)
 - clarify contribution and even rewrite the paper to make the core contribution more obvious
 - comparison to other training regularization techniques that prevent overfitting
limitations: Checklist points to the conclusion, but the conclusion does not really address the limitations and potential negative societal impact of the work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Motivated by the facts of 
(1) the post-normalization of GCP lacks theoretical understanding and 
(2) the high computational complexity of GCP normalization, 
the paper proposes:
- a theoretical analysis of matrix power normalization in the GCP networks and extend it to an adaptive normalization method
- an adaptive channel dropout to reduce the computational complexity.

The proposed method has been verified on ImageNet and iNaturalist datasets using various scale architectures. 
strengths_and_weaknesses: Strengths
- The proposed method is simple and efficient for classification tasks. 
- Experiments are solid on multiple datasets, especially have experimented with different scale architectures.
- Good theory analysis with ablation studies.
questions: - Is the GCP the bilinear pooling? I think it could be better to explain these two academic terms and illustrate their relationships in the paper.
limitations: None.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper focuses on the post-normalization of the pooling strategy leveraging second-order statistics, i.e., global covariance pooling (GCP).  The key contributions are twofold: 1) the paper shows that post-normalization in matrix power normalization [27] tries to balance representation decorrelation and information preservation through empirical analysis; 2) the paper proposes a linear complexity pre-normalization DropCov of GCP based on an adaptive channel dropout. 

The proposed approach has demonstrated its effectiveness on several image classification benchmarks including ImageNet, ImageNet-C, ImageNet-A, stylized ImageNet, and long-tailed classification with different backbones (CNN, transformer). 

strengths_and_weaknesses: **Strength**: 
* [S1]: I think the main strength of this work is the experiments. The paper provides diverse experiments on different datasets with different backbones to validate the proposed DropCov. 

* [S2]: The paper is presented very well and reads very well. I read quite carefully but did not find any typos. The paragraphs and sections are also self-contained and organized well.

* [S3]: The analysis of MPN (Sec.2.1) is intuitive, and the empirical results are interesting.

**Weakness**:
* [W1]: I think the main drawback of this work is the significance. I can acknowledge the improvement is clear, the idea is simple. But as GCP is computationally more expensive than 1st order pooling, not much attention is paid to the field.

* [W2]: I think the analysis on MPN is a little shallow. the experiments are on a single dataset with a network. It would be interesting to see the effect of power on different sizes of datasets with different capacities of models. 
 
* [W3]: For the adaptive power normalization, the improvement with the proposed formulation is quite marginal. Moreover, I am wondering whether extra visualization would provide some insight. For example, it might be possible to visualize samples with important/small alpha to verify corollary 1 further.

* [W4]: Equation 5 is complicated and not intuitive. For example, the cosine similarity between the feature importance and feature correlation is weird. Also, this similarity might be negative and cause the probability larger than 1? Equation 6 might be confused as well. The dropout should be only applied once on $X$ (not separately for $X$ and $X^T$)?

* [W5]: It would be more convincing to conduct different dropout ratios (rather than a fixed ratio of 0.5) to validate the effect of adaptive dropout. 

* [W6]: It is not apparent to understand the intuition behind the experiments of “element” and “channel” dropout. For example, in lines 275-276, the paper emphasizes applying one type of dropouts is equivalent to considering only feature correlation or feature importance. It would be better to provide a more detailed explanation on this part.  




questions: Please refer to the weaknesses.

limitations: No potential negative societal impact.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper focuses on studying matrix power normalization in global covariance pooling operation. The authors provide a detailed analysis in the effect of different power parameters on matrix power normalization as well as other post normalizations, in terms of feature decorrelation and information preservation, empirically showing the necessary of using a good tradeoff between the two properties. To reduce the computation complexity of post normalization, the authors introduce Dropconv which performs channel dropout to achieve feature decorrelation. Experiments show the ability of the proposed strategy  on performance improvements on both ResNet and ViT backbones.
strengths_and_weaknesses: Strengths:
1. The authors provide a detailed analysis in the effect of the power parameter of matrix power normalization on feature decorrelation and information preservation, and consequential impact on model performance empirically, which is valuable for related research area. 

2. The paper is generally well written and easy to follow, although some statements need to be explained and clarified.

3. Extensive experiments on representative traditional and modern backbones are conducted to validate the method. 

Weaknesses:
1. Some descriptions are less rigorous and need to be clarified:

The first corollary (line 120) states that setting power parameter 0.5 is optimal according to Eq. 2 and Fig. 1, while it lacks either theoretical proof or comprehensive empirical studies. Is the behavior of power parameter fairly invariant to dataset or network backbone? 

For the proposed Adaptive Power Normalization, the power parameter is optimized according to eigenvalues of feature covariance, while the features are updated during training. How does it realize in practice? There is a similar question for the dropout probability in Eq. 5. Could the authors provide their training behavior if they are variable during training? If the parameters are adaptive with model training from scratch, primitive parameters may be far from optimal values. On the other hand, if the variance of probability distribution is rather small, it may indicate less necessary for tuning it during training. 

2. The motivation of performing dropout on convolution channels is not new. The main difference is specializing the strategy with global covariance pooling. 

3. Equipping DropCov into baseline backbones (e.g., ResNet50) brings significant increase on parameter size and computational overhead. 
questions: 1. For the experiments in Table 2, the counterpart of DropChannel seems analogous to ACD except that the probability in ACD is determined by features, while the performance gap is so significant and even higher than DropElement. Could the authors analyze the phenomenon? How does the DropChannel and DropElement perform when using different dropout probability beyond 0.5?

2. It is unclear to me why computational complexity is linear for DropCov while quadratic to element-wise methods (in Table 1).

limitations: The paper analyzes post-normalization for global covariance pooling unit and introduces channel dropout to improve it, which is not restricted to certain downstream cv tasks. It seems like no potential negative societal impact.
review_title: null

Meta-review: metareview: Both reviewer Fzo6, reviewer VWPn and reviewer 5eq4 have concerns and been questions regarding equation 5. Please clarify the clarifications on the paper and add intuition and more discussion of Eq. 5. 

The paper and comments from the authors indicate that dropout base regularizations are effective (Maxdropout, Maxout and Decov also outperforms GCP). This does mean that a large part of the benefits of the proposed method (no inference processing, lower complexity) are the result of that dropout/regularization in training, which takes away from the contributions. 

Overall, I think the paper is borderline, leaning to acceptance, as the proposed DropConv does out perform other dropout/regularization methods and I believe the paper might benefit the community. I'd strongly encourage the authors to review their manuscript and address the reviewers’ concerns as best as possible in the revised manuscript. 
confidence: Less certain
Decision: Accept