DropCov: A Simple yet Effective Method for Improving Deep Architectures
Authors: Qilong Wang
Mingze Gao
Zhaolin Zhang
Jiangtao Xie
Peihua Li
Qinghua Hu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Global covariance pooling
post-normalization
adaptive channel dropout
deep convolutional neural networks
vision transformers

Abstract: Previous works show global covariance pooling (GCP) has great potential to improve deep architectures especially on visual recognition tasks, where post-normalization of GCP plays a very important role in final performance. Although several post-normalization strategies have been studied, these methods pay more close attention to effect of normalization on covariance representations rather than the whole GCP networks, and their effectiveness requires further understanding. Meanwhile, existing effective post-normalization strategies (e.g., matrix power normalization) usually suffer from high computational complexity (e.g., $O(d^{3})$ for $d$-dimensional inputs). To handle above issues, this work first analyzes the effect of post-normalization from the perspective of training GCP networks. Particularly, we for the first time show that \textit{effective post-normalization can make a good trade-off between representation decorrelation and information preservation for GCP, which are crucial to alleviate over-fitting and increase representation ability of deep GCP networks, respectively}. Based on this finding, we can improve existing post-normalization methods with some small modifications, providing further support to our observation. Furthermore, this finding encourages us to propose a novel pre-normalization method for GCP (namely DropCov), which develops an adaptive channel dropout on features right before GCP, aiming to reach trade-off between representation decorrelation and information preservation in a more efficient way. Our DropCov only has a linear complexity of $O(d)$, while being free for inference. Extensive experiments on various benchmarks (i.e., ImageNet-1K, ImageNet-C, ImageNet-A, Stylized-ImageNet, and iNat2017) show our DropCov is superior to the counterparts in terms of efficiency and effectiveness, and provides a simple yet effective method to improve performance of deep architectures involving both deep convolutional neural networks (CNNs) and vision transformers (ViTs).

DropCov: A Simple yet Effective Method for Improving Deep Architectures
Qilong Wangh Mingze Gao', Zhaolin Zhang', Jiangtao Xie?, Peihua Li?, Qinghua H3,.
'Tianjin University, China, Dalian University of Technology, China, Haihe Laboratory of Information Technology Application Innovation, Tianjin, China
glwangStju. edu, en, gaomingze@t.ju edu. zz19@tju. edu. en
ji jiangtaoxiemaal dlut. edu. en, peihual cediuallidllt.... en, huginghua@tjj. edu. en
Abstract
Previous works show global covariance pooling (GCP) has great potential improve deep architectures especially on visual recognition tasks, where post- normalization of GCP plays very important role in final performance. Although several nostloormalizttion strategies have been studied, these methods pay more close attention effect of normalization on covariance representations rather than the whole GCP networks, and their effectiveness requires further understanding. Meanwhile, existing effective cost-normalizaton strategies (e.g., matrix power normalization) usually suffer from high computational complexity (e.g., ((d)) for d-dimensional inputs). To handle above issues, this work first analyzes the effect nostloormalization from the perspective of training GCP networks. Particularly, we for the first time show that effective post-normalization can make a good trade-off between representation decorrelation and information preservation for GCP, which are crucial to alleviate over wver-ftting and increase representation ability of deep improve existing post-
GCP networks, respectively. Based on this finding, normalization methods with some small modifications, providing further support observation. Furthermore, this finding encourages pre-normalization method for GCP (namely DropCov), which develops an adaptive channel dropout features right before GCP, aiming to reach trade-off between representation decorrelation and information preservation in a more efficient way. Our DropCov only has linear complexity of Od), while being free for inference. Extensive experiments on various benchmarks (i.e., ImageNet- K, ImageNet-C. ImageNet-A, Mitiieedmmmeeett,, and iNat2017) show our DropCov is superior the counterparts in terms of efficiency and effectiveness, and provides a simple yet effective method to improve performance of deep architectures involving both deep convolutional neural networks (CNNs) and vision transformers (ViTs).
propose novel
Introduction
Global covariance pooling (GCP) has shown remarkable potential to improve performance of deep
architectures in a variety of tasks, especially visual recognition BI
23
30
g
48
36 47 51]
One of core differences among various deep GCP methods nontalaomaiizooinn for covariance representations, which plays crucial role in final performance. The existing post-normalizuile approaches can be roughly categorized into clement-wise post-normalization methods [3] 25
and structure-wise ones 23 29 28
25
36 47), which concentrate on normalizing entries and
"Qinguua Hu is the corresponding author and is with Engineering Research Center of City intelligence and Digital Governance, Ministry of Education of the People's Republic of China.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

eigenvalues of covariance matrices, respectively. Some researches (e.g.. (30) perform two kinds of normalizations simultaneously. Although several nost-normalization strategies have been studied, behavior of different approaches seem vary significantly and there is a lack of an intuitive and unified interpretation. Therefore, the true mechanism lying behind these post-normalization methods (especially under deep architectures) is still an open problem.
Existing works understand the effect of post-normalization methods of GCP from different aspects. Specifically, 3I 25] claim that their element normalization strategies can suppress the bursti- ness of covariance representations as in the classical bag-of- -foddds [37] or actually perform a kind
of whitening on representations. DeepO,P [23] and MPN-COV [29] develop post-normalization methods to exploit geometry of covariances by considering the fact that space of covariance forms a
Riemannian manifold [344, Besides, MPN-COV shows that matrix square-root (sqrt) normalization accounts for a robust covariance estimation 455), while 30 claims spectral normalization compen- sates for burstiness of eigenvalues and shows effectiveness of matrix sqrt normalization via empirical comparisons. Although many efforts are made, the aforementioned researches try to explain why GCP works by studying how post-normalizations affect covariance representations, which lack full view on the whole GCP networks. Therefore, this work analyzes effect of post-normalization approaches on optimizing deep GCP networks. Particularly, by taking matrix power normalization [29] as an example, we find that post-normalization tries to handle a paradox between representation decorrela- tion and information preservation for GCP, which are crucial to alleviate over-fitting and increase representation ability of deep GCP networks, respectively. Particularly, effective post-normalizatio approaches can achieve good trade-off, and so lead to a better performance. Based on this finding, we introduce some modified versions of existing post-normalization methods (e.g., adaptive power normalization for MPN-COV, I-LogM for DeepO,P, and linear transformation for B-CNN 31), which achieve further improvement, providing support to our observation.
Another issue of existing post-normalization methods is high computational complexity. Particularly, effective structure-wise post-normalization strategies usually involve singular value decomposition (SVD) of covariance matrices or sequential matrix multiplications, resulting in the computational complexity of (dd) for d-dimensional inputs. Inspired by finding above, this paper proposes an efficient pre- rremaaamaliitioon method for GCP (namely DropCov) based on dropout [26). as it has the natural ability to perform feature decorrelation 44) and reduce over-fitting of training neural networks (26). Specifically, our DropCov develops an adaptive channel dropout on features right before computation of covariance representations, where probability of dropout is adaptively decided by seeking good trade-off between representation decorrelation and information preservation. Compared to existing post-normalization methods, our DropCov achieves superior or competitive performance with a linear computational complexity of O(d), while is only used for training.
To verify the effectiveness of our proposed DropCov, we conduct experiments using various deep architectures including CNNs [17] and vision transformers (ViT) 43
32 55
on ImageNet- 1K (7).
ImageNet-C [18), ImageNet-A [19,,
Imazzeeee [13]. and
iNat2017 [20]. The
contributions of this work are summarized follows: (1) To best knowledge, this work for the first time analyzes the effect of post-normalization on GCP from the perspective of optimizing neural networks, and we show that effective nost-normalization methods can make good trade-off between representation decorrelation (W.z. over-fitting reduction) and information preservation (wez increase of representation ability) for GCP (w.r. deep GCP networks); (2) Based on our finding, we make some small modifications to existing nost-normalization methods (Sec. 2.2 e.g., adaptive power normalization and I-LogM & linear transformation. The comparisons in Sec. 5.2 show our modifications can bring further improvement, giving support on our observation; (3) Profiting from our finding, we propose a novel DropCov method by developing an adaptive channel dropout as pre-normalization of GCP, which has a linear complexity of ((d) and is only used for training. Extensive experiments show our DropCov is superior or competitive to existing nosmalaomaii approaches and provides a simple yet effective method to improve deep architectures.
2 Effect of Post-normalization on Deep GCP Networks
In this section, we first analyze the effect of post-normalization on optimizing deep GCP networks by taking matrix power normalization as an example. To further verify our finding, we spread it to existing post-normalization approaches, and make some modifications according to our observations.

---

A- 10.4
Top- Aosaaay (55)
60
80
Teu [-
Tron par0
0-
Toxtem0 2]
Tax (x-33
65
Toxtord
60
Tuak J- 100
-20
0-
40
60
100
0.1
0.3
0.5
0.7 0.9
1.0
Thxining Fponh
(a) Convergence curves of various
(b) Generalization vs. representation ability
Figure 1: Results of GCP networks with matrix power normalization (MPN) using the backbone of ResNet-18 (d = 128) on ImageNet-1K K. (a) Convergence curves of various o E (0, 1]: (b) Comparison of generalization and representation abilities for various From them of a result in higher training accuracies indicated by blue solid line in (b), w.r. better representation or approximation ability, but suffer from heavier over-fitting (w.r.f worse generalization), aka larger differences of training accuracy over test one indicated by A. In contrast, smaller values of have better generalization (aka smaller A), while obtaining lower classification accuracies (w:Et weaker representation ability). Particularly, a = 0.5 achieves the best trade-off and so leads the best results.
that larger values
2.1 How Does Matrix Power Normalization Impact GCP
Matrix Power Normalization Both previous works 47 36 and our experiments show structure- wise nost-oormalization methods are generally superior to element- wise ones, where matrix power normalization (MPN) usually performs the most competitively. Let X E IR N xd be N d-dimensional features with zero-mean those outputted by the last convolution layer of deep CNNs or the last transformer block of ViT (i.e., final word tokens), GCP computes X X with post-normalization to generate a covariance representation Specifically, GCP with MPN is calculated as
- xx" = UA"
(1)
where Z E Rdd is used for final classification. >> 0 is parameter of power. U and A are the matrix of eigenvectors and the diagonal matrix of eigenvalues of X respectively.
For MPN, power a has a significant effect on the final results. Previous works [29 30 conduct empirical comparison of a, and choose the best one (e.g., a 0.5). However, the effect brought by a is still not very clear. In this work, we first analyze the effect of a E (0, 1] on optimizing deep GCP networks. It is clear from Eqn. that a has the following effect on GCP from the mathematical perspective:
Effect of Power
Z I- a a Representation Decorrelation & Information Loss, z --+X' if a a Information Preservation & Strong Correlation.
(2)
Specifically, when a - - O, the normalized representation Z will be decorrelated and tend to be an identity matrix I. Such representation decorrelation can help optimization of neural networks [27 4] (in particular for combating over-fitting (5), which, however, will lead to information loss for X"X, hurting representation ability of covariances. Particularly, Z with MPN of
0 becomes an identity matrix, and the information lying X completely loses. When & + + 1, information of X will be gradually preserved, maintaining the correlations characterized in the covariance matrices, which makes training of neural networks difficult (e.g., over-fitting). To further verify above observation, conduct experiments using ResNet-18 ImageNet- (more results can refer to supplementary materials). By observing Figs. ((a) and T(b) we draw the following conclusion:
When < 0.5, post- nost-normalization (i.e., MPN) tends to decorrelate representation. Accord- ing to red line of Fig. [(b) smaller values of a reduce over-fitting more significantly and have better generalization (aka smaller differences of training accuracy over test one 3 5). Meantime, indicated by blue lines Fig. T(b) smaller a leads lower accuracies (w.zf weaker representation approximation ability 355 47) due to information loss.
3

---

When > 0.5, post-normalization (i.e., MPN) tends to preserve information. The training accuracies (W.Ef representation or upproximation ability) of deep GCP networks indicated by blue solid line of Fig. [(bb increase along values of a enlarge, but heavier over-fitting (aka larger values of A) occur because effect of representation decorrelation decreases.
According to Eqn. (2) and above observation, we have
Corollary 1. Effective post-normalization (e.g., matrix power normalization (MPN) with 0 < S 1) can achieve a good trade-off between representation decorrelation and information preservation for GCP, which are crucial to alleviate over-fitting and increase representation ability of deep GCP networks, respectively. Particularly, MPN with a = 0.5 achieves the best trade-off for a (0, 1] (without considering other factors), which is proved to be the widely used choice of a [29 30).
The Corollary | Jilluminates the effect of MPN on deep GCP networks from the perspective of model training, and gives intuitive explanation about the choice of parameter
2.2 Extension of Corollary 1 to Existing Post-normalization Methods
To further verify Corollary T we extend it to analyze behavior of existing post-normalization. Accompanied by these analyses, we can improve previous methods with some small modifications.
Adaptive Power Normalization (APN) Above all, Corollary 1 shows MPN with a = 0.5 is the best trade-off for a E (0, 1] if no other factor is considered. To further verify the effect of trade-off, propose an adaptive power normalization (APN) by considering the effect of inputs (i.c.. cigenvalues A of covariances). Specifically, we formulate the trade-off between representation decorrelation and information preservation as the following objective function:
min log
maz
S.t.
max Σ
log
X Σ
(3)
min
X
where Amaz and Amin are maximum and minimum of A, respectively. Particularly, we minimize the ratio of maz to X0 (in logarithmic coordinates) to enforce representation decorrelation. Here, smaller values of the objective function indicate stronger decorrelation, and a = 0 achieves minimum (i.e., zero). On the other hand, we maximize negative entropy of normalized eigenvalues to keep discrepancy among eigenvalues and preserve information, which achieves maximum value (i.e., zero) for a + +00. By considering that previous works 229 30 show relation between performance and is generally convex, we exploit a grid search strategy to obtain suboptimal solution for a E (0, 1] with the interval of 0.01 However, Amin is usually nearby zero, leading to computational instability. Therefore, we optimize the objective function (3)
min [a(log( - log (ma (C, Amin) ))) -
Σ 1=1 log(A 1=1
{ (4)
representation decorrelation
information preservation
where we set minimum of Amin to a constant C' (e.g., 1e-2), and compute the entropy using Top- eigenvalues. 7 is balance parameter, which is set to 1.2. Note that Eqn. 44 is computed efficiently for small K (K < 3 in our work) using about a hundred trials, given the eigenvalues A.
Theoretically, a of MPN could be larger than one, which, however, violates the principle of representation decorrelation. Meanwhile, it disrupts equilibrium between eigenvalues, hurting information inherent covariances. As illustrated in Fig. 2 MPN with a 2 suffers from heavy over-fitting, and is clearly worse than one with a = 0.5.
MPN with
Matrix Logarithm Normalization (LogM) LogM 23 one another popular structure-wise post-
normalization. Mathematically, logarithm has similar behavior with power function of a = 0.3 when eigenvalues > 1, but reverses amplitudes of eigenvalues nor information preservation). However, small eigenvalues frequently occur in deep models (29]. especially for large feature dimension (d). Accordingly, as shown in Fig. LogM achieves similar
(i.e., neither decorrelation
NNote that it is unnecessary to obtain exactly the optimal solution, since performance gap is negligible when difference of a is less than 0.01.

---

results with MPN of a = 0.3 for d 128. For d = 256, GCP network with LogM fails to converge due side effect brought by small eigenvalues (A < 1). Therefore, we modified LogM with shift operation (namely I-LogM) to make all inputs of LogM be larger than one, i.e,, log (x)
< log((A; + c)/e). Our I-LogM brings promising gains when d = 256 (see Sec. 5.2 and Table
slement-wise Normalization (EwN) EwN could be roughly written in unified formulation: [ (1 -T(( + 7, where is vectorization of X I' is mapping function and ||a|| indi- cates a certain norm of a. 33 and 7 perform a linear transforma- tion (LT), Particularly, for element-wiss power normalization (EPN) BT. ff is a signed power function and μ = O. ||a/| is l2 norm, and no LT is performed. For LN (2) is identity mapping and 4l is mean of f()). ||a||. is a le norm. According to above formulation and Corollary [ EwN performs represen- tation decorrelation based on zero-mean (i.e., f(z) (1) and
(5u
2 3.00
Tal (pskd)
Ie ben2
Tor Rr
(1-01)
Tal: Eoght 8-250
40
100
Toikinng Fposh
standardization (i.e., normalized by [lf(z) sulla), while LT Figure 2: Results of a >> 1 & LogM operation aims to recover information. Compared to MPN, element-wise normalization lacks an ef-
fective mechanism to balance representation decorrelation and information preservation. Particularly, as shown in Sec. 5.2, EPN does not work well, since only representation decorrelation is performed. Therefore, we introduce LT for EPN, and it achieves promising improvement (see TJble
3 DropCov: A Simple yet Effective Normalization
3.1 Normalizing GCP via Dropout
Corollary shows the core effect of effective post-normalization (e.g., MPN) lies a good trade- off between representation decorrelation and information preservation for optimizing deep GCP networks. However, MPN suffers from high computational complexity (i.e., (dd) of d-dimensional features), especially for nigh-dimensional inputs. Previous works [1 5 show that dropout can reduce correlation among features. Therefore, one question is naturally raised up: Could we use efficient dropout perform representation decorrelation for GCP? Given probability p of dropout, + 1 leading to sparser, less correlated features but more information loss. On the contrary, p 0 preserves more information while performing less decorrelation. Above observations show consistence with Corollary []] However, there exist two issues to apply dropout for normalizing GCP: (I) how to perform dropout and (2) how choose probability p of dropout. A naive method is to perform dropout with a fixed after covariance representations (as compared in Table but it has several limitations: first, dropout after covariance representations will break their structures (e.g., row i of covariance means correlations among i-th channel and all ones); second, fixed p hardly achieves trade-off between representation decorrelation and information preservation; third, complexity of such strategy is quadratic to feature dimension d (i.e., Odd)), and is still high for large d.
3.2 Pre-Normalization via Adaptive Channel Dropout (ACD)
To handle above issues, this work proposes a pre-normalization method (namely DropCov) by performing channel dropout with an adaptive probability p features X before GCP, which aims to reach a good trade-off between representation decorrelation and information preservation. Since DropCov performs representation decorrelation via an adaptive dropout operation channel features before GCP, it can maintain structure of covariance representations, while having a more efficient, linear computational complexity of ((d). To reach the trade-off with adaptive probability p. compute it by considering two factors: (1) feature dimension (d) and (2) relationship between feature correlation and feature importance. Then, we have
D w
p log(d)
| [
(5)
w = (CII))(GAP())
π = SUM xx"x),
where w and π represent feature importance and feature correlation, respectively. D scaling parameter, which is set to 10 throughout all experiments. Since self-attention mechanisms [2 46 are
5

---

Table 1: Comparison of our DropCov with close related works in terms of formulation, computational
complexity of nost-normalization and usage mode. Please refer to Sec. 4 for the details of symbols.
Method
Formulation
Complexity
Train
Infer.
B-CNN
31
(x(xTr((V(XTX))
(
Element- wise
Post-Norm.
SigmE
2/(1 + e --VX""X))
(
LN 12
B B ->)oo)
O(d))
DeepOyP
23
([LogM(X'X)
O(d")
MPN-COV 29
v((X"x))
O
ftructure-wise
Post-Norm.
IB-CNN E0
(a(sgrt(V(X* (x))() 1)))))))
O(d))
iSQRT-COV
28
V(= (X" X) 13)
0(d")
MaxExp 25
I- XXXX +))*
O(d²)
Ours
DropCov
Y(Y')), Y (X)
O(d)
x
*: Note B and E are parameters, C and @ indicate element- wise multiplication and addition, respectively.
usually used to search important features, we measure feature importance by modifying light-weight channel attention module [46), which involves global average pooling (GAP), a ID convolution with kernel size of (CID3). and softmax function (a). As covariance X naturally characterizes correlation among different features, we compute feature correlation by summarizing the elements along row of x X (i.e., SUM X X)). Note that we restrict p of Eqn. 5 belongs (0, 1).
As for Equ. 5 we adaptively decide probability P of channel dropout for reaching a good trade-off D between representation decorrelation and information preservation, where Tog(d] and inner product of (we, T are designed to consider effect of feature dimension (d) and relationship between feature correlation and feature importance, respectively. In particular, larger feature dimension d usually requires larger dropout probability p. Mcanwhile, feature correlation (i.c., πT and feature importance (i.e., w) are closely related to representation decorrelation and information preservation, respectively, Clearly, larger feature correlation results in stronger representation correlation, while features with larger channel weights contain more important information. Therefore, relationship between feature correlation and feature importance is good indicator to perform channel dropout. Intuitively, if feature correlation is closely related to (i.e., seriously decoupled with) feature importance, it is hard to select features for reaching a good trade-off between representation decorrelation and information preservation, and we need carefully adopt dropout for channel features under the random setting, leading to small p. Otherwise, we can perform dropout more safely to achieve a trade-off, and adopt large p. Note that we show behavior of p during training in supplementary materials. Based on Eqn. (5) our DropCov is achieved by
Y). Y 6X)),
(6)
where 8, is a channel dropout operation with the probability of p. a vectorization operation followed by triangulation, leading to a d(d + d/me-iimenioonal covariance representation Note that our DropCov (6) is only adopted to train GCP networks, which can be easily implemented by standard back propaaropagttion. During inference stage, simply use V(X"X) for final classification.
Related Works
In this section, we review some works closely related to our DropCov. Table[giivss a detailed compar- ison of our DropCov with existing post-normalization methods in terms of formulation, computational complexity of post-normalization and usage mode. For element-wise post-normalization methods, B-CNN 31 performs element- soiment-wise signed square-root (sqrt) followed by & normalization variance representations. (25) proposes to use some surrogate functions (c.g.. SigmE) to handle issue of negative evidence inherent in power function. Layer normalization
(LN) [ is proposed to normal- ize each feature with its mean (μ) and variance (y) followed by a linear transformation. Since these
"Ntte there exist small differences between GCP and bilinear pooling of B B-CNN. Specifically, GCP computes covariance of inputs X with mean of μ. while bilinear pooling computes outer product of inputs X and Y. When X and Y are shared and zero-mean, bilinear pooling captures the same information with one of GCP.

---

methods perform normalization on each element of covariance representations, they have the compu- tational complexity of (d²)) for d-dimensional features X. For structure-wise post-normalization methods, DeepOyP and MPN-COV 29 perform matrix logarithm (LogM) and matrix power function for covariance representations, respectively. IB-CNN (30 combines matrix square-root normaliza- tion with element-wise one, while iSQRT-COV [28] proposes an approximate matrix square-root normalization via fast iterative matrix multiplications. Song et al. further analyze the effect of iSQRT-COV [38] and develop faster variant [39] (25] introduces several spectral power functions
(e.g., MaxExp) for normalizing covariance matrices. These structure-wise approaches involve SVD of covariances sequential matrix multiplications, having the computational complexity of (d ReDro [36 proposes a relation dropout scheme to divide a large-sized covariance into group of small-sized ones, aiming to mitigate the computational issue of matrix normalization (2 30 28
ReDro reduces computational complexity structure-wiss approaches from O(d*) to ( /G))
where G' is the number of groups. Besides, all existing works perform post-normalization methods during both training and inference (Infer.) stages. Different from aforementioned works, our DropCov performs adaptive channel dropout 5 on features X right before GCP, which has linear complexity ((d) and is free during inference stage. Experimental results in Sec. 5.3 show our DropCov with much less computational complexity superior or competitive previous works.
Experiments
In this section, we conduct experiments to evaluate effectiveness of our DropCov in improving deep architectures on image classification tasks. Specifically, we first describe implementation details, and then make ablation study on ImageNet-IK (IN-1K) [7] using backbone of ResNet- 1.. Additionally,
we apply our DropCov to both ResNets (17) and ViT models [43 32 50], while comparing them on
several benchmarks (i.e., IN-1K, ImageNet-C (IN-C) [8]] ImageNet-A (IN-A) [19] and Stylized-
ImageNet (Sty.-IN) (13)) to assess generalization and robustness of our DropCov. Finally, we assess generalization of DropCov models by transferring them to long-tailed species classification 20))
Table 2: Comparisons (% in Top-1 accuracy) of adaptive normalization methods with the fixed ones using ResNet- 18 on ImageNet- IK.
Table 3: Comparisons (% Top-1 accuracy) of
several specific dropout strategies using ResNet- 18 on ImageNet- IK.
Method
d 64 d 128 d 256
256
Method
Element Channel Element Channel
DropElement (p 0.5)
73.4
74.6
74.0
DropChannel (p 0.5)
70.1
73.1
75.1
Large (p = 0.5)
72.2
71.3
71.4
70,9
ACD (Ours)
73.5
75.0
75.2
Small (p = 0.5)
68.4
71.2
66.4
71.5
Uniform (p 0.5)
72.8
72.1
74.0
74.4
MPN (a 0.5)
73.1
74.4
74.9
APN (Ours)
73.3
74.5
75.0
ACD (Ours)
73.5
75.2
Table 4: Comparisons (% in Top-1 accuracy)
Table 5: Comparisons (% in Top-1 accuracy) of DropElement and DropChannel with various drop rates (p) using ResNet- 18 on ImageNet- IK.
of various dropout variants using ResNet-18 ImageNet-IK.
Method
dim=64
dim=256
Channel Dropout
Element Dropout
Maxout
72.1
73.7
P
dim 64 dim 256 dim = 64 dim 256
72.5
71.9
72.3
70.8
Dropconnect 44
70.6
72.5
0.3
72.8
74.7
72.8
72.3
Decov S
72.7
74.0
0.5
70.1
75.1
73.4
74.0
Maxdropout
72.0
70.1
0.7
65.7
72.1
72.1
74.2
DropBlock 14
72.3
74.1
0.9
20.5
54.7
68.2
73.8
ACD(Ours)
73.5
75.2
ACD (Ours)
73.5
75.2
73.5
75.2
5.1 Implementation Details
Specifically, we construct DropCov models based on ResNet following exactly the same schemes in 29
In this work, we apply our DropCov to several ResNets [17 ViT models [4 32 50)
47
Given a CNN backbone, we first introduce a x convolution after the last convolution layer for reducing dimension of features to d, and then replace the original GAP by our DropCov followed by
7

---

a fully-com connected (FC) layer and softmax for classification. All models are optimized using the same nyper-parameter settings and data augmentations as suggested in [7 29]. where stochastic gradient descent (SGD) with initial learning rate (lr) of 0.1 is used to train the networks within 100 epochs and Ir is decayed by 10 every 30 epochs. We combine DropCov with ViT models as in [49] where one FC layer is employed to reduce dimension of final word tokens, and our DropCov replaces GAP and classification token for Swin Transformers [32] and DeiT/T2T-VE ViT
[43 [5], respectively. These
models are trained by using the same hyper-parameter settings in sever equipped with eight Nvidia RTX 3090 GPUs and 128G RAM. Source code (e.g., PyTorch
43 32 50] All programs run
33
PaddlePaddle and Mindspore) will be available at htthtts:/ // github. com/minngaee/DropCCo
5.2 Ablation Study
In this subsection, we conduct experiments on ImageNet-IK IK using lightweight ResNet-18 to assess effect of adaptive probability o of dropout on our method, while compare with several dropout strategies, and finally assess effect of our modifications on existing post-normalization approaches.
Fixed P vs. ACD. Our DropCov performs pre- nreenommalization for GCP by using an adaptive channel dropout (ACD) method, which automatically determines probability p of dropout by considering trade-off between representation decorrelation and information preservation. We compare with two kinds of baseline dropout methods using fixed p = 0.5, i.e., element-wise dropout for covariance representations (DropElement) and channel dropout for input features (DropChannel). As shown in the upper part of Table 2 our ACD achieves the best results for various feature dimensions (d) and clearly outperforms the naive DropElement in Sec. 3.. Besides, we compare with DropElement and DropChannel with various dropout ratios. As compared in Table our ACD is superior to DropEle- ment and DropChannel for all dropout ratios. Particularly, the best dropout ratios are quite different for two methods and various feature dimensions. In contrast, our ACD can adaptively determine probability o of dropout by balancing representation decorrelation and information preservation, while always achieving the best performance, clearly verifying ACD can achieve a better trade-off.
Comparison of Various Dropout Strategies. To verify the effectiveness of our ACD, we compare with several dropout strategies. First, we perform element- wise (Element) dropout on some specific entries of covariance with p = 0.5. Specifically, we rank all entries of covariance in descend- ing order, and then perform dropout on top/bottom/uniform half of the ordered entries (namely Large/Small/Unifomm
We adopt similar strategies to feature channels (Channel), where channel indicators are decided by attention module of our ACD. Note that 'Element' dropout preforms dropout on elements of covariance representations based on values of elements (xx"')+,) which indicates correlation between i-th channel and j-th channel. Therefore, it only considers feature correlation. For 'Channel' dropout, preforms dropout on feature channels based on their weights, which are obtained by attention module of ACD (i.e., w). Therefore, it only considers feature importance. Based on above discussion, 'Element' and 'Channel' dropout methods can be regarded as special cases of ACD, where only feature correlation feature importance are considered, respectively. As compared in Table 3 uniform dropout clearly outperforms those only focus on largest and smallest entries, indicating equilibrium of feature correlation or importance is crucial final performance. Further- more, our ACD is superior to uniform dropout by a clear margin. Besides, we compare with several existing Dropout variants, including Maxout [15] DropConnect [44), Decov [5], maxdropout (9] and DropBlock [14. As shown in Table 4 our ACD clearly outperforms them. Although these methods can prevent overfitting, they are not good making balance between representation decorrelation and information preservation. These results further demonstrate the effectiveness of our ACD.
Effect of Modifications. According to Corollary I we make several modifications to existing post- normalization approaches. Beyond MPN, we propose an APN method by considering eigenvalues A of inputs (see Eqns. 43) and The results are compared in the bottom part of Table 2 where our APN determines by considering effect of inputs while average values of a are 0.53, 0.49 and 0.45 for d = 256,
128 and d 64, respectively. Clearly, average values of a achieved by APN are nearby 0.5, which further account for why 0.5 is the widely used choice of a for MPN. Besides, APN brings 0.1 11% 0.2% gains over MPN with a = 0.5 by considering effect of inputs. We would like to clarify they are non-trivial gains on ImageNet over strong MPN, and the recent works 38 40) also bring similar gains (0. (%0%.) over MPN. Besides, as discussed in See 2.2 we introduce an I-LogM and linear transform (LT) based on BN 22 for improving LogM normalization (DeepO,P)
8

---

Table 6: Comparison of DropCov with several post-normalization in terms of Top-1 accuracy (Acc.)
and running speed using ResNet-18 18 on ImageNet-1K, * According to See 2.2 and Corollary we modify B-CNN and IB-CNN with a linear transform (LT), while modifying DeepO,P with L-LogM.
Method
=
64
=
256
Top- Ace. (%)
Train (us)
Test (JS)
Top-1 Acc. (%)
Train (HS)
Test (us)
Plain GCP
71.1
3,45
1.04
70.0
35.08
11.89
B B-CNN B1
38.3
4.31
1.23
41.1
44.97
14.38
B-CNN +
68.3
4.45
1.28
73.2
47.37
15.24
LN 2
71.7
4.10
1.13
70.2
40.79
13,59
DeepO>P
23
70.1
922.50
910.95
Not Converge
9731.71
9638.71
I-LogM'
71.2
922.53
910.98
72.0
9732.46
9639.65
MPN-COV 29
73.1
925.16
913.87
74.9
9735.11
9642.43
iSQRT-COV 28
73.4
12.35
4.94
75.2
193.65
77.46
IB-CNN
Not Converge
13.93
5.16
36.1
202.38
80.55
IB-CNN LT
70.0
14.16
5.21
72.8
207.48
82.98
DropCov (Ours)
73.5
3.54
1.04
75.2
36.20
11.89
D: Note we compute running speed (μu) of single GCP module with post-normalization on a 2080Ti GPU. The original ResNet-18 with GAP achieves 70.2% in Top-1 accuracy.
and clement-wiss wise post-normalization (i.c., B-CNN and IB-CNN), respectively. As shown Table 6 DeepO,P performs poorly troubled by small eigenvalues (especially for d = 256), while our I-LogM obtains promising gains for both low- and high-dimensional features. Furthermore, B-CNN and IB-CNN achieve very unsatisfied results; in contrast, simple modification brings significant improvement for B-CNN and IB-CNN. Above results clearly verify our finding in Corollary
5.3 Comparisons with Counterparts
Here, we conduct experiments on ImageNet-IK using ResNet-18 for comparing with existing post-
normalization methods in terms of Top-1 accuracy and running speed, including B-CNN (31. LN (2].
P (23], MPN-COV [29] with 0.5, iSQRT-COV (28) and IB-CNN (30) The results of all compared methods are listed in Table 6 where we can see that our DropCov outperforms plain GCP (i.e., GCP without normalization) by 2.4% (5.2%) for d 64 = 256) with similar running time. Compared with element-wise nost-normalization (i.e., LN, B-CNN and IB-CNN), our DropCov achieves significant performance gains while requiring less running time. For structure-wist post- normalization, our DropCov is remarkably superior to matrix logarithm normalization (DeepO,P) in terms of both accuracy and running speed. Meanwhile, DropCov is superior or comparable to MPN-COV and iSQRT-COV using much less running time, especially during inference stage. Above results show our DropCov performs better or on par with the counterparts in terms of efficiency and effectiveness, providing very promising normalization method for deep GCP networks.
5.4 Comparisons with Various Deep Architectures
CNNs. Here we apply our DropCov with d = 128 to various CNNs, including ResNet-34, ResNet- 50 and ResNet-101. Meanwhile, compare DropCov with the original CNNs IN-1K, IN-C, IN-A and Sty.-IN. As shown the upper part of Table 7 DropCov respectively brings about 2.6%, 2.2% and 1.8% gains over ResNet-34, ResNet-50 and ResNet-101
IN-1K, while showing stronger robustness on IN-C, IN-A and Sty.-IN. Particularly, DropCov with ResNet-50 and ResNet- 101 are respectively superior to the original ResNet- 101 and ResNet-152, but having less model complexity.
Swin-T 32 and T2T- ViT-14 [50], and compare the original models well as ViT ViT [ As shown the bottom part of Table 7 DropCov brings about 2.6%, 1.3% and 1.2% gains over DeĪT-S, Swin-T and T2T-ViT- on IN-IK, respectively. Meanwhile, shows stronger robustness on IN-C, IN-A and Sty.-IN. Particularly, DropCov with DeiT-S is superior to DeiT-B and ViT-i, but having less model complexity. Above results show our DropCov provides a simple yet effective method to improve deep architectures, and deep architectures with DropCov achieve better trade-off between accuracy and model complexity.
VIT. For VIT models, we apply our DropCov with d = 128 to D 43

---

Table 7: Comparison of our DropCov with various ResNet and ViT models on four datasets, where results terms of mCE and Top-1 accuracy (%) are reported on IN-C and reminding ones, receptively,
Method
Params.
FLOPs
IN-IK
(N)
IN-A (1)
Sty.-IN (1)
ResNet-34
3.66 G
74.19
77.9
1.63
7.59
ResNet-50
25.6 M
3.86
76.02
76.7
2.47
7.15
ResNet- 101
44.6 M
7.57
77.67
70.3
4.15
9.51
ResNet- [17
60.2 M
11.28
78.13
69.3
5.98
10.09
ResNer-34+DrooCor (Ours)
29.6 M
5.56
(6.81(2.62)
(6.1(6.8)
3.45
11.16(3.57)
ResNat-50+DrronCox (Ours)
32,0 M
6.19
(8.19(2.17)
69.8(6.9)
5.08(2.61)
(.90(2.75)
ResNet-101+DrrpCoyy (Ours)
51.0 M
9.90
79.51
65.8
(3.39)
11.41
DeiT-S 4
22.1 M
4.6G
79.8
54.6
18.9
14.91
Swin-T
28.3 M
4.5G
81.2
62.0
21.6
13.40
T2T-ViT-14 [5
21.5 M
5.2G
81.5
53.2
23,9
15.80
DeïT-B 43
86.6 M
17.6G
82.0
48.5
27.4
17.94
ConViT-B 6
86.5 M
17.7G
82.4
46.9
29.0
19.67
Deip-S+DropCov (Ours)
25.6 M
5.5
82.4(2.6)
52.6(2.0)
31.2
(7.10
Swin-I+DropCov (Ours)
31,6 M
6.0
82.5
54.8(7.2)
33.1
(4.13
12T-ViT-14-DropCov (Ours)
24.9 M
5,4.G
82.7
52.1(1.1)
31.7
18.81(3.01)
5.5 Comparisons on Long-tailed Benchmark
Finally, we transfer our DropCov models iNat2017 20)
Table 8: Comparisons (%) on iNat2017.
and compare with several deep architectures, i.c.. ResNet-
Method
Top-1
Top-5
101, ResNet-112, Inception V3 (3)cV3)
42 with SE [21]
ResNet-101
62.4
84.1
as well as the recently proposed ViT [10 and TransFG [16].
Res
64.2
85.5
Specifically, we perform fine-tuning our DropCov mod-
IncV3
66.3
86.7
els with backbones of 101 and DeiT-S on iNat2017,
ViT-B_16 (IN-21K) 10
68.7
N/A
namely DropCov (ResNet-101) and DropCov (DeiT-S). As shown in Table our DropCov (ResNet-101) obtains 5.5% gains over the original ResNet-101, while outperforming ResNet-153 152 and IncV3 SE. Besides, our DropCov (DeiT-S) respectively outperforms ViT-B and TransFG by 3.7% and 0.7%, which are pre-trained ImageNet-21K. The results clearly verify generalization of DropCow.
TransFG (IN- (IN-21K) [6]
71.7
N/A
DropCov (ResNet-101)
67.9
87.3
DropCov (DeiT-S)
72.4
90.3
6 Conclusion
In this paper, we first analyze the effect of post-normalization on GCP from the perspective of optimizing deep GCP networks. Particularly, find that effective post-normalization methods have a good ability to balance representation decorrelation and information preservation, which can reduce over-fitting and increase representation ability of deep GCP networks, respectively. According to our finding, we introduce some strategies to improve existing normalization methods, further verifying finding. More importantly, propose novel pre-normalization for GCP (namely DropCov) based on adaptive channel dropout, which achieves very competitive performance with linear complexity and training-only mode. Extensive experiments identify that our DropCov provides a simple yet effective solution to improve existing deep architectures on image classification tasks.
GCP normally produces a high-dimensional covariance representation, bringing certain amount of computational cost, but our work makes use of high-dit figh-dimensional GCP more flexible. Meanwhile, our DropCov can achieve clear gains for low-dimensional GCP with little extra computational cost (refer to Table (). Additionally, our DropCov is potentially applicable to compact GCP methods [12] 24] and encourages exploration of other effective dropout strategies, which will be studied in future work.
A Acknowledgment
The work was sponsored by National Natural Science Foundation of China (Grant No.s 622761 86, 61925602, 61971086 and 61732011), CCF-Baidu Open Fund 100.20011P110002000 JAAIXSJLJI-- 2022-010C and Haihe Lab of ITAI (NO. 22HHXCJC00002),
10

---

References
[1] V. Arsigny, P. Fillard, X. Pennec, and N. Ayache, Fast and simple calculus on tensors the Log-Fuclidean
framework. In MICCAI, 2005.
[2] I.. J. Ba, J. R. Kiros, and G. F. Hinton. Layer normalization. CoRR, abs/1607 06450, 2016.
[3] C. M. Bishop. Puttern recognition and machine learning, 5th Edition. Information science and statistics.
Springer, 2007.
[4] G. Chen, P. Chen. Y. Shi, Hsieh, B. Liao, and S. Zhang. Rethinking the usage of batch normalization
and Dropout in the training deep neural networks. CoRR, abs/1905.05928,, 2019.
[5] M. Cogswell, F. Ahmed, R. B. Girshick, I.. Zitnick, and D. Batra. Reducing overfitting in deep networks by decorrelating representations. In ICL.R, 2016.
[6] S. d' Ascoli, H. Touvron, M. L. Leavitt, A. S. Morcos, G. Biroli, and L. Sagun. ConViT: Improving vision
transformers with soft convolutional inductive biases. In ICML, 2021.
[7] Deng. W. Dong. R. Socher, L.-J. Li, K. Li, and Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
[8] A. Diba, V. Sharma, and I.. Van Gool. Deep temporal linear encoding networks. In CVPR, 2017. [9] C. G. dos Santos, D. Colombo, M. Roder, and J. Papa. Maxdropout: Deep neural network regularization based on maximum output values. In ICPR, 2020.
[10] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, Gelly, et al. An image worth 16x16 words: Transformers for image recognition at scale. In ICI.R, 2021.
[11] M. Engin, I.. Wang. I.. Zhou, and X. Liu. DeepKSPD: Learning kemnel-matri--based SPD representation for fine-grained image recognition. In ECCV, 2018.
[12] Y. Gao, O. Beijbom. N. Zhang. and T. Darrell. Compact bilinear pooling. In CVPR, 2016.
[13] R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In ICLR, 2019. [14] G. Ghiasi, T. Lin, and Q V. Le. DropBlock: A regularization method for convolutional networks. In NeurlPS, 2018.
[15] J. Goodfellow, D. Warde-Farley, M. Mirza, A. C. Courville, and Y. Bengio. Maxout networks. In ICMI.,
2013.
[16] J. J.N. Chen, S. Liu, A. Kortylewski, C. Yang. Y. Bai, C. Wang, and A. Yuille. TransFG: transformer architecture for fine- gna-gra recognition. AAAI, 2022.
[17] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
[18] D. Hendrycks and G. Dieuerich. Benchmarking neural network rohustness common corruptions and perturbations. In ICLR, 2019.
[19] D. Hendrycks. K. Zhao, S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. In CVPR, 2021.
[20] G. V. Horn, O. M. Aodha, Y. Song. Y. Cui, C. Sun, A. Shepard, H. Adam, P. Perona, and S. J. Belongie. The iNaturalist species classification and detection dataset. In CVPR, 2018.
[21] J. Hu, I.. Shen, Albanie, G. Sun, and F. Wu. Squcezz-anddexcciation networks. IEEE Trans. Pattern Anal. Mach. Intell., 42(8):201 1-2023, 2020.
[22] S. loffe and C. Szegedy. Batch Normalization: Accelerating deep network training by reducing internal
covariate shift. In ICML, 2015.
[23] C. Ionescu, O. Vantzos, and C. Sminchisescu. Matrix backpropagation for deep networks with structured layers. In ICCV, 2015.
[24] S. Kong and C. C. Fowlkes. Low-rank bilinear pooling for fine- fine-grained classification. In CVPR, 2017.
[25] P. Koniusz, H. Zhang, and F. Porikli. A deeper look at power normalizations. In CVPR, 2018.
[26] A. Krizhevsky, L. Sutskever, and G. E. Hinton. ImageNet classification with deep convolutional neural
networks. NIPS, 2012.
[27] Y. LeCun, I. Kanter, and S. A. Solla. Second order properties of error surfaces. In NIPS, 1990.
[28] P. Li, Xie, Q. Wang, and Z. Gao. Towards faster training of global covariance pooling networks by
iterative matrix square root normalization. In CVPR, 2018.
[29] P. J. Xie, Q. Wang, and W. Zuo. Is second-order information helpful for large-scale visual recognition?
In ICCV, 2017.
[30]
T. Lin and S. Maji. Improved bilinear pooling with CNNs. In BMVC, 2017.
[31]
T. Lin, A. RoyChowdhury, and S. Maji. Bilinear CNN models for fine- fine-grained visual recognition. In ICCV,
2015.
[32] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision
transformer using shifted windows. In ICCV, 2021.
[33] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan. Killeen, Lin, N. Gimelshein,
L. Antiga, A. Desmaison, A. Köpf, E. Z. Yang. Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy,
B. Steiner, L. Fang. J. Bai, and S. Chintala, Pytorch: An imperative style, performafem deep learning
library. In NewrIPS, 2019.

---

[34] X. Pennec, P. Fillard, and N. Ayache. A Riemannian framework for tensor computing. IJCV, 66(1):41-66 2006.
[35] T. Poggio, A. Banburski, and Q. Liao. Theoretical issues in deep networks. Proceedings of the National
Academy of Sciences (PNAS). 117(48):30033-30045 2020.
[36] S. Rahmmnn L. Wang, Sun, and Zhou. ReDro: Efficiently learning large-sized SPD visual representa- tion. In ECCV, 2020.
[37] J. Sánchez, F. Perronnin, T. Mensink, and J. Verbeck. Image classification with the Fisher Vector: Theory and practice. IJCV, 105(3):222- 2212222-,,,222 2013.
[38] Y. Song. N. Sebe, and W. Wang. Why approximate matrix square root outperforms accurate SVD in global covariance pooling? In ICCV, 2021.
[39] Y. Song, N. Sebe, and W. Wang. Fast differentiable matrix square root. In ICLR, 2022.
[40] Y. Song. N. Scbe, and W. Wang. Improving covariance conditioning of the SVD meta-layer by orthogonality.
In ECCV, 2022.
[41] N. Srivastava, G. E. Hinton, A. Krizhevsky, Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting, J. Mach. Learn. Rex., 15(1):192--1955 2014.
[42] C. Szegedy, V. Vanhoucke. loffe, J. Shlens, and Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
[43] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data-efficient image transformers distillation through attention. In ICMI., 2021.
[44] L. Wan, M. D. Zeiler, S. Zhang, Y. LeCun, and R. Fergus. Regularization of neural networks using dropconnect. In ICMI., 2013.
[45] Q. Wang. Li, W. 7no, and I., Zhang. RAID-G: Robust estimation of approximate infinite dimensional
Gaussian with application to material recognition. In CVPR, 2016.
[46] Q Wang. B. Wu, P. Zhu, Li, W. Zuo, and Q. Hu. ECA-Net: Efficient channel attention for deep
convolutional neural networks. In CVPR, 2020.
[47] Q Wang, J. Xie, W. Zuo, I.. Zhang, and P. Li. Deep CNNs meet global covariance pooling: Better representation and generalization. IEEE TPAMI, 43()), 2020.
[48] T. Winterbottom, S. Xiao, A. McLean, and N. A. Moubayed. Trying bilinear pooling in Video-QA. arXiv preprint 2012::2012. 10285, 2020.
[49] Xie, R. Zeng. Q. Wang, Z. Zhou, and Li. ViT: Mind visual tokens for vision transformer. CoRR.
0ts/2104.10935. 2021.
2021.
[50] L. Yuan, Y. Chen, T. Wang. W. Yu, Y. Shi, Z.-H. Jiang, F. E. Tay, J. Feng, and S. Yan. Tokens-to Token ViT: Training vision transformers from seratch on ImageNet. In ICCV, 2021.
[51] J. Zhang, I.. Wang, I.. Zhou, and W. Li. Beyond Covariance: SICE and kernel based visual fcature
representation. Int. J. Comput. Vis., 129(2):300-320, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Abstract and Section [
(b) Did you describe the limitations of your work? [Yes] See Section 6
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See Section 5. and Supplemental Material.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section 5.1 and Supplemental Material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] We share the same settings with compared methods.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)? [Yes] See Section 5.1
12

---

4. If you are using existing assets (e.g., code, data, models) or curating/releasnng new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 5
(b) Did you mention the license of the assets? [No] All of them are publicly available.
(c) Did you include any new assets either in the supplemental material or URL? [No]
We use publicly available datasets.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No] All of them are publicly available.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] We use publicly available datasets.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "mkK0-YOZNA", "reviewer": "Reviewer_h5SV", "summary": "Motivated by the facts of \n(1) the post-normalization of GCP lacks theoretical understanding and \n(2) the high computational complexity of GCP normalization, \nthe paper proposes:\n- a theoretical analysis of matrix power normalization in the GCP networks and extend it to an adaptive normalization method\n- an adaptive channel dropout to reduce the computational complexity.\n\nThe proposed method has been verified on ImageNet and iNaturalist datasets using various scale architectures. ", "questions": "- Is the GCP the bilinear pooling? I think it could be better to explain these two academic terms and illustrate their relationships in the paper.", "limitations": "None.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The proposed method is simple and efficient for classification tasks. \n- Experiments are solid on multiple datasets, especially have experimented with different scale architectures.\n- Good theory analysis with ablation studies.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5yysqDOh1Uh", "reviewer": "Reviewer_Fzo6", "summary": "The main contribution of this paper is to propose Adaptive Channel Dropout (ACD), which performs channel dropout before the covariance computation to obtain good trade-off between representation decorrelation and information preservation. ACD is more efficient and more effective than existing  post-normalization techniques. Since the core of ACD is dropout, it is only used during training, and is not needed in inference. There are other minor contributions, such as Adaptive Power Normalization (APN), which is a slight improvement over matrix power normalization (MPN) [34,42]. The paper also performs more analysis of MPN.", "questions": " - more (theoretical) discussion of equation (5)\n - clarify contribution and even rewrite the paper to make the core contribution more obvious\n - comparison to other training regularization techniques that prevent overfitting", "limitations": "Checklist points to the conclusion, but the conclusion does not really address the limitations and potential negative societal impact of the work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n - ACD is more effective and efficient than existing post-normalization techniques.\n - more extensive analysis of alpha for MPN\n\nWeaknesses\n - One of the main contribution is equation (5), but the intuition/explanation is not very clear. Some more theoretical discussion would also be helpful.\n - The current presentation makes it hard to understand what are the main contributions. There are many things going on: analysis of MPN, LogM. APN. ACD... My take is ACD is the main contribution, but would be helpful to make it clear in the paper\n - The main advantages of ACD seems to come from the fact that it is a dropout-based technique. The fact that its faster and is not required in inference are because it is based on dropout, which is quite different from post-norm/pre-norm. It would be helpful to compare to more regularization techniques in training.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2WqE6OvkFR_", "reviewer": "Reviewer_VWPn", "summary": "This paper focuses on studying matrix power normalization in global covariance pooling operation. The authors provide a detailed analysis in the effect of different power parameters on matrix power normalization as well as other post normalizations, in terms of feature decorrelation and information preservation, empirically showing the necessary of using a good tradeoff between the two properties. To reduce the computation complexity of post normalization, the authors introduce Dropconv which performs channel dropout to achieve feature decorrelation. Experiments show the ability of the proposed strategy  on performance improvements on both ResNet and ViT backbones.", "questions": "1. For the experiments in Table 2, the counterpart of DropChannel seems analogous to ACD except that the probability in ACD is determined by features, while the performance gap is so significant and even higher than DropElement. Could the authors analyze the phenomenon? How does the DropChannel and DropElement perform when using different dropout probability beyond 0.5?\n\n2. It is unclear to me why computational complexity is linear for DropCov while quadratic to element-wise methods (in Table 1).\n", "limitations": "The paper analyzes post-normalization for global covariance pooling unit and introduces channel dropout to improve it, which is not restricted to certain downstream cv tasks. It seems like no potential negative societal impact.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The authors provide a detailed analysis in the effect of the power parameter of matrix power normalization on feature decorrelation and information preservation, and consequential impact on model performance empirically, which is valuable for related research area. \n\n2. The paper is generally well written and easy to follow, although some statements need to be explained and clarified.\n\n3. Extensive experiments on representative traditional and modern backbones are conducted to validate the method. \n\nWeaknesses:\n1. Some descriptions are less rigorous and need to be clarified:\n\nThe first corollary (line 120) states that setting power parameter 0.5 is optimal according to Eq. 2 and Fig. 1, while it lacks either theoretical proof or comprehensive empirical studies. Is the behavior of power parameter fairly invariant to dataset or network backbone? \n\nFor the proposed Adaptive Power Normalization, the power parameter is optimized according to eigenvalues of feature covariance, while the features are updated during training. How does it realize in practice? There is a similar question for the dropout probability in Eq. 5. Could the authors provide their training behavior if they are variable during training? If the parameters are adaptive with model training from scratch, primitive parameters may be far from optimal values. On the other hand, if the variance of probability distribution is rather small, it may indicate less necessary for tuning it during training. \n\n2. The motivation of performing dropout on convolution channels is not new. The main difference is specializing the strategy with global covariance pooling. \n\n3. Equipping DropCov into baseline backbones (e.g., ResNet50) brings significant increase on parameter size and computational overhead. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1CUngiee1q", "reviewer": "Reviewer_5eq4", "summary": "This paper focuses on the post-normalization of the pooling strategy leveraging second-order statistics, i.e., global covariance pooling (GCP).  The key contributions are twofold: 1) the paper shows that post-normalization in matrix power normalization [27] tries to balance representation decorrelation and information preservation through empirical analysis; 2) the paper proposes a linear complexity pre-normalization DropCov of GCP based on an adaptive channel dropout. \n\nThe proposed approach has demonstrated its effectiveness on several image classification benchmarks including ImageNet, ImageNet-C, ImageNet-A, stylized ImageNet, and long-tailed classification with different backbones (CNN, transformer). \n", "questions": "Please refer to the weaknesses.\n", "limitations": "No potential negative societal impact.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**: \n* [S1]: I think the main strength of this work is the experiments. The paper provides diverse experiments on different datasets with different backbones to validate the proposed DropCov. \n\n* [S2]: The paper is presented very well and reads very well. I read quite carefully but did not find any typos. The paragraphs and sections are also self-contained and organized well.\n\n* [S3]: The analysis of MPN (Sec.2.1) is intuitive, and the empirical results are interesting.\n\n**Weakness**:\n* [W1]: I think the main drawback of this work is the significance. I can acknowledge the improvement is clear, the idea is simple. But as GCP is computationally more expensive than 1st order pooling, not much attention is paid to the field.\n\n* [W2]: I think the analysis on MPN is a little shallow. the experiments are on a single dataset with a network. It would be interesting to see the effect of power on different sizes of datasets with different capacities of models. \n \n* [W3]: For the adaptive power normalization, the improvement with the proposed formulation is quite marginal. Moreover, I am wondering whether extra visualization would provide some insight. For example, it might be possible to visualize samples with important/small alpha to verify corollary 1 further.\n\n* [W4]: Equation 5 is complicated and not intuitive. For example, the cosine similarity between the feature importance and feature correlation is weird. Also, this similarity might be negative and cause the probability larger than 1? Equation 6 might be confused as well. The dropout should be only applied once on $X$ (not separately for $X$ and $X^T$)?\n\n* [W5]: It would be more convincing to conduct different dropout ratios (rather than a fixed ratio of 0.5) to validate the effect of adaptive dropout. \n\n* [W6]: It is not apparent to understand the intuition behind the experiments of “element” and “channel” dropout. For example, in lines 275-276, the paper emphasizes applying one type of dropouts is equivalent to considering only feature correlation or feature importance. It would be better to provide a more detailed explanation on this part.  \n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept