Title: Revitalizing SVD for Global Covariance Pooling: Halley’s Method to Overcome Over-Flattening
Venue: NeurIPS 2025
Accepted: yes
Keywords: Global Average Pooling;
Abstract: Global Covariance Pooling (GCP) has garnered increasing attention in visual recognition tasks, where second-order statistics frequently yield stronger representations than first-order approaches. However, two main streams of GCP---Newton--Schulz-based iSQRT-COV and exact or near-exact SVD methods---struggle at opposite ends of the training spectrum. While iSQRT-COV stabilizes early learning by avoiding large gradient explosions, it over-compresses significant eigenvalues in later stages, causing an \emph{over-flattening} phenomenon that stalls final accuracy. In contrast, SVD-based methods excel at preserving the high-eigenvalue structure essential for deep networks but suffer from sensitivity to small eigenvalue gaps early on. We propose \textbf{Halley-SVD}, a high-order iterative method that unites the smooth gradient advantages of iSQRT-COV with the late-stage fidelity of SVD. Grounded in Halley's iteration, our approach obviates explicit divisions by $(\lambda_i - \lambda_j)$ and forgoes threshold- or polynomial-based heuristics. As a result, it prevents both early gradient explosions and the excessive compression of large eigenvalues. Extensive experiments on CNNs and transformer architectures show that Halley-SVD consistently and robustly outperforms iSQRT-COV at large model scales and batch sizes, achieving higher overall accuracy without mid-training switches or custom truncations. This work provides a new solution to the long-standing dichotomy in GCP, illustrating how high-order methods can balance robustness and spectral precision to fully harness the representational power of modern deep networks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nHalley-SVD identifies an intriguing phenomenon: when applied to larger-scale networks or with increased batch sizes, the iSQRT-COV method tends to compress the eigenvalue spectrum of the covariance matrix. Specifically, it reduces larger eigenvalues while amplifying smaller ones, resulting in a more uniform distribution of eigenvalues. This eigenvalue \"flattening\" effect diminishes the discriminative power of the covariance representation, ultimately limiting the model's performance. Through theoretical analysis, Halley-SVD attributes this issue to the iterative update rule inherent in the iSQRT-COV method. To address this limitation, the authors propose Halley-SVD, a modified approach based on Halley's method for approximating the matrix square root. Empirical results demonstrate that Halley-SVD consistently improves performance across various network architectures, highlighting its effectiveness in preserving the discriminability of global covariance features.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n1.Identifies an intriguing phenomenon: when applied to larger-scale networks or with increased batch sizes, the iSQRT-COV method tends to compress the eigenvalue spectrum of the covariance matrix.\n\n2.This work proposes a novel approach to mitigate the performance limitations of iSQRT-COV observed under larger batch sizes or increased model scales.\n\n3.Extensive experimental results are provided, including training accuracy across different model scales for iSQRT-COV, as well as convergence comparisons between iSQRT-COV and MPN-COV. The experimental setup is thorough and supports the validity of the conclusions.\n\nWeaknesses:\n\n1.Concerns Regarding Experimental Data: Although Figures 1 and 2 in the paper provide extensive training accuracy results of iSQRT-COV across various model scales and datasets, several issues remain unclear and raise concerns. \n\n1.1 Specifically, the exact experimental settings used to generate the training results in Figures 1 and 2 are not clearly described. \n\n1.2 Moreover, some inconsistencies in the reported results are noteworthy. For instance, Figure 1 appears to correspond to a ResNet-101 model according to the textual description; however, the peak accuracy and the epoch at which it is achieved in Figure 1(a) seem to differ from those shown in Figure 2(c) for the same configuration. Similarly, the training result of iSQRT-COV with batch size 256 in Figure 1(c) is inconsistent with the corresponding result reported in Table 1 for ResNet-101 with batch size 256. Furthermore, the performance of the Swin Transformer shown in Figure 2(g) also differs from that reported in Table 1.\n\n2.Further Theoretical Analysis:\n\nClarification on the Relationship Between Over-Flattening and Model/Batch Scale: While the paper presents empirical evidence suggesting that the performance of iSQRT-COV deteriorates on larger models and with larger batch sizes due to over-flattening of the covariance spectrum, a more thorough theoretical explanation of this phenomenon would significantly strengthen the paper. \n\n3.Comparison with Recent Normalization for GCP:\n\nThe proposed method can essentially be interpreted as a form of post-regularization applied to Global Covariance Pooling (GCP). Given this perspective, it would be beneficial for the authors to position their work about recent advances in this area. In particular, the paper would be strengthened by comparison with recent methods such as DropCov (NeurIPS 2022)[R1], which also aims to improve GCP representations through simple yet effective normalization techniques.\n\n[R1] Wang Q, Gao M, Zhang Z, et al. DropCov: A simple yet effective method for improving deep architectures[C]. Advances in Neural Information Processing Systems, 2022: 33576-33588.\n\nQUESTIONS:\n1. Clarification on the Over-Flattening Phenomenon and Visualization. The paper defines the over-flattening phenomenon as the tendency of iSQRT-COV to suppress large eigenvalues and amplify small ones during training, thereby leading to a more uniform eigenvalue distribution. To support this claim, the authors provide an eigenvalue spectrum comparison between iSQRT-COV and Halley-SVD in Figure 1(b).\nHowever, upon close examination of the visualization, the distinction between the two spectra appears relatively subtle. It seems that iSQRT-COV may exhibit a larger maximum eigenvalue, while the smallest eigenvalues (on the far right of the x-axis) appear lower than those in Halley-SVD, which may contradict the stated over-flattening effect. This raises questions regarding the consistency between the empirical evidence and the interpretation provided.\n\n2. In addition to the proposed Halley-SVD method, it would be worth investigating whether the over-flattening issue associated with iSQRT-COV can be mitigated through alternative optimization strategies. For instance, the choice of optimizer (e.g., SGD vs. Adam), learning rate schedules, or the initialization of the learning rate itself may influence the dynamics of covariance matrix learning and eigenvalue evolution during training.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGlobal Covariance Pooling is designed to replace the last global average pooling in classification backbone networks. Among various GCP methods, the paper improves iSQRT-COV. iSQRT-COV uses a square root matrix of the covariance matrix for network output, which is computed by approximation based on Newton-Schulz. In this framework, the proposed Halley-SVD replaces Newton-Schulz with Halley iterative approximation method. Although Halley-SVD costs more computation than iSQRT-COV, it solves problems of iSQRT-COV and demonstrates performance improvements in image classification tasks on various network architectures.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The motivation of the paper is well described with various formulas and experiments.\n- The benefits of Halley-SVD are well supported by theoretical evidence.\n\nWeaknesses\n- Outdated topic\n  - The topic is old and outdated. The most relevant papers were presented 7-8 years ago: iSQRT-COV [CVPR 2018] and MPN-COV [ICCV 2017]. \n  - The most recent comparison method, SVD-Pade [ICCV 2021], is difficult to consider recent.\n  - As a researcher in ImageNet classification, I believe this topic (GCP) is definitely out of the mainstream. Thus, it is hard to impact and contribute to the community.\n  - As mentioned in the introduction, a few recent papers [39, 32, 36] use GCP. However, their topics—[32] video recognition, [45] graph neural network, and [39] color image steganalysis—are significantly different from this paper's evaluation: image classification. Overall, it is hard to resolve concerns on outdating and significance. \n- Computation costs\n  - The computation cost is a significant weakness of GCP. It improves performance but reduces speed. As reported in Table 4, GCP requires x1.5 - x2.0 costs for less than 1.0 pp. improvements, which underperforms performance-speed trade-off in recent image classification research. It significantly limits the practical value of GCP.\n  - As reported in Table 4, the proposed method, Halley-SVD, costs more time than iSQRT-COV. Compared to iSQRT-COV, Halley-SVD increases the latency by ~50% and reduces throughput by ~25%. It has the smallest throughput in GCP methods. I don't think it is acceptable overhead for the marginal performance improvements.\n- Limited novelty and marginal improvements\n  - Halley-SVD shares the objective with iSQRT-COV: approximates the square root matrix. Thus, it naturally limits the novelty. It can be a better approximation but can't be a novel pooling method.\n  - The performance improvements reported in Table 1 look marginal. Without a large batch size, the gaps between iSQRT-COV and Halley-SVD are below 0.3 pp. On a large batch setting, improvements increase but are still limited below 0.9 pp., and I doubt the value of the large batch setting since it achieves performance degradation in most cases.\n\nQUESTIONS:\n- In Figure 1, iSQRT-COV achieves poor last performance due to late flattening. But, in Table 1, iSQRT-COV shows comparable performance with other methods. Why are these two results different? Is there any special way to prevent the flattening?\n- The formula shows that Halley-SVD uses matrix inversion in its iteration. How could it handle matrix inversion? Does it have affordable computation?\n- ViT and Swin transformer performance aligns with recent recipes: DeiT and Swin. But, ResNet performance looks poor. Is Halley-SVD still effective on recent training recipes like RSB?\n  - [RSB] ResNet strikes back: An improved training procedure in timm\n- As far as I know, the large batch setting maximizes GPU utilization and extremely accelerates training, which does not align with GCP's computation overheads. Why is the large batch setting important for GCP?\n- Can Halley-SVD improve the performance of recent GCP papers  [39, 32, 36]? Is there any empirical evidence on it?\n\nLIMITATIONS:\nI recommend to move the computation comparison in Table 4 to the main paper. Since the computation is the major issue in GCP and is addressed in the main section of previous papers: iSQRT-COV and SVD-Pade, I think it should be discussed in the main paper to address the limits of GCP.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper revisits Global Covariance Pooling (GCP) for visual recognition, identifying that the widely used Newton–Schulz method (iSQRT-COV) over-flattens the covariance eigenspectrum under deep or large-batch training, limiting accuracy. To address this, the authors propose Halley-SVD, a higher-order square-root iteration that avoids unstable SVD gradients, better preserves large eigenvalues, and introduces no extra hyperparameters. Experiments on ImageNet (ResNet-50/101, ViT-B/16, Swin-T) show comparable results to iSQRT-COV under standard settings and up to 0.9 pp Top-1 improvement with large batches.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- *Technical soundness* The paper presents a rigorous derivation of the Halley update and its gradient. Analytical comparisons with Newton–Schulz on diagonal Σ demonstrate that Halley-SVD reduces shrinkage on large eigenvalues (λ).\n- *Extensive empirical evaluation* The method is tested on both CNNs and Transformers with batch sizes ranging from 256 to 4 096. Implemented in double precision, it systematically reports speed–accuracy trade-offs.\n- *Timely relevance* The work addresses two prominent topics: large-batch training and second-order pooling. Halley-SVD offers a principled alternative to the common ad hoc “switch-to-SVD-late” heuristics.\n________________________________________\n### Weaknesses\n- *Marginal gains with significant overhead* Halley-SVD delivers small accuracy improvements (typically <1 pp, Table 1) over iSQRT-COV and SVD-Padé, but incurs considerably higher memory and runtime costs (up to 33 % and 29 %, respectively; see Table 4). For many practical settings, such trade-offs may favor simpler alternatives.\n- *Internal inconsistency* In Figure 2(c), the blue curve (labeled as iSQRT-COV in panel (h)) peaks near 78 % but collapses to approximately 60 % by epoch 60. However, Table 1 reports 78.3 % /77.8% for the same method and backbone at epoch 100 with a batch size of 256/2048. A rebound from 60 % to 78 % under the stated cosine schedule seems implausible and is not currently explained.\n- *Misinterpretation of $\\kappa_{\\text{flat}}$* – Equation (2) defines $\\kappa_{\\text{flat}}$ as GM/AM. By the AM–GM inequality, a smaller value indicates more anisotropy, not “more uniformity.” Thus, the claim that “$\\kappa \\downarrow \\Rightarrow$ over-flattening” is mathematically inverted.\n- *Incomplete ablations* The impact of Halley iteration depth ($K=3\\ldots15$) under large-batch regimes is not explored, and the viability of using fp32 remains untested.\n\nQUESTIONS:\n- *On cost-effectiveness*\nGiven that Halley-SVD offers modest accuracy gains (≤ 1 pp) while incurring substantially higher memory and runtime costs (as shown in Table 4), could you elaborate on realistic scenarios where such trade-offs are justified?  I suggest moving Table 4 to the main paper for a fairer comparison with iSQRT-COV and SVD-Padé.\n- *On learning curve inconsistency*\nIn Figure 2(c), the curve labeled iSQRT-COV drops to ~60 % by epoch 60, yet Table 1 reports 78.3 % at epoch 100 for the same method/backbone. How is this sharp recovery explained under the cosine learning rate schedule? \n- *On interpretation of $\\kappa_{\\text{flat}}$*\nEquation (2) defines $\\kappa_{\\text{flat}}$ as GM/AM. By the AM–GM inequality, a smaller $\\kappa$ implies greater anisotropy. Yet the manuscript states that “$\\kappa \\downarrow \\Rightarrow$ over-flattening,” which appears mathematically inverted. Could you clarify this discrepancy?\n- *On missing ablations*\nHave you examined the effect of varying Halley iteration depth ($K = 3 \\ldots 15$) under large-batch conditions? Also, is there any evidence that using fp32 meaningfully changes the observed trade-offs?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper proposed a high-order iterative method Halley-SVD that unites the smooth gradient advantages and late-stage fidelity, designed to be robust and efficient for GCP. In the experiment, the proposed Halley-SVD is shown to outperform various SVD-based solutions.

STRENGTHS AND WEAKNESSES:
Strengths
Global Covariance Pooling is important for various tasks in deep learning. 
The paper systematically analyzes the "Over-Flattening" phenomenon of iSQRT-COV and proposes the corresponding method to solve it.
Comprehensive experiments and ablation studies demonstrate the robustness and effectiveness of the proposed method.

Weaknesses
The completeness of this work}: In my view,  Halley-SVD combines two significant strengths: the smooth gradient advantages in the early training stage and late-stage fidelity. However, the paper mainly focus on how to gain the smooth gradient advantages in the early training stage for SVD based method. There is relatively less discussion regarding the second advantage. (Only a few words are described in Section 2.3.) The addition of some relevant experimental arguments would enhance the paper.
The efficiency of the method}: As can be seen in the Table~4 of the appendix, the computational efficiency of Halley-SVD seems to be insufficient.
The missing detailed illustration: The Fig2.h presents results without specifying the underlying network architecture and dataset.
The missing discussion of limitation: The discussion of the limitations of this method would make the paper more comprehensive.

QUESTIONS:
Please see the Weaknesses.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02

comments: SUMMARY:
This paper introduces Halley-SVD, a high-order iterative method that combines the smooth gradient characteristics of iSQRT-COV with the high-precision convergence of SVD. The authors first observe that iSQRT-COV suffers from severe "over-flattening" when applied to deeper networks or with larger batch sizes. To address this issue, the paper proposes a refined SVD-based iterative approach that mitigates early gradient explosion and prevents the over-flattening observed in iSQRT-COV. Extensive experiments across various network architectures have been conducted to validate the effectiveness of the proposed method.

STRENGTHS AND WEAKNESSES:
Strength

1. The analysis and proposed Halley-SVD are well-motivated and supported by theoretical proof.
2. The proposed Halley-SVD demonstrates better performance in large-batch training regimes.
3. The paper is well-written and easy to follow.

Weakness

1. [1] should be included for comparison in Table 1.
2. Can the presented method be utilized in another way, for example, channel attention or AdaIN? How about a local version of Halley-SVD that achieves more representative features in other vision tasks?
3. How about the latency, peak mem, and throughput during inference?

[1] Wang, Qilong, et al. "Towards a deeper understanding of global covariance pooling in deep learning: An optimization perspective."

QUESTIONS:
See weaknesses

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-05