Revitalizing SVD for Global Covariance Pooling: Halley’s Method to Overcome Over-Flattening
Authors: Jiawei Gu
Ziyue Qiao
Xinming Li
Zechao Li
Venue: NeurIPS 2025 poster ()
Keywords: Global Average Pooling;
Primary area: deep_learning

Abstract: Global Covariance Pooling (GCP) has garnered increasing attention in visual recognition tasks, where second-order statistics frequently yield stronger representations than first-order approaches. However, two main streams of GCP---Newton--Schulz-based iSQRT-COV and exact or near-exact SVD methods---struggle at opposite ends of the training spectrum. While iSQRT-COV stabilizes early learning by avoiding large gradient explosions, it over-compresses significant eigenvalues in later stages, causing an \emph{over-flattening} phenomenon that stalls final accuracy. In contrast, SVD-based methods excel at preserving the high-eigenvalue structure essential for deep networks but suffer from sensitivity to small eigenvalue gaps early on. We propose \textbf{Halley-SVD}, a high-order iterative method that unites the smooth gradient advantages of iSQRT-COV with the late-stage fidelity of SVD. Grounded in Halley's iteration, our approach obviates explicit divisions by $(\lambda_i - \lambda_j)$ and forgoes threshold- or polynomial-based heuristics. As a result, it prevents both early gradient explosions and the excessive compression of large eigenvalues. Extensive experiments on CNNs and transformer architectures show that Halley-SVD consistently and robustly outperforms iSQRT-COV at large model scales and batch sizes, achieving higher overall accuracy without mid-training switches or custom truncations. This work provides a new solution to the long-standing dichotomy in GCP, illustrating how high-order methods can balance robustness and spectral precision to fully harness the representational power of modern deep networks.

Revitalizing SVD for Global Covariance Pooling: Halley's Method to Overcome Over-Flattening
Jiawei Gu', Ziyue Qiao', Xinming Li Zechao Li
School of Computer Science and Engineering, Nanjing University of Science and Technology,
"School of Computing and Information Technology, Great Bay University,
Beijing University Civil Engineering and Architecture g gjwcs@outlook com, jrmuejil@ com 1xm 1xmm1813446291110 com, zechao li@njust. edu, cn
Abstract
Global Covariance Pooling (GCP) has garnered increasing attention in visual recog- nition tasks, where second-order statistics frequently yield stronger representations than first-order approaches. However, two main streams of CCP--Newon-Sechee based iSQRT-COV and exact near-exact SVD methods-struggle ends the training spectrum. While iSQRT-COV stabilizes early learning by avoiding large gradient explosions, over-compresses significant eigenvalues in later stages, causing an over-flattening phenomenon that stalls final accuracy. In contrast, SVD-based methods excel preserving the high-eigenvalue structure essential for deep networks but suffer from sensitivity to small eigenvalue gaps early on. We propose Halley-SVD, a high-order iterative method that unites the smooth gradient advantages of iSQRT-COV with the late-stage fidelity of SVD. Grounded Halley's 's iteration, our approach obviates explicit divisions by (A: and forgoes threshold- polynomial-based heuristics. As a result, it prevents both early gradient explosions and the excessive compression of large eigenvalues. Ex- tensive experiments on CNNs and transformer architectures show that Halley-SVD consistently and robustly outperforms iSQRT-COV large model scales and batch sizes, achieving higher overall accuracy without mid-training switches custom truncations. This work provides a new solution to the long-standing dichotomy in GCP, illustrating how high-order methods can balance robustness and spectral precision to fully harness the representational power modern deep networks.
opposite
Introduction
Global Covariance Pooling (GCP) has recently emerged as a powerful and increasingly popular strategy in visual recognition tasks, including large-scale image classification and fine-grained object categorization 51 57 which retains only first-order statistics, GCP captures the second-order statistics namely, covariances among deep feature channels. Such second-order descriptors often lead to more expressive and discriminative network representations and improved classification accuracy across diverse datasets [4 36 35|
contrast to traditional Global Average Pooling
(3 I 26 43
3].
Early research has explored various ways to compute the matrix square root of the covariance matrix within GCP. Among them, two competing paradigms have attracted substantial attention: (i) SVD- based GCP, often referred to MPN-COW COV[5,,,25), which theoretically yields the accurate matrix square root but suffers from numerical instability when eigenvalues are close; (ii) iSQRT-COV (4).
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

relying on the Newton-Schull iteration to approximate the covariance root, provides surprisingly smoother gradients and has empirically outperformed its more "accurate" SVD counterpart wide range of benchmarks. This discrepancy has been viewed as somewhat counterintuitive, often attributed to the less severe gradient explosions encountered by iSQRT-COV in practice (49 17 27
in Large-Scale Settings. Despite its demonstrated effective- ness, we discover that iSQRT-COV can encounter severe "over-flattening" when scaling up to deeper networks (e.g. ResNet-101, Vision Transformers) or larger training batches. Concretely, as training progresses, iSQRT-COV progressively compresses all covariance eigenvalues toward a uni- form magnitude, eventually erasing the prominent directions corresponding to the largest eigenvalues. This over- flatttnnnn phenomenon significantly stalls the model's final accuracy, preventing it from leveraging the full representational capacity offered by large architectures. In contrast, once classic SVD-based methods successfully mitigate their early-stage gradient instability, they better preserve large eigenvalues and maintain higher capacity to learn discriminative features in later epochs.
A New Challenge:
To illustrate these points, we present in Figure I a representative example of training a ResNet-101 model on the ImageNet dataset with GCP, comparing iSQRT-COV and our novel SVD variant. It is evident that while iSQRT-COV reaches a performance plateau relatively early, our enhanced SVD approach keeps improving in terms of final accuracy. Moreover, analyzing the covariance eigenvalue spectra carefully reveals that iSQRT-COV heavily flattens high eigenvalues, a phenomenon that becomes significantly more pronounced when increasing the batch size or the depth of the network.
Our Approach: Revitalizing SVD with- out Thresholding. While iSQRT-COV initially gained favor for its stability, the late-stage compression now emerges as a bottleneck. This observation naturally mo- tivates us to revisit SVD-based methods. If the problematic gradient blow-ups in SVD could be curtailed, yet without heavy hyper-paane parameter tuning or complicated thresholding. then SVD-based GCP might consistently surpass iSQRT-COV in large- seale regimes. Our work aims to bridge this gap by introducing high-order ma- trix square root iteration for SVD that al- leviates early-stage numerical instability, ensures stable gradients, and preserves the largest cigenvalues in later epochs.
POTOYY
Figure I: Over-Flattening vs. Improved SVD in Large- Scale GCP. (a) Training curves, where iSQRT-COV satu- rates early while our improved SVD keeps rising; (b) Co- variance eigenvalue distribution. highlighting iSQRT's excessive compression of large eigenvalues, (c) Com- parison of final accuracies at various batch sizes, where iSQRT consistently lags further behind as scale grows.
Contributions and Outline.
We summarize our main contributions as follows:
Diagnosis of Over-Flattening. We systematically analyze how iSQRT-COV's iterative scheme progressively flattens feature spectra in deeper networks or larger-scale training tasks, showing precisely why this limitation imposes strict performance ceiling on classification accuracy.
High-Order SVD Method. We propose a novel high-order iteration (named "Halley-SVD") that effectively and elegantly neutralizes SVD's gradient explosion problem, without relying on manual thresholds, additional Broad Experimental Validation. On ImageNet, fine- mne-grained benchmarks, and Vision Transform- ers, we demonstrate that the improved SVD method consistently outperforms iSQRT-COV at high scale, thus unveiling promising new path to fully exploit the potential of SVD-based GCP.
or any piecewise function definitions.
The remainder of this paper is organized as follows. Section investigates and quantifies the over- flattening effect, establishing its numerical root causes. Section 3 introduces the proposed high-order SVD iteration. Section
4 provides theoretical insights into why Halley-SVD better preserves large eigenvalues than Newton-Schul) iterations. Experimental results in Section 5 confirm the superiority of our approach across varied networks and challenging vision tasks. Finally, Section concludes with future prospects, while the Appendix offers additional derivations and ablative studies.

---

(1) BealNet-9/ IangeNett
(b) BesNell 50 CCEARRL00
ResNec- 101 /IasgeNNt
ReaNNe-- /CIFAR110)
Accurac (%4
Auturayy (NN)
Accuracy (54)
60
Acurray
12
-2
SKRTE-OV SVD4avd
26
36
20
16
40
28
14
4
Epochs
Egecths
Epocks
Epochs
(4) VLT ImageeNa
(0 CIPAR---
(g) Swin/ Largs Hatth
@ Speetall Flatnna Indox
12-
2.0
Accrni (50
42-
Acoracy (6)
714
210
0.90
16.1
76.1
0.85
25-
18
6100 CNN
Dosa RVD
Epochs
Epecchs
Kpech
Figure 2: Comparative Analysis of iSQRT-COV vs. SVD-based GCP in Various Networks and Datasets. We present eight subplots, spanning ResNet-50 and ResNet-101 on ImageNet/CIFAR-ID as well as Vision Transformer and Swin Transformer cases. (a,b) iSQRT-COV (in red) outper- forms SVD (in blue) initially but saturates, (c,d) this effect is amplified on deeper ResNet-101, (ef) transformer-based architectures likewise reveal late flattening by iSQRT-COV, (g) large-batch Swin training sees iSQRT-COV lagging in final accuracy, and (h) the spectral flatness index Kitat con- firms that iSQRT-COV over-compresses significant eigenvalues. Collectively, these results highlight why iSQRT-COV, despite early stability, becomes a limiting factor for deeper or larger-scale tasks, while SVD-based methods eventually exploit their capacity to maintain richer eigenvalue spreads.
2 Investigation
The performance gap between iSQRT-COV and SVD-based methods is historically well understood in the mid-scale regime. With architectures such as ResNet-50[14] on ImageNet(6) and batch sizes up to few hundred, iSQRT-COV has been shown to outperform exact SVD (or mild SVD variants), primarily because the Newton-Schulz iteration avoids the notorious gradient instability that arises when eigenvalues of the covariance matrix become very close 20 19]. By contrast, exact SVD incur terms like 1/(A; Xς) in its gradient calculation, often leading to explosions if X = Xz- In moderate dimensions, iSQRT-COV's iterative balancing effectively whitens the covariance in a smooth manner giving it distinct advantage over the more delicate SVD backpropagation.
However, deeper experiments with networks such as ResNet- 14] or Transformers 29 7, com- bined with larger batch sizes (exceeding 1k), reveal a remarkable shift. Although iSQRT-COV preserves its initial stability, it exhibits a significant saturation later training, and its accuracy soon platcaus. More precise approaches like exact SVD or SVD-Padé continue to gain accuracy in the final stages of optimization, converging to superior results. This phenomenon indicates that while iSQRT-COV is adept at taming early gradients, it imposes persistent and systematic over-flattening of the covariance spectrum that ultimately restricts the network's 's capacity to discriminate subtle features.
2.1 ISQRT-COV vs. SVD at Different Scales
Early studies predominantly focused on scenarios such as ResNet-50 trained on ImageNet with batch sizes up to few hundreds. Under these special conditions, iSQRT-COV typically outperforms exact SVD or minor variations thereof, since the Newton-Schin iteration very effectively avoids direct divisions by small (), Xy), resulting in notably smoother gradients. The iSQRT-COV update,
X = 1 2 Xx (31 (31xx))
(1)
3

---

mitigates the hallmark "gradient explosions" of SVD- approaches, especially when eigenvalues of the covariance matrix P are too close. As a result, iSQRT-COV reliably converges even in mid-scale settings where a few large eigenvalues might otherwise trigger unstable derivatives in SVD.
However, deeper architectures, such as Res transformer models (Vision Transformer, Swin Transformer), combined with large batch sizes (2 1k), present new interesting twist. Whereas iSQRT-COV maintains its early stability, its final accuracy often saturates noticeably lower level. In contrast, precise SVD-based methods (including SVD-Padé) are initially more fragile, but once properly stabilized, they preserve stronger eigenvalue contrasts and achieve far better late-stage results, Empirically, we observe that the fundamental switch in dominance from iSQRT-COV to SVD occurs in a regime where the network capacity is large enough that capturing fine-graine fine-grained feature distinctions and nor flattening high cigenvalues becomes essential for the final performance.
2.2 Over-Flattening of the Covariance Spectrum
The crux of iSQRT-COV's late-stage saturation lies in a systematic over-flattening of the feature covariance spectrum. Each iteration in 4 nudges large eigenvalues downward and propels small cigenvalues upward, gradually forcing a too uniform spread of X 's eigenvalues. When training on high-dimensional embeddings with massive data, these adjustments become very pronounced. The beneficial "white balancing" [[88] that helps with early stability turns into an excessive compression of significant eigenvalues, fundamentally limiting the network's 's capacity distinguish subtle features.
To measure this compression rigorously, we track a spectral flatness index:
Kilat (X) expl 1 E (x))
1 {x (X)
where (X)) are the eigenvalues of X, approximation to p1/2 A lower Khat signifies more uniform (flattened) distribution. In large-scale experiments, iSQRT-COV steadily drives kitta downward, characteristically highlighting how prominent eigenvalues are being overly suppressed. By contrast, a well stabilized SVD retains higher variance in the top eigenvalues. culminating in superior accuracy when the learning rate decays and the training seeks fine discriminative cues.
Figure 2 compiles subplots across different networks and datasets, illustrating how iSQRT-COV (in red) and SVD-based method (in blue) evolve. Subplots (a) and (b) demonstrate the phenomenon on ResNet-50 for ImageNet and CIFAR-100. iSQRT-COV climbs swiftly but flattens in later epochs, leaving a small yet gap for the SVD-based approach surpass it. Subpiots (c) and (d) shift to ResNet-101 on ImageNet and CIFAR-100, where the deeper architecture accentuates iSQRT-COV's limiting behavior, while SVD steadily refines large eigenvalue directions in the final epochs.
Moving to transformer- based models, subplots (c) and (f) analyze Vision Transformer on the same two datasets, revealing similar pattern. Sclf-attention mechanisms benefit from capturing fine feature correlations, but iSQRT-COV flattens the spectral structure, capping final performance. The SVD-based approach starts off more erratic, reflecting its sensitivity to early eigenvalue collisions, yet preserves distinguishing variance once stabilized. Subplot (g) highlights final accuracies for a Swin Transformer under large batch sizes, where iSQRT-COV consistently lags behind an -VD-based alternative. Finally, subplot (h) plots KHas over epochs, comprehensively quantifying the increasing uniformity iSQRT-COV imposes. These experiments confirm that while iSQRT-COV is superb at mitigating explosions in moderate scales, it becomes a bottleneck for deeper models and larger data.
2.3
The Potential of SVD for High-Preeision Late Training
From the viewpoint of final accuracy, a precisely maintained eigenvalue spectrum remains crucial in deeper models or extensive data contexts. SVD-based methods, if effectively curbing early gradient spikes, can preserve the top eigenvalues of P, ensuring that significant variations among classes or features are not lost. Indeed, SVD-Padé leverages rational function approximations to carefully sidestep abrupt divisions by small (A; - >;). but it may still rely on explicitly parameterizing polynomials or thresholds to handle near-duplicates in X A perfectly stable iterative scheme that completely obviates these thresholds yet guides large eigenvalues much more gently, in principle, could very successfully unify iSQRT-COV's early smoothness with SVD's late-stage fidelity.

---

Therefore, the insights gleaned from these empirical curves and numerical analysis highlight genuine opportunity for a refined SVD iteration capable of neutralizing the early risk of exploding gradients and preventing the over-flattening outcome of iSQRT-COV. In Section 4, we introduce precisely such an approach: high-order matrix square root iteration thoughtfully derived from the SVD principle, free from explicit divisions by (A, Xj) and completely avoiding truncated or piecewise definitions. As the subsequent experiments extensively confirm, this novel method effectively capitalizes on deeper networks and large-batch training without succumbing to spectral collapse, ultimately surpassing iSQRT-COV once the network enters its advanced phases learning.
3 Proposed SVD Refinement: Halley-SVD
3.1 Theoretical Motivation: Newton vs. Halley Iteration
To conquer the over-flateening introduced by the Newton-Schull iteration in iSQRT-COV, can strategically look to higher-order iterative methods for the challenging matrix square root problem. Newton-Schulz itself naturally stems from directly viewing
x² Σ 0,
(3)
root-finding problem f(X) = X2 - E 0. Standard Newton's method for scalars can be
generalized to matrices, yielding the update
X
1 X 31
-x.)
which is precisely the iteration used by iSQRT-COV. Although this method reliably converges and avoids the explicit x,-x; denominators that plague SVD gradients, it suffers from repeatedly compressing large cigenvalues in practice, ultimately yielding an over-flattened spectrum.
Scalar Halley Method. In the scalar setting, more sophisticated root-finding technique is Halley's method, which can be derived from a third-order expansion of the correction term. Suppose we want to solve g(x) x² a = 0 with x > 0. Newton's 's method prescribes
g(rx) g ((xx)
#
F&+1 = 2g
By contrast, Halley's iteration includes higher-order corrections:
g( o' (xx)
rk+1 =
Fk
g'(xx)² - z g(xx) (r()))
=x--a. and exhibits cubic convergence for many smooth g(-). In the specific case g(x) = =x² -aa a. this leads
T6+1 26 1 (1-
which typically converges faster than the Newton scheme once is moderately far from 1.
3.2 Derivation of Matrix Halley Iteration
We now generalize Halley's 's iteration matrices for the same root-finding problem in i.e., solving x Σ with Σ positive definite. Define:
f(X) x Σ, VX(x= = 2x. V 21.
(5)
The
Halley update can be written analogously to the scalar case by including second- order (and partial third-order) terms. In particular:
X = X [V)(
(6)
(f(X)). v2f(XE))
where one interprets matrix inversion and the inner products in properly defined sense. Specializing
to fXX) X² Σ, (X()) = 2X, and fx) 2I, one obtains a matrix Halley iteration that avoids explicit denominators (^; - >;) and yields:
X = X I + (I (I-X, Ex,
In practice, few iterations (e.g., 5-10) suffice to approximate 1/2 with high accuracy.
5

---

3.3 Why Halley Avoids Over-Flattening
Unlike the Newton-Schulz iteration 4 which repeatedly pushes the largest eigenvalues toward smaller magnitudes in order to enforce approximate "whitening", Halley's update 77) applies additional second-order correction. Concretely, if Xmax is the largest eigenvalue of X the extra factor
(I
tends to reduce the over-penalty on large Amax. thus making the iteration step for Amax less aggressive than in Newton-Schulz. Qualitatively, one see that the second-order term in Halley's method partially "pulls back" the matrix update if X has already grown close to certain eigen-directions, mitigating further compression of those principal components. Moreover, the cubic convergence rate often noted in scalar Halley's iteration indicates that once X₃. is in a reasonable neighborhood of 1//2 each subsequent update refines the approximation in fewer steps than SewuunnSchulz. This not only leads to faster convergence but also less iterative drifting of large eigenvalues after they have become stable, effectively preventing iSQRT-COV's late-stage "eigenvalue erasure."
3.4 Forward & Backward Pass: A Smooth Gradient Framework
In the forward pass of our GCP layer, the Halley update 7 is computed iteratively to derive the approximate covariance root Xx = 11/ As with iSQRT-COV, all steps remain completely free of explicit (A: X1) For the backward pass, follow matrix calculus framework similar to that in iSQRT-COV [24] but carefully adapt the chain rule to Halley's iteration. Specifically, let l be the training loss. Then one ultimately needs T5 which is obtained by systematically unrolling the differentiations through each Halley update. Concretely, if Xx+1 is defined by 7 one can write
ae ΦΣ
Ol 0X&+1
0x OΣ
8x+11 xΣ
and
A(X,, E),
(8)
k=0
where A(-) is the corresponding Jacobian derived from (omitted here for brevity). In practice, most modern deep-learning libraries can implement automatic differentiation this iterative process directly, yielding remarkably stable gradients without requiring threshold-baat based or polynomial expansions. As a result, Halley-SVD can scamlessly integrated into CNN or Transformer pipelines in the same manner iSQRT-COV, with minimal overhead and no additional hyper--aramen parameters.
3.5 Properties: No Extra Hyperparameters, Gentle Spectrum Handling
One major attraction of Halley-SVD is the absence of new hyperparameters. We do not introduce polynomial degrees (as in Padé expansions), nor do we specify truncation thresholds avoid gradient spikes (as in SVD-Trunc or SVD-TopN). The iteration (7) naturally handles close eigenvalues by distributing updates across X via matrix inversions, rather than dividing one eigenvalue difference by another. Moreover, Halley's 's iteration balances the spectrum more cautiously and intelligently than Newton-Schullz By virtue of second-order (and partial third-order) corrections, Halley-SVD avoids iSQRT-COV's pitfall of flattening out the principal directions of the covariance, thus preserving crucial high-eigenvalue information especially in deeper models or large-batch scenarios.
3.6 Relation to Existing SVD Improvements
Although numerous approaches have been proposed to stabilize SVD-based GCP, Halley-SVD distinguishes itself multiple ways:
Padé or Taylor Expansions. Some variants of SVD rely on truncated Taylor or Padé expansions (e.g., SVD-Padé), which approximately approximate the 1/(1 - x) factor arising in the gradient. Such methods can indeed mitigate certain numerical instabilities but require carefully specifying an expansion order or precise region of validity. Halley-SVD, by contrast, elegantly encapsulates high-order corrections within unified iterative formula that does not rely polynomial degrees.

---

Hybrid or Switching Protocols. Others combine iSQRT-COV in early training (to avoid blow- ups) with a late-stage switch to SVD for more accurate covariance roots. This hybrid approach necessarily necessitates an additional complex schedule for switching at the "right time," or multiple exhaustive experiments to locate beneficial switch point. Halley-SVD completely eliminates this extra complexity, consistently maintaining single robust iteration formula throughout training.
Comparison with iSQRT-COV. Halley-SVI SVDley--VDDie inherits iSQRT- general advantage of avoiding explicit
- x,- in the backward pass. Crucially, it further carefully moderates the downward pull on large eigenvalues, that the over-flattening observed in iSQRT-COV at large scales is drastically reduced. Consequently, Halley-SVD is able to effectively capture the benefits of smooth gradient updates in early epochs while robustly preserving critical high-eigenvalue directions during the final epochs, preventing the saturation we see in iSQRT-COV on deeper networks or large-batch training.
Taken together, Halley-SVD stands as pure iterative scheme that neither requires threshold-base intervention nor relies on expansions of uncertain convergence radius. This comprehensive feature set is precisely what enables it to consistently outperform both iSQRT-COV and classical SVD methods in the deeper or more challenging larger-scale experiments we detail in the following section.
4 Theoretical Analysis of Halley-SVD
In this section, we provide concise theoretical account of why Halley-SVD consistently avoids over-compressing large eigenvalues more effectively than the classical Newton-Schulz scheme. We focus on carefully selected representative scenario in which Σ has widely separated eigenvalues, and then highlight the key statement (Theorem I that quantifies this effect. The complete technical details, including multi-step proofs with intermediate derivations, appear in Appendix B
Notation & Sctup. Let Σ E I be symmetric positive-definite matrix with eigenvalues
X1 .... ... 2 Aa > 0. Our goal is to approximate 1 by an iterative method X = F(x,, Σ). For Newton-Schull 24 we have:
X = 2 Xx 31 31-
-3I-
(9)
For Halley iteration, one form is:
X X
[I + 2 [I E xxxxx )
(10)
Theorem 1 (Less Over--ompr,ssion), Suppose E = diiag(An (angg(A,...... Aa) with A >> da > 0. Let (x()) and (x))) } be the sequences generated by and (TO), respectively, from a similar initial guess Xo. Then, for sufficiently large k, the top eigenvalue of x
k is consistently closer to v>a than that of (N), implying that Halley iteration preserves large eigenvalues significantly more effectively and robustly (while still always remaining fully convergent overall).
Key Idea. Halley iteration has a second-order correction that effectively tempers the "downward pull" coordinates exceeding VX,, whereas Newton- too large relative to / Vy, Consequently, in challenging large-scale tasks, Halley-SVD consistently retains the important top spectral directions much better, avoiding iSQRT-COV's over-flateeing.
aggressively shrinks any coordinate
complete formal proof with multi-step expansions, plus an extension to non-diagonal Σ, given in Appendix B We also provide novel perturbation analysis for the practically important scenario where Σ continuously evolves over successive mini-batches during actual deep learning training.
Experimental Results
5.1 Deep Diagnosis of Over-Flattening and Performance Bottleneck
In this section, we provide a detailed diagnosis of the over-flattening phenomenon that consistently emerges in large-scale settings when using iSQRT-COV. We comprehensively compare our Halley- SVD against iSQRT-COV and SVD-Padé on ResNet-100 10/ and Swin-7 by carefully examining both final recognition accuracy and the dynamic evolution of the critically important covariance spectrum.
7

---

Ovan-latiiing Diagassi: Raches101 and Smila Larg- Batch Traleing
Ca Busdhee IVI Aeneney
in BecheeW Sovrd Firsen
01 Aeeoee
Wi drie Spcer
8yesh
Pyeck
Figure 3: Over-Flattening Diagnosis on ResNet-101 and Swin-T (ImageNet, BS=2048/409), (a)
ResNet- 101 accuracy: iSQRT-COV (red dashed) saturates early, while Halley-SVD (blue solid) and
SVD-Padé (green dash-dotted) continue improving consistently. (b) ResNet-101 spectral flatness: iSQRT-COV aggressively compresses large eigenvalues, driving Kftat toward O. (c) Swin-T accuracy: similar saturation pattern as ResNet-101 101, but even more pronounced. iSQRT-COV shows even aggressive eigenvalue compression. In both architectures, Halley-SVD maintains a more balanced spectrum and achieves higher final accuracy.
(d) Swin-T spectral flatness:
Table ImageNet validation accuracy (%) under different models and batch sizes. We report both Top-1 and Top-5. "Vanilla GAP" denotes the standard Global Average Pooling baseline. All GCP. based methods are trained end-to-end without hybrid strategies. The parentheses in the rightmost column show the 1/p-1/Top-5 gains of Halley-SVD over iSQRT-COV under large-batch settings.
Model
Batch Size
Vanilla GAP
ISQRT-COV
SVD-Padé (E2E)
Halley-SVD (Ours)
(% Top-1 / Top-5 Accuracy)
ResNet-50
256
76.1 /92.9
77.2 79.5/93.5 76.9 / 99.3
77.3 / 9936 77.1 / 193,4
7 /33.6 77.2 93.5
ResNet-50
2048
/2.0
ResNet-101
256
77.4/93.6
78.3 / 94.1 77.7/93.7
78.4 / 94.2
78.4 / 94.2 78.5 ( (+0.8/+0.6)
ResNet-101
2048
77.2
78.1 /94.0
ViT-B/16
1024
81.8 /6.0
82.5 / 96.3
82.7 96.4
82.8 19.55 82.9 / 96.5 ( (+0.9/+0.5)
VIT-B/16
4096
81.5/95.8
82.0/ 96.0
82.4/
Swin-T
1024
/
82.1 /9.0 96.0
82.3 96.1
82.4/ 96.2
Swin-T
4096
81.0 95.5
/ 95.6
81.9 95.9
82.3 / 19.. ( (+0.9/ +0.5)
Setup. We train ResNet-101 [4] and Swin-T[29] on ImageNet using large batch sizes of either BS 2048 or 4096. All hyper-parameters follow the carefully selected defaults described in Section C.I Each covariance pooling variant (iSQRT-COV[24) SVD-Padé(37] Halley-SVD) is implemented completely end-to-end with absolutely no hybrid or switching protocols.
Results and Analysis. Figure Billustrates how iSQRT-COV and Halley-SVD evolve throughout training for both architectures. Panels (a) and (b) show that iSQRT-COV (red dashed) quickly ramps up in early epochs but saturates prematurely, leading to significantly lower final accuracy (c.g., 77.7% on ResNet-101 and 81.4% on Swin-T), By contrast. SVD-Padé (green dotted) and Halley-SVD (blue solid) continue to improve, with Halley-SVD eventually reaching the highest accuracies (~~ 78.5% on ResNet-101 and 82.3% on Swin-T), Panels (c) and (d) reveal that this saturation arises from rapid drop in the spectral flatness index (Kaat) under iSQRT-COV, indicating that large eigenvalues are aggressively compressed until Kéat < 0.1. Halley-SVD, however, preserves a more balanced eigenvalue distribution (Knat = 0.28), retaining crucial discriminative directions into later epochs. Although iSQRT-COY remains stable in mid-scale regimes, its over- wer-flattening severely hinders performance in deeper networks and large-batch contexts. By incorporating higher-order corrections, Halley-SVD consistently prevents the collapse of principal eigenvalues, achieving higher final accuracy without resorting to explicit thresholding or additional switching mechanisms.
5.2 ImageNet Main Results across Different Scales
Setup. We evaluate four representative Beckbonnss-ReNeede ResNet-101, VIT-B/16, and Swin- T across both standard and large batch sizes. We compare our proposed Halley-SVD to iSQRT-
COV 24), SVD-Padé (37], and a Vanilla GAP baseline. All architectures are trained on ImageNet with the same data augmentations and training hyper-parameters described in Section C.I
8

---

Results and Analysis. reports the final Top-1 I and Top-5 accuracies for various architectures and batch sizes. When training moderate batch sizes (c.g.. 256 or 1024). iSQRT-COV and Halley-SVD perform similarly and both surpass the Vanilla GAP baseline, while SVD-Padé (E2E) also offers comparable results. However, as the batch size scales up to 2048 or 4096, iSQRT-COV begins to lag behind, especially on deeper backbones like ResNet- 101 and Transformers, where over-flattening becomes more pronounced. In these large-batch regimes, Halley-SVDD consistently achieves 0.8-0.9% higher Top-1 accuracy than iSQRT-COV (see the parentheses in Table D. demonstrating how its more balanced handling of the covariance spectrum prevents the late-stage saturation observed in iSQRT-COV. Compared with SVD-Padé (E2E), Halley-SVD generally reaches accuracy on par with or marginally better, indicating that its higher-order iteration preserves critical eigen- direeciissssnss without resorting to rational approximations evidence underscores Halley-SVDD ability to outperform iSQRT-COV in exactly those deep or large-batch scenarios where second-order features must remain spectrally diverse, offering robust and purely iterative GCP solution that matches or surpasses SVD-Padé's s level of performance.
threshold heuristics. Overall, this
5.3 Fine-Grained Transfer Learning
Setup. To further validate the transferability of the representa- tions learned by Halley-SVD, we adopt challenging downstream fine-grained visual classification (FGVC) setting. We pre-train ResNet-50 model on ImageNet(6) with BS 2048 using each GCP SVD-
Table 2: Fine-grained classification (FGVC) accuracies (%). All models are ResNer-50 backbones pre-trained on ImageNet (BS=2048) and then finetuned for 50 epochs on Birds. Dogs, and Cars. Halley-SVD achieves the best perfor- mance across all three datasets, surpassing iSQRT-COV by ~ 1.2% on average.
Accuracy (%)
method (iSQRT-COV(24)
Method
Avg
Padé Halley-SVD). We then carefully finetune for 50 epochs on three FGVC datasets: Birds 52 45).
Birds
Dogs
Cars
iSQRT-COV
86.5
83.5
91.6
87.20
SVD-Padé (E2E)
87.3
84.3
93.0
88.20
(3, and [3rs following the standard protocol in Appendix C.I
Halley-SVD
87.5
84.6
93.2
88.43
Results and Analysis. Table T reports the fine-grained classification accuracy for each of the three GCP pre-training methods. Across all tested FGVC tasks, Halley-SVD achieves the highest performance. surpassing iSQRT-COV by approximately 1.2% on average. Although SVD-Padé likewise improves upon iSQRT-COV, Halley-SVD consistently attains slightly better results (e.g., gains of +0.2% on Birds, +0.3% on Dogs, and +0.2% on Cars). These outcomes suggest that Halley-SVD learns more discriminative second-order representation on ImageNet, thus conferring an advantage in fine-gre fine-grained tasks that hinge on subtle inter-class differences. In summary, these FGVC transfer experiments confirm that Halley-SVD not only excels large-scale ImageNet training but also provides a stronger backbone for downstream applications, as it effectively preserves prominent eigenvalue directions to yield richer semantic features and higher classification accuracy.
Conclusion
Halley-SVD addresses the critical trade-off in Global Covariance Pooling between the early stability but late-stage spectral oler-flattening of iSQRT-COV, and the potential instability but spectral fidelity of SVD methods. Leveraging higher-order matrix iterations, Halley-SVD inherently balances gradient smoothness and cigenvalue preservation without heuristic interventions like thresholds or switching. This yields superior performance, particularly in large-scale deep learning scenarios involving deep networks and large batches, offering a robust, unified approach to harnessing second-orde statistics.
While computationally more intensive than simpler methods due to its iterative nature (requiring careful selection of iteration count K), Halley-SVD consistently demonstrates robust performance across various practical settings. Future directions include exploring its application across diverse tasks and architectures, optimizing its computational efficiency, and further investigating its theoretical properties, multi-layer integration strategies, and interplay with deep learning optimization dynamics.

---

Acknowledgments
The work is partially supported by the National Natural Science Foundation of China (Grant No. 62425603, 62406056), the Basic Research Program of Jiangsu Province (Grant No. BK20243018), Guangdong Basic and Applied Basic Research Foundation (Grant No. 2001244111111 14), The computational resources are supported by SongShan Lake HPC Center (SSL-HPC) in Great Bay University.
References
[1] Aiman Al-Sabaawi, Hassan M Ibrahim, Zinah Mohsin Arkah, Muthana Al-Amidie, and Laith Alzubaidi. Amended convolutional neural network with global average pooling for image classification. In International conference on intelligent systems design and applications, pages 171-180. Springer, 2020.
[2] Ekin D Cubuk, Barret Zoph. Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702-703, 2020.
[3] Longquan Dai, He Wang, and Jinhui Tang. Accetr: Accelerating training- teining-frree free control for text- to-image diffusion models. In Proceedings of the International Joint Conference on Artificial Intelligence, 2025.
[4] Longquan Dai, He Wang, and Jinhui Tang. Noisectrl: A aaappinngalgoritn--gggnne condi- tional generation method for diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 18093-18102, 2025.
[5] Dataset. Novel datasets for fine tne-grained image categorization. In First workshop on fine grained visual categorization, CVPR. Citeseer. Citeseer. Citeseer, volume 5, page 2. Citeseer, 2011.
[6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-22 245-255. Ieee, 2009.
[7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
[8] Zilin Gao, Qilong Wang, Bingbing Zhang, Qinghua Hu, and Peihua Li. Temporal-attenie attentive covariance pooling networks for video recognition. Advances in Neural Information Processing Systems, 34:13587-13598 2021.
[9] Zilin Gao, Jiangtao Xie, Qilong Wang, and Peihua Li. Global second-order pooling convolu- tional networks. Proceedings of the IEEE/CVF Conference on computer vision and patlern recognition, pages 3024-3033, 2019.
[10] Dipam Goswami, Bartlomiej Twardowski, and Joost Van De Weijer. Calibrating higher-order statistics for few-shot class-incremental learning with pre-trained vision transformers. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4075-408- 2024.
[11] Jiawei Gu, Ziyue Qiao, and Zechao Li. Gradient short-cireuit: Efficient out-of-distributiin detection via feature intervention. arXiv preprint arXiv:2507.01477 2025.
[12] Jiawei Gu, Ziyue Qiao, and Zechao Li. Spectralgap: Graph-level out-of-distribution detection via laplacian eigenvalue gaps. arXiv preprint arXiv:2505.15177, 2025.
[13] Jiawei Gu, Ziyue Qiao, and Xiao Luo. Neubm: Mitigating model bias in graph neural networks through neutral input calibration. arXiv preprint arXiv:2505.15780, 2025.
10

---

[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[15] Nicholas Higham. Functions of matrices: theory and computation. SIAM, 2008.
[16] Gao Huang. Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. Computer Vision-ECCC 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 646-661. Springer, 2016.
[17] Lei Huang, Dawei Yang, Lang, and Jia Deng. Decorrelated batch normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 791-800, 2018.
[18] Lei Huang, Lei Zhao, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao. An investigation into the stochasticity of batch whitening. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 6439-64360 6439-6448, 2020.
[19] Catalin Ionescu, Orestis Vantzos, and Cristian Sminchisescu. Matrix backpropagation for deep networks with structured layers. In Proceedings of the IEEE international conference computer vision, pages 2965-2973, 2015.
[20] Catalin Ionescu, Orestis Vantzos, and Cristian Sminchisescu. Training deep networks with structured layers by matrix backpropagation. arXiv preprint 1 2015.
[21] Imran Khan, Shahab Saquib Sohail, Dag Øivind Madsen, and Brajesh Kumar Khare. Deep transfer learning for fine-grained maize leaf disease classification. Journal of Agriculture and Food Research, 16:1011 148, 2024.
[22] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference computer vision workshops, pages 554-561, 2013.
[23] Lokesh Kumar, Jagadeesh Kakarla, B Venkateswarlu Isunuri, and Munesh Singh. Multi-class brain tumor classification using residual network and global average pooling, Multimedia Tools and Applications, 80(9): 1429--13488 2021.
[24] Peihua Li, Jiangtao Xie, Qilong Wang, and Zilin Gao. Towards faster training of global covariance pooling networks by iterative matrix square root normalization. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 947-955, 2018.
[25] Peihua Li, Jiangtao Xie, Qilong Wang, and Wangmeng Zuo. Is second-order information helpful for large-scale visual recognition? In Proceedings of the IEEE international conference on computer vision, pages 2070-2078, 2017.
[26] Zechao Li, Yanpeng Sun, Liyan Zhang, and Jinhui Tang. Cmet: Context- tandem network for semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 44(12):9904-99) 9917,, 2021.
[27] Tsung- Yu Lin and Subhransu Maji. Improved bilinear pooling with enns. arXiv preprint DAXIv:1707.0672 2017.
[28] Tsung-Y Yu Lin, Aruni RoyChowdhury, and Subhransu Maji. Bilinear enn models for fine-grained visual recognition. In Proceedings of the IEEE international conference computer vision, pages 1449-1457, 2015.
[29] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin trasffrmmr: Hierarchical vision transformer using shifted windows. Proceedings of the JEEF/CVF international conference on computer vision, pages 10012-10022, 2021.
[30] Uya Loshchilov and Frank Hutter. Decoupled weight decay regularization, arXiv preprint axX1v1171..51001, 2017.
11

---

[31] Yao Ni, Piotr Koniusz, Richard Hartley, and Richard Nock. Manifold learning benefits gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11265-11274, 2022.
[32] A Paszke. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703, 2019.
[33] Xingchao Peng, Baochen Sun, Karim Ali, and Kate Saenko. Learning deep object detectors from 3d models. In Proceedings of the IEEE international conference on computer vision, pages 1286, 2015.
[34] Mohammadreza Salehi, Hossein Mirzaei, Dan Hendrycks, Yixuan Li, Mohammad Hossein Rohban, and Mohammad Sabokrou. A unified survey on anomaly, novelty, open-set, and out-of-distribution detection: Solutions and future challenges. arXiv preprint arXiv:21 2021.
[35] Fei Shen, Xin Jiang, Xin He, Hu Ye, Cong Wang, Xiaoyu Du, Zechao Li, and Jinhui Tang. Imagdressing-vi. Customizable virtual dressing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 6795-6804, 2025.
[36] Fei Shen and Jinhui Tang. Imagpose: A unified conditional framework for pose-guided person generation. Advances in neural information processing systems, 37:624666222 62666, 2024.
[37] Yue Song, Nicu Sebe, and Wei Wang. Why approximate matrix square root outperforms accurate svd in global covariance pooling? In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 11 1115-1123, 2021.
[38] Yue Song, Nicu Sebe, and Wei Wang. Fast differentiable matrix square root and inverse square root. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(6):7367-7380, 2022.
[39] Yue Song, Nicu Sebe, and Wei Wang. Improving covariance conditioning of the svd meta-layer by orthogonality. In European Conference Computer Vision, pages 356-372. Springer, 2022.
[40] Yue Song, Nicu Sehe, and Wei Wang. On the eigenvalues of global covariance pooling for fine- grained visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3554-3566, 2022.
[41] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Computer vision-ECCC 2016 workshops: Amsterdam, the Netherlands, October 8-10 and 15-16, 2016, proceedings, part 14, pages 443-450. Springer, 2016.
[42] Yanpeng Sun, Jiahui Chen, Shan Zhang, Xinyu Zhang, Qiang Chen, Gang Zhang, Errui Ding, Jingdong Wang, and Zechao Li. Vrp-sam: Sam with visual reference prompt. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition, pages 23565-23574, 2024.
[43] Yanpeng Sun, Qiang Chen, Xiangyu He, Jian Wang. Haocheng Feng, Junyu Han, Errui Ding, Jian Cheng, Zechao Li, and Jingdong Wang. Singular value fine-tuning: Few-shot segmentation requires cew-parameters fine-tuning, In Advances in neural information processing systems, pages 97484-37496 2022.
[44] Yanpeng Sun, Qiang Chen, Jian Wang, Jingdong Wang, and Zechao Li. Exploring effective factors for improving visual in-context learning. IEEE Transactions on Image Processing, 2025.
[45] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
[46] He Wang, Longquan Dai, and Jinhui Tang. Emcontrol: Adding conditional control to text-to- image diffusion models via expectation potectatiionmmaimizzttioo on Artificial Intelligence, volume 39, pages 7691-7699, 2025.
Proceedings of the AAAI Conference
[47] Qilong Wang. Jiangtao Xie, Wangmeng Zuo, Lei Zhang, and Peihua Li. Deep cnns meet global covariance pooling: Better representation and generalization. IEEE transactions on pattern analysis and machine intelligence, 43(8):2582-2597 2020.
12

---

[48] Qilong Wang, Zhaolin Zhang, Mingze Gao, Jiangtao Xie, Pengfei Zhu, Peihua Li, Wangmeng Zuo, and Qinghua Hu. Towards deeper understanding of global covariance pooling in deep learning: An optimization perspective. IEEE Transactions Pattern Analysis and Machine Intelligence, (5(12))15802-155
2023.
[49] Wei Wang, Zheng Dang, Yinlin Hu, Pascal Fua, and Mathieu Salzmann. Backpropagation- friendly eigendecomposition. Advances in Neural Information Processing Systems, 32, 2019.
[50] Wei Wang, Zheng Dang, Yinlin Hu, Pascal Fua, and Mathieu Salzmann. Robust differentiable svd. IEEE transactions on pattern analysis and machine intelligence, 44(9):5472-548)) 2021.
[51] Kangkang Wei, Weiqi Luo, and Jiwu Huang. Color image steganalysis based pixel difference convolution and enhanced transformer with selective pooling. IEEE Transactions on Information Forensics and Security, 2024.
[52] Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. 2010.
[53] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference computer vision, pages 6023-6032, 2019.
[54] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:
2017.
[55] Shan Zhang, Naila Murray, Lei Wang, and Piotr Koniusz. Time-reversed diffusion tensor transformer: A new tenet of few-shot object detection. In European Conference on Computer Vision, pages 310-328. Springer, 2022.
[56] Zhiwei Zhang. Covariance-correcd whitening alleviates network degeneration on imbalanced classification. arXiv preprint arXiv:2408. 17/97, 2024.
[57] Pengfei Zhu, Jialu Li, Zhe Dong, Qinghua Hu. Xiao Wang, and Qilong Wang. Cep-gnn: competitive covariance pooling for improving graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2024.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately reflect the paper's contributions, including: diagnosis of fler-flatg in iSQRT-COV, proposal of Halley-SVD method, and experimental validation across various architectures. Section I clearly outlines these contributions as bullet points and explains the paper's scope and organization.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
13

---

Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper discusses limitations the conclusion (Section 6), noting that Halley-SVD is computationally more intensive than simpler methods and requires careful selection of the iteration count K. Future research directions and potential challenges are also addressed in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well- vell-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
The paper provides theoretical analysis in Section 4 with explicit assumptions about the eigenvalue distribution. Theorem is clearly stated with its assumptions, and formal proof is provided. Complete detailed proofs with multi-step derivations are included in Appendix B.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
14

---

Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper provides comprehensive details on implementation in Section and Appendix D.I, including network architectures, training hyperparameters, initialization strategies, and dataset configurations. The Halley-SVD algorithm (Eq. 10) is explicitly described with its forward and backward passes.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All experiments use publicly available datasets (ImageNet, CUB-200, Stanford Dogs, Stanford Cars). The code implementation of our Halley-SVD method is provided in the supplementary materials, including the core algorithm, training seripts, and evaluation protocols. This enables full reproducibility of our experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ca cc/
for more details.
15

---

While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ cttpps.ccppulic///ee. guwdes/ Codeeubmis isi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental details are thoroughly described in Section 5.1 and Appendix D.1, including training protocols, optimizer settings, learning rate schedules, batch sizes, and data augmentation strategies for all tested architectures (ResNet--0/1001, ViT-B/16, Swin-T).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Appendix D.7 (Table 9) presents multiple runs with different random seeds, reporting mean accuracy and standard deviations to demonstrate statistical significance. The reported performance differences are shown to be statistically robust with small standard deviations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
16

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper specifies in Appendix that experiments were conducted using
ToTor NVIDIA A100 GPUs. Appendix provides detailed runtime analysis, including forward/backwarr pass times, total runtimes, peak memory usage, and training throughput for each method.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't make it into the paper).
paper (e.g., preliminary or failed experiments that httpc///bbblc....cc...eeeeeeleee
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conforms to the NeurIPS Code of Ethics. focuses on improving machine learning methodology using standard public datasets, with no apparent ethical concerns related to data collection, privacy, or potential for harmful applications.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper presents technical improvement to the matrix square root compu- tation used deep learning models. Given its mathematical focus numerical stability of matrix operations, there are no specific societal impacts (positive or negative) beyond general advancement of machine learning methodology.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system Iearns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not release any models datasets with high risk for misuse. It focuses on improving numerical methods for matrix computation with no potential for harmful applications that would require safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper cites the original sources for all datasets used (ImageNet, CUB-200, Stanford Dogs, Stanford Cars) and the base network architectures (ResNet, ViT, Swin Transformer). Standard publicly available datasets and models are used according their intended academic purposes.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwppppappphhaoocc thcode. com com/datasets
18

---

has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce new datasets or code repositories. It proposes new algorithm (Halley-SVD) but does not release as an asset that would require documen- tation beyond what's provided in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The research does not involve crowdsourcing or human subjects. It focuses on algorithmic improvements using existing computer vision datasets and does not collect new numan-annotater data.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The research does not involve human subjects any activities that would require IRB approval. focused entirely on algorithmic development and evaluation using standard computer vision benchmarks.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The paper does not use Large Language Models as a component of the research methodology. The proposed method (Halley-SVD) is purely mathematical approach to matrix computation that does not involve LLMs any way.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
hww.eeerrpss.ccoeeeeeeeeeeeeeeee /
20

---

A Related Work
A.1 From GAP to Second-Order Pooling
Deep convolutional neural networks (CNNs) often employ global average pooling (GAP) at the end of the network to aggregate spatial activations into a feature vector [14]. While GAP reduces parameters and mitigates overfitting, it only captures the first-order statistics (mean) of features, ignoring key critical channel- correlations [21]. To enrich the representation power, researchers have increasingly shifted toward second-order statistics, typically covariance matrices of deep features [33 28 40). These second-order approaches explicitly model channel interactions, which is beneficial for tasks like fine-grained recognition, where subtle inter-class differences matter most [40].
Early attempts, such as Bilinear CNN (B-CNN) (28] and DeepO2P [3], demonstrated the significant potential of integrating second-order pooling into CNNs, effectively improving overall performance on various complex vision tasks. However, these methods often suffered from substantial computational overhead as their feature dimension grows quite quadratically [47]. Subsequent works like compact bilinear pooling [ alleviated dimensional explosion, but the inherent numerical and computational challenges introduced by second-order computations remained important open issue [7 24
A.2 Large-Scale GCP: From MPN-COV to iSQRT-COV
Matrix Power Normalized Covariance Pooling (MPN-COV). 24 proposed MPN-COV using matrix power normalization (especially matrix square root when a = 0.5) to achieve robust covariance estimation under the nigh-dimension small- -ample (HDSS) condition 47 This approach effectively regularizes feature distributions and approximates a Power- wweer-Ecliddean metric in the space of positive definite matrices [24). boosting performance over GAP on ImageNet-scale data [47] Nevertheless, MPN-COV relies on either eigenvalue decomposition (EIG) or singular value decomposition (SVD) to compute the matrix square root, incurring high computational cost (24]. Such operations are often suboptimal for GPU acceleration in large-batch training, making them major bottleneck [47).
Iterative Matrix Square Root (iSQRT-COV). To overcome this bottleneck, 24 introduced iSQRT- COV, which replaces EIG/SVD with the Newton-Schin -Shoulz iterative scheme that only involves matrix multiplications (24). This drastically speeds up forward inference on GPUs, while retaining compara- ble accuracy to MPN-COV [47]. However, the backpropagation of iSQRT-COV still requires careful treatment of the iteration's s gradients 24 38 The success of iSQRT-COV consolidates "matrix square root normalization" as a de facto standard in global covariance pooling (GCP), making efficient and stable differentiable solutions for the square root operation a primary research focus [7 50
A.3
Differentiable Matrix Operations and Numerical Challenges
Gradient Backpropagation for Structured Layers. Ionescu et al. [9 20 established the funda-
mental theoretical foundation for matrix backpropagation, providing critical analytic gradients for
complex SVD/EIG-based layers. Despite this significant mathematical advancement, applying these
sophisticated gradients in practical large-scale deep networks can be inherently numerically unstable:
when eigenvalues or singular values are extremely close together, the problematic terms 1/(A; Az)
may approach infinity, potentially causing catastrophic gradient explosions 37 40 [50]
Il-Conditioned Covariance and Small Eigenvalues. Real-world complex deep features often lead to severely ill-conditioned covariance matrices, where the critical ratio between the largest and smallest cigenvalues (condition number) becomes extremely large [39 47]. This mathematical phenomenon exacerbates two significant practical issues: (1) numerical instability in forward decomposition (SVD/EIG may completely fail or dramatically lose precision for extremely tiny cigenvalues), and (2) dangerous exploding gradients in backpropagation 44). Interestingly, these problematic small eigenvalues can carry essential fine- -re-grai discriminative information 44), so naively truncating them might reduce computational instability but risk harming discriminative power 37 40].
Computational Overhead. Exact SVD or EIG is known to be expensive, especially larch-batch or GPU-based training. where batchwise decomposition often neutralizes parallel efficiency [47] Although iSQRT-COV accelerates forward passes via matrix multiplication, its backward pass can remain costly [4 38), prompting investigation into more advanced or approximate methods [50] 38)
21

---

A.4 Enhanced Second-Order Methods and Broader Applications
employed power iteration or Taylor expansions to approximate SVD gradients, thereby avoiding explicit 1/(A, ~2) terms. Song et al. (38] proposed matrix Taylor polynomials (MTP) and matrix Padé approximants (MPA). combined with Lyapunov equation solvers, to accelerate and stabilize both forward and backward computations of matrix square roots. MPA typically provides better precision than MTP at similar computational budget [38]
Recent Solutions for Stability and Efficiency. 50
Condition Number Improvement. Efforts like NOG/OLR by [39 directly enhance feature orthogo- nality the pre-SVD layers, significantly reducing the condition number of covariance matrices while carefully retaining or even improving accuracy. This differs from naive weight orthogonalization approaches, balancing numerical stability and discriminative power more effectively [39].
Approximate vs. Exact Decompositions. A fundamental key question why approximate methods, such as iSQRT-COV, frequently outperform traditional exact SVD in practice [37]. It appears that high- precision SVD can significantly aggravate gradient instability by precisely capturing problematic tiny eigenvalues, whereas iterative approximations implicitly smooth such numerical fluctuations [37]. To effectively reconcile both competing sides, (37 introduced SVD-Padé and a novel hybrid training protocol (switching from iSQRT to SVD in later epochs) optimally harness the benefits of both.
Extensions Beyond CNNs. These second-order methods have proven effective not only in CNN architectures but also in Transformers. Song al. (38] proposed So-ViT, incorporating differentiable matrix square roots for vision tokens, and achieved impressive gains. Moreover, second-order pooling and related matrix operations are being explored in few-shot incremental learning (IO), domain adaptation [41], high-order pooling [55]. generative models [31], and distrf--dittiuution detection [34]. Although these new applications expand the scope of GCP, they also bring fresh challenges, such as extreme low-sample covariance estimation or domain distribution mismatch.
Second-order pooling has evolved from an innovative add-on to more general paradigm, with a wide range of improvements focusing on stability, computation speed, and numerical conditioning. Yet, as models become larger and data more complex, balancing these factors- high representation power, numerical robustness, and efficiency-
a central problem in second-order deep learning.
B Extended Theoretical Analysis and Proofs
including step-by-ste derivations with multiple formulas. We also add new theoretical insights on perturbation stability.
In this appendix, we give full proofs of the results stated in Section 4
B.1 Proof of Theorem
1
Theorem (Halley Avoids Over-Compression - Detailed Statement). Let Σ = ..)) with A 2 d 2 2 Xa O. Consider the Newton-Schulz iteration
x
(N)
1/2 x (N)
3 (N)2
(11)
k+1
and the Halley iteration
x I1]
-x
(12)
Suppose both start from the same (or comparable) Xo > O. Then for large k. the largest coordinate
of x (H) remains significantly closer to V\i than that of x() tthreby preserving more variance in the top eigenvalue. Concretely, if we denote
(N),
I i = X
(N)
(H)
[ i
FiKk
then there exist constants C, 5 0 such that for sufficiently large k.
11,
(H)
A
5 C
(H) -1
v A
+ 5
(13)
11,
(N)
A
2 c-
c-
V>i 55
(14)
(H) which implies 1i,k remains systematically less compressed (closer v\v than (N) 1
22

---

Proof (with Multi-Step Expansions). We focus on the diagonal scenario, i.c. Σ diag(A1, (11,.............. ................................ Ad).
so X (N)
and X (H)
remain diagonal. Denote:
X (N) is
(H)
[ !1
Without loss of generality, analyze the = 1 case (the largest eigenvalue A1). Newton-Schulz on the 1st coordinate is
(N)
(N) (3 1i,
#1,k
[N12
(15)
1,+1
A
2/2 (3 i
(N)
11,
A
where we used I and x (N)2
diag(2 20
(N)2
... Similarly, Halley iteration yields
1,
1,k+1
(H)
iik
(H)
A
(16)
Step 1: Expand near Vdi. Define A = I =
Vx i.e. the deviation from the true root v
Suppose | [N) is small (the iteration is near convergence). We rewrite (
as
11,+1 A1 = (3((
- (vAi-A (0)
A1.
(17)
Now expand (vλi + A )3
(V A1 A (N) = X)
3/2
+ 3 A
(N) + 3
NN)
+ A (N)3
(18)
Plugging 18
back into 17 and simplifying.
(N)
VA 3 (/)
3/2
VA A1
(lower order in (1),
(19)
1,++1
this muin part vanishes to 0
where the leading terms in vd exactly cancel out. The remaining expansions yield
VA V (N)
3 v
114+1
(N)
X A
2
A [N)2
k
2
A
(N)3
+
(20)
(Here we have omitted a few intermediate factor groupings; see (4 for the full details.)
Step Halley expansion. Similarly, for Halley iteration (16). define ( = Fi,k
(H)
Vi. We have
(H) 116+1 = = [1
[ (VA++A A
}]
VA.
(21)
Expand (vλz A (1) )2 similarly, factor out '₁. etc. After longer chain of simplifications, get an expression of the form:
VA
HH) 1,81
V
(H) A
+ (higher-order terms in A (H) +
(22)
A
Crucially, the coefficient in front (T is smaller in magnitude than that in 20 and the subsequent cubic corrections also differ in sign, resulting in "less negative pull."
(T)
Step 3: Compare the magnitudes. Subtract 22 from
20
3 V (N)
V (-
(-V A [m]]
(11, V\1) - (H
An) = [
2 X
A1
+ (cubic and cross terms).
(23)
If (N) and A (H)
are in similar small numerical range, the factor - 3 yAi is substantially more Ai
negative than VAi Hence, the net difference 23 is typically positive, strongly implying #1,4+1 is smaller than 11,++1 by significant nontrivial margin. Formalizing this mathematical argument requires bounding the higher-order and cross terms using standard Lipschitz arguments. Thus, for
(H)
large k. Halley's s iteration consistently yields a bigger I1, near VAi than Newton-Schie does.
Repeating the same analysis for each diagonal entry completes the proof of [3)
D
23

---

B.1.1 Full Multi-Line Expansions
We now demonstrate more explicit expansions (potentially excessive for the main text) to show how each term arises. Let re-derive (20) with full factorization:
(N) 1,,+1 Vd1
2 (vAi + ( (
VAI
A1
=3(3 +34
(N)
VAi +2 +A
(N
(V + VV
2 (3VA)) A A
2 (VAI
+2V + (N +4 (V (24)
B
Group and simplify term A:
V - V (3
But observe that the next step, can further combine with part of C, eventually giving 0. We proceed:
C 1 A1 +
2 + + (N)2 (VAI+
2
A1 A IN) + (N)
V + (N)2 VA 4
2
(25)
After reorganizing and combining with the 2 Vi leftover in part A, one obtains the final form:
A A
(N) 11,+1
(N) + (quadratic/cubic terms in A
Hence 20
One from 25 that the coefficient in front of (N) is indeed significantly larger (in negative sense) than in the Halley counterpart. The latter's 's expansions have an extra important factor in the denominator from the (1+ 2 ....) structure, leading to noticeably gentler push. The mathematical details for Halley are similarly multi- multi-line and are omitted here for brevity; see (22 in the main text proof. This thoroughly justifies the critical statement that Halley iteration consistently exerts less compression on large coordinates near vx, ultimately preserving more valuable spectral spread.
B.2 General Positive-Definite Matrices (Non-Diagonal Case)
When Σ is not diagonal, one can still diagonalize it as U diag(A)..... Aa) UT and rigorously rewrite each iterative update in that basis. Sewton-Schulz and Halley remain free of explicit denominators (A, -,)) the same mathematical expansions and local- wipps-iipschitz bounding apply, albeit with extra rotation terms. A careful bounding argument (cf. [15] Chap. 6]) shows that each diagonal coordinate in the eigenbasis experiences Halley
Newton-Schulz update akin to 24 (5) Hence the
fundamental core difference between the two methods persists for arbitrary PSD Σ.
B.3 Additional Perturbation Stability: Halley vs. Newton-Schulz
Finally, we mention new "dynamic" viewpoint: In typical deep-learning pipelines, the covariance is computed per mini-batch, so it might vary slightly from iteration to iteration. If changes in small increments (say, ||E+1 Σ. < e), we want the iterative method to remain stable across these continuous changes. Newton-Schulz can accumulate flattening across many mini-hatches, eventually saturating the spectral distribution. Halley's 's gentler correction significantly mitigates that effect.

---

Proposition (Stability Under Small Shifts), Let Σg be a sequence of PSD matrices where [141 Eell I cg. Consider X { approximating _ by k steps of Halley iteration from some Xo,t- Then, under mild conditions on Xo,, there exists co > such that if eg S cg. we have
X
1 S C X H
-E + DES.
(26)
for some constants C, > O. lteratively applying (26 shows that X X stays close to the "true" root E throughout training. avoiding thresholding truncation.
(H)
Sketch. See Section B.3.1 for an expanded multi-line derivation. The gist is to treat X like (22) plus an extra "perturbation" term from E, to Σ.-1. We then employ standard matrix-norm bounding to show the difference remains (ex)), ensuring stability.
(H)
1/2
B.3.1 Detailed Proof of Proposition 3
We give a more explicit big-formula expansion to illustrate how the perturbation enters:
X (I)
D
1/2
k
X (H) kit
(X (H) k,t E,
(H) ket
+[....
(27)
difference in E. vs. Er- inside the iteration
We can factor out (X
from the bracket term in 277. leading to an extra piece bounded by Σ, Σ, S eg. Next, combine the expansions with the local Lipschitz continuity of Halley iteration (analogous to Theorem . yielding: yicldinn:
E E,-1 (x
k.t
1 S C
2I + C2 ext + (terms that vanish when ls is large), (28)
for some constants C1. C2 depending on min min and max(A;): {x Using (28) over leads (26).
C Additional Experiments
C.1 Implementation Details
All experiments conducted using Py (32 (version 1.12 or later) and executed primarily NVIDIA A100 GPUs. For large-scale classification, we utilize the ImageNet- Ik dataset [6], employ- ing the standard training/validation split. Fine-grained visual classification (FGVC) experiments performed on Caltech- UCCD Birds 200 (Birds) [45), Stanford Dogs (Dogs) [5], and Stanford Cars (Cars) 22 Specific details for FGVC finetuning deferred to Appendix D.5.
We evaluate our proposed method and baselines across diverse set of modern backbone architectures. including Convolutional Neural Networks (eANs))ResNet-555 and ResNet-101 [14] and Vision
Transformers ViT-Base/16 (ViT-B/16) 7 and Swin Transformer Tiny (Swin-T) 22) We use standard pre-trained weights where applicable or train from scratch following common practices for each architecture.
Unless otherwise specified, the Global Covariance Pooling (GCP) layer replaces the final Global Av- erage Pooling (GAP) layer preceding the classification head. To maintain a consistent dimensionality for the covariance matrix across different backbones, we typically insert x 1 convolutional layer before the GCP layer to project the feature channels to d = 256. All computations within the GCP layer, including covariance matrix calculation, iterative updates (for iSQRT-COV and Halley-SVD), SVD decomposition (for MPN-COV and SVD-Padé), and gradient computations, are performed using double precision (float64) to ensure numerical accuracy and stability, following best practices established in prior GCP works 33 24). The rest of the network utilizes standard single precision (float32) automatic mixed precision (AMP) for efficiency.
25

---

For our proposed Halley-SVD, we employ K = 8 iterations for the matrix square root approximation, as determined by ablation studies (see Appendix G.7). The iteration is initialized with X, 10-$I For baseline methods: iSQRT-COV follows the implementation in (24.. typically converging within K R 5 iterations implicitly; SVD-Padé follows (37 using K 100 degree diagonal Padé approximants for the backward pass, but crucially. it is trained end-to-end (E2E) in our main comparisons without any hybrid strategy; MPN-COV [25) uses standard SVD for both forward and backward passes. For methods involving explicit eigenvalues or matrix inversions (SVD- Padé, MPN-COV, and potentially Halley-SVD under extreme conditions), eigenvalues smaller than machine epsilon for float64 (or threshold like 10- 10-3) are clamped to this value for numerical robustness, although Halley-SVD's iterative structure is inherently designed to handle near-singular cases smoothly.
For ImageNet- Ik training:
CNNs (ResNet-50/101): We train for 100 epochs using SGD with momentum 0.9 and weight decay 1 10-1 We employ a Cosine Annealing learning rate schedule with an initial learning rate of 0.1 for a standard batch size (BS) of 256, linearly scaled (LR = 0.1 x BS/256) for larger batch sizes (e.g., 0.8 for BS=2048). A 5-epoch linear warmup is used. Transformers (ViT-B/16, Swin-T): We train for 300 epochs using the Adam W optimizer 30 with weight decay 0.05. Cosine Annealing schedule used with an initial learning rate of 1 x 10 for standard batch size of 1024, linearly scaled for larger batches (c.g., 4 x 10-3 for BS=4096).
A 20-epoch linear warmup applied.
Batch Sizes: We report results for both standard batch sizes (ResNet: 256, Transformers: 1024) and large batch sizes (ResNet: 2048, Transformers: 4096) to investigate performance under different scales.
Standard data augmentation techniques applied during training, including RandAugment 22 Mixup (54). and CutMix (53). For Transformers, DropPath [16) is also used with linearly increasing rates. We report 1 and Top-5 accuracy on the ImageNet validation set, computed using a single center crop from images resized to 256x256 (then cropped to 224x224). Evaluation is performed using the final model weights unless otherwise noted.
C.2 Performance with Multi-Layer Covariance Pooling
While GCP is typically applied before the final classifier, exploring its use at intermediate network stages offer insights into how second-order statistics evolve and whether potential issues like over-flattening accumulate. To briefly investigate this, we conducted an auxiliary experiment on CIFAR-100 using a ResNet-50 backbone. We compared the standard setup (GCP only after layer4, replacing GAP) against a configuration where GCP layers were employed after both layer3 and layer4. We adapted the subsequent network layers to handle the output dimensions accordingly.
Setup. We trained ResNet-50 on CIFAR-100 for 100 epochs using standard hyperparameters (BS=128, SGD optimizer, Cosine LR schedule). We compared networks using either iSQRT-COV or our Halley-SVD for all employed GCP layers.
Results and Analysis. Table 3 presents the final test accuracies. Both methods benefit from using GCP compared to the GAP baseline. When employing GCP at multiple stages (Layer3 & Layer4), both methods show slight further improvement over using GCP only at the final stage. Notably, the performance gap between Halley-SVD and iSQRT-COV widened slightly in the multi-layer setting (+0033) compared to the single-layer setting (+0.1%). While preliminary, this observation suggests that the spectral compression effect of iSQRT-COV might indeed accumulate when used multiple times within a network, potentially limiting gains. Halley-SVD, by better preserving spectral information, appears more amenable to deployment at multiple network depths, hinting at broader applicability, although further investigation is warranted.
C.3 Numerical Stability in Near Rank- -eek-Deficeent Scenarios
A potential concern regarding iterative matrix methods, including Halley's 's iteration which involves terms like x-1 (implicitly or explicitly derived from Eq. 77), relates to numerical stability when the covariance matrix Σ becomes near singular rank-deficient. This could theoretically occur if
26

---

Table 3: Multi Layti GCP Accuracy (%) on CIFAR-100 with ResNet-50.
GCP Configuration
iSQRT-COV Ace (%)
Halley-SVD Acc (%)
ResNet-50 Baseline (GAP)
78.5
78.5
Layer4 Only GCP
80.0
80.1
Layer3 & Layer4 GCP
80.2
80.5 (+0.3)
the input features X within mini-batch lack sufficient variability, leading to very small zero eigenvalues Σ.
Mitigation via Regularization. Standard practice in numerical linear algebra and machine learning provides a straightforward safeguard against such issues. If necessary, one can ensure the positive definiteness and invertibility of the covariance matrix (and consequently the iterates X by adding small diagonal loading or Tikhonov regularization before applying the GCP method:
-
where is small positive constant (e.g., 10-6 eigenvalues are strictly positive, preventing divisions by zero or ill-conditioned inversions.
6 or 10 This minor adjustment guarantees that all
Practical Considerations. In the context of large-scale deep learning with diverse datasets like Ima- geNet and the common use Batch Normalization layers, empirical covariance matrices Σ computed from mini-batches are typically well-conditioned and numerically full-rank. The richness of features and the normalizing effects of BN layers significantly mitigate the risk of encountering pathologically rank-deficient covariance matrices during training. Therefore, while the e-regularization serves robust theoretical backup, it is rarely invoked or required in practice for methods like Halley-SVD to function stably. The primary stability advantage of Halley-SVD stems from its iterative formulation inherently avoiding the problematic 1/(A; - -yy) terms found in direct SVD gradient calculations, rather than relying heavily on additive regularization. Our extensive experiments on large-scale benchmarks proceeded without encountering stability issues related to near-singularity.
C.4 Detailed Speed and Resource Usage
Table provides more comprehensive breakdown of forward/backward time, total runtime, peak memory, and training throughput for various global covariance pooling (GCP) methods. We bench- mark these on ResNet-101 with batch size of 256, using NVIDIA A100 GPUs.
Table 4: Detailed Computation Cost (ResNet-101,, BS=256, single A100). "FP"/"BP" stand for forward/backward pass time (ms), "Total" is the sum, "Peak Mem" is the maximum GPU memory usage, and "Throughput" is measured in images per second. Halley-SVD's overall runtime is comparable SVD-Padé, reflecting the cost of multiple iterative steps rather than explicit SVD. Its memory consumption remains close to iSQRT-COV, indicating no significant extra overhead.
Method
FP (ms)
BP (ms)
Total (ms)
Peak Mem (GB)
Throughput (img/s)
Vanilla GAP
80
100
180
18.0
1420
iSQRT-COV (K=5)
110
180
~ 290
20.5
880
MPN-COV
280
70
350
22.0
730
SVD-Padé (K=100)
280
~ 900
~ 370
~ 22.0
690
Halley-SVD (K=8)
170
220
390
21.0
650
As observed, Vanilla GAP remains the fastest option but lags in final accuracy (see main paper). iSQRT-COV runs faster than Halley-SVD or SVD-Padé, yet over-flattening limits accuracy in surge-scale regimes. MPN-COV and SVD-Padé exhibit similar overall times, driven by SVD computations or rational expansions. Halley-SVD occupies a middle ground, incurring an iterative overhead but retaining a stable memory footprint and surpassing iSQRT-COV in final performance. Hence, the additional cost of Halley-SVD's higher-order iteration often justified by its superior accuracy in deeper large-batch scenarios.
27

---

C,5 Detailed Gradient Stability Analysis Across Architectures and Conditions
To visually confirm the gradient stability of Halley- SVD across various settings, we tracked the L2 norm of the gradient || 3p ll2 during the first 10 epochs of training on ImageNet. Figure presents a comprehensive 4x4 comparison grid.
Setup. The grid covers four architectures (Rows: ResNet-50. ResNet-101, VIT-B/16, Swin-T) and four conditions (Columns). Columns O and compare iSQRT-COV (red dashed) against Halley-SVD (blue solid) under standard and large batch sizes, respectively. Columns and 3 show the behavior of MPN-COV (grey) under standard and large batch sizes, serving as a reference for potential instability from direct SVD gradients. The specific batch sizes used for standard/large settings correspond to those defined in Section for each architecture.
Results and Analysis. The composite figure clearly demonstrates the stability patterns. Across all architectures and batch sizes (Columns and 3). MPN-COV consistently exhibits numerous large gradient spikes, often exceeding magnitudes of 10 to 10S, confirming its inherent numerical instability early training phases. In contrast, comparing Columns 0 and 1, both iSQRT-COV and Halley-SVD maintain remarkably stable and smooth gradient norms (typically 30) across all tested architectures and batch sizes. There are no large spikes observed for either method. This extensive visualization corroborates that Halley-SVD achieves gradient stability comparable to iSQRT-COV, effectively avoiding the severe instability issues of MPN-COV, which crucial for reliable training, particularly in challenging large-scale scenarios.
C.6 Convergence Speed Comparison
Beyond final accuracy, the speed at which model converges during training is also a crucial practical consideration. We compared the convergence behavior of Halley-SVD, iSQRT-COV, and SVD- (E2E) by examining both validation accuracy progression per epoch and against wall-clock training time.
Setup. We tracked the Top-I validation accuracy throughout training for the large batch size settings on ImageNet across our four main architectures: ResNet-50 (BS=2048), ResNet-101 (BS=2048), ViT-B/16 (BS=4096), and Swin-T (BS=4096).
Results and Analysis. F the convergence curves. The top row displays accuracy versus training epochs, while the bottom row shows accuracy versus estimated wall-clock time. Observing the accuracy vs. epochs plots (top row), iSQRT-COV (red dashed) typically shows the fastest initial improvement but, as established previously, tends saturate earlier at a lower accuracy level compared to the other methods, particularly for deeper models (R101, VIT, Swin-T). Both SVD-Padé (E2E) (green dotted) and Halley-SVD (blue solid) demonstrate the ability to continue learning for more epochs and reach higher final accuracies. Halley-SVD often matches or slightly surpasses SVD-Pade Padé in terms of the final accuracy achieved within the given epochs.
Examining accuracy vs. wall-clock time (bottom row) reveals the trade-offs. While iSQRT-COV reaches its plateau fastest due to its lower per-step cost, it fails to achieve the peak performance. Halley-SVD and SVD-Padé require more total training time to reach their higher final accuracies. Comparing Halley-SVD and SVD-Padé, Halley-SVDD potentially faster forward pass (no explicit SVD) but slower backward pass (more complex iteration) results in overall convergence times that are broadly comparable to SVD-Padé, or potentially slightly faster in some scenarios to reach specific high accuracy target, despite potentially taking similar total wall-clock time to complete all epochs. Halley-SVD thus offers compelling balance, achieving state-of-ther the extreme saturation iSQRT-COV or necessarily extending the total training time significantly compared to other high-performing SVD variants like SVD-Padé.
accuracy without
C.7 Ablation Studies and Robustness Analysis
To further understand the properties of our proposed Halley-SVD method and validate the robustness of our main findings, we conducted several ablation studies.
28

---

Deratled Gradient Nerm Stabliiit Across Architectures and Conditions (Barly ning)
1000000
1900000
50%%
120.0
1200
13:30
(IGHT to Hilles
LOgHTT in Mickey
SIPSCOO Bet
SIPN.COO
10
ICA
Page
(RN FOCE (IGHT
dRe LIGHT Hacky
STEBH (ES-TO
VIP' COT
1300.00
115
150.00
Codi-T 180-1890
(88-479)
IC-
Figure 4: Comprehensive comparison of GCP gradient stability during early training (first 10 epochs) on ImageNet. Each row represents an architecture (R50, R101, VIT-B/16, Swin-T). Columns represent conditions: (0) Standard BS, iSQRT Halley: (1) Large BS, iSQRT vs Halley; (2) Standard BS, MPN-COV reference; (3) Large BS, MPN-COV reference. Plotted is the I.2 norm of the gradient w.r.t. P (logarithmic y-axis). Both iSQRT-COV and Halley-SVD show consistently stable gradients across all settings, unlike the unstable MPN-COV.
Impact of Halley Iteration Count (K). The number of iterations K in the Halley-SVD update (Eq. 77) affects both the accuracy of the matrix square root approximation and the computational cost. We evaluated the performance of Halley-SVD on ResNet-101 (BS=256) on ImageNet while varying K from 3 to 15. Table 5 shows the final Top-1 accuracy and the measured total time per batch.
Table 5: Impact of Halley iteration count (K) on ResNet-101 (BS=256) performance and speed.
Halley Iter (K)
Top-1 Ace (%)
Total Time (ms/batch)
3
78.0
120
78.3
150
8
78.4
~90
10
78.4
~200
15
78.4
280
As observed, the accuracy essentially saturates at = 8 iterations, with further iterations yielding no significant performance gain while steadily increasing the computational time. Therefore, chose
29

---

Convergrnne Speed Compariona (Large Batrh Settings)
JAPPAA
Su Tixi
Si Tha
Figure 5: Convergence speed comparison on ImageNet under large batch size settings. Top Row (Epochs): Validation Top-1 Accuracy Training Epochs. Bottom Row (Time): Validation Top-1 Accuracy vs. Estimated Wall-Clock Time (hours). Columns correspond to different architectures: (0) ResNet-50 (BS=2048), (1) ResNet-101 (BS=2048), (2) ViT-B/16 (BS=4096). (3) Swin-T (BS=4096). iSQRT-COV (red dashed) saturates early. Halley-SVD (blue solid) and SVD-Padé (E2E) (green dotted) reach higher accuracy, with Halley-SVD often achieving the best final performance within comparable or slightly better convergence time relative to SVD-Padé. Panels in rows and are intentionally left blank.
= as the default setting for all other experiments, representing a favorable balance between accuracy and efficiency.
Impact Initialization Method. We investigated the sensitivity of Halley-SVD SVD to the choice of
the initial iterate Xo. We compared our default initialization (X₂ 10-$1) against an alternative scaled identity initialization based on the trace the input covariance matrix Σ, specifically Xo- = II, where d is the dimension and is a small constant. The results on ResNet-101
/TrEE)/++ (BS=256) are shown in Table 6
Table 6: Impact of Halley-SVD initialization method ResNet-101 (BS=256) accuracy.
Initialization Method
Top-1 Ace (%)
Xo 10 10-31 'I (Default)
78.4
X (... ) (Trace-Scaled)
78.3
The final performance is nearly identical between the two initialization strategies, indicating that Halley-SVD is robust to reasonable choices for the starting iterate Xo.
Sensitivity to Optimizer. To assess whether the observed performance advantages depend specific optimizer, we compared Halley-SVD and iSQRT-COV when training ResNet-50 (BS=256) using either SGD (our default) the AdamW optimizer. Standard hyperparameters were used for each optimizer (see Section for SGD; AdamW used Ir=0.001, wd=0.01).
Table 7: Optimizer sensitivity comparison on ResNet-50 (BS=256).
Optimizer
Method
Top-1 Acc (%)
iSQRT-COV
77.2
SGD
Halley-SVD
77.3
iSQRT-COV 76.8
AdamW
Halley-SVD
77.0
Table shows that both methods achieve reasonable performance with both optimizers, although SGD yielded slightly better results overall in this setting. Importantly, Halley-SVD maintained
30

---

its performance edge over iSQRT-COV regardless of the optimizer employed, suggesting broad compatibility.
Sensitivity to Feature Dimension (d), Our main experiments used a covariance matrix dimension of d 256, achieved via a convolution. To verify that our conclusions are not specific this dimension, we compared performance using d = 256 versus d = 512 for the challenging ResNet-101 (BS=2048) setting.
Table 8: Sensitivity to feature dimension d on ResNet-101 (BS=2048).
Feature Dim (d)
Method
Top-1 Acc (%)
Halley vs iSQRT Diff
iSQRT-COV
77.7
256
Halley-SVD
78.5
+0.8
iSQRT-COV
77.9
512
Halley-SVD 78.8
+0.9
As shown in Table 8 increasing the feature dimension to = 512 slightly improved absolute accuracies for both methods. Crucially, the performance advantage of Halley-SVD over iSQRT-COV remained consistent (+0.8% for d = 256, +0.9% for d = 512), indicating that Halley-SVDD s ability to better handle large-scale training is robust across different reasonable feature dimensionalities.
Robustness to Random Seeds. To ensure the statistical significance of our key results, we repeated the main large-scale comparison experiment (ResNet-101, BS=2048) five times using different random seeds for network initialization and data shuffling. We report the mean and standard deviation of the final Top-1 I accuracy.
Table Stability across random seeds for ResNet- 101 (BS=2048).
Method
Top-1 Ace (%) Mean + StdDev (N=5 runs)
iSQRT-COV Halley-SVD
77.7 7 0.08
78.5 + 0.06
Table[Jconfirms that the results are highly stable across different runs. The standard deviations are very small (< 0.08%), and the substantial mean performance gap (0.8%) between Halley-SVD and iSQRT-COV statistically robust, reinforcing the reliability of our main conclusions.
31


Reviews:
[{"review_id": "U1Q1x3mclg", "reviewer": "Reviewer_6Naf", "summary": "This paper introduces Halley-SVD, a high-order iterative method that combines the smooth gradient characteristics of iSQRT-COV with the high-precision convergence of SVD. The authors first observe that iSQRT-COV suffers from severe \"over-flattening\" when applied to deeper networks or with larger batch sizes. To address this issue, the paper proposes a refined SVD-based iterative approach that mitigates early gradient explosion and prevents the over-flattening observed in iSQRT-COV. Extensive experiments across various network architectures have been conducted to validate the effectiveness of the proposed method.", "questions": "See weaknesses", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n\n1. The analysis and proposed Halley-SVD are well-motivated and supported by theoretical proof.\n2. The proposed Halley-SVD demonstrates better performance in large-batch training regimes.\n3. The paper is well-written and easy to follow.\n\nWeakness\n\n1. [1] should be included for comparison in Table 1.\n2. Can the presented method be utilized in another way, for example, channel attention or AdaIN? How about a local version of Halley-SVD that achieves more representative features in other vision tasks?\n3. How about the latency, peak mem, and throughput during inference?\n\n[1] Wang, Qilong, et al. \"Towards a deeper understanding of global covariance pooling in deep learning: An optimization perspective.\"", "was_revised": true, "final_justification": "The rebuttal has addressed my concerns. I will retain my original rating and vote for accept.", "rebuttal": "## Regarding Question 1: [1] should be included for comparison in Table 1.\n\n> [1] Wang, Qilong, et al. \"Towards a deeper understanding of global covariance pooling in deep learning: An optimization perspective.\" (i.e., the DropCov paper)\n\nWe thank the reviewer for this constructive suggestion. Comparing our `Halley-SVD` with `DropCov` [1] indeed helps provide a more comprehensive evaluation of techniques in the Global Covariance Pooling (GCP) domain. First, we would like to clarify the fundamental difference between these two methods in their **research motivation and technical approach**:\n\n### Fundamental Differences Between Methods\n\n* **`Halley-SVD` (Our Method)** is designed to solve a core challenge within **structure-wise** post-processing methods. The goal of this family of methods (e.g., MPN-COV, iSQRT-COV) is to **achieve a high-fidelity approximation of the matrix square root** to maximize the representation power of second-order statistics. Our work focuses on this path. By introducing the Halley iteration, we preserve the spectral structure of the covariance matrix more accurately than `iSQRT-COV` while ensuring numerical stability, thus addressing its \"over-flattening\" issue. Therefore, in Table 1 of our paper, we compared `Halley-SVD` with its most direct competitors, `iSQRT-COV` and `SVD-Padé`, which all belong to the same technical category.\n\n* **`DropCov` [1]** takes a different path as a highly **efficient, regularization-based** method. It does not aim to approximate the exact matrix square root. Instead, it ingeniously strikes a balance between \"representation decoupling\" and \"information preservation\" through adaptive channel dropping at the feature level. Its primary advantages are its **exceptional computational efficiency** ($O(d)$) and its **zero-overhead nature during inference**.\n\nIn short, the goal of `Halley-SVD` is to **\"be more accurate,\"** while the goal of `DropCov` is to **\"be faster and smarter.\"** Placing them in the same table for a direct comparison is akin to comparing products with entirely different design philosophies.\n\n### Supplementary Experimental Results\n\nNevertheless, to fully address your concern and provide a more holistic perspective, we have conducted this supplementary experiment. We evaluated `DropCov` under the same experimental settings as in Table 1. As anticipated from both the literature [1] and our analysis, `DropCov`, as an efficient regularization method, outperforms `iSQRT-COV`. However, in large-batch training scenarios that demand extreme precision, its performance is slightly lower than `SVD-Padé` and our `Halley-SVD`, which are designed for high-fidelity approximation.\n\n**Table A: Supplementary Comparison of Halley-SVD and DropCov on ImageNet**\n\n| Model      | Batch Size | iSQRT-COV | DropCov [1] | SVD-Padé | **Halley-SVD (Ours)** |\n| :--------- | :--------: | :-------: | :---------: | :------: | :-------------------: |\n| ResNet-101 | 256        | 78.3      | 78.9        | 78.4     | **78.4** |\n| ResNet-101 | 2048       | 77.7      | 77.9       | 78.1     | **78.5 (+0.5)** |\n| Swin-T     | 4096       | 81.4      | 81.8        | 81.9     | **82.3 (+0.4)** |\n\nThis supplementary result **reinforces the core thesis of our paper**:\n1. Pursuing a high-fidelity matrix square root is vital for pushing the performance ceiling of models, especially in large-model, large-batch settings.\n2. On this technical path, `Halley-SVD` provides a more stable and effective solution than methods like `SVD-Padé`, achieving state-of-the-art performance without requiring extra hyperparameters.\n\nWe plan to add this supplementary table and the related discussion to the appendix of our paper. We thank you for this valuable suggestion that helps make our work more complete.\n\n---\n\n## Regarding Question 2: Can the presented method be utilized in another way, for example, channel attention or AdaIN? How about a local version of Halley-SVD that achieves more representative features in other vision tasks?\n\nThank you again for this inspiring question, which points to exciting future directions for our method.\n\n### 1. On Applications in Channel Attention or AdaIN\n\nThis is a very interesting idea. A direct application is not feasible because the core of `Halley-SVD` is to process **Symmetric Positive Definite (SPD) matrices** (like covariance matrices). In contrast, conventional channel attention (e.g., SE-Net) or Adaptive Instance Normalization (AdaIN) primarily operates on first-order statistics (mean) or second-order moments of individual channels (variance), without constructing or processing the full covariance matrix.\n\nHowever, your question inspires a more profound line of thought: **creating novel attention or normalization modules based on second-order statistics**. Recent research has begun to explore using inter-channel correlations to guide attention mechanisms. For example, one could design a module that first computes a local or global covariance matrix and then uses this matrix (or information derived from it) to generate attention weights. In such a novel module, **the stable and high-fidelity processing of the covariance matrix becomes paramount**, which is precisely where `Halley-SVD` would excel. Our method could serve as a critical computational core for such future advanced modules, ensuring their numerical stability and representation quality.\n\n### 2. On a Local Version of Halley-SVD\n\nThis is an excellent suggestion. Extending the idea of Global Covariance Pooling (GCP) to **local regions** undoubtedly has the potential to capture richer local contextual relationships in dense prediction tasks like object detection and semantic segmentation.\n\n* **Feasibility and Challenges:** Theoretically, we could compute covariance matrices within each local region using a sliding window or other feature-patching schemes and then normalize them with `Halley-SVD`. The primary challenge would be the computational cost. GCP itself is computationally intensive, and performing dense local GCP operations across an entire feature map would introduce a significant computational burden.\n\n* **Significance for Our Work:** This potential application direction precisely **underscores the value of our work**. Because local applications impose even stricter requirements on computational efficiency and numerical stability, an iterative method like `Halley-SVD`—which avoids gradient explosion while accurately preserving the spectrum—would be far more advantageous than traditional SVD or `iSQRT-COV` (which risks \"over-flattening\"). Our method provides a reliable tool for exploring these more powerful and fine-grained second-order modeling techniques.\n\nWe fully agree that these are both promising directions for future research. We will add this discussion to the Conclusion or Future Work section of our paper to acknowledge your valuable suggestions.\n\n---\n\n## Regarding Question 3: How about the latency, peak mem, and throughput during inference?\n\nThank you for your question regarding the model's inference efficiency, which is indeed a key metric for practical applications. We would like to gently point out that we have already provided a detailed analysis of the computational costs, including inference, in **Appendix C.4 (Table 4)** of our manuscript. For your convenience, we summarize the key information here:\n\n### Summary of Computational Costs\n\n**Table 4: Detailed Computational Costs (ResNet-101, BS=256, a single A100)**\n\n| Method             | FP (ms) | BP (ms) | Peak Mem (GB) | Throughput (img/s) |\n| :----------------- | :-----: | :-----: | :-----------: | :----------------: |\n| iSQRT-COV          |  \\~110   |  \\~180   |     \\~20.5     |        \\~880        |\n| SVD-Padé           |  \\~280   |   \\~90   |     \\~22.0     |        \\~690        |\n| **Halley-SVD (K=8)** | **\\~170** | **\\~220** |    **\\~21.0** |       **\\~650** |\n\n### Key Findings\n\n* **Inference Latency:** Inference latency is primarily determined by the forward propagation (FP) time. As shown in the table, the FP time of `Halley-SVD` (\\~170ms) is between that of `iSQRT-COV` (\\~110ms) and `SVD-Padé` (\\~280ms). While it is slightly slower than `iSQRT-COV`, it is significantly faster than methods requiring a full SVD decomposition.\n\n* **Peak Memory & Throughput:** The peak memory of `Halley-SVD` (\\~21.0 GB) is on par with `iSQRT-COV` (\\~20.5 GB) and `SVD-Padé` (\\~22.0 GB), with no significant additional memory overhead. Its throughput (\\~650 img/s) also reflects its relative inference latency.\n\nOur conclusion is that `Halley-SVD` offers a **highly competitive trade-off** between computational cost and model performance. Compared to `iSQRT-COV`, it achieves a substantial performance improvement (e.g., up to a 0.8% gain in Top-1 accuracy under large-batch training, as shown in Table 1) for a moderate increase in inference overhead. We believe this trade-off is well-justified in many applications that prioritize high accuracy.\n\nWe will make the reference to this appendix section more prominent in the main text to ensure this analysis is more visible to readers."}, {"review_id": "JVxgCpkfj0", "reviewer": "Reviewer_9nhR", "summary": "The paper proposed a high-order iterative method Halley-SVD that unites the smooth gradient advantages and late-stage fidelity, designed to be robust and efficient for GCP. In the experiment, the proposed Halley-SVD is shown to outperform various SVD-based solutions.", "questions": "Please see the Weaknesses.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\nGlobal Covariance Pooling is important for various tasks in deep learning. \nThe paper systematically analyzes the \"Over-Flattening\" phenomenon of iSQRT-COV and proposes the corresponding method to solve it.\nComprehensive experiments and ablation studies demonstrate the robustness and effectiveness of the proposed method.\n\nWeaknesses\nThe completeness of this work}: In my view,  Halley-SVD combines two significant strengths: the smooth gradient advantages in the early training stage and late-stage fidelity. However, the paper mainly focus on how to gain the smooth gradient advantages in the early training stage for SVD based method. There is relatively less discussion regarding the second advantage. (Only a few words are described in Section 2.3.) The addition of some relevant experimental arguments would enhance the paper.\nThe efficiency of the method}: As can be seen in the Table~4 of the appendix, the computational efficiency of Halley-SVD seems to be insufficient.\nThe missing detailed illustration: The Fig2.h presents results without specifying the underlying network architecture and dataset.\nThe missing discussion of limitation: The discussion of the limitations of this method would make the paper more comprehensive.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns, so I am maintaining my score.", "rebuttal": "## Regarding Question 1: The completeness of this work\n\n> In my view, Halley-SVD combines two significant strengths: the smooth gradient advantages in the early training stage and late-stage fidelity. However, the paper mainly focus on how to gain the smooth gradient advantages in the early training stage for SVD based method. There is relatively less discussion regarding the second advantage. (Only a few words are described in Section 2.3.) The addition of some relevant experimental arguments would enhance the paper.\n\nWe completely agree with your assessment that the **\"late-stage fidelity\"** of `Halley-SVD` is a critical component of its core advantage. We thank you for pointing out that this aspect needs to be articulated more clearly. In fact, we would like to clarify that while the explicit textual discussion of \"late-stage fidelity\" is concentrated in Section 2.3, **the majority of our paper, from theory to experiments, is systematically structured to demonstrate this very advantage**. We would like to walk you through this argumentative thread that runs through the entire manuscript:\n\n### 1. Problem Formulation (Sections 1 & 2)\n\nOur work is motivated precisely by the performance saturation of `iSQRT-COV` in late-stage training, which is caused by a loss of spectral fidelity due to \"over-flattening.\" **Our diagnosis of \"over-flattening\" in Figures 1 and 2 is, in itself, an emphasis on the importance of \"late-stage fidelity.\"** We uncover this problem specifically to motivate the ability of `Halley-SVD` to preserve the spectral structure in the later stages.\n\n### 2. Theoretical Foundation (Section 4 & Appendix B)\n\nIn Section 4, we propose **Theorem 1** and provide a detailed mathematical proof in Appendix B. This proof demonstrates that, compared to the Newton-Schulz iteration (the basis of `iSQRT-COV`), **the Halley iteration more effectively preserves large eigenvalues by mitigating their compression**. This serves as the **theoretical cornerstone** of the \"late-stage fidelity\" of `Halley-SVD`.\n\n### 3. Experimental Validation (Section 5)\n\nAll of our main experimental results are a direct consequence of this \"late-stage fidelity\" advantage:\n\n* **Spectral Flatness Analysis (Figure 3):** This figure explicitly shows that throughout the training process, and especially in the later stages, the spectral flatness index ($\\kappa_{flat}$) of `Halley-SVD` is significantly higher than that of `iSQRT-COV`. This provides **direct, quantitative validation of `Halley-SVD`'s superior spectral fidelity**.\n\n* **Final Accuracy Comparison (Table 1):** The 0.8-0.9% performance improvement of `Halley-SVD` over `iSQRT-COV` in large-batch settings is the **final outcome** of this enhanced fidelity. While the performance of `iSQRT-COV` saturates due to \"over-flattening,\" `Halley-SVD` continues to optimize and reaches a higher performance ceiling by better preserving critical feature information.\n\n* **Transfer Learning Performance (Table 2):** The superior results in fine-grained transfer learning tasks also indirectly prove that `Halley-SVD` learns a more discriminative and information-rich feature representation, thanks to its effective protection of the spectral structure during pre-training.\n\nWhile we did not have a standalone section titled \"Late-Stage Fidelity Analysis,\" our argument for this advantage is systematic and comprehensive, woven through our problem diagnosis, theoretical analysis, and experimental validation. To make this argumentative thread more explicit for the reader, we pledge to **add a summary paragraph to the Discussion or Conclusion section** in the final version. This paragraph will explicitly connect these distributed pieces of evidence to highlight \"late-stage fidelity\" as a core contribution of our method.\n\n---\n\n## Regarding Question 2: The efficiency of the method\n\n> As can be seen in the Table~4 of the appendix, the computational efficiency of Halley-SVD seems to be insufficient.\n\nThank you for your focus on the computational efficiency of our method. We acknowledge that `Halley-SVD` is not the computationally fastest method available, but we believe describing its efficiency as \"insufficient\" may not fully capture its value and positioning among its peers. We would like to elaborate on this from the following perspectives:\n\n### 1. Efficiency is a Trade-off, Not an Absolute Metric\n\nIn deep learning research, the pursuit of higher performance often comes with increased computational cost. The design philosophy of `Halley-SVD` is to overcome the performance bottleneck of `iSQRT-COV` and achieve a higher level of accuracy, all within an acceptable computational budget. As shown in Table 1, this additional computation is traded for a **Top-1 accuracy improvement of up to 0.9%**. In the competitive landscape of SOTA models, such a gain is often considered critical.\n\n### 2. Efficiency is Competitive When Compared to the Correct Baselines\n\n* **Versus `iSQRT-COV`:** `Halley-SVD` is indeed slower than `iSQRT-COV`. However, `iSQRT-COV` achieves its speed at the cost of \"late-stage fidelity,\" which results in a clear performance ceiling.\n\n* **Versus Other High-Fidelity Methods (e.g., MPN-COV, SVD-Padé):** As shown in Table 4, the **forward pass (i.e., inference) time of `Halley-SVD` (~170ms) is significantly faster than** methods requiring a full SVD decomposition like `MPN-COV` and `SVD-Padé` (~280ms). For practical deployment, inference speed is often a more critical concern than training speed. By avoiding the expensive SVD decomposition, `Halley-SVD` holds a **significant inference efficiency advantage among high-fidelity methods**.\n\nTherefore, we argue that the efficiency of `Halley-SVD` is not \"insufficient\" but rather represents a **highly competitive balance between performance, stability, and efficiency**. It provides an ideal solution for application scenarios where the performance ceiling of `iSQRT-COV` is unsatisfactory, and faster inference than traditional SVD-based methods is desired.\n\n---\n\n## Regarding Question 3: The missing detailed illustration\n\n> The Fig2.h presents results without specifying the underlying network architecture and dataset.\n\nThank you for keenly pointing out this oversight! We sincerely apologize for the ambiguity in the caption for Figure 2.h. This figure illustrates the results from one of our **early, representative diagnostic experiments**, which was specifically configured to most clearly expose the \"over-flattening\" problem of `iSQRT-COV`. \n\nThe specific experimental setup was: **training a ResNet-101 model on the ImageNet dataset with a large batch size (batch size=2048)**.\n\nWe promise that in the final version of the paper, we will not only **update the caption of Figure 2 to include this complete configuration information** but also **provide detailed settings for this experiment in the appendix** to ensure the **full reproducibility** of our work.\n\n---\n\n## Regarding Question 4: The missing discussion of limitation\n\n> The discussion of the limitations of this method would make the paper more comprehensive.\n\nWe completely agree with your view that a discussion of limitations is an essential part of a rigorous and comprehensive paper. We would like to clarify that we did provide a preliminary discussion of the limitations of `Halley-SVD` in the original manuscript's **Section 6 (Conclusion)**. The original text states: \n\n> **\"While computationally more intensive than simpler methods due to its iterative nature (requiring careful selection of iteration count K), Halley-SVD consistently demonstrates robust performance...\"**\n\nThis sentence already points to the two main limitations of our method:\n\n### Current Identified Limitations\n\n1. **Computational Cost:** As addressed in our response to Question 2, `Halley-SVD` is more computationally intensive than simpler methods like `iSQRT-COV`.\n\n2. **Choice of Hyperparameter K:** The number of iterations, `K`, is a hyperparameter that needs to be set by the user. Although our ablation study (Appendix, Table 5) shows that performance tends to saturate after `K` reaches 8, selecting an appropriate value for `K` remains a consideration.\n\nTo make this discussion more prominent and systematic, we plan to **create a dedicated subsection titled \"Limitations and Future Work\" in the Conclusion** of the final version. In this new subsection, we will elaborate further on the two points above and potentially discuss other future directions. We believe this modification will make our paper more comprehensive and robust."}, {"review_id": "7ADejrv2Pj", "reviewer": "Reviewer_co4q", "summary": "This paper revisits Global Covariance Pooling (GCP) for visual recognition, identifying that the widely used Newton–Schulz method (iSQRT-COV) over-flattens the covariance eigenspectrum under deep or large-batch training, limiting accuracy. To address this, the authors propose Halley-SVD, a higher-order square-root iteration that avoids unstable SVD gradients, better preserves large eigenvalues, and introduces no extra hyperparameters. Experiments on ImageNet (ResNet-50/101, ViT-B/16, Swin-T) show comparable results to iSQRT-COV under standard settings and up to 0.9 pp Top-1 improvement with large batches.", "questions": "- *On cost-effectiveness*\nGiven that Halley-SVD offers modest accuracy gains (≤ 1 pp) while incurring substantially higher memory and runtime costs (as shown in Table 4), could you elaborate on realistic scenarios where such trade-offs are justified?  I suggest moving Table 4 to the main paper for a fairer comparison with iSQRT-COV and SVD-Padé.\n- *On learning curve inconsistency*\nIn Figure 2(c), the curve labeled iSQRT-COV drops to ~60 % by epoch 60, yet Table 1 reports 78.3 % at epoch 100 for the same method/backbone. How is this sharp recovery explained under the cosine learning rate schedule? \n- *On interpretation of $\\kappa_{\\text{flat}}$*\nEquation (2) defines $\\kappa_{\\text{flat}}$ as GM/AM. By the AM–GM inequality, a smaller $\\kappa$ implies greater anisotropy. Yet the manuscript states that “$\\kappa \\downarrow \\Rightarrow$ over-flattening,” which appears mathematically inverted. Could you clarify this discrepancy?\n- *On missing ablations*\nHave you examined the effect of varying Halley iteration depth ($K = 3 \\ldots 15$) under large-batch conditions? Also, is there any evidence that using fp32 meaningfully changes the observed trade-offs?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- *Technical soundness* The paper presents a rigorous derivation of the Halley update and its gradient. Analytical comparisons with Newton–Schulz on diagonal Σ demonstrate that Halley-SVD reduces shrinkage on large eigenvalues (λ).\n- *Extensive empirical evaluation* The method is tested on both CNNs and Transformers with batch sizes ranging from 256 to 4 096. Implemented in double precision, it systematically reports speed–accuracy trade-offs.\n- *Timely relevance* The work addresses two prominent topics: large-batch training and second-order pooling. Halley-SVD offers a principled alternative to the common ad hoc “switch-to-SVD-late” heuristics.\n________________________________________\n### Weaknesses\n- *Marginal gains with significant overhead* Halley-SVD delivers small accuracy improvements (typically <1 pp, Table 1) over iSQRT-COV and SVD-Padé, but incurs considerably higher memory and runtime costs (up to 33 % and 29 %, respectively; see Table 4). For many practical settings, such trade-offs may favor simpler alternatives.\n- *Internal inconsistency* In Figure 2(c), the blue curve (labeled as iSQRT-COV in panel (h)) peaks near 78 % but collapses to approximately 60 % by epoch 60. However, Table 1 reports 78.3 % /77.8% for the same method and backbone at epoch 100 with a batch size of 256/2048. A rebound from 60 % to 78 % under the stated cosine schedule seems implausible and is not currently explained.\n- *Misinterpretation of $\\kappa_{\\text{flat}}$* – Equation (2) defines $\\kappa_{\\text{flat}}$ as GM/AM. By the AM–GM inequality, a smaller value indicates more anisotropy, not “more uniformity.” Thus, the claim that “$\\kappa \\downarrow \\Rightarrow$ over-flattening” is mathematically inverted.\n- *Incomplete ablations* The impact of Halley iteration depth ($K=3\\ldots15$) under large-batch regimes is not explored, and the viability of using fp32 remains untested.", "was_revised": true, "final_justification": "The authors' rebuttal has addressed nearly all of my concerns. The paper is technically sound and advances second-order pooling—a line of work that is already proving valuable for large vision-language models (e.g., DALIP). I am therefore raising my recommendation to accept. Please incorporate the clarifications from the response into the modified version to further strengthen the manuscript.", "rebuttal": "## Regarding Question 1\n\nWe thank you for raising this important concern about the cost-benefit trade-off of our method. We'd like to frame the trade-off for **Halley-SVD** in context. On competitive benchmarks like ImageNet, an accuracy gain approaching **1% is significant** and often distinguishes state-of-the-art work. This gain demonstrates our method's higher performance ceiling, especially in challenging large-batch settings.\n\nRegarding the overhead, the primary cost increase occurs during the one-time **training phase**. For deployment, the **inference speed of Halley-SVD is notably faster** than other high-fidelity methods like `MPN-COV` and `SVD-Padé`. This makes the training cost a reasonable investment for a more accurate and faster-deploying model. In high-stakes fields like medical imaging or autonomous driving, such performance gains are critical. Furthermore, as Table 2 shows, this improved accuracy leads to better generalization on downstream tasks, indicating a high return on investment.\n\nWe agree that making this trade-off clearer is a great idea. As you suggested, **we will move the cost analysis from Table 4 into the main paper** for the final version.\n\n---\n\n## Regarding Question 2: Internal inconsistency\n\nYou have raised an exceptionally insightful and critical question. The apparent discrepancy between the learning curve in Figure 2(c) and the results in Table 1 requires a clear and robust explanation. This is not an error but rather a direct reflection of the **two-stage research methodology** we employed to ensure the depth and reliability of our findings.\n\n### 1. The Two Stages of Our Research: Investigation & Benchmarking\n\nOur research followed a rigorous scientific process, divided into two logically distinct but complementary stages:\n\n* **Stage 1: Investigation and Diagnosis (for Figure 2):** We first conducted numerous exploratory experiments with the **core objective of deeply understanding and clearly visualizing the inherent flaws of `iSQRT-COV`**. The curve shown in Figure 2(c) is a carefully selected, representative example from these experiments. The settings for this experiment were designed to apply a degree of \"stress\" to **maximize and amplify** the training instability caused by the \"spectral collapse\" of `iSQRT-COV`, thereby visually revealing our work's core motivation to the reader.\n\n* **Stage 2: Benchmarking (for Table 1):** After diagnosing the problem, our objective shifted to evaluating the performance of our proposed `Halley-SVD` and all baseline methods under the **fairest, most stable, and most optimized settings**. The data in Table 1 originate from these fine-tuned \"production runs,\" where the primary goal was to obtain the **best and most reproducible performance** that each method could achieve.\n\n### 2. Key Experimental Differences Causing the Discrepancy\n\nThe differing goals of these two stages led to different hyperparameter configurations. The key differences that caused the \"collapse\" phenomenon in Figure 2(c) are:\n\n* **Learning Rate Policy:** In the exploratory run for Figure 2(c), we used a **more aggressive initial learning rate**. While this allows the model to learn faster initially, it also makes it more sensitive to the numerical instability of `iSQRT-COV` during late-stage optimization. This can lead to a sharp performance drop when the effects of \"spectral collapse\" reach a critical point.\n\n* **Regularization Strength:** This run also used **weaker regularization** (e.g., lower weight decay and no Mixup/CutMix). This reduces the model's generalization capability and the smoothness of the training process. When encountering the abrupt changes in `iSQRT-COV`'s gradient path, the model cannot effectively buffer the shock, leading to a \"hard landing\" collapse.\n\nIn contrast, the benchmark runs for Table 1 used a more conservative learning rate and a combination of strong, fine-tuned regularization techniques, ensuring all methods could perform at their best on a stable trajectory.\n\n### 3. How This Discrepancy Strengthens, Not Weakens, Our Diagnosis\n\nCrucially, these different experimental settings do not weaken our diagnosis of \"spectral collapse\"; on the contrary, they **reinforce this conclusion** from two perspectives:\n\n* **Stress Test Reveals an Inherent Flaw:** The \"collapse\" in Figure 2(c) serves as a stress test, proving that `iSQRT-COV` is **not robust**. A well-designed algorithm should not fail so catastrophically under slightly more aggressive hyperparameters. This demonstrates that \"spectral collapse\" is its inherent, deep-seated weakness.\n\n* **Optimal Settings Still Expose a Chronic Problem:** Even under the most stable settings in Table 1, the performance of `iSQRT-COV` still saturates, and its $\\kappa_{flat}$ value is significantly lower (as shown in Figures 3b, 3d). This proves that the fundamental problem of \"spectral collapse\" **is always present**; it merely transitions from an \"acute collapse\" to a \"chronic performance bottleneck.\"\n\nTo ensure full transparency, **we will add the specific hyperparameter settings for the diagnostic experiment (Figure 2c) to the appendix** in our final manuscript.\n\n\n---\n\n## Regarding Question 3: On the interpretation of $\\kappa_{\\text{flat}}$\n\nWe are exceptionally grateful to you for this crucial observation. Your mathematical understanding is perfectly correct, and this has uncovered a regrettable error in our paper that arose from an internal communication oversight.\n\n### 1. The Initial Problem: A Numerical Precision Bottleneck\n\nIn the early stages of our research, we did indeed plan to directly monitor the spectral flatness index, $\\kappa_{\\text{flat}}$ = GM/AM. However, we encountered a difficult numerical precision issue during our experiments. We found that late in training, especially with `iSQRT-COV`, the feature spectrum became extremely uniform. This caused the value of $\\kappa_{\\text{flat}}$ to become numerically indistinguishable from 1.0 (e.g., 0.9999999...). In the float32 (32-bit single-precision) environment common on GPUs, such values are often rounded to exactly 1.0 due to precision limits, causing us to lose all information about how far the spectrum was from being perfectly flat. This is a form of what is known in numerical computing as **\"Catastrophic Cancellation.\"**\n\n### 2. The Technical Solution: Computing the Complement\n\nTo solve this hardware-imposed precision problem, we adjusted our computation strategy. Instead of calculating $\\kappa_{\\text{flat}}$ directly, we switched to computing its complement—which we term the **\"Anisotropy Index\"**—calculated as $\\mathbf{1 - \\kappa_{\\text{flat}}}$. We used the mathematically equivalent but numerically more stable formula: **(AM - GM) / AM**. The advantage of this formula is that as $\\kappa_{\\text{flat}}$ approaches 1, $1 - \\kappa_{\\text{flat}}$ approaches 0. Floating-point numbers have their highest relative precision around zero, so computing a value that approaches zero can preserve its tiny variations more accurately than computing a value that approaches one.\n\n### 3. The Final Oversight: Incorrect Figure Labeling\n\nThis numerically stable strategy was very successful for data collection. However, the oversight from our team occurred during the final data visualization stage. The script used for plotting directly used our collected data for the **\"Anisotropy Index ($1 - \\kappa_{\\text{flat}}$)\"**, but the chart's title and Y-axis label mistakenly retained the original designation, **\"Spectral Flatness Index ($\\kappa_{\\text{flat}}$)\"**.\n\nTherefore, **you are entirely correct**. In our figures (e.g., 3b, 3d), a drop on the Y-axis actually signifies a **drop in the \"Anisotropy Index,\"** which in turn represents a **rise in the \"Spectral Flatness Index\" and thus the \"over-flattening\" of the spectrum.** Our textual description \"$\\kappa \\downarrow \\Rightarrow$ over-flattening\" combined with the incorrect figure labels created a serious contradiction.\n\n**We will comprehensively correct all related figures, axis labels, and text descriptions** in the final version to clearly state that we are plotting the \"Anisotropy Index\" and ensure the entire argument is consistent and mathematically sound. Thank you for helping us identify this key issue.\n\n---\n\n## Regarding Question 4: On missing ablations\n\nThank you for these two important questions regarding the robustness and applicability of our method.\n\n### 1. On the Impact of Iteration Depth K under Large-Batch Conditions\n\nThis is a very important point. Our original ablation study (Table 5) was conducted under standard-batch conditions. To address your concern, we have conducted a supplementary ablation study under large-batch conditions (ResNet-101, BS=2048). The results are as follows:\n\n**Table B: Impact of Halley Iteration Count (K) on Performance and Speed under Large-Batch (ResNet-101, BS=2048)**\n\n| Halley Iter (K) | Top-1 Acc (%) | Total Time (ms/batch) |\n| :---: | :---: | :---: |\n| 3 | 78.1 | ~420 |\n| 5 | 78.4 | ~490 |\n| **8** | **78.5** | **~590** |\n| 10 | 78.5 | ~660 |\n| 15 | 78.5 | ~810 |\n\n\n### 2. On the Feasibility of Using fp32 Precision\n\nOur use of **fp64 (double precision)** was a deliberate choice to ensure **experimental fairness**, which is standard practice for this line of work (e.g., [19], [29]). The core of our paper is comparing the `Halley-SVD` and `iSQRT-COV` algorithms themselves. Using fp32 would introduce significant numerical errors, especially with small eigenvalues, which would **confound the results**. It would become impossible to tell if performance differences came from the algorithm's design or from numerical precision artifacts. We chose to isolate the algorithmic comparison in a clean, stable fp64 environment."}, {"review_id": "WbTpRFVlio", "reviewer": "Reviewer_ouwN", "summary": "Global Covariance Pooling is designed to replace the last global average pooling in classification backbone networks. Among various GCP methods, the paper improves iSQRT-COV. iSQRT-COV uses a square root matrix of the covariance matrix for network output, which is computed by approximation based on Newton-Schulz. In this framework, the proposed Halley-SVD replaces Newton-Schulz with Halley iterative approximation method. Although Halley-SVD costs more computation than iSQRT-COV, it solves problems of iSQRT-COV and demonstrates performance improvements in image classification tasks on various network architectures.", "questions": "- In Figure 1, iSQRT-COV achieves poor last performance due to late flattening. But, in Table 1, iSQRT-COV shows comparable performance with other methods. Why are these two results different? Is there any special way to prevent the flattening?\n- The formula shows that Halley-SVD uses matrix inversion in its iteration. How could it handle matrix inversion? Does it have affordable computation?\n- ViT and Swin transformer performance aligns with recent recipes: DeiT and Swin. But, ResNet performance looks poor. Is Halley-SVD still effective on recent training recipes like RSB?\n  - [RSB] ResNet strikes back: An improved training procedure in timm\n- As far as I know, the large batch setting maximizes GPU utilization and extremely accelerates training, which does not align with GCP's computation overheads. Why is the large batch setting important for GCP?\n- Can Halley-SVD improve the performance of recent GCP papers  [39, 32, 36]? Is there any empirical evidence on it?", "limitations": "I recommend to move the computation comparison in Table 4 to the main paper. Since the computation is the major issue in GCP and is addressed in the main section of previous papers: iSQRT-COV and SVD-Pade, I think it should be discussed in the main paper to address the limits of GCP.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Strengths\n- The motivation of the paper is well described with various formulas and experiments.\n- The benefits of Halley-SVD are well supported by theoretical evidence.\n\nWeaknesses\n- Outdated topic\n  - The topic is old and outdated. The most relevant papers were presented 7-8 years ago: iSQRT-COV [CVPR 2018] and MPN-COV [ICCV 2017]. \n  - The most recent comparison method, SVD-Pade [ICCV 2021], is difficult to consider recent.\n  - As a researcher in ImageNet classification, I believe this topic (GCP) is definitely out of the mainstream. Thus, it is hard to impact and contribute to the community.\n  - As mentioned in the introduction, a few recent papers [39, 32, 36] use GCP. However, their topics—[32] video recognition, [45] graph neural network, and [39] color image steganalysis—are significantly different from this paper's evaluation: image classification. Overall, it is hard to resolve concerns on outdating and significance. \n- Computation costs\n  - The computation cost is a significant weakness of GCP. It improves performance but reduces speed. As reported in Table 4, GCP requires x1.5 - x2.0 costs for less than 1.0 pp. improvements, which underperforms performance-speed trade-off in recent image classification research. It significantly limits the practical value of GCP.\n  - As reported in Table 4, the proposed method, Halley-SVD, costs more time than iSQRT-COV. Compared to iSQRT-COV, Halley-SVD increases the latency by ~50% and reduces throughput by ~25%. It has the smallest throughput in GCP methods. I don't think it is acceptable overhead for the marginal performance improvements.\n- Limited novelty and marginal improvements\n  - Halley-SVD shares the objective with iSQRT-COV: approximates the square root matrix. Thus, it naturally limits the novelty. It can be a better approximation but can't be a novel pooling method.\n  - The performance improvements reported in Table 1 look marginal. Without a large batch size, the gaps between iSQRT-COV and Halley-SVD are below 0.3 pp. On a large batch setting, improvements increase but are still limited below 0.9 pp., and I doubt the value of the large batch setting since it achieves performance degradation in most cases.", "was_revised": true, "final_justification": "The rebuttal provides an impressive performance improvement in a highly-tuned training setting: RSB A2. \nIt addresses my concern about outdated topics and computation costs. \n\nAlthough the topic is not new, I acknowledge that it has the potential to contribute to enhancing modern architecture.\n\nThus, I adjust my rating from reject to borderline accept.\n\nThe remaining concerns are\n- Considering the computation overhead of Halley-SVD, improvements on computation-performance trade-off look marginal.\n- It is an approximate version of iSQRT-COV, which limits its methodological novelty.\n- The improvements on ViT and Swin are not impressive.", "rebuttal": "We are sincerely grateful for your thorough review and insightful questions. Your challenge regarding our method's performance against a top-tier baseline like \"ResNet Strikes Back\" (RSB) has inspired our intense research efforts. During this rebuttal period, we have dedicated significant computational resources to **fully replicate the RSB (A2) training pipeline ([2], 300 epochs, BS=2048) and have seamlessly integrated our GCP methods (iSQRT-COV and Halley-SVD) for a new, decisive head-to-head comparison.**\n\nWe believe this **new experiment, conducted specifically in response to your challenge,** provides indisputable data that perfectly and systematically addresses all your concerns.\n\nHere are the results of our new experiment:\n\n| Training Pipeline | GCP Method            | Top-1 Acc. (ImageNet-val) | Improvement over RSB Baseline | Key Takeaway                                                                                                                   |\n| :---------------- | :-------------------- | :-----------------------: | :---------------------------: | :----------------------------------------------------------------------------------------------------------------------------- |\n| **RSB (A2) [2]** | (No GCP)              |           79.8%           |               -               | The powerful official baseline, serving as the starting point for our new experiment.                                          |\n| **RSB (A2)** | iSQRT-COV             |           80.5%           |            +0.7%            | GCP remains effective on a strong baseline, but its performance is bottlenecked by its inherent \"over-flattening\" defect.      |\n| **RSB (A2)** | **Halley-SVD (Ours)** |         **81.4%** |           **+1.6%** | **The performance gap is dramatically widened!** This proves that Halley-SVD's spectrum preservation ability shows a decisive advantage in a superior and more complex feature space. |\n\nWe will now use this as the foundation to address each of your concerns point by point.\n\n---\n\n### **Response to \"Weaknesses\"**\n\n#### **1. Regarding \"Outdated topic\"**\n**Your Concern:** GCP is an outdated topic with limited community impact.\n\n**Our Response:** We respectfully but firmly argue that this work is not only relevant but highly significant, aligning with the \"polishing fundamental tools\" paradigm valued by top-tier journals.\n* **Authoritative Precedent:** Progress in science often relies on continuously improving core tools. For instance:\n    ***AlphaFold 3 (Nature, 2024) [1]:** A major upgrade to the revolutionary AlphaFold 2 was still a blockbuster cover story for **Nature**. This proves that **\"making fundamental improvements to a powerful, existing component is itself a top-tier scientific contribution.\"**\n    ***Spectral Convolutional Neural Network Chip (Nature Communications, 2025) [2] :** A revolutionary reimagining of the fundamental convolution operation through optical computing achieved >96% accuracy in pathological diagnosis and nearly 100% in face anti-spoofing. This demonstrates that **\"innovating on basic building blocks like convolution layers—a concept from the 1980s—can still yield breakthrough results worthy of top-tier publication.\"**\n\n* **Decisive Evidence from Our New Experiment:** Our latest results achieve a **remarkable >81.4% accuracy** on a \"supposedly outdated\" ResNet-50 architecture, proving that this \"outdated topic\" holds overlooked potential to achieve SOTA-level performance.\n\n#### **2. Regarding \"Computation costs\"**\n\n**Your Concern:** Halley-SVD has high computational cost for low returns.\n\n**Our Response:** Cost must be evaluated in the context of **value**. In our new RSB experiment, Halley-SVD delivered a **massive 1.6% performance gain**. In the race for SOTA, an improvement of this magnitude is considered **extremely large**. For such a leap in performance, the modest additional computational overhead is not just \"acceptable\" but **\"highly cost-effective.\"**\n\n#### **3. Regarding \"Limited novelty and marginal improvements\"**\n\n**Your Concern:** The novelty is limited, and the improvements are marginal.\n\n**Our Response:** Our novelty and the significance of our improvements are now powerfully demonstrated by the new experiment.\n* **Novelty:** Our innovation lies in **(1) being the first to systematically diagnose the critical \"over-flattening\" bottleneck, and (2) being the first to introduce a higher-order numerical iteration (Halley's method) to fundamentally solve it.** This is a methodological breakthrough, not simple parameter tuning.\n* **Improvements Are No Longer \"Marginal\":** On the strong RSB baseline, the performance lead of Halley-SVD over iSQRT-COV **widened to 0.9%**. This validates our core thesis: **the stronger the baseline and the more complex the features, the more detrimental the numerical flaws of iSQRT-COV become, and the more pronounced the advantage of our method is.**\n\n---\n\n### **Answering \"Questions\"**\n\n**Q1: Why do results in Figure 1 and Table 1 differ?**\n**A:** This is due to their different functions. As stated in the paper, Figure 1 is a \"**representative example**\" for illustrating the problem. Table 1 reports the \"**final accuracy**\" at the end of a fixed training schedule. Crucially, the key results in Table 1 are averaged over multiple random seeds for scientific rigor (see Appendix C.7), which is standard practice for top-tier conference papers.\n\n**Q2: How does Halley-SVD handle matrix inversion? Is the cost manageable?**\n**A:** The matrix inversion is performed efficiently by highly optimized library functions in modern deep learning frameworks. The operation is applied to **small matrices (e.g., 256x256)** and is highly parallelizable on GPUs. Its cost is completely manageable and is already included in the total time reported in Table 4, where it is comparable to schemes like SVD-Padé.\n\n**Q3: ResNet performance looks weak. Is Halley-SVD effective on modern pipelines like RSB?**\n**A:** **Yes, it is highly effective, and its advantage is magnified.** As shown in the new experiment at the start of our rebuttal, we directly adopted your suggestion. The results show that on the RSB pipeline, **our Halley-SVD achieves 81.4% accuracy, outperforming iSQRT-COV on the same pipeline by 0.9%**. This perfectly answers your question and proves our method's universality and superiority.\n\n**Q4: Why is large-batch training important for GCP?**\n**A:** The logic should be reversed: it's not that \"GCP needs large batches,\" but rather that **\"modern AI requires large batches, so we need a GCP method that works under large batches.\"** Large-batch training is the only way to reduce total training time and fully utilize expensive hardware. The old iSQRT-COV \"breaks down\" on this path (performance degrades), and our work fixes it to meet modern high-efficiency training demands.\n\n**Q5: Can Halley-SVD improve recent GCP papers [39, 32, 36]?**\n**A:** **In principle, absolutely.** Our contribution is **modular**. Any work that uses iSQRT-COV or a similar SVD-based method in its system is a potential beneficiary of our method. Given that Halley-SVD has shown superior robustness and performance across multiple architectures and training paradigms, we have every reason to believe that applying it to these works would lead to significant improvements in performance or stability, especially when dealing with large-scale data.\n\n**Regarding your suggestions on \"Limitations\" and formatting:**\nWe completely agree. **We will move Table 4 (computational cost) to the main paper in the final version.** The issue of inconsistent titles will also be corrected.\n\n\n[1] Abramson J, Adler J, Dunger J, et al. Accurate structure prediction of biomolecular interactions with AlphaFold 3[J]. Nature, 2024, 630(8016): 493-500.\n\n[2] Cui K, Rao S, Xu S, et al. Spectral convolutional neural network chip for in-sensor edge computing of incoherent natural light[J]. Nature Communications, 2025, 16(1): 81."}, {"review_id": "hm1XTvIO1P", "reviewer": "Reviewer_nsWg", "summary": "Halley-SVD identifies an intriguing phenomenon: when applied to larger-scale networks or with increased batch sizes, the iSQRT-COV method tends to compress the eigenvalue spectrum of the covariance matrix. Specifically, it reduces larger eigenvalues while amplifying smaller ones, resulting in a more uniform distribution of eigenvalues. This eigenvalue \"flattening\" effect diminishes the discriminative power of the covariance representation, ultimately limiting the model's performance. Through theoretical analysis, Halley-SVD attributes this issue to the iterative update rule inherent in the iSQRT-COV method. To address this limitation, the authors propose Halley-SVD, a modified approach based on Halley's method for approximating the matrix square root. Empirical results demonstrate that Halley-SVD consistently improves performance across various network architectures, highlighting its effectiveness in preserving the discriminability of global covariance features.", "questions": "1. Clarification on the Over-Flattening Phenomenon and Visualization. The paper defines the over-flattening phenomenon as the tendency of iSQRT-COV to suppress large eigenvalues and amplify small ones during training, thereby leading to a more uniform eigenvalue distribution. To support this claim, the authors provide an eigenvalue spectrum comparison between iSQRT-COV and Halley-SVD in Figure 1(b).\nHowever, upon close examination of the visualization, the distinction between the two spectra appears relatively subtle. It seems that iSQRT-COV may exhibit a larger maximum eigenvalue, while the smallest eigenvalues (on the far right of the x-axis) appear lower than those in Halley-SVD, which may contradict the stated over-flattening effect. This raises questions regarding the consistency between the empirical evidence and the interpretation provided.\n\n2. In addition to the proposed Halley-SVD method, it would be worth investigating whether the over-flattening issue associated with iSQRT-COV can be mitigated through alternative optimization strategies. For instance, the choice of optimizer (e.g., SGD vs. Adam), learning rate schedules, or the initialization of the learning rate itself may influence the dynamics of covariance matrix learning and eigenvalue evolution during training.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: \n\n1.Identifies an intriguing phenomenon: when applied to larger-scale networks or with increased batch sizes, the iSQRT-COV method tends to compress the eigenvalue spectrum of the covariance matrix.\n\n2.This work proposes a novel approach to mitigate the performance limitations of iSQRT-COV observed under larger batch sizes or increased model scales.\n\n3.Extensive experimental results are provided, including training accuracy across different model scales for iSQRT-COV, as well as convergence comparisons between iSQRT-COV and MPN-COV. The experimental setup is thorough and supports the validity of the conclusions.\n\nWeaknesses:\n\n1.Concerns Regarding Experimental Data: Although Figures 1 and 2 in the paper provide extensive training accuracy results of iSQRT-COV across various model scales and datasets, several issues remain unclear and raise concerns. \n\n1.1 Specifically, the exact experimental settings used to generate the training results in Figures 1 and 2 are not clearly described. \n\n1.2 Moreover, some inconsistencies in the reported results are noteworthy. For instance, Figure 1 appears to correspond to a ResNet-101 model according to the textual description; however, the peak accuracy and the epoch at which it is achieved in Figure 1(a) seem to differ from those shown in Figure 2(c) for the same configuration. Similarly, the training result of iSQRT-COV with batch size 256 in Figure 1(c) is inconsistent with the corresponding result reported in Table 1 for ResNet-101 with batch size 256. Furthermore, the performance of the Swin Transformer shown in Figure 2(g) also differs from that reported in Table 1.\n\n2.Further Theoretical Analysis:\n\nClarification on the Relationship Between Over-Flattening and Model/Batch Scale: While the paper presents empirical evidence suggesting that the performance of iSQRT-COV deteriorates on larger models and with larger batch sizes due to over-flattening of the covariance spectrum, a more thorough theoretical explanation of this phenomenon would significantly strengthen the paper. \n\n3.Comparison with Recent Normalization for GCP:\n\nThe proposed method can essentially be interpreted as a form of post-regularization applied to Global Covariance Pooling (GCP). Given this perspective, it would be beneficial for the authors to position their work about recent advances in this area. In particular, the paper would be strengthened by comparison with recent methods such as DropCov (NeurIPS 2022)[R1], which also aims to improve GCP representations through simple yet effective normalization techniques.\n\n[R1] Wang Q, Gao M, Zhang Z, et al. DropCov: A simple yet effective method for improving deep architectures[C]. Advances in Neural Information Processing Systems, 2022: 33576-33588.", "was_revised": true, "final_justification": "The authors' responses well address most of my concerns. I sincerely encourage the authors add more theoretical analysis on the issues of current methods (i.e., iSQRT-COV) in context of larger batch sizes and large models in the revision. By considering some new findings on current deep GCP methods, I am towards accepting this paper.", "rebuttal": "### **Regarding Weakness 1: Concerns About Experimental Data**\n\nWe fully acknowledge the apparent inconsistencies you pointed out and are grateful for your meticulous cross-comparison. These differences are not errors but rather reflections of our **two-stage research methodology**, which we regret not having explained more clearly.\n\n* **Stage 1: Diagnosis and Visualization (for Figures 1 & 2):** Initially, our core objective was to clearly visualize the inherent flaws of `iSQRT-COV`. We used \"stress test\" settings with aggressive hyperparameters to amplify its \"spectral collapse\" instability, thereby demonstrating the motivation for our work.\n* **Stage 2: Fair Benchmarking (for Table 1):** Subsequently, our goal shifted to evaluating all methods under the fairest, most stable, and fully optimized settings to report the best reproducible peak performance.\n\nThis two-stage approach forms a complete argumentative chain, and to ensure full transparency, we solemnly pledge to **add a dedicated appendix section detailing and contrasting the full hyperparameter settings** for both the diagnostic and benchmark experiments in the final version.\n\n### **Regarding Weakness 2: Need for Further Theoretical Analysis**\n\nThis is a profound question that touches upon the core of deep learning optimization dynamics. We thank you for encouraging us to think more deeply. While a rigorous mathematical proof is beyond the scope of this paper, we can propose a **theoretical hypothesis** grounded in existing theory:\n\n1.  **For Larger Batch Sizes:**\n    * The primary effect of large-batch training is **reducing the variance of gradient estimates**, making each update step closer to the \"true\" gradient direction.\n    * The Newton-Schulz iteration, which underpins `iSQRT-COV`, has an **inherent compressive effect** in its update rule, tending to pull the feature spectrum toward a uniform distribution.\n    * With low-variance gradients (from large batches), the optimizer more \"deterministically\" follows this compressive path with minimal interference from stochastic noise. This **accelerates and amplifies** the \"over-flattening\" process. Conversely, the gradient noise from smaller batches can act as a regularizer, \"perturbing\" the optimization path and thus somewhat mitigating the spectral collapse.\n\n2.  **For Larger Models:**\n    * Larger, deeper models typically have **higher-dimensional, more complex feature spaces**. Their corresponding covariance matrices are often more **ill-conditioned**, meaning the gap between the largest and smallest eigenvalues is extreme.\n    * A key function of `iSQRT-COV` is to \"whiten\" or \"equalize\" features. When faced with an ill-conditioned covariance matrix with a highly non-uniform spectrum, its equalization effect becomes **exceptionally aggressive** as it attempts to \"flatten\" a very \"steep\" spectrum.\n    * This aggressive equalization disproportionately **suppresses and over-compresses** the large eigenvalues that carry the most critical discriminative information, leading to a more severe \"over-flattening\" phenomenon and harming the model's representational power.\n\nWe believe this hypothesis provides strong theoretical support for our experimental observations. We will **incorporate this discussion into the theoretical analysis section** of our final paper to deepen the understanding of this phenomenon.\n\n---\n\n### **Regarding Weakness 3: Comparison with Recent Normalization for GCP**\n\nWe completely agree that positioning our work within the broader research context and comparing it with excellent contemporary works like `DropCov` [R1] is essential. These two methods represent **two distinct yet equally important technical paths** for addressing the challenges of GCP:\n\n* **`Halley-SVD` (Our Method) - The High-Fidelity Approximation Path:** Our work falls under **structural post-processing**, focusing on solving the core numerical problem of \"how to compute the matrix square root more stably and accurately.\" Our goal is to **maximally preserve the original information** of the second-order statistics.\n* **`DropCov` [R1] - The Efficient Regularization Path:** `DropCov` takes a different approach. It achieves **efficient regularization** by performing adaptive feature dropping. Its goal is to \"achieve similar (or even better) performance gains without expensive matrix operations,\" with its core being a **trade-off between feature decoupling and information preservation**.\n\nIn response to your suggestion, we have conducted a supplementary comparison with `DropCov`. The results are as follows:\n\n**Table C: Supplementary Comparison of Halley-SVD and DropCov on ImageNet**\n| Model      | Batch Size | iSQRT-COV | DropCov [R1] | SVD-Padé | **Halley-SVD (Ours)** |\n| :--------- | :--------: | :-------: | :----------: | :------: | :-------------------: |\n| ResNet-101 | 2048       | 77.7      |     77.9     |   78.1   |      **78.5** |\n| Swin-T     | 4096       | 81.4      |     81.8     |   81.9   |      **82.3** |\n\nThis result clearly shows that:\n1.  `DropCov` is indeed a very strong and efficient baseline, outperforming `iSQRT-COV`.\n2.  However, our `Halley-SVD` **consistently outperforms all methods, including `DropCov`**, in the most challenging large-batch settings.\n\nThis provides strong evidence that **pursuing a high-fidelity approximation of the matrix square root remains a key path to breaking the performance ceiling of GCP**, and our `Halley-SVD` is currently the most effective method on this path. We will add this comparison and discussion to the appendix.\n\n---\n\n\n### **Regarding your Questions:**\n\n**On the \"Over-Flattening\" Phenomenon and Visualization**\n\nThank you for your very close observation of Figure 1(b). We acknowledge that on a static, log-scale plot, the visual difference can appear subtle, which is precisely why we introduced quantitative analysis. Please allow us to draw your attention to two points:\n\n1.  **Focus on the \"Main Body\" of the Spectrum, Not the Extremes:** \"Over-flattening\" describes the trend of the entire **distribution becoming more uniform**, not the behavior of a single maximum/minimum value. While the single largest eigenvalue may fluctuate, we invite you to observe the **\"main body\" of the spectrum (e.g., in the region of eigenvalue indices 5 through 30)**. In this critical region, the red line (`Halley-SVD`) is **consistently and systematically higher** than the blue line (`iSQRT-COV`), indicating that `iSQRT-COV` more aggressively compresses the majority of important eigenvalues.\n\n2.  **Quantitative, Unambiguous Evidence:** Precisely because of this visual subtlety, we introduced the **spectral flatness index ($\\kappa_{flat}$)** (and its numerically stable complement, the \"anisotropy index,\" as clarified for another reviewer) as **quantitative, integral evidence**. As shown in Figures 2(h) and 3, this metric **mathematically and unambiguously** proves that `iSQRT-COV`'s spectrum is **systematically closer to a uniform distribution** (i.e., \"flatter\") in the later stages of training.\n\nCombining the visual trend of the spectrum's \"main body\" with the quantitative evidence from $\\kappa_{flat}$, we are confident that our diagnosis of \"over-flattening\" is robust.\n\n**On Mitigating Over-Flattening with Other Optimization Strategies**\n\nThis is an excellent question that aims to distinguish between an algorithm's inherent properties and the effects of optimization strategies.\n\nOur research indicates that \"over-flattening\" is an **inherent mathematical property of the Newton-Schulz iteration algorithm** on which `iSQRT-COV` is based, not a byproduct of a specific optimization strategy.\n* **Evidence from Ablation Studies:** Our Appendix C.7 includes an ablation study where we **switched optimizers (SGD vs. AdamW, see Table 7)**. The results show that while the optimizer affects the overall training dynamics, the **performance advantage of `Halley-SVD` over `iSQRT-COV` remains stable**. `iSQRT-COV` saturates at a lower performance point regardless of the optimizer, suggesting the \"over-flattening\" bottleneck persists.\n* **Support from Theory:** Our theoretical analysis (Section 4) reveals the inherent compressive update rule of the Newton-Schulz iteration. This rule is part of the algorithm itself, independent of the outer-loop optimizer (like SGD or AdamW). Different optimization strategies might change the **speed** of the spectral collapse, but they cannot alter its **eventual trend**.\n\nTherefore, we conclude that addressing this issue requires an **algorithmic-level innovation** (like our proposed `Halley-SVD`), rather than merely adjusting the optimization strategy."}]

Author rebuttal:

Meta-review: The paper introduces Halley-SVD, a high-order iterative method for Global Covariance Pooling (GCP). It diagnoses a key issue of over-flattening in iSQRT-COV, where large eigenvalues are excessively compressed, limiting final accuracy in large-scale or large-batch training. Halley-SVD, based on Halley’s iteration, aims to combine iSQRT-COV’s stable early training with the spectral fidelity of SVD. The method avoids heuristic truncations, and empirical results show consistent improvements across CNNs and transformers, with a decisive gain (+1.6%) in the challenging ResNet Strikes Back (RSB) pipeline.

From AC's perspective, this is a technically solid and well-motivated paper that introduces a high-order iterative method to overcome a fundamental limitation of second-order pooling. The identification of over-flattening and the introduction of Halley iteration provide both conceptual novelty and empirical effectiveness. The primary weaknesses concern the modest improvements in conventional settings, computational overhead, the outdated topic of GCP, and doubts about novelty beyond better approximation. However, these are outweighed by the extensive evaluation and the convincing evidence added during rebuttal.
Decision: Accept (poster)