A Coefficient Makes SVRG Effective
Authors: Yida Yin
Zhiqiu Xu
Zhiyuan Li
Trevor Darrell
Zhuang Liu
Venue: ICLR 2025 Poster (2025)
Keywords: Optimization; Variance Reduction; SGD
Primary area: optimization
TL;DR: Introducing a coefficient to control the variance reduction strength in SVRG makes it effective for deep networks.

Abstract: Stochastic Variance Reduced Gradient (SVRG), introduced by Johnson & Zhang (2013), is a theoretically compelling optimization method. However, as Defazio & Bottou (2019) highlight, its effectiveness in deep learning is yet to be proven. In this work, we demonstrate the potential of SVRG in optimizing real-world neural networks. Our empirical analysis finds that, for deeper neural networks, the strength of the variance reduction term in SVRG should be smaller and decrease as training progresses. Inspired by this, we introduce a multiplicative coefficient $\alpha$ to control the strength and adjust it through a linear decay schedule. We name our method $\alpha$-SVRG. Our results show $\alpha$-SVRG better optimizes models, consistently reducing training loss compared to the baseline and standard SVRG across various model architectures and multiple image classification datasets. We hope our findings encourage further exploration into variance reduction techniques in deep learning. Code is available at github.com/davidyyd/alpha-SVRG.

Published as a conference paper at ICLR 2025
A COEFFICIENT MAKES SVRG EFFECTIVE
Yida Yin"
Zhiqiu Xu²
Zhiyuan Lix
Trevor Darrell
Zhuang Liu
l UC Berkeley University of Pennsylvania
*TTIC
"Meta Al Research
ABSTRACT
Stochastic Variance Reduced Gradient (SVRG), introduced by Johnson Zhang (2013), is theoretically compelling optimization method. However, Defazio & Bottou (2019) highlight, its effectiveness in deep learning is yet to be proven. In this work, we demonstrate the potential of SVRG in optimizing real-world neural networks. Our empirical analysis finds that, for deeper neural networks, the strength of the variance reduction term in SVRG should be smaller and decrease as training progresses. Inspired by this, we introduce a multiplicative coefficient a to control the strength and adjust through linear decay schedule. We name our method a- SVRG. Our results show a-SVRG better optimizes models, consistently reducing training loss compared to the baseline and standard SVRG across various model architectures and multiple image classification datasets. We hope our findings encourage further exploration into variance reduction techniques in deep learning. Code is available at g thub. com/ dav dyyd/allypppaaoo -SVRG.
INTRODUCTION
A decade ago, Johnson & Zhang (2013) proposed a simple approach for reducing gradient variance in SGD-Stochase Variance Reduced Gradient (SVRG). SVRG keeps a snapshot model and uses to form a variance reduction term to adjust the gradient of the current model. This variance reduction term is the difference between the snapshot's stochastic gradient and its full gradient on the whole dataset. Utilizing this term, SVRG can reduce gradient variance of SGD and accelerate it to almost as fast as the full-batch gradient descent in strongly convex settings.
Over the years, numerous SVRG variants have emerged. Some focus on further accelerating con- vergence in convex settings (Xiao & Zhang, 2014; Lin et al., 2015; Defazio, 2016), while others are tailored for non-convex scenarios (Allen-Zhu & Hazan, 2016; Reddi et al., 2016; Lei et al., 2017; Fang et al., 2018). SVRG and its variants have shown effectiveness in optimizing simple machine learning models like logistic regression and ridge regression (Allen-Zhu, 2017; Lei et al., 2017).
Despite the theoretical value of SVRG and its subsequent works, they have seen limited practical success in training neural networks. Most SVRG research in non-convex settings is restricted to modest experiments: training basic models like Multi-Layer Perceptrons (MLP) or simple CNNs on small datasets like MNIST and CIFAR-10. These studies usually exclude evaluations on more capable and deeper networks. More recently, Defazio & Bottou (2019) have exploited several variance reduction methods, including SVRG, deep vision models. They found that SVRG fails reduce gradient variance for deep neural networks because the model updates so quickly on the loss surface that the snapshot model becomes outdated and ineffective at variance reduction.
In this work, we show that adding a multiplicative coefficient to SVRG's 's variance reduction term can make it effective for deep neural networks. Our exploration is motivated by an intriguing observation: SVRG can only reduce gradient variance in the initial training stages but increases it later. To tackle this problem, we mathematically derive the optimal coefficient for the variance reduction term to minimize the gradient variance. Our empirical analysis then leads to two key observations about this optimal coefficient: (1) as the depth of the model increases, the optimal coefficient becomes smaller; (2) as training advances, the optimal coefficient decreases, dropping well below the default coefficient of 1 in SVRG. These findings help explain why a constant coefficient of in SVRG, while initially effective, eventually fails to reduce gradient variance.
Based on these observations, we introduce a linearly decaying coefficient a to control the strength of the variance reduction term in SVRG. We call our method a-SVRG and illustrate it in Figure 1.

---

Published as a conference paper at ICLR 2025
loss
high
low
(a) SVRG
(b) a-SRGG
snapshot model
snapshot stochastic gradient
snapshot full gradient
variance reduction
model stochastic gradient
variance reduced gradient
Figure 1: SVRG vs. a-SVRG. Both SVRG (left) and a-SVRG (right) the difference between snapshot stochastic gradient (gray) and snapshot full gradient (blue) to form variance reduction term (orange), which modifies model stochastic gradient (black) into variance reduced gradient (red). But a-SVRG employs a coefficient a to modulate the strength of the variance reduction term. With this coefficient, a-SVRG reduces the gradient variance and results in faster convergence.
n-SVRG decreases gradient variance stably across both early and late training and helps optimize the model better. We evaluate a-SVRG range of architectures and image classification datasets. a-SVRG achieves a lower training loss than the baseline and the standard SVRG. Our results highlight the value of SVRG in deep learning. We hope work offer insights about SVRG and stimulate more research in variance reduction approaches for optimization in neural networks.
MOTIVATION: SVRG MAY NOT ALWAYS REDUCE VARIANCE
SVRG formulation. We first introduce the basic formulation SVRG. We adopt the following notation: Ć is the iteration index, 0f represents the current model parameters, and ff((-) denotes the gradient of loss function f for the i-th mini- bati. In SVRG's original work (Johnson & Zhang, 2013), this corresponds to the i-th data point. When the subscript is omitted, ff(-) represents the full gradient across the entire dataset. A key concept in SVRG is the snapshot model, represented as @rest It is snapshot of the model previous iteration before l. We store its full gradient V (fPest). This snapshot is taken periodically. SVRG defines the variance reduced gradient g as follows:
(.')
(e f ((
variance reduction term
Intuitively, SVRG uses the difference between the mini-batch gradient and full gradient of past model modify the current mini-batch gradient. This could make g better aligned with the current full gradient V V f(e) and thus stabilize each update.
SVRG was initially introduced in the context of vanilla SGD. Recent work (Dubois- Taine et al., 2021; Wang & Klabjan, 2022) has integrated SVRG into alternative base optimizers. Following them, we input the variance reduced gradient of into the base optimizer and ensure a fair comparison by using the same base optimizer for SVRG and the baseline. We also follow the practice in Defazio & Bottou (2019), taking snapshot for SVRG once per training epoch.
Gradient variance. Our goal is assess SVRG's effectiveness. in reducing gradient variance. To this end, we gather mini-batch gradients, denoted as {gfli E
by performing back-
name metric 1*
formula
description
NCN Eisr (1 Σd Var(gi..
the directional variance of the gradients
Ialloo
metric 2t
the variance of gradients
each component
metric 3$
Amox( Dot (o) (gt FT)
the magnitude of the most significant variation
Table 1: Metrics. g" the mean of the mini-batch gradients g indexes the k-th component of gradient gixx References: * Liu et al. (2023), t Defazio & Bottou (2019), I Jastrzebski et al. (2020)
2

---

(metric 2) Published as a conference paper at ICLR 2025
propagation checkpoints of the model at the iteration t with randomly selected N mini-batches. For SVRG, each these gradients is modified based on Equation 1. To present a comprehensive view, employ three metries from prior studies quantify gradient variance in Table 1.
In this part, we compare our three metrics. Metric 1 calculates the cosine distance between pairwise mini-batch gradients, therefore only capturing variance in gradient directions rather than gradient magnitudes. This is very important for scale-invariant optimizers, such as Adagrad (Lydia Francis, 2019) and Adam (Kingma & Ba, 2015). In contrast, metric focuses both gradient directions and magnitudes by summing the variance of each component of gradients. This metric has been the standard tool measure gradient variance in various optimization literature (Allen-Zht 2016; Defazio & Bottou, 2019). Metric 3 considers the largest eigenvalue in gradient covariance matrix, characterizing the most dominant part in gradient variance. We also average each gradient variance metric across three runs, with shaded regions in figures representing the standard deviation.
Hazan,
SVRG's effect on gradient variance. To understand how SVRG affects training, we examine two simple models: a linear layer (Logistic Regression) and 4-layer Multi-Layer Perceptron (MLP-4). train them over 30 epochs on CIFAR-10. We compare SVRG to a baseline using only SGD,
grad. var. (metric 1)
0.45
grad. var. (metric 2)
0.30
0.15
2
0.00
5
10
15 epochs
20
25
30
5
10
15 epochs
20
25
30
0.45
grad. (metric
SGD
0.30
train loss
SVRG
0.15
0.00
1.6
10
20
255
30
5
10
15 epachs
20
25
30
epochs
Figure 2: SVRG Logistic Regression. SVRG effectively reduces the gradient variance for Logistic Regression, leading to a lower training loss than the baseline.
We plot Logistic Regression's 's gradient variance (top two and bottom left) and training loss (bottom right) in Figure 2. For Logistic Regression, SVRG can reduce the gradient variance throughout the entire training process and achieve a lower training loss than the baseline.
grad, var. (metric 1)
0.45
0.30
0.15
0.00
0
5
10
15 epochs
25
30
10
20
25
30
epochs
0.48
grad, (metric
0.32
train loss
SVRG
0.16
0.8
0.00
0.2
10
20
25
30
5
10
20
25
30
epochs
epochs
Figure 3: SVRG on MLP-4. In the first few epochs. SVRG reduces the gradient variance for MLP-4, but afterward, SVRG increases it, well above the baseline, As result, SVRG exhibits a higher training loss than the baseline the end of training.
3

---

Published as a conference paper at ICLR 2025
In contrast, for MLP-4, SVRG may not always reduce gradient variance. As shown in Figure 3, SVRG can only decrease the gradient variance for the first five epochs but then increases it. Consequently, SVRG has a larger final training loss than the baseline. This indicates that the increase in gradient variance caused by SVRG hinders the convergence of MI.P-4's training loss.
This surprising empirical observation in slightly deeper model leads to question whether SVRG may alter the gradient too excessively at certain phases training. Can we mitigate this adverse effect? We explore these questions starting from a theoretical framework in the next section.
A CLOSER LOOK AT CONTROL VARIATES IN SVRG
Control variates (Lavenberg et al., 1977) is technique initially developed in Monte Carlo methods to reduce variance. We aim to estimate the expected value of random variable X. The variance of this estimate usually depends on Var(X). To form less variate estimate X variate Y that correlates with X and a coefficient ( to regulate the influence of Y and EE[]
use control
X = X a(Y B[Y]).
This estimate remains unbiased for any value of a. The coefficient that minimizes the variance of the estimate can be derived as:
Cov(X, Y) a" = Var(Y)
p(X, Y) (x)
o(Y)'
where pXX, Y) represents the correlation coefficient between X and Y; a--) denotes the standard deviation. The derivation is detailed in Appendix A. The minimized variance becomes Var(X") (1 - p(X, Var).. The higher the correlation is, the lower the variance of the estimate is.
Note that SVRG uses control variates to reduce variance in each component of the gradient. This variance reduction occurs each iteration t. Take closer look at Equation and 2: the model stochastic gradient fe(O') is the random variable X; the snapshot stochastic gradient (=(Pa))) is the control variate and the snapshot full gradient f f@rw)) is the expectation EY
A key difference between SVRG and control variates is that SVRG omits the coefficient a, defaulting 1. This is possibly because the gradient distribution does not change drastically in strongly convex settings (Johnson & Zhang, 2013). Yet, SVRG's subsequent studies, even those addressing non-convex cases, have neglected the coefficient and formulated their theories based on Equation
Motivated by this, we introduce a time-dependent coefficient vector a' E IR in SVRG:
g V (('))) a' (V)i( (((P))))
where c represents the clement-wise multiplication.
Optimal coefficient. We adopt the same gradient variance definition as Defazio & Bottou (2019) (metric 2 in Table 1) and aim to determine the optimal a that minimizes at each iteration. Specifically, objective is to minimize the sum of variances across each component of g Let k index the k-thh component ot and the k-th component of the gradient f.). For clarity, we omit the minc-batch index i. This be formally expressed as follows:
Var(g",) = Σ min Var(g",)) k=1 of
min
(5)
We can switch the order of minimization and summation in Equation because the variance of the k-th component of the gradient only depends on the k-th component of the coefficient. Applying Equation yields the optimal coefficient of:
Cov CCvvVV_.e (OPma))
((VS..(0')
0
p(Vf (f (f.)(O))) (.
(6)
Var( f. (mstt)
A stronger correlation between the snapshot and model gradients leads to a larger optimal coefficient.
For small networks like MLP-4, calculating the optimal coefficient each iteration is feasible by gathering all mini-batch gradients for both the current and snapshot models. For larger networks, however, this method becomes impractical; we will address this challenge later in the paper.
4

---

Published as a conference paper at ICLR 2025
optimal coefficient
1.0
optimal coefficient
1.0
0.7
0.7
Logistic Regression A
0.4
0.4
MLP-2
MLP-4
0.1
0.1
5
6
7
10
8
9
10
epochs
epochs
(a) SGD
(b) Adam
Figure 4: Optimal coefficient. At the start of each epoch, a snapshot is taken. Consequently, the optimal coefficient initiates value of and results in periodic upward jump.
Observations on optimal coefficient. To explore how the optimal coefficient evolves in normal training setting, we train 2, and 4-layer MLPs (Logistic Regression, MLP-2, and MLP-4) using SGD and AdamW (Loshchilov & Hutter, 2019) on CIFAR-10 without using SVRG. Given the small size of these models, we can analytically compute the optimal coefficient at each iteration. We plot its mean value over all indices k in Figure 4. We can make two notable observations as below.
Observation I: a deeper model has a smaller optimal coefficient. For Logistic Regression, the optimal coefficient remains relatively stable, hovering near 1. For MLP-2, the coefficient deviates from 1, dropping to about 0.6. For MLP-4, it decreases more sharply, reaching approximately 0.4.
Observation 2: the average optimal coefficient of a deeper model in each epoch generally decreases training progresses. This suggests that each epoch's average correlation between snapshot gradients and model gradients (p( V f (OPmt)) ffx(((')) in Equation 6) decreases as the model becomes better trained. We further analyze this decreasing trend of the correlation term in Appendix D.3.
These observations shed light why the standard SVRG struggles to reduce gradient variance or training loss in later training stages (Figure 3). A default coefficient of proves to be too high, and the weakening correlation between snapshot and model gradients necessitates smaller coefficient. Without a suitable coefficient, gradient variance may increase, leading to oscillations SGD.
Optimal coefficient's effect gradient variance. We evaluate whether optimal coefficient make SVRG more effective in reducing gradient variance. Specifically, we use SVRG with optimal coefficient train MLP-4 by computing optimal coefficient (Equation 6) and adjusting the gradient (Equation 4) at each iteration. In Figure 5, we compare SVRG with optimal coefficient to an SGD baseline. Using the optimal coefficient enables SVRG reduce gradient variance the early stages of training without uplifting it later. This yields consistently lower training loss than the baseline.
grad. var. (metric 1)
0.45
var. (metric 2)
0.30
0.15
0.00
0
5
10
15 epochs
20
23
30
0
5
10
15 epochs
20
25
30
grad. var. (metric 3)
0.48
2.0
SGD
0.32
train loss
1.4
SVRG (opt. coef.)
0.16
0.00
5
10
15
20
25
30
10
15 epochs
20
25
30
epochs
Figure 5: SVRG with optimal coefficient on MLP-4, SVRG with the optimal coefficient reduces gradient variance stably and achieves lower training loss than the baseline SGD.
5

---

Published as a conference paper at ICLR 2025
4 a-SVRG
From our analysis above, it becomes clear that the best coefficient for SVRG is not necessarily 1 for deep neural networks. However, computing the optimal coefficient at each iteration would result in complexity of full-batch gradient descent. This approach quickly becomes impractical for larger networks like ResNet (He al., 2016). In this section, we show how using a preset schedule of values can achieve similar effect of using the computed optimal coefficients.
Q-SVRG. Given the decreasing trend (Figure 4) and the computational challenge, we propose to apply linearly decreasing scalar coefficient for SVRG, starting from initial value ag and decreasing to This our main method in this paper. We name a-SVRG, More results of enabling a-SVRG only during the early stage of training in Appendix C.4. The pseudocode for a-SVRG with SGD and Adam as base optimizers is provided Appendix G.
grad. var. (metric 1)
0.45
var. (metric 2)
0.30
0.15
0.00
0
10
20
25
30
0
10
15 epochs
20
25
30
epochs
grad. var. (metric 3)
0.48
2.0
SVRG (opt. coef.)
0.32
train loss
1.4
a-SVRG
0.16
0.00
0.2
$
10
1.5
25
30
10
15
20
25
30
epochs
epochs
Figure 6: a-SVRG on MLP-4, Q-SVRG behaves similarly to SVRG with optimal coefficient.
To evaluate how well a-SVRG matches SVRG with optimal coefficient, we train an MI.P-4 using o-SVRG and compare it SVRG with optimal coefficient. For all experiments in this section, set ag = 0.5. The results are presented in Figure 6. Interestingly, a-SVRG exhibits gradient variance trend that not much different from SVRG with optimal coefficient. Similarly, the training loss of a-SVRG is only marginally larger than that of SVRG with optimal coefficient.
0.38
grad. var. (metric 2)
grad. var. (metric
0.28
2.3
0.18
1.3
0.08
0.3
1'0
is epochs
20
2'5
5
10
20
2'5
30
epochs
grad, var. (metric 3)
0.30
1.7
AdamW
0.21
train loss
1.2
e-SVRG
0.12
0.03
5
10
15
20
2'5
30
10
15
20
25
30
epochs
epochs
Figure 7: a-SVRG with Adam on MLP-4. a-SVRG can lower the gradient variance at the first 10 epochs, leading faster convergence than the baseline AdamW.

---

Published as a conference paper at ICLR 2025
a-SVRG with AdamW. Since AdamW (Loshchilov & Hutter, 2019) is widely used optimizer in modern neural network training, we assess the performance of a-SVRG with AdamW. We change the base optimizer in a-SVRG to AdamW and use it to train MLP-4 a-SVRG to a baseline using only AdamW. W. As shown in Figure 7, a-SVRG has a noticeable gradient variance reduction initially and achieves consistent lower training loss for MLP-4 than the baseline.
CIFAR-10. We compare
a-SVRG on deeper networks. We further study the effectiveness of a-SVRG with A on real- world neural architectures, moving beyond simple MLPs. To this end, we train modern ConvNet architecture, Con-NeXi-Feemto (Liu et al., 2022; Wightman, 2019), on CIFAR- 10 using the default Adam W optimizer. We compare a-SVRG the baseline using vanilla Adam W Figure a-SVRG can reduce gradient variance during the first 10 epochs (zoom-in plot of Figure 8) and then maintain it at the same level as the baseline. As result, the training loss of a-SVRG converges much faster than the baseline. This demonstrates the potential of aVSV in optimizing more complex models. We further explore this with additional experiments next.
grad, var. (metric 1)
0.48
grad. var. (metric 2)
0.32
3.6
0.16
1.8
0.00
0.0
50
100
150
200
250
300
50
100
200
250
300
epochs
epochs
grad. var. (metric 3)
1.5
AdamW
1.0
train loss
2.0
+ cVSR
0.5
1.7
0.0
1.4
50
100
150
200
250
300
50
100
150
200
250
300
epochs
epochs
Figure 8: a-SVRG on ConwNeXt-Femto. a-SVRG can reduce the gradient variance for Cen Femto during the first 10 epochs (zoom-in plot) without increasing it later on. Consequently, a-SVRG can decrease the training loss at faster rate than the baseline AdamW.
EXPERIMENTS
5.1 SETTINGS
Datasets. We evaluate a-SVRG using ImageNet-IKK classification (Deng et al., 2009) as well as smaller image classification datasets: CIFAR-100 (Krizhevsky, 2009), Pets (Parkhi et al., 2012), Flowers (Nilsback& & Zisserman, 2008), STL-10 (Coates et al., 201 Food-101 (Bossard et al., 2014), DTD (Cimpoi al., 2014), SVHN (Netzer al., 20,1), and EuroSAT (Helber al., 2019).
Models. We use recently proposed vision models on ImageNet- IK, categorized into two groups: (1) smaller models with 5-19M parameters, including Co-FNeXt-F (Wightman, 2019; Liu al., 2022), VIT-T/16 (Dosovitskiy et al., 2021), Swin-F (Liu et al., 2021b), and Mixer-S/32 (Tolstikhin al., 2021); (2) larger models featuring 86M and 89M parameters: ViT-B/16 and Con ConvNeXt-Fi is also evaluated all smaller image classification datasets.
Training. We report both final epoch training loss and top-1 validation accuracy. Our basic training setting follows ConvNeXt (Liu et al., 2022), which uses AdamW. Both SVRG and a-SVRG also use Adam as the base optimizer. On small datasets, we choose the best Qq from (0.5, 0.75, 1). We find the coefficient is robust and does not require extensive tuning. Therefore, for ImageNet-1K, we set ag to 0.75 for smaller models and 0.5 for larger ones. Other training settings for a- aVSV remain the same as the baseline. Further experimental settings can be found in Appendix B.
7

---

Published as a conference paper at ICLR 2025
ConvNeXt-F
VIT-T
Swin-F
Mixer-S
VIT-B
ConvNeXt-B
training loss
AdamW
3,487
3.443
3.427
3.149
2.817
2.644
+ SVRG
3,505
1.018
3,431
4.012
3.389
4.038
3.172
1.023
3.309
1.492
3.113
1.469
+--SVRG
3,467
4.020
3.415
4.028
3.392
4.035
3.097
J.052
2.806
4 4.011
2.642
4.002
validation accuracy
AdamW 76.0
73.9
74.3
76.4
81.6
83.7
SVRG
75.7
4.0.3
74.3
10.4
74.3
1..0
74.5
1.9
78.0
03.6
80.8
2.9
+ +--SVRG
76.3
0.3
74.2
0.3
74.8
0.55
76.1
J.0.3
81.6
0.0
83.1
0.6.
Table 2: Results on ImageNet- The standard SVRG increases the training loss for most models, whereas a-SVRG consistently decreases it for all models.
CIFAR-I00
Pets
Flowers
STL-10
Food-101
DTD
SVHN
EuroSAT
training loss
AdamW
2.66
2.20
2.40
1.64
2.45
1.98
1.59
1.25
SVRR
2.94
0.22
3.42
1 1.22
2.26
00.14 1.90 1 0.26 3.03
0.58
2.01
1..03
1.64
10.05
1.25
0.00
a-SVRG
2.62 0.004
1.96
0.224
2.16
0.24 1.57 0.07 2.42 0.03
1.83
00.15
1.57
0.00
1.23
0.02
validation accuracy
AdamW 81.0
72.8
80.8
82.3
85.9
57.9
94.9
98.1
SVVG
78.2
22.8
17.6
55.2
82.6
1.8 65.1
17.2 79.6
6.3
57.8
40.1
95.7
0.8
97.9
J02
a-SVRG 81.4
0.4
77.8
5.00
83.3
2.5 84.0
1.7
85.9
0.0
61.8
139
95.8
0.9 98.2 00.
Table 3: Results on smaller classification datasets. While the standard SVRG mostly hurts the performance, a-SVRG decreases the training loss and increases the validation accuracy.
5.2 RESULTS
Table 2 presents the results of training various models on ImageNet-IK, The standard SVRG often increases the training loss, especially for larger models. In contrast, a-SVRG decreases the training loss for both smaller and larger models. This also supports our earlier finding that deeper models benefit from lower coefficient values, and using a default coefficient of impedes convergence.
Table 3 displays the results of training ConvNeXt-F on various smaller datasets. The standard SVRG generally elevates the training loss and impairs the generalization. On the contrary, a-SVRG lowers the training loss and improves the validation accuracy across all small datasets. We have provided additional experiment results to demonstrate a-SVR''s effectiveness Appendix C.
Note that lower training loss in a-SVRG does not always lead to better generalization. For smaller models, lower training loss usually directly translates to higher validation accuracy. larger models (Mixer-S, ViT-B, and ConvNeXt-B), additional adjustments to regularization strength may be needed for better generalization. This is out scope for a-SVRG as an optimization method but warrants future research co-adapting optimization and regularization.
5.3 ANALYSIS
We analyze various components in a-SVRG. In the following experiments, use initial value ag = 0.5 and ConvNeXi-F on STL-10 as the default setting. Because the standard SVRG is ineffective as discussed above, omit it and only compare a-SVRG to AdamW baseline.
Coefficient value. We investigate the impact of the initial value of the coefficient au for aVRG We vary it between 0 and 1 and observe its effect on the training loss. The results presented in Figure 9. The favorable range for initial values in a-SVRG quite broad, ranging from 0.2 to 0.9. This robustness indicates o-SVRG requires minimal tuning in the practical setting.
8

---

Published as a conference paper at ICLR 2025
1.65
1.64
1.63
AdamW a-SVRG
2.00
1,94
AdamW
train loss
1.62
1.62
train loss
1.85
a-SVRG
1,59
1.59
1.59
1.69
1,74
1,71
1.59
1.58
1.70
1.64
1,62
56
1.56
1,58
1.53
0.1
0.5
128 batch size
512
coefficient
Figure 9: Coefficient value. a-SVRG is effec-
Figure 10: Batch size. a-SVRG's effectiveness is observed for larger batch sizes.
tive with wide range of coefficient values.
Batch size. Since the batch size controls the variance among mini-batch data, we change the batch size to understand how it affects a-SVRG. We also scale the learning rate linearly (Goyal et al., 2017). The default batch size is 128. In Figure 10, can see that a-SVRG leads lower training loss when the batch size is larger, but it is worse than the baseline when the batch size is smaller. This may stem from the weakening correlation between snapshot gradients and model gradients as the batch size decreases. Therefore, a sufficiently large batch size is essential for a-SVRG.
Coefficient schedule. By default, our a-SVRG uses a linearly decreasing schedule adjust the coefficient. Below we explore other schedules and illustrate them in Figure 1.. Global schedules only decay the coefficient acmoss epochs and keep as a constant within an epoch. In contrast, double schedules also model the local decay in each epoch (Figure 4) by initiating the coefficient at and decreasing to an ending value specified by the global decay. More details on them are in Appendix
1.0
coeffccient
linear
coefficient
0.7
quadratic geometric
0.7
d-linear
0.4
d-quadratic
d-geometric
0.1
0.1
2
3
5
8
9
10
8
9
10
epoch
epoch
(a) global schedules
(b) double schedules
Figure 11: Different coefficient schedules with ao = 0.75. Each global schedule (left) maintains a static coefficient within epoch and applies a coefficient decay only at the end of each epoch. In contrast, each double schedule (right) also adjusts the coefficient within an epoch.
Table presents the results of a-SVRG using each schedule. a-SVRG with double schedules surprisingly have higher training loss than the AdamW baseline (1.64). This is possibly because the coefficient within an epoch sometimes overestimates the optimal coefficient and therefore increases gradient variance. In contrast, a-SVRG with global schedules consistently achieves a lower training loss than the baseline (1.64) regardless the choice of any initial coefficient.
train loss
linear
quadratic
geometric
d-linear
d-quadratic
d-geometric
<xo 0.5
1.59
1.61
1.62
2.07
1.97
1.81
an 0.75
1.57
1.58
1.58
2.07
2.00
1.93
0u = 1
1.57
1.56
1.57
2.00
1.97
1.88
Table 4: Schedules. Bold indicates the lowest training loss among different schedules using the same initial coefficient (row). a-SVRG with global schedules outperforms that with double schedules.
Inner loop size. The inner loop size specifies the number of iterations between two consecutive snapshot captures. We vary it from to 312 iterations to understand its effect on a-SVRG. The default value is 39 iterations (one epoch). Figure 12 illustrates a-SVRG has a lower training loss than the baseline even with larger inner loop size, where the snapshot is relatively distant from the current model. On the other hand, a smaller inner loop size results lower training loss but requires additional training time, full gradient must be calculated each time snapshot is taken.

---

Published as a conference paper at ICLR 2025
1.65
1,58
train loss
1.55
1.55
1.55
1.54
1.48
1.49
1.45
1.43
AdamW
1.37
a-SVRG
1.35
256
inner loop size
Figure 12:
Inner loop size. Although a greater inner loop size leads to a weakening correlation between the model gradients and the snapshot gradients, a-SVRG can still lower training loss.
RELATED WORK
Variance reduction in optimization. There are a range of methods aiming at reducing gradient variance by directly modifying stochastic gradient. Initial works (Johnson & Zhang, 2013; Schmidt et al., 2016) focus on simple convex settings. Subsequent research enhances these methods (Defazio et al., 2014a; Mairal, 2015; Lin et al., 2015; Defazio, 2016; Allen- 2017; Lin et al., 2018) or handles finite sums in non-convex landscapes (Allen-Zhi & Hazan, 2016; Nguyen et al., 2017: Fang al., 2018; Li & Li, 2018; Cutkosky & Orabona, 2019; Elibol et al., 2020; Zhou et al., 2020; Kavis et al., 2022). For these methods, we either need to store all gradient with respect to each individual data point (Defazio et al., 201 4b; Shalev-Shwartz & Zhang, 2013; Li al., 2021) calculate full gradient periodically (Johnson & Zhang, 2013; Fang et al., 2018). Gower et al. (2020) provide comprehensive review for variance reduction methods. While these studies focus on theories of SVRG, we primarily explore the practical utility of SVRG for real-world neural networks.
One of the most relevant works us is MARS (Yuan et al., 2024), which also demonstrates that using a coefficient helps the variance reduction methods optimize modern neural networks. In contrast, our work studies how coefficient makes SVRG effective through step-by-step controlled experiments.
Implicit variance reduction. Apart from methods that explicitly adjust the gradient, there are variance reduction techniques that implicitly reduce gradient variance through other means. A variety of optimizers (Zeiler, 2012: Kingma & Ba, 2015; Dozat, 2016; Lydia & Francis, 2019; Loshchilov Hutter, 2019; Liu et al., 2021 a; 2024; Chen et al., 2023) utilize momentum to mitigate gradient variance. They achieve this by averaging past gradients exponentially, thus stabilizing subsequent updates. Lookahead optimizer (Zhang et al., 2019) reduces gradient variance by only updating the model once every A iterations. Dropout (Hinton et al., 2012) is also found reduce gradient variance and better optimize models when used at early training (Liu et al., 2023).
7 CONCLUSION
Over the past decade, SVRG has been a method with a significant impact on the theory of optimization. In this work, explore the effectiveness of SVRG in training real-world neural networks. Our key insight is the optimal strength for the variance reduction term in SVRG is not necessarily 1. It should be lower for deeper networks and decrease as training advances. This motivates us introduce a-SVRG: applying a linearly decreasing coefficient to SVRG. a-SVRG leads to a steady reduction in gradient variance and optimizes models better. Our experiments show that a-SVRG consistently achieves a lower training loss compared to both bascline and the standard SVRG. Our results motivate further research of variance reduction methods neural networks training.
Acknowledgement. We would like to thank Kaiming He, Aaron Defazio, Zeyuan Allen-Zhu, Kirill Vishniakov, Huijin Ou, Jiayi Xu, Shuyi Wang, and Zekai Wang for valuable discussions and feedback.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Zeyuan Allen-Zhu. Katyusha: The first direct acceleration of stochastic gradient methods. Symposium on Theory of Computing, 2017.
Zeyuan Allen-Zhu and Elad Hazan. Variance reduction for faster non-convex optimization. In ICML, 2016.
Lukas Bossard, Matthicu Guillaumin, and Luc Van Gool, Food-101-ming discriminative compo- nents with random forests. In ECCV, 2014.
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hicu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoe Le. Symbolic discovery optimization algorithms. arXiv preprint arXiv:2302.06655, 2023..
Mircea Cimpoi. Subhransu Maji. lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, 2014.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis single-layer networks in unsupervised feature learning. In AISTATS, 2011.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPR Workshops, 2020.
Ashok Cutkosky and Francesco Orabona. Momentum-basee variance reduction in non-convex sgd. In NeurlPS, 2019.
Alex Damian, Tengyu Ma, and Jason Lee. Label noise sgd provably prefers flat global minimizers. In NeurIPS, 2021.
Aaron Defazio. simple practical accelerated method for finite sums. NeurlPS, 2016.
Aaron Defazio and Lon Bottou. On the ineffectiveness of variance reduced optimization for deep learning. arXiv preprint 1/Xiv:1//2.004529, 2019.
Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. In NeurlPS, 2014a.
Aaron Defazio, Tibrio Caetano, and Justin Domke. Finito: A faster, permutable incremental gradient method for big data problems. In ICMI, 2014b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
Timothy Dozat. Incorporating Nesterov Momentum into Adam. In ICLR, 2016.
Benjamin Dubois-Tain Taine, Sharan Vaswani, Reza Babanezhad, Mark Schmidt, and Simon Lacoste- Julien. Svrg meets adagrad: Painless variance reduction. arXiv preprint arXiv:2102.09644 2021.
Melih Elibol, Lihua Lei, and Michael Jordan. Variance reduction with sparse gradients. In ICLR, 2020.
Cong Fang, Chris Junchi Li, Zhouchen Lin, and Tong Zhang. Spider: Near-optimal non-convex optimization via stochastic path integrated differential estimator. In NewrIPS, 2018.
Robert Gower, Mark Schmidt, Francis Bach, and Peter Richtrik. Variance-reduced methods for machine learning. In IEEF, 2020.
11

---

Published as a conference paper at ICLR 2025
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet in 1 hour. arXiv preprint arXiv:1706.02677 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
Geoffrey Hinton, Nitish Srivastava, Alex Krizhevsky, llya Sutskever, and Ruslan R Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580, 2012.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. ECCV, 2016.
Stanislaw Jastrzebski, Maciej Szymczak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. The break-even point optimization trajectories of deep neural networks. In ICI.R, 2020.
Rie Johnson and Tong Zhang, Accelerating stochastic gradient descent using predictive variance reduction. In NeurIPS, 2013.
Ali Kavis, Stratis Skoulakis, Kimon Antonakopoulos, Leello Tadesse Dadi, and Volkan Cevher. Adaptive stochastic variance reduction for non-convex finite-sum minimization. In NewrIPS, 2022.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Tech Report, 2009.
Stephen Lavenberg, Thomas L Moeller, and Peter D Welch. The application of control variables to the simulation of closed queueing networks. In Proceedings of the 9th conference on Winter simulation-Volume 1, 1977.
Lihua Lei, Cheng Ju, Jianbo Chen, and Michael Jordan. Non-convex finite-sum optimization via scsg methods. In NeurIPS, 2017.
Zhiyuan Li, Tianhao Wang, and Sanjeev Arora. What happens after sgd reaches zero loss? -a mathematical framework. In ICLR, 2022.
Zhize Li and Jian Li. A simple proximal stochastic gradient method for nonsmooth nonconvex optimization. In NearIPS, 2018
Zhize Li, Slavomír Hanzely, and Peter Richtárik. Zerosarah: Efficient nonconvex finite-sum opti- mization with zero full gradient computation. arXiv preprint arXiv:2103.01447, 2021.
Hongzhou Lin, Julien Mairal, and Zaid Harchaoui. A universal catalyst for first-order optimization. In NeurlPS, 2015.
Hongzhou Lin, Julien Mairal, and Zaid Harchaoui. Catalyst acceleration for first-order convex optimization: from theory to practice. JMLR, 2018.
Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-traaee pre-training. In ICLR, 2024.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In ICLR, 2021a.
Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei. Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021b.
Zhuang Liu, Hanzi Mao, Chao Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.
A convnet for the 2020s. In CVPR, 2022.
12

---

Published as a conference paper at ICLR 2025
Zhuang Liu, Zhiqiu Xu, Joseph Jin, Zhiqiang Shen, and Trevor Darrell. Dropout reduces underfitting. In ICML, 2023.
Iyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Agnes Lydia and Sagayaraj Francis. Adagradan optimizer for stochastic gradient descent. JMLR, 2019.
Julien Mairal. Incremental majorization-miniiizaiii optimization with application large-scale machine learning. SIAM Journal on Optimization, 2015.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop, 2011.
Lam Nguyen, Jie Liu, Katya Scheinberg, and Martin Tak. Sarah: A novel method for machine learning problems using stochastic recursive gradient. ICML, 2017.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics & Image Processing, 2008
Omkar Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, 2012.
Sashank Reddi, Ahmed Hefny, Suvrit Sra, Barnabas Poczos, and Alex Smola. Stochastic variance reduction for nonconvex optimization. In ICMI,, 2016.
Mark Schmidt, Nicolas Le Roux, and Francis Bach. Minimizing finite sums with the stochastic average gradient. arXiv preprint 1 2016.
Shai Shalev-Shwartz and Tong Zhang. Stochastic dual coordinate ascent methods for regularized loss minimization. JMLR, 2013.
Samuel Smith, Erich Elsen, and Soham De. On the generalization benefit of noise in stochastic gradient descent. In /CML, 2020.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In CVPR, 2016.
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai Thomas Un- terthiner, Jessica Yung. Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mip-mixer: An all-mip architecture for vision. In NeurlPS, 2021.
Ruiqi Wang and Diego Klabjan. Divergence results and convergence of variance reduced version of 2022.
adam. arXiv preprint
Ross Wightman. GitHub repository: Pytorch image models. GitHub repository, 2019.
Lin Xiao and Tong Zhang. proximal stochastic gradient method with progressive variance reduction.
SIAM Journal on Optimization, 2014.
Huizhuo Yuan, Yifeng Liu, Shuang Wu, Xun Zhou, and Quanquan Gu. Mars: Unleashing the power of variance reduction for training large models. arXiv preprint arXiv:2411.10333 2024.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
Matthew Zeiler. Adadelta: An adaptive learning rate method. arXiv preprint arXiv: 2012.
Hongyi Zhang, Moustapha Cisse, Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
Michael Zhang, James Lucas, Geoffrey Hinton, and Jimmy Ba. Lookahead optimizer: k steps forward, step back. In NeurlPS, 2019.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmenta- tion. In AAAI, 2020.
Dongruo Zhou, Pan Xu, and Quanquan Gu. Stochastic nested variance reduction for nonconvex optimization. In NewrIPS, 2020.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A DERIVATION OF THE OPTIMAL COEFFICIENT
We present the full derivation the optimal coefficient for control variates:
min Var(X*) = min Var(X aY)
(7)
min Var(X) 2aCov(X, Y) a Vara()..
(8)
Differentiating the objective with respect to we can determine the optimal coefficient a":
2a (Y (Cov(X, Y) = 0,
(9)
Cov(X, Y) Var(Y)
(10)
Lastly, we can plug the definition of the correlation coefficient:
(X. Y) Cov(X. Y)
(11)
o(X)o(Y)
into the optimal coefficient and rewrite Equation 10 as:
a' = p(X, Y) (X)
(12)
oYY
B EXPERIMENTAL SETTINGS
Training recipe. Table 5 outlines training recipe. It is based the setting in ConvNeXt (Liu et al., 2022). For all experiments, the base learning rate is set at 4e-3, except for training ConvNeXt-F on ImageNet-IK using a-SVRG, where increasing it to 8e-3 reduces training loss very much.
config
value
weight init
trunc. normal (0.2)
optimizer
AdamW
base learning
4e-3
weight decay
0.05
optimizer momentum
81, 82 0.9, 0.999
learning rate schedule
cosine decay linear
warmup schedule
randaugment (Cubuk al., 2020)
(9, 0.5)
mixup (Zhang al., 2018)
0.8
cutmix (Yun al.. 2019)
1.0
random erasing (Zhong et al., 2020)
0.25
label smoothing (Szegedy et al., 2016)
0.1
Table 5: Our basic training recipe, adapted from ConvNeXt (Liu et al., 2022).
Hyperparameters. Table 6 lists the batch size, warmup epochs, and training epochs for each dataset. Note that these hyperparameters selections are done on the AdamW baseline. For each dataset, we
CIFAR-100
Pets
Flowers
STL-10
Food101
DTD
SVHN
FuroSAT
ImageNet-1K
batch size
1024
128
128
128
1024
128
1024
512
4096
warmup epochs
50
100
50
50
100
40
training epochs
300
600
300
300
200
Table 6: Hyperparameter setting.
14

---

Published as a conference paper at ICLR 2025
set the batch size proportional its total size and tune the training epochs to achieve reasonable performance. The warmup epochs are set to one-fifth or one-sixth of the total training epochs.
We do not use stochastic depth (Huang al., 2016) small models. For larger models, adhere to the original work (Dosovitskiy et al., 2021; Liu et al., 2022), using a stochastic depth rate of 0.4 for ViT-B and 0.5 for ConyNeXt-B. In these models, maintain consistent stochastic pattern between the current model and the snapshot at each iteration (Defazio & Bottou, 2019).
C ADDITIONAL RESULTS OF a-SVRG
In this section, provide additional experiment results to demonstrate the effectiveness of a-SVRG. This includes full results of a-SVRG with different initial coefficients On on small image classification datasets (Appendix C. training with three random seeds (Appendix C.2), full learning rate search (Appendix C.3), and applying Q-SVRG only in the early training stage (Appendix C.4).
C.I DIFFERENT INITIAL COEFFICIENTS
Table 7 presents the performance of ConvNeXt-F trained with a-SVRG using different initial coefficients ag on various image classification datasets. a-SVRG consistently reduces the training loss of ConvNeXt-F and enhances the validation accuracy on most datasets, regardless of the choice of initial coefficient ag. This demonstrates the robustness of a-SVRG the initial coefficient.
CIFAR-100
Pets
Flowers
STI.-10
AdamW
2.659
2.203
2.400
1.641
SVRG
2.937
10.278
3,424
1.221
2.256
00.144
1.899
0.258
a-SVRG (au 0.5)
2.622
4.0037
1.960
40.243
2.265
4.0.35
1.583
4.0.58
a-SVRG (ag = 0.75)
2.646
4.0.133
2.004
4..199
2.162
4.0.38
1,568
0.007
a-SVRG (an = 1)
2.712
0.005
1.994
1,0.209
2.259
1.0.141
1.573
0.0068
Food-101
DTD
SVHN
EuroSAT
AdamW
2,451
1,980
1.588
1.247
SVRG
3.026
0.557
2.009
1.0029
1.639
1.0051
1.249
0.002
a-SVRG (oo 0.5)
2,423
4.0028
1.865
40.115
1.572
4.0.16
1.243
40.004
a-SVRG (ag 0.75)
2.461
0.0010
1.829
4.0151
1.573
1.0.15
1.237
4.0010
a-SVRG (Ao 1)
2.649
0.198
1.790
0..190
1.585
1.0003
1.230
40.017
(a) training loss
CIFAR-100
Pets
Flowers
STL-10
AdamW
81.0
72.8
80.8
82.3
SVRG
78.2
128
17.6
155.2
82.6
f 1.8
65.1
4 17.2
a-SVRG (Au = 0.5)
81.4
10.4
77.8
1.00
83.3
2.5
83.5
1.2
oVSVR (oo 0.75)
80.6
40.4
76.7
3.9
82.6
1.8
84.0
1.7
a-SVRG (ao 1)
80.0
11.
77.3
1..5
81.9
1.1
84.0
1.7
Food-101
DTD
SVHN
Euro
AdamW
85.9
$7.9
94.9
98.1
+ SVRG
79.6
46.
57.8
J0.1
95.7
+..8
97.9
10.
+ a-SVRG (am 0.5)
85.9
10.0
57.0
40.9
95.4
10.1
+ SVSG (an 0.75)
85.0
10.9
60.3
12.4
95.7
10.8
98.2
10.1
+ a-SVRG (oo 1)
83.8
22.
61.8
13.9
95.8
1.9
98.2
10.1
(b) validation accuracy
Table 7: Results on smaller classification datasets with different initial coefficients. While SVRG negatively affects performance most of these datasets, a-SVRG consistently reduces the training loss and improves the validation accuracy for almost any initial coefficient on each dataset.
15

---

Published as a conference paper at ICLR 2025
C.2 STANDARD DEVIATION RESULTS
Here we run the AdamW baseline and a-SVRG in Table 3 with random seeds. Table 8 presents the results. a-SVRG decreases the mean training loss and improves the mean validation accuracy. The mean difference is usually larger than one standard deviation, indicating the reliability of a-SVRG.
CIFAR-100
Pets
Flowers
STL-10
AdamW
2.645 0.013
2.326
0.088
2.436 0.038
1.660
0.017
+a-VVG
2.606 + 0.017
2.060 0.071
2.221 0.042
1.577 = 0.022
Food-101
DTD
SVHN
EuroSAT
AdamW
2.478 0.021
2.072
0.066
1.583 0.005
1.259 + 0.017
+ +-SVVG
2.426 + 0.007
1.896 = 0.075
1.572 - 0.011
1.239 = 0.016
(a) training loss
CIFAR-100
Pets
Flowers
STL-10
AdamW
81.02 + 0.07
70.61 + 1.55
80.33 + 1.01
80.80 = 1.46
+a-SVRG
81.07 0.22
76.37 + 1.06
84.15 1.15
83.65 0.92
Food-101
DTD
SVHN
EuroSAT
AdamW
85.29 0.47
56.21 + 1.19
94.29 0.67
97.91 + 0.12
+ a-SVRG
85.45 0.43
61.44 + 0.35
94.94 0.60
98.13 + 0.07
(b) validation accuracy
Table 8: Results of a-SVRG with AdamW with standard deviation.
In Section 5, primarily AdamW as the base optimizer study a-SVRG. Here we switch the base optimizer in a-SVRG from AdamW W to SGD. Specifically, we train a ResNet-18 (He et al., 2016), which by default uses SGD train, on the small image classification datasets. Following He et al. (2016) 6), we use an initial learning rate of 0.1, which is divided by 10 when the error plateaus, momentum of 0.9, and weight decay of le-4. Other settings in training recipe remain the same as Table 5. Table 10 details other hyperparameters, such as batch size and training epochs.
CIFAR-100
Pets
Flowers
STL--10
SGD
3.118 0.035
2.706
0.095
2.822
0.058
1.763 0.032
+ a-SVRG
3.087 + 0.011
2.655 0.1
2.699 0.049
1.725 = 0.043
Food-101
DTD
SVHN
EuroSAT
SGD
3.424 0.015
2.589 0.032
1.789 0.029
1.449 0.029
+ a-SVRG
3.397 +: 0.006
2.543 0.039
1.764 0.014
1.412 + 0.011
(a) training loss
CIFAR-100
Pets
Flowers
STL-10
SGD
75.43 0.88
71.25 1 4.74
65.92 1 4.24
76.09 1: 1.09
+ +a-SVRG
75.93 0.83
71.89 + 4.84
74.98 3.14
78.55 2.54
Food-101
DTD
SVHN
EuroSAT
SGD
60.58 2.00
57.53 + 1.25
91.10 1.31
95.31 0.46
+ +VSRRG
62.89 0.87
58.44 + 1.05
91.60 0.59
96.17 + 0.17
(b) validation accuracy
Table 9: Results of a-SVRG with SGD smaller datasets with standard deviation.
16

---

Published as a conference paper at ICLR 2025
We compare a-SVRG (ag 0.5) equipped by linear decreasing schedule to the baseline using only SGD. The results, shown Table 9, are based on the average runs with different random seeds. As see, a-SVRG consistently outperforms the SGD baseline across all datasets.
CIFAR-100
Pets
Flowers
STI.-10
Food101
DTD
SVHN
EuroSAT
batch size
1024
128
128
128
1024
128
1024
512
training epuchs
150
150
200
200
Table 10: Hyperparameter setting for a-SVRG with SGD on ResNet-18.
C.3 DIFFERENT LEARNING RATES
a-SVRG with AdamW. In Section and a-SVRG. However, each method's optimal learning rate might be different. Here we sweep the base learning rate over the range [Ie-2,5e-3, le-3, 5e-4, le-4) for both methods. As shown in Figure 13, a-SVRG (ag 0.5) can decrease training loss better than vanilla AdamW in most cases.
the same base learning rate of 4e-3 for both Adam
4.0-
3,788
3.4
23
AdamW
train loss
3.6
a-SVRG
7,855
train loss
3.0
2,855
2,575
2,688
090
3.074
2,873
2,352
2.8
2.655
2.624
2,680
2.611
2,806
2,987
2.2
1.774
2.130
23002
AdamW
a-S a-SVRG
2.4
1.8
le-2
Se-3
Te.3
5e-4
le.4
1e.2
5e.3
le.3
Se44
1e44
learning rate
learning rate
(a) CIFAR-100
(b) Pets
3.637
AdamW
AdamW
train loss
3.0
3.5
a-SVRG
2,652
2.876
3.458
train loss
2.0
1.8
+-SVVR
1.672
1,732
1.782
1.914
1.879
2.484
2,370
2,294
2,461
2,485
2,618
L,99
1379
L641
1.689
2.0
1.4
le-2
5e-3
le.3
5e-4
le.4
le.2
5e.3
Te33
5e4
1e-4
learning rate
learning rate
(c) Flowers
(d) STL-10
4.6
3.2
train loss
AdamW a-SVRG
3,818
train loss
2.9 2.776
AdamW
a-SVRG
2.865
3.4
2.6
2.697
2.367
2.417
2.447
2.610
2,710
2.740
2.3
23312
2.172
2.284
2,253
2212
2,398
2.555
2.156
le-2
5e.4
le.4
le-2
le-3
5e-4
leamnnng rate
learning rate
(e) Food-10 101
(f) DTD
2.2-
1.7
AdamW
1.946
AdamW
train loss
+-SVRG
train loss
+-SVRG
1.8
1,702
1.904
352
1.645
1.6
1.578
1.665
1.584
13373
1.625
1.633
1.4
1.389
1,373
1.327
1.328
1.366
1.4
1324
1.318
1353
le-2
Se-3
le-3
5e-4
le-4
le-2
5e-3
le-3
5e-4
le-4
learning rate
learning rate
(g) SVHN
(h) EuroSAT
Figure 13: a-SVRG with AdamW at different learning rates.
17

---

Published as a conference paper at ICLR 2025
a-SVRG with SGD. We also sweep the base learning rate for the results of Table 9 using SGD the base optimizer. We compare vanilla SGD to a-SVRG (ag = 0.5). The results are shown in Figure 14. In most learning rates, a-SVRG can achieve a lower training loss than SGD.
train loss
3.6
SGD
a-SVVRG
3,488
3.704
3.693
train loss
3.0
3.2
066
SGD
3.009
3.082
3.171
5.062
3.4
3.478
+-SVRG
3,120
3,118
3.311
2.6
271
2655
2.603
3.0
3087
2.326
20-1
16-1
4e-2
20-2
lo-2
20-1
16-1
46-2
20-2
16-2
learning rate
learning rate
(a) CIFAR-100
(b) Pets
3.6
3,541
2.0
3,381
3,400
SGD
train loss
3.4
3,233
3,224
3.336
3/435
train loss
1.9
+a-SVRG
1.814
3.2
3:207
1,740
17788
1:754
3,023
3.061
SGD
eVSV
1.7
1.666
1.25
1.31
746
2.8
1.6
2e-1
le-1
4e-2
2e-2
le-2
26-1
le-1
4e-2
2e-2
1e-2
learning rate
learning rate
(c) Flowers
(d) STL-10
4.4
3.2 3.118
train loss
SGD
a-SVRG
3,840
4.031
4.028
train loss
3.0
SGD
*-SVRG
3.8
3,636
3.824
2.8
2,770
2.717
66
3.391
3,424
3.607
2,586
2.653
2.706
3.2
3.349
3,397
2.356
2.346
2.585
20-1
16-1
20-2
lo-2
46-2
20-2
16-2
learning rate
learning rate
(c) Food-101
(f) DTD
2.4
1.7
2,248
SGD
1.597
-
SGD
1,556
train loss
2.
*+SVRR
train loss
*-SVRG
1,998
2.0
1.854
1.5
1.449
T.530
1.8
1.764
1,788
1.242
1.737
1,794
1,779
E351
1.4
1412
1.401
1.469
2c-1
le-1
4e-2
20-2
1o-2
20-1
10-1
4e-2
20-2
16-2
learning rate
learning rate
(g) SVHN
(h) EuroSAT
Figure 14: a-SVRG with SGD at different learning rates.
18

---

Published as a conference paper at ICLR 2025
C.4 USING SVRG DURING THE INITIAL PHASES OF TRAINING
Compared with SGD and AdamW, W, both standard SVRG and a-SVRG require computing the snapshot stochastic gradient V V. and snapshot full gradient V (((pu))). This leads to about twice the computational cost of the baseline methods. Nevertheless, in Section 4, we find that a-SVRG can only reduce gradient variance during the initial training epochs and then maintains a similar level as the baseline methods. This motivates us to apply a-SVRG only at the early phases of training, and we refer this approach as early a-SVRG.
To evaluate this approach, we use early a-SVRG to train ConvNeXt-Femicc on various image classification datasets. We use the default linear decay with an initial coefficient ag 0.75 to schedule early a-SVRG, but a-SVRG is only applied during the first 10% of training and is disabled thereafter. Moreover, we add transition epoch where the coefficient decreases from its original final value to zero. We find this crucial for maintaining the stability of momentum in the base optimizer.
CIFAR- 100
Pets
Flowers
STL-10
A
2.659
2.203
2,400
1.641
a-SVRG
2.646
1.0.13
2.004
1..199
2.162
4.0238
1.568
4..073
+ early a-SVRG
2.644
1.0015
2.190
10.013
2.328
4 1.007
1.616
4 0.025
Food-101
DTD
SVHN
EuroSAT
AdamW
2.451
1,980
1,588
1.247
+ a-SVRG
2.461
1.0010
1.829
40.151
1.573
10.015
1.237
1..010
+ carly a-SVRG
2.444
4.0007
1.918
10.062
1.583
4.0005
1.240
0.000
(a) training loss
CIFAR- 100
Pets
Flowers
STL-10
baseline
81.0
72.8
80.8
82.3
a-SVRG
80.6
40.4
76.7
3.9
82.6
t 1.8
84.0
1 1.7
eerlly a-SVRG
81.0
0.00
74.6
1.8
83.6
2.28
82.8
0.5
Food-10 101
DTD
SVHN
Euro
baseline
85.9
57.9
94.9
98.1
a-SVRG
85.0
40.9
60.3
2.4
95.7
0.8
98.2
10.1
+ early a-SVRG
85.9
0.0
62.3
1.4
0.9
98.0
10.1
(b) validation accuracy
Table 11: Early a-SVRG smaller classification datasets.
Figure 11 shows the results. We can see early a-SVRG consistently reduces training loss across different datasets. Furthermore, we observe that the validation accuracy achieved by early a- SVRG is sometimes higher than that of standard a-SVRG. This phenomenon is likely because disabling a-SVRG in the later training stages allows the presence of benign gradient noise during optimization. Such noise may drive the model toward local minima that exhibit better generalization properties (Smith et al., 2020; Damian et al., 2021; Li al., 2022).
19

---

Published as a conference paper at ICLR 2025
D ADDITIONAL RESULTS OF OPTIMAL COEFFICIENT
We provide further results on optimal coefficient SVRG (Equation 6) below: gradient variance reduction other datasets with SVRG using optimal coefficient (Appendix D.I), the impact of data on optimal coefficient (Appendix D.2), and the evolution of the correlation between model gradients and snapshot gradients during training (Appendix D.3).
D.1 SVRG WITH OPTIMAL COEFFICIENT ON OTHER DATASETS
Throughout the experiments in Section 3, we mainly study the behavior of SVRG with optimal coefficient on CIFAR-I0 10 dataset. To show the generality of our approach for SVRG, below monitor the optimal coefficient on other image classification datasets. We train MLP-4 on Flowers / EuroSAT with SGD and SVRG using optimal coefficient for 15 / 25 epochs. As shown Figure 15, SVRG with optimal coefficient can consistently reduce gradient variance and achieve lower training loss than the baseline SGD.
grad. var. (metric 1)
0.6
grad. var. (metric 2)
0.4
10
5
12
epochs
epochs
grad. var. (metric 3)
2.7
4.8
1.8-
train loss
3.2
0.9
1.6
SGD
+VVR (opt. coef.)
0.0
12
I's
12
is
epochs
epochs
(a) Flowers
grad. var. (metric 1)
grad. var. (metric 2)
0.2
1.4
0.1
0.7
0.0
10
I's
20
2'
15
20
25
epochs
epochs
(metric 3)
0.6
SGD
0.4
train loss
SVRG (opt. coef.)
0.2
1.3
0.0
0.8
3
10
15
20
25
10
15
20
25
epochs
epochs
(b) EuroSAT
Figure 15: SVRG with optimal coefficient on other datasets.
D.2 EFFECT OF DATA ON OPTIMAL COFFFICIENT
Below we conduct experiments on CIFAR-10 and CIFAR-100 to understand how the number of object classes in datasets affect optimal coefficient. Specifically, train 1, 2, and 4- MI.Ps
20

---

Published as a conference paper at ICLR 2025
(Logistic Regression, MLP-2, and MLP-4) on each of the two datasets using SGD (without SVRG)
and compute the optimal coefficient (Equation 6) during the training. Figure 16 shows the results.
optimal coefficient
1.0
optimal coefficient
1.0
0.7
0.7
Logistic Regression
0.4
0.4
MLP-2
MLP
0.1
0.1
2
11
epochs
epochs
(a) CIFAR-10
(b) CIFAR-I00
Figure 16: Effects Data on Optimal Coefficient.
The optimal coefficient of each model trained on CIFAR-100 is lower than that on CIFAR-10. This possibly because CIFAR-100 has 10x more object classes than CIFAR-I0 and therefore the correlation between the snapshot model gradient and the current model gradient is weaker. Thus, we recommend using a smaller coefficient when the training dataset includes larger number of classes.
D.3 CORRELATION BETWEEN MODEL GRADIENTS AND SNAPSHOT GRADIENTS
In Section 3, we find each epoch's 's average optimal coefficient decreases as training progresses. Here, seek understand whether the standard deviation ratio a (0/)) (V f (OPut)) or the correlation f. (@Pa)), f (0') ) in Equation 6 contributes more to this observation. We plot these two values separately in Figures 17 and 18. We can see the standard deviation ratio is relatively stable during the training, whereas the correlation decreases very much during the training.
1.4
1.4
1.0
std. ratio
1.0
Logistic Regression
sid.
0.6
MLP-2
MLP4
0.2
0.2
2
3
10
11
3
4
5
10
epachs
epachs
(a) SGD
(b) Adam W
Figure 17: Standard deviation ratio. The ratio between the standard deviations of the model gradients and the snapshot gradients oscillates around but is relatively stable overall.
1.0
correlation
correlation
0.7
0.7
Amma
Logistic Regression.
0.4
0.4
MLP-2
MLP-4
0.1
0.1
2
10
11
2
3
4
5
10
epochs
epochs
(a) SGD
(b) Adam
Figure 18: Correlation. The correlation between the snapshot gradients and the model gradients decreases the training progresses.
E ALTERNATIVE SCHEDULES FOR a-SVRG
In this part, we provide the exact mathematical formulation of each schedule studied in Section 5.3.
21

---

Published as a conference paper at ICLR 2025
Notations. We decompose the global iteration index into epoch-wite wise index and an iteration index é within an epoch. We also denote the total number of training epochs as T and represent the number of iterations in one epoch as M.
Linear schedule. This is our default scheduling approach. The coefficient decreases linearly across epochs and keeps as constant within an epoch:
s(t),
iisarr = 000(1
(13)
T
Other global schedules. We also consider quadratic decay and geometric decay:
L quadratic
00 (T' s(t))?
(14)
72
Ottinal
geometric
oxp
2
(15)
an
Double schedules. In Figure 4 of Section 3, within each epoch, the coefficient starts from and decreases over time. Motivated by this local behavior, we introduce three additional schedules that combine both the local and global decay: d(ouble) -linear, d-quadratic, and d-geometric. In addition to the global decay, each double schedule has another local decay for each epoch that initiates at I and decreases to an ending value specified by the global decay.
Mincar (1
+ Fainnrr
(16)
d-lincar
M
local decay
quaadraric
(1 0 quadratic
M (M - 1)² +o + quaddatii
(17)
local decay
dggoomettiic
(a
goometric
+ Ord-limal
(18)
local docay
F CONVERGENCE COMPARISON
We compare the training loss convergence of the AdamW baseline and a-SVRG on ImageNet- IK (Figure 19) and small classification datasets (Figure 20). It is observed that a-SVRG can consistently decrease training loss and deliver faster convergence.
7.0
7.0
7.0
AdamtNi
AdamW/
AcamW
train boa
5.8
SVRG (ou 0.77)
train loss
or SVRG (o. 0.75)
train loes
5.8
SVRG (ou 0.76
30
100
150
200
250
3/0
30
103
150
203
250
300
50
100
150
200
250
300
epochs
epachs
epochs
(a) ContNNettFF
(b) ViT-T
(c) Swin-F
7.0
7
7.0
AdaniW
AdantW
Adaen
train loes
o-SVRG (ov 0.76g
train loss
o SVRO (2) 0.)
train loes
SVRG (e, - 0.5)
4.6
100
150
200 250
350
30
100
150
230
300
50
100
150
200
250
300
apochs
opachs
epachs
(d) Mixer-S
(e) ViT-B
(f) Con
Figure 19: Training loss with AdamW and a-SVRG on ImageNet-1K (Table 2).
We set O'imal 0.01 ensure that the geometric schedule eventually decreases to sufficiently small value. Note that Graal can not he zero, it serves the base of the exponent. The same rule applies Equation 18.
22

---

Published as a conference paper at ICLR 2025
4.7
4.7g
4.7
Mam
Adow
Adamii
3.9
O-VRR (1g 0.5g
--SVR (1ng 0075)
urin U8
3.9
VRS (og
3.0
3.1
2.3
2.3
2.3
50
100
1.0
200
250
300
50
103
150 spochs
200
250
300
50
100
130 spochs
200
250
300
spochs
(a) CIFAR-100
3.7
3.7
3.7
A
Adann
Adamn
3.1
oVRVRG (ou 0.5)
tmin loo
SVRG (1w 0.75)
loss
u-S VRG
1.9
1.9
1.9
200
300
4.00
300
600
100
200
300
400
300
600
100
200
300 spochs
403
300
500
spocts
spocls
(b) Pets
4.8
4.5
4,8
A
AdsmW
Adamii
3.9
O-VVRG (4) 0.5)
wain loss
3.9
o-SVIU 0.7))
uain loss
2.0
3.0
2.8
1000
200
300 spocts
400
500
600
150
200
300
500
600
100
260
300 spochs
400
500
600
spochs
(c) Flowers
2.9
2.5
AdanW
Adan
Adami
1min baa
2.0
SVRG (4k
Irain kos
1
SVRG
tain kos
2.0
SVRG (o,
2540
150 spocls
200
opochs
opocls
(d) STL-10
Adam/W
Adom
AdamiW
1950 1000
3.9
SVRG (do
usin los
3.9
SVRG (.70)
usin lol
3.9
SVRG
2.11
3.1
250
300
50
50
130
250
spochs
epochs
spochs
(e) Food-101
Alanw
AdansW
Inin look
SVRG (su 0.5)
train Loos
SVRG (su 0.79)
crain loos
&VSVG (eo
2.4
1.6
1000
300
400
300
100
100
300
600
100
200
300 spochs
403
350
spochs
(f) DTD
2.3
2.5
Adam/ G-SVRG (ng 0.3)
AdsmW
Adamli
Insin look
2.0
a --SVR (mg 0.75)
usiin boss
2.0
o-VVRG (ng =1)
0.7
train
3.7
1.4
1.4
1.4
255
50 epochs
75
150
25
50 spochs
7/
epochs
(g) SVHN
2.3
2.5
2.5
AdannW
AdannW
Adamni
o SVRG (cu 0.5)
trann Joa
a SVRG (0.50.7))
& SVRG (0a -1
Inde
1.5
1.1
1.1
30
100 spochs
150
Z0o
30
150
300
30
160 spacts
150
300
epocti
(h) EuroSAT
Figure 20: Training loss with Adam W and a-SVRG smaller classification datasets (Table 7).
23

---

Published as a conference paper at ICLR 2025
G PSEUDOCODE FOR a-SVRG
For clarity, we adopt the following notations: 8 as the epoch index, t as the iteration index within each epoch, & as the sampled index at iteration t, ' as the epoch length, M the iteration length within each epoch, as the total number of training data, and rff as the learning rate. We illustrate a-SVRG with SGD in Algorithm and with Adam W Algorithm 2:
Algorithm 1 a-SVRG with (G T, M, Hine ) )2z0 Ha
Input: initialized model parameters 08. epoch length iteration length within each epoch M. learning rates {(ni Jt-u M- Jamú scheduled coefficients {{af] Jimu 1 weight decay À
0 =60
for = to T 1-116 do
(f = 1 >. (0pm))
for = too M 1 do
1t ud
Uniform (1. (1..... n
g Vs, (0:) o (V fú (0pm)) ((((pa)))
0 0: T. 20;
end for
0p 0i
end for
Output: final model 0,,
Algorithm 2 a-SVRG with AdamW M, (br) Jt= 1: {(ai) 1s=0 . 31.32,1.
Input: initialized model parameters 0g epoch length T, iteration length within each epoch M, learning rates Hin 1:2 - scheduled coefficients {{a? 1 y momentums 31, B2, weight decay À
0 = 0g
m -
do for = 0 to
f( pmu) = 1 D (f
for t .M do
Rdd Uniform(1, (1.
96 VS,
a ( ft. (0mma) - V v((Opa))
m = 3im + (1 Bz)g
v 32 + (1 (2)
0 0 VI Votc X0;
end for
pust 0
end for
Output: final model 0
24


Reviews:
[{"review_id": "AjdnUrW5QH", "reviewer": "Reviewer_QXir", "summary": "This paper revisit SVRG in deep learning, showing how reasoning more carefully about control variates. The paper starts by a nice recap of what is believed to be variance reduction and SVRG performance in deep learning, and then guides the reader to the \"additional parameter\" solution. The authors complement their finding (i.e. a simple modification of SVRG can help training) experimentally on a diverse range of datasets in vision, using many architecture.", "questions": "See above", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "I like the paper, it is a simple idea yet presented very clearly:\n\n1) The graphics and plots help the reader get the idea behind the paper. The many presented variance metrics also are very helpful in convincing the reader of the approach.\n\n2) The authors go step by step, showing improved performance with \"optimal variance reduction\" and later showing why a simple coefficient can close most of the gap. This is very important, makes the investigation very scientific.\n\n3) There are many experiments presented in the corresponding section.", "weaknesses": "I think this could be a very interesting easy reference for modern SVRG. I believe though some additional points can help delivering the idea in a more solid way:\n\n1) As far as I understand, you are always using Adam with lr 3e-4 in your experiments. This is a quite nice lr - but it is well-known that Adam might profit from additional tuning. I do not believe your point here should be \"you are better than Adam\" - many papers overclaim performance as their main point, but you show much more. I think here what could be helpful is a plot of final accuracy or training loss over learning rare - comparing both with SGD and Adam. Showing this for 1 learning rate is not enough. Note that you do not have to show you are better than Adam, but just that you are better than SVRG and close most of the gap.\n\n2) Figure 5 (SVRG with optimal alpha) is a bit weak: not much improvement over SGD id shown. Do you have an explanation for this? In a way this is the \"best we can get\" - yet in this specific example Adam, SGD and SVRG (with or withouth alpha) perform quite similar! the variance metrics also confirm that not much variance reduction is happening. I would personally try to find another example or compare this performance on multiple examples.\n\nIf the authors can show some improvements around these points, I am happy to raise my score!\n\n-------\n\nAuthors addressed part of my concerns, and will likely improve experiments further, increased to accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GjMIsO5myz", "reviewer": "Reviewer_Nknp", "summary": "This paper observes the ineffectiveness of SVRG due to the decay of the relevance between the batch gradient and the snapshot. The paper then proposes $\\alpha$-SVRG, with $\\alpha$ as a conditioning multiplicator to account for the decaying relevance. The approach is shown to be effective in multiple experiments.", "questions": "In the schedule of Eq. (9), why do we use a constant of $10^{-2}$ instead of other constants? Can this constant also be tuned?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The flow of the paper is easy to follow.\n2. There is adequate discussion about the experiment results.\n3. The perspective of the gradient preconditioning is interesting.", "weaknesses": "1. The design of the scheduling of $\\alpha$ may require more discussions and refinements. Specifically,\n- The effect of $\\alpha_0$. The following three points are a bit inconsistent as a whole: (a). From Figure 4, the optimal $\\alpha_0$ seems to be almost the same for different models. (b) From Table 4 and Figure 9, it seems that $\\alpha\\approx0.75$ may be a better choice that $\\alpha_0=0.5$ (at least for the small model that is used). (c) In the experiment that yields Table 3, however, $\\alpha-0.5$ for small models and $\\alpha_0=0.75$ for larger ones.\n- The schedule of $\\alpha$. It may be interesting to analyze different types of scheduling of $\\alpha$ based on the perspective of preconditioning in Appendix G.\n2. For the experiment that yields Figure 4, it would be better if the authors could conduct experiments on different datasets and try to analyze the effect of data on the optimal $\\alpha$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZeNDq3kich", "reviewer": "Reviewer_9C3X", "summary": "This paper proposed the $\\alpha$-SVGR algorithm, which introduces a linearly decaying coefficient to adjust the scale of the correction term in SVRG. The proposed algorithm is inspired by the observation that the original SVRG does not effectively reduce the variance in the late training phase. To show the effectiveness of $\\alpha$-SVRG, the authors conducted extensive experiments across various datasets and model architectures.", "questions": "In Figure 5, the gradient variance of $\\alpha$-SVRG is not lower than SGD in the late phase in terms of any metric. Does the improvement of $\\alpha$-SVRG in training loss, if any, come from the reduced variance in the early phase? If this is the case, would it be more effective to employ $\\alpha$-SVRG only during the initial phase and switch to standard AdamW or SGD for the later stages?\n\nTo further elaborate my question, the role of gradient noise in helping generalization is widely discussed in the literature (e.g., [1]-[5]). In the late phase of training, we even need to encourage more helpful, benign noise that drives the model parameter to better generalizing minima. From this perspective, why not altogether remove the correction term in SVRG in the late phase?\n\n[1] Samuel Smith, Erich Elsen, and Soham De. On the generalization benefit of noise in stochastic gradient descent. ICML'2020.\n\n[2] Alex Damian, Tengyu Ma, and Jason D. Lee. Label noise SGD provably prefers flat global minimizers. NeurIPS'2021.\n\n[3] Zhiyuan Li, Tianhao Wang, and Sanjeev Arora. What happens after SGD reaches zero loss?–a mathematical framework. ICLR'2021.\n\n[4] Guy Blanc, Neha Gupta, Gregory Valiant, and Paul Valiant. Implicit regularization for deep neural networks driven by an ornstein-uhlenbeck like process. COLT'2020.\n\n[5] Xinran Gu, Kaifeng Lyu, Longbo Huang, Sanjeev Arora. Why (and When) does Local SGD Generalize Better than SGD? ICLR'2023.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. This paper is well-written with clear logic. It first identifies the problem with the original SVRG by monitoring the gradient variance during training. Then, it addresses the identified issue by introducing an extra coefficient that adjusts the scale of the SVRG correction.\n\n2. The evaluation is comprehensive. The authors not only conducted extensive experiments across various datasets and model architectures but also explored other possible schedules for the introduced hyperparameter $\\alpha$. However, I still have some concerns about the effectiveness and extra cost of $\\alpha$-SVRG (see weaknesses).", "weaknesses": "**Major**\n1. The effectiveness of $\\alpha$-SVRG: In Table 2, compared with the baseline $\\alpha$-SVRG's improvement in training loss and validation accuracy is marginal. \n\n2. The computation and memory cost of $\\alpha$-SVRG: $\\alpha$-SVRG requires additional computations for the full gradient $\\nabla f(\\theta^{\\text{past}})$ and the mini-batch gradient at $\\theta^{\\text{past}}$, $\\nabla f_i(\\theta^{\\text{past}})$. Additionally, it necessitates maintaining $\\theta^{\\text{past}}$ and $\\nabla f(\\theta^{\\text{past}})$ in memory. The paper fails to address or discuss these significant costs.\n\n\nIn summary, while the paper offers a good starting point for investigating why SVRG underperforms in deep learning, the enhancements to the original SVRG model, represented by $\\alpha$-SVRG, do not sufficiently justify the substantial additional costs and provide only limited benefits compared to standard SGD or AdamW.\n\n\n**Minor**:\n1. The authors are advised to include the pseudo-code for AdamW + $\\alpha$-SVRG rather than merely describing its use of AdamW as the base optimizer.\n\n2. Typos in eqs (7)-(9).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "45G9MUfmqW", "reviewer": "Reviewer_4HSX", "summary": "The paper studies a modification of the variance-reduced method SVRG. The authors attempted to fix the issue of the applicability of variance-reduced methods in deep learning which was noticed by Defazio & Bottou, 2019. The modification consists of adding a scaler to the variance term in the update of SVRG. The authors explain this change by considering an example of the reconstruction of one random variable X from another variable Y correlated to X. The authors provide the convergence guarantees for the proposed modification of SVRG and demonstrate its effectiveness for several DL benchmarks.", "questions": "- Could you please elaborate how $\\alpha$-SVRG update is plugged in AdamW? Do you perform $\\alpha$-SVRG update on the stochastic gradient which is then used in AdamW? Or do you perform $\\alpha$-SVRG trick for the updates of AdamW? In general, this would be beneficial to add a pseudocode at least in the appendix how to combine the proposed $\\alpha$-SVRG update with any optimizer.\n\n- I don't understand the connection of $\\alpha$-SVRG and preconditioning. In (38) the authors define a matrix A. Could you provide the exact form for A? I believe the SVRG update cannot be presented as a linear mapping and therefore this connection seems to be misleading.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to follow. \n\n- There is a clear intuition as to why there should be a scaler smaller than 1 in the update of SVRG from the example in (2). The authors support this change by computing theoretical values of the scaler $\\alpha$ for logistic regression and MLPs. It turns out that the values of  $\\alpha$ drop to the end of the training for MLP and stay stable around $1$ for logistic regression. This empirical evaluation separates convex logistic regression (where SVRG is known to perform well) and non-convex MLP (where SVRG doesn't perform well).\n\n- The proposed $\\alpha$-SVRG method was tested on several DL workloads demonstrating its potential improvement over other baselines like SGD, SGD+SVRG, and AdamW.\n\n- The authors provide potentially useful $\\alpha$-schedulers that gradually decrease the values of $\\alpha$. This is explained by the fact that vanilla SVRG can reduce the variance at the beginning of the training (where the stochastic gradients are more aligned) and fails in the end.", "weaknesses": "- SVRG-type methods do not suit the DL framework well not only because of the failure to effectively reduce the variance but also because of the necessity of full gradient computation from time to time. From this perspective, methods Zero-SARAH are more interesting to explore. \n\n- The convergence of $\\alpha$-SVRG is presented under the assumption of bounded gradients $\\|\\nabla f_i(x)\\|\\le \\sigma$. In my opinion, this assumption is strong and doesn't allow to show the theoretical benefit of using $\\alpha$-SVRG instead of vanilla SVRG. \n\n- The authors claim that $\\alpha$-SVRG decreases the variance. I don't understand why this is the case. For example, based on Figure 5 $\\alpha$-SVRG can reduce the variance only in the beginning (where vanilla SVRG can do that as well) but afterward, the variance of $\\alpha$-SVRG is similar to SGD which doesn't have any variance reduction mechanism inside. Moreover, the fluctuations of empirical variance of $\\alpha$-SVRG are higher than that of SGD. Therefore, I believe it would make sense to average the variance across several random seeds to see the statistical error (it might happen that in the beginning $\\alpha$-SVRG was a bit lucky). This would be also beneficial for the train loss dynamics to understand how significant the improvement. For example, for some workloads, the improvement in Table 3 is in the second digit after the dot which most likely is within the standard deviation which I believe cannot be used as significant (the results in table 8 seem to support my comment).\n\n- The comment above also relates to the claims of the form \"...can reduce gradient variance significantly...\" (see line 353 and other lines like this). First, since the results of one run are presented it is not possible to measure the significance (it might be that the differences in training dynamics of the algorithm and its $\\alpha$-SVRG version are within standard deviation). Second, how is \"significance\" measured for the proposed 3 variance metrics? Can the change 0.1 be considered large? What is the threshold for being \"significant\"?\n\n- The results in Figure 10 are controversial to the claims of the paper. For small batch sizes you expect $\\alpha$-SVRG to perform better since the variance should be larger. However, AdamW without any variance reduction mechanism handles the variance better than.\n\n- The authors use various deep learning techniques like warmup in the training and the improvement might come from a weird combination of all of those techniques with $\\alpha$-SVRG update. In my view, it is important to understand if $\\alpha$-SVRG procedure alone indeed helps to reduce variance.\n\nTo summarize, in my view, the authors didn't provide enough empirical evidence to showcase the usefulness of $\\alpha$-SVRG even though the idea of adding a scaler is intuitively logical.  The improvement is typically marginal (within standard deviation) and comes with a significant increase in memory usage and full gradient computation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)