On the Performance Analysis of Momentum Method: A Frequency Domain Perspective
Authors: Xianliang Li
Jun Luo
Zhiwei Zheng
Hanxiao Wang
Li Luo
Lingkun Wen
Linlong Wu
Sheng Xu
Venue: ICLR 2025 Poster (2025)
Keywords: Momentum Method
Stochastic Gradient Descent
Z-Transform
Frequency Domain Analysis
Deep Learning
Primary area: optimization
TL;DR: Momentum method is a time-variant filter for gradients.

Abstract: Momentum-based optimizers are widely adopted for training neural networks. However, the optimal selection of momentum coefficients remains elusive. This uncertainty impedes a clear understanding of the role of momentum in stochastic gradient methods. In this paper, we present a frequency domain analysis framework that interprets the momentum method as a time-variant filter for gradients, where adjustments to momentum coefficients modify the filter characteristics. Our experiments support this perspective and provide a deeper understanding of the mechanism involved. Moreover, our analysis reveals the following significant findings: high-frequency gradient components are undesired in the late stages of training; preserving the original gradient in the early stages, and gradually amplifying low-frequency gradient components during training both enhance performance. Based on these insights, we propose Frequency Stochastic Gradient Descent with Momentum (FSGDM), a heuristic optimizer that dynamically adjusts the momentum filtering characteristic with an empirically effective dynamic magnitude response. Experimental results demonstrate the superiority of FSGDM over conventional momentum optimizers.

Published as a conference paper at ICLR 2025
ON THE PERFORMANCE ANALYSIS OF MOMENTUM METHOD: A FREQUENCY DOMAIN PERSPECTIVE
Xianliang Li Jun -u, Zhiwei Zheng*), Hanxiao Wang24, Li Luo", Lingkun Wen
2,6 Linlong Wu', Sheng Xui
'Shenzhen Institutes of Advanced Technology. Chinese Academy of Sciences University of Chinese Academy of Sciences SUniversity of California, Berkeley
'Institute Automation, Chinese Academy of Sciences SSnn Yat-sen University
"Shanghai Astronomical Observatory, Chinese Academy of Sciences yinleung. weyeleellll..... com, zhiwei wewegeeeeeeeeeeeeeyeeeee.e..eeee edu, wanghanxxiaclBemailss www.:lss...s.cc........ luoli33emaile 12. rysu.edu....
University of Luxembourg
{j. s.lnng,sheng
wenlingkum@shao.aco ac..neeeeesaa...........
ABSTRACT
Momentum-based optimizers are widely adopted for training neural networks. However, the optimal selection of momentum coefficients remains elusive. This uncertainty impedes clear understanding of the role of momentum in stochastic gradient methods. In this paper, we present a frequency domain analysis frame- work that interprets the momentum method as a time-variant filter for gradients, where adjustments to momentum coefficients modify the filter characteristics. Our experiments support this perspective and provide deeper understanding of the mechanism involved. Moreover, our analysis reveals the following significant findings: high-frequency gradient components are undesired in the late stages of training; preserving the original gradient in the early stages, and gradually am- plifying oww-freueeccy gradient components during training both enhance per- formance. Based on these insights, we propose Frequency Stochastic Gradient Descent with Momentum (FSGDM), heuristic optimizer that dynamically ad- justs the momentum filtering characteristic with an empirically effective dynamic magnitude response. Experimental results demonstrate the superiority of FSGDM over conventional momentum optimizers.
INTRODUCTION
Momentum has achieved great success in deep learning applications when combined with Stochastic Gradient Descent (SGD) (Robbins Monro, 1951). Among various momentum methods (Polyak, 1964: Nesterov, 1983; Van Scoy al., 2017; Ma & Yarats, 2018; Kidambi et al., 2018), one of the most prevalent variants is the momentum method utilized within Stochastic Gradient Descent with Momentum (SGDM) (Sutskever al., 2013; Paszke et al., 2019), which can be expressed as:
Standard-SGDM (decoupled) m, u, UcDt. It xt-1 a.m,
(1)
where St denotes the gradient at iteration 6, my is the momentum buffer, and H, represents the learnable parameters. The momentum coefficients u, and Uj control the influence of the previous momentum and the current gradient, respectively, and a is the learning rate. For these time-variant momentum coefficients, multistage setting has been commonly adopted in the machine learning community (Aybat et al., 2019; Kulunchakov & Mairal, 2019; Liu et al., 2020). Throughout this paper, refer to this formulation, which decouples the two momentum coefficients, Standard- SGDM. In contrast, another prevalent variant couples the two momentum coefficients using the Exponential Moving Average (EMA) method (Gardner Jr, 1985). leading the formulation of EMA- SGDM:
EMA-SGDM (coupled): m4 4,74-1 + (1-4))9. 4e)ee It It-1 a
(2)
#: Equal contribution. f: Corresponding author.
'Our implementation of FSGDM available https: pettss//ggiten //tpt////.tttt/t//// com/

---

Published as a conference paper at ICLR 2025
where u, [0, 1) is the momentum coefficient. Notably, this coupled momentum formulation is a special case of the decoupled one, i.e., Standard-SGDM with v = 1 - Ue. Our experiments show performance gaps between these two formulations. Moreover, how the momentum coefficients change over time can significantly affect the test accuracy (see Section 3). The existence of these
two distinct momentum formulations and their differing performances raise two primary questions in modern deep learning:
1. Decoupling Coupling: Should the coefficients u, and be decoupled or coupled?
2. Temporal Variation: How should the momentum coefficients evolve over time during training to achieve better model performance?
For Question 1, some literature has investigated the convergence of the coupled method (Mai & Johansson, 2020; Li et al., 2022). Liu et al. (2020) argued that coupling the coefficients leads only to constant scaling difference. Wang al. (2024) further demonstrated that the mathematical equiv- alence between EMA-SGDM and Standard-SGDM can be achieved by adjusting the momentum coefficients and the learning rates in coupled way. However, in practice, learning rate schedules are typically independent of momentum coefficient tuning during network training. On the other hand, popular frameworks like Py (Paszke et al., 2019) adopt decoupled momentum strategy by default. In our framework, we tackle the first question from the frequency domain perspective, revealing the relationship between the coupled and decoupled constructions.
Regarding Question 2, prior research offered diverse opinions on how the momentum coefficients should vary over time, Some studies preferred fixed decoupled momentum coefficients (Yan et al., 2018; Liu et al., 2018; Yu et al., 2019). commonly selecting de values as 0.9 and i value as 1. Liu et al. (2020) highlighted the benefits of stagewise learning rate schedules in EMA-SGDM, noting that u, either remain constant increase along with the stagewise adjustments. Conversely, Smith (2018) demonstrated that decreasing the momentum coefficients while increasing the learning rate improves test performance. Moreover, Adaptive momentum methods (Kingma & Ba, 2014; Reddi et al., 2018; Luo et al., 2019; Chen el al., 2018) proved the convergence of decreasing coupled momentum coefficients in the context of online convex optimization. Nonetheless, consensus regarding the optimal time-variate pattern of the momentum coefficients has yet to be reached.
To answer these questions, one has to understand how the momentum method affects the training process. Goh (2017) analyzed the momentum method from the aspect of convergence and dynam- ics. Several prior studies (Cutkosky & Orabona. 2019; Ma & Yarats, 2018) speculated that averaging past stochastic gradients through momentum might reduce the variance of the noise in the parameter update, thus making the loss decrease faster. Polyak (1964); Rumelhart et al. (1986) argued that the EMA momentum can cancel out oscillations along high-curvature directions and add up contribu- tions along low-curvature directions. From the signal processing perspective, the EMA method acts as discrete low- low-pass filter for smoothing out high-frequency fluctuations while retaining the low- frequency baseband pattern of the signal (Gardner Jr, 1985). These points of view bring us a new insight into connecting the momentum update processes with the specific filters. In this aspect, the momentum methods with different coefficient selections can be interpreted in a unified frequency domain analysis framework, whereby Questions 1 and 2 resolved.
In this paper, we propose a novel frequency domain analysis framework to address the two questions and provide a deeper understanding of the role of momentum in stochastic optimization. To the best of our knowledge, this paper, for the first time, reveals the fundamental difference between Standard-SGDD SGDM and EMA-SGDM and uncovers the effects of the dynamic momentum coefficients clearly from the frequency domain perspective. This perspective not only explains the difference between various momentum methods but also provides practical guidelines for designing efficient optimizers. Accordingly, we introduce FSGDM, an optimizer that dynamically adjusts momentum filter characteristics during training. Experiments show that FSGDM outperforms traditional SGD- based momentum optimizers.
FREQUENCY DOMAIN ANALYSIS FRAMEWORK
This section introduces the background of Z-transform (Zadeh, 1950) in signal processing and then proposes new frequency domain analysis framework for momentum methods.
2

---

Published as a conference paper at ICLR 2025
2.1 Z-TRANSFORM AND
APPROXIMATION
Frequency analysis is crucial technique for understanding how systems react to varying fre- quency components of input signals. Specifically, for discrete-time linear time-invariant systems, Z-transform is leveraged to examine how systems attenuate or amplify signals at specific frequen- cies. especially in the study of system stability, pole-zero behavior, etc. (Oppenheim al., 1996).
Interestingly, in neural network training, the momentum update process at time t can be seen as recursive filter where the gradient 9t and the momentum my act as input and output signals, re- spectively. The momentum coefficients affect the gradient adjustments across different frequency components. The high- fegh-freeeencyy gradient components correspond large and more abrupt changes in the gradient; while the low-frequency components indicate smooth and more gradual adjustments.
However, one key issue is that the momentum system can be inherently time-variant, as its coef- ficients may change stagewise throughout the training process. This variability makes it difficult to apply traditional Z-transform analysis. To overcome this, inspired by the Zadeh (1961); Jury (1964), approximate the system as time-invariant in each discrete interval stage. By holding the momentum coefficients constant over every interval, we construct a time-invariant quasi quasi-stationary system (Hubner & Tran-Gia, 1991), enabling to apply the Z-transform validly.
In our following analysis framework and our later optimizer design, we follow this multistage strat- egy for changing momentum coefficients. Particularly, for predefined stage whose length is de- noted by 6, the momentum coefficients are redefined using the floor function to ensure they remain constant over the whole stage:
" u[[t/5] x 5) and et = =[(t//) x 6),
where (())),(())))) are the continuous dynamic sequence functions with respect to t. While there are multiple sequences with different designs, in this paper, we use the following increasing and decreasing sequences:
1 4+1
Increasing utt) or vtt)
Decreasing u(t) or e(t)
(4)
t ++p p²
t ++
where y and are the increasing and decreasing factors In Appendix C.I, also examined the test set performance using other kinds of dynamic sequences. Under the above settings, for a given stage k (k- I.... N), with E [(k 1)á, kō 1]. the momentum system becomes:
m, = u + 4k:t
(5)
where ue u(( 1)8) and e2 v((k 1)8) are constants for the duration of the k-th stage. Additionally, we set the total number of stages, denoted by N, to a constant value of 300 for all the experiments in this paper.
2.2 FREQUENCY DOMAIN ANALYSIS OF THE MOMENTUM METHOD
In this subsection, we introduce our frequency domain analysis framework and analyze the impacts of the momentum method on neural network training. We first apply Z-transform, denoted by Z, to Equation 5:
M(=) = u M(2)) + G(=) where G(2) (9.), M(2) Z mm,)). and - (M)) 7(m-11) To obtain the frequency response of the momentum system during stage k. we evaluate the transfer function (x) on the unit circle (Oppenheim et al., 1996):
(6)
H&
(z)
2k
Hn(w) = 1 u.e jw
Ukk
G(2)
where w E [0, π is the normalized angular frequency of the real-value signal. The frequency re- sponse of the momentum system describes how the input gradient signal G(z) is altered to the output momentum signal M (w) when it passes through the system. Note that this transfer function is valid for the entire duration of the k-th quasi-stationary stage.
Note that different from the increasing sequence, the numerator of the decreasing sequence is f f 1. This design avoids the zero gradients the first training stage.
3

---

Published as a conference paper at ICLR 2025
Magnitude Response. The magnitude response of the momentum system in the k-th stage can be calculated by taking the magnitude of Hy(w):
Hy(w)] = VI 2uh COS w + u
(8)
The magnitude response describes the amplitude scaling effect of the system at different frequen- cies. It indicates how the momentum system amplifies or attenuates different frequency components during each stage. This characteristic of the momentum system plays a key role in affecting the optimization process. Notably, when [Hx(w)]
the momentum system attenuates the signals with frequency w; when H& (w)) > 1, the momentum system amplifies the signals with w. Conse- quently, we divide the momentum systems into two categories: Orthodox Momentum Systems and Unorthodox Momentum Systems.
Orthodox Momentum Systems are the ones whose amplitude of the magnitude response will not surpass 1, like the EMA-SGDM (2). This kind of momentum system only shows attenuating char- acteristics. Specifically, the momentum system behaves high-pass filter when My < O. Additionally, when 44 gets close to 1, the momentum system will prefer attenuate the gradient components with high frequencies. The visualization of the (dy- namic) magnitude responses of orthodox momentum systems is in Section 3.1 and Appendix C.2.
low-pass filter when u2 and
For Unorthodox Momentum Systems where the amplitude of magnitude response will surpass 1, such as selecting ", = 0.9 and i = 1 in Standard-SGDM (1), the momentum system possesses both amplifying and attenuating characteristics.
this paper, we refer to these kinds of unortho- dox filters low/high-par passsgghhpssssseeee gain filters. Specifically, the momentum system behaves gain filter when Us > 0, 02- = I and a high-past gain filter when ug < 0, 2 = 1. Additionally, if ug is close 1, the momentum system attenuates high-frequency gradient components while strongly amplifying low- cow-frequency components; if ug is close to 11, the momentum system atten- uates low-frequency gradient components while strongly amplifying high-frequency components. The visualization of the (dynamic) magnitude responses of unorthodox momentum systems is in Section 3.2 and Appendix C.2.
low-pass
To demonstrate the momentum effects from the frequency perspective, in Figure 1, we compare an original sinusoidal signal, a noisy version injected with Gaussian noise, and the signal after applying the momentum method (which is called momentum signal for short) in the time domain. The red curve represents the noisy signal, the black dashed curve corresponds to the original noise-free true signal, and the cyan curve shows the momentum signal. We can see that different selections of and e2 significantly affect the amplifying or attenuating effects of the momentum system.
Hotar
Tre Stgul
Trus Signal
Trai Stgual
(a) Dynamic Low-pass Filter
(b) Dynamic High-pass Filter
(c) Low-pass Gain Filter
(d) High-pass gain Filter
Figure 1: Visualization of different filters towards the noisy sinusoidal signal. (a) uk = 0 Up 1 with the system gradually shifting from an all-pass filter to a narrow low-pass filter; (b) ug, with the system gradually shifting from an all-pass filter to narrow high-pass filter: (c) = 0.9, Ug 1, which indicates the momentum behaves like a low-pass gain filter with amplification on low-frequency gradient components; (d) Uu -0.9, Us = 1, which indicates the momentum behaves like high-pass gain filter with amplification on high-frequency components. The amplifying and attenuating effects of different momentum systems are verified.
ug = 10-1,, 14
Similarly, we also have the phase response of the momentum system (see Appendix A). While the phase response of the momentum only provides limited insights, understanding the behavior of the magnitude response across stages essential for analyzing the time-variant characteristics of the momentum system. By plotting the dynamic magnitude response value [w]) the normalized
4

---

Published as a conference paper at ICLR 2025
angular frequency axis for each stage k, track how the requeney--depender behavior of the multistage momentum system evolves. This provides valuable insights into the amplifying or atten- uating characteristics of the momentum system. Further results the comparisons of momentum systems with different dynamic magnitude responses are presented the next section.
DYNAMIC MAGNITUDE RESPONSE OF THE MOMENTUM SYSTEMS
In this section, we present an empirical study to discover the influence of the momentum coeffi cients by comparing the test performance momentum systems with different dynamic magnitude responses. We train VGG (Simonyan & Zisserman, 2014) on the CIFAR-I0 (Krizhevsky et al., 2009) dataset and ResNet50 (He al., 2016) on the CIFAR- 100 dataset using different momentum coefficients, while keeping all other hyperparameters unchanged. For each experiment, we report the mean and standard error (as subscripts) of test accuracy for 3 with random seeds from 0-2. The detailed experimental settings can be found in Appendix D. The experimental results in CIFAR- 10 show high similarity to those in CIFAR- Thus, here. we mainly focus on the analysis based on CIFAR-100 100 and defer the experimental results of VGG16 on CIFAR-10 in Appendix C.3.
3.1 ORTHODOX MOMENTUM SYSTEMS
3.533
Teepety Insken
(a) Dynamic Magnitude Response
(b) Magnitude Response
(c) Dynamic Magnitude Response
momerian Lant
mwevedem more Guol
gradken narm (andel)
gudken beart
mamenian Lell
www.
awevedenn Guedin
(d) Norm Comparisons
(c) Norm Comparisons
Norm Comparisons
Figure 2: (Up) Analysis of the (dynamic) magnitude responses in the early and late training stages for EMA-SGDM with low-pass momentum defined in Equation 9. The solid lines denote the mag- nitude responses the early stages, and the dashed lines denote the magnitude responses in the late stages. (Down) The comparison between the gradient norms and momentum norms for EMA- SGDM with low-pass momentum. Left Column: increasing sequence. Middle Column: fixed se- quence. Right Column: decreasing sequence.
We first focus the orthodox momentum systems with the following two main types: low-pass and high-pass momentum, defined as:
Low-pass
1 + uz)g:, High-pass m, -0,7,-1 + (1 u. uz)ge.
where i, [0, 1) can be set as increasing, decreasing sequences, or fixed value. For time-variam momentum systems, different strategies of u, result in different time-variant filtering characteris- ties during training. According to Section 2.1, scaling the increasing and decreasing factors af- fects the changing rates of u, In the following, we demonstrate the dynamic magnitude responses, comparisons between gradient norms and momentum norms, and test accuracy results orthodox momentum systems under different u, sequences
Note that selecting / 100 and 10 lead to long stage of the super narrow-band filter. To avoid this problem, we select j 103, 10 10 and 10?, 10 10* in this paper.
5

---

Published as a conference paper at ICLR 2025
Example 1: Low-Pass Momentum. We first explore the effect of increasing, fixed, and decreasing uy sequences in low-pass momentum. Figure 2(a) 2(c) show the corresponding dynamic magnitude responses over time. With increasing ue, the system transits from all-pass to progressively nar- rower low-pass filter, gradually attenuating high-frequency components. Larger results in slower transitions. Decreasing dy shows reverse behavior, with larger resulting in slower transitions. ot with a fixed value maintains a constant filter, with larger leading to more aggressive smoothing and noise reduction characteristics. The norm comparisons in Figure 2(d) 2(1) show that the mo- mentum norms in low- -a-ssss momentum systems are always less than corresponding gradient norms. Larger ue // and smaller j lead to more reduced momentum norms, which validates the timi-variant filtering characteristics of orthodox momentum systems.
Test accuracy results in Table reveal that increasing or fixing 4t can achieve higher accuracy compared to applying decreasing sequences of 44 In particular, momentum systems with proper increasing sequences of uy can outperform those with fixed ue- We also find that larger v results in poorer model performance. These phenomena indicate that gradually attenuating high-frequency components during training improves test set performance, while excessive suppression of low- frequency gradient components in early stages and retention of high-frequency components in late stages degrade model performance.
High-Pass Momentum. High-pass momentum systems exhibit symmetric dynamic magnitude responses and similar norm comparisons, compared to their low-pass counterparts (see Figure 6 in Appendix C.2). With increasing "t, the system shifts from an all-pass to narrow high-pass filter, progressively attenuating low-frequency components. Decreasing sequences act in reverse. Fixed sequences with larger u, lead to more aggressive attenuation of low-frequency com- ponents. The comparison of gradient norms and momentum norms can be found in Appendix C.2.
Example 2:
Test accuracy in Table shows that dynamic high-pass systems with larger μ and smaller y yield better top-1 accuracy performance. When selecting fixed values, momentum systems with larger up perform more poorly. These results confirm that suppressing low-frequency gradient components is harmful. Moreover, high-pass systems generally outperform low-pass systems when applying decreasing strategies with the same v, suggesting that nigh-freueecc components play a crucial role in the early training stages, which is also supported by the studies in Appendix C.4.
From Examples and 2, we empirically verify that high-frequency gradient components are detri- mental in late training stages, while their preservation in early stages leads to higher test accuracy, which matches the viewpoint that gradient noise has generalization benefit early in training (Smith et al., 2020).
Table I: Top-1 ACC. (%) comparisons of different momentum coefficient strategies of orthodox momentum systems of ResNet50 on C00.R-100.
Increusing Factor (μ))
Fixed Value (Mr)
Decreasing Factor (v)
Parameters
lk
10k
100k
0.3
0.6
0.9
100
1k
10k
Low-pass
77. 17.17
77 77.06..14
76.86
76. 78mm
76.82 65.14
76. g6.8
72.58
70.53a.1 75.92..12
69.69a75 70.99
High-pass
51.599.78
67.55 07.55p.22
74.72a.06
72.46
53.43
76.82025
3.2 UNORTHODOX MOMENTUM SYSTEMS
Unorthodox momentum systems allow magnitude responses larger than 1, meaning they can both attenuate and amplify gradients in different frequency bands. We focus on two main types: low-pass gain and high-pass gain momentum, defined as:
Low-pass gain m. = 4 + ge, High-pass gain my = --,77,1-- + St:
(10)
where E [0, 1) can follow increasing, fixed, decreasing sequences. For simplification reasons, we use the PoTorch setting with 8, = 1. We show the dynamic magnitude responses, comparisons between gradient norms and momentum norms, and test accuracy results of unorthodox momentum systems under different it sequences as follows.
Example 3: Low-Pass Gain Momentum. In low-pass gain momentum, the system transits from
an all-pass to a narrower low-pass gain filter as u, increases, amplifying frequency components while attenuating high-frequency components. Figure 3(a) 3(() show the corresponding dynamic

---

Published as a conference paper at ICLR 2025
13
(a) Dynamic Magnitude Response
(b) Magnitude Response
(c) Dynamic Magnitude Response
UMGID moreemun nernig TeD)
(UHEE)) mar ETEAT Femmite BUT ma
grod en nom tamcods
gIn BoY tel:
mor ennur perignn
20m
C6
150
122
(d) Norm Comparisons
Norm Comparisons
Norm Comparisons
Figure 3: (Up) Analysis of the (dynamic) magnitude responses in the early and late training stages for Standard-SGDD with low-pass gain momentum defined in Equation 10. The solid lines denote the magnitude responses in the early stages, and the dashed lines denote the magnitude responses in the late stages. (Down) The comparison between the gradient norms and momentum norms for Standard-SGDM with low-pass gain momentum. Left Column: increasing sequence. Middle Column: fixed sequence. Right Column: decreasing sequence.
magnitude responses over time. A large μe corresponds to the slow shifts. Decreasing reverses the trend, heavily amplifying low-frequency components early and relaxing this effect over time. Fixed maintains constant filters, in which larger ue amplifies low-frequency components more aggressively. Figure 3(d)- 3(f) demonstrate larger momentum norms compared to gradient norms, indicating the amplification effects in gain filters. Larger u, and smaller yu lead to more reduced momentum norms, which validates the time- variant filtering characteristics of orthodox momentum systems. Test results in Table 2 indicate that increasing with appropriate μ outperforms the sce- narios using fixed and decreasing sequences of ue, We also find that smaller v yields worse accuracy in test sets. From these results, we conclude that amplifying low-frequency gradient components and properly attenuating
ones, improves test set performance.
Example 4: High-Pass Gain Momentum. High-pass gain momentum mirrors the dynamic magni- tude response behavior of low-pass gain systems (see Figure in App C.2). Increasing u, gradually amplifies tigh-freeuency gradient components and attenuates low-frequency ones. Decreasing и4 reverses this pattern, heavily amplifying high-frequency components early on. Fixed constructions more aggressively amplify high-frequency components for larger 14, The comparison of gradient norms and momentum norms can be found in Appendix C.2. Test accuracy in Table 2 shows that fixed constructions with larger u, and decreasing u, with larger u perform worse. These findings confirm that amplifying nigh-frequency gradients in training might be undesirable.
From Examples 3 and 4, we empirically verify that proper amplification in unorthodox momen- tum systems can improve model performance, particularly when amplifying low-frequency gradient components.
Table Top-1 ACC. (%) comparisons of different momentum coefficient strategies of unorthodox momentum systems of ResNet50 on C00AR-100.
Increasing Factor (I))
Fixed Value (u²)
Decreasing Factor (v)
Parameters
Ik
10k
100k
0.9
100
10k
Low-Pass Gain
80.48pm 75.47p21
16.10
78.02ao3
78.00 78.01o.4
79.51o.15
79.71
70.37
18.18 74.00
71.53
High-Pass Gain
74.54
75.97am
75.68p.18
74.56m.00
73.77au
76.41p.1
68 88.90p82

---

Published as a conference paper at ICLR 2025
3.3 DISCUSSION
The differences in norm comparison and test accuracy between orthodox and unorthodox momentum systems validate the distinctions between EMA-SGDM and Standard-SGDM, While EMA-SGDM possesses attenuating filter effects, Standard-SGDM can both amplify and attenuate different fre- quency gradient components. Moreover, our findings indicate that with appropriate momentum coefficients, Standard-SGDM consistently outperforms EMA-SGDM, showing the advantages of decoupling momentum coefficients, which answers Question 1.
Regarding Question 2, the test results show that decoupled momentum coefficients with a properly increasing U, and fixed D2 can achieve better performance. In particular, our empirical findings re- veal the following insights in training convolutional neural networks (CNNs): (1) gehhfrreueecy gradient components are undesired in the late stages of training; (2) preserving the original gradi- ent the early stages leads improved test set accuracy; (3) gradually amplifying low- owwfirquency gradient components enhances performance. Furthermore, we find that these insights are also adapt- able in various learning areas (see Section 5). Based on these insights, it may be possible to design a more effective optimizer by appropriately adjusting the momentum coefficients.
4 FREQUENCY-BASED OPTIMIZER
As suggested by our frequency domain analysis framework, achieving better test performance is equivalent to finding an appropriate dynamic filter-changing pattern for momentum systems. Based on this idea, propose FSGDM, heuristic optimizer that dynamically adjusts momentum filtering characteristics. Furthermore, to explore the potential optimal strategies of our proposed FSGDM based on the findings in Section 3.3, several sets of experiments in various deep-learning tasks are conducted.
4.1 FREQUENCY STOCHASTIC GRADIENT DESCENT WITH MOMENTUM
Generally, determining the best optimization strategy by tuning H2 and it according to our frequency domain analysis challenging. In the field of signal process- ing. how to select the best filters for different problems still an open problem. However, we can design a bet- ter optimizer based on the findings in Section 3.3. Still, there are infinite dynamic magnitude responses that can meet the requirements of the aforementioned findings. Based on Occam's Razor principle, we provide a mini- malist form of our proposed optimizer in Algorithm 1, where Σ is the total gradient update steps the whole training process determined by the epoch number and the size of the dataset, c is a scaling factor, L. : IRd - R is the loss for the f-th step, Ç--1 denotes minibatch drawn from the training data, and N is the number of stages. μ and parameters that dominate the filtering characteristic of FSGDM. Moreover, since is a function of Σ, the dynamic magnitude response can be inherited when Σ varies. In particular, we have the following proposition.
Algorithm 1: FSGDM
Input: Σ, C, N;
Initialization: mo. p = cC.
5 5= E/N;
for each 1..,,... do
u(t) zta* ul [1//8 x8);
m u Ugti
It -t-1 - a,m,i
end
adjustable
Proposition 1. By fixing the number of stages N and the scaling factor e, the dynamic magnitude response of Algorithm keeps invariant with respect to changes the total number of training steps.
The proof of Proposition deferred Appendix B.3. By this, we show that the dynamic mag- nitude response of well-performed FSGDM can be adaptable various tasks. In the following subsection, we explore the optimal scaling factor e and momentum coefficient for FSGDM.
4.2 EMPIRICAL EXPLORATION OF OPTIMAL SETTINGS FOR FSGDM
As discussed in Section 3, different choices of e and significantly affect the filtering character- istics of FSGDM. To understand their impact on optimization performance and to identify optimal parameter settings, conduct comprehensive empirical study.
8

---

Published as a conference paper at ICLR 2025
Specifically, empirically explore the optimal parameter selection of FSGDM across three dif- ferent image classification tasks by first sweeping and v within the ranges (0, 1) and [0.5, 3], respectively. Specifically, conduct three sets of experiments using the same codebase (See Ap- pendix D for more training details): (1) training ResNet18 for 100 epochs on CIFAR- 10, (2) training ResNet34 for 100 epochs Iiny-ImageNer (Le & Yang, 2015), and (3) training ResNet50 for 300 epochs on CIFAR-100. We also explore the optimal parameter selection on one natural language processing task in Appendix C.7. By finding the parameter selections with better test performance in different tasks, we try to empirically summarize the law of optimal parameter selection.
CIAALE- WEE
They mag.hatteeeooomm..m Ne1
CHAR Resie550
23-
Optimal Zone
Op timal Zone
Optimal Zone
Manuruu Oxe Pobooi
Coe Coefidorr
528597
5.234
334,42
6.823
32.074
28383
S3.487
Jeass
Scoling Facterc
Scalin TaItEr
Scaling Factar
Figure 4: The Top-1 test errors of training ResNet18 CIFAR-10, ResNer34 on Tiny-ImageNet and ResNet50 on CIFAR-I00. The results show that the optimal parameter selections across these three training settings exhibit high similarity. The black points denote the parameter selections with better test performance. The optimal zone of the parameter selection is circled red.
The results in Figure 4 show that there exists optimal zone where relatively better test accuracy results can be achieved. When the momentum coefficient is fixed, the accuracy shows an initial increase followed by decline as the scaling factor increases. In Appendix C.8, we plot the magnitude responses and the test accuracy results of the black points in Figure and find that these parameter selections have similar dynamic magnitude responses and test accuracy curves. Thus, we assume the parameter selections with similar dynamic magnitude responses will lead to close performance. More discussions are in Appendix C.8.
EXPERIMENTS
To verify the generalization of the proposed FSGDM, we perform a large-scale comparison across vision classification tasks, natural language processing (NLP) tasks, and reinforcement learning (RL) tasks. We compare the test performance of FSGDM and conventional SGD-based momentum optimizers, including Standard-SGDN, and EMA-SGDM. We set SGDM, and 4, = 0.9 for EMA-SGDM, which are the common momentum coefficient selections in training neural networks. For fair comparison and convenience, we one of the black points in the optimal zone Figure 4, for FSGDM. Note that other combinations of and the optimal zone can also be selected. For the other adjustable parameters in Algorithm we set N to 300 as mentioned at the end of Section 2.1, and set Σ as the number of total training steps. Notably, since our focus is on comparing the performance of different optimizers, do not fine-tune every parameter for each model but use the same hyperparameters across all models for convenience. See Appendix D for more experimental details.
0.9, V = 1 for Standard-
0.033,
which is
Table 3: Performance on Image Classification Experiments
Dataset Model
CIFAR-10
CIFAR-I00
Tiny-ImageNet MobileNet
ImageNet ResNet50
VGG16
ResNet18
ResNet50
DenseNet121
ResNet34
EMA-SGDM
93.71pm
94.19
76.84no6
78.18
62.28n.17
55.00u.m
74.24mm
Standard-SGDM
94.08pm
95.570.6
79.71a2s
80.49h.9
67.51nms
76.660
FSGDM
94.19
95.660.07
81.44006
81.14
67.74..0
59.61om
76.91eos
Image Classification. We perform four sets of experiments with different datasets in computer vi- sion tasks and use various CNN architectures for training them. Specifically, select: (a) VGG16 and ResNet18 for CIFAR-10; (b) ResNet50 and DenseNet121 (Huang et al., 2017) for CIFAR- (c) ResNet34 and MobileNet (Howard, 2017) for Tmny-Ima (d) ResNet50 for ILSVRC 2012

---

Published as a conference paper at ICLR 2025
ImageNet Russakovsky al. (2015). For each task, report the mean and standard error (as subscripts) of test accuracy for 3 runs with random seeds from 0-2. The results in Table 3 show that our FSGDM consistently achieves better test set performance. Additionally, we observe that Standard-SGDM steadily outperforms EMA-SGDM, which aligns with our discoveries in Sec- tion 3.3.
Natural Language Processing. We conduct experiments on the IWSLT14 German-Engling trans- lation task (Cettolo et al., 2014) to represent NLP tasks, widely used benchmark in the community. Specifically, we train six different models encompassing a variety of architectures: two convolution- based models, FConv (Gehring et al., 2017) and LightCony (Wu al., 2019); two LSTM-based models, vanilla LSTM (Hochreiter & Schmidhuber, 1997) and LSTM-W (Wiseman & Rush, 2016); and two Transformer- based models (Vaswani et al., 2017) with different sizes, Transformer-tiny and Transformer. Model performance is reported using BLEU scores, where higher scores indicate bet- ter performance, and we summarize all results in Table 4. Compared with the baseline optimizers, FSGDM outperforms all others in this task across six different models. This shows the effectiveness of our optimizer improving translation quality. Moreover, the consistent improvement highlights the robustness of FSGDM and its ability to generalize across different neural network structures in natural language processing tasks.
Table 4: Performance IWSLT14 Dataset
Model
FConv
LightConv
LSTM
LSTM-W
Trunsformer-tiny
Transformer
EMA-SGDM Standard-SOii SGDM
13.97p.01
10.56..01
4, 4.99p.0
1.20p
5.17
6.27p.01
27. 27.4
33.05om:
28. 120.06
24.66..0
18. 16.03
31.500.05
FSGDM
28.30n.01
33.44a.uz
29.27
27.41a.u
19.94au7
32.40n.a5
Hallhhatth-ol
Ant-ot
9000
5000
EMA SGOM
EMA EMA-3CDM
EMA--BDMM
RS0DM
SSCD
8000
FaGDM
5000
Standieni-SCio
4000
Estcado Rowadd Roward
2540
3000
2003
1200
2000
1003
1000
3.M
3.15
410
3.1
3.14
Timesstegn
Timestips
Timesdeggs
Figure 5: The reward curves of EMA-, Standard-SGDMM, and FSGDM on three MuloCo tasks.
Reinforcement Learning. We evaluate FSGDM on PPO (Schulman et al., 2017), of the most popular policy gradient methods in reinforcement learning. We replace the default Adam opti- mizer (Kingma & Ba, 2014) in PPO with FSGDM, Standard-SGDM, and EMA-SGDM. We test the three optimizers on Walked2d-v4, HalfCheetah-v4, and Ant-V4, which are continuous control environments simulated by the standard and widely-used engine, MuloCo (Todorov et al., 2012). Following standard evaluation, we run each game under 10 random seeds (range from 0-9) and test the performance for 10 episodes every 30,000 steps. All experiments are conducted using the Tian- shou codebase (Weng et al., 2022), widely known RL framework. Figure 5 presents the results on three tasks, where the solid line represents the average episode rewards during evaluation, and the shaded region indicates the 75% confidence interval. It can be easily observed that on three test games, our FSGDM achieves higher rewards than Standard-SGDD and EMA-SGDM.
CONCLUSIONS
This paper proposes a frequency domain analysis framework for the momentum method. Based on the proposed framework, we find that different selections of momentum coefficients correspond to different filter characteristics of the momentum methods. Performance will be significantly different under different time-varian momentum coefficients. Furthermore, we develop a heuristic optimizer named FSGDM which outperforms the conventional SGD-based momentum optimizers in various learning tasks. Future work may explore the best filtering strategy for all general scenarios and extend the frequency domain analysis framework to other optimizers such Adam.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We would like to especially thank Prof. K. C. Ho for many enlightening discussions. The work of Xianliang Li and Sheng Xu was supported by the National Natural Science Foundation of China (62273327) and the Shenzhen Science and Technology Program (KCXFZ2021 1020165003005), The work of Linlong Wu was supported by Luxembourg FNR CORE METSA project C22/1S/13311632.
REFERENCES
N. S. Aybat, A. Fallah, M. Gurbuzbalaban, and A. Ozdaglar. A universally optimal multistage accelerated stochastic gradient method. Advances Neural Information Processing Systems, 32, 2019.
M. Cettolo, J. Niehues, S. Stüker, L. Bentivogli, and M. Federico. Report the 11th IWSLT evaluation campaign. In Proceedings of the 11th International Workshop on Spoken Language Translation:
Evaluation Campaign, pp. 2-17, 2014.
X. Chen, S. Liu, R. Sun, and M. Hong. On the convergence of class of Adam-type algorithms for non-convex optimization. International Conference Learning Representations, 2018.
A. Cutkosky and F. Orabona. Momentum-based variance reduction in non-convex SGD. Advances in Neural Information Processing Systems, 32, 2019.
E. S. Gardner Jr. Exponential smoothing: The state of the art Journal of forecasting, 4(1):1-28, 1985.
Gehring, M. Auli, Grangier, D. Yarats, and N. Dauphin. Convolutional sequence to sequence learning. In International Conference Machine Learning, pp. 1243-1252. PMLR, 2017.
G. Goh. Why momentum really works. Distill, 2017. doi: 10.23915/distil..000
K. He, X. Zhang, S. Ren, and Sun. Delving deep into rectifiers: Surpassing human-level per- formance imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1026-1334, 2015.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Hochreiter and J. Schmidhuber, Long short-term memory. Neural Computation, 1997.
A. G. Howard. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.04861 2017.
G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700-4708, 2017.
Hubner and P. Tran-Gia. Quasi-stationary analysis finite capacity asynchronous multiplexer with modulated deterministic input. ITC-13, Copenhagen, 1991.
E. I. Jury. Theory and application of the z-transform method. Electronics and Power, 11(3):291-299, 1964.
R. Kidambi, P. Netrapalli, P. Jain, and Kakade. On the insufficiency of existing momentum schemes for stochastic optimization. In 2018 Information Theory and Applications Workshop (ITA), pp. 1-9, IEEE, 2018.
D. P. Kingma and J. L.. Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2014.
A. Krizhevsky, G. Hinton, et al. Learning multiple layers features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
11

---

Published as a conference paper at ICLR 2025
A. Kulunchakov and J. Mairal. A generic acceleration framework for stochastic composite opti-
mization. Advances in Neural Information Processing Systems, 32, 2019.
Y. Le and X. Yang. Tiny imagenet visual recognition challenge. CS23IN, 7(7):3, 2015.
X. Li, M. Liu, and F. Orabona. On the last iterate convergence of momentum methods. In Interna- tional Conference on Algorithmic Learning Theory, pp. 699-717. PMLR, 2022.
T. Liu, Z. Chen, E. Zhou, and T. Zhao. A diffusion approximation theory of momentum SGD in nonconvex optimization, arXiv preprint
2018.
Y. Liu, Y. Gao, and W. Yin. An improved analysis of stochastic gradient descent with momentum. In Advances Neural Information Prucessing Systems, volume 33, pp. 18261-18271, 2020.
I. Loshchilov and Hutter. SGDR: Stochastic gradient descent with warm restarts. International Conference Learning Representations, 2016.
L. Luo, Y. Xiong, Y. Liu, and X. Sun. Adaptive gradient methods with dynamic bound of learning rate. International Conference on Learning Representations, 2019.
J. Ma and D. Yarats. Quasi-hyperbolic momentum and Adam for deep learning. In International Conference on Learning Representations, 2018.
V. Mai and M. Johansson. Convergence of stochastic gradient method with momentum for non- smooth non-convex optimization. In International Conference on Machine Learning, pp. 6630- 6639. PMLR, 2020.
Y. Nesterov. A method for solving the convex programming problem with convergence rate o (1/k2). In Dokl Akad Nauk SSSR, volume 269, pp. 543, 1983.
A. V. Oppenheim, A. S. Willsky, and S. H. Nawab. Signals & systems (2nd ed.). Prentice-Hie Hall, Inc., USA, 1996. ISBN 0138147574.
A. Paszke, S. Gross, Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, perf-macformance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
B. Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computa- tional Mathematics and Mathematical Physics, 4(5):1-17, 1964. ISSN 0041-5553.
J. Reddi, S. Kale, and S. Kumar. On the convergence of Adam and beyond. In International Conference on Learning Representations, 2018.
H. Robbins and S. Monro. A stochastic approximation method. The Annals of Mathematical Statis- tics, pp. 400-407, 1951.
D. E. Rumelhart, G. Hinton, and R. J. Williams. Learning internal representations by error prop- agation, parallel distributed processing, explorations in the microstructure of cognition, ed. de rumelhart and j. mcclelland. vol. 1. 1986. Biometrika, 71(599-607):6, 1986.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 1 115:211-252. 2015.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXivv1707.06331 2017.
K. Simonyan and Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: arXiv:1409.1 1409. 1556, 2014.
L. N. Smith. A disciplined approach to neural network hyper-paramettrrs Part --learning rate, batch size, momentum, and weight decay. arXiv preprint arXiv:1803.09822 2018.
S. Smith, Elsen, and S. De. On the generalization benefit of noise in stochastic gradient descent.
In International Conference on Machine Learning, pp. 9058-9067. PMLR, 2020.
12

---

Published as a conference paper at ICLR 2025
1. Sutskever, J. Martens, G. Dahl, and G. Hinton. On the importance of initialization and momentum in deep learning. In International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 1139-1147. PMLR, 2013.
E. Todorov, T. Erez, and Y. Tassa. MuJoCo: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, IEEE, 2012.
B. Van Scoy, R. A. Freeman, and M. Lynch. The fastest known globally convergent first-order method for minimizing strongly convex functions. IEEE Control Systems Letters, 2(1):49-54, 2017.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, t. Kaiser, and Polo- sukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
R. Wang. S. Malladi, T. Wang, Lyu, and Li. The marginal value of momentum for small learning rate SGD. In International Conference on Learning Representations, 2024.
J. Weng, H. Chen, D. Yan, K. You, A. Dubureq, M. Zhang, Y. Su, H. Su, and J. Zhu. Tianshou: A highly modularized deep reinforcement learning library. Journal of Machine Learning Research, 23(267): 2022.
S. Wiseman and A. M. Rush. Sequence-to-seuee torticee-t--ceeeee to-seuennee learning beam-search optimization. Pro- ceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016.
F. Wu, A. Fan, A. Baevski, Y. N. Dauphin, and M. Auli. Pay less attention with lightweight and dynamic convolutions. International Conference on Learning Representations, 2019.
Y. Yan, T. Yang. Z. Li, Q. Lin, and Yang. A unified analysis of stochastic momentum methods for deep learning. Proceedings of the Twenty-Seventh International Joint Conference Artificial Intelligence, 2018
H. Yu, R. Jin, and S. Yang. On the linear speedup analysis of communication efficient momentum SGD for distributed non-convex optimization. In International Conference Machine Learning, pp. 7184-7193. PMLR, 2019.
L. A. Zadeh. Frequency analysis of variable networks. Proceedings of the IRE, 38(3):291-299, 1950.
Zadeh. Time-varying networks, Proceedings of the IRE, 49(10): 1961.
13

---

Published as a conference paper at ICLR 2025
A PHASE RESPONSE
The phase response of the momentum system in the k-th stage can be written as,
arg (H)(H))W) = arg(m*) tan (1
Ug sin w
(11)
BA cosw
where arg(.) .) the argument operator. For any real value Uk. arg(u)) = if un > 0 and arg (vx) = π Uy 0; for any E [0, π and uy E (-1, 1), tan (ug sin w/ ug cos w)) E (- 3). The phase response describes the phase-shifting effect of the momentum system at different frequencies. In the context of gradient-based optimization, the phase shift indicates change in the optimization direction. Therefore, when < 0, the phase shift of the momentum adds up an extra rad on the shifted direction, indicating that the direction of the update is greatly reversed, which can lead to oscillations, instability, or divergence in the optimization process. Thus, it is necessary to select positive us when applying momentum methods.
B ADDITIONAL DERIVATIONS AND PROOF
B.1 DERIVATION OF EQUATION
HH((W) V (H
Uk
2k
-ue
ju
uge
use
ju
use
uhe -jwejw
up (cos w sin - us (cosw + fososw+jsin jisnw) + ug(coss w + +in? w)
2uh COS w + u
2uz COS w + u
B.2 DERIVATION EQUATION 11
arg(H)(w)) arg(uk) arg(1 use -w)
arg(Uu)) arg ((1 un cos w) + j(u sin
arg(U)) - tan -1 ug sin w
Uy COS
B.3 PROOF OF PROPOSITION
According to Algorithm 1, the momentum coefficient in the k-th stage (k: 1.2,.... ,N)
(k: 1)8
(k: 1)8
(k: 1)8
k- 1
u4
(12)
(k: - 1)6
(k (k- 1)6 E/c (A: - 1)6 cNo
k: k cN
This guarantees that the number of training steps, which may be different when choosing other training strategies or changing datasets, independent of 24 when the scaling factor c and the number of stages N are already determined.
14

---

Published as a conference paper at ICLR 2025
C ADDITIONAL EXPERIMENTS
In this section, present several supplementary experiments. The detailed experimental settings are shown Appendix D.
C.I DYNAMIC SEQUENCE CONSTRUCTION
There are infinite increasing decreasing sequences. In this part, compare the test set perfor- mance of the sequence mentioned in Equation with four other dynamic sequences. Specifically, compare with the following four dynamic increasing sequences within Algorithm 1:
Linear:
uu()= = a1d;
Exponential: u(I) = =1
Sine: ((t) sin (azt);
Logarithmic: u(t) In(a,t);
where to a4 are scaling coefficients. For fair comparison, we adjust the coefficients to keep the u, of all sequences unchanged in the beginning and ending stages. To make other types of sequences unique, we keep the of different dynamic sequences nearly unchanged in the beginning and ending stages. Table displays their test accuracy results after 300 epochs of training on CIFAR- 100 using ResNet50. We ran each experiment under different random seeds (0, 1, 2). Clearly, the dynamic sequence we use in Equation 4 shows its superiority over other constructions.
Table 5:
Top-1 ACC. (%) comparisons of using linear, exponential, sine, logarithmic, and quences when adopting FSGDM.
Dynamic Sequence Type
Ours
Linear
Exponential
Sine
Logarithmic
ACC-1 (%)
81.440.06
78.24b,1
80.38..04
78.76p29
78.70p.09
Specifically, (a1, 02 az 01) = (8.271 x 10 6, P,3.793 x 10 5, 1.125 x 10 9.1994 x 10 $).
C.2 ADDITIONAL FIGURES OF HIGH-PASS MOMENTUM SYSTEMS ON CIFAR- 100
This subsection provides the figures of the dynamic magnitude responses and norm of high-pass (gain) momentum systems mentioned Section Figure and Figure 7 show the magnitude re- sponses and norm comparisons of high-pass and high-pass gain momentum systems, respectively. The high-pass (gain) momentum systems preserve or even amplify rapidly fluctuating gradient com- ponents, leading to sharp oscillations in gradient norm curves and momentum norm curves across iterations.
C.3 ADDITIONAL EXPERIMENTS OF VGG16 CIFAR-10
In this subsection, provide experiments of training VGG16 CIFAR-10. The experimental settings follow Section 3 and Appendix D. From the test accuracy in Table and Table 7, we ob- serve that the test performances and norm comparisons in different momentum methods in training VGG16 on CIFAR-10 are similar to those in training ResNet50 on CIFAR-100. This similarity implies that the empirical findings in Section 3 applicable various CNNs.
Table 6: Comparison of Top-1 Accuracy (%) among different momentum coefficient methods in orthodox momentum systems using VGG16 on CIFAR-10.
Increasing Factor (μ)
Fixed Value (wz)
Decreasing Factor (2)
Parameters
1k
10k
100k
0.3
0.6
0.9
100
Ik
10k
Low-Pass
93.78u.12 93.80
93.79k.9
03.680.18
93. 93.64p.0s
92.33a.04 93.7
90. 01.89
90.56p.1
High-Pass
90.02ans
92.64p.0
93.41nml
93.52n.16
92.71m.07
90.32p.07
93.86n.00
93.73p.08
93.38om
15

---

Published as a conference paper at ICLR 2025
Tripunccy (ndann
Fiequrny F160041
(a) Dynamic Magnitude Response
(b) Magnitude Response
(c) Dynamic Magnitude Response
sradee svre Gested
gladkct mare laminng
cosdeer sare Gosted sertlee
mamerion (antol) pork (1-3et
nameraan
200
(d) Norm Comparisons
Norm Comparisons
(1) Norm Comparisons
Figure 6: (Up) Analysis of the (dynamic) magnitude responses in the early and late training stages for EMA-SGDM with high-pass momentum defined in Equation 9. The solid lines denote the mag- nitude responses in the early stages, and the dashed lines denote the magnitude responses in the late stages. (Down) The comparison between the gradient norms and momentum norms for EMA- SGDM with high-pass momentum. Left Column: increasing sequence. Middle Column: fixed sequence. Right Column: decreasing sequence.
Table 7: Comparison of Top-1 Accuracy (%) among different momentum coefficient methods in unorthodox momentum systems using VGG16 on CIFAR-10.
Increasing Factor (1))
Fixed Value (14)
Decreasing Factor (v)
Parameters
1k
10k
100k
0.9
10k
Low-Pass Gain
84.01o.1
94.19q.m
93.85
93 93.86
93.98u.09
94.08
92.00gas
92.27a.12
92.97o25
High-Pass Gain
93.340.3
93.56p.06
93.79
93.71o.1
93.466.06
93.33002
93. 33.79..07
93.330.12
93.05
C.4 THE EARLY STAGES OF TRAINING
This subsection focuses the test performance affected by the momentum coefficients in the very early training stages. We plot the test accuracy curves for the first 10 epochs of different momentum systems in Section 3 and study the early behaviors different momentum systems.
Figure 8 demonstrates the early test accuracy curves of different momentum coefficient methods. For orthodox momentum systems, preserving the original gradient (i.e., all-pass momentum sys- tem, low-pass momentum system with an increasing and high-pass momentum system with an increasing 4z) or attenuating high fighffreyeeccy gradient components(i.e., static low-pass momentum system with u, = 0.9) results in better initial performance, while greatly attenuating high- high-frequency gradient components (i.e., low-pass momentum system with decreasing uz) or attenuating low- pass components (i.c., static high-pass and high-pass momentum system with a decreasing u,) lead to bad test performance at the beginning.
On the other hand, for unorthodox momentum systems, preserving the original gradient (i.e., all- pass momentum system, low-pass gain momentum system with an increasing u, and high-pass gain momentum system with increasing uz) achieve better early performance, while greatly amplifying high-frequency gradient components (i.e., static high-pass gain momentum system and high-pass gain momentum system with decreasing wx) leads to bad initial accuracy results.
16

---

Published as a conference paper at ICLR 2025
Trqpancc (naland
an (oaliee
Fiequrey F100e41
(a) Dynamic Magnitude Response
(b) Magnitude Response
(c) Dynamic Magnitude Response
(irk.e mommetiin bomm Dast
srodeer sare Gosted: semiloeidd
tacch
20
(d) Norm Comparisons
Norm Comparisons
(1) Norm Comparisons
Figure 7: (Up) Analysis of the (dynamic) magnitude responses in the early and late training stages for Standard-SGDD with high-past pass gain momentum defined Equation 10. The solid lines denote the magnitude responses in the early stages, and the dashed lines denote the magnitude responses in the late stages. (Down) The comparison between the gradient norms and momentum norms for Standard-SGDM with high-pass gain momentum. Left Column: increasing sequence. Middle Column: fixed sequence, Right Column: decreasing sequence.
trapemseed
Greenn poled
Ireenn Cer
41
Alamiset
seted
Test Accuracy (15)
pin miee Alam
28
Epoch
Epoch
(a) Orthodox Momentum Systems
(b) Unorthodox Momentum Systems
Figure 8: The first 10 epochs of the test accuracy curves with different momentum coefficient meth- ods. We choose 1 for both increasing and decreasing factors (μr and v) in dynamic momentum systems and = 0.9 for static momentum coefficient.
These observations significantly validate that preserving the original gradient in early stages en- hances test performance, which matches the findings in Section 3. Additionally, our proposed FS- GDM retains the all-pass characteristic and possesses the same quick start property in test accuracy curves.
17

---

Published as a conference paper at ICLR 2025
Compurts
(a) LP2HP
(b) HP2LP
(c) LPG2HPG
(d) HPG2LPG
Figure 9: The magnitude response curves of Stage 1, 150, 300 in different momentum systems.
C.5 COMPARISON WITH SPECIAL MOMENTUM SYSTEMS
In this subsection, we investigate the test performance of the following four types of momentum systems: 1) low-pass to high-pass momentum system (LP2HP); 2) high-pass to low-pass momen- tum system (HP2LP); 3) low-pass gain to high-pass gain momentum system (LPG2HPG); 4) high- pass gain to low-pass gain momentum system (HPG2LPG). Their dynamic magnitude responses are shown in Figure 9. Note that the maximum values H (w)) | of these four systems are the same as the default setting in FSGDM. We each experiment under 3 different random seeds (0-2). Table 8 displays the test accuracy results of four types of momentum systems and FSGDM. Our proposed FSGDM outperforms all four special momentum systems. Specifically, the test accuracy of the mo- mentum systems shifting from high-pass to low-pass is better than that shifting from low-pass to high-pass. This indicates that compared to the low-frequency gradient components, high-frequency components are more undesired in the late training stages, which supports the finding in Section 3.
Table 8: Comparison of Top-1 Accuracy (%) among the low-pass to high-pass, high-pass to low- pass, low-pass gain to high-pass gain, high-pass gain to low-pass gain momentum systems and FSGDM.
Dynamic Magnitude Response
FSGDM
LP2HP
HP2LP
LPG2HPG
HPG2LPG
ACC-I (%)
81.44p.06
74.77pa1
77.00 77.00m13
72.60a.8
78.91
C.6 TRAINING WITH EXTREME MOMENTUM CORFFICIENTS
Fregancy tradier
Freguncy
Pretuantyy tedkeess
(a) Extreme in EMA Setting
(b) Extreme with
Extreme & with We 0.9
Figure 10: The magnitude responses of different u, and with extreme value ranges. (a): EMA- SGDM; (b), (c): Standard-SGDM.
Why do researchers usually choose u, 0.9 or v; instead larger values? From the frequency domain perspective, we discover that 1) when is extremely close to 1 in EMA-SGDM, the mo- mentum system will behave like a super narrow low-pass filter, with an extreme reduction in most of the fighfrreuencc gradient components; 2) when is extremely close to 1 in Standard-SGDM, the momentum system will behave like a super narrow low-pass gain filter, with a reduction high- frequency gradient components and high amplification in narrow band of oow-frequncyy gradient
18

---

Published as a conference paper at ICLR 2025
components, 3) when i is larger than 1 in Standard-SGDM, the attenuation of high-frree high-frequency gra- dient components then reduced. We speculate that all these poor filtering characteristies of the momentum systems will lead to bad test performance. Figure 10 displays the magnitude response of these three situations. As shown in Figure 11, the test performance results validate our previous speculations and support our frequency domain analysis framework.
Looch
(a) Extreme in EMA Setting
(b) Extreme with 12
Extreme vs with de 0.9
Figure 1I: The test accuracy curves of different ue and By in extreme value ranges. (a): EMA-
SGDM; (b), (c): Standard-SGDM.
C.7 ADDITIONAL EXPLORATION OF OPTIMAL SETTINGS FOR NLP TASKS
In this subsection, provide experiments that explore the optimal parameter selection of FSGDM for the IWSLT14 translation task by training LSTM-W and Transformer-tiny The experimental settings follow Section and Appendix D.
3.0
Optimal Zone
Optimal Zone
2.5-
25.843
17.00
2.0
24.817
17.005
20.031
18:221
Vereitmm
21.612
20.
12 86
20.030
0.5
0.51
61
0.01
63
Soa lis Factor
Scaling
Figure 12: The BLEU scores of training LSTM-W and Transformer-tiny IWSLT14 German- English translation task. The results show that the optimal parameter selections across these two training settings exhibit high similarity. The black points denote the parameter selections with better test performance. The optimal zone of the parameter selection is circled in blue.
The results Figure 12 indicate that similar optimal zones can be observed on the NLP task. When the momentum coefficient is fixed, the BLEU score shows an initial increase followed by decline as the scaling factor increases, which highly consistent with the results in Section 4.2. In addition, we find that the empirical insights discussed in Section 3.3 are also applicable to various deep learning models beyond CNNs, as well as NLP tasks.
C.8 OPTIMAL ZONE OF FSGDM
In this subsection, we go deeper into the optimal zone. We suspect that the similarity of the dynamic magnitude responses may lead to close test set performance. The dynamic magnitude responses of the black points with different parameters in the optimal zone (Figure are shown in Figure 13. We train ResNet50 on CIFAR- 100 and visualize the training losses and the test accuracy curves of different points in the optimal zone. The results shown in Figure 14.
19

---

Published as a conference paper at ICLR 2025
2ags
Sege
2agam
2age - 100
2egt 200
Sage 50
2ags
2ege
2ags
2age -20
Z0ge 220
Sage 220
Zage
Sage
Zage
Ireguncy (0a8004
(a) c= 0.016, 0.5
(b) c c= 0.033, U 1.0
(c) c = 0.051, v 1.5
Sage m
2age -
2age
2ags
2ege
2aga
2age 156
2oge 200
2age 150
2ags
2ags
2apa
Sage 300
2uge 300
Sage 300
Tropeteral Iiakan
0.069, 2.0
0.088, 2.5
0.107, 3.0
Figure 13: The dynamic magnitude responses of the black points in the optimal zone.
2-0.107 v-0.0
v=0
v-0
1.0.051v=11
Training Loss
v=0.033.v=1.0
v
Test Ncaraccaaa
- V= (81 20%
(80 98%)
v- (81.215)
0.051, (81 (81.70%) 70%]
0.016 - 0.5 (81 35%)
Epoch
Epech
(a) Training Losses
(b) Test Accuracy
Figure 14: The training losses and test accuracy different parameter settings in the optimal zone.
From the training loss and accuracy curves, we find that the optimization processes of different black points in the optimal zone resemble each other. According to the existing parameter settings of the black points, one can find that the mathematical relationship between e and in training ResNet50 on CIFAR-100 is approximately 30.992 R 11++
C.9 ABLATION STUDY DIFFERENT BATCH SIZE
This subsection provides the ResNet50 training experiments on CIFAR- with different batch size settings. We compare the Top-1 accuracy of the test set by using our FSGDM with 0.033, 1, Standard-SGDM with u, 0.9, v 1, and EMA-SGDM with u, = 0.9, as shown Table 9. The test results show that our FSGDM consistently outperforms popular conventional SGD-based momentum optimizers.
"This relationship can be better approximated and generalized with continued experimentations across di- tasks.
20

---

Published as a conference paper at ICLR 2025
Table 9: Comparison of Top-1 Accuracy (%) among the FSGDM, Standard-SGDM, and EMA- SGDM with different batch size settings.
Batch size
64
128
256
EMA-SGDM
79.42a.11
76.84g.t
69.030.39
Handarr-SGDDM
79.55o.13
79.7
78.960.3
FSGDM
80.92..13
81.44..06
80.34..01
D EXPERIMENTAL SETTINGS
D.I TRAINING SETTINGS FOR VISION CLASSIFICATION TASKS
We use custom training code based on the PyTorch tutorial code for all our visual classification experiments (including the experiments in Section 3, Section 4.2 and Section 5) We choose the the learning rate experiments on CIFAR- 10, CIFAR- 100, and Tiny- Imageeme,,,e, and 1x 1x10- for ImageNet. All models used are simply following their paper's original architecture, and adopt the weight initialization introduced by He et al. (2015). Additionally, we train 300 epochs for experiments on CIFAR- 10 and CIFAR-100 and train 100 epochs for Tiny- Tiny-ImageNet and ImageNet. We 128 batch size for experiments on CIFAR-10. 10, CIFAR-100, and Tiny-ImageNet, and 256 for ImageNet. All experiments are conducted RTX 4090 A GPUs.
CnnianAnn (Loshchilov & Hutter, 2016) as our training scheduler. Additionally, we set
10 for all experiments, while the weight decay is set 5 10 4 for
Data Augmentation. For experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNer, we adopt PyTorch's RandomCrop, followed by random horizontal flips. Specifically, the random crop size is set to 32x32 for CIFAR-I0 10 and CIFAR- 100 and set to 64x64 for Tiny-Imagehi ImageNet. For experiments on ImageNet, we adopt PyToreh RandomResizedCrop, cropping to 224x224 followed by random horizontal flips. Test images use fixed resize to 256x256 followed by center crop to 224x224. At last, a data normalization adopted to input images.
D.2 TRAINING SETTINGS FOR NATURAL LANGUAGE PROCESSING TASKS
All models used in our experiments are directly adopted from the FairSeq 5 framework. We retain the original architecture of each model and train models for 100 epochs using a single NVIDIA RTX 4090 GPU. We set the maximum batch size to 4,096 tokens and apply gradient clipping with a threshold of 0.1. The baseline learning rate is set to 0.25, and for the optimizer, we use a weight decay of 0.0001.
D.3 TRAINING SETTINGS FOR REINFORCEMENT LEARNING TASKS
For the experiments RL tasks, we not make any changes except for replacing the original Adam optimizer with Handard-SGDM.. EMA SGDM, and our proposed FSGDM. To ensure faimesss, use Tianshou's (Weng et al., 2022) default hyperparameters for PPO training. However, since SGD- based optimizers highly sensitive the learning rate, searched for suitable learning rates across the three games, ultimately setting 10-2, 10-2 and 10-3 for Walker2d-v4, HalfCheetah-v4, and Ant-v4, respectively.
21

---

Published as a conference paper at ICLR 2025
E CHALLENGES IN THE FREQUENCY DOMAIN ANALYSIS FOR ADAPTIVE OPTIMIZERS
Algorithm 3: Adam
Algorithm 2: RMSprop
Input 31. 32, E, mo,20;
Input 32, 6, Uoi
for each t = 1.2,..... do
for each 1,.,... do
9t = (C((0111(-11))
(L G-1) et - (1 82)ge
my = -1m4-1 111110 (1 B1 gei
Vt = $20 + (1 $2)
It 24-1 - a19t/ /n9t:/VUt t);
my 1-31 12 1-83
end
Tt Et-1 a,mi (v c);
end
In this section, make discussion the potential challenges for the extension of the frequency domain analysis framework to adaptive optimizers like RMSprop and Adam as shown in Algo- rithm and 3. The first-moment estimate of Adam is in the form of EMA and thus acts low-pass filter. However, the second-moment estimate presents additional obstacles for frequency domain analysis in the following ways:
1. The second-mone moment estimates of Adam and RMSprop involve the squared gradient term g., resulting in nonlinearity that complicates the direct application of the Z-transform.
2. Adam introduces both the first- and second-moment estimates (me and we), and adopts m./(v + e) as the update step. This intricate interaction between m, and et also makes the analysis more challenging.
At this stage, we believe that our argument regarding the three insights discussed in Section 3.3 is also applicable to other optimizers. However, it remains unclear how the different frequency gradient components in the model parameter updates are processed by the Adam optimizer. We anticipate that resolving these issues will provide deeper insight.
22


Reviews:
[{"review_id": "g2LLGTmhqh", "reviewer": "Reviewer_AKLA", "summary": "The paper proposes a signal processing framework for the analysis of momentum in stochastic gradient descent (SGD). Specifically, the authors consider two variants of momentum SGD: coupled (where the previous and the current step proportions add up to one) and decoupled (where the current step has the weight of one, and the previous step can have the momentum term tweaked independently). Furthermore, the authors use the signal processing viewpoint to gain insight on how the momentum coefficients should evolve over the course of neural network training. Finally, based on the insights regarding the training behavior under various momentum settings, the authors propose a new frequency-based variant of momentum SGD, and show on a wide variety of problems that the proposed variant outperforms the conventional momentum SGD (both coupled and decoupled).", "questions": "A few mistakes that I picked up need to be fixed:\n* Page 2, line 95: SGD-base -> SGD-based\n* Page 6, line 321: “test results of orthodox momentum…” - do you mean unorthodox?\n* Page 7, line 351: “A Large” - “large” should not be capitalized\n\nLegend on the figures is in a very small font, which makes it rather hard to read.\n\nTables 1 and 2 refer to various \"Experiments\" (1, 2, 3, and 4). While I realize that the experiments are defined in the main body of the text, I think the tables can be made more easily interpretable on their own by rather using a descriptive name (high-pass VS low-pass gain, etc.) per row.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "**Originality:** The paper brings the signal processing perspective to the understanding of SGD, which is invaluable. Seeing that a lot of the progress in ML is purely empirical, the systematic and thorough approach the paper takes is both refreshing and useful. Viewing momentum term contributions as corresponding to low/high-pass filters opens the door to apply various insights from signal processing directly to neural network training. Thinking of momentum as attenuating and amplifying various gradient frequencies also provides a useful intuition, and maps well to existing state-of-the-art optimizers such as Adam.\n\n**Observations:** The authors note that decoupled rather than coupled approach to momentum is more beneficial. Further, the authors observe the following set-up as beneficial to generalization performance: retaining the original gradient early in training; focusing on high-frequency gradients early in training, and suppressing them closer to the end of training; amplifying low-frequency gradients at the end of training. Perhaps the insights gained can lead to further studies into generalization behavior of neural networks.\n\n**Quality and Clarity:** The paper is very well-written, with only a few minor typos. The methodology is clear. A lot of the results are pushed into appendices, which indicates that the authors have done an excessive amount of work towards the paper. I feel bad that such extensive experimentation is becoming a de facto standard for top-tier conferences. Nevertheless, there is evidence that the authors went above and beyond the expectation.\n\n**Significance:** The paper asks fundamental questions, and provides useful answers to these questions. It offers a significant new perspective that can aid the design of new, more efficient optimizers.", "weaknesses": "An important work on how momentum works was published in 2017 in the Distill web-based journal: https://distill.pub/2017/momentum/\nI believe that this paper needs to feature in the related literature. Also, in case the authors are not aware of it, I believe they will enjoy reading it!\n\nAs mentioned above, the experiments conducted are truly comprehensive and span a wide range of problems and architectures. However, what is missing from the experiments is the presence of the (arguably) state of the art approach to NN training, i.e., the Adam optimizer. For the reinforcement learning experiments, the authors explicitly state that they have replaced Adam with their method as well as coupled and decoupled versions of momentum SGD. However, based on the insights, it seems that Adam succeeds by doing exactly what the paper prescribes as necessary: amplify low-frequency gradients while attenuating high-frequency gradients. A discussion of Adam in the context of the proposed paradigm, even on a superficial level, would benefit the discussion, in my opinion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fiwoFpbxuK", "reviewer": "Reviewer_Whrt", "summary": "This paper introduces a frequency-domain analysis framework for understanding momentum-based optimizers, viewing them as time-variant filters that adaptively shape gradient characteristics through momentum coefficients. Based on these insights, the authors propose FSGDM, a novel optimizer that dynamically adjusts momentum to enhance generalization, outperforming conventional momentum-based methods in experiments.", "questions": "Why the Z-transform is employed in the learning settings, while many other transforms can also be deployed?", "limitations": "", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper is clear and straightforward, making it easy to follow. The core idea is both simple and effective, showcasing an elegant solution that achieves strong results without unnecessary complexity.", "weaknesses": "1. In traditional signal processing, many derivations provided in the paper are well-established and widely recognized. \n2. Quantitative comparisons within this field remain relatively limited, often lacking comprehensive metrics or benchmarks to evaluate performance across different methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VKu1ID55j2", "reviewer": "Reviewer_RGqa", "summary": "The paper explores the momentum-based optimization methods commonly used in training neural networks through a frequency domain perspective. It shows a new interpretation of momentum as a time-variant gradient filter, where the value of momentum coefficients determines the filtering of different frequency components of gradients. The analysis shows that high-frequency gradient components decrease performance in the later training stages, while preserving original gradients in the early stages and amplifying low-frequency components gradually improves generalization.\n\nBuilding on these insights, the paper introduces Frequency Stochastic Gradient Descent with Momentum (FSGDM), a heuristic optimizer that adjusts the momentum coefficients dynamically to enhance training performance. Experimental results on image classification and NLP tasks demonstrate that FSGDM outperforms conventional momentum-based optimizers.", "questions": "Suggestions:\n* It’s unclear how robust the optimizer is to hyperparameter tuning across different datasets and tasks. Without this information, practitioners might face challenges in adopting FSGDM in real-world applications. Therefore I suggest to include a dedicated analysis or appendix section detailing the sensitivity of FSGDM’s performance to variations in these hyperparameters. It would be useful to identify hyperparameter settings that generalize well across tasks, if any.\n\nQuestions:\n* Since Adam and RMSprop uses both gradient and curvature of the error surface\n  ** how might the frequency domain analysis framework be extended for Adam or RMSprop?\n  ** What challenges do you anticipate in applying this framework to those methods?\n  ** Do you believe similar insights about gradient frequency components would apply, or would different principles emerge?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\n* The paper explores the impact of momentum on gradient frequency domain. This work is very original and has not been well explored before. The exploration shows that the momentum update is usually a low pass filter which gets more and more narrower low pass filter in the later part of the training.\n\nQuality:\n* The derivations based on signal processing principles, particularly the application of Z-transforms, provide a good theoretical foundation. The analysis effectively distinguishes between conventional momentum methods (like Standard-SGDM and EMA-SGDM), clarifying their distinct impacts on gradient components from frequency domain perspective.\n* The experiments are comprehensive and well-designed, covering image classification and language translation. The results consistently show that FSGDM outperforms traditional momentum optimizers in both accuracy and convergence speed, demonstrating the practical relevance of the proposed method.\n\nClarity:\n*  The paper follows a clear structure, beginning with the theoretical formulation of the problem, progressing to the design of the FSGDM optimizer, and concluding with comprehensive experimental results.\n* The use of visuals, such as plots of frequency response and performance comparisons, enhances understanding. The inclusion of tables summarizing performance across different tasks is also helpful for comparing results at a glance.\n\n\nSignificance:\n* By integrating frequency domain insights into momentum adjustment, this work not only advances the understanding of optimization mechanisms but also opens the door for further research.\n* The empirical results demonstrate that FSGDM consistently achieves better performance than standard momentum-based optimizers, indicating that the proposed method can be a strong alternative across different neural network task", "weaknesses": "Significance:\n* While the Frequency perspective of momentum based weight updates is very impressive, the proposed FSGDM (Frequency Stochastic Gradient Descent with Momentum) does not appear to be very significant because of the following 2 reasons\n  1. Setting schedules for momentum coefficient and learning rates during training is a very common practice in NN training. In the proposed FSGDM method we still need to set bunch of hyperparameters like scaling factor c, momentum coefficient v. Therefore FSGDM is not very different from the tradition momentum method in terms of hyperparameter tuning.\n  2. The optimal value of scaling factor c and momentum coefficient v is determined for different datasets and network architectures (as shown in fig 4) and is used for the results shown in the experiment section. While some standard coefficients are used for standard SGDM and EMA SGDM. This shows that the results might be slightly biased.\n* The frequency domain analysis in this paper remains somewhat limited in its applicability to broader optimization contexts. The framework primarily addresses momentum-based methods in the context of SGD-like algorithms, but it does not extend to other popular optimization strategies like Adam or RMSprop, which also incorporate adaptive mechanisms and momentum-like components. Therefore I suggest extending the frequency domain analysis for other popular optimization methods like Adam and RMSprop would make this paper much more significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)