Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learning
Authors: Ronglong Fang
Yuesheng Xu
Venue: NeurIPS 2024 poster (2024)
Keywords: deep neural network
spectral bias
multi-grade deep learning
Primary area: deep_learning_architectures
TL;DR: This paper addresses the spectral bias issue of deep neural networks by multi-grade deep learning.

Abstract: Deep neural networks (DNNs) have showcased their remarkable precision in approximating smooth functions. However, they suffer from the {\it spectral bias}, wherein DNNs typically exhibit a tendency to prioritize the learning of lower-frequency components of a function, struggling to effectively capture its high-frequency features. This paper is to address this issue. Notice that a function having only low frequency components may be well-represented by a shallow neural network (SNN), a network having only a few layers. By observing that composition of low frequency functions can effectively approximate a high-frequency function, we propose to learn a function containing high-frequency components by composing several SNNs, each of which learns certain low-frequency information from the given data. We implement the proposed idea by exploiting the multi-grade deep learning (MGDL) model, a recently introduced model that trains a DNN incrementally, grade by grade, a current grade learning from the residue of the previous grade only an SNN (with trainable parameters) composed with the SNNs (with fixed parameters) trained in the preceding grades as features. We apply MGDL to synthetic, manifold, colored images, and MNIST datasets, all characterized by presence of high-frequency features. Our study reveals that MGDL excels at representing functions containing high-frequency information. Specifically, the neural networks learned in each grade adeptly capture some low-frequency information, allowing their compositions with SNNs learned in the previous grades effectively representing the high-frequency features. Our experimental results underscore the efficacy of MGDL in addressing the spectral bias inherent in DNNs. By leveraging MGDL, we offer insights into overcoming spectral bias limitation of DNNs, thereby enhancing the performance and applicability of deep learning models in tasks requiring the representation of high-frequency information. This study confirms that the proposed method offers a promising solution to address the spectral bias of DNNs. The code is available on GitHub: \href{https://github.com/Ronglong-Fang/AddressingSpectralBiasviaMGDL}{\texttt{Addressing Spectral Bias via MGDL}}.

A Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learning
Ronglong Fang, Yuesheng Xu' Department of Mathematics and Statistics, Old Dominion University (rfang002, ylxu)oodu.edu
Abstract
Deep neural networks (DNNs) have showcased their remarkable precision in ap- proximating smooth functions. However, they suffer from the spectral bias, wherein DNNs typically exhibit tendency to prioritize the learning of lower-frequency components of function, struggling to effectively capture its high-frequency features. This paper is to address this issue. Notice that a function having only low frequency components may be well-represented by shallow neural network (SNN), a network having only a few layers. By observing that composition of low frequency functions propose to learn a function containing high-frequency components by composing several SNNs, each of which learns certain freeueequccy information from the given data. We implement the proposed idea by exploiting the multi-grade deep learning (MGDL) model, recently introduced model that trains DNN incremen- tally, grade by grade, a current grade learning from the residue of the previous grade only SNN (with trainable parameters) composed with the SNNs (with fixed parameters) trained in the preceding grades as features. We apply MGDL synthetic, manifold, colored images, and MNIST datasets, all characterized by presence of fighffrequeccy features. Our study reveals that MGDL excels at representing functions containing high-frequency information. Specificallyy, the neural networks learned in each grade adeptly capture some low-frequency infor- mation, allowing their compositions with SNNs learned in the previous grades effectively representing the high-frequency features. Our experimental results underscore the efficacy of MGDL in addressing the spectral bias inherent DNNs. By leveraging MGDL, we offer insights into overcoming spectral bias limitation DNNs, thereby enhancing the performance and applicability of deep learning models in tasks requiring the representation of high-frequency information. This study confirms that the proposed method offers promising solution to address the spectral bias of DNNs. The code is available on GitHub: Addressing Spectral Bias via MGDL
effectively approximate nighffrequency function,
Introduction
Deep neural networks (DNNs) have achieved tremendous success in various applications, including computer vision [5), natural language processing (38), speech recognition (26), and finance [27]. From mathematical perspective, the success is mainly due to their high expressiveness, evidenced by theoretical demonstrations showing their capability to approximate smooth functions with arbitrary approximation tool were [1 [6] 22 25 35| 45 47 49]. However, it was noted in [30] 42 that the
precision 3
10 [4]
17
48 Various mathematical aspects of DNNs
recently investigated
standard deep learning model, which is called single grade deep learning (SGDL) in this paper,
"Corresponding author; yixu@odu. edu
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

trains DNN end-to-end leading to learning bias towards now-freueecyy functions. While this bias may explain the phenomenon where DNNs with a large number of parameters can achieve low generalization error 7 40 41 DNNs trained by SGDL struggle to capture
components within a function even though they can well-represent its low-frequency components. This bias may potentially limit the applicability of DNNs to problems involving high-frequeng features, such as image reconstruction [12 24 37,. seismic wavefield modeling [39], high-frequency wave equations in homogenization periodic media [9], and high energy physics [28]. Especially, in medical image reconstruction such as PET/SPECT, high-frequenc components play a crucial role in determining image resolution, which is critical in clinical practice, as higher resolution leads to earlier and more accurate disease diagnosis.
There have been some efforts to address this issue. A phrase shift DNN was proposed in [5), where the original dataset was first decomposed into subsets with specific frequency components, the high- frequency component was shifted downward to a low-frequency spectrum for learning, and finally, the learned function was converted back to the original high frequency. An adaptive activation function was proposed in [18] to replace the traditional activation by scaling it with a trainable parameter. A multiscale DNN was introduced in [6 23), in which the input variable was first scaled with different scales and then the multiscale variables were combined to learn a DNN. It was proposed in 136 first to map the input variable to Fourier features with different frequencies and then to train the mapped data by DNNs. All these approaches mitigate the spectral bias issue of DNNs to some extent.
Despite of encouraging progresses made in mitigating the spectral bias of DNNs, practical learning with DNNs remains a persistent challenge due to the bias, especially for learning from higher- dimensional data. This issue deserves further investigation. We propose to address this issue by understanding how a high-frequency function can be more accurately represented by neural networks. On one hand, it has been observed 77 30 31 that function having only low frequency components can be well represented by shallow neural network (SNN), network having only few layers. On the other hand, the classical Jacobi-Anger identity expresses a complex exponential of a trigonometric function as a linear combination of its harmonics that can contain significant high-frequency components. Even though the complex exponential function and the trigonometric function both are of low frequency, their composition could contain high frequency components. This motivates us to decompose a function containing high- frequencies as a sum-compositiee form of low-frequency functions. That is, we decompose it into sum of different frequency components, each of which is further broken down to a composition of low-frequency functions. In implementing this idea, we find that the multi-grad, grade deep learning (MGDL) model recently introduced in [43 44 matches seamlessly for constructing the sum-composition form for a function of It is the purpose of this study introduce the general methodology in addressing the spectral bias issue of DNNs and implement it by employing MGDL as a technical tool. We demonstrate the efficacy of the proposed upproach in four experiments with one- dimensional synthetic data, two- dimensional manifold data, wwo-dimensional colored images, and very sigh-dimensionall modified National Institute of Standards and Technology (MNIST) data. Our numerical results endorse that the proposed approach can effectively address the spectral bias issue, leading to substantial improvement in approximation accuracy in comparison with the traditional SGDL. training approach.
Contributions of this paper include: (a) We propose a novel approach to address the spectral bias issue by decomposing a function containing high-frequencies as a sum of different frequency components, which are represented as compositions of low-frequency functions. (b) We investigate the efficacy of MGDL in decomposing a function of high-frequency into its "sum-composition" form of SNNs. (c) We successfully apply the proposed approach to synthetic data in 1 and 2 dimensions and real data in and 784 dimensions, showing that it can effectively address the spectral bias issue.
2 Proposed Approach and Multi-Grade Learning Model
We introduce a novel approach to tackle the spectral bias issue and review the MGDL model.
We begin with quick review of the definition of DNNs. A DNN is a successive composition of an activation function composed with linear transformation. Let IR denote the set of all real numbers, and d, s be two positive integers. A DNN with depth D consists of an input layer, - I hidden layers, and an output layer. Let ND 21, DJ. For j E (O)UN, UN let d, denote the number of neurons in the j-th hidden layer with do ;= d and dp == s. We use W, [R id,x and by gd

---

Figure 1: Spectrum comparison of f f= fi f2 fi + fs f2 o fi fa o fs o f2 fi and fz: Amplitude versus one-side frequency plots for function (Left) and ʃj for j N4 (Right). The function ʃ is of high- frequency and functions fy all are of low-frequency.
0.30
0.10
0.08
fa
Ampliude
Ampliude
f
0.04
0.04
0.00
0.02
0.00
0.00
175 200
100
200
Frequency (f1c)
Frequency [H1z]
to represent the weight matrix and bias vector, respectively, for the j-th layer. By a : R + we denote an activation function. When is applied to a vector, it means that is applied to the vector componentwise. For an input vector x : [21, .2,...,, .,, Fa]
E Rdd, the output of the first layer is (W1x + b1). For DNN with depth D > 3, the output of the (j + 1) 1-tth
defined by Hi (x) hidden layer can be identified as a recursive function of the output of the j-th hidden layer, defined as H.y+1 (x) := (W j+1 H, (x) + by+1), for E ND-2. Finally, the output of the DNN with depth D is an s-dimensional vector- valued function defined by
No (WW,, by) bi)f= = (x)())= = WD (x) + bn.
Suppose that data samples D == {xe.Y) are chosen. The loss on D is defined as
<( (W,, b, by :D) = 2N |x NN ((W,) by b :) (xx)
1
(2)
The traditional SGDL model to minimize the loss function L defined by (2) with respect θ = [W,, by D2 which yields the optimal parameters O == {W,, j, b b and the corresponding DNN No :). When D is relatively small, for example, D < 5. we call ND an SNN. It is well-recognized that training an SNN notably easier than training a DNN.
We motivate the proposed idea by simple example. We consider the function f(x), x E [0, 1]. whose Fourier transform is shown in Figure [ (Left), where the Fourier transform defined by 120 dx. To compute the Fourier transform of f defined on [0. 1), we extend
f(t) = - (xx)e=(xxxxx to the entire real line by assigning its value to be zero for x & [o, 1]. Observing from Figure I (Left), the function ff has significant high- feeeufryyeeccy components, with frequencies varying from to 200. The function f can be represented as
f(x) (x) + (f2 o fi) + (fs f2 o (i)
f2 fi)(),, E [0, 1],
(3)
where o denotes the composition of two functions. Note that the Fourier transforms fj. = 1,2, 3, 4, displayed in Figure (Right). Clearly, the functions f j 1, 3, 4, are of low-frequency, with frequencies mainly concentrating on the interval [0, 50). This example surely demonstrates that function of high-ffrequeny can be expressed a sum of compositions lower-frequency functions. This observation leads to the proposed approach of addressing the spectral bias of DNNs to be studied in this paper. The legitimacy of the proposed idea may be reinforeed by the Jacobi-Anger identity (2. which expresses a complex exponential of trigonometric function as a linear combination of its harmonics. Even though both the complex exponential function and the trigonometric function are of low-frequency, their composition contains many Jacobi-Anger identity, the identity named after the 19th-century mathematicians Carl Jacobi and Carl Theodor Anger. It has the form
components. We now review the
ta sin(
00
In(a)e inbx
(4)
where denotes the imaginary unit and (a) denotes the n-th Bessel function of the first kind, see details in 22] Taking the real part of the both sides of the Jacobi-Anger identity (4), we obtain that
cos( a sin((x)) E
(a) (nbxx)..
(5)
The left-hand side of (5) composition of two low-frequency functions cos(ax) and sin having frequencies a/(2)) and b/(2x), respectively, while the right-hand side is a linear combination of cos(nbx) with n taking all integers. The high-frequency of the composition can be estimated by rule of thumb. Specifically, the left-hand side of (5) is frequency-modull
sinusoidal signal
3

---

32 33
with its frequencies spreading on an interval centered at zero. It follows from the well-known
Carson bandwidth rule 29 321, regarded as a rule of thumb, that more than 98% frequencies are located within the interval |- (ab + b)/(2m) ) (ab + (2x]] Therefore, the highest frequency of
cos(a can be well-estimated by 2x which is greater than the frequencies of cos (ax) and
ab+6
(bx) when a > 0 and b > 1. These suggest that a composition of two ore-freuennyy functions may lead to a high-freen figh-freyeency function.
The example presented earlier, together with the Jacobi-Anger identity, inspires us decompose given function into a sum of different frequency components, each of which is a composition of lower- frequency functions, a decomposition similar to equation (3) for the function of represented in Figure (Left). In other words, for a function g of high-frequency, we decompose it in a "sum-composition" form as
Σ K
A:
--1 94.
(6)
where j=1 93 == Ok .... o 92 o 91, and g3. J E NR. are all of wwwffrqueecy. The function f represented in 33) is a special example of (6). In the context of approximation by neural networks, we prefer expressing 9j by SNNs, as a function having only low-frequency components can be well- represented by an SNN. The MGDL model originated in 45 44 furnishes exactly the decomposition (6). with each 9j being an SNN. We propose to employ MGDL to learn the decomposition (6). where the low-frequency function 9j is represented by an SNN.
It is worth explaining the motivation behind the MGDL model. MGDL was inspired by the human education system which is arranged in grades. In such system, students learn complex subject in grades, by decomposing it into sequential, simpler topics. Foundational knowledge learned in previous grades remains relatively stable and serves as a basis for learning in a present and future grades. This learning process can be modeled mathematically by representing a function that contains higher-frequency components by ssum-oompoiiiion" " form of -wwfrequency functions. MGDL draws upon this concept by decomposing the learning process into multiple grades, where each grade captures different levels of complexity.
We now review the MGDL model that learns given data D := {xx,ye) N Following 43 we split DNN with depth D into L grades, with L < D. each of which learns an SNN ND., defined
as D.. with depth Dr, from the residue N { of the previous grade, where 1 < Dr < D and denote the parameters to be learned in grade l. We define recursively 91 xO):x) == ND, (θ1; x), 91+1 (01+1;x) := ND (9t+1:) o (D.-1
Etn Dj = D + I. - 1. Let O2 == , b)} ب= j=1
Di
o ( (xx), for E NL-1. and the loss function of grade I by
C DO: = 2N Σ
le| gi (Or; x 112
where 07 [W). bo.],,bb)) b- are the optimal parameters learned by minimizing the loss function Li with respect to θj The residues are defined by c/ := ye and c/+1 of 9: (0;; xt). for I N
l E NN. When minimizing the loss function 2 DD) of grade parameters J E N₁ learned from the previous / - 1 grades are all fixed and 1 o... a HDD (0): serves as a feature "basis". After T. grades are learned, the function Ir learned from MGDL is the summation of the function learned in each grade, that is.
IL (101) Ex_, (O);
(8)
where (x] := ND. (O):-) o -1 ....o HD, (e]: and HD for 1 < k S L and NDz are SNNs learned in different grades. Thus, MGDL enables to construct the desired "sum- coumooompoiition" form (6). When L = 1, MGDL reduces to the traditional SGDL model.
In MGDL, we use the mean squared error (MSE) loss function. It was established in 4 that when the loss function is defined by MSE, MGDL either learns the zero function or results in a strictly decreasing residual error sequence (see, Theorem in Appendix A Since the regression problems conducted in this paper naturally align with MSE losses, it a suitable choice. In practice, MGDL can also be applied with other loss functions, such as cross-entropy loss, when solving classification problems. MGDI,, the computation cost remains relatively consistent across all grades. For x == (D.-1-107
1 o... o HD,,1(07; -)) (xxe, we recursively
4

---

let xetx := xe, x :- - -xx xxx-l____ k-1 k == 2.3...... When training grade 1, we use the output of grade - 1, denoted x along the residual ey, which are already obtained. The training dataset in grade 1 consists of { e
N
t-1' This dataset is used to train a new shallow network, which is independent of the previous I - grades. Moreover, x can be computed recursively, ensuring that the computation cost for each grade remains relatively consistent.
MGDL avoids training a DNN from end to end. Instead, it trains several SNNs sequentially, with the current grade making use the SNNs learned from the previous grades as a feature and composing it with new SNN to learn the residue of the previous grade. This allows MGDL to decompose function that contains higher-frequener in a form of 6 with gj being a SNN learned from grade j. In this way. higher-frequene manner. Note that the training time of MGDL increases linearly with the number of grades. This makes MGDL effective and scalable solution for tackling complex tasks. MGDL is an adaptive approach by nature. When the outcome of the present grade is not satisfactory, we can always add a new grade without changing the previous grades.
components in the data can be effectively learned in grade-by-grade
is worth noting that while ResNet [15] also has sum-composition form, MGDL differs from significantly. The "Composition" for ResNet refers to composition of layers, while that for MGDL emphasizes the composition of the SNNs sequentially learned in the previous grades. Moreover, ResNet learns all parameters of the entire sum of DNNs at once, training it from end to end, whereas MGDL learns the sum incrementally, grade by grade, in each grade training an SNN composed with the feature (the composition of the SNNs learned in the previous grades). MGDL also differs from the relay backpropagation approach proposed in [34], where a DNN is divided into multiple segments, each with its own loss function. The gradients from these losses are then propagated to lower layers of their respective segments and all segments are optimized all together by minimizing the sum of the losses. While MGDL trains SNNs in a multi-grade manner, each of which learns from the residue of the previous grade, freezing the previously learned SNNs (serving as features or "bases").
MGDL is principle applicable to various models, including standard DNNs, convolutional neural networks, and ResNet. In this paper, we demonstrate its feasibility by applying it to standard DNNs.
3 Numerical Experiments
In this section, we study MGDL empirically in addressing the spectral bias issue of SGDL. We consider four examples: Subsections 3.1, 3.2, and 3.4 investigate regression on synthetic, manifold, and MNIST data, respectively, for which the spectral bias phenomena of SGDL are identified in [30]. Section 3.3 deals with regression on colored images, which were studied in (36) by using the Fourier features to mitigate the spectral bias. Our goal is to compare the learning performance of MGDL with that of SGDL on these datasets and understand to what extent MGDL can overcome the spectral bias exhibited in SGDL.
The loss functions defined in 2 for SGDL and 7 for MGDL are used to compute the training and validation loss when D is chosen to be the training and validation data, respectively. We use the relative squared error (RSE) to measure the accuracy of predictions obtained from both SGDL and MGDL.. Assume that A is trained neural network. For prediction value ý. == ->xxx) at x we define RSE= == Σ Ily; - yelle//2 E llyell When D represents the training, validation, and testing data, RSE is specialized as TIRSE, VaRSE, and TeRSE, respectively.
Details of the numerical experiments conducted in this section, including computational resources, the network structure of SGDL and MGDL for each example, the choice of activation function, the optimizer, parameters used in the optimization process, and supporting figures are provided in Appendix B
3.1 Regression on the synthetic data. In this experiment, we compare the efficacy of SGDL and MGDL in learning functions of four different types of high-frequencies.
The experiment setup is as follows. Given frequencies := (KI, (2,,, RM) with corresponding amplitudes a := (a1, a......, a.M). and phases 9 := (P1,, P2,...... (aar), we consider approximating the function À : [0. 1] + R defined by
X(x) = Σ, M
j=1 a sin (2T + pp)) x x [0
by neural networks learned with SGDL and MGDL. We consider four settings, in all of which we choose M := 20, Kj ;= 10j and y u(, 2T)) for j e N2o. where U denotes the uniform
5

---

0.32
0.25
0.90
0.29
0
04
Frequency (H2)
Frequency (H62]
Froquency [22)
Froquency [82]
Figure 2: Amplitude versus one-side frequency plot for the learned functions learned across four grades of MGDL for settings 1-4.
Table I: Comparison of SGDL and MGDL: Accuracy
setling
model
ima
Imin
batch size
lime(()
TrRSE
VaRSE
TeRSE
SGDL
10 "
10
256
32, 401
1.5 10
1.3
1.2 10
MGDL
10-4 4
10 10-4
256
27, 817
5.9 x 10
1.9 x 10
1.7 10
SGDL
5 10-4
10
256
32,055 053
5.8 x 10-3
5.9 x 10
5.7 x 10
2
MGDL
10-4
10
256
27, 22.628
1.5 10
4.0 x 10
6.5 10
SGDL
10-3
10
256
31, 808
9.6 x 10-2
8.3 10-2
1,1 10
3
MGDL
10-4
10
256
87,
5.2 10
3.2 3,2x10' 10-5
2.1 x 10
SGDL
5 10-5
10
256
41, 063
7.9 10
x
7.7 10
4
MGDL
10 10-4
10
Full
9,875
1.1 x 10
2.2 x02110
1.3 x 10 10-3
distribution and the random seed set to be 0. In settings 1, 2, and 4, we choose respectively the 1.05-0.. 0.05 amplitudes 0j == 1, aj 1.05-0.055j. 0ry (x) == e cos(jx), and aj := 0.05j, for j N2o- Note that in setting 3, we explore the case where the amplitude varies as a function of x for each component. The amplitudes versus one-side frequencies for across the four settings are depicted in Figure 7 in Appendix B.1 For all the four settings, the training data consist of pairs (xe, >(xe) ((x)))sse ReNoonn" Where x, are equally spaced between 0 and 1. The validation and testing data consist of pairs (x,, (rxx((xe) 2oo where Xy 's are generated from a random uniform distribution on [0, 1], with the random seed set to be 0 and 1, respectively.
Numerical results for this example are reported in Figures and 3 as well as and (in Appendix B.I]. and Table Figure e displays the amplitude versus the one-side frequency of the functions learned across four grades of MGDL in settings 1-3, and five grades in setting 4. In all the four settings, MGDL exhibits a pattern of learning low frequency components in grade 1, middle- frequency components in grade 2, and high- high-frequency components in grades 3, 4 (and 5 for setting 4). We and H HD, (0): -) The SNN Ni learned in grade represents low-frequency component of X, the SNNS Na and N2 learned in grades and 3, composed with Hi and H2 о Hi respectively, represents Ni. learned in grade 4, composed with H3 H5 o Hi represents the highest-frequenge component of X. This fact is particularly evident in setting 4 (see, the fourth subfigure Figure 2). where the amplitude within the data increases with the frequency. For settings 2, and 3, grade 4 does not learn much. This is because for the functions of these three settings, the amplitudes of frequecies are proportionally small. However, grade 4 is still important for these settings. As shown in Figure (right), grade 4 reduces the loss from 10 10-5 for setting and from 10-4 10 for settings 2 and 3. This indicates that if we want high precision, we need to include grade 4. For setting 4, we need grade 5 to learn its highest frequency component. These findings suggest that MGDL particularly well-suited for learning the high-frequency components of the function.
let .i == :N,
components of A. Likewise, the SNN
0.36
G
0.25
0.25
Q
0.30
$ 6.30
0.25
0.30
13
0,000
0.50
1420
2000
110000 240 HAESO FINSOW
50 103 150 200 Recuency THe)
250 200 26o -00
500
103
1.50
303 203 300
(S0OL)
Epochs
Propuency
Figure 3: Comparison of SGDL (left) and MGDL
Figure 4: Amplitude vs. one-side frequency plot for the learned function across four grades of MGDL: settings 1 (left) and 2 (right) with q = 0.
(right): training and validation loss across settings 1-4.

---

Figure B compares the progress of the training and validation losses against the number of training epochs for SGDL and MGDL across the four settings. We observe that when learning a task involving high-frequency components by SGDL, the training and validation losses decrease slowly due to the spectral bias of DNN. While the same task is learned by MGDI,, the learning process progresses through distinct grades. In grade 1, MGDI. primarily learns low-frequency, resulting in slow decrease in loss. In grade 2, the training loss and validation loss both decrease more rapidly due to the use of the composition of SNN NW with the feature Hi. facilitating in learning fighefreeuency features. This accelerated learning aspect of MGDL is further evidenced in grades and (as well as grade 5 for setting 4). Table [ compares the accuracy achieved by SGDL and MGDL. Within comparable or even less training time, MGDL increases accuracy, measured by TeRSE from 10 to 10 10-5. 10 to 10 10-1 to 10 10-5, and 10 to 10-3 in settings 1, 2, 3 and 4, respectively. Across the four settings, TeRSE values are reduced by factor of 592 ~ 7, 058. These comparisons highlight MGDL's advantage in effectively learning high-frequency oscillatory functions.
Figure 8 in Appendix B.T depicts the functions, the Fourier domain, learned by SGDL (row 1) and MGDL (row 2) across the four settings, demonstrating that MGDL has a substantial reduction in the 'spectral bias' exhibited in SGDL. This is because nigh-frreuency components are learned in higher grade, where they are represented as the composition of a low-frequency component with the low-frequency components learned in the previous grades, and each grade focuses solely on learning low-frequency component by an SNN. We also include Figure 9 in Appendix to compare the spectrum evolution between SGDL (Ist row) and MGDL (2nd row) across settings I-4. Notably, although in iterations SGDL and MGDL both learn fow-frequency components first and then followed by middle and high-frequency components, MGDL learns in grade by grade, exhibiting significant outperformance.
3.2 Regression on the manifold data. The second experiment compares regression by SGDL and MGDL on two-dimensional manifold data, studied in 30 but with twice higher frequencies.
The goal of this experiment to explore scenarios where data lies on lower-dimensional manifold lisher-dimensional space. Such data is commonly referred to as manifold data 4 Let y be an injective mapping from [0, 1jTh + Rd with m S dl and M := >[0 1](²) denote the man- ifold data. A target function T .M- + R defined on the manifold can be identified with function X == roy defined on [0, 1)m Regressing the target function F therefore equivalent to finding ʃ Rdd - TR.
embedded within
such that f o y matches X. Following 330. we set m == 1, d == 2, and choose the mapping 7 as
(x) := [1 + sin(2mxx)) /2] (cos (cos(2xxx), sin( (2mx)), x E [0, 1],
(10)
for a nonnegative integer q. Clearly, 7y : [0, 1] + R², and M := 7i ([O, 1J) defines the manifold
corresponding to flower-shaped curve with petals when g > 0, and a unit circle when
0. Sup-
pose that A : [0, 1] - IR is the function defined by
( Our task is to learn a DNN f : R2 + such
that f "y matches A. We consider two settings for A. In settings 1 and 2. we choose ay == 0.025 and ax := E cos(jx) for j E N40- respectively. For both the settings, we choose K, := 10j and 40; (000 2m) for E Nao with the random seed set to be 0, and consider the cases where q := and == O. Note that the smaller q is, the more difficult the learning task is. The training data consists of pairs (7x( )(xe))eeN.zpor where Xx's 's are equally spaced between 0 and 1. The validation and testing data consist of pairs (%/ X(xe)) (
ReNanoo" where x 's are generated from random uniform distribution on [0, 1]. with random seed set to be 0 and 1, respectively.
Numerical results for this example are reported in Figures 4 and 10 (in Appendix (B.2), and Table 2 Figure Iillustrates the frequency of functions learned across four grades of MGDL for settings and 2, where g == O. In both of the settings, MGDL exhibits a pattern of learning low-frequency components in grade middle-freoe freddueefr components in grade 2, and high-frequency components in grades 3 and 4. Therefore, the high-frequency components within the function mainly learned in higher grades, in which the learned function is a composition of the SNNs learned from several grades. That is, MGDL decomposes a high-frequency component as the composition of several lower-frequency components, facilitating effectively learning high frequency features within the data.
Table 2 compares the approximation accuracy achieved by SGDL and MGDL for settings 1 and 2. For SGDL, reducing the value of q makes the learning task for both settings more challenging, due to the spectral bias of DNNs. When == 4, 0 in setting 1 and := 0 in setting 2, learning becomes especially challenging for SGDL. In such cases, MGDL significantly outperforms SGDL by achieving higher accuracy in approximately half to one-third of the training time for both settings. Figure 5 displays the training and validation loss for SGDL and MGDL. Figure [0] illlstrattes the
7

---

Table Accuracy comparison: SGDL
versus MGDL.
setting
method
Emaz
Emin
batch size
time (S)
TIRSE
VaRSE
TeRSE
SGDL
10
10
1024
28,835 832
2.7 10
2.4 10
2.8 x 10
4
MGDL
10-3
10-4
Full
15,519
10 5
1.8 10
1.1 10
SGDL
10
10
1024
29,05 051
5.5 x 10-1
5.4 10
5.4 10
0
MGDL
10-3
10-4
Full
15,969
1.9 x 10
2.7 10
2.1 10
SGDL
10-3
10-6
512
4,083
2.0 10
1.8 10-3
2.3 x 10
4
MGDL
10-3
10
Full
11,067
8.5 x 10 -
1.4 10
4.2 10
2
SGDL
10
512
41,941
1.1
10
1.8
10
1.0 10"
MGDI
10
10
Full
11,027
3.7 x 10-5
2.2 x 10
8.7 x 10-5
127
55-1
L255
voleation
voleition
3
brsking
valcutton.
tnsking.
culcation
Epocts (56OLI
Epccho INGDL)
Epochs (5GDL)
Epoors (MGDU
Figure 5: Comparison of the training and validation loss for SGDL (1st and 3rd subfigures) and MGDL (2nd and 4th subfigures) settings (1st and 2nd subfigures) and (3rd and 4th subfigures).
spectrum evolution throughout the learning process for settings 1 and 2. We observe that MGDL outperforms SGDL in accuracy during the learning process. Table 2 and Figure T0 consistently demonstrate that MGDL exhibits substantial advancement in addressing the spectral bias.
3.3 Regression on two-dimensional colored Images. In the third experiment, we compare perfor- mance of MGDL and SGDL for regression of two-dimensional color images.
We test the models with the 'cat' image from website Cat, and the 'sea' and 'building" images from the Div2K dataset [. The input to the models is the pixel coordinates, and the output is the corresponding RGB values. The training dataset consists of regularly spaced grid containing 1/4 of the image pixels, while the test dataset contains the full image pixels. We use peak signal-to- noise ratio (PSNR) defined in (12) evaluate the accuracy of images obtained from MGDL and SGDL. PSNR values computed over the train and test dataset are denoted by TrPSNR and TePSNR, respectively.
Numerical results for this experiment are presented in Table Figure 6 and Figures II 13 Appendix B.3 Table 3 the PSNR values of images obtained from MGDL and SGDL. For MGDL, it is evident that adding more grades consistently improves the image quality, as both training and testing PSNR values increase substantially with the addition of each grade across all images Cat, Sea, and Building. MGDL outperforms SGDL by 2.35 ~ 3.93 dB for the testing PSNR values. Specifically, for images Cat, Sea and Building, MGDL surpasses SGDL by 2.35, 3.93 and 2.88 dB, respectively. This demonstrates the superiority of MGDL in representing images compared to SGDL. Figure O illustrates the training and testing PSNR values during the training process for the three images. is evident that MGDL facilitates a smoother learning process as more grades are added, in comparison to SGDL. This observation aligns with the results presented in Table 3
(in
Predictions of each grade of MGDL and of SGDL are illustrated in Figures I1| 12, and 13 for images Cat, Sea, and Building, respectively. In all cases, grade captures only the rough outlines of the images, representing the lower-frequency components. As we progress to grades 2, 3, and more details corresponding to the higher-frequency components are added. This demonstrates the effectiveness of MGDL in progressively learning high-frequency features within the images. Moreover, the image quality achieved with MGDL notably superior that with SGDL.
3.4 Regression on MNIST data. In our fourth experiment, we apply MGDL to regress on high- dimension data, the MNIST data. We compare the training and validation loss, and TeRSE for SGDL and MGDL, when learning high-frequency features from the data.
We set up this experiment following (30], with a focus on comparing performance of SGDL and MGDL in learning tigh-frequency features. We choose the handwriting digits from MNIST dataset 21, composed of 60,000 training samples and 10,000 testing samples of the digits "0" through

---

SGDI
SGDI
SCIL
28
Training PSMR
Training PSNR
Taining PSNR Testing
Testing
24-
PSNH
PSMR
W
20-
10
4000
10000
12
2000
6000
4000
8000 10000
Epochs
Epochs
Epochs
(a) Cat: SGDL
(b) Sea: SGDL
(c) Building: SGDL
MGDL
MGDL
MCDL
Training
Testing PSNR.
26-
Thating
Trairing PSML
Training PSWR Testing PSNR
PSNI
PSNN
14-
Epochs
(d) Cat: MGDL
(e) Sea: MGDL
(l) Building: MGDL
Figure 6: Comparison of PSNR values for SGDL and MGDL on images Cat, Sea, and Building:
SGDL (a)-(c), MGDL (f)-(f).
Table 3: PSNR comparison: SGDL versus MGDL.
image
method
grade
learning rate
time (s)
TrPSNR
TePSNR
10
38
20.45
20.41
2
10
40
22.97
22.67
MGDL
cat
5 x 10
40
25.14
23.71
5 x 10
28
28.59
24.18
SSGL
5 T0 10"3
77
21.93
21.83
10
160
18.62
18.62
2
10 10-3
173
21.57
21.50
MGDL
10
174
24.17
23.42
4
10
182
27.31
24.32
"SGDL"
10
685
20.43
20.39
1
5 X 10-3
171
17.30
17.29
10
181
18.97
18.95
MGDL
building
3
10
182
20.86
20.67
10-3
182
23.36
21.97
SGDL
10
7422
19.12
19.09
"9". We represent each digit j = 0, .,. by a one-hot vector Cj+1 E R100 whose (j + 1)-th component is one and all others are zero, and denote by To : 784 - 10 the classifier, which is piecewise constant function defined by To (x) Cj-1 if x represents the digit j. We split the available training samples to form the training and validation data, denoted as Do = {xe, To T0(xx))oeNNno.o and
Do = {x'e To(x') ( iEN..... respectively, with Rerain 45, 000 and Rival == 15, 000, and use the testing samples as the testing data, denoted as D' = {x",, Tx(" e)) with Neest = 10, 000.
Clearly, Do, Do' and ID" are subsets of {10, 18784, (es+1)}=00 Letting (x)(x) == sin
radial wave defined the input space 784 we define the target function by (x) : To (x(x) (1 + BBb (x)) where is the frequency of the wave and 3 is the amplitude. Note that 7g and Bxyr contribute respectively the lower- lower-frequency and high-frequent
corresponding
components (regarded as noise) of the target function TB.r, as discussed in [30]. The modified training and
validation data denoted by Dg = {xe, T3 (xxe) EEN., crain
and D/ (x/, Ta (x/))EE
respectively, are used to train DNNs. Our goal is to use SGDL and MDGL to regress the modified data a through minimizing their respective training loss, compare their robustness to noise. The training loss is evaluated on De.x and validation loss is on D/ 8.x. TrRSE, VaRSE, and TeRSE are
evaluated on De. D D' g and D' respectively, noting that DD test data without noise.

---

Table 4: Accuracy comparison: SGDL versus MGDL with
method
time (8)
TrRSE
VaRSE
TeRSE
SGDL
3, 3,298
3.1 X 10
4.1 X 10
1.1 X 10
MGDL
3, 109
3.5 x 10-1
3.9 x 10
8.0 x 10-2
SGDL
3,333
3.0 x 10
x.1'X00'0
1.1 T0
5
MGDL
3.461
3.5 x 10-1
3.9 x 10
7.8 x 10
SGDL
199
3.1 10
4.1x100'1
1.0 x 10
10
MGDL
3,448
3.5 x 10-1
3.9 x 10-1
7.8 10-2
SGDL
3,168
3.0 x 10
4.1 10-1
1.1 X 10
50
MGDL
3,484
3.5 x 10-1
3.9 x 10-1
7.9 X 10 10"2 2
We choose the amplitude 3 from {0.5, 1,3,1, 5). For each 8, we vary the frequency x. from (1,3,5, 5, 7, 10, 50}. These choices of 3 and result in functions TB..c that have higher frequen- cies than those studied in [O). Figures [4 and [5] (in Appendix (B.4) compare the training and validation Ioss versus training time of SGDL (structure (3) and MGDL (structures (14) and (T5) respectively) for different values frequency & and amplitude (depicted in the figure). We observe that for small 8, for example, 3 = 0.5, the results of the two models are comparable. When 3 = 1, the training loss for SGDL keeps decreasing as training time increases, while the validation loss initially decreases and starts to increase after training of 1,600 seconds, and unlike SGDL, both the training loss and validation loss for MGDL keep decreasing. This trend indicates that over-fitting phenomenon occurs for SGDL and suggests that MGDL has superior generalization capability. Further increasing the value of 3, for example, 3 = 3 and B = 5, the over-fitting phenomenon occurs in both of the models. An increase in / corresponds to higher proportion of high-frequency components within T- This increased proportion of the high-frequency component significantly impacts the validation loss of SGDL, a trend also observed in MGDL. However, the effect is comparatively less with MGDL than with SGDL. We present in Table 4 TrRSE, VaRSE, and TeRSE of SGDL (structure (13) and MGDL (structure (44) for 3 1. where the results are obtained by choosing bmin 10 and
tmaz == 10 and the batch size to be 'Full', for all cases of and for both SGDL and MGDL. We observe that while TrRSE of SGDL is smaller than that of MGDL, both VaRSE, and TeRSE of SGDL are larger than the corresponding values of MGDL, suggesting occurrence of overfitting with SGDL. MGDL's improvement in accuracy, measured by TeRSE, about 27 29%. This experiment reveals that MGDL is promising model for learning nigher-dimensional data with higher proportion of high-frequency features.
Conclusion
By observing that a fighffrequency function may be well-represented by composition of several lower oowerfirequency functions, have proposed a novel approach to Iearn such function. The proposed approach decomposes the function into a sum multiple frequency components, each of which is compositions of lower-frequency functions. By leveraging the MGDL model, we can express high-frege tigh-freueecy components by compositions of multiple SNNs of low-frequency. We have conducted numerical studies of the proposed approach by applying it to mie-dienniionll synthetic data, two-dimensional manifold data, colored images, and sighee-diimessionll MNIST data. Our studies have concluded that MGDL can effectively learn high-frequency components within data. Moreover, the proposed approach is easy to implement and not limited by dimension of the input variable. Consequently, offers a promising approach to learn high-frequency features within (high-dimensional) dataset.
Limitation: Mathematical understanding of the spectral bias of DNNs is absent. Theoretical foundation for MGDL to address the spectral bias issue needs to be established. Numerical studies preliminary, limited to four examples. More extensive numerical experiments will be conducted in our future work.
A Acknowledgments: Y. Xu is indebted to Professor Wei Cai of Southern Methodist University for insightful discussion of the spectral bias of DNNs. Xu is supported in part by the US National Science Foundation under grant RMS-2208386, and by the US National Institutes of Health under grant R21CA263876.
10

---

References
[1] Agustsson, E., & Timofte, R. (2017) Núre 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition Workshops, pp- 126-135.
[2] Arfken, G. B.. Weber, H. J., & Harris, E. (2011) Mathematical Methods for Physicists: A Comprehensive Guide. Orlando, FL: Academic press.
[3] Arpit, D., Jastrzebski, S., Ballas, N., Krueger. D., Bengio, E.,, Kanwal, M. S.,... & Lacoste-Julien, S. (2017) A closer look at memorization in deep networks. In International Conference Machine Learning, pp. 233-242.
[4] Bengio, Y., Courville, A.C., Vincent, P. (2012) Representation learning: review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8): 1798-1828
[5] Cai, W., Li, X., & Liu, L. (2020) A phase shift deep neural network for high frequency approximation and wave problems. SIAM Journal on Scientific Computing, 42(5): A3285-A3312.
[6] Cai, W., & Xu. Z. Q. J. (2019) Multi-scalc deep neural networks for solving high dimensional pdes. arXiv preprint Xivv 1910.11710.
[7] Cao, Y., Fang, Z., Wu, Y., Zhou, D., & Gu, Q. (2019) Towards understanding the spectral bias of deep learning. International Joint Conference on Artificial Intelligence, pp. 2205-2211.
[8] Carson, J. R. (1922) Notes on the theory modulation. Proceedings of the Institute of Radio Engineers 10(1): 57-64.
[9] Craster, R. V., Kaplunov, J., Pichugin, V. (2010) High-frequency homogenization for periodic media. Proceedings of the Royal Sociery A: Mathematical, Physical and Engineering Sciences, 466(2120); 2341- 2362.
[10] Cybenko, G. (1989) Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2(4): 303-314.
[11] Daubechies, I., Verer, R., Foucart, S., Hanin, B., & Petrova, G. (2022) Nonlinear approximation and (deep) ReLU networks. Constructive Approximation. 55(1): 127-172.
[12] Fang. R., Xu, problem. Journal of Scientific Computing, 100(2): 58.
Yan, M. (2024) Inexact fixed-point proximity algorithm for the to sparse regularization
[13] Glorot, X., & Bengio, Y. (2010, March) Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirleenth International Conference on Artificial Intelligence and Statistics. pp. 249-256.
[14] Goodfellow, L,, Bengio, Y., & Courville, A. (2016) Deep Learning. Cambridge, MA: MIT Press.
[15] He, K.. Zhang, X., Ren, S., & Sun, J. (2016) Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 770-778.
[16] Huang. J., Jiao, Y., Li, Z.. Liu, S., Wang, Y. & Yang, (2022) enror analysis generative adversarial networks for learning distributions. Journal of Machine Learning Research, 23(116): 1-43.
[17] Hornik, K., Stinchcombe, M., & White, H. (1989) Multilayer feedforward networks are universal approxi- mators. Neural Networks, 2(5): 359-366.
[18] Jagtap, A. D., Kawaguchi, K., & Karniadakis, G. E. (2020) Adaptive activation functions accelerate convergence in deep and 109136.
neural networks. Journal Computational Physics, 404:
[19] Jiang, J.. Xu, Y. (2024) Deep neural network solutions oscillatory Fredholm integral equations. Journal of Integral Equations and Applications 36(1): 23-55.
[20] Kingma, D.P.& Ba, J. (2015) Adam: A method for stochastic optimization. Proceedings of the 3rd International Conference on Learning Representations (ICLR).
[21] LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) Gradient-basse learning applied to document recognition. Proceedings of the IEEE 86(11): 2278-2324.

---

[22] Li. Y., Lu, S., Mathé, P., & Pereverzev, S. V. (2024). Two-layer networks with the ReLU* activation Function: Barron spaces and derivative approximation. Numerische Mathematik, 156(1): 319-344.
[23] Liu, Z., Cai, W. & Xu, Z. Q. J. (2020) Multi-scale deep neural network (MscaleDNN) for solving Doisson-Boltzmane equation in complex domains. Communications in Computational Physics, 28(5): 1970- 2001.
[24] Lu, Y., Shen, L., & Xu, Y. (2010) Integral equation models for image restoration: high accuracy methods and fast algorithms. Inverse Problems, 26(4): 045006.
[25] Mhaskar, H. N., & Poggio. T. (2016) Deep shallow networks: An approximation theory perspective. Analysis and Applications, 14(06): 829-848.
[26] Nassif, A. B., Shahin, I,, Attili, I., Azzeh, M., & Shaalan, K. (2019) Speech recognition using deep neural networks: A systematic review. IEFE Access, 7: 19143-19165.
[27] Ozbayoglu, A. M., Gudelek, M. U., & Sezer, O. B. (2020) Deep learning for financial applications: A survey. Applied Soft Computing. 93: 106384.
[28] Perkins, D. H. (2000) Introduction to High Energy Physics. Cambridge: Cambridge University Press.
[29] Pieper, R. J. (2001) Laboratory and computer tests for Carson's 's FM bandwidth rule. In Proceedings of the 33rd Southeastern Symposium on System Theory (IEEE), pp.. 145-149.
[30] Rahaman, N., Baratin, A., Arpit, D., Draxler, F., Lin, M., Hamprecht, F., Bengio, Y., & Courville, A. (2019) On the spectral bias of neural networks. In International Conference on Machine Learning, pp. 53.1530-55 5301-5310
[31] Ronen, B., Jacobs, D., Kasten, Y., & Kritchman, S. (2019) The convergence rate of neural networks for learned functions of different frequencies. Advances in Neural Information Processing Systems, 32.
[32] Schulze, J.P., Kraus, M., Lang, U., & Ertl, T. (2003) Integrating pre- re-iineegattonn into the shear-warp algorithm. In IEEE VGTC / Eurographics International Symposiun on Volume Graphics, pp. 109-118.
[33] Shanmugam, K. S. (1979) Digital and Analog Communication Systems. New York: Wiley.
[34] Shen, I.., Lin, Z., Huang, Q. (2016) Relay hackpropagation for effective learning of deep convolutional neural networks. European Conference on Computer Vision, pp. 467-482.
[35] Shen, Z., Yang, H.,& & Zhang, S. (2022) Optimal approximation rate of ReLU networks in terms of width and depth. Journal de Mathématiques Pures et Appliquées, 157: 101-135.
[36] Tancik, M., Srinivasan, P., Mildenhall, B., Fridovich-Keil S., Raghavan, N., Singhal, U. ... & Ng. R. (2020) Fourier features let networks learn high frequency functions low dimensional domains. Advances in Neural Information Processing Systems, pp. 7537-7547.
[37] Tang, X., Schmidtlein. C. R., Li, S., Xu, (2021) An integral equation model for PET imaging. International Journal of Numerical Analysis and Modeling, 18(6): 834-864.
[38] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017) Attention is all you need, Advances in Neural Information Processing Systems, pp. 5998-6008.
[39] Wu, T., & Xu, Y. (2022) Inverting incomplete Fourier transforms by sparse regularization model and applications in seismic wavefield modeling. Journal of Scientific Computing, 92(2): 48.
[40] Xu, Z. J. (2018) Understanding training and generalization in deep learning by fourier analysis. arXiv preprint
[41] Xu. Z. Q. J., Zhang, Y., & Luo, T. (2022) Overview frequency principle/spectral bias in deep learning. arXiv preprint arXiv:2201.07399
[42] Xu, Z. J., Zhang, Y. & Xiao, Y. (2019) Training behavior of deep neural network in frequency domain. International Conference on Neural Information Processing, pp. 264-274.
[43] Xu, Y. (forthcomingg Multi deep learning. Communications on Applied Mathematics and Computa- tion, arXiv preprint arXiv:2302.00150 published on February 2023.
[44] Xu, Y. (2023) Successive affine learning for deep neural networks. ArXiv, abs/2.055007996, July 11, 2023.
[45] Xu, Y. & Zhang, H. (2022) Convergence of deep convolutional neural networks. Neural Networks. 153: 553 563.563.
12

---

[46] Xu, Y. & Zhang, H. (2024) Convergence of deep ReLU networks. Neurocomputing 571: 127174.
[47] Xu, Y. Zhang, H. (2024) Uniform convergence of deep neural networks with Lipschitz continuous activation functions and variable widths, IEEE Transactions on Information Theory, 70: No. 10, October 2024.
[48] Zhang. C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2021) Understanding deep learning (still) requires rethinking generalization. Communications of the ACM 64(3): 107-115.
[49] Zhou, D. X. (2020) Universality of deep convolutional neural networks. Applied and Computational Harmonic Analysis, 48(2): 787-794.
A Analysis of the Multi-Grade Deep Learning Model
It was established in 43 that when the loss function is defined in terms of the mean square error, a grade of MGDL either learns the zero function or results in a strictly decreasing residual error sequence.
Theorem 1. Let D be a compact subset of IR* and L2,,, R denote the space of t-dimensional vector-valued square integral functions on D. Iff L2 (2, RRt, then for all 1,2,....
Σ feee, f = Ni o Ni o
where No is the SNN learned in grade of MGDL, and for i 1,2,... either f(+1 = 0 or
|| < ||e|||-
All numerical examples presented in this paper validate Theorem !
B Supporting material for Section 3
We provide in this appendix details for the numerical experiments in Section 3 including computa- tional resources, network structures of SGDL and MGDL, the choice of activation function, well as the optimizer and parameters used in the optimization process, and some supporting figures.
The experiments conducted in Sections 3. 3.2, and 3.4 of Section 3 were performed on X86 64 server equipped with an Intel(R) Xeon(R) Gold 6148 CPU @ 2.4GHz (40 slots) or Intel(R) Xeon(R) CPU ES- v3 @ 2.30GHz (32 slots). In contrast, the experiments described in Section 3.3 of Section 3 were performed on X86_6e server equipped with AMD 7543 @ 2.8GHz (64 slots) and AVX512, 2 Nvidia Ampere A100 GPU.
We choose ReLU as the activation function as in 30 for all the four experiments. For each experiment, for SGDL we test several network structures and choose the one produces the best performance. We then design MGDL, having the same total number of layers and the same number of neurons for each layer as the chosen SGDL structure, but their parameters are trained in multiple grades as described in Section 2 Details of the network structure will be described for each experiment.
The optimization problems for both SGDL and MGDL across the four experiments are solved by the Adam method [20] with 'Xavier' initialization [I]. In Sections 3. 3.2, and 3.4, the learning rate lx for the k-th epoch decays exponentially with each epoch [19]. calculated as 44 == m ->k where 7 = (1/K) (/(4ma///mmm)) represents the decay rate, with K being the total number of training epochs, maa and I'mn denoting the predefined maximum and minimum learning rates, respectively. In Section 3.3, we employed fixed learning rate for both SGDL and MGDL, as numerical results indicate that the exponential decay learning rate performs poorly. We observe that when a network structure of SGDL is split into multiple grades of MGDL, the combined computing time required to train all grades in MGDL, each for K epochs, is comparable to the computing time needed to train the SGDL with K epochs, due to only SNNs involved in MGDL. Therefore, in all the four experiments, we train SGDL and all grades of MGDL for the same K epochs. We test SGDL and MGDL with the same set of the algorithm parameters, including imxi Amans batch size, and K, for all the experiments. Optimal parameters were selected based on the lowest VaRSE value for Sections 3.1 and 3.2, the highest PSNR value for Section 3.3, and the lowest validation loss for Section 3.4, within the range of parameters to be described for each example.
13

---

Frequency
Frequency
Prequency
Figure Amplitude versus one-side frequency plots for across settings 1-4.
B.1 Section 3.1
For settings 1, 2, and 3, the network structure for SGDL is
[1] + [256] 8 + [1].
(11)
where [n] N indicates N hidden layers, each with n neurons. The network structure for each grade
of MGDL is given by
Grade [1] + [256] x + [1]
Grade 2: [1] + [256] F x 2 + [256]
x 2 + [1]
Grade [1] + [ 4 [256]
2 [1]
Grade 4: [1] + [256] x 6 - [256]
x 2 + [1].
Here, [n]F indicates layer having n neurons with parameters, trained in the previous grades, remaining fixed during the training of the current grade. We employ this notation across the numerical experiment section without further mentioning, For setting 4, the most challenging case, we employ deeper structure for SGDL, with two more hidden layers in addition (II). Correspondingly, for MGDL we add one more grade:
Grade 5: [1] + [ x 8 + [256] x 2 + [1].
We now describe the search range the parameters for all the four settings. We let 4 := {10- 1003 10-1101--- 6). 12 == {10-5 3 10-4, 10- 10-4), 1a = {10 10 6 10 7} and 17 the pair (4min; Lmax) from all possible cases in the set (1g 12) U (1, x 44). The batch size is chosen from 256, 512 or the full gradient (denoted by 'Full') for each epoch. The total epoch number is set to be 30,000.
(5 x 10-5, 10-51- Then for both SGDL and MGDL,
The supporting figures for this experiment include:
Figure
7
presenting amplitude versus one-side frequency plots for À across settings 1-4;
Figure
comparing the amplitude of one-side frequency for SGDL and MGDL across
settings -4;
Figure
9
illustrating the evolution of spectrum comparison between SGDL and MGDL for
settings
1-4.
In Figure the colors in these subfigures show the measured amplitude of the network spectrum at the corresponding frequency, normalized by the amplitude of the same frequency. The colorbar is clipped between O and 1, indicating approximation accuracy from the worst to the best when changing from 0 to 1.
B.2 Section 3.2
The network structure that we use for SGDL
[2] + [256] 8 + [1],
and the grade network structure for MGDL is
Grade [2] + [256] x 2 + [1]
Grade 2: [2] + [256]r 2 + [256] 2 [1]
Grade [2] + [256] 4 + [256]
[1]
Grade 4: [2] + [256] 6 + [256] 2 + [1].
14

---

Femoltuc
Femolbuc
Frequency
Frequenccy
Procuenccy
Ampladde
E.
Frequency [H2]
Prequency [H2)
Frequency [H2)
Prequency [H2]
Figure 8: Comparison of SGDL (Ist row) and MGDL (2nd row): Amplitude across settings 1-4.
10000
30000
Excdh (SDDL)
10000 19000
Eppers (5DDL)
20030
(SDD))
20000
Epcetcc SSDD
20000
10000
Exodh
10000
150
201
102
1500
50
1.00
200
105
150
750
Proquancy [H9]
Proqunncy [H2]
Proquency [H9]
Prequanccy [HO]
01
Fraquency [Mir]
Prequency (MK)
Froquency [MT)
Prequency [MV]
Figure 9:
Comparison of SGDI. row) and MGDL. (2nd row) for settings 1-4 of Section 3.1: The
evolution of spectrum.
For choices of Imin and Imazi we let 1 = {10- 10 1,-5,10-00 10-1)
and I2
{1 x 10 3 5 x 10-4, 1 x 10 10-4). For both SGDL and MGDL,, we test the pair (Iminy lmaz) from all possible cases in the set Tn T2, the batch size is chosen from 512, 1024, or the full gradient for each epoch, and the total number of the epochs K" set to be 30,000.
The supporting figures for this experiment include Figure 10 illustrating the evolution of spectrum comparison between SGDL and MGDL for settings 1-2 of Section 3.2. The mean of colorbar in Figure i0 consistent with that in Figure
B.3 Section 3.3
The network structure for SGDL
[2] > [256] x 12 [3]
and that for MGDL is
Grade I: [2] + [256] x 3 + [3]
Grade 2: [2] [256] + [256] x 3 - [3]
Grade 3: [2] - [256] x - [256] x - [3]
Grade 4: [2] [256] x + [256] x 3 [3]
For SGDL and all grades MGDL,, we select the learning rate from the set
10-3, 5x
10-1 11101 choose the full gradient for each epoch, and set the total epoch number K to be 10, 000.
15

---

30030
100.00
Exoctt (BRL
Epento EOR 10000 20000
Exocdts (SGU
Epochh (S0L)) 19000 70000
205
04
9.4
300
200
300
Figure 10: Comparsion SGDL (Ist row) and MGDL (2nd row) for settings and Section 3.2: The evolution of spectrum (the first and second columns for the learned functions manifolds 7g with = and = 0, respectively, for setting 1, while the third and fourth columns for setting 2).
500
500
(a) G I: 20.41
(b) G 2: 22.67
G 3: 23.71
200
100
100
200
200
300
500
200
.02
-00
200
(d) 24.18
(e) SGDL: 22.84
(0) Ground Truth
Figure 11: Comparison of MGDL and SGDL for image Cat. (a)-(d): Predictions of MGDL for grades
1-4, with the corresponding testing PSNR values indicated in the subtitles. (e): Prediction of SGDL. with testing PSNR displayed the subtitle. (f): Ground truth image
The quality of the reconstructed image in Section 3.3 is evaluated by the peak signal-to-noise ratio (PSNR) defined by
n x 2552
PSNR == 10 log10
(12)
||v VI
where v is the ground truth image, v is reconstructed image, n is the number of pixels in v. and ||-I|r denotes the Frobenius norm of matrix.
The supporting figures for this experiment include Figures I1 which display the predicted image for MGDL and SGDL corresponding to the Cat, Sea, and Building images, respectively.
16

---

390
600
100
200
(a) 18.62
(b) G 2: 21.50
(c) G 3: 23.42
030
(d) G 4: 24.32
SGDL: 20.39
(f) Ground Truth
Figure 12: Comparison of MGDL and SGDL for image Sea. (a)-d)) Predictions of MGDL for grades 1-4, with the corresponding testing PSNR values indicated in the subtitles. (c): Prediction of SGDL with testing PSNR displayed in the subtitle. (f): Ground truth image
1730
(a) 17.29
(b) G 18.95
(c) G 20.67
224
206
700
1000
2000
1008
2000
200
200
300
150
2000
1250
3000
1751
200
159
500
150
2000
1250
1000
D00
200
(d) 4: 21.97
SGDI:: 19.09
(f) Ground Truth
Figure 13: Comparison of MGDL and SGDL. for image Building. (a)-(d): Predictions of MGDL for grades 1-4, with the corresponding testing PSNR values indicated in the subtitles. (c): Prediction of SGDL with testing PSNR displayed in the subtitle. (f): Ground truth image
17

---

SGDL
MGDL
SGDL
MGDL
614
Lo55
Loss
035
303
030
able
1809
2080
2000
a0e
Training time [s]
Training time [s]
Training time [s]
Training time [s]
(b)
SGDL
MGDL
SGDL
6
2.5
Lo55
Loss
10
2i03
wid
Training time [s]
Training time [s]
Training time [s]
Training time [s]
(c)
(d)
Figure 14: Comparison of SGDL and MGDL with structure (14): training (dash curve) and validation
(solid curve) loss versus training time for various values of B and (a) 3 0.5, (b) B = 1, (c) 3, (d) 5.
B.4 Section 3.4
The network structure for SGDL is
784] + [128] x 6 + [10].
(13)
For MGDL, we consider two grade splittings. In the first splitting, we split network (33 into three grades, each with two hidden layers. The structure of MGDL for this splitting is as follows:
Grade 1: [784] + 128] x 2 - [10]
Grade 2: [784] 128 2 128
[10]
(14)
Grade 3: [784] + 128] F x + 128]
x - [10].
In the second splitting, we split network 13) into six grades, with each grade containing one hidden layer. The structure for MGDL for this case is as follows:
Grade 1: [784] + 128] - [10]
Grade 2: [784] + [128] F + [128]
[10]
Grade 3: [784] + [128]r x > [128] + 110]
(15)
Grade 4: [784] + [128) x [128]
[10]
Grade 5: [784] + [128] x [128]
- [10]
Grade 6: [784] [188] F 5 128
- 110]
For choices of parameters Eman and tmez. we let 1 := both SGDL and MGDL, we test (fmmn: fmar) from all possible cases in the set J x Iz, choose the batch size from 512, 1024, the full gradient for each epoch, and set the total number of the epochs K be 2,000.
and Le = (10-3, 10-4). For
The supporting figures for this experiment include Figure 14, which compares the training and validation loss for SGDL and MGDL with structure (14) Figure 15 which compares the training and validation loss for SGDL and MGDL with structure [5
18

---

SGDL
MGDL
SGDI
MGDL
614
Lo55
Loss
035
303
030
2000
able
sWe
200
200
129
Training time [s]
Training time [s]
Training time [s]
Training time [s]
(b)
SGDL
MGDL
10
1906 Tile Training time [s]
via
2003 Training time [s]
Training time [s]
Training time [s]
(d)
Figure 15: Comparison of SGDL and MGDL with structure 15 training (dash curve) and validation
loss (solid curve) versus training time for varies values of 3 and (a) B = 0.5, (b) B 1. (c) B = 3, (d) 3 = 5.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim the paper's contributions accurately and clearly in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations after the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings,
19

---

model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The loss functions for SGDL and MGDL are provided. In each experiment, we clearly state experimental settings, including how to choose training/validdtiionnessssgg data. The structure of SGDL and MGDL, and the parameters required in training are provided in The computer code available online through the GitHub link provided in the abstract.
Justification:
B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
20

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The computer code is available online through the GitHub link provided the abstract. A README.1x file containing guidelines for using the code for each example is also provided.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https http://gnideeeeoeeeobbeea
//nips
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
21

---

Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes)
Justification: The loss functions for SGDL MGDL are provided. each experiment, we clearly state experimental settings, including how we choose raining/validation/eesstt data. The structure of SGDL and MGDI,, and the parameters required in training are provided in
B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide tables of relative mean square error accuracy and PSNR values, along with figures showing the training/validation loss and PSNR values throughout the training process.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma eтror bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the experiments compute resources in Appendix B
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
22

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.ce httpp//ps...pppbbtt//ccciciiiiia
Answer: [Yes]
Justification: The research is conducted under the guidance of Code Of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impacts in the conclusion section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
23

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We utilize the Div2K and MNIST datasets and properly cite their creators.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
25


Reviews:
[{"review_id": "vcqytlrkcF", "reviewer": "Reviewer_vDtJ", "summary": "DNNs suffer from spectral bias, which is the tendency to prioritize learning of low-frequency functions. The authors propose a method called MGDL, which tackles the spectral bias problem by changing the network parameterization and loss function such that each layer in the network learns the residual of the function from the previous layers. This framework forces higher layers in the network to learn high frequency residuals rather than an additive low frequency function. The authors justify their method using regression experiments on synthetic and MNIST data.", "questions": "(2) Is there a reason you are constrained to the MSE loss? Does this idea work on other loss functions? \n\n(3) I am confused by Figure 3. I see that this is a plot of the functions learned by each layer in the frequency domain, but what is the main take away? For example, Grade 4 does not seem to be learning anything in the first two settings, whereas according to my intuition it should be learning the highest frequencies.", "limitations": "Adequately addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea presented is clear and intuitive. \n\n - I found section 2 to be easy to understand despite the many different notations used.\n\n- The experiments are organized in a logical order. Figure 4 clearly shows the advantage of MGDL in modeling functions with different frequency components.\n\n- Lastly the authors are honest in acknowledging the lack of theoretical justification of MGDL.", "weaknesses": "(1) The main weakness (and probably the most obvious) is the question of scalability. There is a heavy focus in this paper on synthetic experiments. The only real world experiments are done on MNIST. I am uncertain as to whether this work has the potential to be adopted for any real applications (in computer vision or otherwise). In other words, I think the impact is limited due to the lack of more complex experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the referee's constructive comments and insightful suggestions on this study. Below, we reply the weaknesses and questions term by term.\n\nResponse to weaknesses (1):\n\nThe referee's criticism is well-taken. This study is to provide insights of MGDL and proof-of-concept numerical examples to understand how MGDL captures the high-frequency components of the target function. This paper represents our initial step in demonstrating the effectiveness of the MGDL method. From what we have learned, MGDL is scalable, since in general its training time increases linearly as the number of grades increases. We plan to apply MGDL to more real-world applications where capturing high-frequency components is crucial.\n\nResponse to Question (2):\n\nThe idea of MGDL can also be applied to other loss functions. We applied recently MGDL to image classification using the cross entropy loss and it works well. The reason for which we confine ourselves in this paper to the MSE loss is that for an MSE loss, it has been shown in (Xu, Y. (2023). Multi-grade deep learning. arXiv preprint arXiv:2302.00150.) that the loss decreases as the number of grades increases and regression problems naturally lead to MSE losses.\n\nResponse to Question (3):\n\nFigure 3 is to show how MGDL learns the functions defined by equation (8) with different amplitudes, shown in Figure 2. Columns 1, 2, 3 and 4 have constant, decreasing, increasing and variable amplitudes, respectively. For each of the columns, Grade 1 learns lower-frequency components (blue), Grade 2 learns middle-frequency components (orange), Grade 3 learns higher-frequency components (green), and Grade 4 learns the highest-frequency components (red). Since the function of Column 1 has constant amplitudes, grades 1-3 basically learn most of its useful information. Column 2 shows a similar behavior. However, Column 3 shows that highest-frequency components learned by Grade 4 are significant, since the function has increasing amplitudes, that is, higher-frequencies have larger amplitudes.  Your observation is correct that in the first two settings, Grade 4 does not learn much. This is because for the functions of these two settings, the amplitudes of higher-frequencies are {\\it proportionally} small. However, Grade 4 is still important for these two settings. As shown in Figure 5, Grade 4 reduces the loss from $10^{-4}$ to $10^{-6}$. That indicates that if we want a high precision, we need grade 4. \n\nFor setting 3, Grade 4 is very significant. This function is the one among all four settings most difficult for SGDL to learn. Figures 3 and 4 show that MGDL learn higher-frequency (with large amplitudes) much more effectively than SGDL, see Column 3 of Figure 4 (Row 1 for SGDL and Row 2 for MGDL). MGDL can learn most of the highest-frequency components but SGDL cannot.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NkVDbgTPZ1", "reviewer": "Reviewer_GtgE", "summary": "This paper tackles the spectral bias of deep neural networks by proposing a method to capture high-frequency data effectively. The authors assert that previous works have not adequately addressed spectral bias, emphasizing its significance. They propose that high-frequency functions can be decomposed into a sum-composition of low-frequency functions. To implement this, they employ a multi-grade deep learning (MGDL) model to achieve the sum-composition functions from the original function. The authors provide empirical evidence of the effectiveness of this approach in three distinct scenarios, including high-frequency noise-injected MNIST.", "questions": "Please see the above weaknesses.", "limitations": "- Limitations are provided", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper is easy to follow and well-written.\n- The logical progression from the motivation that high-frequency components can be estimated by the sum of composites of low-frequency components is sound.\n- The proposed method would likely address scenarios involving high-frequency noises or data containing high-frequency components.", "weaknesses": "- The paper fails to highlight why addressing spectral bias is significant. Recent network architectures perform well in various \ncomputer vision fields without specifically addressing this issue, where the datasets they use mostly contain high-frequency noises or noisy data themselves. From my perspective, the advantage of this paper in handling high-frequency components primarily benefits specific synthetic scenarios, such as those provided by the authors. I believe there would be more compelling examples to underscore the importance of the matter where the claim is significant. I recommend that the authors provide more practical scenarios where not addressing spectral bias leads to tangible failures, along with a rationale for why it is important to capture high-frequency elements.\n\n- While Figure 1 demonstrates the effectiveness of sum-composition in shallow deep neural networks, it does not provide a rationale for freezing components in each composition as done in MGDL. Based on this motivation alone, other learning methods beyond MGDL could potentially achieve the same or even better results. In this regard, why was MGDL exclusively used for the experiments? Considering that a residual network (ResNet) also employs a sum-composition approach, why wasn't it included in the experimental analysis?\n\n-  I believe the major problem with this paper lies in the experiment section. It provides insufficient experimental evidence to substantiate the authors' claims.\n  - First of all, as aforementioned, the scope of the experiments is very limited, involving only three synthetical scenarios and experiments with a fixed network architecture for each scenario. Additionally, comparisons are made solely with SGDL.\n  - MGDL appears to capture diverse frequency successfully ranges through multiple grades (whereas SGDL does not). However, I suspect this outcome may stem from learning instabilities caused by a simplistic network architecture lacking 1) normalization layers; 2) skip connections (i.e., shortcuts); and 3) freezing some layers for MGDL. In this regard, a deep SGDL is highly likely to encounter training failures due to gradient issues, whereas MGDL freezing some layers may offer more training stability in reducing loss. Therefore, I believe that any methods or networks that train shallow networks sequentially could potentially benefit in a manner similar to [1]. All experiments should be reperformed to provide more convincing experimental support. \n    - [1] Relay Backpropagation for Effective Learning of Deep Convolutional Neural Networks, ECCV 2016\n \n- The experiments on high-dimensional data are limited to MNIST, which does not convincingly demonstrate the challenges of high-dimensional data as claimed. For example, the standard ImageNet training protocol uses 224x224 images, resulting in a dimension of 50,176 when vectorized,\n\n- Figure 8 contains very small fonts, which should be refined for better readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the referee's constructive comments and insightful suggestions on this study. Below, we reply the weaknesses term by term.\n\nResponse to weakness item (1):\n\nWe are grateful to the referee for this suggestion. This study is to provide insights of MGDL and proof-of-concept numerical examples to understand how MGDL captures the high-frequency components of the targeted function. This paper represents our initial step in demonstrating the effectiveness of the MGDL method. In future studies, we plan to apply MGDL to real-world applications where capturing high-frequency components is crucial. Many physical problems involve finding high-frequency solutions in complex domains, such as seismic wavefield reconstruction, high-frequency wave equations in homogenization periodic media, and high energy physics. In such applications, accurately learning high-frequency components within a target function is crucial, making these areas potential applications for MGDL method.\n\nResponse to weakness item (2):\n\nWe agree with the referee that motivation for the MGDL method can be better explained in this paper. Below, we briefly state the motivation of MGDL method (as discussed in Xu, Y. (2023). Multi-grade deep learning. arXiv preprint arXiv:2302.00150.).\nThe motivation behind MGDL is to tackle complex tasks by decomposing them into sequential, simpler tasks, inspired from the human learning process. In human learning, knowledge is learned grade by grade, with each grade building on the knowledge learned from previous one. The foundational knowledge from earlier grades remains relatively stable and serves as a basis for more advanced learning in later grades. This principle motivates the design of the MGDL method. There are two reasons for freezing the components learned in previous grades. First, we observe that foundational knowledge remains relatively stable throughout the human learning. Second, by freezing these components, we only need to train a shallow network, which avoids training a deep network.\n\nResNet also employs a sum-composition approach. The difference between ResNet and MGDL is that ResNet learns a complex task at once, training a deep network from end to end, whereas MGDL learns a complex task incrementally, grade by grade, training a shallow network at each grade. \n\nMGDL is a principle that can be applied to various existing learning models, including standard deep neural network (SGDL) and ResNet. This paper represents our initial step in demonstrating the effectiveness of  MGDL, and thus we have simply applied MGDL to the standard deep neural network. More extensive numerical experiments, including MGDL applied to CNN and ResNet, will be conducted in our future work.  \n\nResponse to weakness item (3):\n\nThe referee's criticism is well-taken. We plan to apply MGDL to more real-world applications where capturing high-frequency components is crucial.\n\nResponse to weakness item (4):\n\nFor the synthetic scenarios, we fixed network structure to 8 hidden layers and varied the width of each hidden layer from 32, 64, 128, 256. The experiments showed that a width of 256 yields best performance for both SGDL and MGDL. Consequently, we adopted this configuration for the architecture described in the paper.  For the MNIST dataset, we used a fixed network architecture with 6 hidden layers and experimented with layer widths of 128 and 256. The performance results were similar for both widths. Therefore, we selected a width of 128 for the final network configuration. \nIn our future study, we will compare MGDL with more advanced networks, such as CNNs and ResNet. \n\nResponse to weakness item (5):\n\nThe difficulty of SGDL in learning the high-frequency components have been discussed and verified in several references, including Rahaman, Nasim, et al. 'On the Spectral Bias of Neural Networks,' International Conference on Machine Learning, PMLR, 201 and Xu, Zhi-Qin John, Yaoyu Zhang, and Yanyang Xiao. 'Training Behavior of Deep Neural Networks in the Frequency Domain,' Neural Information Processing: 26th International Conference, ICONIP 2019. This is the main motivation for us to address this issue.\n\nIn this study, we employed  a simple network architecture without using the normalization and skip connections to ensure that the observed improvements are only due to the change in networks from SGDL to MGDL.  We will incorporate these techniques, such as the normalization and skip connection, on SGDL and MGDL in our future studies. \n\nWe appreciate the referee to provide the reference [1].  The method in [1] also involves training several shallows networks, but it trains several shallow networks at once. In contrast, MGDL trains several shallow networks sequentially, freezing the previous network (serving as features or ``bases'') as we move on to train the next grade.  We plan to compare performance of the method in [1] and MGDL in our future work.\n\nResponse to weakness item (6):\n\nThank the referee for the suggestion. We plan to apply MGDL to more challenging datasets in our future studies.\n\nResponse to weakness item (7):\n\nThanks for pointing this out. We will modify Figure 8.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jwySDIyLfx", "reviewer": "Reviewer_otLk", "summary": "This paper addresses the issue of spectral bias in deep neural networks (DNNs), where DNNs tend to prioritize learning lower-frequency components of a function, struggling with high-frequency features. The proposed solution involves decomposing high-frequency functions into compositions of low-frequency functions using a novel approach termed Multi-Grade Deep Learning (MGDL). MGDL incrementally trains DNNs, each grade focusing on learning low-frequency information from the data. The efficacy of MGDL is demonstrated on synthetic, manifold, and MNIST datasets, showing improvements in capturing high-frequency components compared to traditional methods.", "questions": "1. Is the MGDL method mentioned in the paper applicable to other model architectures such as CNNs and Transformers?\n2. Does the training time of MGDL increase linearly with the number of grades?\n3. Please add the following citation as it is one of the earlier methods proposed to alleviate slow convergence for high frequencies:\n   Cai W, Xu Z Q J. Multi-scale deep neural networks for solving high dimensional PDEs. arXiv preprint arXiv:1910.11710, 2019.", "limitations": "See weeknesses.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Quality: The paper provides a thorough theoretical foundation and supports its claims with experimental results, although these are not entirely convincing.\nClarity: The paper is generally well-organized, with clear explanations and effective use of figures and mathematical expressions.", "weaknesses": "Originality: lacks new theoretical contributions for MGDL method. \nPerformance and Stability: There are concerns about the performance and stability of MGDL, as evidenced by loss spikes and lack of significant loss reduction in real-world tasks (Figure 8).\nMinor Punctuation Errors: For example, On Table 1,\"32,401\" and \"27, 817\".\nImage Readability: The legends of figures in this paper are too small, making them difficult to read and interpret.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the referee's constructive comments and insightful suggestions on this study. Below, we reply the weaknesses and questions term by term.\n\nResponse to weakness:\n\nThe aim of this study is to show that MGDL can be used to address the spectral bias issue of deep neural networks by providing  proof-of-concept numerical examples to illustrate how MGDL captures the high-frequency components of the target function. This paper represents our initial step in demonstrating the effectiveness of the MGDL method.  We plan to apply MGDL to more real-world applications where capturing high-frequency components is crucial.\n\nWe are afraid that the referee misunderstood the punctuation in Table 1. The number `32,401' in the table indicates that it takes 32401 seconds (approximately 9 hours) to train the model. \n\nWe will modify the legends of figures to improve the readability.\n\nResponse to question (1):\n\nWe have recently applied MGDL to CNNs for an image classification problem, showing that the test accuracy improves as grade increases. We haven not yet applied MGDL to transformers,  but we plan to explore this in our future work.\n\nResponse to question (2):\n\nIf the network structure and the number of training epochs are consistent across all grades, then the training time of MGDL increase linearly with the number of grades. At each grade, MGDL trains a shallow network, which makes it advantageous computationally in training, since training of a shallow network is much easier and more stable than training a deep network.\n\nResponse to question (3):\n\nThanks for bringing this reference to our attention. We will discuss the relevance of this reference in our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are grateful to the referees for their constructive comments and insightful suggestions on this study. Various points made by the referees are valuable. A `global' reply to the referees is as follows. 

(1) MGDL is mainly motivated from three considerations. First of all, it is inspired by the human education system. Human education is arranged in grades. In such a system, students learn a complex subject in grades, by decomposing it into sequential, simpler topics. Foundational knowledge learned in previous grades remains relatively stable and serves as a basis for learning in a present and future grades. This learning process can be modeled mathematically by representing a function that contains higher-frequency components by a 'sum-composition' form of low-frequency functions. Namely,
for a function $g$ of high-frequency, we decompose it in a 'sum-decomposition' form as
$$
    g=\sum_{k=1}^K \bigodot_{j=1}^k g_j,
$$
where
$
\bigodot_{j=1}^k g_j:=g_k\circ\cdots\circ g_2\circ g_1,
$ 
with $\circ$ denoting the composition of two functions, and $g_j$, $j\in\mathbb{N}_k$, are all of low-frequency. The lower-frequency functions $g_j$ can be effectively represented by shallow networks. This is the second consideration. The third consideration is that a shallow network can be effectively trained but it is more involved to train a deep network since it is highly non-convex with large number of parameters.

(2) MGDL avoids training a deep network from end to end.  Instead, it trains several shallow networks sequentially, with the current grade making use the shallow network learned from the previous grade as features and composing it with a new shallow network to learn the residue of the previous grade. MGDL decomposes a function that contains higher-frequency in a form of the above displayed equation, with $g_j$ being a shallow network learned from grade $j$. In this way, higher-frequency components in the data can be effectively learned in a grade-by-grade manner.
    
MGDL involves training several shallow neural networks instead of one deep neural network. This design makes MGDL more efficient, as the training time of MGDL increases linearly with the number of grades. This linear increase in training time makes MGDL an effective and scalable solution for tackling complex tasks. 

MGDL is an adaptive approach by nature. When the outcome of the present grade is not satisfactory, we can always add a new grade without changing the previous grades.

(3) The aim of this study is to understand empirically how  MGDL captures high-frequency components of a targeted function by using synthetic examples and the MNIST dataset. Specifically, our numerical examples in section 3.1 and 3.2 illustrate that Grade 1 learns lower-frequency components, Grade 2 learns middle-frequency components, Grade 3 learns higher-frequency components, and Grade 4 learns the highest-frequency components. These examples demonstrate that higher grades are efficient to capture higher-frequency components within the targeted function.

(4)  We plan to apply MGDL to other real-world problems where abstraction of higher-frequency information from given datasets is essential.

(5) MGDL is a principle that can be applied to various existing learning models, including standard deep neural network, CNN, and ResNet. Based on the improvements observed in our numerical experiments, we believe MGDL is a promising method in addressing the spectral bias issue.

Meta-review: Thanks for your submission to NeurIPS.

This was very much a borderline paper.  On the one hand, reviewers appreciated the clarity of the paper and seemed to feel it was an interesting contribution overall.  On the other hand, multiple reviewers were concerned about the experiments; namely, that they were limited to small-scale and/or synthetic data.  Other criticisms were mostly minor and were addressed in the rebuttal, but overall the biggest concern seemed to be with respect to empirical evidence.  The authors responded that the present paper is an "initial step" and that further investigation would be required.

In general, this paper could go either way.  It would have been good to see some additional experiments performed during the rebuttal phase.  But, I do like the overall idea of the paper and feel it would be worth having at the conference.  So I am going to recommend acceptance for this paper.

In the final version, please do try to address the comments of the reviewers as much as possible.
Decision: Accept (poster)