Title: Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: deep neural network

spectral bias

multi-grade deep learning
Abstract: Deep neural networks (DNNs) have showcased their remarkable precision in approximating smooth functions. However, they suffer from the {\it spectral bias}, wherein DNNs typically exhibit a tendency to prioritize the learning of lower-frequency components of a function, struggling to effectively capture its high-frequency features. This paper is to address this issue. Notice that a function having only low frequency components may be well-represented by a shallow neural network (SNN), a network having only a few layers. By observing that composition of low frequency functions can effectively approximate a high-frequency function, we propose to learn a function containing high-frequency components by composing several SNNs, each of which learns certain low-frequency information from the given data. We implement the proposed idea by exploiting the multi-grade deep learning (MGDL) model, a recently introduced model that trains a DNN incrementally, grade by grade, a current grade learning from the residue of the previous grade only an SNN (with trainable parameters) composed with the SNNs (with fixed parameters) trained in the preceding grades as features. We apply MGDL to synthetic, manifold, colored images, and MNIST datasets, all characterized by presence of high-frequency features. Our study reveals that MGDL excels at representing functions containing high-frequency information. Specifically, the neural networks learned in each grade adeptly capture some low-frequency information, allowing their compositions with SNNs learned in the previous grades effectively representing the high-frequency features. Our experimental results underscore the efficacy of MGDL in addressing the spectral bias inherent in DNNs. By leveraging MGDL, we offer insights into overcoming spectral bias limitation of DNNs, thereby enhancing the performance and applicability of deep learning models in tasks requiring the representation of high-frequency information. This study confirms that the proposed method offers a promising solution to address the spectral bias of DNNs. The code is available on GitHub: \href{https://github.com/Ronglong-Fang/AddressingSpectralBiasviaMGDL}{\texttt{Addressing Spectral Bias via MGDL}}.

Addressing Spectral Bias of Deep Neural Networks by Multi-Grade Deep Learning

Abstract

Deep neural networks (DNNs) have showcased their remarkable precision in approximating smooth functions. However, they suffer from the spectral bias , wherein DNNs typically exhibit a tendency to prioritize the learning of lower-frequency components of a function, struggling to effectively capture its high-frequency features. This paper is to address this issue. Notice that a function having only low frequency components may be well-represented by a shallow neural network (SNN), a network having only a few layers. By observing that composition of low frequency functions can effectively approximate a high-frequency function, we propose to learn a function containing high-frequency components by composing several SNNs, each of which learns certain low-frequency information from the given data. We implement the proposed idea by exploiting the multi-grade deep learning (MGDL) model, a recently introduced model that trains a DNN incrementally, grade by grade, a current grade learning from the residue of the previous grade only an SNN (with trainable parameters) composed with the SNNs (with fixed parameters) trained in the preceding grades as features. We apply MGDL to synthetic, manifold, colored images, and MNIST datasets, all characterized by presence of high-frequency features. Our study reveals that MGDL excels at representing functions containing high-frequency information. Specifically, the neural networks learned in each grade adeptly capture some low-frequency information, allowing their compositions with SNNs learned in the previous grades effectively representing the high-frequency features. Our experimental results underscore the efficacy of MGDL in addressing the spectral bias inherent in DNNs. By leveraging MGDL, we offer insights into overcoming spectral bias limitation of DNNs, thereby enhancing the performance and applicability of deep learning models in tasks requiring the representation of high-frequency information. This study confirms that the proposed method offers a promising solution to address the spectral bias of DNNs. The code is available on GitHub: Addressing Spectral Bias via MGDL .

1 Introduction

Deep neural networks (DNNs) have achieved tremendous success in various applications, including computer vision [15], natural language processing [38], speech recognition [26], and finance [27]. From a mathematical perspective, the success is mainly due to their high expressiveness, as evidenced by theoretical demonstrations showing their capability to approximate smooth functions with arbitrary precision [3, 10, 14, 17, 48]. Various mathematical aspects of DNNs as an approximation tool were recently investigated in [11, 16, 22, 25, 35, 45-47, 49]. However, it was noted in [30, 42] that the standard deep learning model, which is called single grade deep learning (SGDL) in this paper,

trains a DNN end-to-end leading to learning bias towards low-frequency functions. While this bias may explain the phenomenon where DNNs with a large number of parameters can achieve low generalization error [7, 40, 41], DNNs trained by SGDL struggle to capture high-frequency components within a function even though they can well-represent its low-frequency components. This bias may potentially limit the applicability of DNNs to problems involving high-frequency features, such as image reconstruction [12, 24, 37], seismic wavefield modeling [39], high-frequency wave equations in homogenization periodic media [9], and high energy physics [28]. Especially, in medical image reconstruction such as PET/SPECT, high-frequency components play a crucial role in determining image resolution, which is critical in clinical practice, as higher resolution leads to earlier and more accurate disease diagnosis.

There have been some efforts to address this issue. A phrase shift DNN was proposed in [5], where the original dataset was first decomposed into subsets with specific frequency components, the highfrequency component was shifted downward to a low-frequency spectrum for learning, and finally, the learned function was converted back to the original high frequency. An adaptive activation function was proposed in [18] to replace the traditional activation by scaling it with a trainable parameter. A multiscale DNN was introduced in [6, 23], in which the input variable was first scaled with different scales and then the multiscale variables were combined to learn a DNN. It was proposed in [36] first to map the input variable to Fourier features with different frequencies and then to train the mapped data by DNNs. All these approaches can mitigate the spectral bias issue of DNNs to some extent.

Despite of encouraging progresses made in mitigating the spectral bias of DNNs, practical learning with DNNs remains a persistent challenge due to the bias, especially for learning from higherdimensional data. This issue deserves further investigation. We propose to address this issue by understanding how a high-frequency function can be more accurately represented by neural networks. On one hand, it has been observed [7, 30, 31] that a function having only low frequency components can be well represented by a shallow neural network (SNN), a network having only a few layers. On the other hand, the classical Jacobi-Anger identity expresses a complex exponential of a trigonometric function as a linear combination of its harmonics that can contain significant high-frequency components. Even though the complex exponential function and the trigonometric function both are of low frequency, their composition could contain high frequency components. This motivates us to decompose a function containing high-frequencies as a sum-composition form of low-frequency functions. That is, we decompose it into a sum of different frequency components, each of which is further broken down to a composition of low-frequency functions . In implementing this idea, we find that the multi-grade deep learning (MGDL) model recently introduced in [43, 44] matches seamlessly for constructing the sum-composition form for a function of high-frequency. It is the purpose of this study to introduce the general methodology in addressing the spectral bias issue of DNNs and implement it by employing MGDL as a technical tool. We demonstrate the efficacy of the proposed approach in four experiments with one-dimensional synthetic data, twodimensional manifold data, two-dimensional colored images, and very high-dimensional modified National Institute of Standards and Technology (MNIST) data. Our numerical results endorse that the proposed approach can effectively address the spectral bias issue, leading to substantial improvement in approximation accuracy in comparison with the traditional SGDL training approach.

Contributions of this paper include: (a) We propose a novel approach to address the spectral bias issue by decomposing a function containing high-frequencies as a sum of different frequency components, which are represented as compositions of low-frequency functions. (b) We investigate the efficacy of MGDL in decomposing a function of high-frequency into its 'sum-composition' form of SNNs. (c) We successfully apply the proposed approach to synthetic data in 1 and 2 dimensions and real data in 2 and 784 dimensions, showing that it can effectively address the spectral bias issue.

2 Proposed Approach and Multi-Grade Learning Model

We introduce a novel approach to tackle the spectral bias issue and review the MGDL model.

We begin with a quick review of the definition of DNNs. A DNN is a successive composition of an activation function composed with a linear transformation. Let R denote the set of all real numbers, and d, s be two positive integers. A DNN with depth D consists of an input layer, D -1 hidden layers, and an output layer. Let N D := { 1 , 2 , . . . , D } . For j ∈ { 0 }∪ N D , let d j denote the number of neurons in the j -th hidden layer with d 0 := d and d D := s . We use W j ∈ R d j × d j -1 and b j ∈ R d j

0.10

0.10

f

0.08

0.08

2

itude

0.06

Amplitude

e

0.06

3

Am

0.04

0.04

0.02

0.02

0.00

0.00

0

25

50

75

100 125 150 175 200

0

25

50

75

1001

125 150 175 200

Frequency[Hz]

Frequency[Hz]

to represent the weight matrix and bias vector, respectively, for the j -th layer. By σ : R → R we denote an activation function. When σ is applied to a vector, it means that σ is applied to the vector componentwise. For an input vector x := [ x 1 , x 2 , . . . , x d ] ⊤ ∈ R d , the output of the first layer is defined by H 1 ( x ) := σ ( W 1 x + b 1 ) . For a DNN with depth D ≥ 3 , the output of the ( j +1) -th hidden layer can be identified as a recursive function of the output of the j -th hidden layer, defined as H j +1 ( x ) := σ ( W j +1 H j ( x ) + b j +1 ) , for j ∈ N D -2 . Finally, the output of the DNN with depth D is an s -dimensional vector-valued function defined by

Suppose that data samples D := { x ℓ , y ℓ } N ℓ =1 are chosen. The loss on D is defined as

The traditional SGDL model is to minimize the loss function L defined by (2) with respect to Θ := { W j , b j } D j =1 , which yields the optimal parameters Θ ∗ := { W ∗ j , b ∗ j } D j =1 and the corresponding DNN N D (Θ ∗ ; · ) . When D is relatively small, for example, D < 5 , we call N D an SNN. It is well-recognized that training an SNN is notably easier than training a DNN.

We motivate the proposed idea by a simple example. We consider the function f ( x ) , x ∈ [0 , 1] , whose Fourier transform is shown in Figure 1 (Left), where the Fourier transform is defined by ˆ f ( t ) := ∫ ∞ -∞ f ( x ) e -i 2 π tx d x . To compute the Fourier transform of f defined on [0 , 1] , we extend f to the entire real line by assigning its value to be zero for x / ∈ [0 , 1] . Observing from Figure 1 (Left), the function f has significant high-frequency components, with frequencies varying from 0 to 200 . The function f can be represented as

where ◦ denotes the composition of two functions. Note that the Fourier transforms ˆ f j , j = 1 , 2 , 3 , 4 , are displayed in Figure 1 (Right). Clearly, the functions f j , j = 1 , 2 , 3 , 4 , are of low-frequency, with frequencies mainly concentrating on the interval [0 , 50] . This example surely demonstrates that a function of high-frequency can be expressed as a sum of compositions of lower-frequency functions. This observation leads to the proposed approach of addressing the spectral bias of DNNs to be studied in this paper. The legitimacy of the proposed idea may be reinforced by the Jacobi-Anger identity [2], which expresses a complex exponential of a trigonometric function as a linear combination of its harmonics. Even though both the complex exponential function and the trigonometric function are of low-frequency, their composition contains many high-frequency components. We now review the Jacobi-Anger identity, the identity named after the 19th-century mathematicians Carl Jacobi and Carl Theodor Anger. It has the form

where i denotes the imaginary unit and J n ( a ) denotes the n -th Bessel function of the first kind, see details in [2]. Taking the real part of the both sides of the Jacobi-Anger identity (4), we obtain that

The left-hand side of (5) is a composition of two low-frequency functions cos( a x ) and sin( b x ) , having frequencies a/ (2 π ) and b/ (2 π ) , respectively, while the right-hand side is a linear combination of cos( nb x ) with n taking all integers. The high-frequency of the composition can be estimated by a rule of thumb. Specifically, the left-hand side of (5) is a frequency-modulated sinusoidal signal

[32, 33], with its frequencies spreading on an interval centered at zero. It follows from the well-known Carson bandwidth rule [8, 29, 32], regarded as a rule of thumb, that more than 98% frequencies are located within the interval [ -( ab + b ) / (2 π ) , ( ab + b ) / (2 π )] . Therefore, the highest frequency of cos( a sin( b x )) can be well-estimated by ab + b 2 π , which is greater than the frequencies of cos( a x ) and sin( b x ) when a > 0 and b > 1 . These suggest that a composition of two low-frequency functions may lead to a high-frequency function.

The example presented earlier, together with the Jacobi-Anger identity, inspires us to decompose a given function into a sum of different frequency components, each of which is a composition of lowerfrequency functions, a decomposition similar to equation (3) for the function f represented in Figure 1 (Left). In other words, for a function g of high-frequency, we decompose it in a 'sum-composition' form as

where ⊙ k j =1 g j := g k ◦ · · · ◦ g 2 ◦ g 1 , and g j , j ∈ N k , are all of low-frequency. The function f represented in (3) is a special example of (6). In the context of approximation by neural networks, we prefer expressing g j by SNNs, as a function having only low-frequency components can be wellrepresented by an SNN. The MGDL model originated in [43, 44] furnishes exactly the decomposition (6), with each g j being an SNN. We propose to employ MGDL to learn the decomposition (6), where the low-frequency function g j is represented by an SNN.

It is worth explaining the motivation behind the MGDL model. MGDL was inspired by the human education system which is arranged in grades. In such a system, students learn a complex subject in grades, by decomposing it into sequential, simpler topics. Foundational knowledge learned in previous grades remains relatively stable and serves as a basis for learning in a present and future grades. This learning process can be modeled mathematically by representing a function that contains higher-frequency components by a 'sum-composition' form of low-frequency functions. MGDL draws upon this concept by decomposing the learning process into multiple grades, where each grade captures different levels of complexity.

We now review the MGDL model that learns given data D := { x ℓ , y ℓ } N ℓ =1 . Following [43], we split a DNN with depth D into L grades, with L < D , each of which learns an SNN N D l , defined as (1), with depth D l , from the residue { e l ℓ } N ℓ =1 of the previous grade, where 1 < D l < D and ∑ L l =1 D l = D + L -1 . Let Θ l := { W l j , b l j } D l j =1 denote the parameters to be learned in grade l . We define recursively g 1 (Θ 1 ; x ) := N D 1 (Θ 1 ; x ) , g l +1 (Θ l +1 ; x ) := N D l +1 (Θ l +1 ; · ) ◦ H D l -1 (Θ ∗ l ; · ) ◦ . . . ◦ H D 1 -1 (Θ ∗ 1 ; · )( x ) , for l ∈ N L -1 , and the loss function of grade l by

where Θ ∗ l := { W l ∗ j , b l ∗ j } D l j =1 are the optimal parameters learned by minimizing the loss function L l with respect to Θ l . The residues are defined by e 1 ℓ := y ℓ and e l +1 ℓ := e l ℓ -g l (Θ ∗ l ; x ℓ ) , for l ∈ N L -1 , ℓ ∈ N N . When minimizing the loss function L l (Θ l ; D ) of grade l , parameters Θ ∗ j , j ∈ N l -1 , learned from the previous l -1 grades are all fixed and H D l -1 -1 (Θ ∗ l -1 ; · ) ◦ . . . ◦ H D 1 -1 (Θ ∗ 1 ; · ) serves as a feature or 'basis". After L grades are learned, the function ¯ g L learned from MGDL is the summation of the function learned in each grade, that is,

where g l (Θ ∗ l ; x ) := N D l (Θ ∗ l ; · ) ◦ H D l -1 -1 (Θ ∗ l -1 ; · ) ◦ . . . ◦ H D 1 -1 (Θ ∗ 1 ; · )( x ) , and H D k -1 -1 for 1 ≤ k ≤ L and N D L are SNNs learned in different grades. Thus, MGDL enables us to construct the desired 'sum-composition" form (6). When L = 1 , MGDL reduces to the traditional SGDL model.

In MGDL, we use the mean squared error (MSE) loss function. It was established in [43] that when the loss function is defined by MSE, MGDL either learns the zero function or results in a strictly decreasing residual error sequence (see, Theorem 1 in Appendix A). Since the regression problems conducted in this paper naturally align with MSE losses, it is a suitable choice. In practice, MGDL can also be applied with other loss functions, such as cross-entropy loss, when solving classification problems. In MGDL, the computation cost remains relatively consistent across all grades. For x l ℓ := H D l -1 -1 (Θ ∗ l -1 ; · ) ◦ H D l -2 -1 (Θ ∗ l -2 ; · ) ◦ . . . ◦ H D 1 -1 (Θ ∗ 1 ; · )( x ℓ ) , we recursively

let x 1 ℓ := x ℓ , x k ℓ := H D k -1 -1 (Θ ∗ k -1 ; · ) ◦ x k -1 ℓ , k = 2 , 3 , . . . , n. When training grade l , we use the output of grade l -1 , denoted as x l ℓ along the residual e l ℓ , which are already obtained. The training dataset in grade l consists of { ( x l ℓ , e l ℓ ) } N ℓ =1 . This dataset is used to train a new shallow network, which is independent of the previous l -1 grades. Moreover, x l ℓ can be computed recursively, ensuring that the computation cost for each grade remains relatively consistent.

MGDL avoids training a DNN from end to end. Instead, it trains several SNNs sequentially, with the current grade making use the SNNs learned from the previous grades as a feature and composing it with a new SNN to learn the residue of the previous grade. This allows MGDL to decompose a function that contains higher-frequency in a form of (6), with g j being a SNN learned from grade j . In this way, higher-frequency components in the data can be effectively learned in a grade-by-grade manner. Note that the training time of MGDL increases linearly with the number of grades. This makes MGDL an effective and scalable solution for tackling complex tasks. MGDL is an adaptive approach by nature. When the outcome of the present grade is not satisfactory, we can always add a new grade without changing the previous grades.

It is worth noting that while ResNet [15] also has a sum-composition form, MGDL differs from it significantly. The 'Composition' for ResNet refers to composition of layers, while that for MGDL emphasizes the composition of the SNNs sequentially learned in the previous grades. Moreover, ResNet learns all parameters of the entire sum of DNNs at once, training it from end to end, whereas MGDL learns the sum incrementally, grade by grade, in each grade training an SNN composed with the feature (the composition of the SNNs learned in the previous grades). MGDL also differs from the relay backpropagation approach proposed in [34], where a DNN is divided into multiple segments, each with its own loss function. The gradients from these losses are then propagated to lower layers of their respective segments and all segments are optimized all together by minimizing the sum of the losses. While MGDL trains SNNs in a multi-grade manner, each of which learns from the residue of the previous grade, freezing the previously learned SNNs (serving as features or 'bases').

MGDL is a principle applicable to various models, including standard DNNs, convolutional neural networks, and ResNet. In this paper, we demonstrate its feasibility by applying it to standard DNNs.

3 Numerical Experiments

In this section, we study MGDL empirically in addressing the spectral bias issue of SGDL. We consider four examples: Subsections 3.1, 3.2, and 3.4 investigate regression on synthetic, manifold, and MNIST data, respectively, for which the spectral bias phenomena of SGDL are identified in [30]. Section 3.3 deals with regression on colored images, which were studied in [36] by using the Fourier features to mitigate the spectral bias. Our goal is to compare the learning performance of MGDL with that of SGDL on these datasets and understand to what extent MGDL can overcome the spectral bias exhibited in SGDL.

The loss functions defined in (2) for SGDL and (7) for MGDL are used to compute the training and validation loss when D is chosen to be the training and validation data, respectively. We use the relative squared error (RSE) to measure the accuracy of predictions obtained from both SGDL and MGDL. Assume that N is a trained neural network. For a prediction value ˆ y ℓ := N ( x ℓ ) at x ℓ , we define RSE := ∑ N ℓ =1 ∥ ˆ y ℓ -y ℓ ∥ 2 2 / ∑ N ℓ =1 ∥ y ℓ ∥ 2 2 . When D represents the training, validation, and testing data, RSE is specialized as TrRSE, VaRSE, and TeRSE, respectively.

Details of the numerical experiments conducted in this section, including computational resources, the network structure of SGDL and MGDL for each example, the choice of activation function, the optimizer, parameters used in the optimization process, and supporting figures are provided in Appendix B.

3.1 Regression on the synthetic data. In this experiment, we compare the efficacy of SGDL and MGDL in learning functions of four different types of high-frequencies.

The experiment setup is as follows. Given frequencies κ := ( κ 1 , κ 2 , . . . , κ M ) with corresponding amplitudes α := ( α 1 , α 2 , . . . , α M ) , and phases φ := ( φ 1 , φ 2 , . . . , φ M ) , we consider approximating the function λ : [0 , 1] → R defined by

by neural networks learned with SGDL and MGDL. We consider four settings, in all of which we choose M := 20 , κ j := 10 j and φ j ∼ U (0 , 2 π ) for j ∈ N 20 , where U denotes the uniform

Amplitude

0.5

0.4

G

1

Amplitude

0.5

0.4

G 1

G 2

G 3

Amplitude

0.35

0.30

0.25

G 2

G 3

G 1

Amplitude

0.25

0.20

G 1

G 2

G 3

0.3

G

2

0.3

G 4

0.20

G 4

0.15

G 4

0.2

G4

G

3

0.2

0.15

0.10

0.10

G 5

0.1

0.1

0.05

0.05

0.0

0.0

0.00

0.00

0

25

50

75100125150175200

0

25

5075100125150175200

0

25

50 75 100125 150175200

0

25

50 75 100 125 150175 200

Frequency[Hz]

Frequency[Hz]

Frequency[Hz]

Frequency[Hz]

distribution and the random seed is set to be 0 . In settings 1 , 2 , 3 , and 4 , we choose respectively the amplitudes α j := 1 , α j := 1 . 05 -0 . 05 j , α j ( x ) := e -x cos( j x ) , and α j := 0 . 05 j , for j ∈ N 20 . Note that in setting 3 , we explore the case where the amplitude varies as a function of x for each component. The amplitudes versus one-side frequencies for λ across the four settings are depicted in Figure 7 in Appendix B.1. For all the four settings, the training data consist of pairs { x ℓ , λ ( x ℓ ) } ℓ ∈ N 6000 , where x ℓ 's are equally spaced between 0 and 1 . The validation and testing data consist of pairs { x ℓ , λ ( x ℓ ) } ℓ ∈ N 2000 , where x ℓ 's are generated from a random uniform distribution on [0 , 1] , with the random seed set to be 0 and 1, respectively.

Numerical results for this example are reported in Figures 2 and 3, as well as 8 and 9 (in Appendix B.1), and Table 1. Figure 2 displays the amplitude versus the one-side frequency of the functions learned across four grades of MGDL in settings 1-3, and five grades in setting 4. In all the four settings, MGDL exhibits a pattern of learning low-frequency components in grade 1, middle-frequency components in grade 2, and high-frequency components in grades 3, 4 (and 5 for setting 4). We let N ∗ j := N D j (Θ ∗ j ; · ) and H ∗ j := H D j -1 (Θ ∗ j ; · ) . The SNN N ∗ 1 learned in grade 1 represents a low-frequency component of λ , the SNNs N ∗ 2 and N ∗ 3 learned in grades 2 and 3, composed with H ∗ 1 and H ∗ 2 ◦ H ∗ 1 , respectively, represents higher-frequency components of λ . Likewise, the SNN N ∗ 4 , learned in grade 4, composed with H ∗ 3 ◦ H ∗ 2 ◦ H ∗ 1 represents the highest-frequency component of λ . This fact is particularly evident in setting 4 (see, the fourth subfigure in Figure 2), where the amplitude within the data increases with the frequency. For settings 1, 2, and 3, grade 4 does not learn much. This is because for the functions of these three settings, the amplitudes of higher-frequencies are proportionally small. However, grade 4 is still important for these settings. As shown in Figure 3 (right), grade 4 reduces the loss from 10 -2 to 10 -5 for setting 1 and from 10 -4 to 10 -6 for settings 2 and 3. This indicates that if we want a high precision, we need to include grade 4. For setting 4, we need grade 5 to learn its highest frequency component. These findings suggest that MGDL is particularly well-suited for learning the high-frequency components of the function.

100

101

100

101

0.30

0.25

G 1

G 2

0.35

0.30

G 2

G 1

Loss

10-1

10-2

103

10-4

training, 2

training, 3

training. 1

validation, 1

validation, 2

S

10-1

10-2

103

10-4

training. 1

validation, 1

training. 2

validation, 2

training, 3

litude

Am

0.10

0.20

0.15

G 3

G 4

Am

itu

P

0.25

0.20

0.15

0.10

G 3

G 4

10-5

training, 4

validation, 3

10-5

validation, 3

training. 4

0.05

0.05

validation, 4

validation, 4

0.00

0.00

10-6

0

10000

20000

30000

10-6

0

0000090000090000090000000000009

0

50100150200250300350400

0

50

100150200250300350400

Epochs (SGDL)

Epochs (MGDL)

Frequency[Hz]

Frequency[Hz]

Figure 3 compares the progress of the training and validation losses against the number of training epochs for SGDL and MGDL across the four settings. We observe that when learning a task involving high-frequency components by SGDL, the training and validation losses decrease slowly due to the spectral bias of DNN. While the same task is learned by MGDL, the learning process progresses through distinct grades. In grade 1, MGDL primarily learns low-frequency, resulting in a slow decrease in loss. In grade 2, the training loss and validation loss both decrease more rapidly due to the use of the composition of SNN N ∗ 2 with the feature H ∗ 1 , facilitating in learning high-frequency features. This accelerated learning aspect of MGDL is further evidenced in grades 3 and 4 (as well as grade 5 for setting 4). Table 1 compares the accuracy achieved by SGDL and MGDL. Within a comparable or even less training time, MGDL increases accuracy, measured by TeRSE from 10 -1 to 10 -5 , 10 -3 to 10 -6 , 10 -1 to 10 -5 , and 10 -1 to 10 -3 in settings 1, 2, 3 and 4, respectively. Across the four settings, TeRSE values are reduced by a factor of 592 ∼ 7 , 058 . These comparisons highlight MGDL's advantage in effectively learning high-frequency oscillatory functions.

Figure 8 in Appendix B.1 depicts the functions, in the Fourier domain, learned by SGDL (row 1) and MGDL (row 2) across the four settings, demonstrating that MGDL has a substantial reduction in the 'spectral bias' exhibited in SGDL. This is because high-frequency components are learned in a higher grade, where they are represented as the composition of a low-frequency component with the low-frequency components learned in the previous grades, and each grade focuses solely on learning a low-frequency component by an SNN. We also include Figure 9 in Appendix B.1 to compare the spectrum evolution between SGDL (1st row) and MGDL (2nd row) across settings 1-4. Notably, although in iterations SGDL and MGDL both learn low-frequency components first and then followed by middle and high-frequency components, MGDL learns in grade by grade, exhibiting significant outperformance.

3.2 Regression on the manifold data. The second experiment compares regression by SGDL and MGDL on two-dimensional manifold data, studied in [30] but with twice higher frequencies.

The goal of this experiment is to explore scenarios where data lies on a lower-dimensional manifold embedded within a higher-dimensional space. Such data is commonly referred to as manifold data [4]. Let γ be an injective mapping from [0 , 1] m → R d with m ≤ d and M := γ ([0 , 1] m ) denote the manifold data. A target function τ : M→ R defined on the manifold can be identified with function λ := τ ◦ γ defined on [0 , 1] m . Regressing the target function τ is therefore equivalent to finding f : R d → R such that f ◦ γ matches λ . Following [30], we set m := 1 , d := 2 , and choose the mapping γ as

for a nonnegative integer q . Clearly, γ q : [0 , 1] → R 2 , and M := γ q ([0 , 1]) defines the manifold corresponding to a flower-shaped curve with q petals when q > 0 , and a unit circle when q = 0 . Suppose that λ : [0 , 1] → R is the function defined by (9). Our task is to learn a DNN f : R 2 → R such that f ◦ γ q matches λ . We consider two settings for λ . In settings 1 and 2, we choose α j := 0 . 025 j and α j ( x ) := e -x cos( j x ) for j ∈ N 40 , respectively. For both the settings, we choose κ j := 10 j and φ j ∼ U (0 , 2 π ) for j ∈ N 40 with the random seed set to be 0 , and consider the cases where q := 4 and q := 0 . Note that the smaller q is, the more difficult the learning task is. The training data consists of pairs { γ q ( x ℓ ) , λ ( x ℓ ) } ℓ ∈ N 12000 , where x ℓ 's are equally spaced between 0 and 1 . The validation and testing data consist of pairs { γ q ( x ℓ ) , λ ( x ℓ ) } ℓ ∈ N 4000 , where x ℓ 's are generated from a random uniform distribution on [0 , 1] , with random seed set to be 0 and 1, respectively.

Numerical results for this example are reported in Figures 4-5, and 10 (in Appendix B.2), and Table 2. Figure 4 illustrates the frequency of functions learned across four grades of MGDL for settings 1 and 2, where q := 0 . In both of the settings, MGDL exhibits a pattern of learning low-frequency components in grade 1 , middle-frequency components in grade 2 , and high-frequency components in grades 3 and 4 . Therefore, the high-frequency components within the function mainly learned in higher grades, in which the learned function is a composition of the SNNs learned from several grades. That is, MGDL decomposes a high-frequency component as the composition of several lower-frequency components, facilitating effectively learning high frequency features within the data.

Table 2 compares the approximation accuracy achieved by SGDL and MGDL for settings 1 and 2. For SGDL, reducing the value of q makes the learning task for both settings more challenging, due to the spectral bias of DNNs. When q := 4 , 0 in setting 1 and q := 0 in setting 2, learning becomes especially challenging for SGDL. In such cases, MGDL significantly outperforms SGDL by achieving higher accuracy in approximately half to one-third of the training time for both settings. Figure 5 displays the training and validation loss for SGDL and MGDL. Figure 10 illustrates the

101

r0t

101

101

100

p0t

100

100

101

10-1

SS07

10-1

SS07

1-0t

102

102

20-2

training, 4

102

training, 4

103

training.4

validation, 4

Mula

103

training, 4

validation,4

10-3

training. 0

validation, 0

validation,4

10 -3

training, 0

validation, 0

validation, 4

10-4

training, 0

validation, 0

104

validation,0

training,0

20~4

00000

Epochs(SGDL)

20000

30000

10~4

0

G 1

0000E

Epochs(MGDL)

G 2

30000

30000

G 4

30000

105

0

10000

Epochs(SGDL)

20000

30000

10-5

0

G 1 30000 G 2 30000 G 3 30000 G 4 30000

Epochs(MGDL)

spectrum evolution throughout the learning process for settings 1 and 2. We observe that MGDL outperforms SGDL in accuracy during the learning process. Table 2 and Figure 10 consistently demonstrate that MGDL exhibits a substantial advancement in addressing the spectral bias.

3.3 Regression on two-dimensional colored Images. In the third experiment, we compare performance of MGDL and SGDL for regression of two-dimensional color images.

We test the models with the 'cat' image from website Cat, and the 'sea' and 'building' images from the Div2K dataset [1]. The input to the models is the pixel coordinates, and the output is the corresponding RGB values. The training dataset consists of a regularly spaced grid containing 1 / 4 of the image pixels, while the test dataset contains the full image pixels. We use peak signal-tonoise ratio (PSNR) defined as in (12) to evaluate the accuracy of images obtained from MGDL and SGDL. PSNR values computed over the train and test dataset are denoted by TrPSNR and TePSNR, respectively.

Numerical results for this experiment are presented in Table 3, Figure 6, and Figures 11-13 (in Appendix B.3). Table 3 compares the PSNR values of images obtained from MGDL and SGDL. For MGDL, it is evident that adding more grades consistently improves the image quality, as both training and testing PSNR values increase substantially with the addition of each grade across all images Cat, Sea, and Building. MGDL outperforms SGDL by 2 . 35 ∼ 3 . 93 dB for the testing PSNR values. Specifically, for images Cat, Sea and Building, MGDL surpasses SGDL by 2.35, 3.93 and 2.88 dB, respectively. This demonstrates the superiority of MGDL in representing images compared to SGDL. Figure 6 illustrates the training and testing PSNR values during the training process for the three images. It is evident that MGDL facilitates a smoother learning process as more grades are added, in comparison to SGDL. This observation aligns with the results presented in Table 3.

Predictions of each grade of MGDL and of SGDL are illustrated in Figures 11, 12, and 13 for images Cat, Sea, and Building, respectively. In all cases, grade 1 captures only the rough outlines of the images, representing the lower-frequency components. As we progress to grades 2, 3, and 4, more details corresponding to the higher-frequency components are added. This demonstrates the effectiveness of MGDL in progressively learning high-frequency features within the images. Moreover, the image quality achieved with MGDL is notably superior to that with SGDL.

3.4 Regression on MNIST data. In our fourth experiment, we apply MGDL to regress on highdimension data, the MNIST data. We compare the training and validation loss, and TeRSE for SGDL and MGDL, when learning high-frequency features from the data.

We set up this experiment following [30], with a focus on comparing performance of SGDL and MGDL in learning high-frequency features. We choose the handwriting digits from MNIST dataset [21], composed of 60,000 training samples and 10,000 testing samples of the digits '0' through

(a) Cat: SGDL

(b) Sea: SGDL

(c) Building: SGDL

(d) Cat: MGDL

(e) Sea: MGDL

(f) Building: MGDL

SGDL

SGDL

SGDL

28

24

28

Training PSNR

26

Training PSNR

Training PSNR

26

Testing PSNR

24

Testing PSNR

22

Testing PSNR

PSNR

24

PSNR

22

PSNR

20

22

20

20

M

18

18

18

16

16

14

16

0

2000

4000

6000

800010000

12

0

2000

4000

6000

800010000

14

0

2000

4000

6000

800010000

Epochs

Epochs

Epochs

MGDL

MGDL

MGDL

28

24

28

26

Training PSNR

Testing PSNR

26

24

Training PSNR

Testing PSNR

22

Testing PSNR

Training PSNR

PSNR

24

22

PSNR

22

20

PSNR

20

20-

18

18

18

16

16

16

14

0

10000

20000

30000

40000

12

0

10000

20000

30000

40000

14

0

10000

20000

30000

40000

Epochs

Epochs

Epochs

'9'. We represent each digit j = 0 , 1 , . . . , 9 by a one-hot vector e j +1 ∈ R 10 , whose ( j + 1) -th component is one and all others are zero, and denote by τ 0 : R 784 → R 10 the classifier, which is a piecewise constant function defined by τ 0 ( x ) := e j +1 if x represents the digit j . Wesplit the available training samples to form the training and validation data, denoted as D 0 := { x ℓ , τ 0 ( x ℓ ) } ℓ ∈ N n train and D ′ 0 := { x ′ ℓ , τ 0 ( x ′ ℓ ) } ℓ ∈ N n val respectively, with n train := 45 , 000 and n val := 15 , 000 , and use the testing samples as the testing data, denoted as D ′′ 0 := { x ′′ ℓ , τ 0 ( x ′′ ℓ ) } ℓ ∈ N ntest with n test := 10 , 000 . Clearly, D 0 , D 0 ′ and D ′′ are subsets of { [0 , 1] 784 , { e j +1 } 9 j =0 } . Letting ψ κ ( x ) := sin(2 πκ ∥ x ∥ 2 ) , corresponding to a radial wave defined on the input space R 784 , we define the target function by τ β,κ ( x ) := τ 0 ( x ) (1 + βψ κ ( x )) , where κ is the frequency of the wave and β is the amplitude. Note that τ 0 and βψ κ contribute respectively the lower-frequency and high-frequency components (regarded as noise) of the target function τ β,κ , as discussed in [30]. The modified training and validation data denoted by D β,κ := { x ℓ , τ β,κ ( x ℓ ) } ℓ ∈ N n train and D ′ β,κ := { x ′ ℓ , τ β,κ ( x ′ ℓ ) } ℓ ∈ N n val , respectively, are used to train DNNs. Our goal is to use SGDL and MDGL to regress the modified data D β,κ through minimizing their respective training loss, to compare their robustness to noise. The training loss is evaluated on D β,κ and validation loss is on D ′ β,κ . TrRSE, VaRSE, and TeRSE are evaluated on D β,κ D ′ β,κ , and D ′′ 0 , respectively, noting that D ′′ 0 are test data without noise.

We choose the amplitude β from { 0 . 5 , 1 , 3 , 5 } . For each β , we vary the frequency κ from { 1 , 3 , 5 , 7 , 10 , 50 } . These choices of β and κ result in functions τ β,κ that have higher frequencies than those studied in [30]. Figures 14 and 15 (in Appendix B.4) compare the training and validation loss versus training time of SGDL (structure (13)) and MGDL (structures (14) and (15) respectively) for different values of frequency κ and amplitude β (depicted in the figure). We observe that for small β , for example, β = 0 . 5 , the results of the two models are comparable. When β = 1 , the training loss for SGDL keeps decreasing as training time increases, while the validation loss initially decreases and starts to increase after training of 1,600 seconds, and unlike SGDL, both the training loss and validation loss for MGDL keep decreasing. This trend indicates that over-fitting phenomenon occurs for SGDL and suggests that MGDL has a superior generalization capability. Further increasing the value of β , for example, β = 3 and β = 5 , the over-fitting phenomenon occurs in both of the models. An increase in β corresponds to a higher proportion of high-frequency components within τ β,κ . This increased proportion of the high-frequency component significantly impacts the validation loss of SGDL, a trend also observed in MGDL. However, the effect is comparatively less with MGDL than with SGDL. We present in Table 4 TrRSE, VaRSE, and TeRSE of SGDL (structure (13)) and MGDL (structure (14)) for β := 1 , where the results are obtained by choosing t min := 10 -5 and t max := 10 -4 , and the batch size to be 'Full', for all cases of κ and for both SGDL and MGDL. We observe that while TrRSE of SGDL is smaller than that of MGDL, both VaRSE, and TeRSE of SGDL are larger than the corresponding values of MGDL, suggesting occurrence of overfitting with SGDL. MGDL's improvement in accuracy, measured by TeRSE, is about 27 ∼ 29% . This experiment reveals that MGDL is a promising model for learning higher-dimensional data with a higher proportion of high-frequency features.

4 Conclusion

By observing that a high-frequency function may be well-represented by composition of several lower-frequency functions, we have proposed a novel approach to learn such a function. The proposed approach decomposes the function into a sum of multiple frequency components, each of which is compositions of lower-frequency functions. By leveraging the MGDL model, we can express high-frequency components by compositions of multiple SNNs of low-frequency. We have conducted numerical studies of the proposed approach by applying it to one-dimensional synthetic data, two-dimensional manifold data, colored images, and higher-dimensional MNIST data. Our studies have concluded that MGDL can effectively learn high-frequency components within data. Moreover, the proposed approach is easy to implement and not limited by dimension of the input variable. Consequently, it offers a promising approach to learn high-frequency features within (high-dimensional) dataset.

Limitation : Mathematical understanding of the spectral bias of DNNs is absent. Theoretical foundation for MGDL to address the spectral bias issue needs to be established. Numerical studies are preliminary, limited to four examples. More extensive numerical experiments will be conducted in our future work.

Acknowledgments: Y. Xu is indebted to Professor Wei Cai of Southern Methodist University for insightful discussion of the spectral bias of DNNs. Y. Xu is supported in part by the US National Science Foundation under grant DMS-2208386, and by the US National Institutes of Health under grant R21CA263876.

References

Agustsson, E., & Timofte, R. (2017) Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , pp. 126-135.

Arfken, G. B., Weber, H. J., & Harris, F. E. (2011) Mathematical Methods for Physicists: A Comprehensive Guide. Orlando, FL: Academic press.

Arpit, D., Jastrz˛ ebski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., ... & Lacoste-Julien, S. (2017) A closer look at memorization in deep networks. In International Conference on Machine Learning , pp. 233-242.

Bengio, Y., Courville, A.C., & Vincent, P. (2012) Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence , 35 (8): 1798-1828.

Cai, W., Li, X., & Liu, L. (2020) A phase shift deep neural network for high frequency approximation and wave problems. SIAM Journal on Scientific Computing , 42 (5): A3285-A3312.

Cai, W., & Xu, Z. Q. J. (2019) Multi-scale deep neural networks for solving high dimensional pdes. arXiv preprint arXiv:1910.11710.

Cao, Y., Fang, Z., Wu, Y., Zhou, D., & Gu, Q. (2019) Towards understanding the spectral bias of deep learning. International Joint Conference on Artificial Intelligence , pp. 2205-2211.

Carson, J. R. (1922) Notes on the theory of modulation. Proceedings of the Institute of Radio Engineers 10 (1): 57-64.

Craster, R. V., Kaplunov, J., & Pichugin, A. V. (2010) High-frequency homogenization for periodic media. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences , 466 (2120): 23412362.

Cybenko, G. (1989) Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems , 2 (4): 303-314.

Daubechies, I., DeVore, R., Foucart, S., Hanin, B., & Petrova, G. (2022) Nonlinear approximation and (deep) ReLU networks. Constructive Approximation , 55 (1): 127-172.

Fang, R., Xu, Y., & Yan, M. (2024) Inexact fixed-point proximity algorithm for the ℓ 0 sparse regularization problem. Journal of Scientific Computing , 100 (2): 58.

Glorot, X., & Bengio, Y. (2010, March) Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , pp. 249-256.

Goodfellow, I., Bengio, Y., & Courville, A. (2016) Deep Learning . Cambridge, MA: MIT Press.

He, K., Zhang, X., Ren, S., & Sun, J. (2016) Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 770-778.

Huang, J., Jiao, Y., Li, Z., Liu, S., Wang, Y ., & Yang, Y . (2022) An error analysis of generative adversarial networks for learning distributions. Journal of Machine Learning Research , 23 (116): 1-43.

Hornik, K., Stinchcombe, M., & White, H. (1989) Multilayer feedforward networks are universal approximators. Neural Networks , 2 (5): 359-366.

Jagtap, A. D., Kawaguchi, K., & Karniadakis, G. E. (2020) Adaptive activation functions accelerate convergence in deep and physics-informed neural networks. Journal of Computational Physics , 404 : 109136.

Jiang, J., & Xu, Y. (2024) Deep neural network solutions for oscillatory Fredholm integral equations. Journal of Integral Equations and Applications 36 (1): 23-55.

Kingma, D.P., & Ba, J. (2015) Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR) .

LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) Gradient-based learning applied to document recognition. Proceedings of the IEEE 86 (11): 2278-2324.

Li, Y., Lu, S., Mathé, P., & Pereverzev, S. V. (2024). Two-layer networks with the ReLU k activation function: Barron spaces and derivative approximation. Numerische Mathematik , 156 (1): 319-344.

Liu, Z., Cai, W., & Xu, Z. Q. J. (2020) Multi-scale deep neural network (MscaleDNN) for solving Poisson-Boltzmann equation in complex domains. Communications in Computational Physics , 28 (5): 19702001.

Lu, Y., Shen, L., & Xu, Y. (2010) Integral equation models for image restoration: high accuracy methods and fast algorithms. Inverse Problems , 26 (4): 045006.

Mhaskar, H. N., & Poggio, T. (2016) Deep vs. shallow networks: An approximation theory perspective. Analysis and Applications , 14 (06): 829-848.

Nassif, A. B., Shahin, I., Attili, I., Azzeh, M., & Shaalan, K. (2019) Speech recognition using deep neural networks: A systematic review. IEEE Access , 7 : 19143-19165.

Ozbayoglu, A. M., Gudelek, M. U., & Sezer, O. B. (2020) Deep learning for financial applications: A survey. Applied Soft Computing , 93 : 106384.

Perkins, D. H. (2000) Introduction to High Energy Physics . Cambridge: Cambridge University Press.

Pieper, R. J. (2001) Laboratory and computer tests for Carson's FM bandwidth rule. In Proceedings of the 33rd Southeastern Symposium on System Theory (IEEE) , pp., 145-149.

Rahaman, N., Baratin, A., Arpit, D., Draxler, F., Lin, M., Hamprecht, F., Bengio, Y ., & Courville, A. (2019) On the spectral bias of neural networks. In International Conference on Machine Learning , pp. 5301-5310.

Ronen, B., Jacobs, D., Kasten, Y., & Kritchman, S. (2019) The convergence rate of neural networks for learned functions of different frequencies. Advances in Neural Information Processing Systems , 32.

Schulze, J.P., Kraus, M., Lang, U., & Ertl, T. (2003) Integrating pre-integration into the shear-warp algorithm. In IEEE VGTC / Eurographics International Symposium on Volume Graphics , pp. 109-118.

Shanmugam, K. S. (1979) Digital and Analog Communication Systems . New York: Wiley.

Shen, L., Lin, Z., & Huang, Q. (2016) Relay backpropagation for effective learning of deep convolutional neural networks. In European Conference on Computer Vision , pp. 467-482.

Shen, Z., Yang, H., & Zhang, S. (2022) Optimal approximation rate of ReLU networks in terms of width and depth. Journal de Mathématiques Pures et Appliquées , 157: 101-135.

Tancik, M., Srinivasan, P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., ... & Ng, R. (2020) Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems , pp. 7537-7547.

Tang, X., Schmidtlein, C. R., Li, S., & Xu, Y. (2021) An integral equation model for PET imaging. International Journal of Numerical Analysis and Modeling , 18 (6): 834-864.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017) Attention is all you need. Advances in Neural Information Processing Systems , pp. 5998-6008.

Wu, T., & Xu, Y. (2022) Inverting incomplete Fourier transforms by a sparse regularization model and applications in seismic wavefield modeling. Journal of Scientific Computing , 92 (2): 48.

Xu, Z. J. (2018) Understanding training and generalization in deep learning by fourier analysis. arXiv preprint arXiv:1808.04295 .

Xu, Z. Q. J., Zhang, Y., & Luo, T. (2022) Overview frequency principle/spectral bias in deep learning. arXiv preprint arXiv:2201.07395 .

Xu, Z. Q. J., Zhang, Y., & Xiao, Y. (2019) Training behavior of deep neural network in frequency domain. In International Conference on Neural Information Processing , pp. 264-274.

Xu, Y. (forthcoming) Multi-grade deep learning. Communications on Applied Mathematics and Computation . arXiv preprint arXiv:2302.00150 published on February 1, 2023.

Xu, Y. (2023) Successive affine learning for deep neural networks. ArXiv, abs/2305.07996 , July 11, 2023.

Xu, Y. & Zhang, H. (2022) Convergence of deep convolutional neural networks. Neural Networks , 153 : 553-563.

Xu, Y. & Zhang, H. (2024) Convergence of deep ReLU networks. Neurocomputing 571 : 127174.

Xu, Y., & Zhang, H. (2024) Uniform convergence of deep neural networks with Lipschitz continuous activation functions and variable widths. IEEE Transactions on Information Theory , 70 : No. 10, October 2024.

Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2021) Understanding deep learning (still) requires rethinking generalization. Communications of the ACM 64 (3): 107-115.

Zhou, D. X. (2020) Universality of deep convolutional neural networks. Applied and Computational Harmonic Analysis , 48 (2): 787-794.

A Analysis of the Multi-Grade Deep Learning Model

It was established in [43] that when the loss function is defined in terms of the mean square error, a grade of MGDL either learns the zero function or results in a strictly decreasing residual error sequence.

Theorem 1. Let D be a compact subset of R s and L 2 ( D , R t ) denote the space of t -dimensional vector-valued square integral functions on D . If f ∈ L 2 ( D , R t ) , then for all i = 1 , 2 , . . . ,

where N ∗ l is the SNN learned in grade l of MGDL, and for i = 1 , 2 , . . . , either f i +1 = 0 or

All numerical examples presented in this paper validate Theorem 1.

B Supporting material for Section 3

We provide in this appendix details for the numerical experiments in Section 3, including computational resources, network structures of SGDL and MGDL, the choice of activation function, as well as the optimizer and parameters used in the optimization process, and some supporting figures.

The experiments conducted in Sections 3.1, 3.2, and 3.4 of Section 3 were performed on X86_64 server equipped with an Intel(R) Xeon(R) Gold 6148 CPU @ 2.4GHz (40 slots) or Intel(R) Xeon(R) CPU E5-2698 v3 @ 2.30GHz (32 slots). In contrast, the experiments described in Section 3.3 of Section 3 were performed on X86_64 server equipped with AMD 7543 @ 2.8GHz (64 slots) and AVX512, 2 x Nvidia Ampere A100 GPU.

Wechoose ReLU as the activation function as in [30] for all the four experiments. For each experiment, for SGDL we test several network structures and choose the one produces the best performance. We then design MGDL, having the same total number of layers and the same number of neurons for each layer as the chosen SGDL structure, but their parameters are trained in multiple grades as described in Section 2. Details of the network structure will be described for each experiment.

The optimization problems for both SGDL and MGDL across the four experiments are solved by the Adam method [20] with 'Xavier' initialization [13]. In Sections 3.1, 3.2, and 3.4, the learning rate t k for the k -th epoch decays exponentially with each epoch [19], calculated as t k := t max e -γk , where γ := (1 /K ) ln( t max /t min ) represents the decay rate, with K being the total number of training epochs, t max and t min denoting the predefined maximum and minimum learning rates, respectively. In Section 3.3, we employed a fixed learning rate for both SGDL and MGDL, as numerical results indicate that the exponential decay learning rate performs poorly. We observe that when a network structure of SGDL is split into multiple grades of MGDL, the combined computing time required to train all grades in MGDL, each for K epochs, is comparable to the computing time needed to train the SGDL with K epochs, due to only SNNs involved in MGDL. Therefore, in all the four experiments, we train SGDL and all grades of MGDL for the same K epochs. We test SGDL and MGDL with the same set of the algorithm parameters, including t min , t max , batch size, and K , for all the experiments. Optimal parameters were selected based on the lowest VaRSE value for Sections 3.1 and 3.2, the highest PSNR value for Section 3.3, and the lowest validation loss for Section 3.4, within the range of parameters to be described for each example.

0.5

0.5

0.5

0.5

0.4

0.4

0.4

0.4

Amplitude

Amplitude

Amplitude

Amplitude

0.3

0.3

0.3

0.3

0.2

0.2

0.2

0.2

0.1

0.1

0.1

Mw

0.1

0.0

0.0

0.0

0.0

0

25

5075 100 125 150175200

0

25

5075 100125150175200

0

25

5075 100 125 150 175 200

0

25

50 75 100 125 150 175 200

Frequency[Hz]

Frequency[Hz]

Frequency[Hz]

Frequency[Hz]

B.1 Section 3.1

For settings 1, 2, and 3, the network structure for SGDL is

where [ n ] × N indicates N hidden layers, each with n neurons. The network structure for each grade of MGDL is given by

Here, [ n ] F indicates a layer having n neurons with parameters, trained in the previous grades, remaining fixed during the training of the current grade. We employ this notation across the numerical experiment section without further mentioning. For setting 4, the most challenging case, we employ a deeper structure for SGDL, with two more hidden layers in addition to (11). Correspondingly, for MGDL we add one more grade:

We now describe the search range of the parameters for all the four settings. We let I 1 := { 10 -4 , 10 -5 , 10 -6 } , I 2 := { 10 -3 , 5 × 10 -4 , 1 × 10 -4 } , I 3 := { 10 -5 , 10 -6 , 10 -7 } and I 4 := { 5 × 10 -5 , 10 -5 } . Then for both SGDL and MGDL, we test the pair ( t min , t max ) from all possible cases in the set ( I 1 × I 2 ) ∪ ( I 3 × I 4 ) . The batch size is chosen from 256 , 512 or the full gradient (denoted by 'Full') for each epoch. The total epoch number K is set to be 30,000.

The supporting figures for this experiment include:

Figure 7, presenting amplitude versus one-side frequency plots for λ across settings 1-4;

Figure 8, comparing the amplitude of one-side frequency for SGDL and MGDL across settings 1-4;

Figure 9, illustrating the evolution of spectrum comparison between SGDL and MGDL for settings 1-4.

In Figure 9, the colors in these subfigures show the measured amplitude of the network spectrum at the corresponding frequency, normalized by the amplitude of λ at the same frequency. The colorbar is clipped between 0 and 1 , indicating approximation accuracy from the worst to the best when changing from 0 to 1 .

B.2 Section 3.2

The network structure that we use for SGDL is

and the grade network structure for MGDL is

0.5

0.5

0.5

0.5

0.4

0.4

0.4

0.4

Amplitude

0.3

Amplitude

Amplitude

0.3

Amplitude

EO

0.2

0.2

0.2

0.2

0.1

0.1

0.1

0.1

0.0

0.0

0.0

0.0

0

25

50 75 100 125 150 175 200

0

25

5075 100 125150 175200

0

25 50 75 100 125150175200

0

25 50 75 100 125 150 175 200

Frequency [Hz]

Frequency [Hz]

Frequency [Hz]

Frequency [Hz]

0.5

0.5

0.5

0.5

0.4

0.4

0.4

0.4

Amplitude

Amplitude

Amplitude

Amplitude

0.3

0.3

0.3

0.2

0.2

0.2

0.2

0.1

0.1

0.1

0.1

0.0

0.0

0.0

0.0

0

25 50 75 100 125 150 175 200

0

25

5075 100 125 150 175 200

0

25

55075100125150175200

0

25

50 75 100 125 150 175 200

Frequency [Hz]

Frequency [Hz]

Frequency [Hz]

Frequency [Hz]

0000

00000

00000

0000

1.0

1.0

1.0

1.0

Epochs (SGDL)

20000

- 0.8

Epochs (SGDL)

20000

0.8

Epochs (SGDL)

20000

0.8

Epochs (SGDL)

20000

0.8

0.6

0.6

0.6

0.6

10000

0.4

10000

0.4

10000

0.4

10000

0.4

0.2

0.2

0.2

0.2

50

100

150

200

0.0

50

100

150

200

0.0

50

100

150

200

0.0

o

50

100

150

200

0.0

Frequency [Hz]

Frequency [Hz]

Frequency [Hz]

Frequency [Hz]

G 4

= 1.0

G 4

1.0

G 4

= 1.0

G 5

1.0

G 3

- 0.8

G 3

0.8

G 3

0.8

G 4

- 0.8

MGDL

G2

0.6

MGDL

G2

0.6

MGDL

G2

0.6

MGDL

G3

0.6

0.4

0.4

0.4

G2

0.4

G 1

0.2

G 1

0.2

G 1

0.2

G 1

0.2

50

100

150

200

0.0

50

100

150

200

0.0

50

100

150

200

0.0

50

100

150

200

0.0

Frequency [Hz]

Frequency [Hz]

Frequency [Hz]

Frequency [Hz]

For choices of t min and t max , we let I 1 := { 10 -4 , 10 -5 , 10 -6 } and I 2 := { 1 × 10 -3 , 5 × 10 -4 , 1 × 10 -4 } . For both SGDL and MGDL, we test the pair ( t min , t max ) from all possible cases in the set I 1 × I 2 , the batch size is chosen from 512 , 1024 , or the full gradient for each epoch, and the total number of the epochs K is set to be 30,000.

The supporting figures for this experiment include Figure 10, illustrating the evolution of spectrum comparison between SGDL and MGDL for settings 1-2 of Section 3.2. The mean of colorbar in Figure 10 is consistent with that in Figure 9.

B.3 Section 3.3

The network structure for SGDL is

and that for MGDL is

For SGDL and all grades of MGDL, we select the learning rate from the set { 10 -2 , 5 × 10 -3 , 10 -3 , 5 × 10 -4 , 10 -4 } , choose the full gradient for each epoch, and set the total epoch number K to be 10 , 000 .

00000

00000

00000

00000

- 1.0

1.0

1.0

1.0

Epochs (SGDL)

1000020000

0.8

Epochs (SGDL)

20000

0.8

Epochs (SGDL)

20000

0.8

Epochs (SGDL)

20000

0.8

0.6

0.6

0.6

0.6

0.4

0000T

0.4

00000

0.4

10000

0.4

0.2

0.2

0.2

0.2

100

200

300

400

0.0

o

100

200

300

400

0.0

100

200

300

400

0.0

100

200

300

400

0.0

Frequency [Hz]

Frequency [Hz]

Frequency [Hz]

Frequency [Hz]

G 4

- 1.0

G 4

1.0

G 4

1.0

G 4

1.0

0.8

G 3

0.8

G3

0.8

G 3

0.8

MGDL

G2

0.6

MGDL

G 2

0.6

MGDL

G2

0.6

MGDL

G2

0.6

0.4

0.4

0.4

0.4

G 1

0.2

G 1

0.2

G 1

0.2

G 1

0.2

100

200

300

400

0.0

100

200

300

400

0.0

100

200

300

400

0.0

100

200

300

400

0.0

Frequency [Hz]

Frequency [Hz]

Frequency [Hz]

Frequency [Hz]

(a) G 1: 20.41

(b) G 2: 22.67

(c) G 3: 23.71

(d) G 4: 24.18

(e) SGDL: 22.84

100

100

100

200

200-

200

000

400

400

400

500

500

500

0

100

200

300

400

500

0

100

200

300

400

500

0

100

200

300

400

500

0

100

100-

100

200

200-

200

300

000

400

400

400

500

500

500

0

100

200

300

400

500

0

100

200

300

400

500

0

100

200

300

400

500

The quality of the reconstructed image in Section 3.3 is evaluated by the peak signal-to-noise ratio (PSNR) defined by

where v is the ground truth image, ˆ v is reconstructed image, n is the number of pixels in v , and ∥·∥ F denotes the Frobenius norm of a matrix.

The supporting figures for this experiment include Figures 11-13, which display the predicted image for MGDL and SGDL corresponding to the Cat, Sea, and Building images, respectively.

(a) G 1: 18.62

(b) G 2: 21.50

(c) G 3: 23.42

(d) G 4: 24.32

(e) SGDL: 20.39

0

0

200

200

200

400

400

400

600

600

600

800

800

800

1000

1000

1000

1200

1200

1200

0

250

500

750

10001250

1500

1750

2000

0

250

500

750

10001250

1500

1750

2000

0

250

500

75010001250

1500

1750

2000

0

0

0

200

200

200

400

400

400

600

600

600

800

800

800

1000

1000

1000

1200

1200

1200

0

250

500

750

1000

1250

1500

1750

2000

0

250

500

750

1000

1250

1500

1750

2000

0

250

500

750

1000

1250

1500

1750

2000

(a) G 1: 17.29

(b) G 2: 18.95

(c) G 3: 20.67

(d) G 4: 21.97

(e) SGDL: 19.09

200

200

200

400

400

400

600

600

600

800

800

800

1000

1000

1000

1200

1200

1200

250

500

750

1000

1250

1500

1750

2000

0

250

500

750

1000

1250150017502000

0

250

500

750

1000

125015001750

2000

0

200

200

200

400

400

400

600

600

600

800

800

800

1000

1000

1000

1200

1200

1200

250

500

750

1000

1250

1500

1750

2000

0

250

500

750

1000

1250

15001750

2000

0

250

500

750

1000

1250

1500

1750

2000

(a)

(b)

(c)

(d)

SGDL

MGDL

SGDL

MGDL

0.20

0.20

0S'0

0S'0

0.18

0.18

K=1

K =1

0.45

0.45

K = 1

K = 1

0.16

0.16

0.40

0.40

SS07

0.14

5507

0.14

Loss

SS07

0.12

0.12

K =5

K = 5

0.35 -

SE'O

K=5

K = 5

0.10

0.10

K =7

0.30

0E'O

K = 7

0.08

80°0

K =10

0.25

0.25

-11

K =10

K =7

0.06

0.06

K =10

K =10

0

Training time [s]

1000

2000

3000

0

Training time [s]

1000

2000

3000

0.20

0

Training time [s]

1000

2000

3000

0.20

。

Training time [s]

1000

2000

000E

SGDL

MGDL

SGDL

MGDL

3.0

3.0

8.0

8.0

2.8

2.8

K =1

7.5

7.5

K=1

2.6

2.6

K=1

7.0

7.0

K = 1

S507

2.4

Loss

2.4

SS07

6.5

S507

6.5

2.2

2.2

K =5

K=5

6.0

6.0

K =5

K = 5

2.0

2.0

5.5

5.5

K =7

1.8

1.8

K =7

5.0

5.0

K =7

K =10

4.5

4.5

K =10

1.6

1.6

K =10

K =10

0

Training time [s]

1000

2000

3000

0

Training time [s]

1000

2000

3000

4.0

0

Training time [s]

1000

2000

3000

4.0

Training time [s]

1000

2000

000E

B.4 Section 3.4

The network structure for SGDL is

For MGDL, we consider two grade splittings. In the first splitting, we split network (13) into three grades, each with two hidden layers. The structure of MGDL for this splitting is as follows:

In the second splitting, we split network (13) into six grades, with each grade containing one hidden layer. The structure for MGDL for this case is as follows:

For choices of parameters t min and t max , we let I 1 := { 10 -4 , 10 -5 } and I 2 := { 10 -3 , 10 -4 } . For both SGDL and MGDL, we test ( t min , t max ) from all possible cases in the set I 1 × I 2 , choose the batch size from 512, 1024, or the full gradient for each epoch, and set the total number of the epochs K to be 2,000.

The supporting figures for this experiment include Figure 14, which compares the training and validation loss for SGDL and MGDL with structure (14); Figure 15, which compares the training and validation loss for SGDL and MGDL with structure (15).

(a)

(b)

(c)

(d)

SGDL

MGDL

SGDL

MGDL

0.20

0.20

0S'0

0.50

0.18

0.18

K=1

K =1

0.45

0.45

K = 1

K = 1

0.16

0.16

0.40

0.40

SS07

0.14

5507

0.14

Loss

SS07

0.12

0.12

K =5

0.35 -

0.35

K=5

K=5

K = 5

0.10

0.10

K =7

0.30

0.30

K = 7

0.08

80°0

K =10

0.25

0.25

K =10

K =7

0.06

0.06

K =10

K =10

0

1000

2000

3000

0

1000200030004000

0.20

0

1000

2000

3000

0.20

1000200030004000

Training time [s]

Training time [s]

Training time [s]

Training time [s]

SGDL

MGDL

SGDL

MGDL

3.0

3.0

8.0

8.0

2.8

2.8

K =1

7.5

7.5

K =1

2.6

2.6

K = 1

7.0

7.0

K = 1

S507

2.4

2.4

SS07

6.5

S507

6.5

K =3

2.2

2.2

K =5

K=5

6.0

6.0

K =5

K = 5

2.0

2.0

5.5

5.5

K =7

1.8

1.8

K = 7

5.0

5.0

K =7

K =10

4.5

4.5

K =10

1.6

1.6

K =10

K =10

0

Training time [s]

1000

2000

3000

Training time [s]

1000200030004000

4.0

0

Training time [s]

1000

2000

3000

4.0

Training time [s]

10002000

00OE

4000

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We claim the paper's contributions accurately and clearly in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations after the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings,

model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The loss functions for SGDL and MGDL are provided. In each experiment, we clearly state experimental settings, including how to choose training/validation/testing data. The structure of SGDL and MGDL, and the parameters required in training are provided in B. The computer code is available online through the GitHub link provided in the abstract.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The computer code is available online through the GitHub link provided in the abstract. A README.txt file containing guidelines for using the code for each example is also provided.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The loss functions for SGDL and MGDL are provided. In each experiment, we clearly state experimental settings, including how we choose training/validation/testing data. The structure of SGDL and MGDL, and the parameters required in training are provided in B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide tables of relative mean square error accuracy and PSNR values, along with figures showing the training/validation loss and PSNR values throughout the training process.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the experiments compute resources in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research is conducted under the guidance of Code Of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the societal impacts in the conclusion section.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We utilize the Div2K and MNIST datasets and properly cite their creators.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the issue of spectral bias in deep neural networks (DNNs), where DNNs tend to prioritize learning lower-frequency components of a function, struggling with high-frequency features. The proposed solution involves decomposing high-frequency functions into compositions of low-frequency functions using a novel approach termed Multi-Grade Deep Learning (MGDL). MGDL incrementally trains DNNs, each grade focusing on learning low-frequency information from the data. The efficacy of MGDL is demonstrated on synthetic, manifold, and MNIST datasets, showing improvements in capturing high-frequency components compared to traditional methods.\n\nSTRENGTHS:\nQuality: The paper provides a thorough theoretical foundation and supports its claims with experimental results, although these are not entirely convincing.\nClarity: The paper is generally well-organized, with clear explanations and effective use of figures and mathematical expressions.\n\nWEAKNESSES:\nOriginality: lacks new theoretical contributions for MGDL method. \nPerformance and Stability: There are concerns about the performance and stability of MGDL, as evidenced by loss spikes and lack of significant loss reduction in real-world tasks (Figure 8).\nMinor Punctuation Errors: For example, On Table 1,\"32,401\" and \"27, 817\".\nImage Readability: The legends of figures in this paper are too small, making them difficult to read and interpret.\n\nQUESTIONS:\n1. Is the MGDL method mentioned in the paper applicable to other model architectures such as CNNs and Transformers?\n2. Does the training time of MGDL increase linearly with the number of grades?\n3. Please add the following citation as it is one of the earlier methods proposed to alleviate slow convergence for high frequencies:\n   Cai W, Xu Z Q J. Multi-scale deep neural networks for solving high dimensional PDEs. arXiv preprint arXiv:1910.11710, 2019.\n\nLIMITATIONS:\nSee weeknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the spectral bias of deep neural networks by proposing a method to capture high-frequency data effectively. The authors assert that previous works have not adequately addressed spectral bias, emphasizing its significance. They propose that high-frequency functions can be decomposed into a sum-composition of low-frequency functions. To implement this, they employ a multi-grade deep learning (MGDL) model to achieve the sum-composition functions from the original function. The authors provide empirical evidence of the effectiveness of this approach in three distinct scenarios, including high-frequency noise-injected MNIST.\n\nSTRENGTHS:\n- This paper is easy to follow and well-written.\n- The logical progression from the motivation that high-frequency components can be estimated by the sum of composites of low-frequency components is sound.\n- The proposed method would likely address scenarios involving high-frequency noises or data containing high-frequency components.\n\nWEAKNESSES:\n- The paper fails to highlight why addressing spectral bias is significant. Recent network architectures perform well in various \ncomputer vision fields without specifically addressing this issue, where the datasets they use mostly contain high-frequency noises or noisy data themselves. From my perspective, the advantage of this paper in handling high-frequency components primarily benefits specific synthetic scenarios, such as those provided by the authors. I believe there would be more compelling examples to underscore the importance of the matter where the claim is significant. I recommend that the authors provide more practical scenarios where not addressing spectral bias leads to tangible failures, along with a rationale for why it is important to capture high-frequency elements.\n\n- While Figure 1 demonstrates the effectiveness of sum-composition in shallow deep neural networks, it does not provide a rationale for freezing components in each composition as done in MGDL. Based on this motivation alone, other learning methods beyond MGDL could potentially achieve the same or even better results. In this regard, why was MGDL exclusively used for the experiments? Considering that a residual network (ResNet) also employs a sum-composition approach, why wasn't it included in the experimental analysis?\n\n-  I believe the major problem with this paper lies in the experiment section. It provides insufficient experimental evidence to substantiate the authors' claims.\n  - First of all, as aforementioned, the scope of the experiments is very limited, involving only three synthetical scenarios and experiments with a fixed network architecture for each scenario. Additionally, comparisons are made solely with SGDL.\n  - MGDL appears to capture diverse frequency successfully ranges through multiple grades (whereas SGDL does not). However, I suspect this outcome may stem from learning instabilities caused by a simplistic network architecture lacking 1) normalization layers; 2) skip connections (i.e., shortcuts); and 3) freezing some layers for MGDL. In this regard, a deep SGDL is highly likely to encounter training failures due to gradient issues, whereas MGDL freezing some layers may offer more training stability in reducing loss. Therefore, I believe that any methods or networks that train shallow networks sequentially could potentially benefit in a manner similar to [1]. All experiments should be reperformed to provide more convincing experimental support. \n    - [1] Relay Backpropagation for Effective Learning of Deep Convolutional Neural Networks, ECCV 2016\n \n- The experiments on high-dimensional data are limited to MNIST, which does not convincingly demonstrate the challenges of high-dimensional data as claimed. For example, the standard ImageNet training protocol uses 224x224 images, resulting in a dimension of 50,176 when vectorized,\n\n- Figure 8 contains very small fonts, which should be refined for better readability.\n\nQUESTIONS:\nPlease see the above weaknesses.\n\nLIMITATIONS:\n- Limitations are provided", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDNNs suffer from spectral bias, which is the tendency to prioritize learning of low-frequency functions. The authors propose a method called MGDL, which tackles the spectral bias problem by changing the network parameterization and loss function such that each layer in the network learns the residual of the function from the previous layers. This framework forces higher layers in the network to learn high frequency residuals rather than an additive low frequency function. The authors justify their method using regression experiments on synthetic and MNIST data.\n\nSTRENGTHS:\n- The idea presented is clear and intuitive. \n\n - I found section 2 to be easy to understand despite the many different notations used.\n\n- The experiments are organized in a logical order. Figure 4 clearly shows the advantage of MGDL in modeling functions with different frequency components.\n\n- Lastly the authors are honest in acknowledging the lack of theoretical justification of MGDL.\n\nWEAKNESSES:\n(1) The main weakness (and probably the most obvious) is the question of scalability. There is a heavy focus in this paper on synthetic experiments. The only real world experiments are done on MNIST. I am uncertain as to whether this work has the potential to be adopted for any real applications (in computer vision or otherwise). In other words, I think the impact is limited due to the lack of more complex experiments.\n\nQUESTIONS:\n(2) Is there a reason you are constrained to the MSE loss? Does this idea work on other loss functions? \n\n(3) I am confused by Figure 3. I see that this is a plot of the functions learned by each layer in the frequency domain, but what is the main take away? For example, Grade 4 does not seem to be learning anything in the first two settings, whereas according to my intuition it should be learning the highest frequencies.\n\nLIMITATIONS:\nAdequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-29"}