Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
Authors: Kexuan Shi
Hai Chen
Leheng Zhang
Shuhang Gu
Venue: ICML 2025 poster (2025)
Keywords: Spectral Bias
Implicit Neural Representations
Training Dynamics
Inductive Gradient Adjustment
Primary area: deep_learning->everything_else
TL;DR: Spectral Bias, Implicit Neural Representations, Training Dynamics

Abstract: Implicit Neural Representations (INRs), as a versatile representation paradigm, have achieved success in various computer vision tasks. Due to the spectral bias of the vanilla multi-layer perceptrons (MLPs), existing methods focus on designing MLPs with sophisticated architectures or repurposing
training techniques for highly accurate INRs. In this paper, we delve into the linear dynamics model of MLPs and theoretically identify the empirical Neural Tangent Kernel (eNTK) matrix as a reliable link between spectral bias and training dynamics. Based on this insight, we propose a practical **I**nductive **G**radient **A**djustment (**IGA**) method, which could purposefully improve the spectral bias via inductive generalization of eNTK-based gradient transformation matrix. Theoretical and
empirical analyses validate impacts of IGA on spectral bias. Further, we evaluate our method on different INRs tasks with various INR architectures and compare to existing training techniques. The superior and consistent improvements clearly validate the advantage of our IGA. Armed with our gradient adjustment method, better INRs with more enhanced texture details and sharpened edges can be learned from data by tailored impacts on spectral bias. The codes are available at: [https://github.com/LabShuHangGU/IGA-INR](https://github.com/LabShuHangGU/IGA-INR).

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
Kexuan Shi
Hai Chen 23
Leheng Zhang' Shuhang Gu'
Abstract
to parameterize discrete signals in implicit and contin- uous manner. Benefiting from the continuity and implicit nature, INRs have gained great attention as a versatile signal representation paradigm (Luo et al., 2025) and achieved state-of-the-art performance across various computer vision tasks such as signal representation (Sitzmann al., 2020; Liang al., 2022; Saragadam et al., 2023; Kazerouni et al., 2024; Zhang et al., 2025), 3D shape reconstruction (Zhu et al., 2024; Zhang et al., 2024), and novel view synthe- sis (Mildenhall et al., 2021; Müller et al., 2022).
Implicit Neural Representations (INRs), as ver- satile representation paradigm, have achieved suc- cess in various computer vision tasks. Due the spectral bias of the vanilla multi-layer per- ceptrons (MLPs), existing methods focus on de- signing MLPs with sophisticated architectures or repurposing training techniques for highly ac- curate INRs. In this paper, we delve into the linear dynamics model of MLPs and theoreti- cally identify the empirical Neural Tangent Kernel (eNTK) matrix as reliable link between spectral bias and training dynamics. Based on this in- sight, we propose practical Inductive Gradient Adjustment (IGA) method, which could purpose- fully improve the spectral bias via inductive gen- eralization of eNTK- -ased gradient transforma- tion matrix. Theoretical and empirical analyses validate impacts of IGA on spectral bias. Fur- ther, we evaluate our method on different INRs tasks with various INR architectures and com- pare to existing training techniques. The supe- rior and consistent improvements clearly validate the advantage of our IGA. Armed with our gradi- ent adjustment method, better INRs with more enhanced texture details and sharpened edges can be learned from data by tailored impacts spectral bias. The codes are available at: http://ggttbb.coomnaaaSSlaaaaugu
However, obtaining high-precision INRs is non-trivial. MLPs with ReLU activations often fail to represent high- frequency details. Such tendency of MLPs to represent simple patterns of target funetion is referred to as spectral bias (Xu, 2018; Rahaman et al., 2019). To improve the performance of INRs, great efforts have been made to al- leviate or overcome this bias of MLPs. These efforts have primarily focused on how MLPs are constructed, such as complex input embeddings (Tancik et al., 2020; Müller et al., 2022; Xie al., 2023) and sophisticated activation func- tions (Sitzmann al., 2020; Ramasinghe & Lucey, 2022: Saragadam al., 2023; Zhu et al., 2024; Shenouda et al., 2024). Recently, two studies find that training dynamics adjustments based on traditional training techniques, i.e.. Fourier reparameterized training (FR) (Shi et al., 2024a) and batch normalization (BN) (Cai et al., 2024), can overcome the spectral bias. Compared to the above approaches empha- sizing architectural refinements, these methods could offer considerable representation performance gains while barely increasing the complexity of inference structures.
Despite these works serendipitously revealing the great po- tential of training dynamics in alleviating spectral bias for better INRs, the underlying mechanism driving their im- provements remains unclear. Moreover, there is no clear guidance the choice of Fourier bases matrix for reparam- eterization batch normalization layer to confront spectral bias with varying degrees in various INR tasks. Lack of reli- able insight and guidance results in unstable and suboptimal improvements across different scenarios and INR models (Essakine et al., 2024), hindering their broader applications.
1. Introduction
The main idea of implicit neural representations (INRs) is us- ing neural networks such as multi-layer perceptrons (MLPs)
'School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China "Snn Yat-sen University, Guangzhou, China 'North China Institute of Computer Systems Engineering. Correspondence to: Shuhang Gu <shuhanggu @ com..
Proceedings of the 42 International Conference on Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Notably, a stream of works (Ronen al., 2019; Tancik al., 2020; Shi et al., 2024a; Cai et al., 2024; Zhu et al., 2024)

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
adopts the spectrum of the Neural Tangent Kernel (NTK) matrix (Jacot et al., 2018) to analyze spectral bias. Taking step further, Geifman et al. (2023) have made first attempt to modify the spectrum of the NTK matrix to improve con- vergence speeds, but limited to the toy MI.P and restricted synthetic data. These prior works imply the potential of the NTK matrix to guide adjustments in training dynamics for better INRs. However, the NTK matrix is known for the general lack of analytical expression and the quadratic growth in memory consumption as the data size increases (Xu et al., 2021; Novak et al., 2022; Mohamadi et al., 2023), especially in INRs, which involve various activation func- tions and dense predictions. Thus, how to effectively adjust training dynamics of MLPs to purposefully overcome spec- tral bias in INRs remains an open and valuable problem.
synthetic data and range of implicit neural represen- tation tasks. Compared to existing training dynamics approaches, our method enables tailored improvements in mitigating the spectral bias of standard MLPs, result- ing in implicit neural representations with enhanced high-frequency details.
2. Related Work
Implicit neural representations. Implicit neural represen- tations (INRs) representing the discrete signal as an implicit continuous function by MLPs have gained lots of attention. Comparing to traditional grid-based methods, INRs offer remarkable accuracy and memory efficiency across various representation tasks such as 1D audio representation (Kim et al., 2022), 2D image representation (Klocek et al., 2019; Strümpler al., 2022), 3D shape representation (Park et al., 2019; Martel et al., 2021), novel view synthesis (Mildenhall et al., 2021; Saratchandran et al., 2024). However, vanilla MLPs with ReLU activations fail to represent complex sig- nals. Therefore, various modifications have been studied. One category focuses on input embedding. such as using Po- sitional Encoding (PE) (Mildenhall al., 2021) or learned features (Takikawa al., 2021; Martel et al., 2021; Xie et al., 2023). A different category of modifications concen- trates on better activations; for instance, periodic activations (Sitzmann et al., 2020; Fathony et al., 2020; Zhu et al., 2024) and Gabor wavelet activations (Saragadam et al., 2023) pro- vide more accurate and robust representations. Considering broader framework, these works can be summarized as efforts enhance the representational capacity of INRs or MLPs (Yüce et al., 2022). Recently, another category, dis- tinet from those previously discussed, focuses on the train- ing process of MLPs. Shi et al. (2024a) find that learning parameters in the Fourier domain, i.e., Fourier reparame- terized training (FR), can improve approximation accuracy of INRs without altering the inference structure. Cai et al. (2024) propose that the classic batch normalization layer (BN) can also improve the performance of INRs.
In this paper, we delve into the linear dynamics model of MLPs and propose an effective training dynamics adjust- ment strategy guided by theoretical derivation to purpose- fully alleviate the spectral bias issue. Our strategy allows for tailored impacts on spectral bias, achieving more stable im- provements. Specifically, these impacts on spectral bias are tailored by purposefully adjusting the spectrum of the NTK matrix, which closely connects the spectral bias with train- ing dynamics. Given that the NTK matrix is not available in most cases, we theoretically justify the empirical NTK (eNTK) matrix (Novak et al., 2022) as a tractable surrogate. Empirical evidence corroborates these theoretical results. Based on this insight, we further propose an inductive gra- dient adjustment (IGA) method, which could purposefully improve the spectral bias via inductive generalization of the eNTK-based gradient transformation matrix with millions of data points. We give both theoretical and empirical anal- ysis of our method, demonstrating how it tailors impacts on spectral bias. Besides, our IGA method can work with previous structural improvement methods such as Positional Encoding (Tancik et al., 2020) and periodic activations (Sitz- mann et al., 2020). We validate our method in various vision applications of INRs and compare it to previous training dynamics adjustment methods. Experimental results show the superiority of our IGA method. Our contributions are summarized as follows:
Spectral bias. Spectral bias refers to the learning bias of MLPs to prioritize learning simple patterns or low-frequency components of the target function. Great efforts have been made to dissect this bias. Rahaman et al. (2019) find that spectral bias varies with the width, depth and input mani- fold of MLPs. Xu (2018) attribute this bias of MLPs with Tanh activation to the uneven distribution of gradients in the frequency domain. From the perspective of linear dynamics model of MLPs, spectral bias is induced by the uneven spec- trum of the corresponding Neural Tangent Kernel (NTK) matrix (Jacot et al., 2018; Arora et al., 2019; Ronen et al., 2019). Inspired by this theoretical result, a series of works (Tancik et al., 2020; Shi et al., 2024a; Cai et al., 2024) try to elucidate spectral bias by observing spectral distribution of
We connect spectral bias with linear dynamics model of MLPs and derive a training dynamics adjustment strategy guided by a theoretical standpoint to purpose- fully improve the spectral bias.
We propose a practical inductive gradient adjustment method which could purposefully mitigate spectral bias of MLPs even with millions of data points. Theoretical and empirical analyses show how our method tailors impacts on spectral bias.
We present comprehensive experimental analyses on
2

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
the NTK matrix and find that PE, FR and BN help to attain more even spectrum. In this paper, we adopt the insight of attaining more even spectrum of the NTK matrix. For the first time, we show that modifying the eNTK matrix spectrum can have the similar impacts on spectral bias and propose a practical gradient adjustment method, which im- proves spectral bias of INRs via inductive generalization of eNTK-based gradient transformation matrix.
ries (e.g., SSIM, MS-SSIM (Wang et al., 2003)) in INRs. Following this insight, recent works (Tancik et al., 2020; Bai et al., 2023; Shi et al., 2024a; Cai et al., 2024) suggest that MLPs with a more even spectrum of K are less affected by spectral bias.
3.2. Inductive Gradient Adjustment
As previously discussed, such an uneven spectrum of K leads to slow convergence of high frequency components. Naturally, adjusting the evenness of the spectrum of K is expected to tailor the improvement of spectral bias. Follow- ing this intuition, we focus on the adjustment of eigenvalues of K, i.e., K-based adjustment. Inspired by the theoretical insights of Geifman et al. (2023), this can be achieved by multiplying transformation matrix S as follows':
3. Method
In this section, we first review the linear dynamics model of MLPs, indicating the potential of adjusting the NTK spec- trum to overcome spectral bias in INRs. Then we analyze the VTK-based adjustment and show that the NTK matrix's s intractability renders it impractical. By theoretical analy- sis, justify the eNTK matrix as tractable proxy, but it succumbs to dimensionality challenges as data size grows. Thus, we propose practical gradient adjustment method that improves spectral bias via inductive generalization of eNTK-base based gradient transformation matrix.
(2) where S = > (A)()))))) {9:);=1 denote the po- tential eigenvalue transformations. As driven by the training dynamics of Eq. 2, the spectrum of the modified kernel ma- trix K can be theoretically approximated as {A.)))
O+11 O - Vf ()Sr,,
Although Eq. 2 offers theoretical approach to adjust impacts of training dynamics on spectral bias, Eq. 2 is almost infeasible in practical INRs tasks. The first chal- lenge is that an analytical expression for K is difficult to derive (Xu et al., 2021; Novak et al., 2022; Wang et al., 2023; Shi et al., 2024b). Specifically, the analytical expres- sion [ Go) T Vef(X; 0o)] of K becomes in- tractable as the depth of the network increases. The second challenge is that the size of K grows quadratically with the number of data points (Mohamadi et al., 2023). Considering the Kodak image fitting task (Saragadam et al., 2023; Shi et al., 2024a), it requires decomposing a matrix on the order of 10 entries, which occupies over 8192GiB (gibibytes) of memory if stored in double precision.
3.1. Connecting spectral bias with training dynamics
Without loss of generality and for simplicity, we consider a scalar discrete signal. We define the vector of signal val- ues as y = (yi....,,N' E R.V. and the corresponding coordinate vectors (æ (......, N denoted by X E N xd The INR of this signal is parameterized as f f(x; +), where f(; Q) denotes MLP. Parameters θ E IE are optimized with respect 1/4 (, -)=))²² via gradient descent. The residual (f θ2) y:) ==) at time step t is denoted as rt. Training dynamics of r. can be approximated by the linear dynamics model: r = (I - 1-1KKrr ) under the conditions of wide networks and small learning rates 17 (Du et al., 2018; Arora al., 2019; Lee et al., 2019). K is the NTK matrix defined by Fe, Ve]((X; θo) Vef(X;:0) On)], where Ve fx; Qa) E R With the eigenvalue decom- position of K ΣN λ.,;o, and recurrence unfolding of the linear dynamies model, training dynamies of r2 are characterized as follows (Arora et al., 2019):
To overcome the first challenge, numerous studies (Xu et al., 2021; Novak et al., 2022; Mohamadi et al., 2023) have studied the eNTK matrix K defined by V(x; O,) Ve, 0x) and Geifman et al. (2023) show that replacing K with K (K-based) in Eq. 2 can accelerate convergence on toy settings. However, whether K K-based adjustment has the similar impacts on spectral bias in Eq. 2 and remains consistently effective in gen- eral settings, is an open question. To justify feasibility, we prove that eigenvectors of K as well as eigenvalues converge to the corresponding parts of K as the network width increases. Namely, impacts on spectral bias by K- based adjustment can be approximately equivalent to that of K -basedd adjustment. The detailed theoretical analysis presented in our Theorem 3.1. Then, empirically vali- date the aforementioned equivalence and demonstrate that
-il V i=1 (1 -0
||rl|22
- 11
v)2
(1)
Eq. shows that convergence rate of f(x θ) at the projec- tion direction v.T with larger X; is faster. For vanilla MLPs, projection directions related high frequencies are consis- tently assigned to small eigenvalues, while those related to low frequencies correspond to larger eigenvalues (Ronen et al., 2019; Bietti & Mairal, 2019; Heckel & Soltanolkotabi, 2020). Such an uneven spectrum of K leads to extremely slow convergence rates to high frequency components of the signal, leading to a reduction in average approximation accuracy (e.g., MSE loss) as well as frequency-awar met-
'This matrix is also known the pre-conditioned matrix in the field of optimization.
3

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
1.2
Two-laeer MLPs (Width=1024)
Epocts [x10"
0.211r
2.0
0.412z
L8
1.2-
Two-layer MLPs (Width=4096)
Epocaa (M 204)
0.2Hz
1.0
0.4H2
1.0
SGD-
SGD 5
24
6.
08
3
04
na
1.0-
SGD
SGD4
Epocas
04
0.6
0a
MSE Logs
0.8
S
SGD-5,
5
5,
0.8
SCID
SGD-5,
3
0.6
Egoctts (x:07
0.8H3
1.0
Epochs (x10)
1.8Ff
0.6
12 (x109)
0.9H4
Epoccla (M300))
1.6ff
0a
0.4
0.6
0.4
67
6
Egochs (x10")
5
5
5,
0.2
Epochs (x10*)
10
15
20
5
8
5
Figure/ 1. Evolution of approximation error with training iterations in time and Fourier domain. Line plots visualize MSE loss curves of
MLPs with 1024 and 1096 neurons optimized by four gradient adjustments, i.e., I, S, S, S.. The shaded area of lines indicates training fluctuations. Heatmaps show the relative error Au Eq. 6 on four frequency bands. Details are the experiment I of Sec.
K-based adjustment can tailor the improvement of spectral bias on more general settings in Sec. 4. The formula of K- K-based adjustment is as follows:
Algorithm 1 Inductive Gradient Adjustment (IGA)
Input: Mini-batch X, = { 2/) J model f(, O2)
Output: Updated parameters 2x+1
O O - -V, O,)Sr.
(3)
// Sample a subset
where K = - = E. (o(00/AA))))
Sample X C X2 based on the strategy in Sec. 3.4 // Compute gradients and empirical kernel Compute G. f θτ) and then K. = G Ge // Construct the transformation matrix Compute S. based on K. using the method in Sec. 3.4 // Generalize the adjustment to the full mini-batch Adjusted gradient: Jr = >i=1 f(,, S.rS.r; update the parameters by: Oc+1 +- O -1 77 return Or+1
Although adjustment based on K instead of K achieves sim- ilar effects to K while avoiding intractability, it still encoun- ters the curse of dimensionality as the data size N increases. One common approach
estimate K using sampled data, which is efficient for tasks where K indirectly impacts train- ing without affecting gradients, such as neural architecture search (Park et al., 2020) or adaptive module (Shi et al., 2024b). However, how to effectively utilize K from sam- pled data to directly adjust the training dynamics on the pop- ulation data remains a problem. In Theorem 3.2, we show that training dynamics r could be estimated via inductive generalization of the linear dynamics model of sampled data and analyzable error. Inspired by this theoretical foundation, we propose to inductively generalize the gradient adjustment based on sampled data to the gradients of the population data. Specifically, considering an input set X = { of N coordinates in Idd as the population data, X can be divided into n groups (N = =np), where the j-th group X and corresponding labels are {(ar, ) Then we sample one data point from each group to form the sampled set Xe. where |x,| is defined by K. = ve.f(X.; [V..((ee O,). We construct the corresponding transformation matrix S.. Then inductive adjustments S. are generalized to gradients of the population data X as follows:
And the j-th elements of these vectors ri all belong to the group X Therefore, our inductive gradient adjustments are generalized from one sampled data point of X to the remaining (p - 1) data points of X for all j = 1, =I....,.. n.
3.3. Theoretical Analysis
In this subsection, we provide theoretical analysis to jus- tify the feasibility of our IGA. In Theorem 3.1, we prove that adjustments based on eNTK matrix K asymptotically converge to those based on NTK matrix K as the network width increases. Then we delve into the estimate of training dynamics via inductive generalization of linear dynamics model of sampled data in Theorem 3.2.
< N. The eNTK matrix on X
In specific, we build upon previous training dynamics works (Du al., 2018: 8; Arora et al., 2019; Lee al., 2019; Geifman et al., 2023) and analyze a two-layer network f(æ; Q) with width m. Without loss of generality, we assume the training set { {xt, 31 J.M and adopt the notations in Eq. 2-4. Detailed analysis settings are in Appendix A. Based on these settings, the following Theorem 3.1 show that K K-based adjustments achieves similar effects to K- --based adjustments.
1.+1= = 0 - E' 1=1 Ve.f O,)
(4)
where rú is the residual of X; = (a/) v/ 33ا= S. ELs (3(())))AA K. = Ei= 7, We intro- duce the implementation details of Eq. 4 in Sec. 3.4 and summarize the IGA method in the form of an algorithm block 1.
Theorem 3.1. (informal) Assuming that o,ū;; 0 for E [N]. Let (9)( be a set of Lipschitz continu- ous functions, with learning rate n < min! (ainf(mmxxgg (max(g((A)) + (max(g.(A)) + min(g.(A))) -1) for all e >> 0, there always exists M > O. such that m > M, for
As shown in equation 4, all r are linearly transformed by S.. Please note that for each vector ri (where = 1.....,) p).
min(g:
its j-th element is scaled by the j-th column of the matrix S..
E [N], we have: lg. (A:) - (x,) < 61, 1| 6,1 < E2.

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
ReLU
ReLU-S,-2
ReLU-S,-8
SIREN
SIREN 5,
SIREN 5,
Precuescy (the)
1.0
1.0
1.0
1
1.10 3
10
10
0.8
0.8
20-
200
0.6
30
0.6
30
0.6
30
0.6
0.4
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
200
0.2
30
Epocis (x10%)
Epochs (x10*)
Egochs (x100))
Epochs (x10')
Egochs (x10*)
Epochs (x10')
Figure 2. Progressively amplified impacts on spectral bias of ReLU and SIREN by increasing the number of balanced eigenvalues via S.
when p 8, i.e., IGA, ReI.U denotes the ML. with ReL.U optimized via vanilla gradients; ReI.U-S,-2 denotes the ReI.U optimized via
gradients adjusted by S. with balanced eigenvalues. Details and more results are in the experiment 2 Sec. 4 and Appendix B.
ReLU by balancing eigenvalues 1.0
ReLU by balancing 4 eigenvalues
ReLU by balancing 6 eigenvalues
ReLU by balancing eigenvalues
at == 20Hz
at k 20H?
1.0
at kt 20Hz
1.0
20Hz
1.0
0.8
0.8
0.8
0.6-
0.6
0.6
0.6-
JGA 1
IGA 1
IGA
IGA
0.4-
IGA-4
0.4
IGA-4
0.4
IGA-4
0.4
JGA-4
0.2
ICA-8
IGA-8
ICA-8
IGA-8
Baseline
Basellne
Baseline
Baseline
0.0
Epochs (x10=)
8
0.0
EEochs (x102)
10
0.0
Epoccs (x10')
10
0.0
0
Spochs (x10²)
SIREN by balancing 2 eigenvalues
SIREN by balancing eigenvalues SIREN by balancing eigenvalues
SIREN by balancing eigenvalues
De k = 40Hz
1.0
1G41
4 at 40Hz
1.0
ICA-1
k 40Hz
1.0
IGA-1
4x at k 40HZ
1.0
IGA--
0.8-
IG4-4
0.8-
IGA-4
0.8
IGA-
0.8
IGA-4
IG4-8
IGA-8
IGA-8
IGA-8
=
Boseline
Baseline
Baseline
0.6
Baseline
0.4
0.4
0.2
0.2
0.2
0.2
0.0
Epochs (x10²) 8
10
0.0
Epochs (x10*)
0.0
Epochs (x10*)
0.0
Epachs (x10*)
10
Figure 3. Relative error A curves of ReLU at 20Hz and SIREN at 40Hz with varying balanced eigenvalues and group size p. IGA-1 denotes that p= = 1, i.e., K-based gradient adjustment; Baseline denotes vanilla gradients. The shaded area of lines indicates training fluctuations, represented by twice the standard deviation. Details and more results are the experiment 2 of Sec. 4 and Appendix B.
thus
of at at time step L; r denotes the residual on Xe al time step t; f(x" Ve, Qt) is the ji-th row in Ke-
(1 - 11--n9i(A))) (A))) - (1- re)21 < Es,
Theorem 3.2 demonstrates that generalizing Ke, induced from sampled data Xe, to population data X provides an inductive estimation of overall training dynamics. The in- ductive estimate error E + E2 decreases as the network width increases, the consistency bound decreases, or both. This theoretical insight encourages us to generalize the transformation matrix S. to gradients of population data X, thereby tailoring impacts on spectral bias via Ke, S.. We provide empirical validation of Theorem 3.2 in Sec. 4, with its formal version and proof in Appendix A.
where 62, E3 and are of the same order as + O.
Theorem 3.1 illustrates that eigenvalues and eigenvectors of K converge one-to-one to eigenvalues and eigenvectors of K. That is, Š impacts not only the convergence rates Lg (A similar to those of S but also the corresponding convergence directions (z:). This theoretical results justify that the K-based adjustment is approximately equivalent to K-based adjustment. We further provide the empirical evidence in Sec, 4 and detailed proof of formal version in our Appendix A. Next, we analyze how training dynamics of X can be estimated from sampled data X.-
3.4. Implementation Details
Sampling Strategy. As illustrated in Sec. 3.2, we need to sample X, from groups of the population data X to ob- tain Ke, S.. In practice, we first form n groups by simply considering p adjacent input coordinates for each group. Specifically, for ID and 2D inputs, groups are defined by oveeoverlgp intervals and patches, respectively. For N d-dimensional input coordinates (d 2 3), typically repre- sented tensor of shape (N1..... na, d), we flatten the first d dimensions into a 2D tensor X' of shape (N, d) and then group it along the first dimension of X' as ID signals, Then, the input with the largest residuals in each group are selected to form the sampled data X.. To better illustrate the above process, we provide diagrams in Fig. 6. Details
Theorem 3.2. (informal) X is partitioned into n groups X, = {ai,.... = and N = np. For X,, the sam- pled point denoted as x² and these points form X. = {x0 Dj=1, < N. There exists e >> 0, for j E [n] and i,,12 E [p]. such that max(1(f(2) ; 0t) - v (aal,, 0.) -. IVl. [Z),) -Ve,)-,,, (ar < c. Then, for any Iú, assumed that x; belongs to Xj, such that:
((rr)) - ( TV. " Drrrl < 61 +2,
where 6 = O(c"))) ); c2 = 0(m-3/2); O(-) denotes equivalence; (Art); = (r41 - rt); denotes the residual
5

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
of hyperparameterss n and p will be introduced in the ex- periment section. The ablation study of n and p is in the Appendix F, As shown in our wide experiments, the above strategy is simple yet effective and easy to implement.
and outputs ʃe at frequency & to show spectral bias:
A = - [FDlee()))((( where FD denotes the discrete Fourier transform.
(6)
Experiment 1. In this experiment, as discussed above, we investigate the K- K-based and K- K-based adjustments. There- fore, the accurate computation of NTK matrix K is the key. Given that the accurate computation of K is non-trivial, we adopt a two-layer MLP with a fixed last layer as in Arora et al. (2019); Ronen et al. (2019), whose Kij can be com- puted by the formula i This architecture facilitates theoretical analysis but has lim- ited representational capacity for complex functions. Con- sidering these concerns, we constructed the following sim- ple function ((oo(()), sin(0) S + RA, widely used in prior works (Ronen et al., 2019; Geifman et al., 2023):
Construction Strategy. As illustrated in Eq. and Theo- rem 3.2, we adjust the evenness of the spectrum of K. via the transformation matrix S. to purposefully improve the spectral bias. We have K.- Ei= o and assume that A1 > >>.>>>>>>. Am > O. To adjust the evenness of spec- trum of Ke, Se is constructed to balance the eigenvalues of different eigenvectors of Ke; impacts are tailored by managing the number of balanced eigenvalues. For vanilla gradient descent, S. is constructed as follows:
1 (x: Xj + 1)( axccos(x; x,) )),
end
S. = Σ (A imstort
Σ
L.V. (5)
[E[start,edd]
sin(0.4x0) + sin(0.877) + sin(1 (1.6x0) + sin (3.2mm)) (7)
Generally, start is fixed at 1; end represents the controlled spectral range. Larger end values have stronger impacts on spectral bias. We demonstrate this effect in experiment 2 of Sec. 4. For Adam, we utilize Àend+1 instead of Astart to balance eigenvalues in Eq. 5 for better convergence, re- sulting in uniform eigenvalues of the modified Ke across the (end- - start 2) eigenvectors. Then, momentum and adaptive learning rates are updated according to the adjusted gradients. More discussion are in Appendix D.
We set N = 1022 for input samples via uniformly sampling 1024 θ in [0, 2T). We vary the width of the MLP from 1024 to 8192. We adopt the Identity matrix I (that is, the vanilla gradient) and a series of S, S.S. S. that start = 1 and end ranges from 10 to 14. Following our sampling strategy, we set p = 8 and n = 128, indicating that S, is only 1/64 the size of S, S. All MLPs are trained separately with the same fixed learning rates by SGD for 20K iterations. All models are repeated by 10 random seeds. We visualize the results of the baseline model and the adjustments with end = 14 in Fig. 1. More results can be found in Appendix B.
Multidimensional Output Approximation. In practical INRs tasks, the output is multidimensional, such as for color images. Compared to the data size N, it is trivial but still affects efficiency. Thus, we simply compute the Jacobian matrix Ver(X;;) by sum-of-logits, which is shown to be efficient in Mohamadi et al. (2023); Shi et al. (2024b).
Analysis of Experiment 1. In Fig. 1, line plots show the convergence trends of MLPs by K-based (S), K-based (S) and inductive gradient (S.) adjustments, compared to vanilla gradient (I). Heat maps with a consistent color scale visualize their spectral impacts, where darker colors indicate higher errors. Note that color bars of I darken rapidly from 0.4Hz, highlighting severe spectral bias in MLPs trained with vanilla gradients. All the adjustments introduce lighter shades over 0.4Hz and 0.8Hz, effectively improving spectral bias, therefore resulting in faster MSE convergence. Please note that while adjustments by S, Š and S. exhibit slight difference in MSE loss trends and impacts on spectrum in the case of width 1024, the differences are barely noticeable for width 4096. This result aligns with previous theoret- ical analysis that increasing width m reduces differences between K-based and K-based adjustments (Theorem 3.1) and errors induced by inductive generalization (Theorem 3.2). Moreover, despite the presence of difference, S and S. effectively improve spectral bias and the approximation accuracy, indicating that K successfully links spectral bias with training dynamics and our IGA method is effective.
4. Empirical Analysis on Simple Function Approximation
In this section, we aim to explore and provide empirical evidence for the theoretical results presented in Sec. 3.3. In experiment we investigate the effects of K-based and --based adjustments on spectral bias and observe that their differences diminish as the network width m increases (The- orem 3.1). Training dynamics curves further suggest that increasing width m reduces estimate error 61 + 62 from inductive generalization (Theorem 3.2). In experiment 2, we validate -bsed adjustments and IGA on more gen- eral settings, demonstrating that IGA purposefully improves spectral bias, with its impacts controllable by managing the number of balanced eigenvalues in Eq. 5. This suggests the effectiveness of inductive generalization in estimating the training dynamics (Theorem 3.2) on general settings.
Experiment 2. In this experiment, we aim to validate K- based adjustments and our IGA on more general settings. For IGA, we also aim to show that impacts on spectral
To better analyze, we follow Xu (2018); Shi et al. (2024a) and compute the relative error AS between target signal f
6

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
PE (24.60dB)
PE+FR (26.63dB)
PE+BN (25.45dB)
PE+IGA (28.06dB)
Figure 4. Visual examples of 2D color image approximation results by different training dynamics methods. Enlarged views of regions labeled by red boxes are provided. The residuals of these regions in the Fourier domain are visualized through heatmaps in the top right corner. The increase in error corresponds to the transition of colors in the heatmaps from blue to red. Detailed settings are in Sec. 5.
Table 1. Average metrics of 2D color image approximation results by different methods. The detailed settings can be found in Sec. 5.1. Per-scene results are provided Table 8-11. Note: SIREN+BN is omitted due to our finding their incompatibility, neither does the original work (Cai et al., 2024) demonstrate the feasibility.
Average Metric
ReLU
PB
SIREN +FR
Vanilla
+FR
+BN
+IGA
Vanilla
-FR
+BN
+IGA
Vanilla
+IGA
PSNR T
21.78
22.14
22.55
23.00
28.64
31.65
28.78
32.46
32.65
32.61
33.48
SSIM f
0.4919
0.5004
0.5126
0,7832
0.8167
0.8822
0.8975
0.8991
MS-SSIM f
0.6521
0.6800
0,7090
0.7383
0,9466
0.9564
0.9554
0,9752
0,9818
0.9820
0.9847
LPIPS
0.6302
0.6315
0.6182
0.5549
0.2225
0. 0.1869
0.2346
0.0938
0.0807
0.0813
0.0668
bias can be tailored by managing the number of balanced eigenvalues under varying p. Therefore, we consider two practical INR models, i.e., MI.Ps with ReI.U (Rel.U) and MLPs with Sine (SIREN) (Sitzmann et al., 2020) like Shi et al. (2024a). To analyze tailored impacts under varying p, we use settings in Rahaman al. (2019) and construct a ID function f R + IR. with abundant spectrum follows:
our theoretical strategy and practical outcomes, supporting the extension of IGA to more general applications.
5. Experiment on Vision Applications
In this section, we apply our inductive gradient adjustment (IGA) method to various vision applications of INRs. The superior and consistent improvements demonstrate the broad applicability and superiority of IGA.
sin (20mz) + sin( 407x) + sin(60 + sin (80Tx)
+ sin( (100mz)) + sin( 120mx).
(8)
5.1. 2D Color image approximation
SIREN is trained to regress the fx) with 2048 values uni- formly sampled in [-1, For ReLU, we halve the fre- quency of each component due to its limited representation capacity (Yüce et al., 2022). For IGA, we set p to 1, 4 and 8 and vary end from 1 to 7 construct the corresponding Se. When p- 1, IGA degenerates to K K-based adjustment. 256-width architecture for ReLU and SIREN, optimized by Adam for 10K iterations with a fixed learning rate. All models are repeated by 10 random seeds. In Fig. 2, we adopt heatmaps of relative error As evolution to show impacts on spectral bias. In Fig. 3, visualize A curves for some frequencies to precisely demonstrate impacts of IGA convergence rates.
Single natural image fitting has become an ideal test bed for INR models (Saragadam et al., 2023; Xie et al., 2023; Shi et al., 2024a), as natural images contain both low- and high-frequency components (Chan & Shen, 2005). In this experiment, we attempt to parameterize the function & IR2 - R3, R3,x -> Φ(x) that represents discrete im- age. Following Saragadam et al. (2023), we establish four- layer MLPs with 256 neurons per hidden layer. We test three MLPs architectures, i.e., MLPs with ReLU activations (ReLU), ReLU with Positional Encoding (PE) (Tancik et al., 2020) and MLPs with periodic activation Sine (SIREN) (Sitzmann et al., 2020). These three models are classic base- lines in INRs and are widely used for comparison (Sitzmann et al., 2020; Saragadam et al., 2023; Shi et al., 2024a). Re- sults on more activations can be found in Appendix E. We test on the first 8 images from the Kodak 24 (Franzen, 1999). each containing 768 x 512 pixels. For these images, K has approximately 10t1 entries, rendering decomposition and multiplication infeasible during training. Following our sam- pling strategy, we group each image into non-overlapping 32 32 patches (p 1024) and sample data with the largest residuals to form the X.. We construct Š, with end = 20 for SIREN and PE and end = 25 for ReLU due to its se- vere spectral bias. Therefore, we generalize the inductive
We adopt four mundhiddee--laye
Analysis of Experiment 2. In Fig. 2, for K-based adjust- ment and IGA, as more balanced cigenvalues are introduced, the proportion of lighter regions in heatmaps of both ReLU and SIREN grows, while the relative error curve decreases more rapidly in Fig. 3. These observations elearly illustrate that impacts on spectral bias by K-based adjustment and our IGA can be amplified by increasing the number of bal- anced eigenvalues on general settings. Further, as shown in Fig. 3. although the relative error curves of ReLU ex- hibit subtle differences as increases, our IGA effectively amplifies impacts on spectral bias with more balanced eigen- values. These results demonstrate the consistency between
7

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
Thai Ground truth
PE (0. (897) PE+FR (0. (0.9929) PE+BN (0.9916) PE+IGA (0. (0.9943)
Figure 5. Visual examples of 3D shape representation results by different training dynamics methods. Five images on the right correspond to the enlarged views of the red-boxed area of ground truth and four models, respectively. More results can be found our Appendix Table Intersection over Union (IOU) and Chamfer distance of 3D shape representation by different methods. The detailed settings can be found in Sec. 5.2, Per-scene results are provided in Table 12 and 13.
Average
ReI.U
PE
SIREN +FR
Metric
Vanilla
+FR
+BN
+GA
Vanilla
+FR
+BN
+IGA
Vanilla
-IGA
IOU
9.647e-1
9.654e.1
9
9.733e-1
9.942e11
9.961e.2
9
9.970c-1
9.889e 1 9.866e-1
9.897e11
Chamfer Distance 4 5.936e-6
5.8676-6
5.610e-6
5.487e-6
5.123e-6
5.111e-6 5,1
5.108e-6
5.688e-6
5.1636-6
5.157e-6
gradient adjustments, derived from less than 0.1% of the sampled data, to the entire population. For all architectures, we train baseline with vanilla gradients and with our IGA (+IGA) by Adam optimizer. The learning rate schedule follows Shi et al. (2024a). Detailed training settings can be found in Appendix G. Current training dynamics methods, i.e., Fourier reparameterizztioo (+FR) (Shi et al., 2024a) and batch normalization (+BN) (Cai et al., 2024), are compared. Their hyperparameters follow publicly available codes, and we make every effort to achieve optimal performance. In Table 1, we report average values of four metrics of differ- ent INRs over the first images of Kodak 24, where LPIPS values are measured by the "alex" from Zhang al. (2018).
ability to model complex topologies. In this section, we evaluate IGA on this task using five 3D objects from the public dataset (Martel et al., 2021; Zhu et al., 2024) and objects are sampled over a 512 grid following Saragadam et al. (2023); Shi et al. (2024a). For IGA, following our sam- pling strategy for high-dimensional data, we set p 256 and randomly draw 512 groups and generalize the inductive gradient adjustments on these groups in each iteration. The architectures of MLPs are consistent with Experiment 5.1. We use the Adam optimizer to minimize the l loss between voxel values and INRs approximations. For a fair compari- son, the same training strategy is adopted for FR, BN and IGA. Detailed settings can be found in our Appendix H.
As shown in Table 1 and Fig. 4, PE with our IGA (PE+IGA) not only achieves the best improvements on average approx- imation accuracy: PSNR values compared to PE, PE+FR, and PE+BN, but also excels in frequency-aware metrics: SSIM, MS-SSIM, with a more precise representation of high-frequenen details instead of being overly smooth like other methods. Concretely, shown in the spectra located in the top right corner of Fig. 4, improvements of IGA are uniformly distributed across most frequency bands. In contrast, while BN and FR increase PSNR values, most improvements are near the origin, i.e., in the low-frequency range. Furthermore, as shown in Table 8- IGA achieves consistent improvements across almost all scenarios and metrics. While FR and BN achieve competitive results in some scenarios, their performance is generally less stable and occasionally leads to suboptimal or negative outcomes, These observations indicate that our IGA method allows for more balanced convergence rates of a wide spectral range with millions of data. More results are in Appendix G.
In Table 2, we report the average intersection over union (IOU) and Chamfer Distance metrics of five objects for reference. Under our training settings, baseline models opti- mized by vanilla gradient have converged to a favorable op- timum, significantly outperforming prior works (Saragadam et al., 2023; Cai et al., 2024; Shi et al., 2024a). Nevertheless, our IGA method enables models to explore superior optima by purposefully improving the spectral bias, thereby leading to improvements in representation accuracy such as sharper edges in Fig. 5. More results are in our Appendix H.
5.3. Learning 5D neural radiance fields
Learning neural radiance fields for novel view synthesis, i.c., NeRF, is the main application of INRs (Mildenhall et al., 2021; Saragadam et al., 2023; Xie et al., 2023; Shi et al., 2024a). The main process of NeRF is to build a neural radiance field from a 5D coordinate space to RGB space.
Specifically, given a ray: i from the camera into the scene, the MLP takes the 5D coordinates (3D spatial locations and 2D view directions) of points along the ray inputs and outputs the corresponding color e and volume density a. Then e and a are combined using numerical volume render- ing to obtain the final pixel color of the ray i. Following our
5.2. 3D shape representation
3D shape representation by Signed Distanced Functions (SDFs) is widely used for testing INR models (Saragadam et al., 2023; Cai et al., 2024; Shi et al., 2024a), due to its
8

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
sampling strategy, We place the 2D view directions last dur- ing flattening and set p as the number of sampling points per ray. That is, each ray is treated as group. For each group, we sample the point with the maximum integral weight. We apply our method to the original NeRF (Mildenhall et al., 2021). The SeeRF-PyTorche codebase (Yen-Chen, 2020) is used, and we follow its default settings for all methods. More details can be found in Appendix
6. Conclusion
In this paper, we propose an effective gradient adjustment strategy to purposefully improve the spectral bias of multi- layer perceptrons (MLPs) for better implicit neural repre- sentations (INRs). We delve into the linear dynamics model of MLPs and theoretically identify that the empirical Neu- ral Tangent Kernel (eNTK) matrix connects spectral bias with the linear dynamics of MLPs. Based on eNTK ma- trix, we propose our inductive gradient adjustment method via inductive generalization of gradient adjustments from sampled data. Both theoretical and empirical analysis are conducted to validate impacts of our method on spectral bias. Further, we validate our method on various real-world vision applications of INRs. Our method can effectively tailor improvements spectral bias and lead to better repre- sentation for common INRs network architectures. We hope this study inspires further work on neural network training dynamics to improve application performance.
Table 3. Average metrics of 5D neural radiance fields by different methods. Per-scene results are provided in Table. 14.
Metrics
NeRI
N
NeRF+BNI
NeRF+IGA
PSNR 1
31.23
31.35
31.37
31.47
SSIM 1
0,953
0.954
0.956
0,955
LPIPS
0.029Akx
0.028
0,050
0.027
7 Use values reported the paper (Cai et al., 2024).
Table 3 lists average metrics of four methods on the down- scaled Blender dataset (Mildenhall et al., 2021). Our IGA method achieves the best results among these methods. The average improvement of our method is up to 0.24dB, which is nearly twice that of FR and BN. Visualization results in our Appendix I show that our IGA enables NeRF to capture more accurate and high-frequency reconstruction results.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
5.4. Time and memory analysis
In this subsection, we analyze the training time and memory cost of our IGA in the aforementioned experiments. For both vanilla gradient descent and IGA, we present the average per-iteration training time and memory cost over all models within each experiment, as summarized in Table 4. We only report the results of adjustments using the complete eNTK matrix (+Full eNTK) for ID simple function approximation, due to its excessive memory cost (exceeding 80GiB) and prohibitive training time in other INR tasks.
A Acknowledgments
This work was supported by National Natural Science Foun- dation of China (No. 62476051) and Sichuan Natural Sci- ence Foundation (No. 2024NSFTD0041..
References
Arora, S., Du, S., Hu, W., Li, Z., and Wang, R. Fine-grained analysis of optimization and generalization for overpa- rameterized two-layer neural networks. In International Conference on Machine Learning, pp. 322-332. PMLR, 2019.
Table 4. Average per-iteration training time and memory cost by different methods. "Vanilla" denotes standard training using con- ventional gradients; "+Full eNTK" denotes training with gradi- ent adjustment based on the complete eNTK matrix K; "+IGA" denotes training with inductive gradient adjustment. The abbre- viations "ID", "2D", "3D", and "5D" are used to represent the experiments on simple functions, 2D images, 3D shapes, and neu- ral radiance fields, respectively.
Bai, J., Liu, G.-R., Gupta, A., Alzubaidi, L., Feng, X.-Q.. and Gu, Y. Physics-informed radial basis network (pirbn): A local approximating neural network for solving nonlin- ear partial differential equations. Computer Methods in Applied Mechanics and Engineering, 415: 16290, 2023.
INR tasks
Training time (s) +Full eNTK
Memory cost (MB) -Full eNTK
Vanilla
+IGA
Vanilla
+IGA
1D
0.028
0.185
0.044
965
5861
1574
2D
0.061
0.088
Bietti, A. and Mairal, J. On the inductive bias of neural tan- gent kernels. Advances in Neural Information Processing Systems, 32, 2019.
3D
0.118
0.169
2291
2468
5D
0.159
0.286
9451
11033
Overall, IGA averages 1.56x the training time and 1.34x the memory of the baseline but achieves greater improve- ments (on average, 2.0x those of prior FR and BN). Com- pared to "+Full eNTK", IGA reduces the training time and memory consumption by at least 4x.
Cai, 7., Zhu, H., Shen, Q. Wang, X., and Cao, X. Batch normalization alleviates the spectral bias in coordinate networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 25160- 25171, 2024.
9

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
Chan, F. and J. Image processing and analysis: vari- ational, PDE, wavelet, and stochastic methods. SIAM, 2005.
Kingma, D. Adam: A method for stochastic optimization. arXiv preprint arXiv: 141.66884 2014.
Klocek, S., Maziarka, L.,, Wołezyk, M., Tabor, J., Nowak, J., and Smieja, M. Hypernetwork functional image represen- tation. In International Conference Artificial Neural Networks, pp. 496-510. Springer, 2019.
Deng, J., Dong, W., Socher, Li, Li, K., , and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Du, S. S., Zhai, X., Poczos, B., and Singh, A. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint /0..:v://0000005:. 2018.
Essakine, A., Cheng, Y., Cheng, C.-W., Zhang, L.., Deng, Z., Zhu, L., Schönlieb, C.-B., and Aviles-Rivero, A. Where we stand with implicit neural representations? technical and performance survey. arXiv preprint arXiv:2411.03688. 2024.
Lee, J., Xiao, L.., Schoenholz, S., Bahri, Y., Novak, R., Sohl- Dickstein, J., and Pennington, J. Wide neural networks of any depth evolve as linear models under gradient descent. Advances neural information processing systems, 32, 2019.
Fathony, R., Sahu, A. K., Willmott, D., and Kolter, J. Z. Mul- tiplicative filter networks. In International Conference on Learning Representations, 2020.
Liang, R., Sun, H., and Vijaykumar, N. Coordx: Accel- erating implicit neural representation with split mlp architecture. arXiv preprint 2rXiv:2201.12425 2022.
Franzen, R. Kodak lossless true color image suite. ht attps: 1999.
Luo, Y. Zhao, X., and Meng, D. Continuous representa- tion methods, theories, and applications: An overview and perspectives, 2025. URL abs hbbs/2505.155
//r0k. us/ aas/graphiess /kooakk// kodak/,
Geifman, A., Barzilai, D., Basri, R., and Galun, M. Con- trolling the inductive bias of wide neural networks by modifying the kernel's spectrum. arXiv preprint arXiv:2307.1453/ 2023.
Martel, J. N., Lindell, D. B., Lin, C. Z., Chan, E. R., Mon- teiro, M., and Wetzstein, G. Acorn: Adaptive coordinate networks for neural scene representation. arXiv preprint 100:22055.0288,, 2021.
He, K., Zhang. X., Ren, S., and Sun, Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., and Ng, R. Nerf: Representing scenes as neural radiance fields for view synthesis. Communica- tions of the ACM, 65(1):99-106, 2021.
Heckel, R. and Soltanolkotabi, M. Compressive sensing with un un-trained neural networks: Gradient descent finds a smooth approximation. In International Conference on Machine Learning, pp. 4149-4158. PMLR, 2020.
Mohamadi, M. A., Bae, W., and Sutherland, D. A fast, well-founded approximation to the empirical neural tan- gent kernel. In International Conference Machine Learning, pp. 25061-25088 PMLR, 2023.
Jacot, A., Gabriel, F., and Hongler, Neural tangent ker- nel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Müller, T., Evans, A., Schied, C., and Keller, A. Instant neural graphics primitives with multiresolution hash encoding. ACM transactions on graphics (TOG), 41(4): 1-15, 2022.
Kazerouni, A., Azad, R., Hosseini, A., Merhof, D., and Bagci, U. Incode: Implicit neural conditioning with prior knowledge embeddings. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1298 1307, 2024.
Novak, R., Sohl-Dickstein, J., and Schoenholz, S. S. Fast finite width neural tangent kernel. In International Con- ference on Machine Learning, pp. 17018-17044, PMLR, 2022.
Kim, J., Lee, Y., Hong. S., and Ok, J. Learning continuous representation of audio for arbitrary scale super resolution. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3703-3707. IEEE, 2022.
Park, D. S., Lee, J., Peng, D., Cao, Y., and Sohl-Dickstein, J. Towards nngp- gngg-guided neural architecture search. arXiv preprint arXiv:201 1.06006, 2020.
10

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
Park, J. J., Florence, P., Straub, J., Newcombe, R., and Love- grove, S. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 165-174, 2019.
Strümpler, Y., Postels, J., Yang, R., Gool, L. V., and Tombari, F. Implicit neural representations for image compression. In European Conference on Computer Vision, pp. 74-91. Springer, 2022.
Takikawa, T., Litalien, J., Yin, K., Kreis, K., Loop, C., Nowrouzezahrai, D., Jacobson, A., McGuire, M., and Fidler, S. Neural geometric level of detail: Real-time rendering with implicit 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 11358-11 11367, 2021.
Rahaman, N., Baratin, A., Arpit, D., Draxler, F. Lin, M., Hamprecht, F., Bengio, Y., and Courville, A. On the spec- tral bias of neural networks. In International conference on machine learning, pp. 5301-5310 PMLR, 2019.
Ramasinghe, S. and Lucey, S. Beyond periodicity: Towards unifying framework for activations in coordinate-mlps In European Conference Computer Vision, pp. 142- 158. Springer, 2022.
Tancik, M., Srinivasan, P., Mildenhall, B., Fwidovich- Keil, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J., and Ng. R. Fourier features let networks learn high fre- quency functions in low dimensional domains. Advances in neural information processing systems, 33:7537-7547, 2020.
Reddi, S. J., Kale, S., and Kumar, S. On the convergence of adam and beyond. arXiv preprint arXiv: 1904:09990009 2019.
Vaswani, A. Attention is all you need. Advances in Neural
Information Processing Systems, 2017.
Ronen, B., Jacobs, D., Kasten, Y., and Kritchman, S. The convergence rate of neural networks for learned functions of different frequencies. Advances in Neural Information Processing Systems, 32, 2019.
Wang. Y., Li, D., and Sun, R. Ntk-sap: Improving neural network pruning by aligning training dynamics. arXiv preprint arXiv:2304. 02200204, 2023.
Saragadam, V., LeJeune, D., Tan, J.. Balakrishnan, G., Veer- araghavan, A., and Baraniuk, R. G. Wire: Wavelet implicit neural representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18507-18516, 2023.
Wang. Z., Simoncelli, E., and Bovik, A. Multiscale struc- tural similarity for image quality assessment. In The Thrity-Seventle Asilomar Conference on Signals, Systems & Computers, 2003, volume 2, pp. 1398-1402 Vol.2, 2003. doi: 10.1 (LLLO///ACSSC.20011222222
Saratchandran, H., Ramasinghe, S., and Lucey, S. From ac- tivation initialization: Scaling insights for optimizing neural fields. In Proceedings of the IEEF/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 413 422, 2024.
Wilson, A. C., Roelofs, R., Stern, M., Srebro, N., and Recht, B. The marginal value of adaptive gradient methods in machine learning. Advances in neural information processing systems, 30, 2017.
Xie, S., Zhu, H., Liu, Z.,, Zhang, Q. Zhou, Y., Cao, X., and Ma, Z. Diner: Disorder-invariant implicit neural representation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 6143-6152, 2023.
Shenouda, J., Zhou, Y., and Nowak, R. D. Relus are suffi- cient for learning implicit neural representations. arXiv preprint arXiv:2406.025299 2024.
Shi, K., Zhou, X., and Gu, S. Improved implicit neural representation with fourier reparameterized training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
Xu, J., Zhao, L.,, Lin, J., Gao, R., Sun, X., and Yang, H. Knas: green neural architecture search. In Inter- national Conference on Machine Learning, pp. 1 11613- 11625. PMLR, 2021.
2024a.
Shi, Y., Chen, Y., Dong, M., Yang. X., Li, D., Wang. Y., Dick, R., Lv, Q. Zhao, Y., Yang, F,, et al. Train faster, perform better: modular adaptive training in over- parameterized models. Advances in Neural Information Processing Systems, 36, 2024b.
Xu, Z. J. Understanding training and generalization in deep learning by fourier analysis. arXiv preprint arXiv: 1000.008..00 008.04495, 2018
Yen-Chen, L. Nerf-pytorch.
baanchenlin//eer-pyyoooo 2020.
Sitzmann, V., Martel, J., Bergman, A., Lindell, D.,, and Wetzstein, G. Implicit neural representations with peri- odic activation functions. Advances in neural information processing systems, 33:7462-7473. 2020.
Yu, Y., Wang. T., and Samworth, J. A useful variant of the davis kahan theorem for statisticians. Biometrika, 102(2):315-35) 2015.

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
Yúce, G., Ortiz-Jiménez, G., Besbinar, B., and Frossard, P. A structured dictionary perspective implicit neural representations. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 19228-19238, 2022.
Zhang. C., Luo, S. T. S., J., Wu, Y-C.,, and Wong, N. Nonparametric teaching of implicit neural representations, 2024. URL https: http::/aaxxiioooo/e
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
Zhang, W., Xie, S., Ren, C., Xie, S., Tang. C., Ge, S., Wang, M., and Wang, Z. Evos: Efficient implicit neural training via evolutionary selector, 2025. URL https: //arxiv. orgg abs/abasaaaa 2412. 10153.
Zhu, H., Liu, Z.. Zhang, Q. Fu, J., Deng, W., Ma, Guo, Y., and Cao, X. Finer++: Building family of suriable-periodi functions for activating implicit neural representation. arXiv preprint arXi::2407.19453 2024.
12

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
Appendix
This appendix is organized as follows. In Sec. A, we provide the formal version and detailed proof of Theorem 3.1 and 3.2 in the main paper. Then, more empirical evidence from Sec. are provided in Sec. B support theoretical results in Sec. 3.3. In Sec. C, the visual illustration of our sampling strategy is provided. In Sec. D, further discussion about our inductive gradient adjustment (IGA) with Adam optimizer referred in Sec. 3.4 is presented. Then, we add experiments of IGA on recent state-of- ofte--of-hheart activations in Sec. E. In Sec. F, the ablation study on sampling strategy is provided. Lastly, we give detailed training settings and per-scene results of our three vision applications in Sec. G, H, respectively.
A. Theoretical Analysis
In this section, we first introduce the detailed framework for theoretical analysis. Then, we provide the formal statements and proof's of Theorem 3.1 and 3.2 of Sec. 3.3.
A.1. Analysis framework
For the sake of simplicity and to focus on the core aspects of the problem, we follow the previous works (Lee al., 2019; Geifman et al., 2023; Arora et al., 2019), and perform theoretical analysis on the basis of the following framework: assuming that the training set (æt, Yi ) is contained in one compact set, a two-layer network fa;; Q) with m neurons is formalized as follows to fit these data:
fx; vm Σ a.o + br);
where the activation function a satisfies that (o()),, ||o'lll supatx (o' (z) - o' (2'11/= - I'1 < xc. The param- eters of f(x;; Q) are randomly initialized with No(,, ) except for biases initialized with NN co), where co = 1 (/ The loss function is EE=((((;; O.) ve) Then, the NTK matrix K and the empirical NTK matrix K is defined as follows:
K Re, JX: Ve,
(10)
K = Ve.f Ve. (x)
(11)
where (e)(X; E RexN; is the number of parameters. We assume that K, K are full rank, indicating that they are positive definite. This assumption generally holds due to the complexity of neural networks. Then we have the following standard orthogonal spectral decomposition: K Ei A,u,n, and K = E A T, which eigenvalues are indexed
in descending order of magnitude. We construct the corresponding transformation matrices as S = Ein A.
(((Ae) d. and
$ E gx().)) H. through set of Lipschitz continuous functions {a)()) From Jacot et al. (2018); Du et al. (2018); Arora et al. (2019); Geifman et al. (2023), we have that:
X;
lim K
K
F ====0.
(12)
M-
A.2. Basic Lemmas
In this section, we show some basic lemmas. Lemma A.1 describes the local properties of f(z, Q). Based on Lemma A.1, we derive Lemma A.2, A.3 and A.4, which characterize training dynamics of f(x, Q) under different adjusted gradients. Lemma A.1. (Modified from Lemma A.l of (Geifman et al., 2023)) For bounded matrices, i.e., I,S, there a K > 0 such that for every C > O, with high probability over random initialization the following holds: YO, 19,6cc E B(O,, Cm - b) at time step t:
R 11 - 6|
[[(vef(X) - vof(X)) S
(13)
Vm
[[Vo((X)Al]r
S
K
(14)
mm.
where can be I, S.
13

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
As discussed in Lee al. (2019); Geifman et al. (2023), the core of Lemma A. is the requirement that the matrix A is bounded. Therefore, Lemma A. clearly holds. Then, to better illustrate our core theoretical analysis, present the following Lemma A.2, A.3 and A.4. Please note that similar version of Lemma A.2 has been proved in (Geifman et al., 2023). However, it focuses on T++11 induced by only S and the details of the eigenvalues between P1+1 and r, are omitted. Therefore, our Lemma A.2, A.3 and A,4 characterize the dynamics in one iteration from ry to T++1 and eigenvalues induced by NTK-based and ANTK-basee adjustments and vanilla gradient, respectively.
Lemma A.2. The parameters are updated by: D - n..-vo..fle f Sr)Srr with Lj (min(g: (A)) + max (g. (A))) For > O, there always exists M > 0, when > M, such that with high probability over the random initialization, have that:
rr+11l2 1
Σ (1 ng.(A))) r2) E((), E(),,
(15)
where r, = (a 0i) - 2 and [E(() <
Proof. By the mean value theorem with respect to parameters θ we can have that:
r-+1 = T1+1 Tr + r, =
- 61) +r, = (o. = (I k + ( K)Sr, + (Vo,, - Vo:j(X)) Vo.f(X)Sr,
+
where O2 lies on the line segment 0,0 We define that E'(1) A B. For || we have that:
||||| S ( K)Sr, || S 1 K
|||||||||/|l
< nLRol K K| (2) e
2
where (1) follows Geifman et al. (2023) that S and r, are bounded by constants and Ro, respectively; (2) follows the equation 12.
For |B]], we have that:
1/122 Ro m
[B| vll (e)=(x,66)ll/Veee/(()))
|0,
06,12
7 Ro 110,
Or+1ll 1|2 S n Ro
11²x0
ee, ( (X,
Sr) (11)
33322
m
m
where (1) follows the Lemma A.1. Therefore, we have that r++1 EN ng. (A:)) re)o; + E((t;; there always
exits M1 > O, such that m Mi. [I)) < || B|| Further, have that:
N
|| 113 Σ (1 n..) r.) I (+) + (E) Σ (A:)) re)
N
= Σ (1 (o)
+ 2g (1)
(r+ + E(())
{())
For E(t), as T141 is bounded by Ro, there always exists M > M1, such that: [E(t) < 311' (1)1l² + 2R ('))) <.
Lemma A.3. The parameters are updated by: 0 = 0, V Sr, with 17 (maz(g (X)) + min(g (a))) -!. For > O, there always exists M > O, when m > M, such that with high probability over the random inirialization, we have that:
N
P 1
Σ (1 - no. n9 (5.)) Tr. E())
(16)
where r, = (f( Q.) - N and [((t) <c
14

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
Proof. By the mean value theorem with respect to parameters O we can have that:
T1+1 Tt+1 T2 + Ve, ( (0.+1 θ2 r. Ve, (0 +re 1 ( (XV, - Ver
(X)S
(t)
For (1), we follow the same technique in the proof of A.2. Then we can have that:
x² 1 -1 0/1l2 1
(1)I S v/lVe,
Ve,
||
nA² Ro
R2
10, 0 1|
m
m3322
Therefore, we have that re+1
Σ Ei n994 (2:)) rz)ō, + (t): there always exits M₁ > 0, such that m > M2.
(()( 6. Further, we have that:
N
N
r
11
(x)
(0) + + (t) T Σ (1 (g)() T
N (1 - (AA))
(t) + (t) (r, = (1))
{(t)
For E(1), as T is bounded by Ro, there always exists M > M2, such that:
{(t) S (+)l + 2R (1))l <c
Lemma A.4. The parameters are updated by vanilla gradient descent. For > O, there always exists M > 0, when m > M, such that with high probability over the random initialization, we have that:
N
f(x,, 0+41) - f(x;, O₂) (o,,, 0,)
Σ
(a,,((,,,))000 - -1)) Et())
(17)
where (g((t) <c
Proof. By the mean value theorem with respect to parameters O we can have that:
f(a,, O++1) = f 01) (0++1 - +.) + f(x;, O2)
Verf(e; 1,) I-n Σ
Ve,f(a, O,, 0.) u;)] f(x,, O,)
5=1
N
Ve, 0,)
Ve, (,) 02) v,) + f(x;, O1) (tt).
For (()), we have that:
N = = [V]; 02) Ve,
O,) - -y))
< Ve; 0/) Ve,f(x, 0,)1121 Σ Ve, 0i)(fe, 0,) 9;)l,a
Ve; 0%) Ve.f(z, θ Ve. (x. 0,)11211re112
<a)) x² 1 O1l12 (2) 3 3 R2
m
3/2
15

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
where (1); (2) follow the Lemma A.1 and r, is bounded by Rg: Therefore, there exists M 0. such that M, we have that [((()] <
Lemma A.2 and lemma A.3 show the training dynamies of the residual r; by NTK-based and eNTK-bbsed gradient adjustments. Equation 15 shows that {9.(A,)]i= control the decay rates of different frequency components of the residual T, as eigenvectors of K are the spherical harmonics (Ronen et al., 2019). Lemma A.4 show the training dynamics of residual at data point æ; by vanilla gradient descent. Despite eNTK-based and NTK-based gradient adjustments have the similar form, they differ in the cigenvector directions U, which leads to errors. We give a theoretical analysis to this error in Theorem 3.1.
A.3. Proof of Theorem 3.1
Theorem A.5. (formal version of Theorem 3.1 from the paper) The following standard orthogonal spectral decomposition
Aw and K = N > X satisfying that v, ù; > 0 for E [N]. We denoted as max; {min (A, A1+1, A4+1 {++2]]= G. Let [== be a set of Lipschitz continuous functions, with the Supremum of their Lipschitz constants denoted by k with y < min (m + min(g:(A)) (A))) (max (X)) mi)(g:(A))) J. for > 0, there
exists: K = Dimi i=
always exists M > 0, such that m > foror i [N]. we have that (ge(),) (x()a)] E1, 1/v, - ui E2i furthermore,
have that:
1(1 - n9. (g. (A.)) T rr)² (1 79. (A:)) r.) ) < <3, || - -i+1ll < E4,
where E1 = ke; E2 = (23/2 Ey & (Kiee + (w + 1)e² + (1² + Gv)e); Es = =eN(( LouRo v. k. Rg are constants.
+ n + +2) + 2e;
G
Proof. We define that the perturbation of K as E K- K. As previously discussed, we have that lim,, 000 K K F 0. Therefore, for any 0 there always exists Mo. such that m Mo. we have that || symmetric matrices, E is still real symmetric matrix. This indicates that:
E. As K, K are real
N
Ell2 - - (x)2 S
(x)= tracee ET ETE) = |
where X is the largest eigenvalue of E.
By Wely's inequality, when m > Mo. for i = 1,... N, we have that [Λ, - A² S |E < As g:A) is Lipschitz continuous function with the Lipschitz constant k, for i E [N], we have that:
(g:(A:) - (Ro) S k|A; 3el < ke.
Therefore, for any 0 < e, there always exists Mo. such that (A()) - g(Re)) < ke. By the Corollary 3. of Yu et al. (2015) that a variant of Davis- Kahan theorem, we have that:
23/2 E
23/2
||:; - 16:-vs11= S
i [N].
G
G
Then, we have that:
[(1 no: (A: (2))) - (11-n9 (g)))) 21
(1
(T)))))) =I(1 (19((a))) r2) ng. ng.( - (1-))=11--08__ll1 (Ar)) (1 (1)- T2
(1
no.
(1 - (1.-(,)))))) r. + 0, re)( r. 8, r.) + (nn(a((A)) - ((()) r1)2
For A, we have that:
A < - 11 (Ax) - IIoTT + 5. o o. 1llllrel1
1 (Ax)l (2|10T1 || + -| 1o 0x]12l|1r113 < [G v +
16

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
As S' is fixed, have that constant to bound ||ol]| for i [N] Therefore, for 0, there always exists M, such that Mo, have that A +
For B, we have that:
B < nlg(())) (c()i)( rz)² chn(llo) - o. I1lrell [ellll cknRo 23/2
G c c+
Further, we have that:
8Ro
(1 1g. (A)) (x.)))
Tr) (.)T))))))))))))))))))))))))))))
(kne + (c + + (v²
Go)c).
G²
Following these analytical approaches, is trivial have the following result:
16
(11 - (A.)))) r,)u, - (1 (g "T) < el
+ nyu² Ro) + 2.
G
Combing this result with the Lemma A.2 and A.3, there always exists M max MI, M2). have that:
|re+1 Frtill Σ
(U,
(1=))(A)) -
+ (OII
Ro
< <EN((
+ nnku² Ro) + + +1 1()
G
< eN(( 16oRo
+ nnu Ro) + +)+++
G
A.4. Proof of Theorem 3.2
Theorem A.6. (formal version of Theorem 3.2 from the paper) ( partitioned into n groups, where each group X; = {0) {,,,,...,"") and N npp For each group, one data point is sampled denoted as x² and these n data points form Xe = {x 1= & N. There exists > 0, for j I,..., n and any 1 S 21 S such that - c. Then, for any Dy, assumed that x; belongs to
I(f(=l,, 0,) 1 v (f(x), 0.)
33o such that:
(K + n2/
qu R3
AAf 02) ve.f(=",,)) Σ (o, r')ll < <e(
+
m
3322
where A,(a; = (r++1 rt); denotes the dynamics of s(x;, 0) at time step 1; Ro t are constants; fe. θ) T Te.f(x) θ) is the entry in (i.i.j). Proof. From the equation 17 of lemma A.4, we consider the following error:
N
Vo,f O.)
T
Σ
0.) vi) ve.f(x't T Σ (,)(((()))))))))))) v'1l
3=1
Following previous analysis technique, this error can be bounded by A + B. For A, we have that:
N
A
(aoi T)
Ve, (a, (f) Q.) y;)
1=1
Ve, (0r ..)IlI
Ve, Ja, O,) (f) (.) -
xRo (c s
mm
17

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
For B, we have that:
B Ve, ,0,)
O,) -1) -p Σ
Ve, (,))))))(()))) - wl))
j=1
- (o.
Ve, (.)((a')))((',,, 00) - y8y)
Ve, saa O.) ,,, 0x)-1()1 (18)
j=1
< Vo.f (6
f(f(("), O1) - -801 + Ve. 9111)
=
$ / θt) 11 Ro + k S e( n//2 Rox k²
vm
m
We define that c'= + B. By the lemma A.4, we have that:
f(z,, O441) f(x,, O2) Ve.f (0) O
fe.f (()(==)))())))))-))))))))))))))) y')l + + (t))
where 6'1 S c(
+ x²2) RoR
S (
): :E(((0] v
n Rd 3/3
Based on this, the following vector form can
vm
vm
m
v/m
be derived:
r rit = pk,rt E + Ett)
(19)
where r/+1 (f(x), v.) rf (f(x',))) 0,) y²)
Further, from the derived expression equation 18, we observe that for different r/41 we can replace rf with rî as the pairwise cancellation of some error terms.
B. Empirical analysis on simple function approximation
In this section, we first visualize more results of the experiment 1 in our paper. Subsequently, we illustrate impacts of IGA on spectral bias of ReLU and SIREN with varying numbers of balanced eigenvalues under varying group size p from the experiment 2. As can be seen in Fig. 9, the same trend as our observations in our paper, with the width increases, the differences on MSE curves and spectrum between gradient adjustments by S, S and S. gradually diminish when end = 12, 14. This is consistent with our Theorem 3.1 and the analysis of our Theorem 3.2. As shown in Fig. 10, 11 and 12, impacts on spectral bias are gradually amplified by increasing the number of balanced eigenvalues of S., which is also consistent with the observations in our paper. The above empirical results corroborate our thcoretical results and show that our IGA method provides chance to amplify impacts on spectral bias by increasing the number of balanced eigenvalues under varying group size p.
C. Visual illustration for the sampling strategy of IGA
In this section, we aim to provide a visual illustration of the sampling strategy introduced in Sec. 3.4. As introduced in the main text, for 1D and 2D inputs, we sample data with the largest residuals in each non-overlapping interval and patch, respectively. For higher-dimennional settings like d-dimensional input coordinates (d 2 3), represented as a tensor of shape (r1.....,77, n2,, d), we flatten the first d dimensions into a 2D tensor X' of shape (N, d). where N nı X ... x na (which evident), and then group it along the first dimension of X'. Then, the input with the largest residuals in each group are selected to form the sampled data X.. We take d = 3 as example to further illustrate sampling strategy. Considering a 3D "Thai" statue, we first sample data over 512 x 512 x 512 volume grid with each voxel within the "Thai" volume assigned 1, and voxel outside the "Thai" volume assigned 0 following Saragadam et al. (2023); Shi et al. (2024a). The 3D coordinates (I),..., Fn, x {vi... }ou? x { Zng? } of voxels defined by the "Thai" volume are the inputs of INRs. We denote these coordnates as X. Generally, X is represented as tensor of shape (n1, n2, ng. 3) and n1, n2; ng depends on the resolution of the aforementioned volume grid and the object volume, demonstrated in Fig. 6,
18

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
visualize X cube with dimensions 7 T22 x nig, composed of N cubical elements. Each cubical element contains its corresponding coordinate. For example, if we define the лi -i-dimmnnooon the x-axis, the ny-dimension as the y-axis, and the vaa-dimension as the z-axis (with positive directions indicated in Fig. 6), the red-marked cubical element contains the corresponding coordinate (Ix,, V1, 2ng). Then, we use the flatten operation to unfold the cube sequentially along the dimensions, obtaining X'. Note that the flatten operation preserves adjacency. Therefore, we directly group X' and sample one element from each group to form Xe, marked green in Fig. 6.
This sampling strategy simple yet effective. The adjacency during grouping helps obtain a smaller consistency bound e, thereby reducing the estimate error. Meanwhile, the flatten operation maintains adjacency while reducing computational overhead. Our extensive experimental results validate the effectiveness of this strategy- Note that there might be more sophisticated strategies via clustering or segmentation, which diverge from the primary focus of our work. We leave this for future research. The ablation study about hyperparameters n and of the strategy are provided in Sec. F.
x: (fc. Kin x X/y Pep
Flatten
X
X,:
Volume grid
Input coordinates X of shape (2..3)))).33))
Tensor shape (n, (m.
Sampled data X, shape
Figure 6. Visual illustration for the sampling strategy of IGA. The detailed description of the above diagram is in Sec, C.
D. Discussion about IGA in Adam
In the current field of deep learning, Adam (Kingma, 2014) as a variant of vanilla gradient descent has been proven to be one of the most effective methods empirically (Deng et al., 2009; He et al., 2016; Vaswani, 2017) and widely adopted by current INR models (Sitzmann et al., 2020; Mildenhall al., 2021; Saragadam et al., 2023; Zhu et al., 2024; Shi et al., 2024a). However, due to the introduction of adaptive learning rates and momentum, the characterization of training dynamics like Eq. 17 induced by Adam becomes exceptionally challenging and remains an open problem. Intuitively, momentum is the linear combination of previous gradients, which implies that momentum has the similar direction with current adjusted gradients. This suggests that momentum is inherently compatible with our IGA. Therefore, we adjust gradients first by IGA and then update the momentum and learning rates by traditional Adam. Adaptive learning rates typically result in larger update steps for parameters (Kingma, 2014; Wilson et al., 2017; Reddi et al., 2019). For better convergence, we utilize dend+1 to balance cigenvalues equation 5. Our extensive empirical studies in INRs (Table 8, 12, 14 in our paper) demonstrate the compatibility of IGA with the Adam optimizer.
E. IGA on more activations
In the main text, we have evaluated our IGA on ReLU, PE (Tancik et al., 2020) and SIREN (Sitzmann et al., 2020). In this section, we apply our IGA to the most recent, meticulously crafted state-of-the-al activations with periodic variable, i.e., Gauss (Ramasinghe & Lucey, 2022), WIRE (Saragadam et al., 2023) and FINER (Zhu et al., 2024). For these activations, we follow their initial learning rates by their official guidelines. The same training strategy adopted for these experiments. We set end 20 for all these activations, which are consistent with PE and SIREN. These experiments are conducted on the first 8 images of Kodak 24. We report the average PSNR in the Table. 5. Similar results in the main text, our IGA consistently improves the approximation accuracy for the evaluated activations.
F. Ablation study
As discussed in Sec. 3.4, we conduct ablation studies about the effect of sampling data and the group size p. All experiments are conducted on the first three images of Kodak 24 dataset, and the average PSNR values are reported for reference.
19

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
Table Peak signal to-noise ratio (PSNR 1) of color image approximation results on more sfate-of-the activations with periodic variable. "Gauss" and "WIRE" denote that MLPs of Gaussian and Gabor Wavelet activate function by periodic variable, respectively. The detailed settings can be found in Sec. 5.1 and Sec. E.
Average
Gauss
WIRE
FINER
Metric
Vanilla
+IGA
Vanilla
+IGA
Vanilla
+IGA
PSNR f
33.79
34.45
30.37
31.49
35.12
35,55
F.1. The effect of Sampling data
In this subsection, we examine sampling data selection. We randomly sample one data from each group in each iteration, referring to the resulting MLPs as randomly sampling (RI). Additionally, we implement variant where random sampling occurs only at the beginning. denoting these MLPs randomly sampling initially (RSI). We also introduce sampling based on the largest residual, termed SLR. Table 6 presents the average PSNR results while maintaining the same settings as in Experiment 5.1, demonstrating that our method significantly improves performance across various sampling strategies, with SLR yielding the best average performance. Consequently, we adopt SLR for sampling strategy.
Table Ablation experiments on the effect of sampling points. SLR denotes the sampling based on the largest residual in each iteration. RSI denotes the random sampling initially. RI denotes the random sampling in each iteration.
Method
ReI.U+IGA
PE+IGA
SIREN+IGA
Average
SLR
25.44
34.09
34.85
31.46
RSI
25.52
33.77
33.76
31.02
RI
25.19
34.05
33.69
30.98
Vanilla
24.09
30.46
33,70
29.42
F22 Group size and Training time
The size of the sampling group influences the level of data similarity within the group and hardware requirements. Intuitively, a smaller p reduces errors in estimating training dynamics and allows for more accurate tailored impacts on spectral bias, leading to improved representation performance. However, this leads to a larger matrix K, which results in longer run times and greater memory usage.
To further explore the effect of various p. follow the MLP architecture in Experiment 5.1. All training settings remained. We vary p from 2 to 128 Larger sizes correspond to coarser estimates of the dynamics. We report the average training time of each iteration.
As shown in Table 7, our method under all sampling sizes has the significant improvement comparing to the baseline models. Particularly, when the size p is 128² that means that the size of K, is only 24a, it still improves the representation performance. That means that relatively low-precision estimates are sufficient to achieve representation improvements. Further, as p decreases, representation accuracy improves; however, the extent of this improvement gradually diminishes with smaller group size p. Specifically, reducing the group size p from 64 to 322 nearly doubles the average increment Ang. while the increment becomes negligible when reducing p from 32 to 16 while the time required for p 16 increases fivefold. This indicates that, in practical applications, there is no need to consider more precise sampling intervals; a relatively coarse partition suffices to achieve substantial improvements while maintaining efficiency.
G. Experimental details and per-scene results of 2D color image approximation
In this section, we first include additional experimental details that omitted in the main text due to space constraints. Then we provide the detailed per-scene results for all four metrics and more visualization results.
As illustrated in the main text, we follow the learning rate schedule of Shi et al. (2024a), which maintains a fixed rate for the first 3K iterations and then reduces it by 0.1 for another 7K iterations. For IGA, we set initial learning rates as 5e 3 for ReLU activation and 1e - 3 for Sine activation. For all baseline models, we set initial learning rates as le due to poor performance observed with 5e Full-batch training is adopted.
20

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
Table Ablation experiments on sampling group size p. Awg denotes the average increment across three model architectures. "Time
(ms)" denotes the average time of each iteration.
ReLU+IGA
PE+IGA
SIREN+IGA
Avg
Time (ms)
16a
25.49
34.25
34.87
+2.12
256.51
32
25.44
34.09
34.85
+2.04
56.33
64
25.19
32.54
34.10
+1.19
39.76
128
25.06
32.08
33.75
+0.88
39.61
Vanilla
24.09
30.46
33.70
30.00
We report four metrics to comprehensively evaluate the representation performance, i.e., peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), multi-scale structural similarity index measure (MS-SSIM) and learned perceptual image patch similarity (LPIPS). in Table 8-11. Specifically, PSNR is used to measure average representation accuracy based on MSE, while SSIM, MS-SSIM and LPIPS are more sensitive to the representation of high-frequency information. As shown in Table 8-1 11 all methods improve representation quality across the four metrics. However, the improvements from FR and BN are inconsistent. Although they enhance frequency-sensitive metrics to some extent, they degrade average representation accuracy and even lead to negative effects in certain cases. contrast, our IGA method achieves consistent improvements across almost all scenarios and metrics, demonstrating the superiority of tailor impacts on spectral bias induced by our IGA. We further visualize more representation results Fig. 7, which is consistent with our observation of Fig. that mprovements of IGA are uniformly distributed across most frequency bands.
ReLU+ER
ReLU+BN
ReLU+IGA
PE
PE+FR (32.14dB)
PE-BN (31.31dB)
PE+IGA (35.07dB)
SIREN
SIREN-FR SIREN-IGA
(24.70dB)
(25.3ddB)
(25.49dB) (26.17dB) (31.37dB)
(35.10dB) (34.87dB) (35.80dB)
Figure 7. Visual examples of 2D color image approximation results by different training dynamics methods Kodak 4. Enlarged views of the regions labeled by red boxes are provided. The residuals of these regions in the Fourier domain are visualized through heatmaps in the top line. The increase in error corresponds to the transition colors in the heatmaps from blue red. ReLU+IGA denotes that MLPs with ReLU activations are optimized by our IGA method. Details can be found in Sec. 5.1 and Sec. G.
H. Experimental details and per-scene results of 3D shape representation
In this section, we firstly introduce the detailed training strategy of the 3D shape representation experiment. Then we provide the intersection over union (IOU) and Chamfer distance metric values of each 3D shape and give more visualization.
Following previous works (Saragadam al., 2023; Shi et al., 2024a; Cai et al., 2024), train all models for 200 epochs with learning rate decay exponentially to 0.1 of initial rates. We set the initial learning rate 2e 3 for ReLU and PE; For SIREN, we set the initial learning rate as 5e - 4. FR, BN and our IGA method are adopted the same learning rate with baseline models. We typically set end for ReLU and PE, and end = 14 for SIREN. The per-scene results are reported Table and 13. It can be observed that our IGA method achieves improvements IOU metrics across all objects, and achieves the best performance in average metrics of all objects. We visualize the Lucy object in Fig.
Ground (0.9467)
(0.9465)
(0.9445)
ReIAI-IGA
(0.9571)
(0.9920)
(0.9945)
(0.9907)
(0.9961)
(0.9862)
SIREN-FR (0.9785)
SIREN-IGA
(0.9868)
Figure Visualization of 3D shape representation results and IOU values on the Lucy, ReLU+IGA denotes that the MLPs with ReLU activations are optimized by our IGA method. Details can be found in Sec. 5.2 and Sec. H.
21

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
I. Experimental details and per-scene results of Learning Neural Radiance Fields
In this section, we firstly introduce some training details. Then we provide metrics of each scenario and give more visualization. As previously discussed in Experiment 5.3, we apply our method on the original NeRF model (Mildenhall al., 2021).
Specifically, given a ray i from the camera into the scene, the MLP takes the 5D coordinates (3D spatial locations and 2D view directions) of N points along the ray as inputs and outputs the corresponding color and volume density a. Then c and a are combined using numerical volume rendering to obtain the final pixel color of the ray i. We set each ray as one group. For each group, we sample the point with the maximum integral weight. The original NeRF adopts two models: "coarse" and "fine" models. We adjust the gradients of the "fine" model as it captures more high-frequency details and remain gradients of the "coarse" model.
The "NeRF-PyTorch" codebase (Yen-Chen 2020) is used and we follow its default settings. Besides, we also compare to previous training dynamics methods, i.c., FR and BN. Their hyperparameters follow their publicly available codes. Generally, we set the end- = 25. For more complex scenes, such as the branches and leaves of the ficus, we set the end = 30 to achieve better learning of high frequencies. We report per-scene results in Table 14 and visualize several scenes in Fig. 13.
Two-layer ML.Ps 1024)
Wit22a 9/24
2-3024 5:2-5.10
Twodkyer MLPs (Width 1024)
LIGA 9CD 522
Two-dayer MLPPs (Wtdth-1024)
W M. 9/2-
WX2 BCD
5:0 5o
5015--12
54D-5-3
500-
244
2
20:024 5.0-1.1
Sparte bash
1o
Epects
Two-taye MLPPs (W4.1
W-2005-500-
Twoslsyer MLPs (Width:2048))
W/2555 SrDt
MI 51D
Two-layer MLPs (W1,0h-2048)
1.2
15:
1-
5334
5000
5524
ga..
950 5
520 4,10
18:33
5000
5004,02
gas.
540 14
5024
W 2046
W 193338 2004-000-3
2:13
2..
bet0
Two loyol ML.Ps, (Width-4006)
Two-layer MLPs (Width-4c96)
Two-layer MLPPs (Width=6699)
1.8-
4300--10
530 d,10
2.4
Byoots b
13
10 Spects m117
Epocis 04103
Tho-layse Mi.PPs (Width-011
wa11e son
Was30 $00
Two-layer MLDss (195th-8192)
Watle sab.
Vi a19o 50D
Two-layser (Width-81992
a1 sab.
Wa114 $60
1.2
15
5304
5004
Zax-
s20 5,10
500-5,33
244
Epecio 1
Figure 9. Evolution of approximation error with training iterations on time domain and Fourier domain. The shaded area indicates training fluctuations, represented by twice the standard deviation. Line plots visualize the MSE loss curves of MLPs. Hcatmaps show the relative error Au on four frequency bands. SGD-S.- 10 denotes that the MLP with ReLU optimized using gradients adjusted by S. with end 10 and group size 8. Please zoom in for better review.
22

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
ReLLU-GGA-1-Balance-
ReLU-IGA-I-Balance-
ReLLC-GGA-1-Bllacee-5
ReIUUIGA-1-Ballanee
1.0
1.0
1.0
1.0
10
0.8 _
10
0.8
10
0.8
10
0.8
Frequenc
15-
0.6
8 15
0.6
15
0.6
Freque e
15
0.6
0.4
20
0.4
Freque
0.4
20
0.4
25-
0.2
Fregu
25
0.2
0.2
25
0.2
30
30
30
30
2
4
6
8
10
4
8
10
2
4
6
8
10
2
4
6
8
10
Epochs
(x10
Epochs (x10²)
Epochs (x10²)
Epochs (x10*)
ReLU-IGAA4-Balancee
ReLU-IGA-4-Balanee
ReLU--IAA-4Balancee
1.0
1.0
1.0
1.0
5-
5
5-
0.8
0.8
0.8
10
0.8
Erequen c
0.6
0.6
15
0.6
115
15
0.6
20
0.4
20
0.4
20
0.4
Frequ
20
0.4
25
25
25
25
0.2
0.2
0.2
0.2
4
8
Epochs (x10*)
Epochs (x10²)
Epochs (x10²)
Epochs (x10²)
ReLU-IGA-8-Balann
R
9-a-IGG--BBBlance-66
BILLU-IGA---Balancee
1.0
1.0
1.0
1.0
10
0.8
10
0.8
10
0.8
10
0.8
Frequency en
15
0.6
Freque eneguen
15
0.6
0
15
0.6
15
0.6
0.4
20
0.4
Freque
0.4
Fredhe
20
0.4
25
25
0.2
0.2
0.2
0.2
30
30
30
30
2
4
6
8
10
4
8
10
2
6
8
10
4
6
8
10
Epochs
(x10²)
Epochs (x10²)
Epochs (x10²)
Epochs (x10²)
SIREN IGA Balance
SIREN IGA 1 Balance 4
SIREN IGA Balance
6
Frequency (2))
10
1.0
10
10
1.0
2
10
1.0
20
0.8
20
0.8
20
0.8
20
0.8
30
0.6
30
0.6
30
0.6
30
0.6
0.4
Freques
40
0.4
0.4
40
40
0.4
50
0.2
50
0.2
50
0.2
50
0.2
60.
60
60
60
6
10
4
8
10
2
6
10
Epochs (x103)
Epochs (x10²)
Epochs (x10²)
Epochs (x10*)
BEELN-IG---BBllance
SIREN IGA 4 Balance 4
SIREN IGA 4 Balance 6
SIREN-IGAA4BBllncce
Frequency [Hz]
10
1.0
10
10
1.0
10
1.0
20
20
0.8
0.8
20
0.8
0.6
0.6
0
0.6
8
30
0.6
50
40
0.2
0.4
Freque
50
40
0.4
0.2
Frequep
40
50
0.4
0.2
Freque
40
50
0.2
0.4
60
60
60
10
4
10
Epochs (x10a)
Epochs (x103)
Epochs (x10²)
Epochs (x10²)
BUREN-IGA---Balancee
SIREN-IGA-8 Balance--
FIREN-IGA---Balanne--
SIREN-IGA-8 8-AGG----Balane---
1.0
1.0
1.0
1.0
10
0.8
20
0.8
20
0.8
20
0.8
Frequency
30
0.6 30
0.6
30
0.6
30
0.6
40
0.4
Preque
40
0.4
Frequed
0.4
0.4
0.2
0.2
50
0.2
50
0.2
60
60
60
60
2
4
6
10
4
8
10
2
6
8
10
4
8
10
Epochs (x10²)
Epochs (x10²)
Epochs (x10²)
Epochs (x10²)
Figure 10. Progressively amplified impacts spectral bias of ReLU and SIREN by increasing the number of balanced eigenvalues of S. when the group size p ReLUU denotes that MI. with ReLU optimized using vanilla gradients; ReL.U-IGA-8- 8-alance- denotes that the MI. with ReLU optimized using gradients adjusted by S_ with end 1 (for Adam, there are two balanced eigenvalues) and group size p is 8. Details of experimental setting can found in Sec. 4. Please zoom in for better review.
23

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
ReLUU by balancing 2 eigenvalues
ReLU by bolancing 4 eigenvalues
ReLU by bolancing 6 eigenvalues
ReLU by balancing eigenvalues
atak 5Hz
IGA
1.0
IGA
1.0-
IGA
0.8
1GA-4
0.8
IGA-4
0.8
IGA-5
54
0.8
IGA-
IGA
IGA
IGA
IGA 8
Baseline
Baseline
Baseline
0.4
Dr at
0.4
0.4
0.4
0.2-
0.2
0.2
0.2-
0.0
Epochs (K10=)
Epochs (X10=)
8
Epochs (x10=)
Epochs (x10a)
ReI. by halancing 2 eigenvalues
ReL. by balancing 4 eigenvalues
ReL. by balancing 6 eigenvalues
Rel.2 by balancing eigenvalues
10Hz
1.0-
JGA11
1.0-
JGA-1
1.0-
JGA-1
10H4
1.0-
I6N-1
0.8
0.8
0.8
0.8
IGA-
8
JGA-U
IGA-U
IGA-U
Raseline
Roseline
Roseline
0.6-
Roseline
0.4
0.4
0.4
0.4
0.0
Epochs (x10=)
0.0-
Epochs (x10=)
0.0%
Epochs (x10=)
0.0-
6
Epochs (x10=)
10
ReLU by balancing eigenvalues
ReLU by balancing eigenvalues
ReLU by balancing eigenvalues
ReLU by balancing eigenvalues
15Hz
1.0
1
1.0-
16A1
1.0-
16A1
15t
1.0-
IGA
ICA 4
1GA-
4GA4
0.8
IGA 4
JGA
JGA
IGN
0.5
Baseline
0.6
Baseline
0.6
Baseline
0.6
Baseline
0.4-
0.2
0.2
0.2
0.2
0.0
EEpocbs (x10²)
10
0.0
Epochs (x10>)
10
0.0
Epochs (x10=)
10
0.0-
Epochs (x10=) "
10
ReLU by balancing 2 eigenvalues
ReLU balancing 4 eigenvalues
ReLU balancing 6 eigenvalues
ReLU by balancing elgenvalues
dx at k= 20Hz
at k 20Hz
atak kakk 20Hz
at = 20Hz
0.8
0.8
0.8
0.8
0.6
0.6-
0.6-
0.6-
[GA
[GA 1 IGA-
I3A
IGA
0.4
[GA-4
0.4
0.4
IGA-1
0.4
IGA-4
0.2
[GA
0.2 EGAB
0.2
F3AB
0.2
IGA
Beseline
Beseline
Beseline
Baseline
Epocbs (x103)
Epochs (x103)
0.0-
Epochs (>103)
0.0-
Epochs (x10*)
10
ReLU by balancing 2 elgenvalues
ReLU by balancing 4 eigenvalues
ReLU by balancing 6 eigenvalues
ReLU by balancing eigenvalues
aaat 25Hz
1.0
1.0
0.8
0.8-
0.8-
0.8-
0.4
IGa.4
kt 0.4-
k
0.4-
0.4-
IGA4
KGA-1
IGA
KGA
KGA 8 Beanline
8 0.2
Basaline
Reanline
Baanline
Epachs (x10*)
8
Epachs (x10 8
Epochs (x10)) 8
0.0
Epochs (x10')
10
ReLU by balancing eigenvalues
ReLU by balancing eigenvalues
ReLU by balancing eigenvalues
ReLU by balancing eigenvalues
30Hz
1.0
1.0
1.0
1.0
0.6
0.6-
0.6-
0.6-
[GA-1
[GA-1
KGt-1
IGA-1
0.4
IGA-
0.4
IGA4
0.4
IGA4
0.4
IGA-
[GA-8
0.2
(GA-9
0.2-
1Gh-8
0.2
1G4-8
Baseline
Baseline
Baseline
Baseline
0.0
Epachs (x10==) t0
0.0
Epachs (x10==) 10
0.0
Epachs (x10=)) &
10
0.0
Epachs (X10*)
#
10
Figure/ 11. Comparison of group sizes with varying balanced eigenvalues on Relative Error As at 5, 10, 15, 20, 25, 30Hz of ReLU. I denotes that the group size p is 1, i.e. K -based gradient adjustment; IGA-4 denotes that the group size p is 4 for IGA: Baseline denotes MLPs with ReLU activations optimized by vanilla gradients. The shaded area indicates training fluctuations, represented by twice the standard deviation. Details of experimental setring can be found Sec. 4. Please zoom in for better review.
24

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
SIREN by balancing 2 eigenvalues
SIREN by balancing 4 eigenvalues
STREN by balancing eigenvalues
SIREN by balancing 8 eigenvalues
Ax at.kk = 10Hz
4u k= 10Hz
1.0
ICA
Au a= 1011z
IGA
at 20Hz Au at = 10Hz
1.0
0.8-
JGA-1
0.8
0.8-
KGA-4
0.8-
JGA-4
1GA-8
IG4-5
[GA-8
1GA-8
0.6
Baseline
=
0.8
Baseline
0.6
Basellow
0.6
Baselline
04
0.4
0.2
0.2
0.2
0.2
0.0
Epochs (x10²)
6
10
0.0
0
Epochs (x10²)
10
0.0
Epochs (x10p) 6
10
0.0
Epochs (x10²)
6
10
SIREN by balancing 2 eigenvalues
SIREN by balancing 4 eigenvalues
SIREN by balancing eigenvalues
SIREN by balancing 8 eigenvalues
k 20Hz
1GA-1
at = 20Hz
1.0
1GA-1
at k 2011z
1.0
1GA-
0.0
1GA-4
0.0
IGA-
[GA-4
0.8
1GA-4
ICA-8
IGA-8
IGA-8
1GA-8
0.6
Baseline
0.6
Paseline
0.6
Baselino
0.6
Baseline
0.4
0.4
0.4
0.2
0.2
0.2
0.0
Epochs (x10²)
6
8
10
0.0
0
Epochs (x10*)
10
0.0
Epochs (x10²))
6
8
10
0.0
Epochs (x10*)
8
10
SIREN by balancing 2 eigenvalues
SIREN by balancing 4 eigonvalues
SIREN by balancing eigenvalues
SIREN by balancing 8 eigenvalues
= 30Hz.
1.0-
ICA-1
4x at = 30Hz
1.0
IGA-1
4 k = 3011z
1.0-
[GA-1
30Hr
1..0-
IGA-1
0.8
1AA4
0.8
ICA-4
0.8
IGA-4
0.8
1GA-4
IGA 8
IGA
IGA 8
IGA 8
0.6
Daseline
0.6
Daseline
Baselize
0.6-
Beseline
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0-
10
Epachs (x10=)
Epochs (x10²)
Epochs (x1o²)
Epachs (x10*)
SIREN by balancing 2 eigenvalues
SIREN by bulancing 4 eigenvalues
SIREN by balancing eigenvalues
SIREN by balancing 8 eigenvalues
kt= = 40Hz
ICA-1
at k = 40Hz
1.0
ICA-1
Ae k= 40112
IGA-1
A at k 40Hz
1.0
IGA-1
0.8
JGA 4
0.8
JGA 4
0.a-
IGA 4
0.8-
JGA 4
1GA-8
IG4-8
[GA-0
1GA-8
0.6-
Baseline
0.6-
Baseline
0.6-
Baseline
0.6-
Baseline
0.4
0.2
0.2
0.2
0.2
0.0
Epochs (x10²)
6
10
0.0
0
Epochs (x1)
10
0.0
Epochs (x10²) #
10
0.0
0
Epochs (x10²)
8
10
SIREN by balancing 2 eigenvalues
SIREN by balancing 4 eigenvalues
SIREN by balancing eigenvalues
SIREN by balancing 8 eigenvalues
a 50Hz
1GA-1
= 50Hz
1.0-
1GA-1
at k= 5011z
1GA11
50Hz
1.0
1GA
0.8-
IGA-1
0.8-
IGA-1
[GA-1
0.8-
I6A-1
1CA-8
ICA-S
IGA-8
1GA-8
0.6
Baseline
0.6
Baseline
0.6
Baseline
0.6
Baseline
at k
0.4
0.4
0.4
0.2
0.2
0.0
Epochs (x
6
8
10
0.0
0
Epochs (x10²)
10
0.0
Epochs (x10*))
8
10
0.0
Epochs (x10²)
8
10
SIREN by balancing 2 cigenvalues
SIREN by balancing 4 eigenvalues
SIREN by balancing eigenvalues
SIREN by balancing 8 eigenvalues
k= = 60Hz
1.0-
IGA-1
= 60Hz
1.0-
IGA-1
Ax at ka 40112
1.0-
IGA-1
A at = 60Hz
1.0-
IGA-1
0.8
1CA-4
0.8
IGA-
0.8-
IGA-4
0.8
1GA-4
JGA 8
IGA 8
IGA U
IGA 8
0.6-
Daseline
at =
0.6
Baseline
Baselize
0.6-
Baseline
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
4 Epochs (x10²)
10
0.0
Epochs (x10=)
10
Epochs (x10+))
4
8
10
0.0
Epochs (x10=)
10
Figure 12. Comparison of group sizes with varying balanced eigenvalues on Relative Error Ag at 10, 20, 30, 40, 50, 60Hz of SIREN.
IGA-1 denotes that the group size p is 1, i..., K-based gradient adjustment; IGA-4 denotes that the group size is 4 for IGA, i.e., inductive gradient adjustment; Baseline denotes the STREN optimized hy vanilla gradients. The shaded area indicates training fluctuations, represented by twice the standard deviation. Details of experimental setting can be found in Sec. 4. Please zoom in for better review.
25

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
Table 8. Peak signal-to-noise ratio (PSNR 1) of 2D color image approximation results by different methods. The detailed settings can be
found in Sec. 5.1.
Method
Kodak
Kodak2
Kodak3
Kodak4
Kodak5
Kodakó
Kodak7
Kodak8
Average
ReLU
19.78
26.62
25.88
24.70
17.70
21.98
21.52
16.07
21.78
ReLU+PR
19.96
26.63
26.53
25.31
18.05
22.03
22.13
16.52
22.14
ReLJU+NN
20.13
26.97
26.91
25.49
18.65
22.42
22.64
16.91
22.51
ReLU+IGA
20.42
27.71
28.20
26.17
18.81
22.69
22.96
16.99
23.00
PE
26.07
32.51
32.80
31.37
24.60
27.28
31.74
22.79
28.64
PE+ER
26.95
32.23
33.92
32.14
26.63
28.49
32.71
24.85
29.74
PE+BN
26.50
31.42
32.21
31.31
25.45
27.82
31.54
23.95
28.78
PE+IGA
29.17
35.18
37.91
35.07
28.06
31.27
36.60
26.41
32.46
SIREN
29.61
35.19
36.31
35.10
29.74
31.01
36.73
27.50
32.65
SIREN+FR
30.00
34.73
36.95
34.87
29.72
30.79
36.31
27.52
32.61
SIREN+IGA
30.10
35.85
38.60
35.80
30.13
31.94
37.68
27.73
33.48
Table Structural Similarity Index Measure (SSIM 1) of 2D color image approximation results by different methods. The detailed settings can he found in Sec. 5.1.
Method
Kodak1
Kodak2
Kodak3
Kodak4
Kodak5
Kodakó
Kodak7
Kodak8
Average
ReLU
0.2978
0.6455
0.7235
0.6199
0.2797
0,4531
0.5681
0.2785
0.4833
ReLU+FR
0.3063
0.6450
0.7310
0.6302
0.2920
0.4546
0.5789
0.2972
0,4919
ReLU+BN
0.3150
0.6461
0.7230
0.6319
0.3198
0.4635
0.5896
0,3145
0.5004
ReLU+IGA
0.3298
0.6628
0.7564
0.6387
0.3195
0.4753
0.5946
0.3232
0,5126
PE
0.7193
0.8126
0.8634
0.8077
0.7371
0.7553
0.8911
0.6790
0.7832
PE+ER
0.7653
0,8082
0.8819
0,8266
0,8074
0.7953
0.8947
0.7546
0,8167
PE+BN
0.7552
0.7985
0.8428
0.8198
0.7897
0.7880
0.8778
0.7525
0,8030
PE+IGA
0.8458
0.8806
0.9402
0.8994
0.8542
0.8778
0.9462
0.8132
0.8822
SIREN
0.8715
0.8948
0.9168
0.9060
0.9040
0.8689
0.9534
0.8643
0.8975
SIREN+FR
0.8781
0.8869
0.9308
0.9011
0.8971
0.8797
0.9536
0.8654
0.8991
SIREN+IGA
0.8830
0.9070
0.9492
0.9173
0.9110
0.8996
0.9606
0.8688
0.9121
Table 10. Multi-Scale Structural Similarity Index Measure (MS-SSIM 1) of 2D color image approximation results by different methods. The detailed settings can be found in See. 5.1.
Method
Kodak1
Kodak2
Kodak3
Kodak4
Kodak5
Kodakó
Kodak7
Kodak8
Average
ReLU
0,5270
0.7839
0.8414
0.7741
0.4697
0.6534
0.6646
0,5031
0.6521
ReLU+FR
0.5454
0.7973
0.8621
0.7964
0.5164
0,6679
0.7024
0.5523
0,6800
ReLU+BN
0.5837
0.8075
0.8581
0.8047
0.5943
0.6929
0.7331
0.5976
0.7090
ReLU+IGA
0.6205
0.8364
0.8947
0.8249
0.6208
0.7290
0.7591
0.6211
0.7383
PE
0.9356
0.9518
0.9668
0,9489
0,9382
0.9331
0.9744
0.9242
0,9466
PE+FR
0.9493
0,9494
0,9713
0,9559
0,9566
0,9470
0.9755
0.9464
0,9564
PE+BN
0,9491
0.9507
0.9615
0,9560
0,9561
0.9481
0.9726
0,9491
0,9554
PE+GAA
0,9705
0,9720
0.9880
0,9788
0,9713
0.9692
0.9895
0,9622
0.9752
SIREN
0,9808
0.9817
0.9851
0.9829
0.9845
0.9714
0.9918
0.9761
0.9818
SIREN+FR
0.9797
0.9788
0.9877
0.9819
0.9810
0.9781
0.9926
0.9766
0.9820
SIREN+IGA
0.9815
0.9851
0.9895
0.9873
0,9860
0.9803
0.9921
0.9759
0.9847
26

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
Table 11. Learned Perceptual Image Patch Similarity (LPIPS 4) of 2D color image approximation results by different methods. The detailed settings can be found in See. 5.1.
Method
Kodak1
Kodak2
Kodak3
Kodak4
Kodak5
Kodakó
Kodak7
Kodak8
Average
ReLU
0.7571
0.5392
0.4000
0.5547
0.7788
0.5854
0.6172
0,8089
0.6302
ReLU+FR
0.7692
0,5513
0.3871
0,5362
0.7829
0,6110
0.6131
0.8008
0.6315
ReLU+BN
0.7654
0,5471
0.3865
0.5458
0.7188
0,5959
0.5859
0.7999
0.6182
ReLU+IGA
0.6857
0.4770
0.2927
0,4730
0.6695
0,5609
0.5278
0.7527
0.5549
PE
0.2803
0,2092
0.1062
0,2128
0,2643
0.2517
0.1218
0.3322
0,2223
PE+FR
0.2547
0.2278
0,0905
0.1897
0,1566
0,2179
0.1154
0.2426
0.1869
PE+BN
0,3161
0,2605
0.1474
0.2065
0.2393
0,2592
0.1554
0,2926
0,2346
PE+IGA
0,1363
0.0938
0.0256
0.0956
0.1056
0.1068
0.0310
0,1559
0.0938
SIREN
0,1052
0.0878
0.0661
0.0994
0.0654
0.0967
0.0271
0.0980
0.0807
SIREN+FR
0.0928
0.0888
0.0465
0.1121
0.0661
0.1258
0.0317
0.0865
0.0813
SIREN+IGA
0.0895
0.0746
0.0246
0.0858
0.0628
0.0853
0.0199
0.0921
0.0668
Table 12. Intersection over Union (IOU) of 3D shape representation by different methods. The detailed settings can be found in Sec. 5.2 and Appendix H.
Method
Thai
Armadillo
Dragon
Bun
Lucy
Average
ReLU
0.9379
0.9756
0.9708
0.9924
0.9467
0.9647
ReLU+FR
0,9428
0.9756
0.9707
0,9914
0,9465
0.9654
ReLU+BN
0,9416
0,9789
0.9727
0.9934
0,9445
0.9662
ReLU+IGA
0.9563
0.9805
0.9793
0.9931
0.9571
0.9733
PE
0,9897
0.9956
0.9953
0.9985
0.9920
0.9942
PE+FR
0.9929
0.9979
0.9964
0.9990
0.9945
0.9961
PE+BN
0.9916
0.9968
0.9954
0.9985
0.9907
0.9946
PE+IAA
0,9943
0,9979
0,9975
0,9990
0,9961
0.9970
SIREN
0.9786
0.9908
0.9937
0.9953
0.9862
0.9889
SIREN+FR
0.9731
0,9935
0,9905
0.9972
0.9785
0.9866
SIREN+IGA
0.9802
0.9920
0.9938
0.9958
0.9868
0.9897
Table 13. Chamfer Distance of 3D shape representation by diffcrent methods. The detailed settings can be found in Sec. 5.2 and Appendix
H.
Method
Thai
Armadillo
Dragon
Bun
Lucy
Average
ReLU 5.6726c-06
5.5805c-06
6.5588c-06
8.0611c-00
3.8048e-06
5.9356c-06
ReLU+FR
5,2954e-06
5.6664e-06
6.5062e-06
8.2686e-06
3.5963e-06
5.8666e-06
ReLU+BN
5.3489c-06
5.4240e-06
5.8546e-06
7.8738e-06
3.5496e-06
5.6102e-06
ReLU+IGA
4.7463e-06
5,4097e-06
5.7295e-06
8,2330e-06
3.3144e-06
5.4866c-06
PE
4.2759c-06
5.2162e-06
5.3788e-06
7.7409e-06
3.0036e-06
5.1231e-06
PE+FR
4.2272e-06
5.2246e-06
5.3514e-06
7.7974e-06
2.9542e-06
5.1109e-06
PE+BN
4.2463c-06
5.2030c-06
5.4122c-06
7.7698c-06
2.9493c-06
5.116le-06
PE+IGA
4.2551c-06
5.1524c-06
5.4319e-06
7.7695c-06
2.9288c-06
5.1076e-06
SIREN
4.2920c-06
7.8573c-06
5.3923e-06
7.9121c-06
2.9870e-06
5.6881e-06
SIREN+FR
4.3588e-06
5,1830e-06 06
5.4719e-06
7.7805e-00
3.0217e-06
5.1632e-06
SIREN+IGA
4.3520e-06
5.2203e-06
5.3757e-06
7.8457e-06
2.9934e-06
5.1574e-06
27

---

Inductive Gradient Adjustment for Spectral Bias in Implicit Neural Representations
Table 14, Per-scene results of learning 5D neural radiance fields by different methods. Details can be found in Sec. 5.3 and Appendix I.
Methods
Ship
Materials
Chair
Ficus
Hotdog
Drums
Mic
Lego
Average
NeRF
29.30
29.55
34.52
29.14
36.78
25.66
33.37
31.53
31.23
LPIPS J SSIM + PSNR
NeRF+FR
29.50
29.70
34.54
29.35
37.08
25.74
33.35
31.55
31.35
NeRF+IGA
29.49
29.87
34.69
29.38
37.22
25.80
33,48
31.83
31.47
NeRF
0.8693
0,9577
0.9795
0.9647
0.9793
0,9293
0.9783
0.9626
0,9526
NeRF+FR
0,8729
0,9600
0.9797
0.9665
0.9792
0,9304
0.9786
0.9626
0,9537
NeRF+IGA
0.8716
0.9619
0.9807
0.9667
0.9801
0,9315
0.9791
0.9648
0.9546
NeRF
0.077
0.021
0.011
0.021
0.012
0.052
0.022
0.019
0.029
NeRF+FR
0,071
0.020
0.011
0.019
0.013
0.050
0.021
0.019
0.028
NeRF+IGA
0.074
0.019
0.010
0.019
0.011
0.049
0.020
0.017
0.027
Ground truth
NeRF (31.53dB)
NeRF+IGA (31.83dB)
Ground truth
NeRF (29.55dB)
NeRF+IGA (29.87dB)
Ground truth
NeRF (25.66dB)
NeRF+IGA (25.80dB)
Figure 13. Visual examples of novel view synthesis results of NeRF and NeRF+IGA. Details be found in Sec. 5.3 and Appendix I.
28


Reviews:
[{"review_id": "UM9WbbDuPx", "reviewer": "Reviewer_7Wmr", "summary": "The paper presents a Neural Tangent Kernel-based approach to improving the spectral bias of implicit neural representations, which have been shown to be biased towards low frequencies. The paper summarizes the NTK theory and proposes a way to estimate the K matrix using a subset of the training samples, making this estimation tractable, compared to computing the actual matrix. This matrix can then be used, similarly to previous work, to steer the training of INRs into having less bias. Theoretical guarantees for the estimation are provided, as well as experiments. The experiments are: two simple experiment on function approximation, useful for an analysis on the proposed approach but not for drawing practical conclusions; an experiment on image representation; an experiment on 3D shape representation; an experiment on neural radiance fields. All the experiments show that, in the proposed settings, the method can be applied to existing methods and outperform them.\n\n##Update after rebuttal\nThe rebuttal addressed most of my concerns, especially the ones about practicality in terms of time and memory requirements. I still believe an auto decoder-based 3D shape experiment would be more effective and useful than the one proposed, but overall I propose acceptance of the paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Time and memory analysis\n***\nThanks for suggesting a more detailed time and memory requirements analysis. We conducted a measurement of the training time and memory of IGA across all experiments. For example, in the **1D simple function approximation**, the vanilla SIREN takes 28ms/iter (ms/iter denotes milliseconds per iteration) with 1009MB memory; SIREN+IGA takes 45ms/iter with 1575MB memory; adjustments (w/o IGA) by the full eNTK matrix, i.e., the eNTK matrix obtained from the population data, takes 198ms/iter with 5881MB memory. In the **2D image approximation**, the vanilla PE takes 74ms/iter with 3609MB memory; PE+IGA takes 87ms/iter with 4167MB memory. The full eNTK adjustment requires >100GB memory, which is practically infeasible. In the **3D shape representation**, the vanilla ReLU takes 113ms/iter with 2163MB memory; ReLU+IGA takes 163ms/iter with 2231MB memory. In the **5D neural radiance fields**, the vanilla NeRF takes 160ms/iter with 9451MB memory; NeRF+IGA takes 286ms/iter with 11033MB memory.\n\nThe complete results can be found in: https://anonymous.4open.science/r/iga-inr-icml2025-3A4D/time_memory.md. Overall, IGA averages 1.56× the training time and 1.34× the memory of the baseline but achieves greater improvements (on average, **2.0× those of prior FR and BN**, as shown in the main text). Compared to the full eNTK matrix, IGA **saves at least 1/4 of the training time and memory**. Thus, our method improves fitting accuracy by enhancing training dynamics without significantly increasing time or memory. Notably, since IGA achieves the same results with fewer iterations, it often allows us to achieve results comparable to baseline methods in less or even shorter time. We present examples in the above link. \n# Practical issue\n***\nEnhancing network performance by improving training dynamics without affecting the inference holds significant practical value. It helps improve inference performance in cases with limited computational resources. NTK theory has made significant progress in analyzing the training dynamics and spectral bias of INR and shows the theoretical potential to adjust dynamics. However, directly applying it to most INR models for better performance remains impractical due to the intractability of NTK matrix. **IGA provides two practical solutions**: first, it proves and validates the effectiveness of eNTK (which is generally computable) in adjusting training dynamics; second, it introduces inductive generalization of the transformation matrix derived from the eNTK matrix of sampled data to adjust the overall training dynamics. With these two solutions, IGA achieves greater improvements (on average, **2.0× those of prior FR and BN**, as shown in the main text) without significant additional costs—requiring only 45% more training time and memory on average compared to the baseline. Please note that FR and BN also cost more training time and memory but with only minor improvement. Thus, IGA is a practical and effective training dynamics adjustment method for INR tasks.\n# Additional experiments\n***\nThanks for suggesting more models to further show IGA's practical usability. As you mentioned, we have included 2D results of IGA with Gauss, WIRE, FINER in our Appendix and now provide 3D shape results. Due to time constraints and considering that MFN is the backbone of BACON, we extend IGA to MFN for both 2D and 3D tasks. Furthermore, for the neural radiance fields, we extend IGA to the well-known DVGO. The results are listed in the table of the following link: https://anonymous.4open.science/r/iga-inr-icml2025-3A4D/new_exp.md. Previous training dynamics methods, i.e., FR and BN, are also compared. From the numerical results, **IGA consistently outperforms FR and BN across 2D, 3D, and 5D tasks**, aligning with the main paper. These results further support our claims. We will add these results in the main paper.\n# Chamfer Distance\n***\nThanks for insightful comments about 3D shape representation. Following your suggestion, we computed Chamfer Distance using the code from “Occupancy Networks” and reported the results in the link: https://anonymous.4open.science/r/iga-inr-icml2025-3A4D/chamfer_distance.md. As you mentioned, Chamfer Distance and IoU can behave differently. Despite this difference, **IGA still outperforms the baseline and other training dynamics adjustment methods, FR and BN.**\n# Discussion on 3D Shape Representation\n***\nThanks for valuable suggestions on the 3D task. The setting of 3D shape representation in the main text is simple but has been adopted by many classic INR works to test the representation capability of INR models, such as PE, SIREN, MFN, Gauss, WIRE, and FINER. Recent works on training dynamics of INR, such as FR and BN, have also used this setting to evaluate the effectiveness of their methods. Therefore, this setting is suitable for demonstrating that IGA can better improve training dynamics to enhance INR’s representation performance for 3D shapes.", "claims_and_evidence": "The claims are mostly supported by the shown evidence. The only exception, in my opinion, being the claim at page 2 line 97, where the proposed method is labeled as \"practical\". While the analyses and experiments indeed show that the method works in the tested scenarios and with the tested baselines, for the method to be convincingly practical an analysis on its additional time and memory requirements would be needed. A partial analysis is provided in the supplementary, table 6. However it is only shown for one experiments, it does not mention memory requirements, and the time impact is shown to be highly sensitive to the choice of p. I believe a more detailed analysis should be provided for all experiments, with data about time and memory requirements being added to the tables in the main paper.", "methods_and_evaluation_criteria": "Yes, the method seems well grounded and justified, providing tangible benefits.", "theoretical_claims": "I checked the claims to the best of my abilities, however I could not verify the proofs due to my limited expertise in NTK theory.", "experimental_designs_or_analyses": "Yes. The experiments are mostly well designed, and they include several standard scenarios, baselines and metrics, which overall are convincing about the potential of the proposed approach. \nE1) As mentioned above, a time and memory requirements analysis is required to support the claim about practicality, but it's currently very limited and relegated to the appendix.\nE2) The methods tested in the main paper (ReLU MLP, MLP with PE, Siren, Vanilla Nerf) do show that the proposed approach improves on vanilla approaches, however more recent and complex methods have been proposed, and a few of them should be used to convince the reader about the practical usability of the proposed framework. The supplementary additionally shows Gauss, WIRE and FINER for image representation. MFN[1]/Bacon[2] should be considered too, as well as for shape representation, and a more recent and powerful NeRF model should be added as well. I believe these results belong to the main paper, as they serve to support the claims made by the authors.\n\nE3) Additionally, the experiment on 3D shape representation is not fully convincing for 2 reasons. 1) While IOU is an important metric, Chamfer Distance should be reported as well. In my experience they can behave differently, and they are both considered standard in this field. 2) As far as I understand, a simple setting is shown where each network is overfitted to a single shape. This is not considered a very useful scenario in practice, since INRs are usually used in an auto-decoder fashion to learn multiple shapes. In the auto-decoder scenario, different methods can actually behave quite differently, so this would be a much more useful experimental setting.\n\n[1] Multiplicative Filter Networks, Rizal Fathony, Anit Kumar Sahu, Devin Willmott, J Zico Kolter, ICLR 2021\n[2] BACON: Band-limited Coordinate Networks for Multiscale Scene Representation, David B. Lindell, Dave Van Veen, Jeong Joon Park, Gordon Wetzstein", "relation_to_broader_scientific_literature": "The theoretical contributions seem well contextualised in the literature. My only point is, as mentioned above, about additional experimental comparisons.", "essential_references_not_discussed": "To the best of my knowledge, no, except for the additional experimental methods mentioned above.", "other_strengths_and_weaknesses": "Strengths:\nS1) The proposed method seems original and well grounded in its analysis\nS2) The paper is well written and easy to follow\n\nWeaknesses:\nW1) Experimental section does not fully convince of the validity of the method (see above)", "other_comments_or_suggestions": "These did not impact the review:\nPage 1 Line 33-35 column 2: it's not clear which methods have an impact on complexity. Some of the mentioned ones don't, as far as I know (such as SIREN)\nPage 6 Line 322 column 1: \"fixed last layer in\" -> \"fixed last layer as in\"?\nPage 6 Line 298-301 column 2: I've found the phrasing confusing", "questions_for_authors": "I would like the authors to comment on my questions about the experimental analysis", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jTqIAPYfyq", "reviewer": "Reviewer_x2oC", "summary": "The goal of the paper is to mitigate spectral bias in implicit neural representations by changing the training dynamics. The paper considers the well-known connection between the neural tangent kernel (NTK) and the linear training dynamics, which reproduces spectral bias through the eigenvalues of the NTK matrix. The main idea is to introduce a preconditioning matrix into the gradient updates during training, where the preconditioner is designed to compensate for the spectral bias in the NTK. There are two theoretical contributions required to realize this idea practically: (1) showing that the empirical NTK is a good approximation of the NTK as network width increases, since the empirical NTK is more practical to compute, and (2) showing a method to approximate the empirical NTK via batchwise computations that are computationally tractable even for large datasets. Combining these theoretical contributions, the paper proposes a practical algorithm called Inductive Gradient Adjustment (IGA), that can be applied to existing INR architectures to mitigate spectral bias during training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Answer for 3D shape and radiance field results\n***\nThanks for recognizing our experiments and evaluation approach. We further analyze IGA by comparing it with Fourier Reparameterization (FR) and Batch Normalization (BN), two key training dynamics adjustment methods for mitigating INR spectral bias. For 3D shape task, compared to the vanilla baseline models, FR and BN achieve an average IoU improvement of $9 \\times 10^{-4}$ and $6 \\times 10^{-4}$, respectively, across five scenes. IGA further achieves an IoU improvement of $4.1 \\times 10^{-3}$, which is on average **5$\\times$ that of FR and BN**. For the neural radiance field, FR and BN achieve 0.12dB and 0.14dB in PSNR, respectively. The average improvement of IGA is up to 0.24dB, which is nearly **2$\\times$ that of FR and BN**, as mentioned in line 417-418, right column. Please note that the improvements achieved by all these methods are non-trivial, especially considering that they are achieved without altering the network inference structure or inference speed.\n\n# Clarification of theorems\n***\nThanks for recognizing our theoretical contributions and careful reading. We apologize for any ambiguity in the statement. For Theorem 3.1, there are two assumptions. The first one is that $v_i^{\\top}\\tilde{v}_i >0$. This assumption is easy to satisfy in practice, as both $\\tilde{K}$ and $K$ are real symmetric matrices, and $\\tilde{K}$ gradually converges to $K$, leading to increasingly similar eigenvectors. The second one is that $\\eta < ( ) $. This assumption is made to satisfy the requirements of the linear dynamic analysis. For Theorem 3.2, the statement is indeed meant to convey that max(quantity A in absolute value, quantity B in norm) < epsilon. As you suggested, removing the brackets '[]' and reordering $p$ would make the expression clearer and exactly result in the $j_i$-th row of $\\tilde{K}_e$. **We will incorporate the above suggestions into the final version.**\n\n# Discussion about the Lipschitz functions $g_i$\n***\nThanks again for careful reading of our theorems and insightful comments. We will revise the Eq. 2 to provide more exposition of $g_i$. Initially, the functions $g_i, i=1,..., N$ are introduced to represent the potential eigenvalue adjustment functions. During the analysis, we found that **a Lipschitz continuity condition on these functions serves as a simple yet effective condition for the equivalence between $\\tilde{K}$-based and $K$-based adjustments**. This condition ensures that, for a set of N Lipschitz functions, there exists a specific Lipschitz constant upper bound, allowing for the existence of a width $m$ such that all adjusted eNTK eigenvalues converge to their corresponding potential adjusted NTK eigenvalues.  Following this theoretical analysis, In practice, we simply map the largest $n$ eigenvalues of the $\\tilde{K}$ to the $(n+1)$st largest eigenvalue of $\\tilde{K}$, keeping the remaining eigenvalues unchanged (please refer to Eq. 5 in our paper). The corresponding operation for the potential corresponding $K$ is mapping the largest $n$ eigenvalues of the $K$ to the $(n+1)$st largest eigenvalue of $K$, and others remaining unchanged. In this setting, the potential errors of these mapped $n$ eigenvalues is dominated by the $(n+1)$st largest eigenvalue. As the eigenvalues from the $(n+1)$-th largest to the smallest are kept unchanged, it follows that their $g_i$functions are the Identity function, with a Lipschitz constant equal to 1. Our wide results and empirical analysis show that the aforementioned simple $g_i$  setting effectively balances the convergence speeds of different frequency components. \n\n# Answer for relation to Broader Scientific Literature\n***\nThanks for your thoughtful comments. The second paragraph indeed aims to highlight the distinction between training dynamics-based methods and architecture-based approaches such as SIREN and WIRE. As you pointed out, our method can be combined with architectural modifications for further benefits, providing a stronger justification. We will revise this in the final version.\n\n\n# Refinements on Writing\n***\nThanks for your valuable writing suggestions. As you pointed out, the first two contributions at the end of the introduction may appear somewhat redundant. We appreciate your suggestion to distinguish **theoretical and algorithmic contributions**. We will revise the manuscript to better highlight our theoretical insights. Follow your suggestions, we will **highlight the empirical NTK approximation earlier** and restructure Section 3.1 into a **“Background” section**. Fig. 9 will be **resized** to match Fig. 10, and **ground truth** will be added to Fig. 4. **Abbreviation definitions** will be included in captions, and **line distinctions** in Fig. 3 will be adjusted. Minor typos and redundancies such as \"improves\" will also be carefully revised.", "claims_and_evidence": "Of the three bullet point contribution claims at the end of the introduction, the first two seem redundant with each other (ie. could be combined into one bullet). An alternative suggestion would be to have these two claims be separated into a theoretical contribution and an algorithmic contribution, since (as described in the summary section of the review) there are interesting theoretical contributions that can be separated from (but help justify and make practical) the algorithmic idea of using a preconditioner matrix to correct the spectrum of the NTK during training.\n\nThe theoretical claims are well validated by the first two toy experiments, and the improvements on the image fitting task are compelling real-world validation.", "methods_and_evaluation_criteria": "Overall I find the methods and evaluation criteria are clear and well thought out. The toy experiments do validate the theory, and the image fitting results are compelling. However, the 3D shape and radiance field results are very similar to prior work, with only a marginal improvement.", "theoretical_claims": "I appreciate the structure of how the theoretical contributions are introduced. Section 3.2 gives a clear overview of what the theoretical contributions are and how they will be put together to create the IGA training procedure. \n\nThat said, the informal statement of Theorem 3.1 is a bit confusing in that it’s not entirely clear where the assumptions end and the statement of the result begins. It’s also not entirely clear where the functions g_i come from; can they be chosen at will from the set of all Lipschitz functions?\n\nThe informal statement of Theorem 3.2 also uses some notation that is not entirely clear. In particular, there is a statement “...such that [quantity A in absolute value], [quantity B in norm] < epsilon.” Does this mean that max(quantity A in absolute value, quantity B in norm) < epsilon? I’m also a bit confused why there are brackets (starting at p and ending at r_e) in the main result, since if these were removed (and if p were reordered) then we would have exactly the j_i’th row of \\tilde K_e in the expression, which might be clearer to read.", "experimental_designs_or_analyses": "The first experiment is a toy 1D signal fitting experiment (in Figure 1) to verify the theoretical claims. In this toy setting where the analytical NTK and empirical NTK are both tractable, this experiment shows that the proposed NTK approximation does provide similar performance to the empirical NTK, which itself does slightly worse but still decently similarly compared to the analytical NTK. This experiment also shows that the approximation quality improves as network width increases from 1024 to 4096.\n\nThe second experiment (Figures 2 and 3) also uses a 1D toy function with known frequency decomposition, and validates the theoretical prediction that changing the number of NTK eigenvalues that are adjusted does have qualitatively the expected impact on spectral bias. \n\nThe remaining experiments show compelling improvements on 2D image fitting and more incremental improvements on 3D shape and radiance field fitting tasks.", "relation_to_broader_scientific_literature": "The second paragraph of the introduction draws a distinction between methods that mitigate spectral bias through model architecture versus through training procedure, and seems to imply that it’s simpler or easier to modify training procedure. I’m not sure why this would be true; many of the existing architecture-based INRs use very simple changes to a standard MLP, often just changing the activation function (e.g. SIREN, WIRE) or adding an input embedding (e.g. Fourier features). Later in the introduction the authors make the point that their proposed IGA training strategy can be used in conjunction with these architectural modifications for further benefit; to me this is a stronger justification for the proposed method.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A; discussed in the other sections of the review.", "other_comments_or_suggestions": "The theoretical contribution is, in my view, stronger than the introduction of the paper led me to expect. In particular, the idea of approximating the empirical NTK to make it practical as a gradient preconditioner is clever and elegant, so I’d like to see it mentioned earlier. \n\nOverall the writing is straightforward to follow, but some of the writing would benefit from copy editing. For example, the last sentence of the abstract might intend to say “tailoring the impacts of spectral bias” instead of “tailored impacts on spectral bias”? There is also some opportunity to tighten up the writing and remove redundancy. For example, in the left column of page 2 there are 6 sentences that describe “purposefully” overcoming spectral bias by adjusting the empirical NTK spectrum. On line 120 (left column), “improve” should be “improves.”\n\nSection 3.1 is a helpful and well-written introduction to spectral bias from an NTK perspective. I might suggest separating it into a “background” section rather than having it as part of the “method” section, since it is a summary of important background rather than a novel contribution.\n\nFigure 1 caption: “on time” should be “in time.”\n\nFigure 3: The line colors are difficult to distinguish without zooming in (particularly the blue and purple lines). The lines might be more easily distinguished either by changing colors or by adding some variation in line style (e.g. dotted, dashed, markers).\n\nFigure 4: The result is compelling, but to improve presentation I suggest (1) including the ground truth image, and (2) including the acronym definitions in the caption.", "questions_for_authors": "It would be helpful to provide a bit of exposition about the g_i below equation 2. Are these free parameters to be chosen when constructing the preconditioning matrix S, or are they constrained or derived in some way? Also, please refer to some questions embedded in other parts of the review.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Iv9GvhFkvm", "reviewer": "Reviewer_ARPB", "summary": "This paper introduces a practical Inductive Gradient Adjustment (IGA) method to address spectral bias in Implicit Neural Representations (INR) by using inductive generalization of the eNTK-based gradient transformation matrix. The effectiveness of IGA is evaluated across a wide range of INR tasks, and a theoretical justification for its impact on spectral bias is also provided.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Discussion on training time\n***\nThanks for your overall recognition of our method, experimental design, and analysis. Following your suggestion, we provide a further discussion about training time of our method as a supplement to Table 6 in our Appendix. Please refer to our response **“Time and memory analysis”** to Reviewer 7Wmr. The complete results can be found in: https://anonymous.4open.science/r/iga-inr-icml2025-3A4D/time_memory.md. Overall, IGA averages 1.56× the training time and 1.34× the memory of the baseline but achieves greater improvements (on average, **2.0× those of prior FR and BN**, as shown in the main text). Compared to the vanilla adjustment method by full eNTK matrix, **IGA saves at least 1/4 of the training time**. Thus, IGA improves fitting accuracy by enhancing training dynamics without significantly increasing time.\n\nThe additional training time overhead comes from the matrix decomposition. As you mentioned, computing the eNTK and performing eigenvalue decomposition for the population data is intractable due to both memory consumption and computational complexity. Thanks to our theoretical foundations (Theorem 3.1 & 3.2), we only need to decompose the eNTK matrix of the sampled data and inductively generalize them to the population data, which allows for a significant reduction in computational cost while maintaining the effectiveness of spectral bias mitigation. Table 6 in our Appendix shows that a small amount of the sampled data can also lead to a nontrivial improvement. \n\n# Discussion on potential solutions\n***\nThanks for your constructive advice. We acknowledge INT as a potentially insightful approach and have cited it in the line 014 (right column) of our paper. We will include a following discussion of such methods in the final version. However, the problems that INT aims to address and its use of eNTK are different from ours. INT aims to address **the costly training of INRs via the sample selection**, while our IGA aims to **improve the spectral bias of INRs via the training dynamics adjustment**. Based on the nonparametric teaching perspective, the eNTK in INT is used to compute the functional gradient；samples with larger functional gradients are prioritized for selection. **Therefore, the specific values are not required for INT, only the relative magnitudes matter.** They found that the functional gradient is positively correlated with the discrepancy between the MLP output and the target function; thus, direct computation of the eNTK can be avoided. \n\nWhile, for IGA, based on the linear dynamics perspective, eigenvalues of the eNTK matrix control the convergence speeds of the MLP on the corresponding eigenvectors; the uneven distribution of eigenvalues leads to varying convergence speeds across feature directions, which is considered as one of the potential causes of spectral bias. IGA aims to uniform these eigenvalues of the underlying eNTK matrix during training to balance the convergence speeds and overcome spectral bias. **Therefore, explicit computation of specific eigenvalues and the corresponding eigenvectors on the eNTK matrix is required for IGA.** \n\nAs you mentioned that the computational cost of the eNTK is enormous, IGA adopts the inductive generalization of the eNTK-based gradient transformation matrix estimated from the sampled data. This avoids the computation of the eNTK matrix for the entire dataset, significantly reducing computational overhead, while still providing a non-trivial improvement. Compared to the vanilla method, i.e, full eNTK matrix, this **saves at least 1/4 of the training time**, as mentioned in the answer \"Discussion on training time\".\n\n# Response for writing-related comments\n***\nThanks for your patient review of our theorem and proofs. $\\Theta(\\cdot)$ in Theorem 3.2 of the main text denotes a tight bound. Concretely, we take the $\\epsilon_2=\\Theta(m^{-3/2})$ in line 215, right column as an example. $\\epsilon_2=\\Theta(m^{-3/2})$ indicates that as the variable $m$ grows large, $\\epsilon_2$ behaves asymptotically like $m^{-3/2}$ up to constant factors. That is, as stated in the formal version of Theorem 3.2 (in Section A.4 of the Appendix), $\\epsilon_2=\\frac{\\eta k^3 R_0^2}{m^{3/2}}$, where  $\\eta, k, R_0$ are constants. Following your suggestions, we will remove the extra parenthesis in Theorem 3.1 (line 186) for readability, define $g_i(\\lambda)$ before Equation 2, and refine our explanation of spectral bias for clarity.", "claims_and_evidence": "The claims are well-articulated and supported by compelling evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are logical and well-founded.", "theoretical_claims": "I have reviewed the proof in this paper, and it appears to be correct.", "experimental_designs_or_analyses": "I have reviewed the experimental design and analyses, and most aspects are well-presented. However, there is limited discussion regarding training time. The use of the eNTK-based gradient transformation matrix to address spectral bias in INR involves computing the eNTK, which is computationally expensive, as noted in Section 3.2. It is important to address this issue and consider potential solutions, such as INT [1] (“Nonparametric teaching of implicit neural representations”).", "relation_to_broader_scientific_literature": "This paper aims to enhance the accuracy of Implicit Neural Representations (INR), which could be advantageous for fields related to INR.", "essential_references_not_discussed": "The references are comprehensive.", "other_strengths_and_weaknesses": "**Strengths**:\n\n- The paper is well-structured and easy to follow.\n\n- The evaluation of IGA is thorough across a wide range of INR tasks.\n\n- The theoretical explanation of IGA's impact on spectral bias is also provided.\n\n**Weaknesses**:\n\n- The notation for $\\Theta$ in Theorem 3.2 is unclear.\n\n- There is limited discussion regarding training time.", "other_comments_or_suggestions": "- There is an extra parenthesis in Theorem 3.1; “(max(gi(λ))” should be written as “max(gi(λ))”.\n\n- The explanation of spectral bias could be clearer, similar to the one provided in [2] (\"Fourier features enable networks to learn high-frequency functions in low-dimensional domains\").\n\n- The term gi(λi) should be defined prior to its introduction in Equation (2).", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a Neural Tangent Kernel (NTK)-based approach, the so-called Inductive Gradient Adjustment (IGA) method, to address the spectral bias in implicit neural representations (INRs). 
The effectiveness of IGA is demonstrated across a wide range of INR tasks.

The authors have done an excellent job during the rebuttal phase, and all reviewers are now convinced of the significance and merit of the contribution, which clearly improves upon existing methods.

As a result, I recommend acceptance of this paper to ICML 2025.

A few minor suggestions for the final version:
- Please consider using proper LaTeX formatting for math operators: use $\sin$, $\min$, $\max$ instead of $sin$, $min$, $max$.
- Please include a time and memory analysis during rebuttal, along with implementation details, to complement the experimental results.
- I personally found it a bit difficult to clearly identify the proposed IGA method in Section 3.2, maybe consider adding an algorithm block or a summary table to make it easier for readers to follow.
Decision: Accept (poster)