Learning Input Encodings for Kernel-Optimal Implicit Neural Representations
Authors: Zhemin Li
Liyuan Ma
Hongxia Wang
Yaoyun Zeng
Xiaolong Han
Venue: ICML 2025 poster (2025)
Keywords: INRs
Generalization
Encoding
Reconstruction
Primary area: deep_learning->other_representation_learning

Abstract: Implicit Neural Representations (INRs) rely heavily on architectural choices for good generalization. Developing theoretically grounded approaches for architecture design remains an active area of research. Via theoretical analysis of the infinite-width limit, we establish a methodology that characterizes INR's generalization by means of kernel alignment. We first formulate the optimal kernel that minimizes pointwise expected squared error, then demonstrate that the Neural Tangent Kernel of the composed function (INR with input encoding) can approximate any positive semidefinite dot-product kernels through input feature mapping adjustments. Building upon these insights, we propose a Kernel Alignment Regularizer (KAR) that naturally integrates with existing INR systems to enhance kernel alignment. We further develop Plug-in Encoding for Aligned Kernels (PEAK) to refine INR models with KAR using learnable input encoding. This work contributes to the ongoing research efforts in bridging theory and practice for principled INR architecture design. Code is available at https://github.com/lizhemin15/KAR.

Learning Input Encodings for Kernel-Optimal Implicit Neural Representations
Zhemin Li Liyuan Ma Hongxia Wang'
Yaoyun Zeng'
Xiaolong Han
Abstract
GT
NTK of UNR * OpinaINT OTEAK
INR
Implicit Neural Representations (INRs) rely heav- ily on architectural choices for good general- ization. Developing theoretically grounded ap- proaches for architecture design remains an active area of research. Via theoretical analysis of the infinite-width limit, we establish a methodology that characterizes INR's generalization by means of kernel alignment. We first formulate the op- timal kernel that minimizes pointwise expected squared error, then demonstrate that the Neural Tangent Kernel the composed function (INR with input encoding) can approximate any posi- tive semidefinite dot-pree product kernels through in- put feature mapping adjustments. Building upon these insights, propose Kernel Alignment Regularizer (KAR) that naturally integrates with existing INR systems to enhance kernel align- ment. We further develop Plug-in Encoding for Aligned Kernels (PEAK) to refine INR models with KAR using learnable input encoding. This work contributes to the ongoing research efforts in bridging theory and practice for principled INR architecture design. Code is available at https//giitab.com//izhemmiin
PEAK
D
fa(y())
Suinty
O
2
Vasilla
KAR
PEAK
Ficoddrr Barbbuse
En is (000)) (ol-ainl reol
(a) Space
(b) Space
Figure 1. Overview of PEAK: (a) Function space representation showing how an INR network fo with learnable encoder y ap- proximates the Ground Truth, where dark blue dots D represent training samples and light blue region represents μ.y the con- ditional distribution from which the ground truth function f" is sampled; (b) Theoretical analysis kernel space illustrating how the Kernel Alignment Regularizer (KAR) guides the Vanilla INR from the initial kernel K towards the optimal kernel K*, PEAK achieves kernel alignment while maintaining data fidelity through the joint optimization of encoder parameters and INR weights.
the generalization ability of INR can be influenced by archi- tectural choices, including activation functions (Sitzmann et al., 2020; Saragadam et al., 2023), network structures (Fathony et al., 2021; Lindell et al., 2022), and input en- codings (Tancik et al., 2020; Müller et al., 2022; Xie et al., 2022; Liu, 2024). Developing systematic strategies for INR architecture design presents an important opportunity for further advancement in the field (Dupont et al., 2022).
1. Introduction
Implicit Neural Representation (INR) has emerged as pow- erful paradigm for continuous signal modeling (Sitzmann et al., 2020), advancing various domains from computer vision to scientific computing via its ability to handle in- verse problems (Mildenhall et al., 2020; Chen et al., 2021; Martin-Brusdd
We investigate this design problem through theoretical anal- ysis. Due to INR's highly non-linear nature, direct analysis proves difficult. Neural Tangent Kernel (NTK) theory of- fers a breakthrough by showing that at infinite width, INR's training dynamics can be precisely characterized by kernel regression (Jacot et al., 2018; Chizat et al., 2019). This fun- damental connection enables us to leverage well-established kernel method theories to analyze and enhance INR's per- formance (Tancik et al., 2020; Yüce al., 2022: Li et al., 2023). Building upon kernel methods, we formulate the optimal kernel that minimizes pointwise expected squared error to measure INR's generalization ability. Finding an INR architecture whose corresponding kernel aligns with the optimal kernel becomes our main focus. This process, known as kernel alignment, has shown promising results in
al., 2021: Wadhwani & Kojima, 2022). At its core, INR achieves these successes by leveraging neural networks to map oin-dineniional input coordinates directly to output values. Although showing great promise,
Equal contribution 'Department of Mathematics, National Uni- versity of Defense Teegnology, Changsha, China. Correspondence to: Hongxia Wang <wanghongxia@@ nudr.cdu.cn>>
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Learning Input Encodings for Kernel-Optimal Implicit Neural Representations
traditional kernel methods (Bordelon al., 2020; Cristian- ini et al., 2001; Jacot al., 2020) and in aligning kernels between tasks and data (Chapelle et al., 2002; Cortes al., 2012; Liu et al., 2016; Liu, 2024). Nevertheless, its potential for INR optimization remains to be fully explored.
strate the effectiveness of the proposed approach in improving the generalization of INRs.
2. Related Works
Implicit Neural Representations. Implicit Neural Rep- resentation (INR) offers continuous, differentiable signal encoding, unlike discrete tensor representations that require interpolation and finite differences (Reddy al., 2021; Sitz- mann et al., 2020). This enables efficient physics simulation, shape optimization, and novel view synthesis (Guan et al., 2022; Mildenhall et al., 2020; Martin-Brualla et al., 2021), Research on improving INR's generalization has led to var- ious architectural advances, from activation functions (Xu et al., 2019; Luo et al., 2021; Cao et al., 2021; Vakevičius et al., 2019; Zhao et al., 2019; Sitzmann et al., 2020) to neural network structure (Fathony et al., 2021; Lindell et al., 2022), and input encoding methods (Tancik et al., 2020; Müller et al., 2022; Xie et al., 2022; Liu, 2024). These advances motivate theoretical exploration of architecture design.
Matching INR's kernel with the optimal kernel remains a challenging inverse problem. Previous work on NTK re- verse engineering has made important theoretical progress, showing that single-layer networks with polynomial activa- tions can approximate any semidefinite dot-product kernel (Simon et al., 2022). However, when moving from the- ory to practice deeper architectures have shown particu- larly promising results (Delalleau & Bengio, 2011; Eldan & Shamir, 2015; Lu et al., 2017), motivating us to explore more flexible architectural choices. In exploring activa- tion functions, designed initialization strategies have proven valuable for maintaining training efficiency and performance (Sitzmann et al., 2020). Drawing from these insights, we focus on input feature mapping since it allows us to leverage existing well well-designed deep architectures and their tuned ini- tializations without modification (Tancik et al., 2020; Müller et al., 2022; Xie et al., 2022: Liu, 2024).
Neural Tangent Kernel Theory. The NTK framework (Jacot et al., 2018; Chizat al., 2019) bridges the gap be- tween neural networks and kernel methods by showing that infinitely wide networks behave as kernel regression. This connection enables the theoretical analysis of neural net- work generalization (Tancik et al., 2020; Li et al., 2023) and expressive power (Yüce et al., 2022). The selection of appropriate kernels for optimal generalization remains a central problem in the field. While single-layer networks with polynomial activations can theoretically approximate any kernel (Simon et al., 2022), recent studies have shown that input feature mapping can significantly alter the NTK's s spectral properties (Tancik et al., 2020).
Guided by these practical considerations, this work estab- lishes a novel theoretical foundation showing that the Neural Tangent Kernel of INR with input encoding can approxi- mate any positive semidefinite dot-product kernel. Based on this insight, we propose Plug-in Encoding for Aligned Kernels (PEAK), which combines learnable encoder with Kernel Aligmment Regularizer (KAR) to enhance kernel alignment. PEAK can be readily incorporated into any exist- ing INR system as a plug-in solution without architectural modifications. Figure 1 illustrates this workflow, where the theoretical analysis in kernel space directs the vanilla INR towards the optimal solution using learnable input encoding and kernel alignment.
Kernel Alignment. Kernel Alignment (KA) has emerged as a powerful principle for improving model generalization by ensuring the kernel used in regression matches the target function's 's optimal kernel (Bordelon et al., 2020; Cristianini et al., 2001; Jacot et al., 2020). This approach has proven successful in various domains, from kernel parameter tuning (Chapelle et al., 2002) to multiple kernel learning (Cortes et al., 2012) and clustering (Liu et al., 2016; Liu, 2024). Recent theoretical advances have established rigorous con- nections between kernel alignment and generalization error bounds (Bordelon al., 2020: Canatar et al., 2021; Wang et al., 2024). Building upon the success of input encoding methods in INR (Tancik et al., 2020; Müller et al., 2022), our work explores the potential synergy between kernel theory and INR practice.
Our kernel alignment approach shows promising improve- ments for INR's generalization on inverse problems. The main contributions of this work include:
Theoretical Foundation: A Kernel Alignment Regu- larizer abbreviated as KAR is proposed in this paper which explores the theoretical connection between INR generalization and kernel alignment, providing a princi- pled approach to enhance INR's performance through kernel alignment.
Algorithm Design: We further propose PEAK algo- rithm, a plug-in solution that combines Iearnable encod- ing with KAR to align INR's 's kernel with the optimal kernel adaptively.
Empirical Validation: Experiments on image inpaint- ing, phase retrieval, and Neural Radiance Field demon-
2

---

Learning Input Encodings for Kernel-Optimal Implicit Neural Representations
3. Plug-in Encoding for Aligned Kernels
where the Neural Tangent Kernel (NTK) of this INR is ex- pressed as:
Previous works have made significant contributions to INR architecture design using innovative empirical approaches. Drawing from these advances, we aim to complement ex- isting methods by establishing theoretical foundations for achieving optimal generalization via kernel alignment in the infinite- width limit. Stemming from this insight, we propose PEAK to bridge theory and practice using learnable input encoding.
xx Vo, So, Vo, (x').
(4)
Here, Ku X) represents rew-or of length N whose i-th element is (x x,) and KJ X) E RNNN whose i-th row is Kg (x, X). Ko indicates the Moore-Penrose pseudoinverse of matrix Ko. The NTK captures the evolu- tion of network predictions during training by measuring how changes in parameters affect the output, providing a powerful tool for analyzing the limiting behavior of INR.
3.1. Approximate an Infinite-width INR with Kernel Regression
3.2. Optimal Kernel
To establish a theoretical foundation for optimal INR de- sign, we first examine the connection between INRs and kernel methods in the infinite-width limit. Recent theo- retical advances have demonstrated that training neural network under certain conditions resembles kernel method at infinite width (Jacot et al., 2018 8; Yang & Littwin, 2021; Golikov et al., 2022). Consider an INR fe: : X + y with L hidden layers:
Equation (3) indicates that given INR fe. we can charac- terize its prediction by fox at any x. Next, we examine pointwise expected squared error as follows,
E [Uo_ (x) (x))))
(5)
where the target function f is sampled from a measure Py over an appropriate function space. The key insight of this section lies in formulating the optimal kernel K* that minimizes this expected error, which will serve theoretical guide for designing enhanced INR architectures.
x(1) = (w(Ox((1)+ +b()), ISESL. [x(x)= = (1(++)x + +b(++))
(1)
where o()) denotes an element-wise activation function, ( = x. ( E | and b(9) E Le represent the weight matrix and bias vector of layer E respectively, and o encompasses all network parameters.
Theorem 3.1. The kernel that minimizes the error defined in equation (5) is expressed
K K* (xxx)) = By- ID (x)()
(6)
To formalize the learning process, let X C Rd denote the d-dimensional imput space and y S R be the output space, 1. Given a training dataset D {(x;; yi) æ; E X E V = 1,... ,N of size N, the INR is trained by minimizing the mean squared error (MSE)
Proof. To begin with, we introduce a 1 x N vector Mx == Ko X) X) (X, X). Substituting equation (3) into the ob- jective function (5) and differentiating with respect to Mx, we obtain
i... no = d, n+++
loss ((0) on D:
V VM, By 1D fx)))
1
N
= VM, W [M, Y)²
+ (()(
(()) = 2
Σ (t {x
(2)
2M 2E 2p =0
We aim to find the optimal parameters 0* such that the trained INR Je- (x) approximates the ground-truth function f"(x) for any input x X. However, as we all know, it is difficult to find the optimal parameters. Frequently, what we
This leads to
M = EI
obtain are only sub- sub-optimal solutions (shown in Figure 1), which leads to the generalization problem of INR.
and therefore
Mx = E Df (x) J(Y (
(7)
Let us denote the matrix of input samples as = (x.....,XN) x Nxd and the vector of output samples as Y = [y,,...,N/] UN]T RN When the width of INR ap- proaches infinity and the training time t + oc, fo, + fe_. which can be characterized by kernel regression (detailed derivation in Appendix A.I):
Note that the Hessian of T. wuyllD with respect to M2 is
HM, = T.YT.
As it is positive semi semi-defiitte (5) reaches minimum at Mg given by (6).
fo (x) = Kx XXX((XXX)X
(3)
3

---

Learning Input Encodings for Kernel-Optimal Implicit Neural Representations
Recalling that Mx = (o(x, x)K3(X (X, X), by compar- ing with (7) and noting that YY K K*(X,)) X) and By (f)
kernely (Jacot al., 2018). Reverse engineering shows that any dot-product kernel can be approximated by carefully initializing single-hidden-layer INR (Simon et al., 2022), Specifically, it requires specialized initialization schemes and restricts us to shallow architectures, preventing us from leveraging the benefits of modern deep networks.
X), we arrive at:
Ko(x, X)K = K (X,X). (8)
This equality holds when
A more practical solution involves utilizing& feature map encoder (x) X' + E. (Tancik et al., 2020; Müller et al., (9) 2022; Liu, 2024). In contrast to the reverse engineering approach, this encoder can be seamlessly integrated into any existing deep architecture without modifying their ini- tialization or depth. The resulting INR becomes fo with its NTK (x, x') = K ((x(xx)(xxx)) The following theorem demonstrates that with appropriate design, such a learnable encoder can achieve the optimal kernel K*:
K" (x,x') = EE--n (x))"
which completes the proof.
Remark 3.2. In this context, we employ the Moore-Penrose pseudo-inverse rather than the regular inverse because the kernel matrices may be singular or ill-conditioned in prac- tice. Additionally, K" in equation (6) is not unique, as the scaled version of K", namely a a > 0, yields the same M2- = (x X K (X. X).
Given dot-pruduct kernels K, K* (see Lemma A.1 for detailed properties), where K : 1- + B S R represents a Lipschitz continuous bijection with Lipschitz constant Cx and K* is continuous on [-1, 1] (i.e., K* E +C([-11])) satisfying [-[-1,1] S B _ R. If B S B, then for any E >> O. there exists a mapping 7s : + RE d² such that
Theorem 3.3.
Since / is sampled from Hy conditioned on D, for any X' and training sample x; S { where we deliberately introduce S to distinguish between the entire domain N' and the training set,
(x, x:) = E. j- D [0
= ,
[k (c) - (c)( <E, cE(-1.1]
sup
Specifically, for x,, Xj E S, we obtain
where Ke (c) = KK(a(xx) (x))) with c= x x' for any unit vectors x, x,x' E X.
(x,, x:) = By- (xx)) y:Us-
Utilizing kernel regression with the optimal kernel K* for- mulated in Theorem 3.1 faces two fundamental limitations. Computing KM (xxx'' = E ID intractable in practice due to lack of direct access to Py- Moreover, even with K available, evaluating K=t requires O(N3) operations, posing computational constraints for large-scale datasets. In comparison, INRs provide alter- native approach that can potentially handle various inverse problems with ((N) complexity during forward propaga- tion. Therefore, we propose to improve INR performance by incorporating theoretical insights from optimal kernel selection while circumventing the computational overhead of explicit kernel regression.
Proof. Since K represents a bijection, its inverse K-1, B - (-1, 1] exists and lic)) = K K-*(K((e)) maps [-1, 1] to itself with K(h(c)) = (c) As the composition of dot-producct kernels, l(()) is clearly a dot-product kernel. By Theorem A.2, h(c) admits a non-negative power series:
(x" remains
h(c)
age", k=
k ak 2 0,
ag 30.
k=0
For any > 0. choose N₂ such that ExYN, as < <//Ck and construct:
3.3. Optimal INR
(x()
v
2b Vax
IR
k=0
Without loss of generality, we assume each input x; sat- isfies ||x:||2 1. This condition can be achieved utiliz- ing feature map F(x) [cos x, sin which ensures [[((x))l2 = 1. Under this normalization, the NTK ex- hibits rotation-invariane
where x xs denotes the k-fold tensor product, and @ represent the direct sum and direct product, respectively. For unit vectors x, x' E X::
meaning K (x,x')== ) ((c) with c= T E 1]. For notational simplicity, we employ the same K to denote both the functions of one variable and two variables. We refer to kernels of this form as dot-product
No Te (x(x) 7x(x') (x = Σ (x x') (x
km0

---

Learning Input Encodings for Kernel-Optimal Implicit Neural Representations
Algorithm 1 PEAK Training Algorithm
we obtain
Input: INR fo. attention network gø., polynomial en-
fex (x) = Kx (X (X,
coder y, initial parameters {Ou, 8y (ay (0)] train- ing data D, grid points Xa. loss Lull, learning rate a, epochs
B Y (YYT)tY
(10)
N
for :0:
do
= Ey = Σ A(xxxx,))
Can(t) 6 Call (0,, (t) 0%) (Eq. (12))
0z-1 f 02 - aV. (!)(t)
It is worth noting that I II (x)) resides in the row space of Y. According to the properties of the Moore- Penrose pseudoinverse, for any vector within the row space of Y, we have vY (YYTT) = This elucidates why the second-to equality in equation (10) holds.
0/-1 <- 03 ao (t)
for = 0: Ny do
aj (t 1) + (t) - aV Ca)())
end for
end for
return Sor (xx,, (a,(T)) n0
In numerous practical applications, such as NeRF (Milden- hall et al., 2020), the ground truth ys is not directly observ- able. In these scenarios, fo.. also remains unknown since it depends on yr. However, we recognize that So, (x;) should converge to the true y; as - co. Consequently, we can sub- stitute both fe.. and y; in equation (10) with fe. (denoted as So for simplicity). yielding (x( Ei= A(x, x,) When incorporating the encoder y, this relationship suggests that the optimal INR fe should satisty
By Lipschitz continuity of K:
N₂
(c) = K
Σ
cxc
- K
Σ k==0
cxck
0
$ CK
OK
< E,
R ((fo(r(x))) A) = (l.f(((()) A(x, X) = O,
for some vector norm || 1|ps where A(x. X)= RIXN, We designate this term as the Kernel Alignment Regularizer (KAR), it enforces alignment between the INR's kernel and the optimal kernel derived from our theoretical analysis. In this work, we adopt || ll2 as our choice of norm.
where the first inequality follows from |c| S 0u 2 O.
Consequently, to achieve optimal INR representation (in terms of kernel alignment) while maintaining fidelity on training data, we formulate the following optimization prob- lem:
Remark 3.4. The key insight enabling this construction is the tensor product property: (x{*)
xx]'x2)* for unit vectors. When N₂ = 1, the encoder takes simple form (x) = Iy v /axx|T. This often suffices in practice since higher-order terms (k: > 2) contribute less as c* = c 0 rapidly approaches zero, particularly in scenarios
(xx*)=
N Σ (((0(7(x)))))) + A J R(Jo(1(x)),
min 0.
where x X2 < 1.
(11)
Remark 3.5. The construction is not unique. We can decompose vag into [ /agar,..... / aggmm for any parti- tion {m with En ag 1, which yields: (x(x) =
where serves as a learnable encoder for kernel alignment and À represents trade-off hyperparameter. The first term ensures fidelity on S, and the integral term constitutes the continuous form of KAR that enforces kernel alignment across the entire input space X.
[Vagai [ag0ml 1== V
3.4. The PEAK Algorithm
We observe that both 7 and A possess inherent structural constraints that must be satisfied, necessitating their pa- rameterization in a srnettre-preseevvng manner. Addi- tionally, the integral in equation (11) requires discretiza- tion for practical implementation. Following the construc- tion established in the proof of Theorem 3.3, we express: 7(x) = D v @a where x*k signifies the k-fold tensor product of x (when = O, it reduces to the sealar (g). In practical applications, we discover that setting N, = yields satisfactory performance in realistic image
Although Theorem 3.3 offers theoretical guidance, direct computation remains challenging since we cannot directly access the underlying distribution μ.y. Nevertheless, we can leverage a key insight: fx) can be characterized via its re- lationships with Y. Following this intuition, we model ر as a discrete distribution where PP (x) = y1) = A (x, x;) for i= 1,2,.... N, with Σn A(x, x;) = 1 and A x') Z 0 (Vx,''s E X) denotes a function that measures the simi- larity between x and x'. Combining equations (3) and (6),
5

---

Learning Input Encodings for Kernel-Optimal Implicit Neural Representations
tasks, resulting in 7(x [yeo yurx]" This simpli- fication proves reasonable since higher-order terms con- tribute diminishingly to the kernel due to (xi x2)* rapidly approaching zero for larger k.
as
R(fe(7(x)), -11f(((())) - A(x, X)f)(7(X))]12
Then, the following equality holds:
The function A (x, x') should exhibit continuity in mapping from X x to R, while satisfying two key conditions: (a)
Σ R(J0(7())):A)- -I A(X, X))]6(((X)))11 XEXO
En=1 A x;) 1, and (b) xxx 2 O, Vx, X.
We parameterize A as:
This result reveals that KAR corresponds the Dirichlet Energy (Li et al., 2022). From a geometric perspective, the learned function A x') characterizes the similarity between points x and x'. Our framework provides a more comprehensive formulation of this relationship through the lens of kernel alignment.
Ag (x, X) softmax(go (x) gor (X)),
where 90' R - IR" is implemented as compact INR.
This elegant formulation naturally satisfies both conditions while enabling flexible coordinate relationships.
Taking 2D image processing as an illustrative ex- ample, we can sample a uniform discrete grid XG from X' with dimensions m x n, where Xa { +) ISSSs m, s.sssssn C X. The complete loss function can be reformulated as:
4. Experiments
We evaluate PEAK from three aspects: (1) verifying the ker- nel alignment properties of PEAK, (2) analyzing the impact of architectural choices, and (3) examining performance in both linear and nonlinear inverse problems, comparing with strong baseline methods, including vanilla MLP (ReLU ac- tivation), Fourier feature networks (Fourier) (Tancik et al., 2020), and Hash (Müller et al., 2022).
N
Call min 0,ta) 3
[Vol1(x, ( (o5) vi)
1=1
Σ
R(fe(T(x, {az} (a)) Ao),
x
4.1. Kernel Alignment Verification
(12) where the second term represents the discretized KAR that ensures kernel alignment on the uniform grid Xg.
The optimization of equation (12) becomes feasible since exhibits differentiability with respect parameters 0, (. (=o 0' when L, fe and S are differentiable with respect to these parameters. This differentiability prop- erty holds for common loss functions such as L2 and INRs equipped with differentiable activation functions like sin(.). Consequently, we can employ gradient-based optimization techniques, such as gradient descent, to minimize equa- tion (12). Upon convergence, we obtain the optimal pa- Nr j=mn - 0' yielding the optimized INR So (Tx(,, {aj) )) Given that facilitates kernel align- ment by guiding the encoder towards the kernel regres- sion solution, we designate this methodology as the Plug- in Encoding for Aligned Kernels (PEAK) algorithm. For notational clarity, we present the complete algorithm in Algorithm 1.
(a) No mapping NTK
(b) Fourier mapping NTK
rameters 0", {a,
0
1
0
0
(c) Optimal NTK
(d) Our NTK
Figure 2. Visualization of Neural Tangent Kernels (NTK) for dif- ferent architectures on the interval [-1, x [-,
The optimal INR is encompassed within equation (11), yet not all solutions to equation (11) necessarily represent the optimal INR. Thus, equation (11) serves as necessary condition for the optimal INR rather than a sufficient one. Notably, our experimental results (Figure 2) demonstrate that the INRs discovered through our algorithm closely ap- proximate the optimal kernel.
To examine how PEAK influences kernel alignment, we con- duct experiments 1D signal fitting task where we can nu- merically compute both kernels. We consider a synthetic sig- nal f (x) = sin(4rx) with x E [-1, 1]. The optimal kernel K is computed using equation (6). while INR's empirical kernel is calculated as K(x, x') = V (ox
Proposition 3.6. Let X X and define R(fo(7(x)), A)
6

---

Learning Input Encodings for Kernel-Optimal Implicit Neural Representations
As shown in Figure 2, standard INR with Fourier features (Figure 2(b)) exhibits shift-invariant properties that are hen- eficial for many tasks. PEAK learns kernel (Figure 2(d)) that shares characteristies with the optimal kernel (Fig- ure 2(c)), suggesting the effectiveness our kernel align- ment approach.
Table 1. PSNR (dR) of reconstructed images by INRs. Results from six standard test images under three missing types: random, patch, and textural.
Image
Missing Type
MLP
Fourier
Hash PEAK
Random
18.64
20.10
19.67
22.60
Baboon
Patch
18.82
22.35
25.44
36.08
Textural
18.69
23.77
28.84
31.41
4.2. Impact of Polynomial Degree
Random
22.38
25.17
26.57
26.85
Boat
Patch
22.22
27.70
26.52
33.26
24.2
Textural
22.79
29.68
33.85
38.96
Random
23.87
26.17
28.51
29.06
24.0
Cameraman
Patch
23.71
28.68
28.19
35.20
PSNR (dB)
Textural
24.03
30.03
33.07
38.02
23.8
Random
21.59
25.39
25.26
27.55
Jetplane
Patch
21.61
26.09
26.59
34.40
23.6
Textural
21.10
30.67
32.65
36.94
23.4
Mean PSNR
Random
20.01
23.52
24.40
26.65
Min/Max PSNR
Lake
Patch
19.71
24.41
21.58
31.93
23.2
Textural
19.66
26.53
32.78
34.08
1
2
4
8
16
32
64
128
Random
22.40
25.52
27.32
27.95
Polynomial Order
Livingroom
Patch
21.99
24.21
27.63
33.76
Textural
22.03
30.00
34.20
38.15
Figure Analysis polynomial order's impact on reconstruction quality. The blue curve shows mean PSNR values, with shaded regions indicating min/max variations.
image with patch missing, compared to 23.71dB (MLP), 28. 28.68dB (Fourier), and 28. 19dB (Hash). Visual results in Figure 4 also show that PEAK helps maintain fine textures and natural transitions in the reconstructed regions.
Following Theorem 3.3, we study how the polynomial de- gree Nr in encoder affects approximation accuracy. We conduct experiments on the Jetplane image using the image reconstruction task with combined random and structured missing patterns. We evaluate polynomial degrees from Ny to N₂ 128. Results in Figure 3 show that while higher degrees can improve performance, the gains become modest after N., = 1, suggesting that balancing performance and complexity a practical choice. Further- more, the influences of the regularization coefficient X, the output dimension r of the regularization network, and the activation function PEAK thoroughly examined in Appendix B.I.
4.4. Nonlinear Inverse Problem: Phase Retrieval
Table 2. PSNR (dB) achieved by various neural network based phase retrieval methods for FPR and GPR with different sample ratios 8.
Image
ML.
Fourier
Hash
PEAK
1.9 28.02
16.25
31.75
40.44
House
1.8 28.72
17.40
28.78
31.77
1.7 26.76
16.92
25.38
28.11
1.9 17.73
16.64
18.88
21.82
FPR (Net-ADM)
Boston
1.8 17.37
15.21
17.86
18.75
4.3. Linear Inverse Problem: Image Reconstruction
1.7 18.13
13.08
15.32
1.9 22.83
20.90
28.96
33.47
We evaluate PEAK on image reconstruction with missing data. This involves predicting unseen regions from partial observations. The linearity of this problem provides clear testbed for studying INR's generalization in the basic set- ting. We test three complex scenarios with missing patterns shown in Figure 4(a): Random (50% pixels randomly re- moved), Patch (structured regions missing), and Textural (complex patterns missing).
Boat
1.8 22.00
15.22
23.79
30.52
1.7 22.12
13.61
22.22
24.90
19.14
21.88
23.03
33.19
House
0,9 19.03
21.57
20.81
33.25
0.8 19.05
20.98
19.20
32.65
1.0 17.95
19.27
17.56
24.97
GPR (Net-GD)
Boston
0.9 17.97
19.03
16.08
24.74
0.8 17.93
18.59
15.63
22.25
1.0 19.97
21.18
19.00
29.92
Boat
0.9 19.91
20.85
17.05
30.05
Results in Table indicate that PEAK improves across different scenarios. For the Baboon image with random missing pixels, PEAK achieves 22.60dB PSNR compared to 18.64dB (MLP). 20. 10dB (Fourier), and 19.67dB (Hash). Similar trends are observed for structured missing patterns, where PEAK reaches 35.20 50BB PSNR on the Cameraman
0.8 19.95
20.36
15.95
29.10
We evaluate PEAK on the nonlinear inverse problem of phase retrieval (PR), which reconstructs a signal using only magnitude measurements obtained from the Fourier trans- form, referred to Fourier phase retrieval (FPR), or Gaus-

---

Learning Input Encodings for Kernel-Optimal Implicit Neural Representations
PSNR (dB)
18.64
20.10
19.67
22.60
Random
PSNR (dB)
23.71
28.68
28.19
35.20
Patch
PSNR (B1))
19.66
26.53
34.08
Textural
(a) Missing
(b) MLP
(c) Fourier
(d) Hash
(c) PEAK
Figure 4. Qualitative comparison of image reconstruction results. Each row represents a different scenario: Baboon with random 50% missing pixels (top), Cameraman with large patch missing (middle), and Lake with textural pattern missing (bottom).
FIPR
GPR
(a) Ground Truth
(b) MLP
(c) Fourier
(d) Hash
(c) PEAK
Figure 5.
The first row displays the reconstruction images (House) of FPR by Net-ADM at 1.9 sampling ratio, while the second row displays the reconstructions (Boston) of GPR by Net-GD 1.0.
sian matrix linear operations, known as Gaussian phase re- trieval (GPR). A reduced number of measurements increases the challenge of PR. with the sampling ratio defined the ratio of measurements to the signal length in each di- mension. Building on this foundation, Net-GD (Jagatap Hegde, 2019a;b) and Net (Ma et al., 2024) com-
bine a deep decoder with GD and ADMM, respectively (see Appendix B.2). Since both the deep decoder and INR untrained networks, replace the former with an INR in Net-GD and Net-ADM, testing different input encodings.
Results in Table 2 show that PEAK provides improvements
8

---

Learning Input Encodings for Kernel-Optimal Implicit Neural Representations
of up 8.69dB in FPR and 1.688BB in GPR, while main- taining stable performance at lower sampling rates. Visual results in Figure 5 indicate that PEAK helps reconstruct both global image structure and local details. These results sug- gest the potential benefits of kernel alignment in nonlinear inverse problems.
(Müller et al., 2022) NeRF using 25, 50, and 100 input views from the NeRF synthetic dataset. PEAK outperforms vanilla Instant-NGP in PSNR (25 views, Table 3), especially with fewer samples. Figure 6 shows PEAK reduces sparse- sample artifacts, demonstrating better generalization.
4.6. The Computational Efficiency of PEAK
4.5. Neural Radiance Field
MLP
SIREN
Fourier
Mash
PLAK
Table Average PSNR (dB) of NeRF by Instant-NGP and our proposed PEAK under different numbers of view perspective sam- ples.
40
20
Synthetic view Sample numbers Instant-NGP PEAK
100
30.60
30.32
10
chair
50
24.97
28.34
25
23.79
24.42
100
32.50
32.76
Time (s)
hotdog
50
29.85
31.34
25
23.32
24.51
Figure PSNR (dB) INRs for the Baboon image reconstruction over time.
100
27.98
28.01
lego
50
26.23
27.03
25
21.74
23.27
100
23.93
24.19
drums
50
19.43
20.08
MLP
SIREN
Fourler
Hash
PEAK
25
16.92
17.66
50
100
24.07
24.08
ficus
50
21.48
21.86
25
18.75
19.65
100
28.35
29.94
mic
50
25.17
26.87
25
21.72
22.39
0.2
0.4
0.6
1.0
1.2
1.6
1.8
para paramneters (x1e7)
Instant-NGP
PEAK
Instant-NGP
PEAK
Figure 8. PSNR (dB) of INRs for the Babnon image reconstruction with varying number of parameters.
chair
We compare PEAK's efficiency with MLP, SIREN (Sitz- mann et al., 2020), Fourier, and Hash on Baboon image reconstruction. PEAK trains faster (Figure 7) and achieves higher PSNR with fewer parameters (Figure 8), outperform- ing alternatives in speed and performance.
29.65
23.49
26.55 dB
hotdog
25.
29.19
23.99 dB
27.37 dB
lego
5. Concluding Remarks
27.08 dB
27.88 dB
24.00 dB
25.22 dB
In this work, we have presented PEAK, a principled frame- work for enhancing INR's generalization through kernel- guided design. Looking forward, we believe understanding the theoretical relationship between kernel alignment and INR's expressiveness could provide deeper insights into neural network design.
mic
25,50 dB
27.09 dB
26.43 dB
27 88 d8
Figure 6. Visual results of Neural Radiance Field by Instant-NGP and our proposed PEAK with 50 view perspectives.
Neural Radiance Field (NeRF) synthesizes novel views from multi-view images. We compare PEAK with Instant-NGP

---

Learning Input Encodings for Kernel-Optimal Implicit Neural Representations
Impact Statement
Dupont, E., Loya, H., Alizadeh, M., Golinikii A., Teh, Y. W., and Doucet, A. COIN+ Neural compression across modalities. arXiv preprint arXiv: 12900, 2022.
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Eldan, R. and Shamir, O. The power of depth in feedforward neural networks. In Conference Learning Theory, December 2015.
Acknowledgements
Fathony, R., Sahu, A. K., Willmott, D., and Kolter, J. Z. Multiplicative filter networks. In ICLR, 2021.
We sincerely thank the amonymous reviewers for their valu- able and constructive feedhack, which has greatly improved our paper. This work was supported by the following grants: Project 2020Y 13504 under the National Key Research and Development Program of China, and Grant 1247 1401 from the National Natural Science Foundation of China.
Golikov, E.,, Pokonechnyy, E., and Korviakov, V. Neural tan- gent kernel: A survey. arXiv preprint arXiv:2208.134 2022.
Guan, S., Deng, H., Wang, Y., and Yang, X. NeuroFluid: Fluid dynamics grounding with particle-driven neural radiance fields. In ICML, 2022.
References
Jacot, A., Gabriel, F., and Hongler, C. Neural tangent kernel: Convergence and generalization in neural networks. In NeurIPS, pp. 8571 2018.
Arora, S., Du, S., Hu, W., Li, Z., Salakhutdinov, R., and Wang, R. On exact computation with an infinitely wide neural net. In NeurlPS, 2019.
Jacot, A., Simsek, B., Spadaro, F., Hongler, C., and Gabriel, F. Kernel alignment risk estimator: Risk prediction from training data. NeurIPS, January 2020.
Bordelon, B., Canatar, A., and Pehlevan, C. Spectrum dependent learning curves in kernel regression and wide neural networks. In ICML, pp. 1024-1034. PMLR, July 2020. Article No. 96.
Jagatap, G. and Hegde, C. Phase retrieval using untrained neural network priors. In NeurlPS Workshop on Solving Inverse Problems with Deep Networks, 2019a.
Canatar, A., Bordelon, B., and Pehlevan, C. Spectral bias and task-model aligmment explain generalization in kernel regression and infinitely wide neural networks. Nature Communicationss 2021.
Jagatap, G. and Hegde, C. Algorithmic guarantees for in- verse imaging with untrained network priors. In NeurlPS, volume 32, 2019b.
Cao, Y., Fang, Z., Wu, Y., Zhou, D.-X., and Gu, Q Towards understanding the spectral bias of deep learning, In IJCAI, 2021.
Li, Z., Sun, T., Wang, H., and Wang, B. Adaptive and implicit regularization for matrix completion. SIIMS, 15 (4):2000-2022, 2022. doi: 10.1137/22M1489222.
Chapelle, O., Vapnik, V., Bousquet, O., and et al. Choosing multiple parameters for support vector machines. Ma- chine Learning, 46: 2002. doi: 1023/AA
Li, Z., Wang, H., and Meng, D. Regularize implicit neural representation by itself. In CVPR, pp. 10280-10289, 2023.
1012450327387.
Chen, Y., Liu, S., and Wang, X. Learning continuous image representation with local implicit image function. In CVPR, pp. 8628-8638, 2021.
Lindell, D. B., Van Veen, D., Park, J. J., and Wetzstein, G. BACON: Band-limited coordinate networks for multi- scale scene representation. In CVPR, pp. 16252-16262, 2022.
Chizat, L., Oyallon, E., and Bach, F. On lazy training in differentiable programming. In NeurlPS, pp. 2937-2947, dec 2019. Published: 08 December 2019.
Liu, X. Incomplete multiple kemnl alignment maximization for clustering, IEEE TPAMI, 46(3)) 2024. doi: 10.1009/PAM...221..3.3
Cortes, C., Mohri, M., and Rostamizadeh, A. Algorithms for learning kernels based on centered alignment. JMLR, 13(1):795-828, March 2012.
Liu, X., Dou, Y., Yin, J., Wang, L.,, and Zhu, E. Multiple kereel k-means clustering with matrix-induced regular- ization. In AAAI, 2016.
S. On kernel-target alignment. In NeurIPS, 2001. Cristianini, N., Shawe- Shawe-Taylor, J.. Elisseeff, A., and Kandola,
Lu, Z., Pu, H., Wang, F., Hu, Z., and Wang, L. The expres- sive power of neural networks: a view from the width. In NeurlPS, pp. 2232-2240, December 2017. Published: 04 December 2017.
Delalleau, O. and Bengio, Shallow vs. deep sum- product networks. In NeurlPS, pp. 666-674, December 2011.
10

---

Learning Input Encodings for Kernel-Optimal Implicit Neural Representations
Luo, T., Ma, Z., Xu, Z.-Q. J., and Zhang, Y. Theory of the frequency principle for general deep neural networks. CSIAM Transactions on Applied Mathematics, 2021.
Wang, C., He, X., Wang, Y., and Wang, J. On the target- kernel alignment: unified analysis with kernel complex- ity. In NeurIPS, 2024.
Ma, L.,, Wang. H., Leng, N., and Yuan, Z. ADMM based fourier phase retrieval with untrained generative prior. Journal of Computational and Applied Mathematics, pp. 115786, 2024.
Xie, S., Zhu, H., Liu, Z., Zhang, Q. Zhou, Y., Cao, X., and Ma. Z. DINER: Disorder-invariant implicit neural representation. In CVPR, November 2022.
Xu, Z.-Q. J., Zhang, Y., Luo, T., Xiao, Y., and Ma, Z. Fre- quency principle: Fourier analysis sheds light on deep neural networks. Communications in Computational Physics, 2019.
Martin-Brune Brualla, R., Radwan, N., Sujjadi, M. S. M., Bar- ron, J. T., Dosovitskiy, A., and Duckworth, D. NeRF in the wild: Neural radiance fields for unconstrained photo collections. In CVPR, pp. 7210-7219, 2021.
Yang, G. and Littwin, E. Tensor Programs Ilb: Architectural universality of neural tangent kernel training dynamics. In ICML, 2021.
Mildenhall, B., Srinivasan, P., Tancik, M., Barron, J. T., Ramamoorthi, R., and Ng, R. NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV, pp- 405-421. Springer International Publishing, 2020. doi: 10. 10.100079978-3-00--5555----
Yüce, G., Ortiz-Jiménez. G., Besbinar, B., and Frossard, P. A structured dictionary perspective on implicit neural representations. In CVPR, pp. 19228-19238, 2022.
Müller, T, Evans, A., Schied, C., and Keller, A. Instant neural graphics primitives with multiresolution hash encoding. ACM TOG, 11(): Jul 2022.
Zhao, P., Yang, Y., and He, Q.-C. Implicit regulariza- tion via hadamard product over parametrization in high- dimensional linear regression. In NeurlPS, 2019.
Reddy, P., Zhang, Z., Fisher, M., Jin, H., Wang, Z., and Mitra, N. J. A multi-implicit neural representation for fonts. In ICML, 2021.
Saragadam, V., LeJeune, D., Tan, J., Balakrishnan, G., Veer- araghavan, A., and Baraniuk, R. G. WIRE: Wavelet im- plicit neural representations. In CVPR, pp. 18507-18516, 2023.
Simon, J., Anand, S., and DeWeese, M. Reverse engineering the neural tangent kernel. In ICML, 2022.
Sitzmann, V., Martel, J. N. P., Bergman, A. W., Lindell, D. B., and Wetzstein, G. Implicit neural representations with periodic activation functions. In NeurlPS, pp. 7537- 7548, 2020.
Tancik, M., Srinivasan, P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J. T., and Ng. R. Fourier features let networks learn high frequency functions in low dimensional domains. In NeurIPS, volume 33, pp. 7537-7547. Curran Associates, Inc., 2020.
Vakevičius, T., Kanade, V., and Rebeschini, P. Implicit regularization for optimal sparse recovery. In NeurlPS, 2019.
Wadhwani, K. and Kojima, T. SqueezeNeRF: Further factorized FastNeRF for memory-efficient inference. In CVPRW, pp. 2916-2925, 2022. doi: 10.1 CVPRW56644.22022.0000

---

Learning Input Encodings for Kernel-Optings Implicit Neural Representations
Appendix
A. Theoretical Analysis
A.1. Kernel Regression and Neural Tangent Kernel
Let X C Rd denote a d-dimensional input space and y C IR be the output space. For a dataset D {(.;,,) y:) æ; E &X, E),i= ,,..., N} of size N, we denote [x,,...,xx xx] E Rixd and = -- yN] RN. The ground-truth function denoted as f :x - V
1.1.1. KERNEL REGRESSION FRAMEWORK
We start from the mean squared error (MSE) loss funetion:
(
- Σ (yr fo(x:))
(13)
The gradient of the loss with respect to the parameters 0 is:
NN Vol(O) - Σ (v. fx(xx
(14)
The continuous- gradient descent dynamics for minimizing the square loss corresponds to the following ordinary differential equation (ODE):
1 N
0, = Vo.
E (v: - (x,
2
N
= Σ (us So, Vo,
A. 1.2. NEURAL TANGENT KERNEL.
When analyzing the training dynamics of neural networks through the lens of kernel regression, particular kernel known as the Neural Tangent Kernel (NTK) naturally arises. For neural network fo with hidden layers, the NTK defined
0fo(x) ((00)()) 0fo(x))
K(x,x')
(15)
00
0e
This kernel function characterizes how the network output at two different input points changes with respect to the parameters during training. Under the gradient flow dynamics, we have:
fo. (x) = 0 fo.
(ur 1wn
(16)
(x(xx)) (X)).
A. 1.3. PROPERTIES OF NTK
For standard feedforward neural networks, the NTK can be computed recursively using:
((++1) (x,+1((x,x) x') 5()) (x, x') (0) (x) x') + +o
(17)
where ( denotes the NTK at layer &, ( represents the expectation of activation function derivatives, and o of are the variances of weights and biases, respectively.
In the infinite-wiet width limit, the NTK exhibits several remarkable properties:
12

---

Learning Input Encodings for Kernel-Optimal Implicit Neural Representations
1. Pre-activations converge Gaussian process:
1o (x) - K(t)).
(18)
2. The empirical kernel converges to a deterministic kernel:
lim K,(x,x)) Ko(x, x)).
(19)
11 Big +00
3. The convergence rate is O(1/v ming wi) (Jacot et al., 2018).
A. 1.4. TRAINING DYNAMICS THROUGH NTK
The evolution of network outputs during training can be described through the NTK:
d di So (x) - K, (xx Y).
(20)
As oo, the solution converges to:
fe Ko(x, [X]xxXXXXXXXXXXXX
(21)
This result demonstrates that in the -nfit limit, neural network training becomes equivalent to kernel regression with the NTK, providing a theoretical foundation for understanding the learning dynamics of deep neural networks.
A.2. Dot-Produch Property
For normalized inputs (||x||2 1), the NTK becomes function of only the dot product between inputs:
Ko(x,x') = Kolc), c=xxx'
(22)
Lemma A.1. For any two unit vectors x, x' on the unit sphere, considering neural network with ReLU activation function, their pre-uctivations at each layer é in an infinitely wide network follow a Gaussian process with covariance K ((0) (c) depending only =x] =xxx x'.
Proof. We prove this result for the case of ReLU activation function. We proceed by induction over layers under the joint infinite-width limit (n1,..., f2 > oc).
Base Case (L 1): The first layer's pre activviivt:ons::
(x) w (())
(23)
with w N D NNO,, of). yield covariance:
(x,(x o²x x' oh = - +
(24)
Inductive Step (E + + 1): Assume (x) GP(O, (2) ((())k(()))))) (c)). For ReLU activation o, the post-activation covariance becomes:
(c)
|
KO())) (()
((((()
Elo(h (1)(h(0())) (x))
+ arcsin
K(O
(25)
2π
where (1) = o + of by normalization. This defines a new dot-product kernel (c). The (E + 1) layer covariance:
k(e+1) (c) + oz.
(26)
thus preserving the dot-product dependence.
13

---

Learning Input Encodings for Kernel-Optimal Implicit Neural Representations
NTK Recursion: Following (Jacot et al., 2018), the NTK decomposes as:
1.
Ko(c)= Σ
II (0) (c).
(27)
where (c) (c) and (c) ( (( (x)))]
the infinite-width limit, both terms become deterministic functions of с (Arora et al., 2019). Note that while this proof focuses on ReLU activation, similar results may hold for certain other activation functions under appropriate conditions.
(h(e-1)
Theorem A.2. For any dot- dot-product kernel K ((x, x') K(x x') defined on the unit sphere S if K continuous and positive definite, it can be decomposed into power series with positive coefficients and their is finite:
((c) Σ
and",
an > 0,
Σ On < 00. n=0
カ=0
Proof. We prove this result three steps:
Step 1: Power Series Expansion
For continuous positive definite kernel [-1, 1]. by Mercer's theorem, it has uniformly convergent eigenfunction expansion. For dot-product kernels on the sphere, these eigenfunctions are polynomials, which gives us the power series expansion:
K(c)=
Σ an
Step 2: Positivity of Coefficients
To prove an > 0, we use the positive definiteness of the kernel. For any f E (
[
[ K(x T ())(x))(()dxdd > 0.
Sg
Substituting the power series expansion and using the fact that ||x||2 = ||y||2 = 1:
DC
(xJ (xx) > J J
On gdd
Since this inequality holds for f and the integrals are non-negative by construction, we must have a, > for all Step 3: Convergence of Coefficients
Finally, we prove that -n= On oo. Since K is continuous on the compact set gd-1, is bounded. In particular, K(1) is finite. At = 1, we have:
ax. Since all a, > 0 and their sum equals the finite value K(1), we conclude that En On < oc.
K(1)=
7= 0
This result guarantees that any dot-product kernel can be approximated arbitrarily well by finite polynomial with positive coefficients.
B. Experimental Details
B.1. Ablation Studies
To better understand the impact of different components in PEAK, we conducted comprehensive ablation studies on three key aspects. All experiments in this section were conducted on the same task setting as in Section 4 image reconstruction on the Jetplane image with combined random and structured missing patterns.
14

---

Learning Input Encodings for Kernel-Optimal Implicit Neural Representations
Effect of Regularization Coefficient We investigated the impact of regularization coefficient À model performance. We examined À (10-5, 10-1, 10-3 10-2, 10-1, 1, 10, 102). As shown in Figure 9a), PEAK achieves optimal performance (PSNR around 23.5dB) when À is between 10-2 and 10-1 Both too small (< 10- and too large (>> 1) values of X lead to significant performance degradation, with PSNR dropping to around 14dB at À = 102.
Output Dimension of Regularization Network We analyzed the effect of varying the output dimension of the regularization network. As shown in Figure 9(),, we explored dimensions r E {10, 50, 100, 200, 300, 400, 500). The results indicate that PSNR improves significantly (from 19.5dB to 23.8dB) as r increases from 10 to 100. However, further increasing beyond 100 yields diminishing returns and even slight performance degradation, with PSNR decreasing to around 22dB at 500. This suggests that 100 achieves the optimal balance between performance and computational cost.
Choice of Activation Function We compared the performance of various activation functions including ReLU, Sigmoid, Tanh, Softmax, HardTTan,, ELU, Leaky ReI.U, and others. Figure (C) shows that most activation functions achieve PSNR values between 23-24dB, demonstrating relatively stable performance. Softmax shows marginally better stability and slightly higher average PSNR across multiple trials. Sigmoid and Softmin demonstrate competitive performance with minimal difference from Softmax. ReL.U and LeakyRel.U show slightly lower performance, potentially due the vanishing gradient problem.
Our ablation studies revealed several key findings:
The choice of regularization coefficient significantly impacts model performance, with optimal values between 10-2 and 10-1
An output dimension of r = 100 achieves the best performance, with larger dimensions showing diminishing returns.
Most modern activation functions perform well, with Softmax showing marginally better stability, followed closely by Sigmoid and Softmin.
These results provide insights into PEAK's optimal configuration and demonstrate our method's robustness across different parameter settings.
B.2. Integration with Net-GD and Net-ADM
For both the Net-GD and Net-ADM frameworks, we replace the original deep decoder network with our INR as follows, where m denotes the number of measurements and n represents the length of the signal to be reconstructed.
Net-GD Integration:
1. The INR fe directly maps coordinates to pixel values
2. Gradient descent updates follow:
O = O. Vo, Vo (X)l -
where A: R" + R the Gaussian random matrix.
Net-ADM Integration:
1. The INR output is treated as the primal variable
2. The ADMM updates follow:
2 arg min
P
[1 So. (X) + 1411
2
0 arg min
||
||AAo(X))
y22 + cl
2 P
P2 - fo(()) + uul1
2m
2
Ur+ u Pz
s0 (X)
15

---

Learning Input Encodings for Kernel-Optimal Implicit Neural Representations
where A Rm C denotes the discrete Fourier transform operator and P IR" R" is the padding operator that extends signals to match target lengths. The Lagrangian multiplier is expressed u, while p serves the penalty
parameter in the augmented Lagrangian framework. The data fidelity term 2m
[ vy² +1
is a smoothing version of 2m [X) - -y where 1 is vector of all ones. We empirically used 1 and
0.001, and took 2y E Ik" as the final reconstruction result.
16

---

Learning Input Encodings for Kernel- Implicit Neural Representations
24
24
22
23
PSNR (dB)
PSNR (dB)
20
18
21
16
Mean PSNR
Mean PSNR
20
Min/Max PSNR
Min/Max PSNR
14
10-5
10-4
10-3
10-2
10-1
10
10a
102
10
50
100
200
300
400
500
(a) Impact of regularization coefficient on model performance. PEAK achieves optimal performance when A is between 10-2 and 10-1
(b) Performance vs. output dimension of the regularization net- work. Output dimension of 100 achieves the best trade-off between performance efficiency. The PSNR the baseline Jetplane image 22.56 dB. In most cases, the utilization of diverse output dimensions within the PEAK framework signifi- cantly surpasses this bascline performance.
24.25
24.00
(dB)
23.75
23.50
23.25
23.00
22.75
Min/Max
RelU
Sigmoid
Tanh
Softmax
HardTanh
ELU
ReLU6
RReLU
Log Sigmold
TanhShrini
Softsign
Softplus
Softmin
Log Softmax
Saftshrink
Sin
Identity
Gaber
Leaky
Activation
(c) Comparison of different activation functions. Softmax shows marginally
better stability, followed closely by Sigmoid and Softmin.
Figure Ablation studies on three key aspects of PEAK: (a) regularization coefficient X, (b) output dimension regularization network, and (c) choice of activation function. The results demonstrate the robustness of our method across different parameter settings and architectural choices.
17


Reviews:
[{"review_id": "4G0xzf5XGv", "reviewer": "Reviewer_QEM1", "summary": "The paper studies Implicit Neural Representation (INRs) from a Neural Tangent Kernel (NTK) perspective. It introduces the *Kernel Alignment Regularizer* (KAR), which encourages alignment between the INR’s NTK and an optimal kernel and *Plug-in Encoding for Aligned Kernels* (PEAK).  PEAK is a method to integrate KAR with INR architectures with learnable input encodings. The authors show that PEAK improves image reconstruction and phase retrieval tasks compared to simple baselines (MLPs, Fourier features, and Hash encoding).\n\n**Update after rebuttal.** Please refer to [my last comment](https://openreview.net/forum?id=Cx80t5FAQJ&noteId=zeP46QL4uf).", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** The choice of the current baselines needs clarification.  \n**A1:** We appreciate your inquiry regarding the choice of baselines. Our PEAK algorithm is designed to find an encoder $\\gamma(\\mathbf{x})$ that enhances the generalization ability of the composed function $f_{\\boldsymbol{\\theta}}(\\gamma(\\mathbf{x}))$. Consequently, we focus on these baselines that also work with $\\gamma(\\mathbf{x})$. The vanilla MLP with $\\gamma(\\mathbf{x})$ serves as a benchmark for basic performance.  The Fourier feature and Hash encoding are among the most influential methods for improving high-frequency representation capability and convergence speed. In response to your feedback, we have extended Table 1 to include a broader range of baselines in [Table R1](https://anonymous.4open.science/r/ICML-Re-3333/Table_R1.pdf), such as SIREN, MFN, BACON, Gauss, WIRE, and FINER. While these more recent methods show improved high-frequency representation ability in their original studies, their generalization ability remains limited, as shown in [Table R1](https://anonymous.4open.science/r/ICML-Re-3333/Table_R1.pdf). Additionally, our KAR can be directly applied to these baselines in a plug-and-play manner. We further evaluate the improvement of KAR on the aforementioned baselines in [Table R2](https://anonymous.4open.science/r/ICML-Re-3333/Table_R2.pdf). The results indicate that KAR enhances the performance of all these baselines, and the numerical results are consistent and not sensitive to the choice of baseline, as KAR estimates the same optimal kernel under the same sampling pattern and image.\n\n**Q2:** Some theoretical results were not properly referenced.  \n**A2:** Thanks for your insightful suggestions from the perspective of a peer in this specialized field. Our main contributions are the introduction of KAR regularization and the PEAK algorithm, both derived from Theorem 3.3. Given the wide-ranging applications of INR across various research domains, our goal is to present the core ideas in a manner that is accessible to researchers with varying levels of prior knowledge, without necessitating consultation of the original literature. Therefore, we have distilled the essential part of these Theorems. In the revised manuscript, we will include proper citations in Appendix A to facilitate readers in finding the related works.\n\n**Q3:** The per-image results in tables in the main are unnecessary and could be aggregated.  \n**A3:** Thank you for pointing this out. We believe that including more image results is important as the performance of INRs is closely tied to the specific characteristics of each image. For instance, in Table 1, PEAK shows the highest improvement on the \"Baboon\" image with a missing patch, due to its left-right symmetry. This symmetry allows PEAK to enhance generalization by effectively learning internal self-similarities within the signal. A more detailed discussion on this will be provided in the revised version.   \n\n**Q4:** The relationship of the work to the findings of Yuce et al. (2022) should be discussed.  \n**A4:** NTK is a useful mathematical tool for analyzing the dynamics of INRs, and many researchers have utilized it to illustrate the properties of INRs. Yuce et al. (2022) analyze INRs from a dictionary learning perspective, which differs significantly from our approach. In contrast, Tancik et al. (2020) provide an earlier analysis of INRs from the NTK perspective, proposing the Fourier feature encoder to improve high-frequency representation capabilities, which we have cited in our paper. To our knowledge, our PEAK is the first work to employ kernel alignment for guiding encoder design. In the revised manuscript, we will include a citation to clarify the similarities and differences between our work and that of Yuce et al. (2022).  \n\n**Q5:** The readability of the paper could be improved.  \n**A5:** We recognize that the paper's density may pose a challenge to readers. To enhance readability, we will move technical proofs to the appendix and focus on providing clearer high-level intuitions in the main text, such as summarizing key concepts and illustrating them with examples. Additionally, we will work on simplifying the exposition of our methods and results.\n\n**Q6:** Notice that, for fully connected networks, depth does not help generalization in the NTK regime (Bietti & Bach, 2020).  \n**A6:** Thank you for your valuable comment regarding the findings of Bietti & Bach (2020). We agree that while depth may not theoretically improve generalization in the NTK regime for fully connected networks, in practical applications—especially when the network width is not infinite—depth can still positively influence model performance. In fact, there is very limited research that relies solely on a single-layer INR for realistic applications. Therefore, we propose an encoder learning algorithm rather than modifying the activation function in a single-hidden-layer INR, as done by Simon et al. (2022).", "claims_and_evidence": "The paper makes convincing claims.", "methods_and_evaluation_criteria": "The methodology is reasonable, but some stronger baselines are lacking, e.g., SIREN, WIRE.", "theoretical_claims": "Proofs appear correct. Several theoretical results were already known and not properly referenced.", "experimental_designs_or_analyses": "The experiments show that PEAK improves over baselines, but it is unclear if the chosen baselines are strong enough. Moreover, the per-image results in tables in the main are unnecessary and could be aggregated.", "relation_to_broader_scientific_literature": "To the best of my knowledge, the proposed PEAK algorithm is novel. However, several theoretical results – presented as novel – are known or well-established in the literature. For instance, all results reported in Appendix A are not novel and not referenced. Moreover, the result in Theorem 3.1 expressing the optimal kernel in terms of the posterior mean kernel is a standard result in Gaussian process and Bayesian nonparametric methods. In particular (6) is the Bayesian optimal kernel which minimizes the MSE.", "essential_references_not_discussed": "The paper is not the first work studying INRs from an NTK perspective. In particular, Yuce et al. (2022) should be discussed.\n\nYuce, Gizem, et al. *A structured dictionary perspective on implicit neural representations*. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.", "other_strengths_and_weaknesses": "The paper is dense, making it hard to follow. For instance, proofs could be moved to the appendix to improve readability, while providing in the main text only clearer high-level intuitions.", "other_comments_or_suggestions": "(L217) Notice that, for fully-connected networks, depth does not help generalization in the NTK regime (Bietti & Bach, 2020).", "questions_for_authors": "1. What motivated the choice of the current baselines?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GOXPYrtE5y", "reviewer": "Reviewer_Z6av", "summary": "The paper summarises NTK-theory related contributions on INRs, and derives the optimal kernel for INRs under certain conditions. It then proposes an algorithm, named PEAK, to approximate a \"Kernel Alignment Regularizer\" and apply it to an INR, so that its kernel is encouraged to move towards the optimal one, improving its generalisation performance. Experiments show that the proposed approach achieves a similar kernel to the optimal one in a simple function approximation scenario. More practical experiments on image reconstruction and phase retrieval show that the proposed approach helps a simple ReLU-based MLP to achieve better results than with Fourier features or hash encoding.\n\n##Update after rebuttal\nThe rebuttal addressed most of my concerns. I am inclined towards acceptance, and expect the authors to incorporate the new results/experiments/considerations in the final version of the paper in case of acceptance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**All the Table/Figure Rx can be found in https://anonymous.4open.science/r/ICML-Re-3333**\n\n**Q1:** The paper should compare PEAK with SIREN, MFN, BACON, Gauss, WIRE, FINER and SAPE, and explore whether the framework can be applied to and improve these methods.  \n**A1:** Thank you for the concern regarding the experimental validity. Actually, we have shown that our PEAK algorithm is not sensitive to the choice of INRs with different activation functions, as illustrated in Appendix Figure 6(c). Furthermore, we have applied PEAK to the INRs you mentioned, except for SAPE, in *[Tables R1](https://anonymous.4open.science/r/ICML-Re-3333/Table_R1.pdf) and [R2](https://anonymous.4open.science/r/ICML-Re-3333/Table_R2.pdf)*. The exclusion of SAPE is due to its significantly different training strategy, which requires additional fine-tuning for a fair comparison. For more details about these experiments and our motivations, please refer to ***A1 to Reviewer QEM1***.\n\n**Q2:** The time and memory requirements of the algorithm are not discussed.  \n**A2:** As shown in *Figure R1*, for the \"Baboon\" image reconstruction task, our proposed PEAK achieves a PSNR of 30 dB in 2 seconds, which benefits from the kernel alignment providing additional self-similarity to accelerate convergence. As shown in *Figure R2*, PEAK achieves a higher PSNR compared to other methods while using the same number of parameters. We will include a more comprehensive discussion of these results in the revised version.\n\n**Q3:** Additional experiments on tasks like 3D shape representation and neural fields.  \n**A3:** Thank you for the suggestion. We have conducted more experiments on neural fields, comparing our methods with Instant-NGP, as NeRF can be time-consuming. Specifically, we performed experiments on the NeRF Synthetic datasets with 25, 50, and 100 view perspective samples, respectively. As shown in *Table R3*, the average PSNR for 25 view perspectives indicates that the PEAK algorithm outperforms the vanilla Instant-NGP. Notably, the improvement is more pronounced with fewer training samples, demonstrating that PEAK effectively enhances generalization capability. *Figure R3* shows that PEAK reduces artifacts caused by sparser samples, indicating its ability to leverage the internal structure of the data to improve performance in downstream tasks with limited data. We will include this 3D experimental data in the revised manuscript to further demonstrate the effectiveness and generalizability of our method.\n\n**Q4:** The supplementary shows some sensitivity to hyperparameter choice. This should be discussed in a limitations section.  \n**A4:** Due to space constraints, we only discuss the polynomial degree of $\\gamma$ in the main text, as it is unique to PEAK. The influence of other parameters (e.g., the regularization coefficient $\\lambda$) is a common consideration for all regularization-based methods. The ablation study on activation functions demonstrates that PEAK is generally insensitive to these choices. While the output dimension $r$ of the regularization network appears to affect the final results, all configurations still outperform the baselines. We will provide the PSNR values of baselines in the supplementary material to avoid potential misunderstandings.\n\n**Q5:** An analysis of the proposed regularizer's impact on low-frequency bias in INRs is needed.  \n**A5:** As shown in *Table R4*  and *Figure R4*, we conducted experiments using a synthetic image.  We sampled a $256\\times 256$ grid from $[-1,1]\\times [-1,1]$ as $\\mathcal{X}$ and generated  $\\mathcal{Y}$ using $\\mathbf{y}_i=\\sin(50\\pi\\sin(\\frac{\\pi}{3}\\cdot\\left\\\\|\\mathbf{x}_i\\right\\\\|_2))\\in\\mathcal{Y}$. We then trained MLP, Fourier, and our proposed PEAK algorithm on $\\mathcal{X}\\times\\mathcal{Y}$. This synthetic image represents a frequency gradient transitioning from low (outer) to high (center, $(0,0)$). The results indicate that the MLP struggles to learn higher frequencies even after 10,000 epochs. The Fourier shows some improvement by initially capturing low frequencies before gradually learning high frequencies, but this process is relatively slow. In contrast, our PEAK algorithm effectively addresses the low-frequency bias, enabling it to learn both low and high frequencies almost simultaneously.\n\n**Q6:** Clarify the number of additional parameters introduced by the second (compact) INR and compare it to a vanilla INR with the same number of parameters.  \n**A6:** The second (compact) INR introduces around additional $\\frac{1}{10}$ of the first (main) INR parameters. We have compared its performance against a vanilla INR with the same total number of parameters. As illustrated in *Figure R2*, our algorithm exhibits superior performance.\n\n**Q7:** \"While\" should be removed; a reference to Figure 3 is missing.  \n**A7:** Thank you for pointing out the incomplete sentence and the missing reference to Figure 3. We will make the necessary corrections in the revised manuscript.", "claims_and_evidence": "The theoretical claims and supported by proofs and analyses, however the experimental claims about the effectiveness of the proposed approach in improving the generalisation of INRs lack experimental support. See below.", "methods_and_evaluation_criteria": "The proposed method seems appropriate and well grounded.", "theoretical_claims": "I checked the claims to the best of my abilities, however I could not verify the proofs due to my limited expertise in NTK theory", "experimental_designs_or_analyses": "Yes. While I find the experiments themselves to be appropriate for the evaluation of generalisation capabilities of the method, I believe that the shown results are not sufficiently convincing. \nE1) SIREN [1] is mentioned in the paper but not used for comparisons in the experiments. Other more recent methods such as MFN[2], BACON [3], Gauss [4], WIRE [5], FINER [6] and SAPE [7] should be compared to, potentially also showing whether the proposed framework can be applied to such methods and improve them. Currently, the experiments convince the reader that the framework works on ReLU MLPs, but they do not support a practical use of the framework.\nE2) On a similar note, time and memory requirements of the algorithm are not discussed, which would be needed to justify its practicality.\n\nE3) Additional experiments on different tasks would also be appropriate, such as 3D shape and neural fields, the latter of which would greatly benefit from better generalisation capabilities. \n\nE4) The supplementary shows some sensitivity to hyper parameter choice. This should be discussed in a limitations section in the main paper.\n\nE5) Since INRs have been shown to have a bias towards low frequencies, how does the proposed regularizer affect this? An analysis would be interesting.\n\nE6) Additionally, a second (compact) INR is used to compute the function A. How many additional parameters does this introduce? Does the proposed approach work better than a vanilla INR that has as many parameters as the total of f + g?\n\n[1] Implicit Neural Representations with Periodic Activation Functions, Sitzmann et al.\n[2] Multiplicative Filter Networks, Rizal Fathony, Anit Kumar Sahu, Devin Willmott, J Zico Kolter\n[3] BACON: Band-limited Coordinate Networks for Multiscale Scene Representation, David B. Lindell, Dave Van Veen, Jeong Joon Park, Gordon Wetzstein\n[4] Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPs, Sameera Ramasinghe, Simon Lucey\n[5] WIRE: Wavelet Implicit Neural Representations, Saragadam et al.\n[6] FINER: Flexible spectral-bias tuning in Implicit NEural Representation by Variable-periodic Activation Functions, Zhen Liu, Hao Zhu, Qi Zhang, Jingde Fu, Weibing Deng, Zhan Ma, Yanwen Guo, Xun Cao\n[7] SAPE: Spatially-Adaptive Progressive Encoding for Neural Optimization, Hertz et al.", "relation_to_broader_scientific_literature": "The theoretical contributions seem well contextualised in the literature. However the experiments do not show relations to prior experimental work.", "essential_references_not_discussed": "See the experimental section of the review.", "other_strengths_and_weaknesses": "Strengths:\nS1) The paper is overall well written\nS2) The method seems original and potentially interesting to use in practice\n\nWeaknesses:\nW1) The experiments do not convince the reader about the validity of the method except with a very basic baseline. The number of additional parameters, as explained above, may also present a weakness as it is not currently discussed.\nW2) Limitations are not discussed.", "other_comments_or_suggestions": "Page 4 Line 212-215 column 1: sentence is not finished, the \"while\" could be removed.\nSection 4.2: missing reference to figure 3, which is not referenced anywhere", "questions_for_authors": "I would like the authors to address my concern about experimental validity, in case I missed some points or scope.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rj7eTAolCC", "reviewer": "Reviewer_DUFF", "summary": "The paper proposes two theoretically-motivated changes to implicit neural representation architecture and training, based on comparisons to the infinite width neural tangent kernel. The first change is a regularization strategy to encourage alignment with the optimal NTK, and the second is a trainable encoding that can be added before an INR to improve kernel alignment. Since the optimal kernel is not computable (due to both lack of access to the true data distribution and computational limitations of evaluating and inverting a large matrix), the idea is to encourage the INR to approximate this optimal kernel without computing it directly. Note that the proposed changes can be applied to any INR as a plug-in modification.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**All the Table/Figure Rx can be found in https://anonymous.4open.science/r/ICML-Re-3333**\n\n**Q1:** A comparison of model size and training/inference time, with a fixed number of trainable parameters, is necessary for a fair evaluation.  \n**A1:** We have conducted a comparison regarding parameter numbers and training time. Our results demonstrate that PEAK converges quickly and achieves the highest PSNR at the same training time, as seen in [Figure R1](https://anonymous.4open.science/r/ICML-Re-3333/Figure_R1.pdf). Additionally, as shown in [Figure R2](https://anonymous.4open.science/r/ICML-Re-3333/Figure_R2.pdf), PEAK maintains the highest PSNR with an equivalent number of parameters compared to other methods. These findings indicate that our PEAK excels in both efficiency and performance.\n\n**Q2:** It may be beneficial to relax the language around alignment with the optimal kernel.  \n**A2:** Thank you for your insightful feedback. We agree that complete alignment with the optimal kernel is unrealistic due to its unidentifiability from discrete data. Our method aims to endow the INR with properties shared by the optimal kernel, rather than asserting full alignment. In the revised paper, we will adjust our wording to more accurately reflect the actual capabilities of the method and avoid overemphasizing alignment.\n\n**Q3:** Some notation is not clear.  \n(1) The notation for the variable A in Theorem 3.3 and Section 3.4 appears inconsistent and requires clarification.  \n(2) The notation for the embedding function $\\gamma$ requires clarification.  \n(3) The notation for the circled plus is used without definition and can represent multiple operations.  \n**A3:** Thank you for pointing out the confusion regarding the notations.  \n(1) We sincerely apologize for the oversight. In fact, the $A$ in Theorem 3.3 and Section 3.4 represents entirely different concepts. In Theorem 3.3, $A$ denotes a subset of the real numbers, reflecting the range space of a kernel, whereas in Section 3.4, $\\mathbf{A}:\\mathcal{X}\\times \\mathcal{X}\\rightarrow \\mathbb{R}$ (denoted in bold) is a function that measures the similarity between input points. To enhance clarity, we will change the notation in Theorem 3.3 to signify a set, and we will move the definition of the function $\\mathbf{A}$ in Section 3.4 to its first mention in the revised manuscript.  \n(2) The initial definition of $\\gamma$ is provided at line 166 in the right column. We will review the text to ensure clarity and reinforce the understanding that $\\gamma$ is a learnable function with specific properties.  \n(3) The $\\oplus$ and $\\otimes$ represent the direct sum and direct product, respectively. We will add more detailed explanations following these symbols.\n\n**Q4:** Encourage the release of the code upon publication, if not before.  \n**A4:** We will make our code publicly available upon publication to facilitate reproducibility and further research in this area.\n\n**Q5:** The supplement on ablation studies and model architecture details should be included in the main paper.  \n**A5:** We appreciate your suggestion regarding the supplementary material on ablation studies and model architecture details. We will ensure to reference this material in the main paper and summarize the key findings to provide readers with a clearer understanding of our experimental design.\n\n**Q6:** Improvements are needed for the clarity of Figure 1.  \n**A6:** Thank you for your feedback on Figure 1. We will follow your suggestions to improve its clarity in the revised version. In Figure 1(a), we will reduce the transparency to better distinguish overlapping dots and lines. In Figure 1(b), the \"orange arrows\" indicate the training progress, i.e. training INR with the vanilla loss function and PEAK, resulting in $f_{\\boldsymbol{\\theta}}(\\gamma(\\cdot))$ and $f_{\\hat{\\boldsymbol{\\theta}}}(\\hat{\\gamma}(\\cdot))$, respectively. The \"green arrows\" represent the application of the NTK theorem to calculate the corresponding NTK values, where $f_{\\boldsymbol{\\theta}}(\\gamma(\\cdot))\\rightarrow K$ (depicted as the black triangle) and $f_{\\hat{\\boldsymbol{\\theta}}}(\\hat{\\gamma}(\\cdot))\\rightarrow \\hat{K}$ (depicted as the orange star).  The \"blue arrows\" illustrate the theoretical analysis that introduces the optimal kernel $K^*$ (blue star). The vanilla INR's corresponding $K$ (black triangle) is far from the $K^*$ (blue star). Our KAR aligns the $\\hat{K}$ with $K^*$ in the kernel space, guiding the INR to optimize $\\gamma$. And the level sets represent the kernel space. We will add further explanation and simplify Figure 1(b) to enhance clarity in the revised version.\n\n**Q7:** Some typos/minor grammatical mistakes. E.g. the sentence on line 214 (left column) is not complete.  \n**A7:** Thank you for pointing out the incomplete sentence. We will conduct a thorough review of the manuscript to correct these typographical and grammatical errors.", "claims_and_evidence": "The claims of improved performance are well substantiated by experiments. The claims of matching the optimal kernel are substantiated with a toy experiment in Figure 2, though in general it seems the proposed method approximates a class of kernels that includes the optimal kernel, but may not exactly match the optimal kernel since that is in general not identifiable from a finite dataset.", "methods_and_evaluation_criteria": "Evaluation on image inpainting (with different types of linear image corruption) and phase retrieval (a classic nonlinear inverse problem, also here for images), is quite compelling. These are reasonable examples to test the method and substantial improvement is shown compared to existing INRs, both quantitatively and qualitatively.\n\nAlthough the quality of results obtained with the proposed method PEAK are impressive, it would be informative to also compare model size (ideally keeping the number of trainable parameters fixed across all models compared) and training/inference time for each method compared. Since PEAK requires adding a trainable embedding layer, I am concerned that it may gain an unfair advantage by increasing model size and/or training time.", "theoretical_claims": "The theoretical claims include a characterization of some properties that must be satisfied by the optimal kernel method for a dataset, and algorithmic and approximation choices to encourage an INR to approximately satisfy these same properties (including a theorem that such an INR exists). Since the optimal kernel itself is not identifiable from a discrete dataset, I would encourage the authors to relax some of the language around alignment with the optimal kernel since it seems the proposed method is instead endowing the INR with some properties that are also shared by the optimal kernel. Nonetheless, the theoretical contribution is valuable as most INR architectures make no attempt at theoretical motivation or characterization. Figure 2 does also validate in a toy setting that this approximation can induce an INR to mimic the optimal kernel in a setting where the optimal kernel is known and computable, which is a compelling illustration of the main idea.\nSome notation is not clear. Specifically:\n- What is A? In Theorem 3.3 A would seem to be a subset of the real numbers reflecting the range space of a kernel. Then A appears in section 3.4 where it would appear to be a function of two arguments, sometimes bold and sometimes not bold. From context my guess is that the bold version is a vectorized version of the non-bold version of A, but that neither A in section 3.4 is related to the A in theorem 3.3. Eventually the bold function A is defined in line 266 (right column), but this is after the reader has been seeing it without definition for almost a page. It would be preferable to explain at first use that A is a function to be learned, that must satisfy certain properties.\n- The embedding function gamma is sometimes denoted with the internal parameters a_j listed explicitly, and sometimes with these parameters implicit in the function. This is a minor issue but at first glance can make it difficult to see that these are all the same function. Since gamma is also introduced and described before the practical choice for its definition/parameterization is given (in line 257, right column), I would suggest mentioning at first use that gamma is, like A, a function to be learned that must satisfy certain properties.\n- The circled plus notation is used (e.g. on line 247, left column) without definition. The same notation can be used for multiple operations (e.g. direct sum, exclusive or, and dilation) so it warrants precise definition here.", "experimental_designs_or_analyses": "Please refer to my comments on methods and evaluation criteria. Specifically, I’d like to see some comparison or discussion of relative model sizes and training/evaluation times in addition to the provided comparisons of quality. I encourage the authors to release their code (upon publication, if not before).", "relation_to_broader_scientific_literature": "Related work is adequately discussed.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A; see the main review.", "other_comments_or_suggestions": "Figure 1 is difficult to understand. Figure 1(a) could be improved by plotting with alpha<1 so that overlapping dots/lines can be distinguished. Figure 1(b) is not clear what it is trying to show (what do arrows mean? What are the level sets showing? What is the black triangle? etc.).\n\nOverall the writing is clear, though there are occasional typos/minor grammatical mistakes. E.g. the sentence on line 214 (left column) is not a complete sentence, though the meaning is still clear.", "questions_for_authors": "My primary question for the authors is about the model sizes and training/inference times for each of the methods compared, to determine whether the improved performance comes at a cost. My leaning towards acceptance assumes that this information will be shared during the rebuttal/revision.\n\nI also have some questions about notation in the theoretical contributions; these can be addressed by revising the exposition there.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VSY0YiTNYy", "reviewer": "Reviewer_i5ua", "summary": "This paper first established a theoretical insight that the neural tangent kernel of implicit neural representation can approximate any positive semidefinite dot-product kernels. Developing on this insight, the paper propose a kernel alignment regularizer to improve the INR system. Experiments show the proposed method performs better than baseline methods in image reconstruction and phase retrieval tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**All the Table/Figure Rx can be found in https://anonymous.4open.science/r/ICML-Re-3333**\n\n**Q1:**  The proposed method should be applied to other more challenging real-world problems.  \n**A1:** Thank you for your valuable suggestion. We have conducted additional experiments on  more complex problems, specifically focusing on 3D neural fields and comparing our methods with Instant-NGP, as NeRF can be time-consuming. We performed experiments on the NeRF Synthetic datasets with 25, 50, and 100 view perspective samples. As shown in [Table R3](https://anonymous.4open.science/r/ICML-Re-3333/Table_R3.pdf), the average PSNR for 25 view perspectives indicates that the PEAK algorithm outperforms the vanilla Instant-NGP. Notably, the improvement is more pronounced with fewer training samples, demonstrating that PEAK effectively enhances generalization capability. Furthermore, [Figure R3](https://anonymous.4open.science/r/ICML-Re-3333/Figure_R3.pdf) shows that PEAK reduces artifacts caused by sparser samples, indicating its ability to leverage the internal structure of the data to improve performance in downstream tasks with limited data. We will include this 3D experimental data in the revised manuscript to further demonstrate the effectiveness and generalizability of our method.", "claims_and_evidence": "110-114: The neural tangent kernel associated with an INR (which is a multi-layer perceptron) captures the evolution of network predictions during training. So by regularizing the NTK, we can control the training of INR. With this insight, the paper introduces a regularizer Eqn. (11) and claims this regularizer can improve the INR. \n\nThe evidence is both theoretical and empirical. The paper shows the proposed regularized loss function yield better performance in image reconstruction and phase retrieval.\n\nThe experiments indeed support the claim made by the paper. However, I personally consider the experiments to be a little simple. The experiments are simple tasks and mainly synthetic. If the proposed method can be applied to other more challenging real-world problems, the paper will be much stronger.", "methods_and_evaluation_criteria": "The method proposed in the paper is simple and effective, and the theoretical foundation behind the regularization loss function is strong. So I believe the motivation and justification of the proposed method is valid. The proposed method is evaluated by two tasks and shows good results. But as mentioned in previous sections, the tasks seem to be too simple and work easily.", "theoretical_claims": "I didn’t carefully check the proof, while the overall idea makes sense to me.", "experimental_designs_or_analyses": "As mentioned in the previous sections, the considered tasks seem to be too simple.", "relation_to_broader_scientific_literature": "The proposed method seems to have a strong relation to reconstruction tasks, which are the foundation of many important tasks, such as unsupervised learning through reconstruction, image super resolution, etc.. From this perspective, the paper provides new insights into a fundamental problems, so I consider this as a significant contribution.\n\nIt would be much stronger if the proposed method is shown to be also effective in some real-world tasks, rather than only the synthetic tasks presented in this paper.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "No", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper develops a theoretically grounded approach to improving implicit neural representations (INRs) via kernel alignment considerations, combining a Kernel Alignment Regularizer (KAR) and a learnable encoding strategy (PEAK). The reviewers agree that the core idea—harnessing Neural Tangent Kernel (NTK) analysis to guide design choices for INRs—is both original and well-motivated. They also commend the thorough theoretical underpinnings, alongside clear empirical gains on image restoration and phase retrieval tasks. Early concerns included the need for broader baseline comparisons, requests for more complex tasks beyond toy and 2D settings, and clarifications on whether the authors had properly cited related theoretical results. In their rebuttal, the authors provided additional experiments, compared against extended sets of recent methods, clarified that the theoretical results align with known NTK findings, and improved the manuscript’s explanations of key notation and motivations.

Overall, the reviewers now lean in favor of acceptance. While a few felt that the original draft could have used additional experiments and clearer presentation of theoretical references, the authors’ revisions and rebuttal materials have addressed most concerns. The consensus is that the paper makes a solid contribution to the literature, with strong theoretical grounding and evidence of practical benefits. The authors are encouraged to incorporate the newly provided experiments and clarifications into the final version for better readability and completeness.
Decision: Accept (poster)