On the Spectral Differences Between NTK and CNTK and Their Implications for Point Cloud Recognition
Authors: Yuanqu Mou
Chang Gou
Haiyang Bai
Jia Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Neural Tangent Kernel
Interpretability of neural networks
Primary area: interpretability and explainable AI

Abstract: The Convolutional Neural Tangent Kernel (CNTK) offers a principled framework for understanding convolutional architectures in the infinite-width regime. However, a comprehensive spectral comparison between CNTK and the classical Neural Tangent Kernel (NTK) remains underexplored. In this work, we present a detailed analysis of the spectral properties of CNTK and NTK, revealing that point cloud data exhibits a stronger alignment with the spectral bias of CNTK than images. This finding suggests that convolutional structures are inherently more suited to such geometric and irregular data formats. Based on this insight, we implement CNTK-based kernel regression for point cloud recognition tasks and demonstrate that it significantly outperforms NTK and other kernel baselines, especially in low-data settings. Furthermore, we derive a closed-form expression that connects CNTK with NTK in hybrid architectures. In addition, we introduce a closed-form of CNTK followed by NTK, while not the main focus, achieves strong empirical performance when applied to point-cloud tasks. Our study not only provides new theoretical understanding of spectral behaviors in neural tangent kernels but also shows that these insights can guide the practical design of CNTK-based regression for structured data such as point clouds.

Published as a conference paper at ICLR 2026
ON THE SPECTRAL DIFFERENCES BETWEEN NTK AND CNTK AND THEIR IMPLICATIONS FOR POINT CLOUD RECOGNITION
Yuanqu Mou
Chang Gou
Department of Computer Science Nanjing University mouyqesma 1 ju. edu. cn
Nanjing University egou8smail nju.. edu.cr en
Haiyang Bai
Nanjing University baihy@smail nju. edu. on
Jia Liu
State Key Laboratory for Novel Software Technology Nanjing University
ABSTRACT
The Convolutional Neural Tangent Kernel (CNTK) offers principled framework for understanding convolutional architectures in the infinite-width regime. However, a comprehensive spectral comparison between CNTK and the classical Neural Tangent Kernel (NTK) remains underexplored. In this work, present detailed analysis of the spectral properties of CNTK and NTK, revealing that point cloud data exhibits stronger alignment with the spectral bias of CNTK than images. This finding suggests that convolutional structures are inherently more suited to such geometric and irregular data formats. Based this insight, we implement CNTK- -based kernel regression for point cloud recognition tasks and demonstrate that it significantly outperforms NTK and other kernel baselines, especially in low-data settings. Furthermore, we derive closed-form expression that connects CNTK with NTK in hybrid architectures. In addition, we introduce a dlosed-fomm of CNTK followed by NTK, while not the main focus, achieves strong empirical performance when applied to point-cloud tasks. Our study not only provides new theoretical understanding of spectral behaviors in neural tangent kernels but also shows that these insights can guide the practical design of CNTK-based regression for structured data such point clouds.
INTRODUCTION
Neural Tangent Kernel (NTK) Jacot et al. (2018) emerged as a critical theoretical framework for understanding the training dynamics and generalization behaviors of deep neural networks Golikov ct al. (2022). Among its variants, the Convolutional Neural Tangent Kernel (CNTK) Arora et al. (2019a) provides kernel-based perspective convolutional neural networks (CNNs), enabling precise analysis of their inductive biases and convergence properties. Building on this foundation, existing studies Geifman al. (2022 Cagnetta al. (2023Barzilai Barzilai al. (2022 & Mairal (2019) have explored the properties of CNTK as a kernel and extended CNTK in various directions Li et al. (2019). However, there has been lack of discussion the differences between CNTK and NTK when applied to the same dataset. This paper aims to address this theoretical issue by investigating the following question:
"Corresponding author. The authors contribute equally this work.

---

Published as a conference paper at ICLR 2026
Under given data assumption, what the spectral differences between NTK and CNTK?
To address this issue, it is first necessary to consider an appropriate data distribution. Existing studies often assume that data is uniformly distributed account the inherent tensor structure of the data. In this paper, we assume that the + 1-dimensional data is distributed with any distribution over the d-dimensional tensor space (with generally non- diagonal covariance matrix), and insignificant across the channel dimension (though not necessarily independent). Under this assumption, we identify two major differences between NTK and CNTK when evaluated on datasets drawn from the same distribution:
high-dimensional sphere, without taking into
(I) For any I >> O, the mean of the eigenvalues mk under L-layer NTK is greater than that under L-layer CNTK, indicating that NTK tends to have larger eigenvalues on average.
(II) For any L, L' O, the spread 3K of the eigenvalue distribution under any L-layer NTK is smaller than that under any L'-layer CNTK, suggesting that NTK exhibits more concentrated spectrum compared to CNTK.
Observation (II) explains why convolutional networks generally achieve better generalization than fully connected networks since the generalization risk can be bounded by the varianc(Bartlett et al.) (2020), while (I) provides theoretical justification for the empirical design choice of adding a few MLP layers after pooling in deep convolutional networks.
Based on this finding, this paper addresses practical application problem:
How do MLPs (NTK) and convolutional networks (CNTK) differ in their influence on data with different levels of relevance?
To address this issue, we marked the metric 0g aligned with the suitability of the convolutional structure for data analysis. By fitting a dataset to our assumed distribution, 3x quantifies the degree to which NTK exhibits a more concentrated spectrum compared to CNTK, as described observation indicator of how differently MLPs and convolutional networks are
(II). This value serves expected to perform on the given data.
In addition, most existing studies CNTK are limited to the commonly used two-dimensional case, with little discussion on more general dimensional settings. Therefore, this paper extends the analyzed the propagation
analysis to convolutional kernels of arbitrary dimension. Furthermore, of CNTK-NTK kernel computations in convolutional networks followed by nonlinear fully connected layers, which is empirical setup. In this work, such setup can be interpreted leveraging the broad spectrum of CNTK eigenvalues (II) and the bias of NTK towards larger shifts (I).
Specifically, as case study, we find that point cloud tasks rely more heavily on convolutional structures compared to image tasks. Motivated by this, we further investigate kernel regression based on one-dimensional convolutions in the context of point cloud tasks.
1.1 CONTRIBUTION
We considered the notation and framework of CNTK under general dimensions that can be connected to NTK. Within this framework, we revealed the spectral differences between NTK and CNTK that are distribution-fres Furthermore, based on the theoretical findings, we proposed the concept of "Convolutional Suitability" for data analysis, which measures how suitable given dataset is for convolutional networks. Finally, we applied CNTK to point cloud tasks for the first time, including kernel regression experiments based CNTK and the application of our theoretical conclusions to point cloud datasets.
1.2 NOTATION
We assume that each point x; comes from a d-order tensor space R! where h; denotes the length along the i-th dimension. Each sample has no channels of such point, forming data point x {4i, T2, ,,,,,..,,,,, } For such tensor data, we do not use boldface notation. In this paper, only data constructed from IR T xI
R d are written in bold to distinguish them. Define the index set
p. q E [[h,]a]. where [h,] = (1, hr), and [[h,]a] = [hy] x [h2] x .... x [ha], with x denoting the Cartesian product. The length of the convolutional kernel is defined (For simplicity, assume
2

---

Published as a conference paper at ICLR 2026
that the convolutional kernel is square.) (p) is denoted as convolution kernel indices centered p. that is, (p) = {[p - P2 + 1 :....... Pa L Pa + [1]
For an activation function o(x), we denote its derivative as 6(x)), and its dual function with respect to a distribution D as ((D)2,,'~D _[o(z) We define @ the Schur (element-wise product, @ as the Kronecker (tensor) product, and as scalar multiplication or standard matrix-matrix multiplication.
CONVOLUTIONAL NEURAL TANGENT KERNEL
In this section, we provide brief introduction to the Convolutional Neural Tangent Kernel (CNTK). For the purpose subsequent generalization, the definition is given in the context of inputs with arbitrary dimensions. not restricted to the two-dimensional case. Following the recursion in Jacot
et al. 2018) Arora et al. (2019)), CNTK for unified CNNs is defined as follows.
Definition 1. Given two samples x' E Rill let p. 9 be a position in [[hs]al with neighbor-
hood N (p). Define the covariance matrix, 0-order expectation and 1-order expectation of f( (x)
and (0) (x") follows:
A 1 (x, = (2,x) Epg (2,")((,x)) e (z',z) Epa e
Epa
211 (x, E (w,)) (0.4 (d())(()).
(1)
( (ww.) (dw))(o))
with recursion for Ep. (r, x²) is:
el)
e(l) c (x,,') tr tr (2g
Define the trace of tensor T E &R R by x as tr(T) = -Epe[(alal Tp.. Recursive foundation is:
(I, x 22 2z.
p.g
When the convolution dimension is d, there will also be interactions between the elements within the tensors at (-layer's 's neurons. For any two samples x, x', this relationship is described by the three tensors A (z, x') E A x x2x2 (x, z²), (xx,'') E IR A XT hy in the NTK field, as mentioned above. Consequently, we can derive the specific recursive formula for CNTK.
Lemma 1. Given two samples x' E hil and position p with its neighbourhood, N(p). The CNTK without pooling is computed as:
K Koop (x, x') ) tr (K (z,
(4)
the CNTK with the global average pooling Pav is computed as: Kup(I, x') Pav (K(4)
(5)
in which K00 xd) is l-layer tensor kernel with recursion for 51 L and p-entry as:
K!l) p.g (I, x') tr (K ( ((1-)) S (x, + p.g
(2,=)(x,x))
(6)
The derivation details provided in the original research al. (011a). Nevertheless, we also provide a derivation in the ceneral-dimensionall case in Appendix B. Note that when we consider the case = 0, corresponding to the structure of MLPs, we have p fixed and N(p) p, which recovers the standard NTK formulation. By default, we consider the CNTK with pooling layers included in the following.
3

---

Published as a conference paper at ICLR 2026
3 CNTK FOLLOWED BY NTK
While the Convolutional Neural Tangent Kernel (CNTK) captures the infinite-width behavior of vanilla convolutional networks, it does not account for architectures commonly used in practice, where a shallow MLP is typically appended after the convolutional backbone. In this section, derive a closed-form expression of the CNTK for a more general architecture: dj -immensionnl convolutional network followed by dd-diimensional convolutional (or fully connected when d2 0) network.
Generally, let the network topology be determined by two sets: {L1, I2} and d2), where d1 > d2. This represents a network consisting of L1 layers of di-dimensional convolutional layers and L2 layers of dz-dimensional convolutional layers, connected by average pooling layer. We can then propose the combined dimension CNTK formally as follows:
Proposition 1. Suppose d1-dimension sub convolutional neural network fi Rlllt, hi xmb 7 RR x and d2 di-dimnnsion sub convolutional neural network fz R M3 x 1 with d1 > d2, ni, = no and h3 = h for E [d2], generalizing convolutional neural network fa 1 xid R IT M x is defined as:
fa(x) = f2 (Pd (Pdy-da (fi(z)))
in which Pdi 2i -1 is local average pooling when fo is standard initialization and which, ni, mz..... 11 - n2. + oc, its corresponding kernel is:
Ke(x,x') Pav
(8)
at the point of connection, for any samples x, x²:
2xh++) = pdi- da (I(LL)
(9)
In fact, such structure can be applied more than once. In practical scenarios, convolutional networks are typically connected to only few MLP layers, and therefore we usually have d2 = 0, which is named CNTK-NTK in the following.
4 SPECTRAL COMPARISON
In this section, we formally present our results about the spectral properties of CNTK and NTK. For d-dimensional tensor space, consider data sampled from ". xno and following H- dimensional distribution (can be any distribution) with covariance matrix a in which H IT 1 h4- We consider convolutional networks with kernel size 1 and average pooling layers.
4.1 THEORETICAL RESULTS
Let CNNT denote the CNTK of an I--layer convolutional neural network, and K K(L)) NTK denote the NTK of an L-layer MLP. We define the statistics about eigenvalues of kernel matrix K as: (
(-L))
mk tr(((())) SK tr(((( K(L)T and 3((2) (m)))) These three quantities reflect
(L)
certain statistical characteristics of the eigenvalues of the kernel matrix: mk denotes the mean of the cigenvalues, 8 represents the mean of the squared eigenvalues, and 3x indicates the dispersion of the eigenvalues. Formally, based on results from matrix analysis literature Cai et Homm & Johnson| (2012), we have:
Lemma 2. Given N x N positive definite matrix K, let its set of eigenvalues be denoted as A E d2 S S..S&A AN, then:
N
N
Amaz Amin
mK =
Ai
SK =
X2
2 NBK.
(10)
N
N2
i=1
4

---

Published as a conference paper at ICLR 2026
0.
Figure I: Experiments on synthetic data, & denotes the correlation of the multidimensional Gaussian distribution and N refers to the Gaussian distribution W. L represents the number of network layers, where L = 1 corresponds to the neural tangent kernel (()) without passing through any network. In the right figure: in the upper part, is always set to 0.5; in the lower part, I. is fixed 5. "CNTK-NTK" refers to the CNTK of depth I followed by an additional two-layer MI..
Furthermore, 1/3k is essentially the same the quantity effective rank defined in Bartlett al. (2020). which was shown to upper-bound bound the generalization error of kernel regression. In our analysis, use 3g as comparative metric assess how NTK and CNTK differ in their generalization ability on data drawn from arbitrary distributions, thereby shedding light on what kinds of data henefit more from convolutional structures. In addition, mk, convergence capacity of the network. Therefore, we also include a discussion on this aspect.
offset term of the kernel matrix, reflects the
Theorem 2. Suppose that the activation function is Relu. When N + oc, the the following holds:
(L) MNTK Z MCNTK-
(L)
(11)
Remark 1. For this conclusion, we do not require the data to come from any specific distribution, nor do assume that N tends to infinity. In addition, the activation function doesn' 't need to be ReLU; is sufficient that the dual of the activation function is non-convex with respect to the diagonal entries. Therefore, the conclusion is general. In fact, relatively strong statement used in the proof K(L)
process-namely, for any I E R TT xno, we have KNTK (VTK(1,2) 2 KCNTK (I. z).
Theorem 3. For any L-layer CNTK and L'-layer CNTK, under any distribution of and any activation function. When N + cc, the following holds:
(12) Remark 2. In fact, we can prove that for CNTK, PLEX = holds for any layer L/, while for NTK, 3 for every layer. This follows purely from the positive semi-definiteness of the covariance matrix under random initialization, and thus holds universally without any assumption on the data distribution.
BNTK S SPNTK =
gEE)
In the analysis of NTK, the magnitude of eigenvalues affects the convergence speed of neural networks Du al. (2018). The value of mı can reflect the convergence speed the network. Meanwhile, Sx indicates the spread of the eigenvalues of the kernel function; small eigenvalue range suggests that the matrix is overly diagonalized, and and/Bettettt al. (2020) has shown that overly diagonalized kernel matrices often indicate poor generalization ability. The above theorem explains why convolutional networks tend to generalize better than MIPss, and why appending fully connected layers after deep convolutional networks accelerate training while maintaining generalization performance.
Proposition 4. Suppose that covariance matrix of x is σh, when N + oc:
m NTT K = mean(diag(
MCNTK = mean(ah)
(13)
NTK mean(a @ HH),
(0)
SCNTK = mean(aH & aH)
(0)
(14)
(()) 8
mean (o @ оH)
2(0) BONTK = 1. (15)
PTK mean(diag( ( diag(on))
5

---

Published as a conference paper at ICLR 2026
Figure 2: Histograms of Eigenvalues on:(a). Point cloud datasets ModelNet10g, ModelNet10, and ModelNet40; (b) Image datasets CIFARIO and MNIST.
Table 1:
Kernel regression experiments conducted on both the point cloud and image datasets, 1dCNTK is applied to the point cloud dataset, while 2dCNTK is utilized on the image dataset. For all experiments, set the number of layers to L =.
Datasets
ModelNet10g
ModelNet10g
ModelNer40
MNIST
CIFAR10
NTK
17.58
16.19
11.10
94.00
59.19
ldCNTK(2(CNTK)
76.65
91.96
60.62
96.00
76.79
Improvement
59.07
75.77
49.52
2.00
17.60
3K
1.25e-3
1.12e-3
1.53e-3
3.38e-2
1.37e-1
((0) is obvious that TTK 2 m CNNTK: $ &NTK 2 SCNTK and BNTK I PCNTK- The equality of all the formulas holds if and only if H is constant matrix.
(0)
(0)
(
This proposition suggests that the shift and distribution of CNTK and NTK eigenvalues can be described by the covariance matrix defined over the tensor input. Although the proposition formally addresses only the initial layer, our simulation experiments indicate that for CNTK and NTK with the same depth (see Figure T). their spectral characteristics can still be effectively captured by the covariance structure of the data.
Based on this proposition, for data sampled from the same distribution, the spectral distribution the NTK cigenvalues is associated with its covariance matrix. We define B measure of the MLP fitness of the data, quantifying how well the data aligns with an MLP architecture. the context of the tensor data considered in this paper, P NTK indicator of how suitable the data for convolutional networks (i.e. Convolutional Suitability).
(U) 1 / can also be interpreted as an
In our simulation, we use data sampled from a Gaussian distribution to analyze the spectral properties of the NTK and CNTK matrices, illustrated in Figure The parameter controls the global correlation of variables, with the covariance matrix given by = tsC + (1 t)s with unit diagonal matrix I and unit constant matrix C' of scale = 0.1. The observed results align precisely with our theoretical predictions.
4.2 IN REAL SCENARIOS
As application of the invariance of 3x under CNTK and its relevance NTK, use 3(0) as an indicator to analyze a given dataset. Assuming the data is drawn from a Gaussian distribution, we conduct kernel regression fitting for both point cloud recognition (with ID convolutions) and image classification (with 2D convolutions). As shown in Figure 2 and Table 1, the spectral gap between CNTK and NTK more pronounced in point cloud tasks than in image datasets. The table further indicates that point cloud tasks intrinsically benefit more from convolutional models- both theoretically, as suggested by smaller values of 3K. and empirically, as confirmed by kernel regression experiments. The inherent unorderedness of point clouds results in weak inter-sample correlations within the tensor space, which in turn yields a near- diar-diaaonal covariance structure and thus lower 9x- Conversely, image data typically exhibits stronger structural correlations, leading to higher values of 3x. This observation implies that convolutional architectures are particularly crucial for effectively modeling point cloud data.

---

Published as a conference paper at ICLR 2026
Ar1
Axc
60-
11
2002
2583 2000
1330
2200
1008 2000
2000
63.00
3300
5000
7003
3000
(9)
Figure 3: In the visualizations, 'pnet' denotes PointNet, 'pnetm' represents PointNetm, and 'pntk' corresponds to PointNTK. Supervised training results are represented with dashed lines. For each dataset, five independent experiments were conducted, and the mean results are reported. The shaded regions indicate the range of outcomes from these five experiments. Each dataset's plot features a subfigure the lower-left corner, showing results for training with a small sample size, and a subfigure in the lower-right comer, illustrating results for training with large sample size.
KERNEL REGRESSION FOR POINT CLOUD RECOGNITION
This section begins our discussion of point cloud tasks. Point cloud tasks were initially introduced by PointNet Qi et al. (2017a), which empirically explored point cloud-based tasks. Given the stronger necessity of convolutional architectures for point cloud data, it is noteworthy that current kernel regression methods rarely focus this domain. We posit that CNTK-based kernel regression offers significant promise for point cloud applications. In the following, we provide an in-depth analysis and empirical evaluation of both CNTK and the combined CNTK-NTK kemnlss point cloud regression tasks.
5.1 EXPERIMENTAL SETUP
It was found that encoders based shared-MLP outperform typical MLPs. The shared-MLP in PointNet essentially corresponds to a one-dimensional convolution with a convolutional kernel size of in our framework. In practice, shared-MIJ typically follows structure in which few layers of MLP are added only after the pooling layers. Many practical convolutional networks connect MLP layers after pooling, which is precisely the purpose of introducing CNTK-NTK. For point cloud recognition, we fully adopt the PointNet architecture to propose the PointNTK, an instantiation of CNTK-NTK, as set up which L1 L2 3 and d1 1, d2 0.
We applied the discussed kernel in the previous section to actual point cloud classification datasets. In the following experiments, we utilized three datasets from ModelNet10 and ModelNer40. The input of ModelNet10 includes both point cloud positional information and tiree-dimensional orientation information. We refer to the experiment using only the point coordinate information from ModelNet10 as ModelNet10g, while the experiment using the complete original data of ModelNet10 is referred to as ModeINet1Og. Details of the experimental setup are provided in the final subsection of this section.
5.2 POINTNTK VERSUS POINTNET
In this section, compare PointNTK with PointNet, where PointNet maintains all its original settings, including max pooling. Both PointNTK and PointNet are seven-layer neural networks. The first four layers one-dimensional convolution layers with kernel size of 1, followed by three MI.P layers.
Additionally, we introduce the following comparisons:
PointNet,: A variant of PointNet with differences: The training loss is replaced with the least-squares loss; II. The pooling layer is replaced with average pooling. Both networks retain their original batch normalization (BN) layers.
7-layer 1dCNTK and 7-layer NTK: These models serve as additional baselines for evaluation, which represents single-CNTK compared with CNTK-NTK. We set their depth to 7 in order
7

---

Published as a conference paper at ICLR 2026
Table 2: Performance on ModelNet1Os, ModelNet10g and ModelNet40 of various structure. The superscript 'f' indicates that only 100 samples from the dataset were used the training set. The superscript 'M' and 'L' indicate 'medium' and large with depth 9 and 15 layers respectively for Id convolution layers.
Datasets
ModeINet10, ModelNet10g
ModeINer10, ModelNet10;
ModelNet40 ModelNet40
PointNet
91.12 + 1.34
68.92
1.05
92.08
1.75
70.22
0.58
88.71 0.54
38.32 0.97
PointNet
89.98 + 2.38 28.22
1.56
90.43
2.01
27.97 = 1.85
83.93 1.67 5.17
1.22
NTK
17.58
11.12
16.19
11.78
11.10
3.69
1dCNTK
76.65
71.03
91.96
81.16
60.62
45.22
PointNTK
86.56
73.68
91.19
78.52
80.47
45.71
91.98 + 0.98
69.58 + 0.62
92.42 = 0.85 67.33 + 2.65 86.93 + 1.16 30.28 + 1.42
PointNet
90,25 + 1.87
13.07 +: 2.05
PointNTK
M
87.56
74.23
92.40
80.04
81.93
47.77
P
91.51 + 0.87
56.52 + 0.37
91.53 + 1.89 57.39 + 3.42
85.32 + 1.20 23.62 + 0.52
88.54 + 2.56
7.43 + 2.39
PointNTK
L
87.67
74.01
92.62
82.38
82.50
47.77
Modelfier10,
Model
Modelleet00
ACc
Acc
Acc 57
L
74
Arc 14
Acc
45
(B)
Figure 4: The ablation experiments on kernel regression based on NTK families across three datasets: the upper part shows the results for different network depths, while the lower part shows the results for different convolution kernel lengths.
to match the depth of PointNTK, which consists of 4 layers of ID convolutional layers and fully connected layers.
We conducted experiments on three datasets, where the superscript f indicates that only 100 fixed samples from the dataset (covering all categories) were used for training, while the entire test set was used for evaluation. The results are summarized in Table 2, with the observations: I.PointNTK outperforms single-layer 1dCNTK with same number of layers; PointNKK performs slightly worse than PointNet and PointNet,, ModelNet10g; significantly underperforms ModelNet40 and ModelNet10g, but outperforms the PointNet and PointNet,, with small-training set in all datasets.
To further analyze this, we conducted experiments showing how the accuracy of kernel regression- based methods and training- baased methods varies with the size of the training set, as illustrated in Figure
5.3 ABLATION STUDY FOR KERNEL SIZE AND DEPTH
In PointNet, the defined shared- is essentially one- dieeennniiooal convolution with kernel size of 1. This concept was also adopted in the kernel regression methods in our earlier experiments. In this section, we perform ablation experiments the kernel size and network depth within NTK-based
8

---

Published as a conference paper at ICLR 2026
kernel regression methods. We conducted experiments on three datasets, using only fixed 500 samples from each dataset for training while testing on the entire test set.
Depthh ablation. We conducted several experiments follows.
1dCNTK: Uses only one-dimensional convolutions for the kernel with depth "Z + 3". We add 3 to keep it consistent with the number of layers in PointNTK, which holds 3-layer MLPs.
PNTK, and ; : PNTK, fixes the first three one miedimensioonl convolutional layers and varies the depth of the subsequent fully connected layers; PNTK, fixes the last three fully connected layers and varies the depth of the preceding one-dimensional convolutional layers.
PN Simultaneously increases the depth of both parts.
From results in the upper part of Figure we observe the following: The design of PointNTK is highly effective. For structures using only one-dimensional convolutions, increasing depth does not yield significant improvements and may even degrade performance: II. Adding fully connected layers at the end of the one- diee-iiensiinnl convolutional structure improves performance across all datasets when the convolutional depth increased; III. Fixing the depth of the one-dimensional convolutional layers while increasing the depth of the fully connected layers reduces regression performance; IV. Simultaneous depth increases in both parts are less effective than increasing the depth of the one-dimensional convolutions alone. Since increasing the depth of the ID convolutional layers in PointNTK improves accuracy, we naturally hypothesize that the same might be true for PointNet. Finally, further compared the results of deeper kernel regression with supervised training, as shown in Table 2.
2) Kernel size ablation. For kernel size as shown in the lower part of Figure 3, all datasets consistently indicate that a kernel size of is optimal. This is intuitive because point clouds are unordered, and when the kernel size exceeds 1, the points associated with the kernel may not be meaningfully related.
5.4 EXPERIMENTAL DETAIL
All the experiments have been implemented using Python 3.7 and Pytorch 1.8 on one NVIDIA RTX 3090 super 24GB. In addition, neither data augmentation nor post-processing (besides global normalization) was applied, which was followed by each experiment. For kernel regression, com- putations were implemented using CUDA code. For supervised training methods, we adopted the parameter settings from PointNet. In small-sample experiments, we fix a subset of training set for all experiments, and all tests were conducted. We note that when the input data remains unchanged, the results of kernel regression are deterministic and unique.
6 CONCLUSION
In this work, we investigate the spectral differences between NTK and CNTK eigenvalues data drawn from an arbitrary distribution. We show that CNTK consistently exhibits a broader eigenvalue distribution and, on average, smaller eigenvalues compared NTK. This provides theoretical evidence for the superior generalization ability of CNTK over NTK. Building on this insight, we define a data-dependent suitability measure based on the spectral range of the first-layer CNTK and NTK, which depends solely on the covariance matrix of the data. Our experiments reveal that point cloud tasks benefit more significantly from convolutional structures. Motivated by this, conduct extensive kernel regression experiments on point cloud datasets and demonstrate that our proposed strategy, which combines CNTK and NTK, is particularly effective this setting, filling an existing gap in CNTK performance point cloud tasks.
There are still some limitations in our work. While our main theoretical results are derived without assuming any specific data distribution, computing 3x for arbitrary L-layer NTK does require distributional assumptions. As a result, in practical applications, we rely on the first layer to evaluate data suitability. However, the computation of spectral properties of L-layer NTK and L-layer CNTK hard. Future investigations into its upper and lower bounds may yield more precise spectral characteristics of the data at layer T..

---

Published as a conference paper at ICLR 2026
7 ACKNOWLEDGEMENT
This work was supported in part by the Jing-Jin-Ji Regional Integrated Environmental Improvement- National Science and Technology Major Project (2025ZD 1208203), the National Natural Science Foundation of China (62332013), and the Collaborative Innovation Center of Novel Software Tech- nology and Industrialization.
REFERENCES
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with infinitely wide neural net. Advances in neural information processing systems, 32, 2019a.
Sanjeev Arora, Simon S Du. Zhiyuan Li, Ruslan Salakhutdinov, Ruosong Wang, and Dingli Yu. Har- nessing the power of infinitely wide deep nets on small-data tasks. arXiv preprint arXiv: 1910.0163, 2019b.
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(44)):0063-30007) 2020.
Daniel Barzilai, Amnon Geifman, Meirav Galun, and Ronen Basri. A kernel perspective of skip connections in convolutional networks. arXiv preprint 1rXiv:2211.144 14810, 2022.
Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. Advances in Neural Information Processing Systems, 32, 2019.
Francesco Cagnetta, Alessandro Favero, and Matthieu Wyart. What can be learnt with wide con- volutional neural networks? In International Conference Machine Learning, pp. 3347-3379. PMLR, 2023.
Zhicheng Cai, Hao Zhu, Qiu Shen, Xinran Wang, and Xun Cao. Batch normalization alleviates the spectral bias in coordinate networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 25160-25171, 2024.
Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. arXiv preprint 1 2021a.
Yilan Chen, Wei Huang, Lam Nguyen, and Wei Weng. On the equivalence between neural network and support vector machine. Advances in Neural Information Processing Systems, 34: 23478 23478-23490, 2021b.
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference machine learning, pp. 1675-1685. PMLR, 2019a.
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterizzed neural networks. arXiv preprint arXiv.1810.02054, 2018.
Simon Du, Kangeheng Hou, Russ R Salakhutdinov, Barnabas Poczos, Ruosong Wang, and Keyulu
Xu. Graph neural tangent kernel: Fusing graph neural networks with graph kemnlss, Advances in neural information processing systems, 32, 2019b.
Ethan Dyer and Guy Gur-Ari. Asymptotics of wide networks from feynman diagrams. arXiv preprint arXiv: 1909. / 1304, 2019.
Andrew Engel, Zhichao Wang, Anand D Sarwate, Sutanay Choudhury, and Tony Chiang. Torch- ntk: A library for calculation of neural tangent kernels of pytorch models. arXiv preprint arXiv:2205. 12372 2022.
Jean-Yves Franceschi, Emmanuel De Bézenac, Ibrahim Ayed, Mickaël Chen, Sylvain Lamprier, and Patrick Gallinari. A neural tangent kernel perspective of gans. In International Conference on Machine Learning, pp. 6660-6704. PMLR, 2022.
10

---

Published as a conference paper at ICLR 2026
Amnon Geifman, Meirav Galun, David Jacobs, and Basri Ronen. On the spectral bias of convolutional neural tangent and gaussian process kernels. Advances Neural Information Processing Systems, 35:1 1253- 11265, 2022.
Eugene Golikov, Eduard Pokonechnyy, and Vladimir Korviakov. Neural tangent kernel: survey. arXiv preprinl arXiv:2208. 13614, 2022.
Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pet: Point cloud transformer. Computational Visual Media, 7:187-199, 2021.
Bobby He, Balaji Lakshminarayanan, and Yee Whye Teh. Bayesian deep ensembles via the neural tangent kernel. Advances in neural information processing systems, 33:1010--022,, 2020.
Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.
Jiri Hron, Yasaman Bahri, Jascha Sohl-Dickstein, and Roman Novak. Infinite attention: Nogp and ntk for deep attention networks. In International Conference on Machine Learning, pp. 4376-4386. PMLR, 2020.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Jaehoon Lee, Samuel Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein. Finite versus infinite neural networks: an empirical study, Advances in Neural Information Processing Systems, 1516-111117.. 2020.
Zhiyuan Li, Ruosong Wang, Dingli Yu, Simon S Du, Wei Hu, Ruslan Salakhutdinov. and Sanjeev Arora. Enhanced convolutional neural tangent kernels. arXiv preprint 1:11vv9911.0000 2019.
Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Evolution neural tangent kernels under henign and adversarial training. Advances in Neural Information Processing Systems, 35: 11642-11657, 2022.
Mohamad Amin Mohamadi, Wonho Bae, and Danica J Sutherland. Making look-ahead active learning strategics feasible with neural tangent kernels. Advances in Neural Information Processing Systems, 35:12542-12553 2022.
Jisoo Mok, Byunggook Na, Ji- Hi-Hoon Kim, Dongyoon Han, and Sungroh Yoon. Demystifying the neural tangent kernel from practical perspective: Can it be trusted for neural architecture search without training? Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 11861-11870, 2022.
Charles R Qi. Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660, 2017a
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. Advances in neural information processing systems, 30, 2017b.
Maxim Samarin, Volker Roth, and David Belius. On the empirical neural tangent kernel of standard winite-width convolutional neural network architectures. arXiv preprint arXiv:2006. 13645, 2020.
Mariia Seleznova, Dana Weitzner, Raja Giryes. Gitta Kutyniok, and Hung Hsu Chou. Neural (tangent kernel) collapse. Advances Neural Information Processing Systems, 36, 2024.
Haozhe Shan and Blake Bordelon. A theory of neural tangent kernel alignment and its influence on training. arXiv preprini arXiv:2105. 14301, 2021.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich- Keil, Nithin Raghavan, Utkarsh Singhal. Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-7547, 2020.
11

---

Published as a conference paper at ICLR 2026
Hugues Thomas, Charles R Qi. Caan-Emma Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE/CVF international conference computer vision, pp. 6411-6420, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why pinns fail train: A neural tangent kernel perspective. Journal of Computational Physics, 449:1 10768, 2022.
Jingjing Xu, Liang Zhao, Junyang Lin, Rundong Gao, Xu Sun, and Hongxia Yang. Knas: green neural architecture search. In International Conference on Machine Learning, pp. 11613-11625. PMLR, 2021.
Ge Yang and Samuel Schoenholz. Mean field residual networks: On the edge of chaos. Advances in neural information processing systems, 30, 2017.
A USE OF LLMS
We only use the large model to modify grammatical emros and polish the articles; the large model has not been involved in any theorem proofs the discovery of ideas.
B RELATED WORK
B. NEURAL TANGENT KERNEL
The Neural Tangent Kernel (NTK), widely used gradient kernel, is first introduced by Jacot al. Jacot et al. (2018). It is formulated as a Gram matrix constructed from gradients and has been shown to be equivalent to training fully-connected neural network under specific parameterization. resembling a kernel method, as demonstrated in Lee et al.
2020) Chen et (202 Over time, the applicability of NTK has been extended to various neural network architectures, including convolu- tional networks Samarin et al. (2020) Arora et al. (219a) Li et al. (2019), residual networks Yang networks Hron et al. (2020). GAN architectures [Fanceschi et al. (2022), and even graph neural networks Du et al. (2019b). Given its ability to capture critical properties of network architectures and datasets, NTK has also been integrated into Neural Archi- tecture Search (NAS) Chen et al. (202 helping to reduce time overhead in NAS processes Xu et al. 2021) Mok et al. (2022). Moreover, NTK has proven useful as a theoretical tool for analyzing input encoding in Coordinate- MLPP [Tancik et all (2020) and convergence properties of PINN Wang et al. 2022), and for explaining the effectiveness of adversarial training strategies Loo et al. 2022
Schoenholz 2017), attention-
active learning strategies Mohamadi et al. (2022), and ensemble learning He et al. 2020)) NTK has
also been used to demonstrate zero training loss in polynomial time for over-parameterized residual
networks Du al. (2019)), further underscoring its significance.
To address the computational complexity of the NTK, study Arora et al. (0019a) simplified this computation. Additionally, during real-world network training, the network width is finite. In finite-width networks, the kernel can change, and some studies [Shan & Bordelon (202T) Seleznova al.) (2024) have qualitatively pointed out the pattern of this variation. The computation of the Neural Tangent Kernel in finite-width networks is more complex, and this is also discussed |yer & Gur-Ari 2019), with implementation details available in Engel et al. (2022). These kernels play crucial role in convergence analysis, particularly in evaluating convergence rates, depending on the minimum eigenvalue of the NTK [Chnn et al. (2021a). The range NTK applications has expanded considerably since its inception. Its constant kernel property allows it to be used to estimate the performance of specific architectures by assessing convergence behavior. Notably, NTK, as a kernel, integrates seamlessly with most kernel methods Arora et al. (2019b).
B.2 POINT CLOUD CLASSIFICATION
Extensive research has been conducted on point cloud analysis, with a primary focus on learning data-driven representations in end-to-end fashion. The inherent flexibility of the point cloud
12

---

Published as a conference paper at ICLR 2026
format has led to the emergence of innovative approaches aimed at representing 3D structures both effectively and informatively. The seminal work PointNet Qi et al. (2017a) used shared-MLP on point cloud data demonstrating that neural networks can directly process global coordinates to extract geometric information. PointNet has since become a cornerstone for numerous point cloud-hased tasks. Building on this foundation, PointNet++ Q et al. (2017b) introduced local aggregation mechanisms, significantly enhancing the ability to capture fine-grained geometric features. To further improve the extraction of local structures, methods such as KPConv TThomas et al. (2019) introduced convolutional operations to 3D data, achieving notable performance gains across diverse benchmarks. More recently, inspired by the Transformer architecture Vaaswani al. (2017), models like PCT [uoo et al. (202T) effectively incorporate self- self-attention to better capture point cloud features, achieving state-of-the-art results in various tasks.
C PROOF OF MAIN RESULTS
C.I UNIFIED CONVOLUTIONAL NEURAL NETWORK
The general d-dimension forward propagation CNNs are defined as follows.
Definition 2. For d-dim convolutional neural network, given lengths of tensor {hiheea dl, lengths of convolutional kernel (si) and widths of neural network {no, "l,...,, nr.) with L-layer The d-dim convolutional neural network (without head) is defined as f : I - R bixnz- with forward propagation:
(() = (z),
(x) = w f (z)
1SIS6-1
(B-1)
f(x) f(-
Input x is a d-tensor from R " denotes space Rd1 gh1xxxx....hh The activation func- tion d(x) is Relu in this paper, The so-called "standard initialization" denotes that the defined convolution kernel w T and bias b(9) IR = All entries of any w are from 2
N (0,
- independently, and all entries of any (5)) are from A (0, of) independently.
(1)) is still called 'matrix' that is defined in [ (RUE_..))
b00 is still called 'vector' defined m1-1xm
in (Rx
" Convolution operator [w f] 12 .... id]
R x 1iI R is defined as:
Σ
ja.
(B-2)
fli i -1/2. + is 22/ + is s2/2]. The operator for matrix still satisfies matrix multiplication. And we suppose that:
3k s.t. is S f(b, b2
=
(B-3)
which is zero-padding.
In neural network applications, the corresponding head is typically added based on the specific task. This paper discusses classification network. We define two types of heads, one of which refers to the pooling layer and the other is naive.
Definition 3. Define pooling function P Tixiiiiii + R², d-dim convolutional neural network for classification task, without pooling in the last layer fwop. is defined as:
(mop( = Pav (W(L), f(x) + b(4)).
(B-4)
(L)) E R x 1 IRLL ((4)
[ (R) ne
with pooling Swp is defined as:
fup(z) =W(L), *P((()) +6())
(B-5)
((4) E 1 Xny (L) E LL.
13

---

Published as a conference paper at ICLR 2026
C.2 LEMMA 1: CNTK FORMULA
We first consider function f IR ixno - R xxii,A,nng 1 without head, as in practice, we suppose that s; s for any i. For entries f0 S L-LL-L and note this symbol omits subseript j that denotes the 5//
J..p As can be seen in the paper, this notation can actually be omitted. in which = {pr, P2, ..,, Pa). Furthermore, we define neighborhood of as N(p) (r - p/ + s 8/2) n 2 Si d]. Kernel of samples x, x' as entry is induced by: K0 Pig (1, V f (x')
V
(w!) ON (1 (11 1) (x)) + + (.(())
V
ANNGl1 (2')) + ()NN())
W
up x oz (z))
w W x (5 (x')) + to
P'EN(p)
d'EN(g)
1 11
12
Σ Σ
(x) (1- ( ( (z))o (1()
71
('E(()) ('EN(n)
E Σ ('EN(()
(x')) (z)) -1: +
(B-6)
In the calculation process of the tangent kernel, the presence of the bias term introduces shift in each recursion step. Therefore, in many tangent kernel computations, the bias term is excluded. This article follows the same approach that O. For convenience, we need some symbols to record intermediate quantities.For
ao
(x, = Σ
x4 3 or
(B-7)
20 (x, = tr (2)0)) (0) N (p) (z, (z,z')).
in which @ denotes tensor product. For 1 < 1 < L - 1, define the covariance matrix, the oth-order gradient term, and the Ist-order gradient term in sequence according the definition order in NTK.
1-(-1)) (z,x) 20(-"") (xx,a") 2p. Epp (x',ar) I// 21(1-)((',,0)))) xx)
cocariance matria:
( p.o (1,
E R2222
0th order term
22ll (x,x²)
E
(.,0)~ (o. (oso))))))) [Q())(v)).
(B-8)
$:(0 (x, = (a.0)~ (o.)) (0. (0. ((())(())]
1th order term:
And the induction:
((l) (x,,)' = tr - (on,
(B-9)
Now can present the recursive formula for the kernel as follows:
K(()) pig
(x,,
1 T1
(+1) 2
Σ Σ
VoS "
(1.1 (1 (1. (c)) (1))))((((1 (1!1-11(2))
F'EN(D) ('E((g))
3ΣW Σ
ox ( (x)) (J (x)) +
1=1 ('E(())) g'E((g)
-K
$000
NN (3, + gll) (x,
N(p).M(1)
(B-10)
14

---

Published as a conference paper at ICLR 2026
Now, consider neural network fwops fup with head (Assume that the final layer has only one output: the case with multiple outputs is analogous) as follows:
K woop (x,, x²) = V/ Swop (x) Vefwop(2x)
m4-1
FL Σ
= PppP av
w
f(x))
VoPao
will f(x')
(B-11)
Σ
R x') P.p
(II h:)²
tr K (2,
K1p( x0) V Fup (x) Vo (ffp(x')
V Σ wi x Pav (f(x)) V9 Σ
w!T x Pav (f(x')
Par (f(z)) Pao (f(x'))
n5-1
Σ
E
#h-1
II 1
h4 122.-u. (e
Pav (x(L)
(zx, x'))
+ Pav
(B-12)
If we fix the last layer, the upper eqaution can be redued to our proposition:
Kup (2, x²) Paw (IC
(B-13)
C.3 PROPOSITION 1: NTK FOLLOWED BY CNTK
The calculation of the composite-NTK only requires knowledge of the propagation of NTK at the pool- ing layer, K Kupp and propagation of covariance, which necessitates computing
Cov (pd) av d2 (f(z)), pdi d22 (f(z))
Σ
Cov (f..... (x),
(x')) (B-14)
11 d hi fdg (1ddg 2nrddd
= pdi dd ()(x,21))
When d2 = 0, it corresponds to the common average pooling layer, and the covariance tensor is:
g(1+1) = Pav (xx(2x()))) (x,()))
(B-15)
D SPECTRAL COMPARISON
(1) Definition 4. Suppose tensor hi we mark x (tp) as the {i,p} entries of the j th samples (0 in which i E [no] and p [[hiaa]. We further define scalar II For any matrix K, let m (K) and s (=tr(KKT)
Assumption 1. We assume that tasks with random tensor Tk.. RR in which all-entries of (k)
Ik. are sampled from a distribution with covariance matrix (kk) E ERl the scalar d denotes covariance of random variable Ix.p and x And for any k + k. any entries of ( and x are independens. Furthermore, we assume any sample ()) of datasets {x(1) Die[N] is sampled from the above distribution independently.
15

---

Published as a conference paper at ICLR 2026
Lemma 3. Given N N positive definite matrix K, let its set of eigenvalues be denoted as A1 S X2 .... S AN. then:
mk
A
SK
(B-16)
N
Proof. For a N x N positive definite matrix K, by researcHHorn & Johnson (2012)
Cai et al.
2024)
1
#
tr(K) mk
(B-17)
N i=1
N
N
1 (A: 222 [(r(K(T) - -1² -22= =N= - -mk
a²
(B-18)
i=1
let Amsn \1. Amax
AN. then:
Σ N A Nsk
1
Amaz
Amin
A mg
(B-19)
Ei=l A
mK
therefore:
Amar
2 NBK.
(B-20)
Amin
D.1 PROOF OF THROREMS CONTENT
Lemma 4. For a set of tensors(datasets) ()) For the L - layer NTK and CNTK. The following
satisfies:
(x, 2 H p.
Σ
x (x, (x,x) 2 (00) (x,
(B-21)
Proof. I. when 1 = 0, as:
(0) (x, x)
1 p.g
(0)
(I, =
710
1
H
Σ
Lip
H²
Σ
Diplig 20 (B-22)
pE[[b]]al
p.gE[[baaa]
II. suppose that ( 1) (x, 2 + Eppe((h.la)
2p. (x, x) satisfies for 2 1, then:
(x, = x(1, (1-1) (x, 2 x(1, H²
Σ
0 p.y (x,)))
p.gEl[kulal
Σ
m(1, x(1, (10) (I,
H
P.
Σ
24..9
(1-
(I, (x) x)
(1
p.o (x (x,x)) x))
H
(1- (z,, x) Eita (x,xx
Σ
(1-1)) (x,
H²
p.gE[lhulal
(B-23)
1 (z, (:00
En (z. x) is similar as:
500 (x. = k(1) 2 (10 (x,
(B-24)
n
16

---

Published as a conference paper at ICLR 2026
Assumption 2. When {p:) o (q.a)' we define the dual-activation function and one-order dual- activation function as k(), c) which is concave about с and increasing about both X, C, that is for any X, 0, C2 E [0, 1]:
(1)
(i)
k(A, ac, + (1-a)c2) 2 c1)(A.1)) + a))((A,2)))
(B-25)
R(A) increasing about X and independent to c1
Notice that for the Relu function it satisfies, as it is linear w.r.t. item
Theorem 5. For set of tensors(datusets) {x(0), ). Let CCNTK in (L)
NTK be denoted as themg of NTK-
(L)
L-layer CNTK and NTK. The following satisfies:
(L) CNTK S (T)
(B-26)
Proof. It satisfies in L 0, suppose it satisfies in L 1, by Lemma 4, the following holds:
(L)
K(l)) (CNTK((, = Pau (1(0) x) o K KONTK (I, ( (x, (x,x))
MCNTK-
$ (x,)), (C, + Pav (z))(z))))
(B-27)
s (1,)) NTK (I, E(L) (z. MNTK.
(L)
Proposition 6. For set of tensors (datasets)
(x.") Let N - co. When K = K NTK- then:
1 ng
mean(diag( (1))
mK
no
(B-28)
1 no
mean(a (1) o (5')),
8K
no
jJ'E[mnl
when K-(0]
CNTK. then:
1 no
mean o (i)
mK
710
j=1
(B-29)
Σ
mean(o (1) & o0"),,
8K
10
jJ'E(mal
in which @ denotes Schur product (element-wise product), & denotes Kronecker product (tensor prodcut).
Proof. When K KNTK-
N tr(k)
N Σ
1
K K(x(0,, x(()
mR
no
(a (xx)))
N
p [
(B-30)
(1)
HoH
(p.P)
DE [[b
1 no
mean(diogy (5)))
110
17

---

Published as a conference paper at ICLR 2026
(rKKT)
Σ
(2
SK
N
N
AJEE[M]
No
n2 Σ
Σ
[rE[ j=l ph]
ng
E
Σ
E
N2
[ [
Σ
E
N2
(B-31)
iPPE
MellNNJJ''RR E(no) moe[[M
~ 0 )t)
E
Σ
Σ
Lim) Urm) U (ia)
H2
E p.gEl[bula)
ng
(
(1))
(p.)) (p.e)
n
E p.qE (helal
1
mean(o (f) @ @062)
n0
when = (0))
C
mK N tr(K)
1 Σ
K(x x0())
N
N
Σ
(in) 20 (in)
N
i= j=l p.gE[[halal
(B-32)
fg
no 2
(m-g)*
p.gE [hhaaal
mean(o(() (i)
no
1
tr(KKT
Σ
K2 (20 x(0))
SK
No
N
(JE[N]
Σ
Σ
(1)
(in) [an](( (0)
(2,)))
N6N2H4
i j=1
pa
Σ
Σ
E
(0)
(
(
N
[= j. Ennal p.s.a
Σ
Σ
n N
(B-33)
[EEN E(m)) p.g.p' a'Ellaalll
O N
+ n²
E
Σ
Σ
(nn)
(m) (0))
JJ'Einal p.gellhaldl
Σ
(0)
6') (pa) ( (p'.9))
6')
n
( p. / d'Ellhulal
1
Σ
mean(a(1) ()) @ (1)))
= 3
no jJ'E[mo]
18

---

Published as a conference paper at ICLR 2026
Lemma 5. For set of tensors (datasets) {r N.
Let N + oc, define the standard spectral
bandwidth 3x m by research Cai et al. (2024):
8K = Σ X2
E Xi
For the foundation:
PONTRI =1
En mean( (1) @ ااا')
(B-34)
BNTKO
E mean(diag( (i) @ diag diag(or("))) S
When we further assume that the covariance matrix is the same across all channels, the conclusion stated in the main text follows.
Theorem 7. For a set of tensors(datasets) {xO). Let N - cc, define the standard spectral sk for the L-layer NTK and T/ ll-yay CNTK. The following satisfies;
bandwidth 3x mix"
g(L) CNTK 3
=
(B-35)
Proof. For this theorem, we obtain a rather strong result; fact, suffices to prove statements g(E')
(L.) PCNTK 1 and NTT < separately. Recall the proof in Proposition 7. It suffices to replace with the random variable appearing before the final linear layer in CNTK (or NTK) to complete the particular
proof, since the above argument relies neither the specific data distribution instance of the covariance matrix.
D.2 UPPER BOUND OF IN NTK AND CNTK WITH RELU
Lemma 6. Suppose that the activation function 6 is /2Rel.. Then in NTK:
(x,x) = (((,,))=<<(>>,,) (x, (x, 1.
(B-36)
Proof. For the step 1, x1 (Ir, x)) follows recursion:
(x, A(T - arccos(A)) + vT X2
c102 (2,z)) क - arccos(A)
(B-37)
(z,)) I(-1) (2, ). thus c102 = D (2) and A 1. There- (())) 11- (2,
In which DAD
fore:
(x,)x)) (x, (0(xx)))) =1
(B-38)
In the case of CNTK for the step I, E (2,
X( - arccos A)) + v1 X2
C102, {((1 (1, = π - arccos((
( pig (z, x)
(B-39)
(1-1) (z,z) (2,
Σ (1-1) (z, p.
In which DAD
ppp
thus 01 -/- (x. (.. xg (1-11) (x, x) and
Lgg
(1-1)
(x, x)
2|1 1) (x,
L (a.x)
0102
19

---

Published as a conference paper at ICLR 2026
similar conclusion can be obtained to the case in the CNTK, in which EA)
_( (x,z)))
and (A) - 0 along with ny + co. While there exists a specific case
y | Ep.. (x) E (x!
in the step? I = 1 due to no is constant that is differ from n2 < k < L). When / 2 1, in the NTK:
IE (2:00 (z,,'))
A(अT arccos(A)) + v1 11-X8 X2
E3
C102
π
Am + 1
- E8
c102
π
(B-40)
E
[N- 1) (Ir, x4)
1 -E
(V (x,
x)) 20))
π
S E
(1 (Ix x
+
1
E (x
π
in which the last step can be splited two steps follows by Jensen inequality (since the function g(x) = v is concave):
/ 12 (x, (x, (U) E (x) m:)) x x=x x'
E
(EE x) E 1))
(x E (Z))))())
E
E
(B-41)
E (Σ(-- (x, x)) E (x',
= E (Iu (x,, x #
similar to the CNTK:
[1) p.q (Ix, x²) = E ( (\TT - arccoss())) + 11- x2
E
C102
Σ (1-1)
(ma) (xr, x²)) + E V 2 ( (1', x²))
E
(B-42)
E Σ 11
(p.0) (x. x²))
+ v E >> (2, (z, E 2 (x' 22)"
# x'
S
2+ E 2P14 (I, x))
+ # E (> (ux, xr) E 2ل11 11 (xz, xx)
x = x¹
E
(00) (2,)))
π - (m=arccos(A)) E
$1, 1, (z,))))))) =E - ( (mrrccos(A))
<1
π
π
(B-43
Now we induce the twice:
( (MiT
\(x - arecos(A)) ((==arccos(A))
+ v1-1V1
- x²
2
E
Σ 20 (2,
=
.2/
E
2² -1 x²
2
XV1-XX X2 + n²
1
72
2
+ 2TdVI- X2 +1
(B-44)
(EU-))))
(I, x²))
(1(111) (zo x x) (od)
2²
x²
x2)
+ T E (1(-1) 20 (2))
?
n²
E
(1
+ EHE (M(-1) (xx, x²) E (
x # x'
+
(EU-
x x=
20

---

Published as a conference paper at ICLR 2026
Table 3: Performance on ModelNet10; of Kernel Regression (KR) and Supervised Learning (SL). All supervised networks are of the vanilla structure. GAP indicates global average pooling is added to the final layer. The exponential parameter of the Gaussian kernel is 1.
Model
D = 5
D = 10
IdCNN
42.18
21.70
SL
IdCNN_GAP
39.54
21.26
1dCNTK
44.38
43.39
IdCNTK_GAP
76.87
76.54
KR
NTK
19.98
14.54
Gaussian Kernel
21.15
similar to the CNTK:
A(T arccos(A)) + v1 - A2
E
2(0
(xc, x1) = EE
π
E
ạ + 1)) (xx,
(
E
(x, x²) Σ - (z', xx))
π²
π²2
E
(>/ )2
+ E
/2// 1) (1,, m
E 12/1 x
x+40 # x'
x
(x+1)²
E
(>>p ) + اا
1 ( (xr x))
IE
(----11 (2, (z,
= x!
x
(B-45)
E EXTRA EXPERIMENTS
E. VANILLA METHODS FOR POINTCLOUD
Based the kernel regression, tested NTK, 1dCNTK, and IdCNTK with global average pooling layer IdCNTKGAP on ModelNet10g. Additionally, we conducted experiments using a traditional Gaussian kernel for comparison. Furthermore, we trained the corresponding vanilla neural network structures based on these kernels.
The results, shown in Table 1, indicate the following:
Vanilla networks without empirical adjustments perform poorly on point cloud data, with performance degrading as network depth increases.
The NTK corresponding to MLPs fails to perform effective kernel regression point cloud data, performing even worse than the Gaussian kernel.
IdCNTK is effective kernel regression for point cloud data, and 1dCNTK with global average pooling significantly outperforms IdCNTK without pooling.
All kemels show no substantial improvement with increasing depth.
21


Reviews:
[{"review_id": "oFeGwkkFBQ", "reviewer": "Reviewer_qF94", "summary": "This paper presents the first systematic comparison of the spectral properties of the Neural Tangent Kernel (NTK) and the Convolutional NTK (CNTK). The authors formally prove that for data with a tensor structure, CNTK consistently exhibits a broader eigenvalue spectrum and a smaller mean eigenvalue compared to NTK. Based on this insight, they propose that CNTK's spectral bias is inherently better suited for geometric data like point clouds. This hypothesis is validated by introducing a metric for \"Convolutional Suitability\" and demonstrating experimentally that point clouds align more strongly with CNTK's properties. Finally, the authors propose PointNTK, a CNTK-based kernel regression method, which achieves strong performance on point cloud recognition, particularly outperforming training-based baselines in low-data settings.", "questions": "- On the Performance of the PointNTK Model: \n    I noticed in Table 1 that for the ModelNet10_6 dataset, the simpler 1dCNTK model slightly outperforms the PointNTK hybrid model. This is counter-intuitive given that the MLP layers are introduced to enhance performance. What is your interpretation of this result? Is it related to dataset-specific properties?\n﻿\n- On the Smooth Decline in the Kernel Size Ablation: \n    Your argument about point cloud unorderedness suggests that a kernel size k > 1 should be highly detrimental. I would have expected a sharp performance drop when moving from k=1 to k=2. Instead, Figure 4 shows a smooth, gradual decline. What is your intuition for this gentle degradation? \n﻿\n- On β_K(0) as a Proxy for Suitability: \n    You use β_K(0) as a practical proxy for \"Convolutional Suitability\" across the entire network. While I understand the computational challenge of evaluating β_K(L) for deep layers, do you have any evidence or theoretical argument suggesting that β_K(0) is a reliable representative? An answer here would strengthen the claim that this initial-layer metric is sufficient.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novel and Insightful Theoretical Contribution: \n\n    The paper provides the first systematic spectral comparison of NTK and CNTK, offering a novel theoretical lens to understand the inductive biases of convolutional architectures. The introduction of mK and βK as quantitative metrics and the concept of \"Convolutional Suitability\" are significant contributions that provide theoretical guidance on how to choose the right network architecture for a given type of data.\n﻿\n\n- Combining theory with practice: \n\n    A key strength of this work is that it clearly connects theory with practice. The authors start from formal proofs to formulate a verifiable hypothesis—that point clouds are more \"convolutionally suitable\" than images—and then compellingly validate this hypothesis through well-designed experiments.\n﻿\n\n- Explanation a Fundamental Question in Point Cloud Processing: \n\n    This paper tackles a highly relevant question in the 3D vision community: why are convolutional-like structures (such as the shared-MLPs in PointNet) so effective for point cloud data. By approaching this from a theoretical kernel perspective, the work provides a novel explanation.", "weaknesses": "- [Minor] Confusing performance:\n﻿\n    The results in Table 1 present a slightly confusing result. For the ModelNet10_6 dataset, the vanilla 1dCNTK (91.96%) outperforms the more complex PointNTK (91.19%). This seems to contradict the paper's motivation for adding MLP layers . A discussion on why this might occur would be beneficial.\n﻿\n- [Minor] Potential Discrepancy Between Ablation Results and the Unorderedness Argument:\n﻿\n    The paper makes a strong argument that a kernel size greater than 1 is detrimental due to the unordered nature of point clouds. Following this logic, one would expect a sharp performance drop when the kernel size increases from 1 to 2. However, the experimental results (Figure 4) show a gradual decline rather than a steep fall. This gentle degradation seems not entirely consistent with the theoretical expectation that aggregating unordered points would cause severe disruption. The paper would be strengthened by a discussion or explanation of this phenomenon . \n﻿\n- [Minor] Practical Limitation of the β_K Metric:\n\n    The use of the initial layer's metric, β_K(0), as a proxy for \"Convolutional Suitability\" is a practical simplification but also a limitation. As Figure 1 shows, β_K evolves with depth. The paper would be stronger if it discussed this evolution or provided bounds on its variation, which would enhance the completeness of the theory.\n﻿\n- [Minor] The Core Concept of \"Convolutional Suitability\" Lacks a Formal Definition\n\n    The paper introduces \"Convolutional Suitability\" as a key concept but fails to provide a formal definition. It is only mentioned that 1 - β_NTK(0) can be \"interpreted as\" this metric. For clarity and to facilitate future work, this concept should be formally defined when it is first introduced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ge2A8JGCLH", "reviewer": "Reviewer_GQJv", "summary": "The paper analyzes the spectral comparison between Convolutional Neural Tangent Kernel (CNTK) and Neural Tangent Kernel (NTK). Through a series of theoretical analysis and validation via synthetic data, the paper concludes that convolutional structures are inherently more suited to irregular point cloud data.", "questions": "What can be some practical implication of the paper given PointNTK performs worse than PointNet?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The theoretical comparison between CNTK and NTK can potentially guide architectural search of point cloud tasks.", "weaknesses": "* The writing of the paper is hard to follow, hard to understand. Though it is a theory paper, the presentation can be made much more accessible by explaining the intuition behind and visualization. Figure 1 seems to have such attempt, but it is not explained well.\n* The conclusion reached by the paper is a well-known fact from empirical experience. The paper only decorates it with some theoretical proof.\n* The conclusion that “convolutional structures are more suited to irregular point cloud data” is supported by Figure 2. But it only has one point cloud dataset and two image datasets, which cannot represent “point cloud” and “image”.\n* In the experiments, the result shows PointNet performs better than PointNTK under all settings. So it is hard to understand what is the practical implication of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9XYwuTsdLH", "reviewer": "Reviewer_qeWv", "summary": "This paper finds that CNTK is more suitable for point cloud data than image data, and thus proposes a CNTK-based kernel regression method for point cloud recognition tasks. Experimental results demonstrate its effectiveness, while providing a new theoretical explanation for the spectral characteristics of NTK.", "questions": "1.The relevant work done in this paper on shared MLP should be extendable to more advanced point cloud models, such as PointMLP. Have the authors made any relevant attempts?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.This paper explains the spectral characteristics of NTK from a new perspective and finds that CNTK is more suitable for point cloud data. The formulas are rigorous and correct, and the experimental results also demonstrate the rationality of this theory.", "weaknesses": "1.This paper only conducts training on the ModelNet dataset. Experiments on ScanObjectNN, which is more in line with real-world scenarios and more challenging, should be added to demonstrate the practicality of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)