Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study
Authors: Yongtao Wu
Zhenyu Zhu
Fanghui Liu
Grigorios Chrysos
Volkan Cevher
Venue: NeurIPS 2022 Accept (2022)
Keywords: Neural tangent kernel
Hadamard product
extrapolation
spectral bias
kernel regression

Abstract: Neural tangent kernel (NTK) is a powerful tool to analyze training dynamics of neural networks and their generalization bounds. The study on NTK has been devoted to typical neural network architectures, but it is incomplete for neural networks with Hadamard products (NNs-Hp), e.g., StyleGAN and polynomial neural networks (PNNs). In this work, we derive the finite-width NTK formulation for a special class of NNs-Hp, i.e., polynomial neural networks. We prove their equivalence to the kernel regression predictor with the associated NTK, which expands the application scope of NTK. Based on our results, we elucidate the separation of PNNs over standard neural networks with respect to extrapolation and spectral bias. Our two key insights are that when compared to standard neural networks, PNNs can fit more complicated functions in the extrapolation regime and admit a slower eigenvalue decay of the respective NTK, leading to a faster learning towards high-frequency functions. Besides, our theoretical results can be extended to other types of NNs-Hp, which expand the scope of our work. Our empirical results validate the separations in broader classes of NNs-Hp, which provide a good justification for a deeper understanding of neural architectures.

Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study
Yongtao Wu, Zhenyu Zhu,
Fanghui Liu,
Grigorios G Chrysos,
Volkan Cevher
EPFL, Switzerland
{ [ffrrst name]. [surname] H@epf..ch
Abstract
Neural tangent kernel (NTK) is powerful tool to analyze training dynamics of neural networks and their generalization bounds. The study on NTK has been devoted to typical neural network architectures, but it is incomplete for neural networks with Hadamard products (NNs-Hp), e.g., StyleGAN and polynomial neural networks (PNNs). In this work, we derive the finite- width NTK formulation for special class of NNs-Hp, i... polynomial neural networks. We prove their equivalence to the kernel regression predictor with the associated NTK, which expands the application scope of NTK. Based on our results, we elucidate the separation of PNNs and spectral bias. Our two key insights are that when compared to standard neural networks, PNNs can fit more complicated functions the extrapolation regime and admit a slower eigenvalue decay of the respective NTK, leading to a faster learning towards high-frequency functions. Besides, our theoretical results can be extended to other types of NNs-Hp, which expand the scope of our work. Our empirical results validate the separations broader classes of NNs-Hp, which provide good justification for deeper understanding of neural architectures.
standard neural networks with respect extrapolation
Introduction
In deep learning theory, neural tangent kernel (NTK) Jacot et al. 2018 is powerful analysis tool that links the training dynamics of neural networks (NNs) trained by gradient descent to kemel regression [Jacot et al., 2018 Arora et al. [019]. NTK provides tractable analysis for several phenomena deep learning, e.g., the global convergence of gradient descent (Chizat et al. 2019 Du et al.] 2 c). the inductive bias behind NNs [Bietti and Mairal 2019 the spectral bias toward different frequency components [Cao al., 2019]Chhoaria et al. 2022 the extrapolation behavior [Xu et al.) [021]. and the generalization ability [Huang et al. 2020). The study on the NTK has been devoted to typical NNs architectures, c.g., fully-connected NNs [Jacot et al. 2018), residual
NNs Tirer et al. 2020 Huang et al. 2020), convolutional NNs Arooa et al. 2019 graph NNs (uu
ct al.
2019b and recurrent NNs Alemohammad et al., 2021].
Recently, NNs with Hadamard products (NNs-Hp), e.g., StyleGAN Karras et al. 2019] polynomial
neural networks [Chrysos et al. 2020), non-local multiplicative networks [abilon et al. 202T have received increasing attention due to their expressivity and efficiency over traditional NNs [Chrysos] al. 2021a| [Camphell and Broun 2000. Su et al. 2020). There have been several works attempting to demystify the success of NNs-Hp. For instance, Fan et al. [2021] prove that second-degree multiplicative interactions allow NNs-Hp to enlarge the set of functions that can be represented exactly with zero error. Choraria et al. [2022] reveal that NNs-Hp with second-degree multiplicative interactions yield a faster learning high-frequency function during training in the NTK regime. Yet, the theoretical analysis of NNs-Hp with high-degree multiplicative interactions is still unclear. More importantly, when using NTK for analysis, only deriving the NTK matrix is not enough. The
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

complete and rigorous proof is achieved by including the stability of empirical NTK during training and the equivalence to kernel regression. This crucial allow for NTK-base analysis of typical NNs [Arora et al. 2019 Tirer et al. 2020 but is still missing for NNs-Hp.
Polynomial neural networks (PNNs) [Chrysos et al. 2021a], a special class of NNs-Hp (Jayakumar et al., 2020). have showcased remarkable performance on a broad range of applications. As step for analyzing NNs-Hp, in this work, take PNNs an example, derive the NTK for PNNs with high-degree multiplicative interactions and present a rigorous proof for the equivalence to the kernel regression predictor. This analysis enables us further examine properties of PNNs theoretical perspective, e.g., the extrapolation [Haley and Soloway 1992. Barnard and Wessels 1992 Xu et al. 2021]. Neural networks have demonstrated a stellar in in-distribution performance but admit some weaknesses in extrapolating simple arithmetic problems [Saxton et al.) 2019] or learning simple functions [Haley and Soloway 1992 Sahoo et al. 2018). Recently, Xu et al. [202T] theoretically and empirically point out that two-layer collecconeeted NNs with ReL.U can only extrapolate to linear functions. The contrast on the in-/out of-distribution performance of standard NNs motivates us to scrutinize the extrapolation performance of PNNs. Additionally, studying the NTK of PNNs also allows us to investigate its spectral bias.
Overall, our main contributions and findings can be summarized as follows:
We derive the NTK formulation for PNNs with high-degree multiplicative interactions, and give concrete bound of the widths requirement for convergence the NTK at initialization, and stability during training, which allows us to bridge the gap among PNNs trained via gradient descent and kernel regression predictor.
We provably demonstrate the extrapolation behavior of PNNs as well as other NNs-Hp, including multiplicative filter networks and non-local multiplicative networks. Our findings highlight that PNNs can extrapolate to unseen data in non-linear way. Besides, the spectral analysis of NTK of PNNs is also given for better understanding. PNNs admit a slower eigenvalue decay when compared to standard NNs, which leads to a faster learning towards high-freee frequeryy functions.
We empirically show the advantage of NNs-Hp over standard NNs in learning commonly used functions, performing arithmetic extrapolation in real-world dataset, and conducting visual analogy extrapolation task. We scrutinize the role of multiplccative interactions in the task of learning spherical harmonics.
2
Background
In this section, we establish the notation, provide overview of the NTK, and summarize the most closely related work in NNs-Hp as well as extrapolation.
2.1 Notation
The core operators and symbols are summarized in Table 2 at Appendix A Vectors (matrices) are symbolized by lowercase (uppercase) boldface letters, e.g., a, A. We the shorthand [n]]
{1,2.... n for a positive integer n. We use { and {y, | to present the input features and their labels of the training set (X,Y) Y) in compact space, where | denotes the cardinality. We symbolize by (xx x²) the neural tangent kernel with respect to input æ and x', the kernel matrix with (()) K (I,, 2)). Next, we denote by 0 the parameter vector, (2(),) the empirical training loss, and K, the empirical NTK Gram matrix at time step t. The following notation used:
K E R x
()(+))
1
Σ
(f(x;;0) - v.)? f(0.)
E
2
(..)
J(O,) = 00
E [|00
K, = (0) x

---

2.2 Neural tangent kernel
Neural networks (NNs) are relevant the kernel method, under proper initialization [Daniely et al. 2016 de G. Matthews et al.] 2018). Jacot et al. (2018) provably demonstrate the equivalence between the trainnn dynamies by gradient descent and kernel regression induced by NTK when employing the l loss. Below, we recall the exact formula regarding the NTK of N-layer (N > 2) fully-connected NNs with ReLU activation functions a. The corresponding NTK K( x') = K N (a, a') could be computed recursively by:
Ko(æ, x²) Eg (z, x²) = x TTx', xx)) = En, x²) + 2K 2x,,((2,0)) (2) Σn (x, x'),
Vn E [ where the covariance En and its derivative Ei, are defined as:
En x') = 2E (B..))-ooo.,))((u)orrr
En x') =
[x- (2, x)
1 ,2") ) vn E [N].
30-1(0,
An =
1 (2.
Furthermore, the aforementioned NTK is extended to residual NNs (Tieer et al. 2020 Huang et al. 2020], convolutional NNs AAror et al. 2019), graph NNs [Du et al. 2019b]. and recurrent analyze the training behavior of the neural network in the over- parameterization regime (Allen-Zhe et al. [019 Chizat et al. 2019 Zou et al., 2020]. For instance, Lee et al. 20119 showcase that NNs under the NTK parameterization trained via gradient descent of any depth evolve to linear models. Meanwhile, the inductive bias of convolutional networks, e.g., deformation stability of the images, has been
NNs (Alemohammad et al. 2021]. One of the roles of such kernel
Du et al., 2
studied in the NTK regime [Bietti and Mairal 2019).
2.3 Neural networks with Hadamard product
The ideas of augmenting NNs with Hadamard products to allow multiplicative interactions can be traced back to at least [Ivakhnenko] 1971) that investigate the learnable polynomial relationships. Most of the early work e.g., Group Method of Data Handling [Ivakhnenko] 1971], pi-sigma network (Shin) and Ghosh| 1991 1 do not scale well for high-dimensional signals. Chrysos et al. (2021b) factorize the weight of NNs-Hp based tensor decompositions
reduce the number of parameters. They exhibit how to convert popular networks, such as residual networks, and convolutional NNs to the form of NNs-Hp. StyleGAN can be also considered as a special type of NNs-Hp [Chrysos et al. 2019]. New efforts have recently emerged to improve the architecture of the network with Hadamard products [Chrysos et al. 2022 Babiloni et al. 2021 Chrysos et al. 2021a]. In this work, we adopt the complementary approach and focus on the extrapolation as well as the spectral bias from theoretical perspective.
2.4 Extrapolation
The study of extrapolation properties of NNs dates at least back to the 90's [Barnard and Wessels 1992 Kramer and Leonard 1990). Experimental results show poor performance of NNs in case of learning simple functions Barnard and Wessels 1992]. Browne 2002] also suggest that fully- connected NNs cannot extrapolate well and then illustrate how the representation of the input impact the extrapolation. Xu et al. [2021] provably present the extrapolation behavior of fully- cully-connected NNs and Graph neural networks. Specifically, they show that two-layer fully-connected NNs with ReLU activation function ex extrapolate to linear function in extrapolation region. Our work exhibits that NNs-Hp can learn high degree nonlinear function. Apart from fully-connected NNs, Martius and Lampert (2016), Sahoo et al. (0018) showcase a novel family of functions with linear mapping and a non-linear transformation, which allows to use sine and cosine as nonlincarities, enabling such networks to learn well in analytical expressions. Note that there exist multiplication units in EQL, which similar to the multiplicative interactions in NNs-Hp. Lastly, extrapolation is often considered in the context of out-of distribbutoin (OOD). There are other types of OOD problems with specific setting among machine learning community (Shen et al., [021]. Domain adaption assumes the source and the target domains lie in the same feature space but with different distributions [Kouw and Loog 2019), which differs from extrapolation. Another category of methodologies to solve the OOD generalization problem, called invariant learning. aims to discover high-level invariance feature from low-level observations through latent causal mechanisms (Arjovsky et al., 2019, Rosenfeld et al.. 2021] We believe our analysis can also encourage the usage of NNs-Hp in these OOD problems.
3

---

3 Analysis of polynomial neural networks
Our analysis admits the following structure: we firstly study the NTK of PNNS in Section 3.1 which allows us to conduct analysis towards extrapolation in Section 3.2 and spectral bias in Section 3.3 In Appendix Fannd of the supplementary, we consider extensions beyond PNNs other families of Hp, e.g. multplicatiiv filter networks and non-local networks with Hadamard product.
3.1 Neural tangent kernel
We now derive the NTK for PNNs, then we bridge the gap between the PNNs trained by gradient descent with respect to squared loss and the kernel regression predictor involving the NTK. The goal of such networks is to learn N-degree (N 2 2) polynomial expansion that outputs f(x) R with respect to the input R., For simplifying the proof. consider the following formulation, which is a reparameterization version of PNNs [Zhu et al. 2022]. The output is given by:
2 V1 - V m
o f(x)
2 ( (WN+1UN), yn V 2 (Wn.r) yn-1s 2, .....: N, (1)
where is the ReLU activation function, each element in WN+1 E Rixm and W E Rmxd vn E [N] independently sampled from (O, 1). Three remarks are place: a) We multiply by the scaling factor,
after each degree to ensure that the norm of the network output preserved at initialization with infinite-width setting. b) ReLU is usually required to increase the performance of Np in experiments [Chrysos et al. 2021b). c) The original formulation before reparameterization that is used in practice can be founded in Appendix A.
Theorem 1. The NTK of N -egree PNNs, denoted by K( x'), can be derived as:
K(x, x') 2N (x, x )x (I, x' x')) + 2(K2(2x, x
(2)
where K1 and 122 are defined by taking the random Gaussian vector Rdd
Ki = w.w.io. (ō(ww xx) o(w x²)) K22 = w-N(o./E-T (a(w TT) a(w" (())
(3)
The proof, which is provided in Appendix B. is based on the standard NTK calculations. Differently from the NTK of fully- fully-connected NNs, the existence of multiplicative interaction in PNNs induces the product form of multiple kernels.
Next, we provide the following theorem that gives concrete requirement for the width of the networks that is sufficient for nonasymptotic convergence to the NTK at initialization,
Theorem 2. (Convergence to the NTK). Consider N N-degree PNNs, and assume that the width 11 (2N for any E (0, 1), then given two inputs x, x' on the unit sphere, with m 24N 2og-2 Iog2N-1 probability at least - 8 over the randomness of initialization, we have that
(Vf(2), vf(z)) K(x, x')l 1.4NNp pep log(2N/8)
m
where p V2" 2N-1
(22 1)/2) 12N-1)/2
Remark: This result exhibits that the inner product of the Jacobian converges to the NTK at initialization, which has not been studied before for PNNs. This theorem allows us to further analyze the extrapolation of networks from the perspective of the NTK. It should be noted that the term and the width are exponential with respect to the degree N. but the degree N is not large in practice, e.g.. at most 15 in Chrysos et al. [021a] Hence the bound is fair and reasonable.
The technical key issue of the proof is to provide probability estimates for the multiplication of several sub-exponential random variables. To this end, we rely on the concentration of sub-Wabbe Weibull random variables [Zhang and Chen 2020) complete the proof, which is deferred to Appendix B.2
Below, we show that under certain conditions, the limiting NTK of PNNs stays constant when training with gradient descent using the squared loss.

---

Theorem 3 (Stability of the NTK during training). Given PNNs in Eq. TT. assume Xuin (K) > 0 and the training data (X', )) a compact space admitting a + for all æ, x E X, then there exist some constants Ro > 0, M 1, and Q > 1 such that for every m > M, when minimizing the squared loss with gradient descent and sufficient small learning rate yo 22(Xmin (K) + Xmax (K)) )- the following inequality holds with high probability over the random initialization of model parameters:
6Q2h Ro
sup || Kr -Kolls S
( (K)
Remark: Eq. (4) shows that K, m-00 Ko- Combining this with Theorem 2 that states Ko m 70000
K, we have K. 700 K. Thus, the equivalence to the kernel regression is established. Note that Theorem is an extension from the corresponding theorem of NNs with residual connection
(Tirer et al. 2020 to PNNs. This property allows us to characterize the training process as kernel
regression.
Regarding the proof of Theorem we firstly introduce the norm control of the Gaussian weight matrices and then derive the local boundness and local Lipschitzness. The last step is to apply the induction rules over different time steps. Details are presented in the Appendix B.3
3.2 Extrapolation behavior
Firstly, we provide the definition of extrapolation from Xu et al. [022 as follows.
Definition 1. Extrapolation occurs when the domain of test samples is larger than the support of the training distribution.
Remark: The definition presented above is different from the one in Balestriero et al. [202]] that claims extrapolation occurs when the test samples fall outside of the convex hull of the training set. Even though these definitions are not completely compatible, both definitions are suitable for our subsequent analysis.
The derived kernel in the previous section enables us to study how PNNs with ReLU activation trained by gradient descent extrapolates. Note that our theorem can also be extended to the raw PNNs without activation function.
Theorem 4 (yy-degree extrapolation of N -edegre PNNs). Suppose we train N -eggre (N 2 2) PNNs
f: IRd R. with infinite winth on {(>;, yi) ) and the network optimized with the squared loss in the NTK regime. For any direction a E Rd that satisfies || max [>,](,),,))))))))))) let xy to and x = Do + lvo with 1 >> and h >> 0 be the extrapolation data points, the output f' + hwo) follows a d-dggeee (7 S N) fanction with respect to h.
Apart from PNNs, we also consummate Xu et al. connected NNs with only two-layer. We provide the following generalized theorem for N-layer (N 2) fully-connected NNs.
[021 that consider the extrapolation of fully-
Theorem 5 (L.inerr extrapolation of N --layer fully-connected NNs). Suppose we train N-layer (N > 2) flloelly-oneeeee NNs / Rd + RR on {x) - For any direction v E Rd that satisfies ||l maxx||=>;]]] ]] To = and æ To + he with 1 and >> 0 are extrapolation data points, the output f + hw) follows a linear function with respect to h.
We have already shown that PNNs extrapolate to a function with specific degree and are more flexible than fully-connected NNs. However, only knowing the information of the degree of the extrapolation function is not enough. Naturally, we might ask under which condition PNNs can achieve successful extrapolation. Below, we build our analysis in the NTK regime and show how the geometry of the training set affects the behavior of PNNs.
Theorem 6 (Condition for exact extrapolation of PNNs). Let fa(x) x Be be the target function with x E &Rd and f E RRdxd Suppose that (a;) |X|
| contains the orthogonal basis {es) and [-r Then if we train two-degree PNNs f NTK regime, we have f(x) x." Bax for all x E Rd.
fx(x)))) with the squared loss in the
This result only considers quadratic functions our proof heavily relies on the construction of the feature map of the NTK, which is harder for the high-degree case.
Remark:
Due to constrained space, the proof of aforementioned theorems can be found in Appendix C.I to C,3
5

---

3.3 Spectral analysis
In this section, we characterize the approximation properties of N- -egeree PNNs in the in-distribution regime. By studying the spectral analysis in the form of a Mercer decomposition, we explicitly show the eigenvalues and eigenfunctions NTK. We firstly introduce some notation. Denote by
[-= (((.k) the spherical harmonics of degree k in d + 1 variables. G/ represents the Gegenbauer
polynomials with respect to the weight function z + (1 - = à and degree k. Finally, denote by F(d, kb) == 2k+d-1
d-1 (k)+d-2).
The following lemma enables us connect spherical harmonics to Gegenbauer polynomials.
Lemma 1. (Frye and Efthimiou 2012 Theorem 4.11] For any x, x, E g", the de-eeree spherical harmonics in d + variables satisfies:
F(d.k)
Σ Yu)())Y -
= ( F(d,k) ((..)))
For any dot product Mercer kernel K", denote by (#x) the eigenvalues associated to the kernl,, we can apply the following Mercer's decomposition in the form of spherical harmonics, and using Lemma 1 we obtain:
K' =
Σ k=0
20
146 Σ J=1
F(d.))
(e))
((x, Σ ph F(d, +44) ((x)))))))))))))))))) k=0
(5)
In order study the decay rate of the eigenvalues, we can express the NTK as the product of multiple kernels and present the decay rate of the eigenvalues of PNNs.
Theorem 7. Consider PNNs with N N-degree (N 2 2) multiplicative interactions and denote by (p=))== the eigenvalues associated to the NTK. Then for k >> d, we have P* = ()(((()))))))))
The proof can be found in Appendix D As comparison, the decay rate for both deep fully- connecce NNs and residual NNs is ((k Belfer et al. 2021]]. Thus, we can see a slower decay rate when inserting Hadamard product into standard NNs, which leads a faster learning towards high-frequency functions.
4
Experiments
Our experiments are organized as follows: We firstly showcase the extrapolation of NNs-Hp in learning some common functions in Section 4.1 Next, we assess the extrapolation performance on non-synthetic dataset in Section 4.2 and conduct the experiment learning spherical harmonics in Section 4.3 Due to the constrained space, the extrapolation in visual analogy task and the spectral bias in image classification task are deferred Appendix E.5 and Appendix [.6] respectively.
4.1 Extrapolation in learning analytically-known functions
These experiments aim to examine the extrapolation behavior of NNs-Hp in regression tasks. Our first experiment includes training the networks via the squared loss to fit several well-known and malytically-knoww underlying functions. During prediction, we sample data points beyond the training regime and observe the extrapolation performance. More details implementation
be found in Appendix E.I We set the target function as fa(r) x + x² 10r + 5 and use four-layer fully-connected NN. As presented in Figure aa) and Figure fully-connede
NN extrapolates linearly while NN-Hp approximates better the extrapolation part of the underlying non-linear function, which are consistent with Theorem 4 and Theorem
Learning fx) = (2x). We choose cleven-layer fully-connected NN. The training set and testing set the same in the previous experiment. Observing Figure I[c) and Figure [[d), find that NN-Hp is more flexible to learn the non-linear function outside the training region while fully-connected NN still extrapolates linearly.
Learning f (x) (1) " (2)) where (1) and (2) the first and second dimension of a: e R². In this task, we choose lhree-layer NNs. Each model is trained with different data distribution, i.c.,

---

Underyynng
Prediction within training region
Prodiction with extrapolstion region
NN-Hip
NN-Fip
3.00
300
35
08
Figure 1: Extrapolation function. The blue curve indicates the training regime while the pink color symbolizes the extrapolation regime. (a) and (b) show the fitting results towards fp(x)) x² + 2² 10x + 5. We can see that NN extrapolates linearly without the Hadamard product (Hp) while NN-Hp is able to extrapolate to the underlying non-linear function nearly. (c) and (d) present the fitting results towards fz(z) cos(2x). Notably, NN-Hp more flexible learn the non-linear function outsides the training region.
Underlying
Prediction within extrapolation region
Training region
NN-Hp
NN-Hp, fixed training region
NN
-2
-2 -2
-1 -2
Training region (TopDown)
Fixed training region (TopDown)
Training region (TopDown)
Figure 2: Fitting results for the underlying function fo(æ) = (z(1)² + (z:22) where x:() and (2)) are the first and second dimension of x E R². Blue points indicate the training regime, pink points symbolize the extrapolation regime, gray points indicate the underlying function. Left: We train NNs-Hp with the training set containing support in all directions, the network is able to extrapolate successfully. Middle: We train NN-Hp with the training wherein two dimensions of the data fixed to be positive, NN-Hp fails to extrapolate. Right. We remove the Hadamard product of the network, which leads to linear extrapolation.
the training set contains support in all directions in the first case while two dimension of the training set are fixed to be positive in the second case. The result is visually depicted in Figure which shows that NNs-Hp can achieve exact extrapolation if the training set contains support in all directions and thereby validates Theorem 6 On the other hand, NNs without the Hadamard product fail to extrapolate to the underlying function due to its linear extrapolation.
4.2 Extrapolation in real-world dataset
In this section, we assess the extrapolation performance beyond synthetic datasets.
Variation of brightness. This experiment conducted on two well-known grayscale image datasets: MNIST dataset [LeCun et al., 1998) and Fashion-MNIS, dataset Xiao et al. [2017). For these two datasets, the original range of the pixel of each image [0, 1), we divided by 10 for the raw training set to construct the new one where the pixels range from 0 to 0.1. During extrapolation, we limit the range of the original testing set to [0, rmax] through division, where Timax E (0.1, 0.2, 0.3, ..., 1.0), as illustrated in the top two panels in Figure Then we feed these images into the trained network
and evaluate the accuracy. More details on implementation can be found Appendix F.2 The
accuracy is summarized in the two bottom plots of Figure] 3 We find that both networks achieve
7

---

similar accuracy in the case Tmax = 0.1 while inserting Hadamard product (Hp) into NN improves the performance during extrapolation.
Arithmetic extrapolation. Now we turn to more challenging task. As human we can usually extrapolate to arbitrarily large numbers in arithmetic. How do the neural networks perform during extrapolation? Following the setup of Bloice et al. 2020). we use MNIST dataset, where there are 100 different two-image combinations of the digits 0 9. We randomly pick up 90 combinations as the training set and the remaining 10 combinations as the extrapolation set. This problem is treated as regression instead of classification for higher error tolerance following Bloice et al. [2020]. addition, if we design the network for the training set and testing set. The network only outputs one single discrete value. However, we still measure the accuracy by rounding the network output. five-layer folne-co NNs and convolution NNs are chosen as the baselines. For comparison, we implement NN-Hp with dense layers and NN-Hp with convolution layers, respectively. More details on the implementation can be found in the Appendix EE.3 The results obtained by three-fold cross validation are summarized in Table
classifier, the number of the class will vary as the change of the splitting
where we can see NN-Hp has better extrapolation behavior in such more difficult task.
1
1
2
2
2
2
2
2
2
81
5
5
27
27
22
17
J7
JA
6
6
6
83
18
83
10
10
01
10
01
88
7
3
7
2
2
Training
Extrapolation
Traiiing
Extrapolation
Accuracy
0.9
Accuracy
0.9
0.8
MN-hip
NN
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.3
0.4
0.6
0.7
0.8
1.0
Fina
(a) Examples and results MNIST dataset.
(b) Examples and results on Fashion-MNIse dataset.
Figure 3: The top two panels show the examples of extrapolation in MNIST dataset and Fashion- MNIST dataset. 7imax varies from 0.1 to 1.0. from left to right, indicating the variation of the darkness of the image. The bottom two panels show the accuracy as Ymax increasing. Both networks achieve similar accuracy in the Tmax = 0.1 while inserting Hadamard product (Hp) into NN improves the performance during extrapolation.
0+2
3 3+4
|+1
I+2
4 4
Network
2
2
Training
Extrapolation
6+
1 9
Unseen pair
Seen pair
Figure 4: A schematic illustration for the task of arithmetic extrapolation.
4.3 Spectral bias learning spherical harmonics
Here, we aim to learn the linear combinations of spherical Harmonics where inputs are sampled from the uniform distribution on the unit sphere. Our experiment follows the setup Choraria et al. 2022), which only considers NNs-Hp with one Hadamard product. The target function is

---

Table 1: Results in the task of arithmetic extrapolation, which aims to predict the target label with regression. 'Interpolation' indicates the accuracy in the seen pairs during training while "Extrapolation indicates the accuracy tested those unseen pairs. Three ways are used for the network output: (a) Rounding, the output is rounded to the nearest integer. (b) Floor/ceiling. A floor and ceiling function applied for the output and if one of those equals to the ground truth label, we treat it as a correct prediction. (c) +1. An error of =1 is allowed. We can observe that NN-Hp has better extrapolation behavior compared with the baselines.
Method
Rounding
Floor/ceiling
=1
Interpolation
0.980 0.002
0.999 0.000
0.999
0.000
NN(Dense)
Extrapolation
0.436 + 0.065
0.805 + 0.042
0.887 + 0.011
Interpolation
0.926 0.031
0.996 0.001
0.999 0.000
NN-Hp (Dense)
Extrapolation
0.554 0.011
0.802 = 0.010
0.889 0.008
Interpolation
0.945 0.983
0.983
0.021
0.994 0.007
NN(Conv)
Extrapolation
0.617 + 0.103
0.918 + 0.016
0.953 + 0.006
Interpolation 0.991 0.002
0.998 1 0.000
0.999 1 0.000
NN-Hp (Conv)
Extrapolation
0.825 0.109
0.948 0.006
0.963 + 0.007
k
k
k= 5
=
k= 12
3-degree
6-degree
9-degree
Redidial Redidcal length (log ccale)
Ressid all projection length (log erali)
Raciddal projection longth (log sosia)
projection
-10
-15
Itarationa
Iteratinn
Figure 5: Comparison of convergence curve of error projection lengths for NNs-Hp with N- degree multiplicative interactions, where (3,6,9) for different order harmonies with K E 31,3, 4, 8, 12} We see the improvement for high-degree interactions in the rate of convergence of error.
defined by: f*(x) 1 ΣLEK A&,x((x, K K (1,3,4,55, where Px(t) denotes NiK) the k-degree Gegenbauer polynomial, are fixed vectors that independently sampled from uniform distribution on unit sphere, and (CC) is the normalizing constant. The error residuals with different K. compared during the training process. Implementation details are deferred to Appendix E.4 In this experiment, we show that increasing the number of multiplicative interactions can improve the rate of convergence of error, presented in Figure
5
Conclusion
This paper examines neural network with Hadamard product with a particular focus on polynomial neural networks from theoretical perspective. The analysis of the NTK paves the way for knowing interesting properties of the networks, such as the extrapolation behavior and the spectral bias. Experimental results in learning analytically-knoww functions validate our hypothesis. We further conduct several experiments in real-world datasets and demonstrate the advantage of inserting Hadamard products into standard neural networks. We believe not only our framework provides good justification for deeper understanding of neural architecture, but it also lays the foundations to investigate other more complicated OOD problems such domain adaption and invariant learning in future work.

---

Acknowledgements
We are thankful to the reviewers for providing constructive feedback. This project has received support from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement number 725594 timedata). This work was sponsored by the Army Research Office and was accomplished under Grant Number W91 W911NF-19-1 0404. This work was supported by Zeiss. This work has received funding from SNF project - Deep Optimisation of the Swiss National Science Foundation (SNSF) under grant number 200021 20112000011..1 This work was supported by Hasler Foundation Program: Hasler Responsible Al (project number 21043).
References
Sina Alemohammad, Zichao Wang, Randall Balestriero, and Richard Baraniuk. The recurrent neural tangent kernel. In International Conference on Learning Representations (ICLR). 2021.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. convergence theory for deep learning via over- parameterizaaton. In International Conference on Machine Leurning (ICML), pages 242-252.
PMLR, 2019.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint 1r070:1907.022 0.09900,..,..... 2019.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in neural information processing systems (NeurIPS), 32, 2019.
Francesca Babiloni, Ioannis Marras, Filippos Kokkinos, Jiankang Deng, Grigorios Chrysos, and Stefanos Zafeiriou. Poly-nl: Linear complexity non-local layers with 3rd order polynomials. In International Conference on Computer Vision (ICCV), 2021.
Francis Bach. Breaking the curse of dimensionality with convex neural networks. The Journal of Machine Learning Research, 18(1):629-68)) 2017.
Randall Balestriero, Jerome Pesenti, and Yann LeCun. Learning in high dimension always amounts to extrapolation. arXiv preprint 0rXiv:2110.09485, 2021.
Etienne Barnard and LFA Wessels. Extrapolation and interpolation in neural network classifiers. IEEE Control Systems Magazine, 12(5):50-53, 1992.
Yuval Belfer, Amnon Geifman, Meirav Galun, and Ronen Basri. Spectral analysis of the neural tangent kernel for deep residual networks. arXiv preprint arXiv:2104.0300 2021.
Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. In Advances in neural information processing systems (NeurIPS), 2019.
Marcus Bloice, Peter Roth, and Andreas Holzinger. Performing arithmetic using a neural network trained on digit permutation pairs. In International Symposium Methodologies for Intelligent Systems, pages 255-264 Springer, 2020.
Antony Browne. Representation and extrapolation in multilayer perceptrons. Neural computation, 14 (7):1739-1754, 2002.
W.M. Campbell and C.C. Broun. Using polynomial networks for speech recognition. In Neural Networks for Signal Processing X. Proceedings of the 2000 IEEE Signal Processing Society Workshop (Cat. No.000H8801), volume 2, pages 795-803 vol.2, 2000. doi: 10.1..0/ 890159.
Ding-Xuan Zhou, and Quanquan Gu. Towards understanding the Yuan Cao, Zhiying Fang, Yue Wu, Ding- Xuan spectral bias of deep learning. arXiv preprint arXiv: 1912.01/9 2019.
Carlitz. The product of two ultraspherical polynomials. Glasgow Mathematical Journal, 5(2): 76-79, 1961.
10

---

Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in neural information processing systems (NeurIPS), 32, 2019.
Moulik Choraria, Leello Tadesse Dadi, Grigorios Chrysos, Julien Mairal, and Volkan Cevher. The spectral bias of polynomial neural networks. In International Conference on Learning Representa- tions (ICLR), 2022.
Grigorios Chrysos, Stylianos Moschoglou, Yannis Panagakis, and Stefanos Zafeiriou. Polygan: High-order polynomial generators. arXiv preprint arXiv:1908.0657 2019.
Grigorios Chrysos, Markos Georgopoulos, and Yannis Panagakis. Conditional generation using polynomial expansions. In Advances in neural information processing systems (NeurIPS), 202 la.
Grigorios G. Chrysos, Stylianos Moschoglou, Giorgos Bouritsas, Yannis Panagakis, Jiankang Deng. and Stefanos Zafeiriou. P-nets: Deep polynomial neural networks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Grigorios G. Chrysos, Stylianos Moschoglou, Giorgos Bouritsas, Jiankang Deng, Yannis Panagakis, and Stefanos P Zafciriou. Deep polynomial neural networks. IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 2021b.
Grigorios G Chrysos, Markos Georgopoulos, Jiankang Deng, Jean Kossaifi, Yannis Panagakis, and Anima Anandkumar. Augmenting deep classifiers with polynomial neural networks. In European Conference on Computer Vision (ECCV), 2022.
Amit Daniely, Roy Frostig, and Yoram Singer. Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity. In Advances in neural information processing systems (NeurIPS), 2016.
Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. In International Conference Learning Representations (ICLR), 2018
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International Conference on Machine Learning (ICML), 2019a.
Simon S Du, Kangcheng Hou, Russ R Salakhutdinov, Barnabas Poczos, Ruosong Wang, and Keyulu Xu. Graph neural tangent kernel: Fusing graph neural networks with graph kernels, Advances in neural information processing systems (NeurIPS), 32, 2019b.
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over- wver-parameterized neural networks. In International Conference on Learning Representations (ICLR), 2019c.
Feng-Lei Fan, Mengzhou Li, Fei Wang, Rongjie Lai, and Ge Wang. Expressivity and trainability of quadratic networks. arXiv preprint arXiv:2110.0608) 2021.
Rizal Fathony, Anit Kumar Sahu, Devin Willmott, and J Zico Kolter. Multiplicative filter networks. In International Conference on Learning Representations (ICLR), 2021.
Christopher Frye and Costas J Efthimiou. Spherical harmonics in p dimensions. arXiv preprint arXiv:1205.3548 2012.
Pamela J Haley and DONALD Soloway. Extrapolation limitations of multilayer feedforward neural networks. In International Joint Conference on Neural Networks, volume 4, pages 25-30. IEEE, 1992.
Kaixuan Huang, Yuqing Wang, Molei Tao, and Tuo Zhao. Why do deep residual networks generalize better than deep feedforward networks?-a neural tangent kernel perspective. Advances in neural information processing systems (NeurIPS), 33:2698 27099, 2020.
Alexey Grigorevich Ivakhnenko. Polynomial theory of complex systems. IEEE trunsactions on Systems, Man, and Cybernetics, 1971.
11

---

Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization neural networks. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett, editors, Advances in neural information processing systems (NeurIPS). Curran Associates, Inc., 2018.
Siddhant M. Jayakumar, Wojciech M. Czarnecki, Jacob Menick, Jonathan Schwarz, Jack Rae, Simon Osindero, Yee Whye Teh, Tim Harley, and Razvan Pascanu. Multiplicative interactions and where to find them. In International Conference on Learning Representations (ICIR), 2020.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review, 2009.
Wouter M Kouw and Marco Loog. A review of domain adaptation without target labels. IEEE fransactions on pattern analysis and machine intelligence, 43(3):766-785, 2019.
Mark A Kramer and JA Leonard. Diagnosis using backpropagation neural networks analysis and criticism. Computers & chemical engineering, 14(12): (1111))::323-13 1338, 1990.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11 (621))22278,322 1998.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems (NeurIPS), volume 32, 2019.
Georg Martius and Christoph H Lampert. Extrapolation and learning equations. arXiv preprint arXiv:1610.0299: 2016.
Quynh Nguyen, Marco Mondelli, and Guido F Montufar. Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep relu networks. In International Conference on Machine Learning (ICML), pages 81 19-8129, 2021.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference Machine Learning 2019.
Elan Rosenfeld, Pradeep Kumar Ravikumar, and Andrej Risteski. The risks of invariant risk mini- mization. In International Conference on Learning Representations (ICLR), 2021.
Subham Sahoo, Christoph Lampert, and Georg Martius. Learning equations for extrapolation and control. In International Conference on Machine Learning (ICML), pages 4442-4450 2018.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. arXiv preprint arXiv: 01557, 2019.
Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards nut-of-distribution generalization: A survey, arXiv preprint arXiv:2 10..20088.330000 13244, 2021.
Yoan Shin and Joydeep Ghosh. The pi-sigma network: An efficient higher-order neural network for pattern classification and function approximation. In International Joint Conference on Neural Networks, 1991.
Jiahao Su. Wonmin Byeon, Jean Kossaifi, Furong Huang. Jan Kautz, and Anima Anandkumar. Convolutional tensor-train Istm for spatio-temporal learning. Advances in neural information processing systems (NeurIPS), 33:13714-13722 2020.
Tom Tirer, Joan Bruna, and Raja Giryes. Kernel-based smoothness analysis of residual networks. arXiv preprint 2rXhv:2009.10008 2020.
12

---

Jan H van Schuppen. Control and System Theory of Discrete- Time Stochastic Systems. Springer, 2021.
Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.3027, 2010.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Taylor Webb, Zachary Dulberg, Steven Frankland, Alexander Petrov, Randall O'Reilly, and Jonathan Cohen. Learning representations that support extrapolation. In International Conference on Machine Learning (ICMI., pages 10136 10136-10146. PMLR, 2020.
Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference Learning Theory, pages 3635-3673. PMLR, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXin: 2017.
Keyulu Xu, Mozhi Zhang, Jingling Li, Simon Shaolei Du, Ken-Ichi Kawarabayashi, and Stefanic Jegelka. How neural networks extrapolate: From feedforward to graph neural networks. In International Conference on Learning Representations (ICLR), 2021.
Huiming Zhang and Song Xi Chen. Concentration inequalities for statistical inference. arXiv preprint arXiv:2011.02256 2020.
Zhenyu Zhu, Fabian Latorre, Grigorios Chrysos, and Volkan Cevher. Controlling the complex- ity and lipschitz constant improves polynomial nets. International Conference on Learning Representations (ICLR), 2022.
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes over- parameterized deep relu networks. Machine learning, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Appendix !
(c) Did you discuss any potential negative socictal impacts of your work? [Yes] See
Appendix H
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] We provide detailed derivations in the appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No] The datasets we are all publicly available and commonly used in image-related tasks. We intend to release the source code upon the acceptance of the paper, which should enable interested practitioners to further improve our framework and verify our results.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We conduct three runs our quantitative experiments,
e.g., Table [
13

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] All our experiments are conducted on single GPU, which is part of our cluster.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A] This is theoretical work; we either use synthetic experiments, e.g. analytic functions the target, or standard image-based datasets, All of the public datasets are publicly available and we cite their respecting papers, where the licenses are mentioned.
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're rre using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "z-9ldjKEsR2", "reviewer": "Reviewer_SBXv", "summary": "This paper studies NNs with Hadamard products in relation to NTKs. Then it compares the NNs-Hp with standard neural networks with respect to spectral bias and extrapolation. Contributions of the paper include both theory and experiments.", "questions": "I would like to see more experiments on extrapolation datasets used in the literature. Specifically, VAEC dataset seems like a good choice as authors have already cited the paper that introduced it.\n\nThere should be discussion and clarification making it clear what authors mean by extrapolation vs OOD.\n\nAll image classification tasks require extrapolation. Authors can experiment with standard image classification datasets to demonstrate effectiveness of their analysis regarding extrapolation. See this paper:\n\nBalestriero, R., Pesenti, J. and LeCun, Y., 2021. Learning in high dimension always amounts to extrapolation. arXiv preprint arXiv:2110.09485.\n\n\nIf authors would like to extrapolate with feedforward networks, they can consider the models trained by Neyshabur on MNIST and CIFAR datasets.\n\nNeyshabur, B., 2020. Towards learning convolutions from scratch. Advances in Neural Information Processing Systems, 33, pp.8078-8088.\n\nIn the introduction subsection on extrapolation, I am not sure why there is emphasis on limitation of other models in non-linear extrapolation. For feedforward networks, one can easily use non-linear activation functions, and that could lead to non-linear extrapolation and interpolation.\n\nI'm also not sure why authors don't consider interpolation properties of their model when they provide such analysis for extrapolation. These seem to belong to a single paper, and not two separate papers?\n\nHave authors considered the fact that for image classification, all testing samples fall outside the convex hull of training set and all classifications require some considerable amount of extrapolation?\n\nWhen authors use the term extrapolation, do they mean any function evaluation outside the convex hull of training set? Sometime authors seem to interchangeably refer to this as out-of-distribution. These two terms are not necessarily equivalent.", "limitations": "I think authors have not discussed the limitations of their work. Their answer to the questionnaire claims that limitations are explained in Section 5, but I don't see such discussion in the single paragraph of Section 5.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThis is an interesting work. The approach and the theory appear to be sound.\n\nWeaknesses:\nEmpirical results can be improved in my opinion. Specifically, for extrapolation there are benchmark datasets that authors have not considered. It would be insightful to see their results on such datasets and the comparison with prior work. For example, authors can experiment with the VAEC dataset provided by Webb et al which they have cited.\n\nWebb, T., Dulberg, Z., Frankland, S., Petrov, A., O’Reilly, R. and Cohen, J., 2020. Learning representations that support extrapolation. In International conference on machine learning (pp. 10136-10146). PMLR.\n\nThere seems to be inaccuracies in the literature review of extrapolation. For example, the statement \"Xu et al. [2021] theoretically and empirically point out that 2-layer fully-connected NNs can only extrapolate to linear function.\" seems inaccurate, because Xu et al make that claim only for NNs with ReLU activation. If someone wants to extrapolate non-linearly, they can use NNs with non-linear activation functions.\n\nThere are more inaccuracies in that literature review. For example, the statement \"On the contrary, NNs usually lack such [extrapolation] ability\" again seems inaccurate. Note that all image classification tasks amount to extrapolation and NNs have the ability to perform such extrapolations. For example, see the reference below from NeurIPS 2002 which uses SVMs to extrapolate and build a classification model for the MNIST dataset.\n\nHaffner, P., 2001. Escaping the convex hull with extrapolated vector machines. Advances in Neural Information Processing Systems, 14.\n\nTherefore, to study extrapolation capability of NNs, one dataset to experiment with is the actual MNIST dataset.\n\nThe remark on lines 145-148 seems unclear and hand wavy. What is the contribution of this paper with regards to Theorem 1 and the value of \\rho?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "p0MD640yi6S", "reviewer": "Reviewer_ckdZ", "summary": "The paper presents a complete study of Neural Tangent Kerel (NTK) for neural networks with Hadamard products (NN-Hps). The equivalence with kernel regression is first established. Based on this result, the authors establish two advantages of NN-Hps over fully-connected NNs: better extrapolation performance and slower-decaying sprectrum.", "questions": "1. I am personally not familiar with Gegenbauer polynomials and spherical harmonics. Could the authors explain the implications of the experiment in subsection 4.3? Specifically, how is the degree of the Gegenbauer polynomials related to the frequency of the information?\n\nMoreover, according to Theorem 6, the decay rate of the spectrum is independent with $$N$$ as long as $$N\\ge2$$. The experiment, however, shows that increasing $$N$$ makes the decay rate slower. Could the authors further explain this contradiction?", "limitations": "The limitations of this paper are well evaluated by the authors.", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The authors establish two advantages of NN-Hps over fully-connected NNs. Both are important and convincing.\n\nWeaknesses:\n1. The remark for Theorem 2 lacks some rigure. If $$\\hat K_t$$ and $$\\hat K_0$$ are both functions of $$n$$, then one cannot write $$\\hat K_t\\to\\hat K_0$$ as $$n\\to\\infty$$. Moreover, should the condition be $$m\\to\\infty$$ rather than $$n\\to\\infty$$?\n\n2. Theorem 5 requires more interpretation. The requirement that $$\\{x_i\\}$$ includes the orthogonal basis is quite weird, which may not hold in practical cases. Moreover, this theorem only studies the case of quardratic functions.\n\n3. The figures of the experiment for learning are a bit confusing, and a better explanation is required for different colors. The figure itself is blurry as well.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "a4wYlrbAU3", "reviewer": "Reviewer_cA3P", "summary": "This paper derives the neural tangent kernel (NTK) of neural networks \"with hadamard products\" (NN-Hp), and proves several related results: \n1) convergence at initialization and during training on least squared regression;\n2) extrapolation outside the domain of training data (OOD);\n3) decay of the eigenspectrum of the kernel.\n\nNN-Hp are basically deep fully connected ReLU neural networks in which the output of each layer is multiplied (element-wise) by the output of another layer processing the input. \nThis kind of architecture is used e.g. in Style-GAN, and is compared with standard deep fully connected ReLU neural networks (NN). \nThe paper proves that NN-Hp, in contrast to NN, extrapolates non-linearly for OOD data, and that the NTK spectrum of NN-Hp decays more slowly that that of NN, implying a prominence of higher frequencies.\nExperiments are shown in support of the theoretical results.\n", "questions": "Is it possible to test standard NN on the experiment of Figure 5 (in addition to NN-Hp)? \nComparing NN and NN-Hp on the problem of fitting harmonics at different frequencies would provide support for Theorem 6, which is currently untested. \nAs far as I understood the eigenfunctions are identical for the kernels of NN and NN-Hp, only the eigenvalues differ, therefore the comparison would be fair.", "limitations": "I could not find any description of the limitations of this work, even though the checklist says \"yes\"", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is very well written and clear.\n- This problem is interesting, it fills a hole in the literature.\n- The coverage of the topic is substantial, the density of results in the paper is high.\n\nWeaknesses:\n- I was a bit disappointed that little discussion was given about the result on the eigenspectrum. What does that imply?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "D8dKGdNZCEJ", "reviewer": "Reviewer_rMyZ", "summary": "This paper derives the NTK of Hadamard-product neural networks (NNs-Hp). Through an analysis of the NTKs of NNs-Hp, the paper shows that NNs-Hp and standard MLPs have different generalization abilities in the extrapolation regime. These understandings are then illustrated using numerical experiments. ", "questions": "My main question is if the authors can clarify the relationship between NNs-Hp and polynomial neural networks (PNNs). It seems to me that NNs-Hp should be used to refer a generic class of neural networks with Hadamard product. This study, however, is not generally applicable to this wide class of networks. Indeed, as the authors pointed out in line 127-128, their study focuses on polynomial neural networks (PNNs). In that case, I would suggest the authors to position their contribution in the context of PNNs and avoid overloading NNs-Hp with PNNs.", "limitations": "The current paper focuses on a specific type of neural network with Hadamard product -- the polynomial neural network. To me, it seems difficult to distinguish whether the phenomenon observed in the numerical experiments is a result of polynomial parametrization or a result of the usage of the Hadamard product in neural networks. It would be great if the authors can clarify this. \n\nIn addition, the evaluation of the extrapolation ability of NNs-Hp are confined to toy-like experiments. In my opinions, some of these extrapolation experiments, such as fitting sine curves or spherical harmonics, may be biased toward giving NNs-Hp an advantage by design: The underlying functions are close to polynomials, and PNNs provide a good inductive bias in these tasks. As a result, while it is good to see that NNs-Hp perform better in these cases, their advantage in out-of-distribution generalization in such scenarios are not too surprising. ", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**: The main contribution of this paper -- the derivation of NTK of Hadamard-product neural networks -- appears novel and meaningful to me. \n\n**Weakness**:\nIt is unfortunate that the paper contains a lot of typos and grammatical glitches. Examples include:\n\nLine 128: while we still use the term NN-Hp for consistence -> consistency\n\nLine 242: we access the extrapolation performance beyond synthetic datasets. -> we assess\n\nLine 280: The analysis of the NTK paves the way for knowing interesting proporties -> properties\n\nIn the caption of Table 1: (a) Rounding, the output is rounded to the the nearest integer. -> extra \"the\"\n\nLine 244: This experiment is preformed on ... -> performed\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept