Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies NNs with Hadamard products in relation to NTKs. Then it compares the NNs-Hp with standard neural networks with respect to spectral bias and extrapolation. Contributions of the paper include both theory and experiments.
strengths_and_weaknesses: Strengths:
This is an interesting work. The approach and the theory appear to be sound.

Weaknesses:
Empirical results can be improved in my opinion. Specifically, for extrapolation there are benchmark datasets that authors have not considered. It would be insightful to see their results on such datasets and the comparison with prior work. For example, authors can experiment with the VAEC dataset provided by Webb et al which they have cited.

Webb, T., Dulberg, Z., Frankland, S., Petrov, A., O’Reilly, R. and Cohen, J., 2020. Learning representations that support extrapolation. In International conference on machine learning (pp. 10136-10146). PMLR.

There seems to be inaccuracies in the literature review of extrapolation. For example, the statement "Xu et al. [2021] theoretically and empirically point out that 2-layer fully-connected NNs can only extrapolate to linear function." seems inaccurate, because Xu et al make that claim only for NNs with ReLU activation. If someone wants to extrapolate non-linearly, they can use NNs with non-linear activation functions.

There are more inaccuracies in that literature review. For example, the statement "On the contrary, NNs usually lack such [extrapolation] ability" again seems inaccurate. Note that all image classification tasks amount to extrapolation and NNs have the ability to perform such extrapolations. For example, see the reference below from NeurIPS 2002 which uses SVMs to extrapolate and build a classification model for the MNIST dataset.

Haffner, P., 2001. Escaping the convex hull with extrapolated vector machines. Advances in Neural Information Processing Systems, 14.

Therefore, to study extrapolation capability of NNs, one dataset to experiment with is the actual MNIST dataset.

The remark on lines 145-148 seems unclear and hand wavy. What is the contribution of this paper with regards to Theorem 1 and the value of \rho?
questions: I would like to see more experiments on extrapolation datasets used in the literature. Specifically, VAEC dataset seems like a good choice as authors have already cited the paper that introduced it.

There should be discussion and clarification making it clear what authors mean by extrapolation vs OOD.

All image classification tasks require extrapolation. Authors can experiment with standard image classification datasets to demonstrate effectiveness of their analysis regarding extrapolation. See this paper:

Balestriero, R., Pesenti, J. and LeCun, Y., 2021. Learning in high dimension always amounts to extrapolation. arXiv preprint arXiv:2110.09485.


If authors would like to extrapolate with feedforward networks, they can consider the models trained by Neyshabur on MNIST and CIFAR datasets.

Neyshabur, B., 2020. Towards learning convolutions from scratch. Advances in Neural Information Processing Systems, 33, pp.8078-8088.

In the introduction subsection on extrapolation, I am not sure why there is emphasis on limitation of other models in non-linear extrapolation. For feedforward networks, one can easily use non-linear activation functions, and that could lead to non-linear extrapolation and interpolation.

I'm also not sure why authors don't consider interpolation properties of their model when they provide such analysis for extrapolation. These seem to belong to a single paper, and not two separate papers?

Have authors considered the fact that for image classification, all testing samples fall outside the convex hull of training set and all classifications require some considerable amount of extrapolation?

When authors use the term extrapolation, do they mean any function evaluation outside the convex hull of training set? Sometime authors seem to interchangeably refer to this as out-of-distribution. These two terms are not necessarily equivalent.
limitations: I think authors have not discussed the limitations of their work. Their answer to the questionnaire claims that limitations are explained in Section 5, but I don't see such discussion in the single paragraph of Section 5.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper derives the NTK of Hadamard-product neural networks (NNs-Hp). Through an analysis of the NTKs of NNs-Hp, the paper shows that NNs-Hp and standard MLPs have different generalization abilities in the extrapolation regime. These understandings are then illustrated using numerical experiments. 
strengths_and_weaknesses: **Strength**: The main contribution of this paper -- the derivation of NTK of Hadamard-product neural networks -- appears novel and meaningful to me. 

**Weakness**:
It is unfortunate that the paper contains a lot of typos and grammatical glitches. Examples include:

Line 128: while we still use the term NN-Hp for consistence -> consistency

Line 242: we access the extrapolation performance beyond synthetic datasets. -> we assess

Line 280: The analysis of the NTK paves the way for knowing interesting proporties -> properties

In the caption of Table 1: (a) Rounding, the output is rounded to the the nearest integer. -> extra "the"

Line 244: This experiment is preformed on ... -> performed


questions: My main question is if the authors can clarify the relationship between NNs-Hp and polynomial neural networks (PNNs). It seems to me that NNs-Hp should be used to refer a generic class of neural networks with Hadamard product. This study, however, is not generally applicable to this wide class of networks. Indeed, as the authors pointed out in line 127-128, their study focuses on polynomial neural networks (PNNs). In that case, I would suggest the authors to position their contribution in the context of PNNs and avoid overloading NNs-Hp with PNNs.
limitations: The current paper focuses on a specific type of neural network with Hadamard product -- the polynomial neural network. To me, it seems difficult to distinguish whether the phenomenon observed in the numerical experiments is a result of polynomial parametrization or a result of the usage of the Hadamard product in neural networks. It would be great if the authors can clarify this. 

In addition, the evaluation of the extrapolation ability of NNs-Hp are confined to toy-like experiments. In my opinions, some of these extrapolation experiments, such as fitting sine curves or spherical harmonics, may be biased toward giving NNs-Hp an advantage by design: The underlying functions are close to polynomials, and PNNs provide a good inductive bias in these tasks. As a result, while it is good to see that NNs-Hp perform better in these cases, their advantage in out-of-distribution generalization in such scenarios are not too surprising.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents a complete study of Neural Tangent Kerel (NTK) for neural networks with Hadamard products (NN-Hps). The equivalence with kernel regression is first established. Based on this result, the authors establish two advantages of NN-Hps over fully-connected NNs: better extrapolation performance and slower-decaying sprectrum.
strengths_and_weaknesses: Strengths:
1. The authors establish two advantages of NN-Hps over fully-connected NNs. Both are important and convincing.

Weaknesses:
1. The remark for Theorem 2 lacks some rigure. If $$\hat K_t$$ and $$\hat K_0$$ are both functions of $$n$$, then one cannot write $$\hat K_t\to\hat K_0$$ as $$n\to\infty$$. Moreover, should the condition be $$m\to\infty$$ rather than $$n\to\infty$$?

2. Theorem 5 requires more interpretation. The requirement that $$\{x_i\}$$ includes the orthogonal basis is quite weird, which may not hold in practical cases. Moreover, this theorem only studies the case of quardratic functions.

3. The figures of the experiment for learning are a bit confusing, and a better explanation is required for different colors. The figure itself is blurry as well.
questions: 1. I am personally not familiar with Gegenbauer polynomials and spherical harmonics. Could the authors explain the implications of the experiment in subsection 4.3? Specifically, how is the degree of the Gegenbauer polynomials related to the frequency of the information?

Moreover, according to Theorem 6, the decay rate of the spectrum is independent with $$N$$ as long as $$N\ge2$$. The experiment, however, shows that increasing $$N$$ makes the decay rate slower. Could the authors further explain this contradiction?
limitations: The limitations of this paper are well evaluated by the authors.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper derives the neural tangent kernel (NTK) of neural networks "with hadamard products" (NN-Hp), and proves several related results: 
1) convergence at initialization and during training on least squared regression;
2) extrapolation outside the domain of training data (OOD);
3) decay of the eigenspectrum of the kernel.

NN-Hp are basically deep fully connected ReLU neural networks in which the output of each layer is multiplied (element-wise) by the output of another layer processing the input. 
This kind of architecture is used e.g. in Style-GAN, and is compared with standard deep fully connected ReLU neural networks (NN). 
The paper proves that NN-Hp, in contrast to NN, extrapolates non-linearly for OOD data, and that the NTK spectrum of NN-Hp decays more slowly that that of NN, implying a prominence of higher frequencies.
Experiments are shown in support of the theoretical results.

strengths_and_weaknesses: Strengths:
- The paper is very well written and clear.
- This problem is interesting, it fills a hole in the literature.
- The coverage of the topic is substantial, the density of results in the paper is high.

Weaknesses:
- I was a bit disappointed that little discussion was given about the result on the eigenspectrum. What does that imply?

questions: Is it possible to test standard NN on the experiment of Figure 5 (in addition to NN-Hp)? 
Comparing NN and NN-Hp on the problem of fitting harmonics at different frequencies would provide support for Theorem 6, which is currently untested. 
As far as I understood the eigenfunctions are identical for the kernels of NN and NN-Hp, only the eigenvalues differ, therefore the comparison would be fair.
limitations: I could not find any description of the limitations of this work, even though the checklist says "yes"
review_title: null

Meta-review: metareview: This paper derives the NTK of Hadamard-product neural networks and shows the different behavior from the standard neural networks in terms of spectral bias in the extrapolation regime. After the author response and author-reviewer discussion, all the reviewers are in support of accepting this paper. Therefore, I recommend acceptance. Regarding the NTK-based optimization analysis, it seems that the following paper is not mentioned, which is a concurrent work with [Allen-Zhu et al.,2019, Chizat et al., 2019, Du et al., 2019a, 2018]. 
[*] Zou et al. Gradient Descent Optimizes Over-parameterized Deep ReLU Networks, Machine Learning, 2020. Please address the missing reference and prepare the camera ready by incorporating the author response.
confidence: Certain
Decision: Accept