Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper provides a theoretical justification for why feature learning (theoretical proxy for that is the F_1 space) may be crucial as opposed to learning with fixed features (theoretical proxy for that is the RKHS with NTK or F_2 space) when using statistical pseudometrics that are computed using the sup over function spaces (e.g. IPM pseudometrics and Stein discrepancy) to train a generative model to match a given training distribution. In particular, they construct sequences of pairs of distributions for which they can prove that as d increases, F_1 pseudometrics yield much higher “distance” between the distributions than F_2 pseudometrics. In particular, the mathematical exercise suggests that distributions that differ in high frequency components benefit more from feature learning.
main_review: Strengths:

The results and the presentation of the mathematical results are clean and solid (however I have not had a chance to check the proofs in detail). The story that feature learning is beneficial as opposed to fixed kernel is not very surprising not only empirically but also theoretically if the reader is familiar with the F_1 vs F_2 literature. Still, showing it for generative modeling explicitly is novel and could be valuable. 

Constructive comments:

In order to make the results more impactful within the ML community however, the results seem to directly suggest an experiment to link back the theoretical insights to the practical motivation: i.e. can you design an experiment that illustrates that the insight from the distribution pairs (different high frequency components) that are indistinguishable as d-> infty for F_2, carries over to the image scenarios that motivate this study?For example, something that immediately comes to mind is that you could remove/project out the high frequency components and see if the FID without feature learning then gets better

Moreover, the presentation could be much improved. Here is some more detailed feedback 

- The figures are quite poorly presented - use different linestyle. The two red curves in Figure 3 - which is which?

- For IPM your theoretical prediction is quite accurate. For Stein discrepancy - it’s not clear that the ratio is actually increasing with d empirically albeit being high in general. I’m missing a discussion on these Stein discrepancy simulations in general

- Section 7 and appendix generally seem relatively underdeveloped - the relevance of assumptions and implications of Theorem 5 should be directly discussed and interpreted in order for the reader to know what to focus on while parsing it and perhaps the proof. Furthermore, the switch from F_2 to Ftilde_2 is not well motivated - what do we learn about F_2 by looking at Ftilde_2 instead? 

- You mention setting k=d near the discussion of the implications of the theorem and later in the experiments set k constant for all d - discussion would help

- Dimensionality - In sections 3, 4, and 7, the inputs are considered to lie in a subset K of the (d+1)-dimensional space (l. 114), and the paper considers statistical distances between distributions \mu and \nu over K, for two special cases: K=R^d \times {1} or K=S^d. However in sections 5 and 6, the distributions \mu_d and \nu_d are defined over S^{d-1}. Might be useful to keep it consistent

Minor comments

- l. 178: define the acronym EBM, and add the relevant references again
- line 171 it seems like eq (6) should read eq (5)
- eq (33) is the same as (6) - was a bit confusing in the main text 

limitations_and_societal_impact: purely theoretical work
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors propose theoretical results about separation between probability metrics with fixed-kernel and feature-learning discriminators in overparameterized two-layer neural network spaces.
main_review: The authors propose theoretical results about separation between probability metrics with fixed-kernel and feature-learning discriminators in overparameterized two-layer neural network spaces.

In Theorem 2 and 3, the authors derive the ratio between feature-learning $F_1$ and fixed-kernel $F_2$ integral probability metrics (IPM), and for $F_1$ and $F_2$ Stein respectively. The authors illustrate that on the hyper-spheres, the $F_1$ IPM/Stein has much larger values than $F_2$ IPM/Stein which seems to infer that the separation of $F_1$ IPM/Stein is better than $F_2$ IPM/Stein (line 170-175 and line 214-217). 
+ Do the $F_1$ IPM/Stein have different range values than $F_2$ IPM/Stein? (since it is unclear whether their values have a same range)
+ Does the separation relate to the range value of a discrepancy d? since with k>1, the discrepancy kd is k-times larger than discrepancy d, does it imply the separation of discrepancy kd is better than that of discrepancy d? 
--> It is better in case the authors give a more formal definition of "separation". How does the "separation" relate to the value of a discrepancy between probability measures?

Some other concerns:
+ in line 108, $F_2 \subset F_1$, could the authors elaborate this property with more details? Can one use this property for $F_1$ and $F_2$ IPM directly since the IPM is maximum over $F$ space?
+ For results in Section 7.2, what is the relation between $F_2$ and $\tilde{F}_2$?
+ In experiments, how does the degree of Legendre polynomial affect the results?

Some minor concerns:
+ For the sliced Wasserstein, I think it is better to cite the work of Rabin et al, 2011 which is the first paper about sliced-Wasserstein.

References:
+ Julien Rabin, Gabriel Peyré, Julie Delon, and Marc Bernot. Wasserstein barycenter and its application to texture mixing. In International Conference on Scale Space and Variational Methods in Computer Vision, pages 435–446, 2011.

------After the rebuttal-----
I thank the authors for the rebuttal. I think that it is better in case the authors improve the clarification about those raised points (I feel that there are several implications that make it hard for the readers to access its contributions. 
From the discussions with other reviewers, the theoretical part is good, but the contribution can be further improved with some well-designed experiments to illustrate the proposed theoretical results.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: The authors study two classes of functions, F1 and F2, which are related to
"infinite-width" two-layer neural nets and fixed-kernel RKHSes. They show
that in the regime where d goes to infinity, the sensitivity of MMDs based
on F2 decreases exponentially comapred to F1 for specific pairs of measures
on the hypersphere S^d. The authors corroborrate this theoretical find with
some simulations.
main_review: While mostly a theoretical contribution, the work here seems like a concrete
addition to understanding the differences between F1 and F2. There are also
some nice connections made to the sliced and max-sliced Wasserstein metrics
which establishes some nice bounds for these quantities.

Originality:
The constructive examples given in the paper seem quite carefully
constructed. While this work builds upon much other recent work, the core
theorems are original and elucidating.

Quality:
The claims in the paper look correct. Outside of a couple ambiguous parts,
the paper was well written.

Clarity:
Despite the difficulty of the subject, the paper was quite pleasant to read.

Significance:
While the paper may not have any immediate practical import, it does made a
theoretical contribution to an important problem in discriminator theory. In
order to bolster this area, I would suggest that the authors expound a bit
more on F1 and F2 and how they related to modern NNs/GANs in a bit more
detail at the beginning of the paper.

Detailed Comments:

L110-111: It might be worthwhile to expound a bit on what "single units"
are, as this is a key fact for claims that F1 is better than F2.

L155: "as x = r zeta_{(d)} with zeta_{(d)}," should belong to S^(d-1)
Theorem 2: The wording here is a little ambiguous. Does "we can choose k >=
0 and gamma_{k,d} in [-1, 1]" mean that we can choose any such values and
(5-7) will hold? Or does this mean we need a speciifc choice of values for
these to hold? Moreover, why does (5) not depend on gamma_{k,d}?

L194: Where is s_{nu} defined?

L276: Can you explain exactly what "by statistical errors" means?
limitations_and_societal_impact: The authors did not include an impact on the societal impact of their work
and should add one!
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper argues that discriminators based on learned features have significantly more discrimination power than discriminators based on fixed features. It does so by defining two function classes that can be interpreted as infinite width one hidden layer neural networks, one corresponding to the learned, or adaptive regime (F_1), and another corresponding to the fixed, or kernel regime (F_2). Various interesting separation and characterization results are shown between F_1 and F_2, using practically relevant metrics.
main_review: The paper is written very cleanly, and it has a clear message and demonstration of that message: learned features have greater discrimination power than a fixed kernel. The theoretical nature of the paper complements previous empirical demonstrations of this message.

The results also appear very clean. I am not familiar with constructions involving spherical harmonics, but within this area, they seem natural. A sequence of pairs of probability distributions defined by the Legendre harmonic is defined which exhibits an exponential gap between the F_1 IPM and the F_2 IPM, and the authors obtain closed form expressions for the IPMs, which is nice. I also found it interesting that Theorem 1 sheds light on when a previously defined "neural net distance" is actually a distance. The separation in terms of Stein discrepancy is also interesting since the Stein discrepancy is another popular metric used in discrimination tasks, and the within polynomial factor correspondences between F_1 IPM and max-sliced Wasserstein distance and between F_2 IPM and sliced Wasserstein distance is an interesting, unifying characterization. 

Since the paper is so clean and has useful results, I vote to accept. I seems like it would be a useful reference to other researchers in this area. 

A question that I have about the technical novelty of the paper is, how different is the usage of spherical harmonics here from the Bach 2017 reference. What did Bach 2017 use spherical harmonics for, and how similar are any constructions involving them between this paper and that one? 
A note: I did not see s_{nu} defined anywhere, in the definition of the kernelized stein discrepancy.
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: The authors present interesting theoretical analysis with wide-reaching implications; they are able to exhibit sequences of distributions which are distinguished by F1 integral probability metrics (based on an infinite width neural network function class that performs "feature learning") but not F2 integral probability metrics (a function class that does not perform "feature learning"), and in doing so they provide theoretical support for the notion that feature learning is an important property for discriminators (e.g. an argument against the use of "MMD GANs", which have previously appeared in NeurIPS).  The analysis is extended to cover Stein discrepancy, again based on F1 and F2.  All reviewers agreed that the paper was well-written and correct.  I believe these results will be of significant interest to the participants of NeurIPS.
Decision: Accept (Oral)