Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper provides an analysis of radial neural networks from the viewpoint of expressivity and model compressivity.
strengths_and_weaknesses: I have appreciated the originality of the work, namely, exploring simultaneously in a same paper the pros and cons of these architectures. 
The main weaknesses is to me a lack of motivations: why should we use these networks instead of the radial basis function networks, for which the theory is more understood? After having read the paper, I am still unsure whether I should use this architecture or another, mostly for the following reason: it is unclear to me what expressivity we loose using those architectures. The authors only prove the ability of the model to represent simple continuous functions, but do not say exactly how expressivity compares to common architectures regarding state-of-the-art expressivity guarantees. Regarding paper clarity, I found the section about network compression hard to understand. The rest of the paper was well-presented. The paper is overall well written, I found few typos.
questions: Why should we use these networks instead of the radial basis function networks, for which the theory is more understood?
Intuitively, one may expect that applying the nonlinearity on the activation norm instead of each component of the activation vector decreases expressivity. The result you provide is very generic; would it be possible to characterize the loss of expressivity for more complicated functions?
How does the number of layers/width needed to approximate a continuous function to accuracy epsilon compare to those needed for other types of architectures?
Could you please give intuition of why radial neural nets are more compressible than other architectures?
Am I right that Q in Thm 7 depends on k? This should be clarified.
limitations: I do not expect this work to have any societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This study introduces a new deep radial neural network, shows its density in the space of `asymptotically affine' continuous functions w.r.t the sup norm, and presents a compression algorithm that has a similar taste with QR decomposition.
strengths_and_weaknesses: Strength:
The proof of the main theorem is constructive as well as simple. In traditional approximation theory, the decomposition of a function into ‘width’ direction has often been investigated. That is, decomposition into coefficients and bases (or frames). On the other hand, deep learning decomposes a function into function composites, a method that has not been well investigated by traditional approximation theory. This study provides a concrete example of how to decompose a nonlinear function in the ‘depth’ direction, and I consider this is highly valuable as a result of pioneering research.

Weakness:
As the authors also pointed out in the conclusion and discussion section, the main theorem (universality) holds only for step-relu function.

questions: Suggestion:
The authors introduced a new class of functions: Asymptotically affine. As pointed out in l.158, this class covers compactly supported functions. As a consequence, Theorem 3 further implies the so-called cc-universality (i.e., the density in the space of all continuous functions w.r.t. the topology of compact convergence.) I recommend the authors to clearly state that the cc-universality is a corollary of Theorem 3.

limitations: The authors have addressed the limitation in Section 8.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: Activation functions are a critical part of the design of neural networks. The choice of activation function in the hidden layers controls how well the network can model a training dataset. In this manuscript, the authors consider so-called radial neural networks, which have similarities to radial basis function neural networks. The authors show universal approximation in both infinite width and depth as well as present a projected gradient descent algorithm for training.  
strengths_and_weaknesses: SIMILARITY TO RBF NNs (WITH LOCAL LINEAR MODEL):
A radial basis function NN is given by (in the 2-layer case):

NN(x) = sum_{i=1}^N a_i*h(||x+b_i||)

It is sometimes convenient to extend a radial basis function NN with a local linear model to obtain (in the 2-layer case): 

NN(x) = sum_{i=1}^N (a_i + w_i^T(x+b_i))*h(||x+b_i||).

This model is more general than the radial neural networks introduced in this manuscript. In particular, set a_i = 0, w_i = c_i*e_i/||x+b_i||, where c_i is a scalar and e_i is the ith unit vector to obtain: 

NN(x) = sum_{i=1}^N c_i (x+b_i)/||x+b_i||*h(||x+b_i||) (= sum_{i=1}^N c_i rho(||x+b_i||)),

which I believe is an expression for 2-layer radial NN (as defined by the manuscript). Therefore, I do not find the NN architecture to be sufficiently novel to support publication.  

TRAINING: 
Sensitivity analysis explains the difficulties of gradient descent for radial basis function neural networks with smooth radial basis activation functions (see Karayiannis, "Gradient descent learning of radial basis neural networks." Proc. Inter. Conf. Neural Netw., IEEE, 1997). Do the authors observe similarly difficulties with radial neural networks? 

EXPERIMENTS: 
Separate from the novelty of the architecture and training, one would hope for experimental results that demonstrate a significant benefit over existing NN architectures (at least with regard to one example or metric).  The three experiments currently do not demonstrate any benefits but verify the theoretical contributions. 

WRITING 
The manuscript is well-written. Two typos are: 

- p1, in (1.1): I think you want rho : R^n->R^n at the start of (1.1). 
- p6, line 195: "strengthend" should read "strengthened"

questions: - Can you better connect the NN architecture to the literature and suggest why the NN architecture is novel? 

- Do the authors observe similarly training difficulties with radial neural networks as radial basis function neural networks?

- Can the authors add experiments to convince readers to use radial neural networks?
limitations: The work has little potential negative societal impact.
review_title: null

Meta-review: metareview: This paper is very strong in some regards -- it gives a clear technical setup, a sophisticated but accessible analysis (impressively so), and solid coverage of related work. At the same time, I believe that it is missing key pieces -- ones that would naturally concern a reader in the NeurIPS community. The paper mainly claims a theoretical advancement, and while the path to establishing it is interesting, the end result is not especially impactful in theory, partly because it is not especially connected to other work (whether theoretical or practical).

On the side of strengths, the paper is very clear and well written, including the technical walkthough. I appreciate the intuitive support that the authors work hard to ensure (e.g. Figure 3), and the grounding in constructed examples (e.g. Figure 4). Beyond merely being useful for following, it was altogether enjoyable to read. The paper is also very thorough and complete in its presentation. There are details on every matter in the appendix.

What's missing roughly comes down to (a) grounding/motivation and (b) effect of the end result. Naturally (a) could strengthen (b), but to comment on these individually:

** Motivation

The neural net architecture studied in this paper is also first introduced in this paper. Why this architecture? It bears some relation to another architecture, namely RBFNs, but the authors stress novely of the definition. (For instance, they highlight that "this specific type of RBFN has not previously appeared in the literature.") In turn, my understanding is that the results do not directly bear on RBFNs of prior interest.

There is separately a clear opportunity to motivate radial neural nets by experiment, as highlighted by two reviewers (g15c and nux4). However the authors contend that "comprehensive empirical studies are ... beyond the scope of this work." Short of a comprehensive study, the paper does not investigate this emprically at all (the only experiments learn the scalar function exp(-x^2)), so this key question remains without even partial evidence.

** Impact of result

Say we assume the architecture is motivated, and consider the theoretical contributions. The end results are (a) a universality theorem and (b) a proof of equivalence between GD in a compressed model and PGD in an uncompressed one.

Regarding (a), I appreciate that proving universality of a strict subclass of networks is (as the author response puts it) "generally harder than proving it for the entire class". However difficulty is not an indication of impact in this case, and many types of networks are universal in some way. The bounded-width aspect makes (a) more technically substantial, and I understand that Reviewer RXP7 finds the result (and techniques) valuable in the context of approximation theory. However, I would not recommend this for NeurIPS on this grounds alone.

Regarding (b), compressibility is an appealing property, but again the impact reduces back to the question of whether the (full) model is useful (theoretically or practically). This is again where even the simplest experiment could go a long way. Better yet, a positive observation here seems like it could have really meaningful consequences!

The reviewers did not come to a consensus on ratings, but there was also no clear argument among them for acceptance. Many of the points from reviews and discussion -- both approvals and concerns -- are reflected above. Others concerns initially raised in reviews were completely and thoroughly addressed by the authors in their response. (For example, reviewer g15c initially questioned the relation to RBFNs, and the authors' reply was comprehensive.)

As a side note, the paper's font deviates from the conference format. This did not factor in my evaluation at all, but as a general tip I would avoid these sorts of modifications.
confidence: Certain
Decision: Reject