Title: Universal approximation and model compression for radial neural networks 
Authors: Iordan Ganev
Twan van Laarhoven
Robin Walters
Venue: ICLR 2023
Keywords: universal approximation
model compression
radial functions
parameter space symmetries
projected gradient descent

Abstract: We introduce a class of fully-connected neural networks whose activation functions, rather than being pointwise, rescale feature vectors by a function depending only on their norm. We call such networks radial neural networks, extending previous work on rotation equivariant networks that considers rescaling activations in less generality. We prove universal approximation theorems for radial neural networks, including in the more difficult cases of bounded widths and unbounded domains. Our proof techniques are novel, distinct from those in the pointwise case. Additionally, radial neural networks exhibit a rich group of orthogonal change-of-basis symmetries on the vector space of trainable parameters. Factoring out these symmetries leads to a practical lossless model compression algorithm. Optimization of the compressed model by gradient descent is equivalent to projected gradient descent for the full model.


Review discussion:
Strength:
1, the proposed radial neural network is novel. The way it generalizes pointwise activation is natural and elegant, that a simple rescaling of vector is actually expressive.
2, the universal approximation results are solid. The proof method is intuitive and resembles classic ideas in literature.
Weaknesses:
1, I don't see any improvement over previous bounds for universal approximation: for the usual Lipschitz function case, the construction of Theorem 3 requires $O((\frac{\sqrt{n}}{\epsilon})^{3n})$ number of parameters (ignoring poly factors). Theorem 5 improves it to the standard $O((\frac{\sqrt{n}}{\epsilon})^{n})$.
2, the compression algorithm is useful only if there are subsequent layers with notably larger width.
However, such neural networks are already sub-optimal theoretically and wouldn't be used in the first place, then there is no need for compression.
Sections 1-4 are well-written and easy-to-follow. Sections 5-6 are felt not tightly linked to the main results, and I think the mathematical writing can be made more succinct in the main-text. A small polishing on the writing of sections 5-6 could be helpful.
NO.
We thank the reviewer for the feedback and comments. We have smoothed out sections 5 and 6 slightly, and would like to respond briefly to two points:
### Bounds
> I don't see any improvement over previous bounds for universal approximation: for the usual Lipschitz function case, the construction of Theorem 3 requires $O((\frac{\sqrt{n}}{\epsilon})^{3n})$
number of parameters (ignoring poly factors). Theorem 5 improves it to the standard $O((\frac{\sqrt{n}}{\epsilon})^{n})$
It is true that the previous bounds for universal approximation are indeed the same as ours.
However, they are for pointwise activations and we generalize these results to a radial activation, step-ReLU.
### Suboptimality
> the compression algorithm is useful only if there are subsequent layers with notably larger width.
However, such neural networks are already sub-optimal theoretically and wouldn't be used in the first place, then there is no need for compression.
We provide a theoretical explanation for the fact that networks with drastically increasing widths are suboptimal in certain domains.   Our explanation involves parameter space symmetries, and also the link between model compression and gradient descent.
At the same time, the suboptimality isn't directly guaranteed by our results since the larger network may train differently.
Indeed, in some domains, networks with increasing hidden dimensions have proven useful for optimization.
Dear Reviewer,
If you do not want to update your score, please at least acknowledge that you have already read the rebuttal.
Strengths:
* Nonlinearities are a fundamental building block of neural network architectures, so new techniques are potentially broadly applicable.
* The writing of the paper is clear and focused within the respective sections.
* The range of contributions is quite broad, ranging from the theoretical results on the proposed networks being universal function approximators to a practical compression method taking advantage of their structure.
Weaknesses:
* Probably as a result of its breadth, the paper somewhat lacks a clear red thread or headline result. I am not really sure who the target audience of the paper is. Is the rotation equivariance a means to the end of constructing neural networks that are easy to compress, i.e. is this a practically minded paper? Do the authors want to establish radial neural networks as a new direction of research?
Then what are promising directions for future work and what specific limitations of current architectures are they meant to overcome?
* There are essentially no quantitative results. I really don't mind if the methods can't be immediately plugged into ResNets or Transformers and give state-of-the-art results, but the paper should at least give a general idea of how well the method works on standard tasks (MNIST, CIFAR, ...). What's the rationale behind choosing the benchmark on the noised MNIST images? Is it expected that radial neural networks perform better on this task?
**Clarity** The technical writing is clear, however the key contribution of the work could be worked out better.
**Quality** The paper is sound as far as I can tell, although I did not go through the derivations or proofs.
**Novelty** The work is new as far as I am aware, I am however unfamiliar with most closely related pieces of work.
NO.
The reviewer's feedback is much appreciated. We address their questions one by one:
### Headline results
> Probably as a result of its breadth, the paper somewhat lacks a clear red thread or headline result.
The headline is: "Radial neural networks are universal approximators.
Moreover, their parameter spaces exhibit rich symmetries that lead to model compression." It is a fair point that the breadth and technical detail obscures this headline somewhat; we have added this headline (and a few others) to the list of contributions at the end of the introduction.
### Rotational equivariance
> Is the rotation equivariance a means to the end of constructing neural networks that are easy to compress, i.e. is this a practically minded paper?
Yes, rotational equivariance is a means to construct compressible networks. Not only that, but it allows one to relate the compression with gradient descent.
So in this sense, this is a practically minded paper aimed towards practitioners as well as theoreticians.
### Radial NNs as a research area
> Do the authors want to establish radial neural networks as a new direction of research?
Then what are promising directions for future work and what specific limitations of current architectures are they meant to overcome?
Yes, we would be delighted to have radial neural networks established as a new direction of research. They are meant to overcome the fact that current architectures are mostly limited to pointwise activations. By pushing the boundaries of the search space for non-pointwise activations, we hope to expand the flexibility of neural networks. Parameter space symmetries feature prominently in the theoretical analysis of radial neural networks; our framework for studying such  symmetries may be relevant to other kinds of activations. These are all reasons that this research direction is promising.
### Noisy MNIST
>   What's the rationale behind choosing the benchmark on the noised MNIST images?
Intuitively, the main practical advantage of radial neural networks (especially those with step-ReLU activations) is that they are better suited for certain tasks with spherical decision boundaries. Our proof-of-concept experiments  supports  this intuition. In more detail, spherical decision boundaries can be expressed in terms of affine transformations of the unit sphere in the input space, and the Step-ReLU activation is designed to cut out the unit sphere from the rest of the space.
> Is it expected that radial neural networks perform better on this task?
Yes, we show that step-ReLU networks outperform pointwise ReLU networks for noisy image recovery.
Another possible area where step-ReLU networks are expected to perform well is in compressed sensing.
Dear Reviewer,
If you do not want to update your score, please at least acknowledge that you have already read the rebuttal.
=============== Strength ===============
The paper manages to contain a rich amount of related references and provide graphical illustration that makes the ideas easy to understand.
From a technical point of view, the asymptotic universal approximation property of radial neural networks is new to me.
Besides, the compressive nature of radial neural networks and its resulting fast optimization are also not studied in literature.
The organization of the paper is good and easy to follow.
=============== Weakness ===============
A strong motivation of the practicality of radial neural networks seems to be lacking. It will be helpful to provide working examples of radial neural networks and indicate that in some important applications, radial neural networks outperform pointwise networks.
The universal approximation theory allows the network size to grow to infinity. It seems unfair to claim that not requiring bounded domain is a contribution. In early asymptotic results, e.g., Cybenko and Barron, neural networks are already shown to be dense in differential function spaces with unbounded domain.
Later results established the rate of approximation requires a compact domain. As there is no rate of approximation provided, considering unbounded domain does not add much technical challenges.
Moreover, the asymptotic results does not provide understanding of potential advantages of radial neural networks over pointwise neural networks in terms of function approximation.
Synthetic data is relatively toy and the comparison with ReLU network is not fully convincing.
On the one hand, synthetic data are low-dimensional and the target function is simple in that the activation function is almost the form of the target function.
On the other hand, Step-ReLU radial network outperforms ReLU MLP might correlate to multiple reasons: 1) different activation functions; 2) different network architectures; 3) hyper parameter tuning. By the way, is it possible to simply perform classification on MNIST using radial and pointwise neural networks?
As mentioned, the paper is relatively clear. Some notations might need further elaboration.
For example, following Theorem 3, $T_i$ and $S_i$ are affine maps of corresponding input and output dimensions. Is it unclear how $T_i$ can lift the input space to a higher dimension using vector addition.
It is better to highlight technical contributions in Section 4 and compare to the universal approximation of pointwise networks to demonstrate any potential advantage if possible.
NO.
We are grateful for the reviewer's thoughtful assessment.
### Motivation of the practicality
> A strong motivation of the practicality of radial neural networks seems to be lacking. It will be helpful to provide working examples of radial neural networks and indicate that in some important applications, radial neural networks outperform pointwise networks.
The main practical advantage of radial neural networks (especially those with step-ReLU activations) is that they are better suited for tasks with spherical decision boundaries. We demonstrate this in a proof-of-concept experiment on noisy image recovery described in Section 7 and Appendix E. The code in the supplemental material provides a working implementation of radial neural networks that allows for the reproduction of these experiments.
The intuition behind this advantage is that spherical decision boundaries can be expressed in terms of affine transformations of the unit sphere in the input space, and the Step-ReLU activation is designed to cut out the unit sphere from the rest of the space.
Additionally, our theory itself gives practical advantages.
Specifically, the model compression result shows that, once a radial neural network is trained, there is automatic compressibility that can be applied.
### Universal approximation
> It seems unfair to claim that not requiring bounded domain is a contribution. In early asymptotic results, e.g., Cybenko and Barron, neural networks are already shown to be dense in differential function spaces with unbounded domain.
While there is much work on universal approximation for pointwise activations, we are not aware of previous work on universal approximation for radial neural networks; to the best of our knowledge, our work is the first contribution in this direction. The fact that we have results for unbounded domains and bounded widths is an extra bonus.
Cybenko and Barron used sigmoid networks, and these are not dense in the space of differentiable functions with unbounded domains. As a counterexample: one cannot approximate sin(x) for all x with a finite sigmoid MLP. One can handle arbitrarily large domains, but not unbounded ones.
Hence, while one can consider functions on unbounded domains, one still needs to choose a compact set on which the approximation will hold; little or nothing can be said outside of this compact set.
By contrast, in our setting, one still needs to choose a compact set to construct the network, but the ultimate approximation holds on all of $\mathbb{R}^n$.
The works we cite on non-compact domains apply for $L_p$ functions. Such functions are asymptotically affine since they must limit to zero in order for the $L_p$-norm to converge;
but the class of asymptotically affine functions includes functions that are not in $L_p$.
More generally, our proof constitutes a significant innovation in approximation theory, since we provide a concrete procedure of how to decompose a nonlinear function in the 'depth' direction, rather than in the 'breadth' direction, which is the standard. The rate of approximation is a valuable suggestion for future work; thanks for pointing it out!
### Experiments
> Step-ReLU radial network outperforms ReLU MLP might correlate to multiple reasons: 1) different activation functions; 2) different network architectures; 3) hyper parameter tuning.
Of these three explanations, our experimental methods point strongly to the first, supporting our hypothesis and the main themes of the paper. The second one is controlled for by the fact that all networks under comparison have the same number of layers and widths;
hence the same number of parameters.
As for the third, we performed hyperparameter tuning before running the experiments.
In fact, we only tuned the learning rate for ReLU network and used the same rate for the step-ReLU, since the latter outperformed the former at any given, common learning rate. We plan to add details about this tuning to Appendix E.
We did run a Step-ReLU network on 1000 noisy MNIST images for image recovery, and it outperformed a pointwise ReLU networks. We include a summary of this experiment in the revision.
### Question about $T_i$
> Is it unclear how can lift the input space to a higher dimension using vector addition.
For simplicity, we consider the case $i=1$, $c_1 = 0$, and $n=2$. The transformation $T = T_1$ is an affine map from the plane $\mathbb{R}^{2 }$ to  $\mathbb{R}^{3}$ taking $z = (u,v)$ to $(u,v, h_1)$.
Hence, it is evident that $T$ lifts the input plane to a plane within a three-dimensional space. (If $c_1 \neq 0$, then one first subtracts $c_1$ from the input space, and then lifts.) Composing $T$ with step-ReLU and then $S = S_1$ takes everything in the $r_1$-ball around the origin to the vector $(0,0,1)$, and takes everything outside that ball to itself.
Hence, this composition lifts the $r_1$-ball to a higher dimension, and leaves everything else unchanged.
Dear Reviewer,
If you do not want to update your score, please at least acknowledge that you have already read the rebuttal.
We are very thankful to each of the reviewers for their careful reading of the paper, and for their detailed comments. Reviewer vdPZ  points out the originality of our work.
Additionally, they note that  the quality of the writing makes the paper easy to follow. Reviewer Vbpq goes on to remark that the contributions are fundamental, the scope is broad, and the writing is focused. Reviewer MQ2f emphasizes the elegance and mathematical rigor of the work.
We have uploaded a revision of the paper. Here is a list of the changes:
* An emphasis on the headline results in the introduction.
* A comparison with $L_p$ functions in the related works section.
* A slight smoothing of sections 4 and 5. This smoothing consists mostly of deletions and small reorganizations, as well as a few additions.
* A description of an additional noisy image recovery experiment involving a larger part of MNIST.
Additions to the text are marked in blue.
Once again, we appreciate the time and effort each of the reviewers put into reading our paper. We look forward to any further feedback they would like to share!

Meta-review:
role: PC
data: metareview:_summary,_strengths_and_weaknesses: In this paper, a new type of neural network architecture is introduced that is structured in a radial manner, meaning that instead of activating individual points within a layer, the entire layer is activated and the output vector is rescaled based on its norm. The authors provide evidence that this architecture is capable of universal approximation and also describe model compression techniques that can be applied to it.

The reviewers raised two concerns, which were not addressed by the rebuttal:

(1) The universal approximation theory is asymptotic and cannot justify any advantage over existing results for unbounded domains, .e.g, Cybenko and Barron. The contribution is less significant than claimed.

(2) The comparison with ReLU networks are not very convincing. It would be helpful to compare radial and pointwise neural networks on a more complex task such as classification on the MNIST dataset.
justification_for_why_not_higher_score: NA
justification_for_why_not_lower_score: NA
summary_of_AC-reviewer_meeting: NA

Decision: Reject