Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: In this paper, the author analysis the (approximate) function class generated by an infinite-width network when the Euclidean norm is bounded. They extend the work of Savarese et al. on the univariable function by introducing the Randon Transform and R-norm to this problem.  The authors finally prove that any function in Sobolev space could be (approximately) obtained by a bounded network. The results achieved implies some generalization performance analysis and the induction error. Also, according to the authors, the difference between R-norm and RKHS norm might lead to the distinct from neural networks and kernel methods.

I would recommend accepting this paper since it might give a good insight into understanding the performance of the network beyond the traditional method.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper gives characterization of the norm required to approximate a given multivariate function by an infinite-width two-layer neural network. An important result is the relation between Radon-transform and the $\mathcal{R}$-norm. This paper also shows application of the norm on some special case.

I suggest this paper being accepted because it provides new insights into the approximation theory for neural networks. The perspective of norm constraint is different from the traditional approximation theory and may serve as a good contribution to the community.

One question is that: in section 4, the equation (19) is differentiated twice to get the equation (20) containing Dirac delta. Although this is intuitively correct, this seems not a strict derivation to my mathematical background. It would be great if the authors can show the strict definition and derivation presented here.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: The paper studies the function space regularization behavior of learning with an infinite-width ReLU network with a bound on the l2 norm of weights, in arbitrary dimension, extending the univariate study of Savarese et al. (2019).

The authors show that the corresponding regularization function is more or less an L1 norm of the (weak) (d+1)st derivatives of the function, and provide a rigorous formal characterization in terms of the "R-norm", which is expressed via duality through the Radon transform and powers of the Laplacian.

In addition, the paper provides a number of implications of this study, such as approximation results through Sobolev spaces, an analysis of the norm of radial bump functions, and a new type of depth separation result in terms of norm as opposed to width.

Overall, this is a strong paper making several interesting and important contributions for our understanding of the inductive bias of ReLU networks. I thus recommend acceptance.

A few comments:
* is it possible to obtain precise characterizations of interpolating solutions in this setting (other than a mere representer theorem with ReLUs), as done in Savarese et al (2019, Theorem 3.3) for the univariate case?

* perhaps the results of Section 5.1 should be contrasted with those of Bach (2017, e.g. Prop. 5), which only require ~ d/2 derivatives instead of ~ d here, albeit with stronger requirements, for essentially the same functional space (though the approximation result is obtained from an associated RKHS, which is smaller).

* are the results on radial bump functions intended to provide insight on approximation or depth separation? what was the motivation behind this section?

Other minor comments/typos:
- after Prop. 1: "intertwining" appears twice
- eq. (22): missing f in l.h.s.
- eq. (23): is the first minus sign needed?
- before Thm. 1: point to which Appendix
- Section 4.1, "In particular, this is what would happen ... d+1": this should be further explained
- Section 4.1, final paragraph, "in order R-norm to be": rephrase
- Section 5.4, "required norm with three layers is finite": which norm? maybe point to a reference? Also, Example 5 could be explained in further detail
- Section 5.5: what is an RKHS semi-norm? you'd always have ||f|| = 0 => f = 0 in an RKHS, by the reproducing property
review_title: Official Blind Review #1

Meta-review: The article studies the set of functions expressed by a  network with bounded parameters in the limit of large width, relating the required norm to the norm of a transform of the target function, and extending previous work that addressed the univariate case. The article contains a number of observations and consequences. The reviewers were quite positive about this article. 
Decision: Accept (Poster)