Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper is about representing functions $\psi : (\mathbb{R}^d)^n \rightarrow \mathbb{R}$ that are symmetric or asymmetric with respect to the permutation group $S_n$.  The aim is to consider neural networks giving only functions that symmetric or asymmetric, and to establish universality results.  The motivation comes from applications such quantum physics or computer vision with permutation symmetries. 

Honestly, I am very unconvinced by this paper.  In particular, the representation for symmetric functions is essentially due to Newton ($d=1$) and to Weyl ($d > 1$).  On the other hand, the representation for asymmetric functions -- dressed up in elaborate notation and the language of Slater determinants -- seems to just be the statement that an asymmetric function is divisible by the Vandermonde determinant (when $d=1$).  Maybe there are interesting results in this paper, but the writing is so informal and apparently rushed, I simply could not tell.  Itemized comments follow.


Page 1, Footnote 4, terminology: I would suggest just sticking to one of “covariant” or “equivariant” throughout the paper

Page 1, Definition 1, notation: Although I understand the authors are using Matlab notation when they write “$\{1 : n \}$ is short for $\{1, \ldots, n\}$”, it would be more standard to abbreviate $[n] := \{1, \ldots, n\}$.

Page 2, usage of $d$, notation: Please define the meaning of $d$ before referring to this notation.

Page 3: “Functions on sets of fixed size n are equivalent to symmetric functions in n variables.”  Isn’t it symmetric functions in n variables restricted to the domain where no two variables take the same value?

Page 4: “Instead of averaging, the minimum or maximum or median or many other compositions would
also work, but the average has the advantage that smooth  lead to smooth  and  , and more
general, preserves many desirable properties such as (Lipschitz/absolute/...) continuity, (k-times)
differentiability, analyticity, etc.”  Could the authors expand on what they mean by minimum, maximum, median or other compositions to (anti)symmetrize functions?  Never heard of that.  Also, it should be remarked that the averaging operator is an orthogonal linear projection onto the subspace of (anti)symmetric functions known as the Reynolds operator in classical group invariant theory; in particular, averaging has the property that it is the identity applied to the functions that are already (anti)symmetric.

Page 4, Footnote 6: “which induces uniform convergence on compacta”. I had to google “compacta” to find out it is the plural of compact subset or compact metric space. Suffice to say this word is not widely used.

Page 4:  Do you mean $\mathcal{G}_{func} := \{g: \mathbb{R}^m \rightarrow \mathbb{R}\}$?  Also could you say more about how the basis templates $\eta_b$ are coming in?

Page 5: It is stated that computing with elementary symmetric polynomials “is numerically more stable” than with power sums.  Why?  Also why do “we need at least $m \geq n$ functional bases for a continuous representation”?

Page 5: “ Every continuous AS function can be approximated/represented by a finite/infinite linear combination of such [Slater] determinants:” Why?

Page 6, Theorem 3, theorem statement: Thus you have reduced the representation of an antisymmetric function in $n$ variables to that of $n$ symmetric functions in $n-1$ variables, at the expense of an $n \times n$ determinant.  It is not clear how computationally useful this could be, since symmetric functions in $n-1$ variables are nontrivial computationally to represent already, e.g., Theorem 2’s approach would involve $n-1$ power sums.  

Page 6, Theorem 3, proof: The actual representation of $\psi({\bf{x}})$ that is constructed in the proof is almost certainly not useful, namely that it is the determinant of the matrix whose rows look like the following:

$$
\begin{pmatrix}
\psi({\bf{x}}) / \Delta({\bf{x}})  & x_1 & x_1^2 & \ldots & x_1^{n-1} 
\end{pmatrix}
$$
$$
\begin{pmatrix}
\psi({\bf{x}}) / \Delta({\bf{x}})  & x_2 & x_2^2 & \ldots & x_2^{n-1} 
\end{pmatrix}
$$

etc. (Sorry, can't get multi-rowed matrices to render properly in OpenReview...)

Crucial question: why is this a good way of expressing antisymmetric functions, in theory or practice?  The content here is that any asymmetric function is divisible by the Vandermonde determinant.  If the larger $d$ representation results depend on this, I don't know how much they tell me...

Page 7: “For $d > 1$, the Fermion nodes $\{{\bf X} :  \psi({\bf X}) = 0\}$ form essentially arbitrary $\psi$-dependent unions of (non-linear) manifolds partitioning $\mathbb{R}^{dn}$ into an arbitrary even number of cells of essentially arbitrary topology [Mit07].”  This language is too intuitive for me.  What does this sentence mean??

Page 7, Theorem 5, proof discussion:  “Whether this generalizes to $n > 2$  and $d > 1$  is an open problem.”  This is confusing.  Does the theorem have a restriction on $n$ and $d$?

Page 7–8, “Equivariant Neural Network” subsection: work of Risi Kondor and coauthors needs to be cited here.

Page 8, Theorem 6: It would help the reader to include in the body of the paper a more precise statement or diagram showing the construction of the approximating EMLP.
review_title: About representing symmetric and asymmetric functions via neural networks.  Not clear the representations are helpful or informative.  Writing too informal and rushed for me to tell

final_score: rating: 6: Marginally above acceptance threshold
review_content: Update:
I have read the authors' responses and other comments.  I still think that the theoretical results on anti-symmetric functions of this submission are novel, which is, however, not well-delivered. A lot of space is wasted to discuss symmetric functions, for which the contributions of this paper are not clear.  I suggest substantially rewriting this paper by only focusing on the anti-symmetric functions. 

===============================================================
Summary:

This paper studies the representation of symmetric and anti-symmetric functions as well as the parameterization with neural networks.  This issue is very important for applying deep learning to solve problems from computational quantum physics as well as point clouds.  

Pros:

This paper is well-organized, in particular the connection to the previous work.  The representations of symmetric and anti-symmetric functions are very important but less-studied problems. As I understand it, this paper makes some interesting contributions to this topic.  Theorem 3 & 5 are very especially interesting, which basically shows that one Slater determinant is enough to approximate any anti-symmetric functions. 

Cons:

While the theorems in this paper look pretty, it is not clear if they have computational advantages over the other ansatzes.  After all, no approximation rates and numerical results are provided. For example, computing Vandermonde determinants is much efficient than computing Slater determinants.  But as suggested in this paper, the approximation with Slater determinants is more efficient than the one with Vandermonde determinants. Combined them together, it is not clear which one will be more practical. Could you comment more on this point?


Other comments:
The authors do not put the proofs in the appendix.  Instead, they provide an extended version of the paper in the supplemental material which includes the proofs.  This makes the reading of the proof more challenging since readers have to dig out the proofs from the long paper. So basically, there are two versions of this paper. 

review_title: A good contribution to the understanding of representing symmetric and anti-symmetric functions

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this paper the authors study the representability of symmetric or antisymmetric functions using neural networks. In particular, we say f: R^n -> R is symmetric/ antisymmetric if f(x)= f(pi(x)) for all pi in S_n or is of the form f(x)= sign(pi) *f(pi(x)) where sign(pi) is the sign of the permutation. Such functions with such symmetries are ubiquitous in quantum physics since if one looks at the wave function of a  system of identical bosons, then they are symmetric and if you look at fermions they are antisymmetric. In this paper they authors try to understand if there exists succinct architectures that can learn/represent functions with such a symmetry property.  In this direction the author makes a few observations For every "nice" antisymmetric function f, there exists a symmetric function Phi and a matrix associated to Phi such that determinant of Phi can be used to represent f.  Then the authors show that the standard Ferminet (the first demonstration of deep learning) can be used to represent determinants of slater determinant and hence an arbitrary continous antisymmetric function.

Overall, this paper is interesting to read but I wouldn't recommend it for acceptance for the following reasons:

1) There are many points which seem like "overselling". The author claims that a polynomial f: R^n -> R has n! many terms and that is too many to approximate and so on. Sure this is true, but the symmetries make a huge difference here. Additionally, there is a rich theory of approximation theory saying we can approximate symmetric functions by low-degree polynomials, so why not just allow the NN to compute this low-degree polynomial which in turn approximates f? 

2) It seems to me that this paper is more of a review, than something novel. Even the author at many places claims that this paper is a thorough review, so its not clear if it clears the bar for ICLR.

3) The main results of this paper are straightforward in some sense. In fact the author doesn't even talk of the main results for the first 5 pages and the main results are simply swept under the rag, and it's not clear to me what is the novelty of the results in that case. The quantum motivation also seems very weak, i prefer a much more solid motivation for why such functions are useful (given the author mentions practicality).

Given these reservations I wouldn't accept it for recommendation.
review_title: review for antisymmetric

final_score: rating: 4: Ok but not good enough - rejection
review_content: Post discussion:
I read the author's response and other reviews. I will stick to my rating and encourage the author to resubmit a revised version focusing on the antisymmetric case. 





Summary:

The paper studies the approximation power and shows universality results for two recent neural network models that represent symmetric and antisymmetric functions. The main contributions of the paper are claimed to be (1) Universality of Fermi-Net (antisymmetric model) with a single Generalized Slater determinant (2) Universality of symmetric MLPs. In both cases, the authors emphasize the fact that the theorems deal with (i) vector inputs rather than scalar inputs, and that (ii) the approximation results are based on smooth polynomials rather than discontinuous functions as done in previous works. 

While the problem the paper targets is interesting and important, the paper missed several important previous works and does not do a good job explaining their novelty with respect to the previous work that was cited. See below.

Contribution 1: I am not an expert on antisymmetric function approximation and didn’t know Ferminet, but contribution 1 seems novel to me, although the authors should make a better job explaining the difference between their results and the original FermiNet paper. It is not entirely clear what was done before and what is new

Contribution 2 and points (i),(ii) were discussed before in several papers. First “Provably Powerful Graph Networks” (NeurIPS 2019) used the power sum multi symmetric polynomials for representing set functions in the same way they are used in this paper.  Second, “On universal equivariant set networks” (ICLR 2020) proves a universal approximation theorem for equivariant set functions based on these polynomials. Given these two works, I am not sure the current paper has any additional contribution.

Strong points:

Understanding the approximation power of invariant/equivariant neural networks is an important goal.
The results on antisymmetric functions are nice

Weak points:

The work is not properly positioned with respect to previous work and pretty much misses all the work done on the approximation of invariant functions since DeepSets. Except for the discussion above, the following should be cited/discussed:

-- “Janossy Pooling: Learning Deep Permutation-Invariant Functions for Variable-Size Inputs” ICLR 2019, which discusses permutation sampling strategies. 

-- “Universal approximations of invariant maps by neural networks” 2018 that discusses symmetrization and approximation of symmetric functions (and function invariant to many other compact groups). 

-- “PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation” CVPR 2017, which also show universal approximation for symmetric functions.

-- “On the Universality of Invariant Networks”, "Universal Equivariant Multilayer Perceptrons" ICML 2019/2020 might also be relevant.

Recommendation:

The paper studies an important problem but missed significant previous work. I believe that the results on antisymmetric function approximation are novel and suggest rewriting the paper focused on these results, with a clear discussion on the contribution with respect to previous works. It might also be good to spend more time on Ferminet while doing so since this model is less known to the machine learning community. In its current form, The paper is not ready for publication.

Minor comments:

The authors added a long version (22 pages) as an appendix. Not sure if this is OK. 


review_title: Interesting topic but missed significant previous works

Meta-review: The paper received reviews from experts in representation of invariant functions. They all have expressed concerns regarding the novelty of the technical contributions, and the lack of appropriate comparisons to existing results. This applies in particular to representation of symmetric functions using neural networks which was largely covered by previous works, as acknowledged by the authors. The authors are encouraged to consider the valuable inputs by the reviewers and revise accordingly. 
Decision: Reject