Title: value: Neural Injective Functions for Multisets, Measures and Graphs via a Finite Witness Theorem
Authors: value: Tal Amir
Steven J. Gortler
Ilai Avni
Ravina Ravina
Nadav Dym
Venue: NeurIPS 2023
Keywords: value: Equivariant Neural Networks
Universal approximation
Geometric deep learning
multiset learning
injective multiset functions
learning on measures. WL test

Abstract: value: Injective multiset functions have a key role in the theoretical study of machine learning on multisets and graphs. Yet, there remains a gap between the provably injective multiset functions considered in theory, which typically rely on polynomial moments, and the multiset functions used in practice, which rely on $\textit{neural moments}$ — whose injectivity on multisets has not been studied to date.

In this paper, we bridge this gap by showing that moments of neural networks do define injective multiset functions,  provided that an analytic non-polynomial activation is used. The number of moments required by our theory is optimal essentially up to a multiplicative factor of two. To prove this result, we state and prove a $\textit{finite witness theorem}$, which is of independent interest. 

As a corollary to our main theorem, we derive new approximation results for functions on multisets and measures, and new separation results for graph neural networks. We also provide two negative results: (1) moments of piecewise-linear neural networks cannot be injective multiset functions; and (2) even when moment-based multiset functions are injective, they can never be bi-Lipschitz.


Review discussion:
The paper provides theoretical explanation why neural networks are successful in providing injective multiset functions that gained popularity since the seminal Deep-Sets paper.
While neural networks are long-known to be universal approximations for $f: R^k -> R^n$. Much less is known about multiset functions represented by neural networks. The main result proven in this paper (Theorem 3.3) shows that depth-1 neural networks can be used to define injective multiset functions, moreover, they achieve currently best-known embedding dimension. The paper also shows that one cannot use piecewise-linear activations to construct injective multiset functions and should use analytic non-polynomial activations instead (for example, sigmoid).
The paper is well-structured and provides a clear comparison to the prior work. The proofs are rigorous and well-written and as far as I can tell are correct.
I think the paper may benefit from slightly more detailed discussion of applications of injective multiset functions.
While injectivity is a natural property that proves to be useful in various setups, I believe it will be nice to include a slightly more detailed discussion of the applications in the introduction. Some example applications are presented in Section 6.
1) The paper shows that MLPs define injective multiset functions if activation is analytic non-polynomial, and fail to define injective multiset functions if activation is piecewise linear. What happens if activation is polynomial? Is it easy to see that such activations fail similarly to piecewise linear activations?
2) I think it worth replacing "up to a multiplicative factor of 2" everywhere in the paper with "up to a multiplicative factor of 2+o(1)" to be mathematically precise. If one is pedantic, 2nd+1 is not within a factor of 2 from nd.
I believe the paper contains several typos:
L47: a Euclidean -> an Euclidean
L129: this paper -> that paper (?)
L456 quality -> qualify.
NA
No ethics review needed.
We thank the reviewer for his/her helpful comments. Below are our responses.
**Response to Weaknesses**
We thank the reviewr for this comment. Should the paper be accepted, we intend to add to the introduction a broader explanation of the importance of injective multiset functions for practical applications.
**Response to Questions**
1. A possible direction to show how polynomial activations fail would be as follows: If $\sigma : \mathbb{R} \to \mathbb{R}$ is a polynomial of degree $r$, then $\text{span} \\{ \sigma(ax+b) \\, \mid \\, a \in \mathbb{R}^d, b \in \mathbb{R} \\}$ is a subset of the span of all polynomials of degree up to $r$ over $\mathbb{R}^d$. Denote the latter by $\mathcal{P}\_{r}$. Using the fact that $\mathcal{P}\_{r}$ is not dense in $\bigcup\_{t=0}\^{\infty} \mathcal{P}\_{t}$, show that for a large enough $n$ there exist $n$ points $x_1,\ldots,x_n \in \mathbb{R}^d$ and a polynomial $q \in \mathcal{P}\_{r'}$ for some $r'>r$ such that: (a) $\sum_{i=1}^n q(x_i) p(x_i) = 0$ for any $p \in \mathcal{P}\_{r}$;
(b) not all of $q(x_1),\ldots,q(x_n)$ are zero. Let $\mu = \sum_{i=1}^n w_i \delta_{x_i}$ be the discrete signed measure with weights $w_i = q(x_i)$.
Then (a) implies that for any embedding $\hat{f}$ comprised of moments of $\sigma(ax+b)$, $\hat{f}(\mu) = 0$, although by (b), $\mu$ is not the zero measure, hence injectivity is violated.
2. We thank the reviewer for this comment. We will replace this statement by a more accurate one.
I appreciate the authors' response to my questions and keep my positive score unchanged.
This is honestly a fantastic paper, I very much enjoy the read, it is clear, pedagogical, and refreshingly honest. The results presented are of great interest for the community.
Although it is an extension of a known results on semi-algebraic sets, as acknowledged by the authors, the extension seems anything but trivial. The large part dedicated to negative results and/or limitations is very much appreciated.
Minor weaknesses; maybe a tad more explanation on the difference between semi-algebraic and sub-analytic proof (I understand both lies on the same underlying proof technique initiated with works on phase retrieval, but with different tools, how would you describe the main difference if possible?), and a bit more outlooks: do you have any more insight what would be a desirable property to show on ReLU networks, since they are definitely not injective, but are used in practice?
Some questions/minor typos:
- l45: nessecary
- equation (***): missing indices $i$ on the $x$'s
- same equations: might be clearer if the $a$ and $b$ were indexed by $i$ and the $x$'s by $j$ (or vice versa)
- thm 3.4: co-existence of $D$ and $D_\theta$
- eq (6) and lines below: in your definition of Wasserstein, do you mean "uniform" weights instead of "unit"? $S_1$ and $S_2$ may have different cardinality, but the associated measure must be normalized to compute (the classical) Wasserstein
- l 288: seleceted
- l 298: Lipshcitz
- the experiment on bi-Lipschitz is not really illustrative of the theory, since functions are not bi-Lipschitz. It is not uninteresting in describing another phenomenon (the "almost" bi-Lipschitzness on finite data, etc), but I wouldn't say that it "corroborates Thm 5.1"
Authors have focused extensively on negative results and limitations.
No ethics review needed.
We thank the reviewer for the supportive and helpful comments. Below are our responses to the questions and weaknesses.
**Response to Weaknesses**
1. We thank the reviewer for the suggestion. We intend to add to the text an explanation of the main challenges and ideas in this generalization.
In a nutshell, The proof of the finite witness theorem in [Dym and Gortler] relies on several nice properties of semialgebraic sets: This family is closed to linear projections, finite unions, finite intersections, and complements.
Moreover, they are always a finite union of smooth manifolds. Using these properties, [Dym and Gortler] prove a finite witness theorem for semialgebraic sets and corresponding functions, called semialgebraic functions. These functions include polynomials, but do not include other analytic functions of interest.
The generalization of the finite witness theorem to the analytic setting relies on the mathematical study of o-minimal systems, which searches for larger families of sets that enjoy the same properties as semialgebraic sets. There are several known o-minimal systems, such as globally subanalytic sets, which do have the same nice properties. For these sets, the generalization of the finite witness theorem is rather straightforward.
However the corresponding collection of globally subanalytic functions still does not include all analytic functions. A crucial observation in the proof is that we can make it carry through also when considering countable unions of globally subanalytic sets.
Consequently, the corresponding functions we can work with include all analytic functions.
2.
Although any finite-size ReLU-activated network is not moment injective, it can be shown that moments of shallow ReLU networks are universal approximators of continuous injective functions on multisets; this is due to the ReLU function being discriminatory, as commented below the statement of Corollary 6.1.
Thus, taking a high enough embedding dimension, it is possible to construct invariant embeddings using ReLU activations that are practically injective. This comes with a caveat, though, as we noted in our response to Reviewer ezv6: Each such embedding has a nonzero-measure subset of the input domain on which it is provably not injective. Should the paper be accepted, we intend to comment on this in the camera-ready version.
**Response to Questions**
We thank the reviewer for the corrections.
1. In Eq. (6) and below, the term $\text{\emph{uniform}}$ is indeed more suitable, as the proof works well with a total mass of 1.
2. We rephrased the text describing the bi-Lipschitzness experiment (Line 296) and removed the statement that the results corroborate Thm 5.1.
Thank you for the rebuttal. I keep my good score as is.
- The paper contains several interesting mathematical results which could be useful to the broader graph/set NN community.
- The paper is well written and generally provides enough context and guidance for the results to be understandable.
- The theoretical results are focused on architectures that are used in practice.
- The experiments provide additional context and intuition for the applicability of the results.
- The paper is upfront about its scope, limitations, and practical applicability.
- The initial motivation of the paper is the moment injectivity of 'practical' MLPs.
While several results are ultimately established, we can see from the experiment on graph separation (and it is clearly stated in the conclusion) that even non-analytic functions just barely fail in a few cases to match 1-WL. This seems to suggest that the practical relevance of the results is a bit questionable, or maybe there are other factors that could be mitigating the non-analyticity of the activations.
- A few extra explanatory lines could be included in the section of the finite witness theorem (or perhaps in the introduction), that provide context about its exact role in the proof. Currently, the theorem shows up a bit abruptly when theorem 3.3 is presented. OK, it can be inferred from the proof of 3.3 that infinitely many equalities are reduced down to a finite number, but I believe the readability of that section could improve if some of the context that is provided in the supplementary material was moved in the main text to make sure that the role of the theorem is very clearly explained and emphasized.
Specifically, when explaining proof 3.3, around line 154 or perhaps right below the statement of 3.4, in subsection 3.1. Those could be good places to provide a few more sentences that restate the purpose of the theorem and explain the context around it (papers by Balan et al., and Dym and Gortler).
I am skeptical about the significance of the results when it comes to practical considerations but the paper is well written and provides mathematical insights that can be relevant to modern deep learning architectures so I lean towards accepting.
We see in the second experiment a somewhat significant difference between leaky ReLU and ReLU. Could that be a matter of randomness from different initialization, or could the properties of the function that could lead to worse outcomes when it comes to matching the 1-WL? I don't see any obvious reason why the leaky activation would be much different.
N/A
No ethics review needed.
We thank the reviewer for his/her constructive review. Below we address the weaknesses and questions raised by the reviewer.
**Response to Weaknesses**
1.
While indeed in our experiments ReLU-based embeddings performed similarly to analytical activations when the embedding dimension was high enough, this can be contrasted by the following fact: For any moment function $\hat{f} : \mathcal{S}_{\leq n} (\mathbb{R}^d)$ based on ReLU-activated networks, there exists some $\delta > 0$ and neighbourhood of radius $\delta$ on which $\hat{f}$ is not injective.
In contrast, for any $\delta>0$, any injective analytical embedding $\hat{f}$ is guaranteed to be bi-Lipschitz on all pairs of point-sets whose distance is at least $\delta$.
Namely, assuming that the domain is compact, for any $\delta>0$ there exist constants $c(\delta),C(\delta) > 0$ such that if $W_2(X_1, X_2) \leq \delta$, then $c(\delta) W_2( X_1, X_2 ) \leq \lVert \hat{f}(X_1) - \hat{f}(X_2) \rVert \leq C(\delta) W_2( X_1, X_2 )$.
The reason why we did not enounter pathological $\delta$-neighborhoods for ReLU in our experiments with high $m$ is because we drew the input clouds randomly rather than explicitly looking for adversarial examples. We intend to clarify on this in the camera-ready version if our paper is accepted.
2. We revised Section 3 to clarify the role of the finite witneess theorem in the proof of moment injectivity.
In the revised version, the theorem is gently introduced to the reader before proving Theorem 3.3, and only then it is applied to prove the result.
Our new proof of Theorem 3.3 is, in our opinion clearer, and places more emphasis on the essential role of the finite witness theorem.
**Response to Questions**
We conjecture that the differences in favor of Leaky ReLU over ReLU (Figure 1) result from the latter having a region where it is identically zero.
An interesting perspective on this is to compare a ReLU, a leaky ReLU and a nonpolynomial analytic activation, by regarding their restrictions to the region x < 0 as polynomials of degree 0, 1 and $\infty$ respectively. In light of Theorem 3.3, it seems plausible that as the degree of the polynomial increases, the likelihood of getting the same output $\hat{f}(X_1)=\hat{f}(X_2)$ for a distinct pair of inputs $X_1,X_2$ should decrease.
Thanks for the rebuttal and the interesting remarks! As far as I am concerned the paper is solid and should be accepted so I maintain my score.

Meta-review:
role: PC
data: comment: value: This paper provides analytic conditions for an MLP to be moment injective. The reviewers unanimously agreed that this is a great paper and worthy of being accepted to NeurIPS, and I concur.

Decision: Accept (spotlight)