SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left\{\left(\left(\boldsymbol {w}, \boldsymbol {X}\right), \left(\boldsymbol {w} ^ {\prime}, \boldsymbol {X} ^ {\prime}\right)\right) \in \mathbb {M} \times \mathbb {M} \mid F \left(\boldsymbol {w}, \boldsymbol {X}; \boldsymbol {a}, b\right) = F \left(\boldsymbol {w} ^ {\prime}, \boldsymbol {X} ^ {\prime}; \boldsymbol {a}, b\right), \forall (\boldsymbol {a}, b) \in \mathbb {W} \right\} = \\ \left\{\left(\left(\boldsymbol {w}, \boldsymbol {X}\right), \left(\boldsymbol {w} ^ {\prime}, \boldsymbol {X} ^ {\prime}\right)\right) \in \mathbb {M} \times \mathbb {M} \mid F \left(\boldsymbol {w}, \boldsymbol {X}; \boldsymbol {a} _ {i}, b _ {i}\right) = F \left(\boldsymbol {w} ^ {\prime}, \boldsymbol {X} ^ {\prime}; \boldsymbol {a} _ {i}, b _ {i}\right), \forall i = 1, \dots , m \right\}. \tag {7} \\ \end{array}
$$
Let  $\mathbf{A} \in \mathbb{R}^{m \times d}$  with rows  $\mathbf{a}_1, \ldots, \mathbf{a}_m$ , and  $\mathbf{b} = (b_1, \ldots, b_m)$ . Suppose that  $\mathbf{A}, \mathbf{b}$  indeed satisfy (7).
Let  $\mu, \mu' \in \mathcal{M}_{\leq n}(\Omega)$  be two measures with parameters  $(\boldsymbol{w}, \boldsymbol{X})$ ,  $(\boldsymbol{w}', \boldsymbol{X}')$  respectively. Equation (7) implies that if the function  $\hat{f}$  of (5) satisfies  $\hat{f}(\mu) = \hat{f}(\mu')$ , then  $(\boldsymbol{w}, \boldsymbol{X})$ ,  $(\boldsymbol{w}', \boldsymbol{X}')$  are not separated by the entire family of functions  $\{F(\cdot; \boldsymbol{a}, b) \mid \boldsymbol{a} \in \mathbb{R}^d, b \in \mathbb{R}\}$ . Since  $\sigma$  is discriminatory, this in turn implies that  $\mu = \mu'$ . This concludes the proof of moment injectivity on  $\mathcal{M}_{\leq n}(\mathbb{R}^d)$ .
If we are only interested in moment injectivity on  $S_{\leq n}(\mathbb{R}^d)$ , it is sufficient to apply the theorem to
$$
\mathbb {M} = \bigcup_ {\boldsymbol {w} \in \{0, 1 \} ^ {n}} \left\{\left(\boldsymbol {w}, \boldsymbol {X}\right) \mid \boldsymbol {X} \in \mathbb {R} ^ {d \times n} \right\},
$$
which is a finite union of affine subspaces of dimension  $D = nd$ . Thus, Theorem 3.4 only requires  $m = 2nd + 1$  to achieve injectivity on  $S_{\leq n}(\mathbb{R}^d)$ . Similarly, when considering  $\mathcal{M}_{\leq n}(\Sigma)$  with a countable  $\Sigma$ , the theorem can be applied to a domain  $\mathbb{M}$  that can be written as a countable union of affine spaces of dimension  $n$ , which yields  $m = 2n + 1$ . Finally,  $S_{\leq n}(\Sigma)$  is a countable union of points, namely zero-dimensional affine subspaces, and therefore  $m = 1$  is sufficient in this case.
3.1 More on the finite witness theorem
The finite witness theorem can be used to prove moment injectivity for functions beyond the activated inner-product form of (5). As an example, we show in the following proposition that Gaussian functions with random parameters are moment injective:
Proposition 3.5. Let  $n, d \in \mathbb{N}$  and set  $m = 2n(d + 1) + 1$ . Let  $\mathbb{W} = (\pmb{y}, \sigma) \in \mathbb{R}^d \times \mathbb{R}_+$ . Then for Lebesgue almost any  $(\pmb{y}_i, \sigma_i)_{i=1}^m \in \mathbb{W}^m$ , the function
$$
f (\boldsymbol {x}) = \left(\exp \left(- \frac {\| \boldsymbol {x} - \boldsymbol {y} _ {1} \| ^ {2}}{\sigma_ {1} ^ {2}}\right), \dots , \exp \left(- \frac {\| \boldsymbol {x} - \boldsymbol {y} _ {m} \| ^ {2}}{\sigma_ {m} ^ {2}}\right)\right)
$$
is moment injective on  $\mathcal{M}_{\leq n}(\mathbb{R}^d)$ .
Proof idea. Any two measures with bounded support can be separated by the moment of a Gaussian function supported on a small ball around a point where the measures disagree. Thus, a measure in  $\mathcal{M}_{\leq n}(\Omega)$  is uniquely defined by the continuous family of all its Gaussian moments. The finite witness theorem then shows that a finite number  $m$  suffices.
The full version of the finite witness theorem (Theorem A.2), discussed in Appendix A, is more general than Theorem 3.4. In this version, the class of sets admissible as  $\mathbb{M}$  is the class of  $\sigma$ -subanalytic sets. While its definition is technically involved (see Appendix A), this class is quite vast: it includes all open sets, all semialgebraic sets (including affine spaces, polygons, and closed  $\ell_2$ -balls), and countable unions thereof. The analyticity assumption on  $F$  is also substantially relaxed to  $\sigma$ -subanalyticity, though this requires an additional condition (13) in the theorem assumptions.
The proof of the finite witness theorem is non-trivial, and we regard it as the main technical contribution of this work. In essence, the proof generalizes a similar result in [9], which only applies to polynomial functions on sets defined by polynomial constraints — known as semialgebraic sets. This class of sets has several nice properties, which the proof in [9] relies on: It is closed under linear projections, finite unions, finite intersections, and complements. Moreover, any semialgebraic set is a finite union of smooth manifolds.
Our generalization from the polynomial to the analytic setting consists of two steps: First, we generalize the theorem to a larger class of sets, called globally subanalytic sets, which are known to be an  $o$ -minimal system — essentially, a family of sets that has the same nice properties of semialgebraic sets mentioned above. This generalization is straightforward; however, it does not allow  $F$  to be an arbitrary analytic function, and thus does not suffice even to prove the weaker version, Theorem 3.4. Our second step is then to observe that our proof carries through also when considering countable unions of globally subanalytic sets, which we name  $\sigma$ -subanalytic sets. This, in turn, paves the way to prove the full version of the finite witness theorem.
Using the more general version of the theorem, we can prove the following proposition, which in particular implies moment injectivity of deep networks, provided that the last activation is analytic:
Proposition 3.6. Let  $\sigma : \mathbb{R} \to \mathbb{R}$  be an analytic non-polynomial function. Let  $n, d \in \mathbb{N}$  and set  $m = 2n(d + 1) + 1$ . Let  $f: \mathbb{R}^d \to \mathbb{R}^L$  be an injective function that is a composition of  $PwL$  functions and analytic functions. Then for Lebesgue almost any  $\pmb{A} \in \mathbb{R}^{m \times L}$ ,  $\pmb{b} \in \mathbb{R}^m$ , the function  $\sigma(\pmb{A}f(\pmb{x}) + \pmb{b})$  is moment injective on  $\mathcal{M}_{\leq n}(\mathbb{R}^d)$ .
In particular,  $F$  could be a neural network that has increasing widths, linear layers with full rank, and injective activations that are either PwL or analytic (such as leaky ReLU or sigmoid). Therefore, Proposition 3.6 shows that increasing the network depth will not have a negative effect on its moment-injectivity. While this may seem trivial, what is not immediate in this formulation is that the embedding dimension  $m$  depends linearly on  $n \cdot d$  rather than  $n \cdot L$ . The reason this is true is that the shallow neural network applied to  $F(\pmb{x})$  will only 'see' inputs that originate from the set  $F\left(\mathbb{R}^d\right)$ , and in Appendix A we show that this is a  $\sigma$ -subanalytic set of dimension  $\leq d$ .
4 Failure of moment injectivity for piecewise-linear functions
In this section, we show that moments of neural networks with piecewise-linear activations (such as ReLU, leaky ReLU and the hard hyperbolic tangent) cannot be injective when the alphabet is infinite, except for some singular cases discussed below.
Proposition 4.1. Let  $d, m$  and  $n \geq 2$  be natural numbers and  $\Omega \subseteq \mathbb{R}^d$  an open set. If  $\psi : \mathbb{R}^d \to \mathbb{R}^m$  is piecewise linear, then it is not moment injective on  $S_{\leq n}(\Omega)$ .
Proof. There exists some open  $U \subset \mathbb{R}^d$  such that  $\psi(\pmb{x})$  is of the form  $\psi(\pmb{x}) = \pmb{A}\pmb{x} + \pmb{b}$  in  $U$ . Let  $\pmb{x}_0 \in U$  and let  $\pmb{d} \neq \pmb{0} \in \mathbb{R}^d$ . For small enough  $\epsilon > 0$ , we have that  $\pmb{x}_0 + \epsilon \pmb{d}$  and  $\pmb{x}_0 - \epsilon \pmb{d}$  are in  $U$ . It follows that the multiset  $\{\{\pmb{x}_0, \pmb{x}_0\}\}$  and  $\{\{\pmb{x}_0 - \epsilon \pmb{d}, \pmb{x}_0 + \epsilon \pmb{d}\}\}$  have the same moments:
$$
\boldsymbol {\psi} \left(\boldsymbol {x} _ {0}\right) + \boldsymbol {\psi} \left(\boldsymbol {x} _ {0}\right) = 2 \left(\boldsymbol {A} \boldsymbol {x} _ {0} + \boldsymbol {b}\right) = \boldsymbol {A} \left(\boldsymbol {x} _ {0} - \epsilon \boldsymbol {d}\right) + \boldsymbol {b} + \boldsymbol {A} \left(\boldsymbol {x} _ {0} + \epsilon \boldsymbol {d}\right) + \boldsymbol {b} = \boldsymbol {\psi} \left(\boldsymbol {x} _ {0} - \epsilon \boldsymbol {d}\right) + \boldsymbol {\psi} \left(\boldsymbol {x} _ {0} + \epsilon \boldsymbol {d}\right).
$$
This proves that  $\psi$  is not moment injective on  $S_{\leq n}(\Omega)$ .
The basic idea behind the above proof is that inside a linear region of  $\psi$ , different multisetts with the same center of mass have the same moments. The same idea can be used to prove failure of moment injectivity of PwL functions on  $\mathcal{M}_{\leq n}(\Omega)$  for any infinite  $\Omega$ , and on  $S_{\leq n}(\mathbb{Z}^d)$ . On the other hand, PwL networks can be moment injective on  $\mathcal{M}_{\leq n}(\Omega)$  with finite  $\Omega$ , as well as on  $S_{\leq n}(\Sigma)$  when  $\Sigma$  is a somewhat pathological infinite countable alphabet. These results are described in Appendix B.
5 Failure of bi-Lipschitzness for general moment functions
In Section 3 we have shown that a neural network  $f: \mathbb{R}^d \to \mathbb{R}^m$  with analytic non-polynomial activation can induce an injective multiset function  $\hat{f}: S_{\leq n}(\mathbb{R}^d) \to \mathbb{R}^m$ . Ideally, we wish such  $\hat{f}$  to be bi-Lipschitz, meaning that there exist constants  $0 < c \leq C$  such that
$$
c \cdot W _ {2} \left(S _ {1}, S _ {2}\right) \leq \| \hat {f} (S _ {1}) - \hat {f} (S _ {2}) \| \leq C \cdot W _ {2} \left(S _ {1}, S _ {2}\right), \quad \forall S _ {1}, S _ {2} \in \mathcal {S} _ {\leq n} (\mathbb {R} ^ {d}), \tag {8}
$$
where  $W_{2}(S_{1}, S_{2})$  is the 2-Wasserstein distance between the two measures  $\mu_{1}, \mu_{2}$  that assign uniform weights to the points in  $S_{1}, S_{2}$  respectively. Unfortunately, we find that any moment function  $\hat{f}$  induced by some  $f: \mathbb{R}^{d} \to \mathbb{R}^{m}$  cannot be bi-Lipschitz, assuming that  $f$  is differentiable in at least one point.
Proposition 5.1. Let  $n \geq 2, d, m \in \mathbb{N}$ , and let  $f: \mathbb{R}^d \to \mathbb{R}^m$  be differentiable at some  $x_0 \in \mathbb{R}^d$ . Then the induced moment function  $\hat{f}: S_{\leq n}(\mathbb{R}^d) \to \mathbb{R}^m$  defined in (1) is not bi-Lipschitz.
Figure 1(b) illustrates the underlying reason for this failure of bi-Lipschitzness, and its relation to the non-injectivity of  $\mathrm{PwL}$  moments: consider a shallow neural network  $f:\mathbb{R}\to \mathbb{R}^{10}$  with ReLU activations, and its induced moment function  $\hat{f} (\{x_1,x_2\}) = f(x_1) + f(x_2)$  on multisets in  $S_{\leq 2}(\mathbb{R})$ . The left-hand side visualizes the ratio  $\sigma_2 / \sigma_1$  of the smallest and largest singular values of the differential matrix  $D\hat{f}$ . The function  $f$  is  $\mathrm{PwL}$ , with four linear regions  $I_{1},\ldots ,I_{4}$  in [0, 1]. The linear regions of  $\hat{f}$  in  $[0,1]^2$  are thus the rectangles  $I_{i}\times I_{j}$ . As seen in the figure, there are degeneracies in the rectangles that intersect the diagonal, as for small enough  $\epsilon$ ,  $\hat{f} (\{x_0 + \epsilon ,x_0 - \epsilon \}) = \hat{f} (\{x_0,x_0\})$  as in the proof of Proposition 4.1. The right-hand side visualizes the same ratio when the analytic SiLU activation is used instead of ReLU. We see that  $D\hat{f}$  is singular on the diagonal. Intuitively, this is because the differentiability of  $f$  implies that it behaves locally like an affine function. This leads to singularities of  $\hat{f}$  on the diagonal, which do not prevent it from being injective, but do prevent it from being bi-Lipschitz. A proof of this phenomenon is given in the appendix. See also Theorem 21 in [4], which independently proved a similar result for general invariant embeddings.
6 Applications: Universal Approximation and Graph Separation
As mentioned in the introduction, injective multiset functions can be used to construct multiset architectures with universal approximation power, and to prove separation results for graph neural networks. In this section, we present some immediate corollaries of our results for these two applications. Proofs are in Appendix D.3.
6.1 Universal approximation of functions on multisets and measures
Our first approximation result focuses on multiset of a fixed size  $n$  with an alphabet  $K \subseteq \mathbb{R}^d$  that is compact. Any such multiset is determined by a choice of  $n$  vectors in  $K$ , possibly with repetitions and irrespective of order. Thus, multiset functions on this space are equivalent to permutation-invariant functions on  $K^n$ . Using the finite witness theorem and a basic topological argument, we prove:
Corollary 6.1. Let  $n, d \in \mathbb{N}$  and set  $m = 2nd + 1$ . Let  $\sigma : \mathbb{R} \to \mathbb{R}$  be an analytic non-polynomial function. Let  $K \subseteq \mathbb{R}^d$  be a compact set. Then there exist  $A \in \mathbb{R}^{m \times d}$ ,  $b \in \mathbb{R}^d$  such that for any continuous permutation-invariant  $f: K^n \to \mathbb{R}$ , there exists a continuous  $F: \mathbb{R}^m \to \mathbb{R}$  such that
$$
f (\boldsymbol {X}) = F \left(\sum_ {j = 1} ^ {n} \sigma (\boldsymbol {A} \boldsymbol {x} _ {j} + \boldsymbol {b})\right), \quad \forall \boldsymbol {X} = \left(\boldsymbol {x} _ {1}, \dots , \boldsymbol {x} _ {n}\right) \in K ^ {n}. \tag {9}
$$
Combining Corollary 6.1 with the universality of MLPs, we get that any continuous permutation-invariant function on  $K^n$  can be approximated by expressions of the form (9) with  $F$  replaced by an MLP. Similar results were obtained for moments of polynomials rather than of MLPs in [45, 42, 9].
It is worth noting that an analogue of Corollary 6.1 cannot hold with a piecewise-linear  $\sigma$ , assuming that  $K$  has a non-empty interior. This is because by Proposition 4.1, any fixed moment function induced by a PwL MLP will not be able to separate all multiset, whereas any two distinct multiset can be separated by some continuous  $f$ . Though, with a PwL  $\sigma$ , one may approximate any given  $f$  to
arbitrary precision, by taking the embedding dimension  $m$  to infinity. In contrast, with an analytic  $\sigma$ , we are able to specify a finite  $m = 2nd + 1$  for which exact equality in (9) is guaranteed.
Since our injectivity results on multiset sets extend to measures, it is natural to seek an extension of the above approximation result to functions defined on measures. Denote by  $\mathcal{P}_{\leq n}(K)$  the space of probability measures supported on  $\leq n$  points in  $K \subseteq \mathbb{R}^d$ , endowed with the 2-Wasserstein metric.
Corollary 6.2. Let  $n, d \in \mathbb{N}$  and set  $m = 2n(d + 1) + 1$ . Let  $\sigma : \mathbb{R} \to \mathbb{R}$  be analytic and non-polynomial. Let  $K \subseteq \mathbb{R}^d$  be compact. Then there exist  $A \in \mathbb{R}^{m \times d}, b \in \mathbb{R}^m$  such that for any continuous (in the 2-Wasserstein sense)  $f : \mathcal{P}_{\leq n}(K) \to \mathbb{R}$ , there exists a continuous  $F : \mathbb{R}^m \to \mathbb{R}$  such that
$$
f (\mu) = F \left(\int_ {\boldsymbol {x} \in K} \sigma (\boldsymbol {A} \boldsymbol {x} + \boldsymbol {b}) d \mu (\boldsymbol {x})\right), \quad \forall \mu \in \mathcal {P} _ {\leq n} (K).
$$
It follows from Corollary 6.2 that any continuous function  $f: \mathcal{P}_{\leq n}(K) \to \mathbb{R}$  with compact  $K \subseteq \mathbb{R}^d$  can be approximated to arbitrary precision by functions of the form  $\hat{f}(\mu) = F\left(\int_{\boldsymbol{x} \in K} \sigma(\boldsymbol{A}\boldsymbol{x} + \boldsymbol{b}) d\mu(\boldsymbol{x})\right)$ , with  $\boldsymbol{A} \in \mathbb{R}^{m \times d}$ ,  $\boldsymbol{b} \in \mathbb{R}^m$ ,  $m = 2nd + 1$ , and  $F$  being an MLP.
6.2 Graph separation
We now discuss the implications of Theorem 3.3 for graph separation, using terminology from [44]. Let  $\mathcal{G}_{\leq n}(\Sigma)$  be the collection of all graphs  $G = (V,E,\pmb{h}^{(0)})$  with at most  $n$  vertices, endowed with vertex features  $\pmb{h}_v^{(0)}\in \Sigma$ , where  $\Sigma \subseteq \mathbb{R}^d$  is a countable alphabet. We consider GIN-like [44] MPNNs that recursively, for  $t = 1,\dots ,T$ , calculate node features  $\pmb{h}_v^{(t)}$  from the previous features  $\pmb{h}_v^{(t - 1)}$  by
$$
\boldsymbol {h} _ {v} ^ {(t)} = \sum_ {u \in \mathcal {N} (v)} \sigma \left(\boldsymbol {A} ^ {(t)} \left(\eta^ {(t)} \boldsymbol {h} _ {v} ^ {(t - 1)} + \boldsymbol {h} _ {u} ^ {(t - 1)}\right) + \boldsymbol {b} ^ {(t)}\right). \tag {10}
$$
After the  $T$  iterations are concluded, a global feature is computed via a readout function:
$$
\boldsymbol {h} _ {G} = \sum_ {v \in V} \sigma \left(\boldsymbol {A} ^ {(T + 1)} \boldsymbol {h} _ {v} ^ {(T)} + \boldsymbol {b} ^ {(T + 1)}\right). \tag {11}
$$
We choose all features  $h_G$  and  $h_v^{(t)}$  for  $1 \leq t \leq T$ , to have the same dimension  $m$ . Based on the fact that MPNNs are equivalent to 1-WL when the multiset functions are injective [44], and on our injectivity results for countable alphabets, we prove that:
Theorem 6.3. Let  $n, d, T \in \mathbb{N}$  and let  $\Sigma \subseteq \mathbb{R}^d$  be countable. Let  $m \geq 1$  be any integer. Let  $\sigma : \mathbb{R} \to \mathbb{R}$  be an analytic non-polynomial function. Then for Lebesgue almost any choice of  $A^{(t)}, b^{(t)}$  and  $\eta^{(t)}$ , the MPNN defined in (10) and (11) assigns different global features to any pair of graphs  $G_1, G_2 \in \mathcal{G}_{\leq n}(\Sigma)$  that can be separated by  $T$  iterations of 1-WL.
Graph separation with continuous features Up to now, we have discussed graphs with node features coming from a countable alphabet. Since Theorem 3.3 applies to multisets with continuous alphabets, it can be applied to separation of graphs with continuous node features as well. In particular, the paper [15] explains how the random semialgebraic multiset function from [9] can be used to construct architectures for graphs with continuous features, whose separation power is equivalent to WL tests. Their focus is on showing that the embedding dimension in their construction depends linearly on the dimension of the graph space, rather than grows exponentially with the number of message-passing iterations  $T$ . Similar results can now be obtained by using our random analytic multiset functions. We leave a full description of these aspects to future work.
Empirical injectivity and bi-Lipschitzness We empirically investigated the injectivity and bi-Lipschitzness of moments of shallow networks of the form (3), by randomly generating a large number of pairs of multisets of  $n$  vectors in  $\mathbb{R}^d$ , and computing the optimal constants  $c, C$  for which (8) holds for the generated pairs. The ratio  $c / C$  for varying activations and embedding dimension  $m$  is shown in Figure 2. Here  $d = 3$  and  $n = 1000$ . Similar qualitative results were obtained for other values of  $d$  and  $n$ ; see Figure 4 in Appendix E.
We observe several interesting phenomena: First, at low embedding-dimensions, the ratio  $c / C$  for PwL networks is exactly zero, indicating that they are not injective even on the finite sample set. In contrast, for analytic activations,  $c / C$  is always positive. Indeed, we expect analytic activations to be injective on a finite sample even with embedding dimension  $m = 1$ , since a finite sample set has an intrinsic dimension of zero. Next, we observe that  $c / C$  naturally improves as  $m$  increases. Finally, we note that even for high  $m$ ,  $c / C$  is rather small.
Indeed, if it were possible to consider all pairs of multisets when computing  $c / C$ , we would get zero for all activations and all embedding dimensions, as follows from Theorem 5.1. For additional details on this experiment, see Appendix E.1.
Graph Separation To validate Theorem 6.3, we conducted the following experiment: we considered 600 graphs from the TUDataset [30]. On each graph we ran three iterations of the WL test, and three iterations of MPNNs with the Graph Convolutional Layers from [18] with different activations and hidden dimensions. Our goal was to check in how many graphs the MPNNs returned a vertex coloring that differs from the coloring provided by the WL test. The results are shown in Figure 1(a). As seen in the table,
with the three analytic activations tested, the vertex coloring of MPNN was always equivalent to 1-WL, even with a hidden dimension of 1. On the other hand, for the three  $PwL$  activations, there were inconsistencies in about  $1\%$  of the graphs, even with a hidden dimension of 50.
We note that while analytic activations fully succeeded in separation, in some cases the separation was rather weak: while the distance between features of non-equivalent nodes computed by the MPNNs was typically around 0.1, the least-separated features had a distance of  $\sim 10^{-7}$ . In future work, it could be interesting to investigate whether MPNNs can be trained to yield larger distances between the features of all non-equivalent nodes. Further details on this experiment appear in Appendix E.2.
We have shown that moments of neural networks with an analytic non-polynomial activation are injective on multisets and measures. We have also shown how this can be harnessed to construct universal approximators for multiset functions, as well as prove separation results for graph neural networks. A key advantage of our approach is that it enables constructing proofs using real models that are used in practice, rather than idealized versions of them as done in previous works.
It may seem tempting, due to our theoretical results, to conclude that analytic activations should perform on multisets better than piecewise-linear activations. We stress that we make no such claim. Indeed, while the separation results in Figure 1(a) corroborate our theory, PwL networks fail to separate only  $1\%$  of the graphs in our experiment. Furthermore, at high embedding dimensions, the empirical bi-Lipschitzness in Figure 2 does not seem to strongly depend on the analyticity of the activation function. Our claim is thus much more modest: we claim that multiset architectures with analytic activations are easier to theoretically analyze, and we hope that pursuing this analysis shall lead to fruitful theoretical and practical insights, which may ultimately benefit multiset architectures with either type of activation.
Lastly, we note that the finite witness theorem, which is presented here as a tool for proving moment injectivity, may prove valuable as a general tool for reducing an infinite number of equality constraints to a finite number, and we believe it will find additional applications beyond the scope of this paper.
Acknowledgements N.D. is partially funded by a Horev Fellowship. T.A, R.R. and N.D. are partially funded by ISF grant 272/23.