Title: Neural Injective Functions for Multisets, Measures and Graphs via a Finite Witness Theorem
Venue: NeurIPS 2023
Accepted: yes
Keywords: Equivariant Neural Networks

Universal approximation

Geometric deep learning

multiset learning

injective multiset functions

learning on measures. WL test
Abstract: Injective multiset functions have a key role in the theoretical study of machine learning on multisets and graphs. Yet, there remains a gap between the provably injective multiset functions considered in theory, which typically rely on polynomial moments, and the multiset functions used in practice, which rely on $\textit{neural moments}$ — whose injectivity on multisets has not been studied to date.

In this paper, we bridge this gap by showing that moments of neural networks do define injective multiset functions,  provided that an analytic non-polynomial activation is used. The number of moments required by our theory is optimal essentially up to a multiplicative factor of two. To prove this result, we state and prove a $\textit{finite witness theorem}$, which is of independent interest. 

As a corollary to our main theorem, we derive new approximation results for functions on multisets and measures, and new separation results for graph neural networks. We also provide two negative results: (1) moments of piecewise-linear neural networks cannot be injective multiset functions; and (2) even when moment-based multiset functions are injective, they can never be bi-Lipschitz.

Neural Injective Functions for Multisets, Measures and Graphs via a Finite Witness Theorem

Abstract

Injective multiset functions have a key role in the theoretical study of machine learning on multisets and graphs. Yet, there remains a gap between the provably injective multiset functions considered in theory, which typically rely on polynomial moments, and the multiset functions used in practice, which rely on neural moments -whose injectivity on multisets has not been studied to date.

In this paper, we bridge this gap by showing that moments of neural networks do define injective multiset functions, provided that an analytic non-polynomial activation is used. The number of moments required by our theory is optimal essentially up to a multiplicative factor of two. To prove this result, we state and prove a finite witness theorem , which is of independent interest.

As a corollary to our main theorem, we derive new approximation results for functions on multisets and measures, and new separation results for graph neural networks. We also provide two negative results: (1) moments of piecewise-linear neural networks cannot be injective multiset functions; and (2) even when momentbased multiset functions are injective, they can never be bi-Lipschitz.

1 Introduction

Multisets are a slight generalization of sets: like sets, they are an unordered collection of elements { { x 1 , . . . , x k } } , but unlike sets, repetitions are allowed. Multisets arise naturally in many machinelearning tasks. They are the natural way to represent point clouds in R 3 , neighborhoods of vertices in graphs, and any other data that has an intrinsic order that is immaterial to the task at hand.

We refer to functions and architectures whose inputs are multisets in R d as multiset functions and multiset architectures . By definition, these functions do not depend on the order in which the multiset elements are given. This is important not only because the order is irrelevant and thus should not affect the output, but also because otherwise a model may overfit the training data by making predictions based on its intrinsic order.

Multiset architectures are typically constructed using a combination of permutation-invariant operations such as sum- and max-pooling [34], attention mechanisms [23] and sorting [46]. One simple and popular approach, pioneered in the seminal Deep-Sets paper [45], employs multiset functions based solely on sum-pooling. Namely, if the elements of all multisets come from some fixed alphabet Ω , any function f : Ω → R m induces a multiset function ˆ f , to which we refer as the moment of f :

While moment functions of the form (1) are simple, they are quite powerful. For example, in [45] it was shown that if Ω is countable, and the multisets have no repetitions (so they are just sets), then for an appropriate f : Ω → R , the induced function ˆ f maps the input sets injectively to R .

glyph[negationslash]

Injectivity is indeed a desired property for multiset functions. The search for such functions stems from the quest to find an architecture that can approximate all multiset functions. Clearly, if ˆ f assigns the same value ˆ f ( S 1 ) = ˆ f ( S 2 ) to two different multisets S 1 = S 2 , then any architecture based on ˆ f will yield a poor approximation of a multiset function that assigns different values to S 1 and S 2 .

The authors of [45] showed that injectivity is not only necessary for approximation, but also sufficient: Under the assumption that the alphabet Ω is countable, if the moment ˆ f of f maps multisets injectively to R m , then any multiset function F can be written as a composition of the form F ( { { x 1 , . . . , x k } } ) = g ( ∑ k i =1 f ( x i ) ) . Motivated by this observation, the authors proposed a neural architecture of this form, with the functions f and g replaced by Multi-Layer Perceptrons (MLPs). This step was justified by the universal approximation power of MLPs.

These intriguing results inspired further research, mainly focusing on seeking injective multiset functions of the form (1) for continuous alphabets such as Ω = R d . Preferably, such functions should (a) have a minimal embedding dimension m while ensuring that f : Ω → R m induces an injective ˆ f ; and (b) be practical to compute. We next summarize some of these results:

For a countable Ω , the Deep-Sets paper as well as [44] showed that an embedding dimension m = 1 is sufficient. For Ω = R , if the multisets contain at most n elements, and f is continuous, then an embedding dimension of m ≥ n is necessary and sufficient for injectivity [41, 6, 42].

For Ω = R d and multisets of size at most n , it was shown in [16] that m ≥ nd is necessary for injectivity. As for an upper bound on the required m , while some polynomials discussed in the literature achieve injectivity with a rather high exponential [3, 26, 37] or polynomial [43] dimension m , recent work [9] achieved injectivity with m = 2 nd +1 , using a polynomial f with randomly chosen coefficients - thus achieving the lower bound essentially up to a multiplicative factor of 2 .

While the above works provide injective multiset functions with optimal or near-optimal embedding dimension, these functions are typically polynomials, and not the MLPs used in practice. As mentioned above, many papers [45, 44, 26] justify this by the fact that MLPs can approximate any function, and thus any polynomial. However, using this argument, we have no control on the number of neurons required for injectivity - which in some cases may be infinite, as we show in Section 4. In this paper, we address this limitation by providing a practical and efficient method to construct functions of the form (1) that are provably injective while having a near-optimal number of neurons. We now state this formally.

1.1 Problem Statement

Let Ω ⊆ R d be a set, to which we refer as an alphabet . Denote by S ≤ n (Ω) the collection of all multisets { { x 1 , . . . , x k } } with x 1 , . . . , x k ∈ Ω and k ≤ n . Any function f : Ω → R m induces a moment function ˆ f : S ≤ n (Ω) → R m as in (1). If ˆ f is injective, we say that f is moment injective on S ≤ n (Ω) .

We also consider a natural generalization from multisets to measures, by identifying each multiset { { x 1 , . . . , x k } } with the measure µ = ∑ k i =1 δ x i , where δ x is the Dirac measure that assigns a unit weight to x . In this generalized setting, the induced multiset function ˆ f of (1) is just the integral of f with respect to the measure µ . More generally, we consider signed measures µ = ∑ n i =1 w i δ x i , with weights w i ∈ R that can be negative, and points x i that belong to an alphabet Ω ⊆ R d . We denote the space 1 of all such measures by M ≤ n (Ω) . A function f : Ω → R m induces a moment function ˆ f : M ≤ n (Ω) → R m defined by

If ˆ f is injective, we say that f is moment-injective on M ≤ n (Ω) . Naturally, injectivity on M ≤ n (Ω) implies injectivity on subsets of this space, such as the space of measures in M ≤ n (Ω) that are probability measures, or that have only positive weights. In particular, if f is moment-injective on M ≤ n (Ω) , then it is moment-injective on S ≤ n (Ω) .

To summarize, the main questions we focus on in this paper are:

Main Questions: (a) Under what conditions is an MLP f moment-injective on spaces of multisets S ≤ n (Ω) or measures M ≤ n (Ω) ? (b) How many neurons are needed to achieve this injectivity?

2 Main Results

Interestingly, we find that the answers to these two questions largely depend on the activation function. Consider shallow neural networks f : R d → R m of the form

with the activation function σ : R → R applied entrywise to Ax + b . Suppose that σ is analytic and non-polynomial; such activations include the sigmoid, softplus, tanh, swish and sin. In Section 3 we show that for a large enough m , such networks f ( x ; A , b ) with random parameters A , b are moment-injective on M ≤ n (Ω) and on S ≤ n (Ω) ; namely, their induced moment functions ˆ f of (2) are injective. This holds for various natural choices of Ω .

The embedding dimension m required in (3) depends on the dimension d of Ω : For Ω = R d , to achieve injectivity on S ≤ n (Ω) or M ≤ n (Ω) , it suffices to take m = 2 nd +1 or m = 2 n ( d +1) + 1 respectively. When Ω is countable, m = 1 or m = 2 n +1 are sufficient (corresponding to d = 0 ). In Appendix C we show that in all these cases, these embedding dimensions are optimal essentially up to a multiplicative factor of two. These results are summarized in Table 1. In Appendix C we also discuss examples where the optimal embedding cardinality for M ≤ n ( R ) is obtained.

At the core of our poof of moment injectivity is a theorem which we name the finite witness theorem . This theorem enables reducing an infinite family of analytic equality constraints { F ( x ; θ ) = 0 | θ ∈ W } to a finite subset { F ( x ; θ i ) = 0 | i = 1 , . . . , m } . This theorem generalizes the results in [9], where a special case of it was proved for semialgebraic domains and functions. The theorem we prove here (see Appendix A) applies to a much wider class of domains and functions, among which are analytic functions. In addition to our main result, we use the finite witness theorem to prove moment injectivity of Gaussian functions (Proposition 3.5) and deep MLPs (Proposition 3.6), and we believe it shall find additional applications beyond those discussed in this work.

Negative Results We also prove two negative results for moment-based multiset functions: We show that in contrast to analytic activations, with piecewise linear (PwL) activations, such as ReLU, leaky ReLU and hard arctan, moment injectivity on spaces of measures M ≤ n (Ω) with infinite Ω is impossible . On multiset spaces S ≤ n (Ω) , moment injectivity with PwL activations can be obtained for some irregular, countable Ω , such as the alphabet Ω = Σ α defined in Appendix B, but not for infinite alphabets that arise naturally, like Ω = R d , Z d or (0 , 1) d . These results are summarized in the bottom row of Table 1. The second negative result is that while moments of MLPs with analytic activations can be injective, they can never be stable in the bi-Lipschitz sense. This points to a possible advantage

ReLU

SiLU

0

0.3

0.25

0

(a)

(b)

Analytic

PwL

Dim

SiLU

Sin

HardTanh

ReLU

一

0

0

0

7

17

7

10

0

0

0

3

7

7

50

0

0

0

+

5

5

100

0

0

0

一

0

0

of injective multiset functions that are not based on moments, but rather on sorting [3] or max-filters [5]. These multiset functions are not only injective but also bi-Lipschitz.

Implications for learning on multisets and graphs The result on moment injectivity of MLPs with analytic non-polynomial activation enables us to improve upon two seminal theoretical results in the study of functions on multisets and graphs:

(a) Universality for multisets. In Corollaries 6.1 and 6.2, we show that any continuous function on a space of multisets or measures respectively can be presented as a continuous vector-to-vector function composed with a moment function ˆ f of an MLP of the form (3). The MLP has the same embedding dimension m as in Table 1. Essentially, this result replaces the moment-injective polynomials traditionally used in the characterization of multiset functions [45, 42] by MLPs.

(b) Separation power of Graph Neural Networks. Famously, the ability of Message-Passing Neural Networks (MPNNs) to separate distinct graphs is at most that of the Weisfeiler-Leman (WL) graph isomorphism test, with equivalence taking place if the multiset functions used in the MPNN are injective [44]. Injective multiset functions are also used in generalizations of this result, such as the equivalence of high-order Graph Neural Networks (GNNs) to high-order WL tests [31, 26], and recent results on geometric GNNs and their corresponding WL tests [15, 16, 25, 33, 8].

Using the fact that an embedding dimension of one is sufficient to achieve injectivity on S ≤ n (Ω) with countable Ω , we show in Theorem 6.3 that standard MPNNs with analytic non-polynomial activations and random parameters have the separation power of WL, even when their architecture only uses a single feature per node. This can be compared on the one hand with the construction in [44], which also requires a single node feature but uses multiset aggregators that are not MLPs, and on the other hand with works that do consider MLPs with ReLU activations [31, 1], but require a number of node features and parameters that depends polylogarithmically on the number of nodes. In contrast, our construction requires a single node feature and a fixed number of parameters (though we have no bound on the number of bits required for achieving separation using floating-point arithmetic). A numerical verification of these results is shown in Figure 1(a), where we show that, on the 600 graphs in the TUDataset [30], MPNNs with three different analytic activations were equivalent to the WL-test even with a single node feature, whereas three different PwL activations were in some cases weaker than WL when a small number of node features was used.

Independently of this work, it was recently proved in [17] that MPNNs with certain analytic activations can separate all trees of depth two, while separation fails with PwL and even piecewise-polynomial activations. Our results here are stronger in that we show separation for all graphs separable by WL, and all analytic non-polynomial activations.

2.1 Notation

We denote vectors by boldface letters, e.g. x , y , and scalars by plain letters x, y . The inner product of a , x is denoted by a · x . Throughout this work, the term measure always refers to signed measures.

3 Moment injectivity with analytic activations

In this section, we prove moment injectivity for MLPs with analytic non-polynomial activations. We begin by showing that for any non-polynomial function σ : R → R , a measure µ ∈ M ≤ n (Ω) is uniquely determined by the integrals of all functions { σ ( a · x + b ) | a ∈ R d , b ∈ R } . When this holds, we say that σ is discriminatory :

glyph[negationslash]

Definition 3.1. Let σ : R → R be a continuous function. We say that σ is discriminatory if for any two signed Borel measures µ, µ ′ on R d that are distinct (i.e. µ = µ ′ ), finite (i.e. | µ ( A ) | , | µ ′ ( A ) | < ∞ for all Borel A ⊆ R d ) and compactly supported, there exist a ∈ R d , b ∈ R such that glyph[negationslash]

The definition of discriminatory activation functions comes from 2 Cybenko's celebrated paper on the universality of MLPs [7], where it was proved that sigmoid-like activations are discriminatory. This, in turn, was used to prove the universality of MLPs with such activations. In later papers [24, 32] it was shown that universality can be achieved by all continuous non-polynomial activations. In the following simple proposition, we use a reverse argument to that used by Cybenko, and show that activations that allow for universality are automatically discriminatory:

Proposition 3.2. Let σ : R → R be a continuous function that is not a polynomial; then σ is discriminatory.

Proof idea. Suppose that ∫ σ ( a · x + b ) dµ = ∫ σ ( a · x + b ) dµ ′ for all a , b . By the universality theorem for shallow MLPs [32], all continuous functions can be approximated by linear combinations of functions of the form σ ( a · x + b ) . Thus, for any continuous function f , ∫ fdµ = ∫ fdµ ′ . Since a measure is uniquely determined by its integrals of all continuous functions, µ is equal to µ ′ .

Next, we shall prove our main result: If σ is analytic and discriminatory, then shallow MLPs of reasonable width with σ as activation are moment injective.

Theorem 3.3. Let σ : R → R be an analytic non-polynomial function. Let n, d ∈ N , and set m = 2 n ( d + 1) + 1 . Then for Lebesgue almost any A ∈ R m × d , b ∈ R m , the shallow MLP f ( x ) = σ ( A · x + b ) is moment injective on M ≤ n ( R d ) ; namely, the function ˆ f : M ≤ n (Ω) → R m given by

is injective.

For moment injectivity on S ≤ n ( R d ) , it suffices to take m = 2 nd +1 . For M ≤ n (Σ) or S ≤ n (Σ) with countable Σ , m = 2 n +1 and m = 1 respectively are sufficient.

Our proof of Theorem 3.3 is based on a separate theorem, which we name the finite witness theorem . This theorem enables us to show that, since any two measures can be discriminated by an integral ∫ σ ( a · x + b ) dµ ( x ) for some choice of parameters a , b , there exists a finite number of witness parameters ( a i , b i ) m i =1 that are sufficient for discriminating between any two measures. This holds under the assumption that the number of points in both measures is bounded. We shall now state a simple version of this theorem, which suffices for proving Theorem 3.3.

Theorem 3.4. (Finite Witness Theorem, simple version) Let M ⊆ R L be a countable union of affine sets, each of which is of dimension ≤ D . Let W ⊆ R D θ be open and connected. Let F ( x ; θ ) : M × W → R be an analytic function. Then for almost any ( θ (1) , . . . , θ (2 D +1) ) ∈ W 2 D +1 , the following set equality holds:

Using the finite witness theorem, we are now ready to prove Theorem 3.3.

Proof of Theorem 3.3. Recall that a signed measure µ ∈ M ≤ n ( R d ) can be parameterized, albeit not uniquely, by a matrix X = ( x 1 , . . . , x n ) ∈ R d × n representing n points in R d , and a weight vector w = ( w 1 , . . . , w n ) , such that µ = ∑ n i =1 w i δ x i . Let M be the space of measure parameters

Similarly, let W be the space of parameters

We now invoke the finite witness theorem. Set m = 2 n ( d +1)+1 , and note that m = 2dim( M ) +1 . Recall that F is analytic. According to Theorem 3.4, for almost any choice of ( a i , b i ) m i =1 ∈ W ,

Let A ∈ R m × d with rows a 1 , . . . , a m , and b = ( b 1 , . . . , b m ) . Suppose that A , b indeed satisfy (7).

Let µ, µ ′ ∈ M ≤ n (Ω) be two measures with parameters ( w , X ) , ( w ′ , X ′ ) respectively. Equation (7) implies that if the function ˆ f of (5) satisfies ˆ f ( µ ) = ˆ f ( µ ′ ) , then ( w , X ) , ( w ′ , X ′ ) are not separated by the entire family of functions { F ( · ; a , b ) | a ∈ R d , b ∈ R } . Since σ is discriminatory, this in turn implies that µ = µ ′ . This concludes the proof of moment injectivity on M ≤ n ( R d ) .

If we are only interested in moment injectivity on S ≤ n ( R d ) , it is sufficient to apply the theorem to

which is a finite union of affine subspaces of dimension D = nd . Thus, Theorem 3.4 only requires m = 2 nd + 1 to achieve injectivity on S ≤ n ( R d ) . Similarly, when considering M ≤ n (Σ) with a countable Σ , the theorem can be applied to a domain M that can be written as a countable union of affine spaces of dimension n , which yields m = 2 n +1 . Finally, S ≤ n (Σ) is a countable union of points, namely zero-dimensional affine subspaces, and therefore m = 1 is sufficient in this case.

3.1 More on the finite witness theorem

The finite witness theorem can be used to prove moment injectivity for functions beyond the activated inner-product form of (5). As an example, we show in the following proposition that Gaussian functions with random parameters are moment injective:

Proposition 3.5. Let n, d ∈ N and set m = 2 n ( d +1) + 1 . Let W = ( y , σ ) ∈ R d × R + . Then for Lebesgue almost any ( y i , σ i ) i m =1 ∈ W m , the function

is moment injective on M ≤ n ( R d ) .

Proof idea. Any two measures with bounded support can be separated by the moment of a Gaussian function supported on a small ball around a point where the measures disagree. Thus, a measure in M ≤ n (Ω) is uniquely defined by the continuous family of all its Gaussian moments. The finite witness theorem then shows that a finite number m suffices.

Define F : M × W → R by

The full version of the finite witness theorem (Theorem A.2), discussed in Appendix A, is more general than Theorem 3.4. In this version, the class of sets admissible as M is the class of σ -subanalytic sets. While its definition is technically involved (see Appendix A), this class is quite vast: it includes all open sets, all semialgebraic sets (including affine spaces, polygons, and closed glyph[lscript] 2 -balls), and countable unions thereof. The analyticity assumption on F is also substantially relaxed to σ -subanalyticity, though this requires an additional condition (13) in the theorem assumptions.

The proof of the finite witness theorem is non-trivial, and we regard it as the main technical contribution of this work. In essence, the proof generalizes a similar result in [9], which only applies to polynomial functions on sets defined by polynomial constraints - known as semialgebraic sets . This class of sets has several nice properties, which the proof in [9] relies on: It is closed under linear projections, finite unions, finite intersections, and complements. Moreover, any semialgebraic set is a finite union of smooth manifolds.

Our generalization from the polynomial to the analytic setting consists of two steps: First, we generalize the theorem to a larger class of sets, called globally subanalytic sets , which are known to be an o-minimal system - essentially, a family of sets that has the same nice properties of semialgebraic sets mentioned above. This generalization is straightforward; however, it does not allow F to be an arbitrary analytic function, and thus does not suffice even to prove the weaker version, Theorem 3.4. Our second step is then to observe that our proof carries through also when considering countable unions of globally subanalytic sets, which we name σ -subanalytic sets . This, in turn, paves the way to prove the full version of the finite witness theorem.

Using the more general version of the theorem, we can prove the following proposition, which in particular implies moment injectivity of deep networks, provided that the last activation is analytic:

Proposition 3.6. Let σ : R → R be an analytic non-polynomial function. Let n, d ∈ N and set m = 2 n ( d + 1) + 1 . Let f : R d → R L be an injective function that is a composition of PwL functions and analytic functions. Then for Lebesgue almost any A ∈ R m × L , b ∈ R m , the function σ ( A f ( x ) + b ) is moment injective on M ≤ n ( R d ) .

In particular, F could be a neural network that has increasing widths, linear layers with full rank, and injective activations that are either PwL or analytic (such as leaky ReLU or sigmoid). Therefore, Proposition 3.6 shows that increasing the network depth will not have a negative effect on its moment-injectivity. While this may seem trivial, what is not immediate in this formulation is that the embedding dimension m depends linearly on n · d rather than n · L . The reason this is true is that the shallow neural network applied to F ( x ) will only 'see' inputs that originate from the set F ( R d ) , and in Appendix A we show that this is a σ -subanalytic set of dimension ≤ d .

4 Failure of moment injectivity for piecewise-linear functions

In this section, we show that moments of neural networks with piecewise-linear activations (such as ReLU, leaky ReLU and the hard hyperbolic tangent) cannot be injective when the alphabet is infinite, except for some singular cases discussed below.

Proposition 4.1. Let d, m and n ≥ 2 be natural numbers and Ω ⊆ R d an open set. If ψ : R d → R m is piecewise linear, then it is not moment injective on S ≤ n (Ω) .

glyph[negationslash]

Proof. There exists some open U ⊂ R d such that ψ ( x ) is of the form ψ ( x ) = Ax + b in U . Let x 0 ∈ U and let d = 0 ∈ R d . For small enough glyph[epsilon1] > 0 , we have that x 0 + glyph[epsilon1] d and x 0 -glyph[epsilon1] d are in U . It follows that the multisets { { x 0 , x 0 } } and { { x 0 -glyph[epsilon1] d , x 0 + glyph[epsilon1] d } } have the same moments:

This proves that ψ is not moment injective on S ≤ n (Ω)

The basic idea behind the above proof is that inside a linear region of ψ , different multisets with the same center of mass have the same moments. The same idea can be used to prove failure of moment injectivity of PwL functions on M ≤ n (Ω) for any infinite Ω , and on S ≤ n ( Z d ) . On the other hand, PwL networks can be moment injective on M ≤ n (Ω) with finite Ω , as well as on S ≤ n (Σ) when Σ is a somewhat pathological infinite countable alphabet. These results are described in Appendix B.

5 Failure of bi-Lipschitzness for general moment functions

In Section 3 we have shown that a neural network f : R d → R m with analytic non-polynomial activation can induce an injective multiset function ˆ f : S ≤ n ( R d ) → R m . Ideally, we wish such ˆ f to be bi-Lipschitz , meaning that there exist constants 0 < c ≤ C such that

where W 2 ( S 1 , S 2 ) is the 2-Wasserstein distance between the two measures µ 1 , µ 2 that assign uniform weights to the points in S 1 , S 2 respectively. Unfortunately, we find that any moment function ˆ f induced by some f : R d → R m cannot be bi-Lipschitz, assuming that f is differentiable in at least one point.

Proposition 5.1. Let n ≥ 2 , d, m ∈ N , and let f : R d → R m be differentiable at some x 0 ∈ R d . Then the induced moment function ˆ f : S ≤ n ( R d ) → R m defined in (1) is not bi-Lipschitz.

Figure 1(b) illustrates the underlying reason for this failure of bi-Lipschitzness, and its relation to the non-injectivity of PwL moments: consider a shallow neural network f : R → R 10 with ReLU activations, and its induced moment function ˆ f ( { { x 1 , x 2 } } ) = f ( x 1 ) + f ( x 2 ) on multisets in S ≤ 2 ( R ) . The left-hand side visualizes the ratio σ 2 /σ 1 of the smallest and largest singular values of the differential matrix D ˆ f . The function f is PwL, with four linear regions I 1 , . . . , I 4 in [0 , 1] . The linear regions of ˆ f in [0 , 1] 2 are thus the rectangles I i × I j . As seen in the figure, there are degeneracies in the rectangles that intersect the diagonal, as for small enough glyph[epsilon1] , ˆ f ( { { x 0 + glyph[epsilon1],x 0 -glyph[epsilon1] } } ) = ˆ f ( { { x 0 , x 0 } } ) as in the proof of Proposition 4.1. The right-hand side visualizes the same ratio when the analytic SiLU activation is used instead of ReLU. We see that D ˆ f is singular on the diagonal. Intuitively, this is because the differentiability of f implies that it behaves locally like an affine function. This leads to singularities of ˆ f on the diagonal, which do not prevent it from being injective, but do prevent it from being bi-Lipschitz. A proof of this phenomenon is given in the appendix. See also Theorem 21 in [4], which independently proved a similar result for general invariant embeddings.

6 Applications: Universal Approximation and Graph Separation

As mentioned in the introduction, injective multiset functions can be used to construct multiset architectures with universal approximation power, and to prove separation results for graph neural networks. In this section, we present some immediate corollaries of our results for these two applications. Proofs are in Appendix D.3.

6.1 Universal approximation of functions on multisets and measures

Our first approximation result focuses on multisets of a fixed size n with an alphabet K ⊆ R d that is compact. Any such multiset is determined by a choice of n vectors in K , possibly with repetitions and irrespective of order. Thus, multiset functions on this space are equivalent to permutation-invariant functions on K n . Using the finite witness theorem and a basic topological argument, we prove:

Corollary 6.1. Let n, d ∈ N and set m = 2 nd +1 . Let σ : R → R be an analytic non-polynomial function. Let K ⊆ R d be a compact set. Then there exist A ∈ R m × d , b ∈ R d such that for any continuous permutation-invariant f : K n → R , there exists a continuous F : R m → R such that

Combining Corollary 6.1 with the universality of MLPs, we get that any continuous permutationinvariant function on K n can be approximated by expressions of the form (9) with F replaced by an MLP. Similar results were obtained for moments of polynomials rather than of MLPs in [45, 42, 9].

It is worth noting that an analogue of Corollary 6.1 cannot hold with a piecewise-linear σ , assuming that K has a non-empty interior. This is because by Proposition 4.1, any fixed moment function induced by a PwL MLP will not be able to separate all multisets, whereas any two distinct multisets can be separated by some continuous f . Though, with a PwL σ , one may approximate any given f to

arbitrary precision, by taking the embedding dimension m to infinity. In contrast, with an analytic σ , we are able to specify a finite m = 2 nd +1 for which exact equality in (9) is guaranteed.

Since our injectivity results on multisets extend to measures, it is natural to seek an extension of the above approximation result to functions defined on measures. Denote by P ≤ n ( K ) the space of probability measures supported on ≤ n points in K ⊆ R d , endowed with the 2-Wasserstein metric.

Corollary 6.2. Let n, d ∈ N and set m = 2 n ( d + 1) + 1 . Let σ : R → R be analytic and nonpolynomial. Let K ⊆ R d be compact. Then there exist A ∈ R m × d , b ∈ R m such that for any continuous (in the 2 -Wasserstein sense) f : P ≤ n ( K ) → R , there exists a continuous F : R m → R such that

It follows from Corollary 6.2 that any continuous function f : P ≤ n ( K ) → R with compact K ⊆ R d can be approximated to arbitrary precision by functions of the form ˆ f ( µ ) = F (∫ x ∈ K σ ( Ax + b ) dµ ( x ) ) , with A ∈ R m × d , b ∈ R m , m = 2 nd +1 , and F being an MLP.

6.2 Graph separation

We now discuss the implications of Theorem 3.3 for graph separation, using terminology from [44]. Let G ≤ n (Σ) be the collection of all graphs G = ( V, E, h (0) ) with at most n vertices, endowed with vertex features h (0) v ∈ Σ , where Σ ⊆ R d is a countable alphabet. We consider GIN-like [44] MPNNs that recursively, for t = 1 , . . . , T , calculate node features h ( t ) v from the previous features h ( t -1) v by

After the T iterations are concluded, a global feature is computed via a readout function :

We choose all features h G and h ( t ) v for 1 ≤ t ≤ T , to have the same dimension m . Based on the fact that MPNNs are equivalent to 1 -WL when the multiset functions are injective [44], and on our injectivity results for countable alphabets, we prove that:

Theorem 6.3. Let n, d, T ∈ N and let Σ ⊆ R d be countable. Let m ≥ 1 be any integer. Let σ : R → R be an analytic non-polynomial function. Then for Lebesgue almost any choice of A ( t ) , b ( t ) and η ( t ) , the MPNN defined in (10) and (11) assigns different global features to any pair of graphs G 1 , G 2 ∈ G ≤ n (Σ) that can be separated by T iterations of 1-WL.

Graph separation with continuous features Up to now, we have discussed graphs with node features coming from a countable alphabet. Since Theorem 3.3 applies to multisets with continuous alphabets, it can be applied to separation of graphs with continuous node features as well. In particular, the paper [15] explains how the random semialgebraic multiset function from [9] can be used to construct architectures for graphs with continuous features, whose separation power is equivalent to WL tests. Their focus is on showing that the embedding dimension in their construction depends linearly on the dimension of the graph space , rather than grows exponentially with the number of message-passing iterations T . Similar results can now be obtained by using our random analytic multiset functions. We leave a full description of these aspects to future work.

7 Experiments

Empirical injectivity and bi-Lipschitzness We empirically investigated the injectivity and biLipschitzness of moments of shallow networks of the form (3), by randomly generating a large number of pairs of multisets of n vectors in R d , and computing the optimal constants c, C for which (8) holds for the generated pairs. The ratio c/C for varying activations and embedding dimension m is shown in Figure 2. Here d = 3 and n = 1000 . Similar qualitative results were obtained for other values of d and n ; see Figure 4 in Appendix E.

We observe several interesting phenomena: First, at low embedding-dimensions, the ratio c/C for PwL networks is exactly zero, indicating that they are not injective even on the finite sample set. In contrast, for analytic activations, c/C is always positive. Indeed, we expect analytic activations to be injective on a finite sample even with embedding dimension m = 1 , since a finite sample set has an intrinsic dimension of zero. Next, we observe that c/C naturally improves as m increases. Finally, we note that even for high m , c/C is rather small.

Indeed, if it were possible to consider all pairs of multisets when computing c/C , we would get zero for all activations and all embedding dimensions, as follows from Theorem 5.1. For additional details on this experiment, see Appendix E.1.

Graph Separation To validate Theorem 6.3, we conducted the following experiment: we considered 600 graphs from the TUDataset [30]. On each graph we ran three iterations of the WL test, and three iterations of MPNNs with the Graph Convolutional Layers from [18] with different activations and hidden dimensions. Our goal was to check in how many graphs the MPNNs returned a vertex coloring that differs from the coloring provided by the WL test. The results are shown in Figure 1(a). As seen in the table,

1

10

100

1000

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

10

-

3

10

-

2

0

Embedding dimension

c/C

Cos

TanH

Sigmoid

HardTanH

ReLU

with the three analytic activations tested, the vertex coloring of MPNN was always equivalent to 1-WL, even with a hidden dimension of 1. On the other hand, for the three PwL activations, there were inconsistencies in about 1% of the graphs, even with a hidden dimension of 50.

We note that while analytic activations fully succeeded in separation, in some cases the separation was rather weak: while the distance between features of non-equivalent nodes computed by the MPNNs was typically around 0 . 1 , the least-separated features had a distance of ∼ 10 -7 . In future work, it could be interesting to investigate whether MPNNs can be trained to yield larger distances between the features of all non-equivalent nodes. Further details on this experiment appear in Appendix E.2.

8 Conclusion

We have shown that moments of neural networks with an analytic non-polynomial activation are injective on multisets and measures. We have also shown how this can be harnessed to construct universal approximators for multiset functions, as well as prove separation results for graph neural networks. A key advantage of our approach is that it enables constructing proofs using real models that are used in practice, rather than idealized versions of them as done in previous works.

It may seem tempting, due to our theoretical results, to conclude that analytic activations should perform on multisets better than piecewise-linear activations. We stress that we make no such claim. Indeed, while the separation results in Figure 1(a) corroborate our theory, PwL networks fail to separate only 1% of the graphs in our experiment. Furthermore, at high embedding dimensions, the empirical bi-Lipschitzness in Figure 2 does not seem to strongly depend on the analyticity of the activation function. Our claim is thus much more modest: we claim that multiset architectures with analytic activations are easier to theoretically analyze, and we hope that pursuing this analysis shall lead to fruitful theoretical and practical insights, which may ultimately benefit multiset architectures with either type of activation.

Lastly, we note that the finite witness theorem, which is presented here as a tool for proving moment injectivity, may prove valuable as a general tool for reducing an infinite number of equality constraints to a finite number, and we believe it will find additional applications beyond the scope of this paper.

Acknowledgements N.D. is partially funded by a Horev Fellowship. T.A, R.R. and N.D. are partially funded by ISF grant 272/23.

References

Anders Aamand et al. 'Exponentially Improving the Complexity of Simulating the WeisfeilerLehman Test with Graph Neural Networks'. In: Advances in Neural Information Processing Systems 35 (2022), pp. 27333-27346.

Radu Balan, Pete Casazza, and Dan Edidin. 'On signal reconstruction without phase'. In: Applied and Computational Harmonic Analysis 20.3 (2006), pp. 345-356.

Radu Balan, Naveed Haghani, and Maneesh Singh. 'Permutation invariant representations with applications to graph deep learning'. In: arXiv preprint arXiv:2203.07546 (2022).

Jameson Cahill, Joseph W Iverson, and Dustin G Mixon. 'Bilipschitz group invariants'. In: arXiv preprint arXiv:2305.17241 (2023).

Jameson Cahill et al. 'Group-invariant max filtering'. In: arXiv preprint arXiv:2205.14039 (2022).

Gabriele Corso et al. 'Principal neighbourhood aggregation for graph nets'. In: Advances in Neural Information Processing Systems 33 (2020), pp. 13260-13271.

George Cybenko. 'Approximation by superpositions of a sigmoidal function'. In: Mathematics of control, signals and systems 2.4 (1989), pp. 303-314.

Valentino Delle Rose et al. 'Three iterations of ( d -1) -WL test distinguish non isometric clouds of d -dimensional points'. In: arXiv e-prints (2023), arXiv-2303.

Nadav Dym and Steven J Gortler. 'Low Dimensional Invariant Embeddings for Universal Geometric Learning'. In: arXiv preprint arXiv:2205.02956 (2022).

Matthias Fey and Jan E. Lenssen. 'Fast Graph Representation Learning with PyTorch Geometric'. In: ICLR Workshop on Representation Learning on Graphs and Manifolds . 2019.

Jean Feydy et al. 'Interpolating between Optimal Transport and MMD using Sinkhorn Divergences'. In: The 22nd International Conference on Artificial Intelligence and Statistics . 2019, pp. 2681-2690.

Hillel Furstenberg. 'Ergodic fractal measures and dimension conservation'. In: Ergodic Theory and Dynamical Systems 28.2 (2008), pp. 405-422.

Allen Hatcher. Algebraic Toplogy . Cambridge University Press, 2002.

Michael Hochman. 'Lectures on fractal geometry and dynamics'. In: Unpublished lecture notes, Available online at http://math. huji. ac. il/mhochman/courses/fractals-2012/coursenotes. june-26. pdf (2012).

Snir Hordan et al. 'Complete Neural Networks for Euclidean Graphs'. In: arXiv preprint arXiv:2301.13821 (2023).

Chaitanya K Joshi et al. 'On the expressive power of geometric graph neural networks'. In: arXiv preprint arXiv:2301.09308 (2023).

Sammy Khalife and Amitabh Basu. 'On the power of graph neural networks and the role of the activation function'. In: arXiv preprint arXiv:2307.04661 (2023).

Thomas N. Kipf and Max Welling. 'Semi-Supervised Classification with Graph Convolutional Networks'. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. URL: https://openreview.net/forum?id=SJU4ayYgl .

Steven G Krantz and Harold R Parks. A primer of real analytic functions . Springer Science & Business Media, 2002.

M. G. Krein and A. A. Nudelman. The Markov moment problem and extremal problems . Translations of mathematical monographs. American Mathematical Society, 1977.

John M Lee. 'Smooth manifolds, Introduction to smooth manifolds'. In: Graduate Texts in Mathematics 218 (2013).

John M Lee and John M Lee. Introduction to Smooth manifolds . Springer, 2003.

Juho Lee et al. 'Set transformer: A framework for attention-based permutation-invariant neural networks'. In: International conference on machine learning . PMLR. 2019, pp. 3744-3753.

Moshe Leshno et al. 'Multilayer feedforward networks with a nonpolynomial activation function can approximate any function'. In: Neural networks 6.6 (1993), pp. 861-867.

Zian Li et al. 'Is Distance Matrix Enough for Geometric Deep Learning?' In: arXiv preprint arXiv:2302.05743 (2023).

Haggai Maron et al. 'Provably powerful graph networks'. In: Advances in neural information processing systems 32 (2019).

Chris Miller. 'Tameness in expansions of the real field'. In: Logic Colloquium . Vol. 1. 2005, pp. 281-316.

Boris Samuilovich Mityagin. 'The zero set of a real analytic function'. In: Mathematical Notes 107.3-4 (2020), pp. 529-530.

Guido F Montufar et al. 'On the number of linear regions of deep neural networks'. In: Advances in neural information processing systems 27 (2014).

Christopher Morris et al. 'TUDataset: A collection of benchmark datasets for learning with graphs'. In: ICML 2020 Workshop on Graph Representation Learning and Beyond (GRL+ 2020) . 2020. arXiv: 2007.08663 . URL: www.graphlearning.io .

Christopher Morris et al. 'Weisfeiler and leman go neural: Higher-order graph neural networks'. In: Proceedings of the AAAI conference on artificial intelligence . Vol. 33. 01. 2019, pp. 46024609.

Allan Pinkus. 'Approximation theory of the MLP model in neural networks'. In: Acta numerica 8 (1999), pp. 143-195.

Sergey N Pozdnyakov and Michele Ceriotti. 'Incompleteness of graph neural networks for points clouds in three dimensions'. In: Machine Learning: Science and Technology 3.4 (2022), p. 045020.

Charles R Qi et al. 'Pointnet: Deep learning on point sets for 3d classification and segmentation'. In: Proceedings of the IEEE conference on computer vision and pattern recognition . 2017, pp. 652-660.

Maithra Raghu et al. 'On the Expressive Power of Deep Neural Networks'. In: arxiv preprint arXiv:1606.05336 (2017).

Walter Rudin. Real and Complex Analysis . McGraw-Hill Science/Engineering/Math, 1986.

Nimrod Segol and Yaron Lipman. 'On Universal Equivariant Set Networks'. In: International Conference on Learning Representations . 2019.

Guillaume Valette. On subanalytic geometry . June 2023.

Lou Van den Dries and Chris Miller. 'Geometric categories and o-minimal structures'. In: (1996).

Cédric Villani et al. Optimal transport: old and new . Vol. 338. Springer, 2009.

Edward Wagstaff et al. 'On the limitations of representing functions on sets'. In: International Conference on Machine Learning . PMLR. 2019, pp. 6487-6494.

Edward Wagstaff et al. 'Universal approximation of functions on sets'. In: Journal of Machine Learning Research 23.151 (2022), pp. 1-56.

Peihao Wang et al. 'Polynomial Width is Sufficient for Set Representation with Highdimensional Features'. In: arXiv preprint arXiv:2307.04001 (2023).

Keyulu Xu et al. 'How Powerful are Graph Neural Networks?' In: International Conference on Learning Representations . 2018.

Manzil Zaheer et al. 'Deep sets'. In: Advances in neural information processing systems 30 (2017).

Yan Zhang, Jonathon Hare, and Adam Prügel-Bennett. 'FSPool: Learning Set Representations with Featurewise Sort Pooling'. In: International Conference on Learning Representations .

A Finite witness theorem

In this section, we state and prove the full version of the finite witness theorem (Theorem A.2), which is more general than Theorem 3.4 stated in the main text. Before laying out the formal definitions and proofs, we begin by describing the context of these results. While this section makes use of notions from algebraic geometry and real analytical functions, it is self-contained and most of it only requires knowledge of elementary calculus and some topology.

The finite witness theorem is essentially a tool for reducing an infinite, continuously parameterized family of constraints p ( z ; θ ) = 0 ∀ θ to a finite subset of constraints p ( z ; θ i ) = 0 , i = 1 , . . . , m , defined by random parameters θ 1 , . . . , θ m . This general approach seems to have originated from the famous proof of uniqueness for phase-retrieval measurements in [2]. In that work, functions p ( z , θ ) : C n × C n → R of the form p ( z ; θ ) = |〈 z , θ 〉| were considered, and it was proved that a finite number of ∼ 4 n random measurements θ i are sufficient to uniquely determine a signal, up to unavoidable global phase ambiguity. The proof in [2] achieves this result by showing that the values p ( z ; θ ) for all possible θ s are sufficient to determine the signal uniquely, and then uses a real algebraic-geometric and dimension-counting argument to show that this continuous family of measurements can be replaced by a finite subset, defined by m ∼ 4 n random vectors ( witnesses ) { θ i } i m =1 , without losing information.

In [9], the authors provide a generalization of the results in [2], by defining conditions under which a continuously parameterized family of functions p ( z ; θ ) that fully determines z up to equivalence, can be replaced by a finite subset p ( z ; θ i ) , i = 1 , . . . , m determined by random witness-vectors θ i . This theorem is based on similar arguments as those used in [2], and on similar assumptions required for machinery from real algebraic geometry. Specifically, sets in [9] are assumed to be semialgebraic , which means that they are finite unions of subsets of R D that can be defined by polynomial equalities and inequalities. This class of sets includes, for example, finite unions of spheres, balls, and convex polyhedra. The functions in the theorem are assumed to be semialgebraic as well, which means that their graphs are semialgebraic sets. This class of functions includes polynomials, as well as rational functions and piecewise-linear functions.

The main theorem in [9] can be essentially 3 formulated as

Theorem A.1. Let M be a semialgebraic set of dimension D , and let F : M × R D θ → R be a semialgebraic function. Define the set

and assume that for all z ∈ M \ N , we have that

then for Lebesgue almost every θ (1) , . . . , θ ( D +1) ,

The notion of dimension used in condition (12) and throughout this section is the Hausdorff dimension , explained in Appendix A.4 below.

Theorem A.1 is similar to the simple version of the finite witness theorem (Theorem 3.4), with four notable differences: (1) The domain M in Theorem 3.4 is a countable union of affine sets; this is not, in general, a semialgebraic set, and thus does not qualify for the conditions of Theorem A.1. (2) The function F in Theorem 3.4 is analytic. Analytic functions are not necessarily semialgebraic. (3) Theorem A.1 requires the extra condition (12), which does not appear in Theorem 3.4. This condition is not required in Theorem 3.4 because with analytic functions, it is always satisfied; however, it will be required in our full version of the theorem, since it admits a more general class of functions. (4) Theorem A.1 deals with sets of the form { z | F ( z , θ ) = 0 , ∀ θ } while Theorem 3.4 deals with sets of the form { ( x , y ) | F ( x , θ ) = F ( y , θ ) , ∀ θ } . This difference is not essential and can be handled by the change of variables z = ( x , y ) and ˜ F ( x , y , θ ) = F ( x , θ ) -F ( y , θ ) .

To address the first two differences, we shall generalize Theorem A.1 to support a large class of domains M and functions F . The admissible domains shall include a vast class of sets, among which are countable unions of semialgebraic sets, as well as all open sets, and sets defined by analytic, rather than polynomial, equations. The admissible functions F shall include semialgebraic functions, analytic functions, and many other types of functions. To achieve this goal, we use results from the study of o-minimal structures (see below), which aim at finding families of sets that have the same tame properties as semialgebraic sets.

A good starting point to achieve this generalization is to consider the family of globally subanalytic sets (formally defined below). This family contains all semialgebraic sets and is an o-minimal system; consequently, it is possible to generalize Theorem A.1 so that the domain M can include all globally subanalytic sets, and the function F could be any globally subanalytic function (meaning that the graph of F is a globally subanalytic set). However, this still will not allow for a general analytic F , nor for M to include countable unions of affine sets.

To address this, we will show that an analogue to Theorem A.1 holds even when considering countable unions of globally subanalytic sets. We call such sets σ -subanalytic sets . To the best of our knowledge, such sets have not been studied to date. The family of σ -subanalytic sets is not an o-minimal structure, since it not closed under taking complements. However, it is closed under linear projections, countable unions, finite intersections and Cartesian products. Moreover, σ -subanalytic sets are countable unions of C ∞ manifolds. We find that these properties are sufficient to generalize Theorem A.1 and obtain a finite witness theorem for the σ -subanalytic category.

Clearly, the class of σ -subanalytic sets is larger than the class of globally subanalytic sets. In particular, it includes countable unions of semialgebraic sets. This enables the domain M in the theorem statement to be a countable union of affine spaces, as in Theorem 3.4 from the main text. This also implies that any open set is σ -subanalytic, since it is a countable union of open balls which are semialgebraic sets.

As for the function F , our theorem admits all σ -subanalytic functions: functions whose graph is σ -subanalytic. This class includes all analytic functions, as well as all semialgebraic functions. Moreover, we show below that it is closed under composition and other elementary operations.

We now state the full version of the finite witness theorem.

Theorem A.2 (Finite Witness Theorem, full version) . Let M ⊆ R p , W ⊆ R q be σ -subanalytic sets of dimension D and D θ respectively. Let F : M × W → R be a σ -subanalytic function. Define the set

Suppose that for all z ∈ M \ N

Then for generic ( θ (1) , . . . , θ ( D +1) ) ∈ W D +1 ,

Moreover, if W is an open and connected subset of R q , and F ( z ; θ ) is analytic as a function of θ for all fixed z ∈ M , then condition (13) is not required, as it is automatically satisfied.

The notion of dimension in (13) is the Hausdorff dimension (discussed below), and the term generic means that the set of ( θ (1) , . . . , θ ( D +1) ) for which (14) fails is a subset of W D +1 whose Hausdorff dimension is strictly lower than dim ( W D +1 ) . In particular, in the common case where W = R q , or is an open subset of R q , we have that the theorem holds for almost any ( θ (1) , . . . , θ ( D +1) ) , with respect to the Lebesgue measure on R q ( D +1) .

Wenowbegin to rigorously define the notions discussed in this section and then prove Theorem A.2. In Appendices A.1 to A.4, we construct the theoretical framework step by step. Then, in Appendix A.5 we prove the theorem, and in Appendix A.6 we show that the simple version of the theorem (Theorem 3.4) follows from the full version. Finally, we present in Appendix A.6 several corollaries to the theorem.

A.1 Definitions and Background

A.1.1 Globally subanalytic sets: an o-minimal structure

We begin by defining analytic functions, subanalytic sets and functions, and various other related concepts that will be required for our discussion. These definitions are taken from [19] and [38].

Definition A.3 (Analytic function [19]) . Let U ⊆ R D be an open set. We say that f : U → R is analytic if for all z ∈ U , there exists an open ball V ⊆ U centered at z , and ( a α ) α ∈ N D such that

and the power series in (15) converges absolutely.

Our next goal is to define the family of globally subanalytic sets, which contains all semialgebraic sets; moreover, it is an o-minimal structure , meaning that it shares the essential tame properties of semialgebraic sets. This requires some preliminary definitions:

Definition A.4 (Semianalytic sets [38]) . A subset E ⊆ R n is called semianalytic if it is locally defined by finitely many real analytic equalities and inequalities. Namely, for each a ∈ R n , there is a neighborhood U of a , and real analytic functions f ij , g ij on U , where i = 1 , . . . , r and j = 1 . . . s i , such that

Example A.5. As shown in Example 1.1.2 in [38], the graph of the analytic function f : (0 , 1) → R defined by f ( x ) = sin(1 /x ) is not a semianalytic set. This is because there is no neighborhood of a = (0 , 0) for which (16) holds. On the other hand, if B ⊆ R n is a closed ball and f : B → R can be extended to an analytic function in an open set containing B , then the graph of f (as a function defined on B ) will be semianalytic.

The example above suggests that the class of semianalytic sets may be too restrictive for our purposes. More importantly, linear projections of semianalytic sets may not be semianalytic [38], so the family of semianalytic sets does not form an o-minimal system. This can be remedied by defining globally semianalytic sets (which don't form an o-minimal system) and then using these sets to define globally subanalytic sets (which do form an o-minimal system):

Definition A.6 (Globally semianalytic sets [38]) . Asubset Z ⊆ R n is globally semianalytic if V n ( Z ) is a semianalytic subset of R n , where V n : R n → ( -1 , 1) n is the homeomorphism defined by

Globally semianalytic sets can be thought of as semianalytic sets that remain semianalytic when 'compactified' by V n . This is useful to rule out bad behaviour as z goes out to infinity. Important examples of globally semianalytic sets include all bounded semianalytic sets and all semialgebraic sets [38].

Globally semianalytic sets are still not closed under linear projection. Their projections are called globally subanalytic sets:

Definition A.7 (Globally subanalytic sets [38]) . A subset E ⊆ R n is globally subanalytic if it can be presented as a linear projection of a globally semianalytic set; more precisely, if there exists a globally semianalytic set Z ⊆ R n + p such that E = π ( Z ) , with π : R n + p → R n being the projection operator that omits the last p coordinates while leaving the remaining coordinates unchanged.

Globally semianalytic sets are by definition globally subanalytic. Globally subanalytic sets do form an o-minimal structure . In particular, they have the following properties:

Proposition A.8 (Properties of globally subanalytic sets [38]) . Let A,B ⊆ R D and C ⊆ R M be globally subanalytic sets. Then:

1. R D \ A is globally subanalytic.

A ∪ B is globally subanalytic.

A ∩ B is globally subanalytic.

A × C is globally subanalytic.

If π : R D → R L is a linear projection, then π ( A ) is globally subanalytic.

A is a finite union of C ∞ manifolds.

The first property in Proposition A.8 is proven in Gabrielov's Complement Theorem; see Theorem 1.8.8 in [38]. The last property is a weaker version of Theorem 1.2.3 in [38]. The remaining properties are proved 4 in [38, Basic properties 1.1.8].

A.2 σ -subanalytic sets

As mentioned above, the fact that globally subanalytic sets form an o-minimal structure is already sufficient to prove a finite witness theorem for this class. However, this class is still too restrictive for our purposes, as it does not allow us to support arbitrary analytic functions F , as the following example shows:

Example A.9. Analogously to semialgebraic functions, we say that a function is globally subanalytic if its graph is globally subanalytic. Consider the function sin( x ) defined on the real line. Then by Example 1.1.7 in [38], this function is not globally subanalytic.

Fortunately, we find that a finite witness theorem can be proven for a larger class of sets: countable unions of globally subanalytic sets. We name such sets σ -subanalytic sets. To the best of our knowledge, this class of sets has not been studied to date 5 . The class of σ -subanalytic sets is rather large: we will show below that it contains all open sets, while the classes discussed previously do not (see Example A.12 below). This is illustrated in the Venn diagram on the left-hand side of Figure 3. Correspondingly, the class of σ -subanalytic functions is larger than the classes of functions considered previously, and in particular it contains any analytic function F defined on any open set. This is illustrated in the Venn diagram on the right-hand side of Figure 3, and will be discussed rigorously in Appendix A.3 below. Thus, moving to σ -subanalytic sets and functions is the crucial step that enables us to achieve our goal of proving a finite witness theorem for analytic functions (though the theorem covers a much larger class).

We now define σ -subanalytic sets and study their properties.

Definition A.10 ( σ -subanalytic sets) . We say that a subset A ⊆ R D is σ -subanalytic if it is a countable union of globally subanalytic subsets of R D .

The class of σ -subanalytic sets is rather large. For example, any open set in R D can be written as a countable union of open balls. Since open balls are semialgebraic sets, it follows that all open sets are σ -subanalytic. Also, any semianalytic set is σ -subanalytic, since semianalytic sets are a countable union of bounded semianalytic sets, and these are globally subanalytic.

Properties of σ -subanalytic sets

As the following proposition shows, σ -subanalytic sets inherit most of the properties enjoyed by globally subanalytic sets, described in Proposition A.8.

Proposition A.11 (Properties of σ -subanalytic sets) . Assume that A,B ⊆ R D and C ⊆ R M are σ -subanalytic sets, then

A ∪ B is σ -subanalytic. More generally, any countable union of σ -subanalytic sets is σ -subanalytic.

globally subanalytic

functions

𝜎𝜎 -

subanalytic

functions

semialgebraic

functions

semialgebraic

sets

globally subanalytic

sets

𝜎𝜎 -

subanalytic sets

A ∩ B is σ -subanalytic.

A × C is σ -subanalytic.

If π : R D → R L is a linear projection, then π ( A ) is a σ -subanalytic set.

A is a countable union of C ∞ manifolds.

Proof. Let A,B ⊆ R D be σ -subanalytic sets. Then A = ∪ n ∈ N A n and B = ∪ m ∈ N B m , where each A n and B m is globally subanalytic. We then have

Acountable union of σ -analytic sets, each of which being itself a countable union of globally subanalytic set, is clearly a countable union of globally subanalytic sets, and therefore is σ -subanalytic by definition.

We have that

and A n ∩ B m is globally subanalytic as the intersection of globally subanalytic sets.

The set A × R M can be presented as

with A m being globally subanalytic. Since globally subanalytic sets are closed to Cartesian products, each A n × R M is globally subanalytic, and their countable union is σ -subanalytic. Finally,

which is σ -subanalytic by part 2.

Follows from globally subanalytic sets being closed to projections, since

A σ -subanalytic set is a countable union of globally subanalytic sets, each of which is a finite union of C ∞ manifolds. Therefore, it is a countable union of C ∞ manifolds.

In comparison to Proposition A.8, we see that we have lost two properties by moving from globally subanalytic sets to countable unions thereof: Firstly, a σ -subanalytic set is a countable union of manifolds rather than a finite one. Secondly, the complement of a σ -subanalytic set may not be σ -subanalytic, as our next example shows. As it turns out, the properties noted in Proposition A.11 are sufficient for our purposes.

Example A.12. The Cantor set is not σ -subanalytic, since it has fractal Hausdorff dimension and thus is not a countable union of manifolds. However, the complement of the Cantor set, which we denote by U , is an open set and hence is σ -subanalytic. This shows that the class of σ -subanalytic sets is not closed to taking complements, as well as countable intersections: this is since the Cantor set is a countable intersection of finite unions of intervals, which are σ -subanalytic. The set U is also an example of an open set that is not semialgebraic or globally subanalytic, since these classes of sets are closed to complements, and do not contain the Cantor set.

A.3 σ -subanalytic functions

Wenowdefine the class of functions our theorem can admit as the function F : σ -subanalytic functions.

Definition A.13 ( σ -subanalytic function) . Let M ⊆ R D be a σ -subanalytic set. We say that f : M → R L is a σ -subanalytic function if its graph is a σ -subanalytic subset of R D + L .

If M ⊆ R D is a semialgebraic set and f : M → R L is a semialgebraic function, then the graph of f is semialgebraic, and thus is σ -subanalytic. Therefore, a semialgebraic function f defined on a semialgebraic domain M is a σ -subanalytic function. A similar result holds also for analytic functions defined on open sets, as shown in the following lemma:

Lemma A.14. If U ⊆ R D is open and f : U → R L is analytic, then f is σ -subanalytic.

Proof. First recall that since U is open, it is a σ -subanalytic set. Next, we can write U as a countable union of closed balls B i , and then the graph of f can be written as a countable union of sets of the form

These subsets of U × R L are compact and semianalytic, and hence they are globally subanalytic.

Properties of σ -subanalytic functions

The following proposition shows that the class of σ -subanalytic functions is closed under composition and elementary arithmetic operations. In particular, this allows for functions that combine analytic and semialgebraic functions via compositions and elementary arithmetic operations.

Proposition A.15. Let A ⊆ R a and B ⊆ R b be σ -subanalytic sets.

(Composition) If f : A → B and g : B → R c are σ -subanalytic functions, then g ◦ f is a σ -subanalytic function.

(Concatenation) If f : A → R m and g : A → R n are σ -subanalytic functions, then h : A → R m + n given by h ( x ) = ( f ( x ) , g ( x )) is a σ -subanalytic function.

(Addition and multiplication) If f, g : A → R are σ -subanalytic functions, then f + g and f · g are σ -subanalytic functions.

Proof.

To prove that g ◦ f is σ -subanalytic, we need to prove that its graph is a σ -subanalytic set. Indeed, this graph is of the form

which is the projection of the intersection

onto the ( x , z ) coordinates. The two sets in the intersection above are Cartesian products of the graph of f (respectively g ) with a σ -subanalytic set, and hence are σ -subanalytic.

The graph of h is the intersection of the sets { ( x , y , z ) | f ( x ) = y } and { ( x , y , z ) | g ( x ) = z } . Each of these two sets is a Cartesian product of a Euclidean space with the graph of a σ -subanalytic function.

The functions f + g and f · g can be presented as the composition of the function R a glyph[owner] x ↦→ ( f ( x ) , g ( x )) , which is σ -subanalytic, with the addition or multiplications functions. Thus, the claim follows.

A.4 Dimension

The proof of Theorem A.2 is based on a dimension-counting argument. Accordingly, we will need an appropriate definition of dimension for σ -subanalytic sets. It is convenient to work with the Hausdorff dimension, which is defined for every subset of a Euclidean space R D , and coincides with the standard notions of dimension for vector spaces and manifolds. The definition of Hausdorff dimension can be found, e.g., in [14]. For our purposes, we will only need to use some of its properties, stated below.

Proposition A.16 (Taken from [14]) . The Hausdorff dimension has the following properties:

The Hausdorff dimension of a k -dimensional C 1 submanifold of R D is k .

If B 1 , B 2 , . . . are subsets of R D then

For any B ⊆ R D and Lipschitz function f : B → R C , dim( f ( B )) ≤ dim( B ) .

Recall that σ -subanalytic sets are countable unions A = ∪ n ∈ N A n of C ∞ manifolds. As we saw, their Hausdorff dimension is just the maximal dimension of all manifolds A n . We shall now see that for such sets, the Hausdorff dimension has two nice properties that do not hold for general sets. These properties will be used in the proof of the finite witness theorem.

The first property is the dimension of Cartesian products. For two general sets A,B , it is not always true that dim( A × B ) = dim( A ) + dim( B ) (see [14]). However, this does hold if A and B are countable unions of manifolds.

Lemma A.17. If A,B are countable unions of C 1 sub-manifolds of R a and R b , then

Proof. By assumption A = ∪ n ∈ N A n and B = ∪ m ∈ N B m , where A n , B m are C 1 manifolds. We have that

is again a countable union of the product manifolds A n × B m , which are known to be of dimension dim( A n ) + dim( B m ) (see e.g. [22]). It follows that the dimension of A × B is equal to

The second nice property of the Hausdorff dimension for countable unions of manifolds, which does not always hold for general sets, is dimension conservation , in the following sense.

Lemma A.18 (Dimension Conservation) . Let S ⊆ R D 1 be a countable union of C ∞ manifolds and f : R D 1 → R D 2 a C ∞ function. Then

For failure of dimension conservation for general sets see, e.g., the discussion in [12].

A good intuition for the concept of dimension conservation comes from the linear case, in which f : R D 1 → R D 2 is a linear transformation and S ⊆ R D 1 is a linear subspace. In this case, the Rank-Nullity Theorem from elementary linear algebra states that

The same statement also holds if we replace f -1 (0) by f -1 ( t ) , with t being any point in the image of f . Thus, for linear transformations, any loss of dimension when moving from a set S to its image, is accounted for by the dimension of the fibers above points in the image. The image conservation in Lemma A.18 is a weaker since it is only an inequality, but it applies to the more general class of σ -semianalytic functions.

Proof of Lemma A.18. We know that S is a countable union of manifolds, and at least one of these, which we denote by S 1 , has the maximal dimension dim( S 1 ) = dim( S ) . This equality shall enable us to argue about dim( S ) by arguing only about dim( S 1 ) .

Let r be the maximal rank of the differential of f | S 1 . Fix some s 1 ∈ S 1 so that the differential of f | S 1 at s 1 has rank r . The set of s ∈ S 1 whose differential has rank r is open, and so there is a neighborhood V of s 1 , such that V ∩ S 1 is a manifold of the same dimension as S 1 , and the restriction of f to V ∩ S 1 has constant rank r . Consider the restriction of f to V ∩ S 1 , denoted by f | V ∩ S 1 . By the constant rank theorem [21], f | V ∩ S 1 is a projection, up to a diffeomorphic change of coordinates. Since diffeomorphisms of manifolds preserve dimensions, and projections (like all linear transformations) satisfy dimension conservation as in (18), this implies that locally we have dimension conservation: for all t ∈ f ( V ∩ S 1 ) ,

Therefore:

We conclude our discussion of the Hausdorff dimension with a natural corollary which will be useful later on for our proof of Proposition 3.6.

Corollary A.19. Let M ⊆ R D be a σ -subanalytic set, and let F : M → R L a σ -subanalytic function. Then the graph and image of F are σ -subanalytic sets, and

Proof. Note that by the definition of σ -subanalytic functions, the graph of f is a σ -subanalytic set. We begin by showing that it has the same dimension as M . Denote by π the projection from

onto the first coordinate. Firstly, since π is a Lipschitz function, it cannot increase the Hausdorff dimension, so

In the other direction, note that for every ( x , F ( x )) ∈ graph( F ) we have that

is a set containing a single point, and thus has dimension zero. Therefore, using Lemma A.18 we have that

and so we have shown that M and the graph of F have the same dimension. Finally, F ( M ) is the projection of the graph of F onto the second coordinate and so it is a σ -subanalytic set. Moreover, as projections cannot increase the Hausdorff dimension, we obtain that

which concludes the proof of the corollary.

A.5 Proof of the finite witness theorem

We are finally ready to prove the full version of the finite witness theorem.

Proof of Theorem A.2. Let m = D +1 . Let F : M × W m +1 → R m +1 be given by

Since F m is a concatenation of σ -subanalytic functions (each such function is the composition of F with a different linear projection), by Proposition A.15 it is also σ -subanalytic. Therefore, the graph of F m , given by A m below, is σ -subanalytic:

glyph[negationslash]

Next, we can intersect A m with the semialgebraic set defined by the equations s 0 = 0 , s 1 = s 2 = . . . = s m = 0 to obtain the σ -subanalytic set glyph[negationslash]

We can then remove by projection the last m +1 coordinates and the θ 0 coordinate, to obtain the set

which is σ -subanalytic as well. Let π and π θ denote the projections

The set π θ ( B m ) ⊆ W m is exactly the set of m -tuples ( θ 1 , . . . , θ m ) that are not sufficient to determine N . To prove the theorem, we thus need to show that the dimension of π θ ( B m ) is lower than dim( W m ) = mD θ . We do so by bounding the dimension of B m .

Let z 0 ∈ π ( B m ) . The set π -1 ( z 0 ) ∩ B m can be presented as the Cartesian product

The set Θ z 0 is also σ -subanalytic: the graph of F

is σ -subanalytic by assumption; intersecting it with the space z = z 0 , s = 0 , and projecting to the θ coordinate, yields the set Θ z 0 . Since Θ z 0 is σ -subanalytic, it is a countable union of manifolds. By the theorem assumption (13), the Hausdorff dimension of Θ z 0 satisfies

Therefore by Lemma A.17 and Equation (19),

So far, z 0 is only assumed to be an arbitrary point in π ( B m ) . Now, fix z 0 ∈ π ( B m ) to be a maximizer of the left-hand side of Equation (20). By the dimension conservation Lemma A.18,

where

where (a) holds by the choice of z 0 , (b) is by Equation (20), and (c) holds since π ( B m ) ⊆ M and dim( M ) = D . Since π θ is Lipschitz, Equation (21) implies that

glyph[negationslash]

as required.

Now, suppose that W is an open and connected subset of R q . We need to show that (13) holds for all z ∈ M \ N . Fix z ∈ M \ N and let f : W → R be given by f ( θ ) = F ( z ; θ ) . Since z glyph[negationslash]∈ N , there exists some θ for which F ( z ; θ ) = 0 . Hence, f is an analytic function defined on an open connected domain, and is not identically zero. Therefore, its zero-set must be dimension-deficient (see [28, Proposition 3]) and so we obtain (13).

This concludes the proof of the theorem.

A.6 Corollaries

In this subsection, we present several corollaries to the finite witness theorem, and show that the simple version Theorem 3.4 follows from it as a special case.

Auseful application of the finite witness theorem, which often arises in invariant learning, is when one wishes to assert that two points x , y ∈ M are indistinguishable by a parametric family of functions F ( · ; θ ) ; namely, that F ( x ; θ ) = F ( y ; θ ) for all θ ∈ W . As the following corollary shows, this can be asserted almost surely by testing whether F ( x ; θ ) = F ( y ; θ ) on 2 dim ( M ) + 1 random θ s.

Corollary A.20 (Separation by Finite Witnesses) . Let M ⊆ R p , W ⊆ R q be σ -subanalytic sets of dimension D and D θ respectively. Let F : M × W → R be a σ -subanalytic function. Define the set

Suppose that for all ( x , y ) ∈ M × M \ N

Then for generic ( θ (1) , . . . , θ (2 D +1) ) ∈ W D +1 ,

Moreover, if W is an open and connected subset of R q , and F ( x ; θ ) is analytic as a function of θ for all fixed x ∈ M , then condition (22) is not required, as it is automatically satisfied.

Proof. Set ˜ M = M × M , and define G : ˜ M × W → R by

Then ˜ M is a σ -subanalytic set of dimension dim( ˜ M ) = 2 D , and G is a σ -subanalytic function. Moreover, if F ( x ; θ ) is analytic as a function of θ for any fixed x ∈ M , then the function θ ↦→ G (( x , y ) ; θ ) is analytic for any fixed ( x , y ) ∈ ˜ M . The result follows from applying Theorem A.2 to G .

Using Corollary A.20, we can now easily prove Theorem 3.4 and Proposition 3.6 from the main text, which we restate here for convenience:

Theorem 3.4. (Finite Witness Theorem, simple version) Let M ⊆ R L be a countable union of affine sets, each of which is of dimension ≤ D . Let W ⊆ R D θ be open and connected. Let F ( x ; θ ) : M × W → R be an analytic function. Then for almost any ( θ (1) , . . . , θ (2 D +1) ) ∈ W 2 D +1 , the following set equality holds:

Proof of Theorem 3.4. Since affine sets are semialgebraic, countable unions of affine sets are σ -subanalytic. Therefore, Theorem 3.4 follows from Corollary A.20. Note that by Corollary A.20, the claim holds for all ( θ (1) , . . . , θ (2 D +1) ) except for possibly a dimension-deficient subset of W D +1 , which is slightly stronger than the 'for almost any' statement in the theorem.

We now prove Proposition 3.6 from the main text.

Proposition 3.6. Let σ : R → R be an analytic non-polynomial function. Let n, d ∈ N and set m = 2 n ( d + 1) + 1 . Let f : R d → R L be an injective function that is a composition of PwL functions and analytic functions. Then for Lebesgue almost any A ∈ R m × L , b ∈ R m , the function σ ( A f ( x ) + b ) is moment injective on M ≤ n ( R d ) .

Proof. Let Y = f ( R d ) ⊆ R L be the image of f . Since f is injective, it naturally induces an injective pushforward map of measures f ∗ : M ≤ n ( R d ) →M ≤ n ( Y ) , defined by

Thus, it remains to show that for Lebesgue almost any A ∈ R m × L , b ∈ R m , the function g ( y ) = σ ( Ay + b ) is moment-injective on M ≤ n ( Y ) . To prove this, our first step is to bound the dimension of Y . By Corollary A.19, since f is a σ -subanalytic function, Y is a σ -subanalytic set, and

Let M be the space of measure parameters over Y with n points:

and let W be the space of parameters

Then M and W are σ -subanalytic sets, dim( M ) ≤ n ( d +1) and thus m = 2 n ( d +1) + 1 is larger or equal to 2 dim ( M ) + 1 . We can now proceed as in the proof of Theorem 3.3:

Define F : M × W → R by

The set W is open and connected, and F ( w , Y ; a , b ) is analytic as a function of ( a , b ) for any fixed ( w , Y ) ∈ M . Therefore, by Corollary A.20, for almost any choice of ( a i , b i ) m i =1 ∈ W , the following set equality holds:

Let A ∈ R m × L with rows a 1 , . . . , a m , and b = ( b 1 , . . . , b m ) . Suppose that A , b indeed satisfy (24). Then, using the same argument as in the proof of Theorem 3.3, we see that the function g is moment-injective on M ≤ n ( Y ) . This concludes the proof.

B Moment Injectivity of piecewise-linear networks

In this appendix, we describe some additional results on moment injectivity of PwL networks. Our results are summarized as follows:

PwL networks are not moment injective on M ≤ n (Σ) , when Σ is an infinite set.

PwL networks are not moment injective on S ≤ n (Ω) when Ω = R d (shown in the main text) or Ω = Z d .

There exist irregular, countable infinite Σ , for which PwL networks are moment injective on S ≤ n (Σ) (but not on M ≤ n (Σ) as mentioned above).

For finite Σ , PwL networks can be moment-injective on both M ≤ n (Σ) and S ≤ n (Σ) . The number of neurons needed for injectivity depends on n and on the size of Σ .

We now prove these results. We begin with discussing infinite alphabets.

B.1 Failure of moment-injectivity for piecewise-linear functions with infinite alphabets

We begin by showing that PwL moment injectivity is never possible on spaces of measures , when the alphabet is infinite :

Proposition B.1. For all natural m,d and n ≥ ( d + 2) / 2 , and Ω ⊆ R d that is not finite, if ψ : R d → R m is piecewise linear, then it is not moment injective on M ≤ n (Ω) .

Proof. The proof is based on the fact that if µ = ∑ n i =1 w i δ x i is supported on a single linear region L of ψ , then, denoting the parameters of the affine function corresponding to the region L by A ∈ R d × m , b ∈ R m , we have that

Thus it is sufficient to find two distinct measures in M ≤ n (Ω) which are supported in the same linear region, and for which the two sums ∑ n i =1 w i and ∑ n i =1 w i x i give the same value. Since ψ has a finite number of linear regions while Ω is infinite, there is some linear region which contains an infinite number of points in Ω . Thus, we can choose d +2 distinct points x 1 , . . . , x d +2 in this region. Since all these points are in R d , they must be affinely dependent, which means there exist weights w 1 , . . . , w d +2 , not all of which are zero, such that

We can now define k = glyph[ceilingleft] ( d + 2) / 2 glyph[ceilingright] , and set µ = ∑ k i =1 w i δ x i and µ ′ = ∑ d +2 j = k +1 ( -w j ) δ x j . According to (25), integration of ψ over µ and over µ ′ gives the same value, while µ = µ ′ , and so ψ is not moment injective.

glyph[negationslash]

If we change the setup of Proposition B.1 and consider spaces of multisets rather than spaces of measures , we get a somewhat more complicate picture: In the following, we give an example of an irregular, infinite, countable alphabet Σ α , for which even the simple identity function x ↦→ x is moment injective. We shall then show that for the more natural alphabet Σ = Z d , PwL moment injectivity is not possible on S ≤ n (Σ) . We define Σ α = { α, α 2 , α 3 , . . . } , where α ∈ R is a transcendental number , which means that it is not the root of any polynomial with rational coefficients. Examples of such numbers include π and e . In this case, we see that the integral ∫ R xdµ ( x ) over any non-zero measure µ = ∑ n i =1 w i δ α i where w i are rational numbers, will not be zero. This implies that the identity function is moment injective on S ≤ n (Σ α ) for any natural n . Note also that if n is fixed, we can just take α = ( n +1) -1 , as suggested in [44]. Finally, note that by choosing α to be positive, we have that the simple piecewise linear MLP ReLU( x ) is moment injective on S ≤ n (Σ α ) as well, as ReLU( x ) = x on the domain Σ α .

We now consider the case Σ = Z d :

Proposition B.2. Let d, n ≥ 2 be natural numbers and Σ = Z d . If ψ : R d → R m is piecewise linear, then it is not moment injective on S ≤ n (Σ) .

Proof. Let us first consider the case where d = 1 . Since the number of linear regions is finite, there exists a linear region L that contains an infinite number of natural numbers. From the convexity of L , it follows that L contains an interval of the form [ N, ∞ ) where N ∈ N . Using the same argument as in Proposition 4.1, we conclude that the moments of the multisets { { 2 N, 2 N } } and { { N, 3 N } } are the same, and thus ψ is not moment injective. The same argument can be applied in the case d > 1 using the identification of Z with the elements in Z d whose first d -1 coordinates are zero.

B.2 Piecewise linear network moment injectivity for sets with finite alphabets

We now consider moment injectivity of PwL networks when the alphabet is finite. Recall that when the activations are analytic, moment injectivity is achievable on S ≤ n (Σ) with a single output neuron. For PwL activations, moment injectivity is also possible, but the required number of neurons depends on the cardinality of the alphabet.

Definition B.3. For given W,L,d ∈ N , we denote by M ( W,L,d ) the maximal number of linear regions that a ReLU network f : R d → R with maximal width W and depth L can have.

Proposition B.4. If n, W, L, d are natural numbers with n ≥ ( d + 2) / 2 , and Σ ⊆ R d is a finite alphabet satisfying | Σ | > M ( W,L,d ) · ( d +1) , then any ReLU neural network ψ : R d → R m of depth L and maximal width W will not be injective on M ≤ n (Σ) . Moreover, for large enough n , assuming that all points in Σ have rational coordinates, such ReLU networks will also not be moment injective on S ≤ n (Σ) .

Proof. Since M ( W,L,d ) < | Σ | d +1 , by the pigeonhole principle, we know that there must exist at least one linear region that contains at least d +2 points in Σ . Denote these points x 1 , x 2 , ..., x d +2 . These points are affinely dependent, and so there exist weights w 1 , . . . , w d +2 , not all of which are zero, such that

glyph[negationslash]

We can now define k = glyph[ceilingleft] ( d + 2) / 2 glyph[ceilingright] , and set µ = ∑ k i =1 w i δ x i and µ ′ = ∑ d +2 j = k +1 ( -w j ) δ x j . According to (25), integration of ψ over µ and over µ ′ gives the same value, while µ = µ ′ , and so ψ is not moment injective on M ≤ n (Σ) .

We now show that moment injectivity fails also on S ≤ n (Σ) when n is large enough, under the assumption that Σ ⊂ Q d . Under this assumption, the above-mentioned points x 1 , x 2 , ..., x d +2 are affinely dependent over Q , and there exist rational weights w 1 , . . . , w d +2 for which (26) holds.

Now we can multiply (26) by an appropriate integer, so that the w i s are all integers. Using these new weights, suppose that n ≥ ∑ i | w i | . Let µ = ∑ i : w i ≥ 0 w i δ x i and µ ′ = ∑ j : w j < 0 ( -w j ) δ x j . Then µ and µ ′ are two distinct measures that assign the same moment to f . Since the weights of µ and µ ′ are all natural numbers, these measures correspond to multisets, where the weights denote the number of repetitions of each element.

Number of linear regions Proposition B.4 gives a lower bound on the number of linear regions needed for moment injectivity. The relationship between a neural network's size and the number of its linear regions is well studied [29, 35]. In particular, it is known that the number of linear regions can be exponentially larger than the number of the parameters. As shown in ([1], Theorem D.6), the maximal number of linear regions M ( W,L,d ) of a ReLU network with maximal width W , depth L , and input dimension d , is bounded from above by CW d · L , where C > 0 is a constant. Joining this together with Proposition B.4, we see that we cannot have moment injectivity if

This implies that the number of neurons required for injectivity is at least logarithmic in the cardinality of the alphabet. Although this is only a lower bound, we believe that this bound can be achieved using the techniques developed in [1].

We conclude this discussion by providing an upper bound for the number of linear regions needed in the simple case that Σ = { glyph[lscript] 1 < glyph[lscript] 2 < . . . < glyph[lscript] S } is a subset of R , and the depth L is one. In this case, our bound (27) states that one cannot attain moment injectivity on M ≤ n (Σ) when 2 C · W < S . In the other direction, we show that moment injectivity can be obtained when W = S .

Proposition B.5. Let Σ = { glyph[lscript] 1 < glyph[lscript] 2 < . . . < glyph[lscript] S } , and let n ≤ S be some natural number. Then there exists a shallow ReLU network of width W = S that is moment injective on M ≤ n (Σ) .

Proof. Choose some t 1 , . . . , t S such that

Consider the shallow ReLU network

Suppose that µ = ∑ S i =1 w i δ glyph[lscript] i and µ ′ = ∑ S i =1 w ′ i δ glyph[lscript] i are two measures such that

The equality of the last coordinate implies

and so w S = w ′ S . Next, we can consider the equality of the ( S -1) th coordinate and show that this implies that w S -1 = w ′ S -1 . Continuing with this argument recursively, we see that the measures must agree.

C Optimal embedding dimension of moment-injective functions

In Appendix C.1, we prove the lower bounds on the embedding dimensions showed in Table 1 for M ≤ n ( R d ) , S ≤ n ( R d ) and M ≤ n (Σ) for countable Σ . The remaining lower bounds in the table are equal to one, and thus do not require a proof. For S ≤ n ( R d ) , these bounds are already known [16, 6, 42], but here we present a slightly different proof, which also applies to spaces of measures that were not considered previously. As a rule, for all the spaces of measures/multisets considered, the lower bound equals to the intrinsic dimension of the space, i.e., the number of continuous parameters required to describe it. For example, S ≤ n ( R d ) is parameterized by n continuous vectors in R d , with a total dimension of n · d , and by a discrete weight vector that has no influence on the dimension, and thus the lower bound in this case is n · d .

An interesting question that remains open is whether the gap between the embedding dimension achieved here with shallow neural networks, and the best lower bound, can be closed completely. For multisets with features in [0 , 1] , [42] showed that the lower bound of n moments can be attained using n polynomial functions. In Appendix C.2, we show that for spaces of measures with features in R , the lower bound 2 n in Table 1 can be attained by 2 n functions that form a T-system (see discussion below). Examples of T-systems include the functions x, x 2 , . . . , x 2 n , as well as 2 n univariate sigmoid neural networks with distinct parameters. Thus, for d = 1 , we know that the lower bounds on the embedding dimension are optimal. For d > 1 , it seems that this question is still open.

C.1 Lower bounds

Our lower bounds are based on the following intuitive fact, which is a simple corollary to Brouwer's invariance of domain theorem:

Proposition C.1. If U ⊆ R D is an open set and F : U → R M is continuous and injective, then M ≥ D .

Proof. Suppose by contradiction that M < D , and let 0 ∈ R D -M be a vector of all zeros. Then the function ˜ F ( x ) = ( 0 , F ( x )) is a continuous injective function of U ⊆ R D into R D and by the invariance of domain theorem ([13], Theorem 2B.3) ˜ F ( U ) should be open. This leads to a contradiction, since arbitrarily small perturbations of the first coordinate of a point ( 0 , F ( x )) in the image of ˜ F will no longer be in the image.

Based on this proposition, we can prove the lower bounds in Table 1. The following lower bound holds for any alphabet Σ .

Theorem C.2. Let Σ ⊆ R d with | Σ | ≥ n . Suppose that f : Σ → R m is moment injective on M ≤ n (Σ) . Then m ≥ n .

Proof. Fix n distinct points x 1 , . . . , x n in Σ . The function

maps R n injectively into the space of measures M ≤ n (Σ) . Since f is moment injective, it follows that the continuous (in fact, linear) function

is injective. Thus, by Proposition C.1 we have that m ≥ n .

When the alphabet is uncountable, we can obtain stronger lower bounds. We now show two lower bounds on the embedding dimension required for moment injectivity of a continuous function f with the alphabet Ω = R d .

Theorem C.3. Let f : R d → R m be a continuous function. Then

1. If f is moment injective on M ≤ n ( R d ) , then m ≥ n ( d +1) .

2. If f is moment injective on S ≤ n ( R d ) , then m ≥ nd .

Proof. Choose n distinct points y 1 , . . . , y n ∈ R d and let r > 0 be small enough so that the distance between any two distinct points y i , y j is larger than 2 r . Then the function

glyph[negationslash]

maps the open set

injectively into M ≤ n ( R d ) . Therefore, the map

is injective, and since it is also continuous, Proposition C.1 implies that m ≥ n ( d +1) .

For the second part of the theorem, where f is moment injective on S ≤ n ( R d ) , we can use a similar argument, the only difference being that we fix w i = 1 . Namely, the map

is continuous and injective on the open set of ( x 1 , . . . , x n ) with | x i -y i | < r for all i , and therefore by Proposition C.1 m ≥ nd .

C.2 T-Systems

We now show how to achieve moment injectivity on M ≤ n ( R ) with an optimal embedding dimension. Recall from Table 1 that for this space, our lower bound on the embedding dimension of continuous moment-injective functions is 2 n , whereas the embedding dimension of the MLPs constructed using our technique is 4 n +1 . We shall now show that this gap can be closed, using the moments of 2 n functions that form a T-system , which we now define.

Definition C.4 ([20]) . Let Ω ⊆ R . We say that the k functions τ i : Ω → R , i = 1 , . . . , k , form a T-system on Ω , if for all pairwise-distinct x 1 . . . x k ∈ Ω , the square matrix

is invertible.

An example of a T-system on Ω = R is the standard monomial basis τ i ( x ) = x i -1 , i = 1 , . . . , k . With this choice, (28) is the Vandermonde matrix, which is invertible, and hence the monomial basis is a T-system.

We know that the moments of the first n elements of the standard monomial basis are injective on S ≤ n ( R ) . The following simple proposition shows that injectivity on all of M ≤ n ( R ) can be achieved, at the cost of increasing the number of moments to 2 n . Moreover, this is true for all T-systems.

Proposition C.5 (Trivial, based on [20]) . If τ = ( τ 1 , . . . , τ 2 n ) is a T-system on Ω ⊆ R , then the induced moment function

is injective on M ≤ n (Ω) .

Proof. Let µ, µ ′ ∈ M ≤ n (Ω) such that ˆ τ ( µ ) = ˆ τ ( µ ′ ) . Then τ ( µ -µ ′ ) = 0 , where µ -µ ′ denotes the difference measure. Since µ -µ ′ is supported on at most 2 n points, we can write

where the points x i are pairwise distinct, and some of the w i may be zero. Our goal is to show that the vector

is all zero. Indeed, the fact that ˆ τ ( µ -µ ′ ) = 0 implies that M τ · w = 0 . And since by assumption M τ is full rank, we have that w = 0 .

Example C.6 (Sigmoid T-systems) . We've seen that the standard monomial basis forms a T-system. Another well-known example [20] is the family of functions

If the numbers a 1 , . . . , a k are pairwise distinct, then τ = ( τ 1 , . . . , τ k ) form a T-system on Ω = R \ { a 1 , . . . , a k } .

We can use these example to obtain an alternative T-system based on the sigmoid activation σ ( x ) = (1 + e -x ) -1 . Firstly, using the injectivity of the exponent function, we can deduce from the fact that τ is a T-system, that for any distinct b 1 , . . . , b k the functions

form a T-system on Ω = R . It follows that the functions

form a T-system as well. In particular, for all pairwise-distinct b 1 , . . . , b 2 n , the function

is moment injective on M ≤ n ( R ) .

Unfortunately, these results cannot be extended to M ≤ n ( R d ) with d > 1 , as it is known that T-systems can only be defined on subsets of R or S 1 [20].

D Proofs

In this section, we provide the proofs omitted from the main text, except for the proofs of Theorem 3.4 and Proposition 3.6, which appear in Appendix A.6.

D.1 Proofs for Section 3

Proposition 3.2. Let σ : R → R be a continuous function that is not a polynomial; then σ is discriminatory.

Proof. Let σ be as in the statement of the proposition, and let µ be a signed Borel measure that is finite and is supported on a compact set K ⊆ R d . Furthermore, assume that ∫ R d σ ( a · x + b ) dµ ( x ) = 0 for all a and b . We need to prove that µ = 0 .

Note that by the linearity of µ , we have that ∫ fdµ = 0 for every f in the space spanned by functions of the form σ ( a · x + b ) . Additionally, since functions in this space can approximate any continuous function on K uniformly ([32]), Propositions 3.3 and 3.8), and µ is compactly supported, we have that ∫ fdµ = 0 for every continuous function. Finally, by the Riesz representation theorem ([36], Theorem 6.19) we know that a signed (and more generally complex) measure on K is defined uniquely by the integrals of continuous functions and thus µ = 0 as required.

Proposition 3.5. Let n, d ∈ N and set m = 2 n ( d +1) + 1 . Let W = ( y , σ ) ∈ R d × R + . Then for Lebesgue almost any ( y i , σ i ) i m =1 ∈ W m , the function

is moment injective on M ≤ n ( R d ) .

Proof. The proof follows the proof of Theorem 3.3 with some modifications. Set m = 2 n ( d +1)+1 . A measure in M ≤ n ( R d ) is determined (albeit not uniquely) by a matrix X = ( x 1 , . . . , x n ) ∈ R d × n that represents n points in R d , and a weight vector w ∈ R n . Let M denote the space of pairs of measure parameters

and let

We prove the proposition by showing that for Lebesgue almost every y 1 , . . . , y m and σ 1 , . . . , σ m ,

As in the proof of Theorem 3.3, equality (**) follows from the finite witness theorem . The equality (*) follows on the one hand from the fact that whenever the measures defined by ( w , X ) and ( w ′ , X ′ ) are the same, necessarily all integrals of functions against these measures are the same, and so F ( w , w ′ , X , X ′ ; y , σ ) = 0 for all y ∈ R d , σ ∈ R + .

glyph[negationslash]

On the other hand, if the measure µ defined by ( w , X ) and the measure µ ′ defined by ( w ′ , X ′ ) , are not the same, then it is sufficient to show that for some choice of y ∈ R d , σ ∈ R + we have F ( w , w ′ , X , X ′ ; y , σ ) = 0 . To see this is indeed the case, choose some y 0 ∈ R d which the non-zero matrix µ -µ ′ assigns a non-zero weight w 0 . Then glyph[negationslash]

and so for small enough σ we have that F ( w , w ′ , X , X ′ ; y 0 , σ ) which concludes the proof.

D.2 Proofs for Section 5

Proposition 5.1. Let n ≥ 2 , d, m ∈ N , and let f : R d → R m be differentiable at some x 0 ∈ R d . Then the induced moment function ˆ f : S ≤ n ( R d ) → R m defined in (1) is not bi-Lipschitz.

Proof. We choose some arbitrary d ∈ R d with unit norm, and focus on multisets of two elements in R d of the form S glyph[epsilon1] = { { x 0 + glyph[epsilon1] d , x 0 -glyph[epsilon1] d } } . We note that the Wasserstein distance W 2 ( S glyph[epsilon1] , S 0 ) is √ 2 glyph[epsilon1] . Thus, it is sufficient to show that

for this implies that there is no positive c for which (8) holds. Indeed, denoting the differential of f at x 0 by J , we have

where the convergence to zero of the last expression follows from the definition of differentiability at a point.

D.3 Proofs for Section 6

We now prove Corollary 6.1:

Corollary 6.1. Let n, d ∈ N and set m = 2 nd +1 . Let σ : R → R be an analytic non-polynomial function. Let K ⊆ R d be a compact set. Then there exist A ∈ R m × d , b ∈ R d such that for any continuous permutation-invariant f : K n → R , there exists a continuous F : R m → R such that

Proof. By Proposition 1.3 in [9], it is sufficient to show that there exists A ∈ R m × d , b ∈ R d such that the permutation invariant function

is orbit separating. This means that any two element in K n that are not related by a permutation will be separated by the function in (31). Any pair of elements not related by a permutation correspond to two distinct multisets in S ≤ n ( R d ) with exactly n points. By Theorem 3.3, for almost every choice of A , b the function in (31) will be injective on S ≤ n ( R d ) , and thus for such choice this function is indeed invariant and orbit separating.

We now prove Corollary 6.2:

Corollary 6.2. Let n, d ∈ N and set m = 2 n ( d + 1) + 1 . Let σ : R → R be analytic and nonpolynomial. Let K ⊆ R d be compact. Then there exist A ∈ R m × d , b ∈ R m such that for any continuous (in the 2 -Wasserstein sense) f : P ≤ n ( K ) → R , there exists a continuous F : R m → R such that

Proof. Our first step is to show that P ≤ n ( K ) is compact with respect to the Wasserstein metric. Since P ≤ n ( K ) is the image of the compact set

under the function

it is sufficient to show that this function is continuous, as the image of a compact set under a continuous map is compact. Thus, given a sequence of ( w ( k ) , X ( k ) ) which converges to some ( w , X ) , we need to show that in Wasserstein space, the measures µ k := ∑ n i =1 w ( k ) i δ x ( k ) i converge to the measure µ := ∑ n i =1 w i δ x i . Since the Wasserstein distance metrizes the weak topology on measures ([40], Theorem 6.9), it is sufficient to see that the integral of every continuous s : K → R against the sequence of measures converge to the limit measure. Indeed:

Thus we have shown that P ≤ n ( K ) is compact.

Next, by Theorem 3.3, for almost every choice of A , b the function

is injective on M ≤ n ( R d ) , and so in particular on the compact subset P ≤ n ( K ) . As q is also continuous, and a continuous injective function defined on a compact set is a homeomorphism, it follows that q -1 : q ( P ≤ n ( K )) →P ≤ n ( K ) is continuous. We then have that f = ( f ◦ q -1 ) ◦ q . We can then use Tietze's extension theorem to extend f ◦ q -1 from its compact domain to a continuous function F defined on all of R m , and we then obtain f = F ◦ q as required.

Theorem 6.3. Let n, d, T ∈ N and let Σ ⊆ R d be countable. Let m ≥ 1 be any integer. Let σ : R → R be an analytic non-polynomial function. Then for Lebesgue almost any choice of A ( t ) , b ( t ) and η ( t ) , the MPNN defined in (10) and (11) assigns different global features to any pair of graphs G 1 , G 2 ∈ G ≤ n (Σ) that can be separated by T iterations of 1-WL.

Proof. The message passing iterations discussed in the theorem define new node features h ( t ) v from the previous node features h ( t -1) v via

This can be rewritten as

The final 'readout' step creates a global graph features from the node features of the last iteration T via

It is known [44] that every pair of graphs G, ˆ G ∈ G ≤ n (Σ) which are not be separated by T iterations of the WL test, will not be separated by our message passing procedure, regardless of the choice of the parameters

We need to prove the opposite direction.

Since the collection of all graph-pairs from G ≤ n (Σ) is countable, it is sufficient to show that any fixed pair of graphs G, ˆ G ∈ G ≤ n (Σ) that can be separated by T iterations of the WL test, can be separated by almost every choice of parameters.

glyph[negationslash]

Let us fix such a pair G, ˆ G ∈ G ≤ n (Σ) which can be separated by T iterations of WL. Note that the final features h G = h G ( θ ) and h ˆ G = h ˆ G ( θ ) obtained from the message passing procedures are an analytic function of the parameters θ , and therefore to show separation for almost every θ it is sufficient to show existence of θ for which h G ( θ ) = h ˆ G ( θ ) .

To show that a single such θ exists, we choose the parameters of the functions recursively in the order ( η (1) , A (1) , b (1) ) , ( η (2) , A (2) , b (2) ) , . . . in which they are applied in the message passing procedure. When choosing the parameters ( η ( t ) , A ( t ) , b ( t ) ) corresponding to the t th step, it is sufficient to show that, if at the previous ( t -1) timestamp, for two different nodes v, w we had glyph[negationslash]

glyph[negationslash]

then h ( t ) v and h ( t ) w will not be the same. Here v and w are nodes in either G or ˆ G .

Our first goal is to choose η ( t ) so that, for all given nodes v, w in G or ˆ G such that (32) is satisfied, we have glyph[negationslash]

To choose η ( t ) in this way, we note that since all previous parameters were already determined, and since we are only interested in a single pair of graphs, there is only a finite number of features

which we are interested in. Since for fixed vectors x 1 , x 2 , y 1 , y 2 of the same dimension with x 1 = x 2 , there can be at most a single η satisfying the equation

we conclude that all but a finite number of η satisfy glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

We choose η ( t ) which satisfies the inequality above. This implies that if the left-hand side in (32) is indeed an inequality h ( t -1) v = h ( t -1) w , then m ( t ) v,u = m ( t ) w,x for all ( u, x ) ∈ N ( v ) ×N ( w ) , which in turn implies the inequality of multisets obtained from w and v ((33). On the other hand, if the left-hand side in (32) is an equality h ( t -1) v = h ( t -1) w and the multisets in the right-hand side of (32) are distinct, then clearly the multisets in (33) are distinct too. Thus, we have proved that we can choose η ( t ) so that (32) implies (33).

glyph[negationslash]

It remains to show that we can choose the parameters of f ( t ) so that, if (33) holds for some nodes v, w of G or ˆ G , then the features h ( t ) v and h ( t ) w obtained from applying f ( t ) to the multisets in (33) will be distinct. Indeed, since we are only requiring f ( t ) to be injective on a finite collection of finite sets, we have that all these multisets are contained in S ≤ n (Σ) for some finite Σ , and therefore by Theorem 3.3 there is a choice of parameters ( A ( t ) , b ( t ) ) which is injective on S ≤ n (Σ) , even when m = 1 . Thus, we obtained a recursive procedure for choosing the parameters θ such that G and ˆ G will be separated, which concludes our proof.

glyph[negationslash]

E Numerical Experiments: Additional Details

E.1 Empirical injectivity and Bi-Lipschitzness

To generate the results shown in Figure 2 (and Figure 4 below), we ran multiple independent test instances in which we generated two random matrices X 1 , X 2 ∈ R d × n , representing two sets of n vectors in R d . With exact details appearing below, X 1 , X 2 were generated such that: (1) Each entry of X 1 and X 2 has expectation zero and a standard deviation (STD) of 1 ; (2) X 1 and X 2 differ in exactly n ∆ randomly chosen columns, with the parameter n ∆ chosen uniformly at random from { 1 , . . . , n } ; (3) each entry of the n ∆ nonzero columns of ∆ X = X 2 -X 1 has expectation zero and STD= ρ , with the parameter ρ itself drawn uniformly (once per test instance) from [ ρ min , ρ max ] . We used ρ min = 0 . 02 , ρ max = 1 . In other words, the relative difference ρ between non-identical columns of X 1 and X 2 is chosen at each instance randomly between 2% and 100% . The motivation for this construction was to test various types and magnitudes of differences between multisets.

We then randomly generated ¯ A ∈ R ¯ m × d and ¯ b ∈ R ¯ m × 1 , from which we took subblocks to be used as parameters for the function f ( x ; A , b ) of Equation (3): for various values of m ∈ [ ¯ m ] , we took A m , b m to be the top m rows of ¯ A and ¯ b respectively, and used them to construct the embedding ˆ f ( X ; A m , b m , σ ) : R d × n → R :

with x i denoting the columns of X . The entries of ¯ A and ¯ b were drawn from Gaussian distributions chosen such that for each row a k of ¯ A and corresponding entry b k of b , and each column x i of X 1 or X 2 , the input to the activation σ , a k · x i + b k , has expectation zero and STD=1; specifically,

For various activation functions σ , we calculated the ratio

with W 2 ( · , · ) being the 2 -Wasserstein distance. We used a Sinkhorn approximation of W 2 ( · , · ) , calculated by the GeomLoss Python library [11]. Finally, for each activation σ and embedding dimension m , we took c and C to be the minimum and maximum of r ( X 1 , X 2 ) respectively over all test instances, and recorded the ratio c/C . In each experimental setting, we ran between 500,000 and 2 million independent instances, depending on the values of d and n . The results appear in Figure 4. It can be seen that a similar behaviour is exhibited across different values of d and n . In particular, all PwL activations have c/C = 0 at low m and all analytic activations have positive c/C in all settings tested.

Probabilistic distributions of data At each instance, as mentioned above, we first chose n ∆ ∼ Uniform [ { 1 , . . . , n } ] and ρ ∼ Uniform [ ρ min , ρ max ] . We then randomly chose n ∆ columns labelled by J , at which X 1 , X 2 should differ. Let I = [ n ] \ J . Denote by X [: , Λ] the subblock of the matrix X with columns indexed by Λ . The entries of X 1 [: , I ] = X 2 [: , I ] were drawn i.i.d. from Normal (0 , 1) . For X 1 [: , J ] and X 2 [: , J ] , we generated two random matrices U, V ∈ R d × n ∆ , each of whose entries are i.i.d. Gaussian with expectation 0, STD= √ 1 -1 12 ( ρ 2 max + ρ max ρ min + ρ 2 min ) and STD= 1 for U , V respectively. We then set

Lastly, we generated the entries of ¯ A and ¯ b from Normal ( 0 , 1 √ 2 d ) and Normal ( 0 , 1 √ 2 ) respectively. We now show that: (i) each entry of X 1 [: , J ] and X 2 [: , J ] has expectation zero and STD=1, (ii) given ρ , each entry of ∆ X [: , J ] has expectation 0 and STD= ρ , and (iii) All inputs to the activation σ have expectation zero and STD=1. Let x 1 , x 2 be two corresponding entries of X 1 [: , J ] , X 2 [: , J ] , and let u, v be their corresponding entries of U, V . Then

1

10

100

1000

10

-

8

10

-

6

10

-

4

10

-

2

0

d

= 3

n

= 10

1

10

100

1000

10

-

8

10

-

6

10

-

4

10

-

2

0

n

= 100

1

10

100

1000

10

-

7

10

-

5

10

-

3

10

-

1

0

n

= 1000

1

10

100

1000

10

-

6

10

-

4

10

-

2

0

m

d

= 100

1

10

100

1000

10

-

6

10

-

4

10

-

2

0

m

Cos

TanH

Sigmoid

HardTanH

ReLU

The empirical ratio c/C of Equation (8) as a function of the embedding dimension m . The results for different sizes of vector-sets n and ambient dimension d are shown. For low m , the piecewise-linear ReLU and HardTanH have c/C exactly zero. See Appendix E.1 for a full description of the experimental setting.

We now calculate the expectation and variance of x 1 , x 2 . Since U , V and ρ are independent, we have that

and by a similar reasoning, E [ x 2 ] = 0 . The variance of x 1 is given by:

By a similar argument, Var [ x 2 ] = 1 . Thus, (i) holds. Let ∆ x = x 2 -x 1 = ρv . Then

Moreover, the conditional variance of ∆ x given ρ is

and thus (ii) holds. Finally, we show that (iii) holds. We already have established that each entry of X 1 and of X 2 has zero mean and STD=1. Let a k , b k be a row or ¯ A and its corresponding entry of ¯ b . Let x i be an arbitrary column of X 1 or X 2 . Then

and by definition E [ b k ] = 0 . Since each ( a k ) j and ( x i ) j are independent random variables with expectation zero, we have that

and by definition Var [ b k ] = 1 2 . Therefore, (iii) holds.

Computational resources All experiments were run on an NVidia A40 GPU with 48 GB of GPU memory.

E.2 Graph separation

We ran the graph separation experiment using the PyTorch Geometric [10] implementation for 'WL convolutions', and the GCN convolutions of [18], as well as their version of the TUDataset [30]. As initialization for the node features, we only took the node degree.

When moving from the abstract mathematical world to finite precision computing, features that are mathematically equal h i = h j could end up having slightly different values. To deal with this, all computations were done in double precision. The final features computed by the MPNN were normalized to have an average norm of one, and two features h i , h j were deemed equal if | h i -h j | < 10 -12 . In Figure 5 we show the minimum and median of the quantity ‖ h i -h j ‖ over all ( i, j ) from all graphs for which this norm was larger than the threshold of 10 -12 . This is shown for the SiLU activation, and the values are shown as a function of the hidden dimension used. We see that the minimal nonzero distance was safely larger than the threshold in all examples. Also note that the minimal distance moderately improves as the hidden dimension increases.

0

20

40

60

80

100

10

11

10

9

10

7

10

5

10

3

10

1

average feature norm

median non-zero distance

minimal non-zero distance

threshold

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies moment-injective functions defined by neural networks. A moment injective function f can be used to define injective multiset function $g(\\{\\{x_1, x_2, ..., x_n\\}\\}) = \\sum_{i = 1}^n f(x_i)$. Study of infectivity of multiset functions is motivated by recent developments in Graph neuran networks and Message-Passing Neural networks.\n\nPrior work shows that when $x_i \\in \\mathbb{R}^d$ moment injective function f should be an embedding of dimension at least nd, moreover one can construct moment-injective polynomial embedding of dimension 2nd+1.\n\nThis paper extends prior work by showing that depth 1 neural networks of width at least 2nd+1 with analytic non-polynomial activation unit defines a moment injective embedding for almost all choices of weights. Moreover, the paper shows that the result can be generalized to neural networks of higher depth (from theoretical perspective depth-1 case is the most interesting, as it involves the least amount of parameters).\n\nSTRENGTHS:\nThe paper provides theoretical explanation why neural networks are successful in providing injective multiset functions that gained popularity since the seminal Deep-Sets paper. While neural networks are long-known to be universal approximations for $f: R^k -> R^n$. Much less is known about multiset functions represented by neural networks. The main result proven in this paper (Theorem 3.3) shows that depth-1 neural networks can be used to define injective multiset functions, moreover, they achieve currently best-known embedding dimension. The paper also shows that one cannot use piecewise-linear activations to construct injective multiset functions and should use analytic non-polynomial activations instead (for example, sigmoid).\n\nThe paper is well-structured and provides a clear comparison to the prior work. The proofs are rigorous and well-written and as far as I can tell are correct.\n\nWEAKNESSES:\nI think the paper may benefit from slightly more detailed discussion of applications of injective multiset functions. While injectivity is a natural property that proves to be useful in various setups, I believe it will be nice to include a slightly more detailed discussion of the applications in the introduction. Some example applications are presented in Section 6.\n\nQUESTIONS:\n1) The paper shows that MLPs define injective multiset functions if activation is analytic non-polynomial, and fail to define injective multiset functions if activation is piecewise linear. What happens if activation is polynomial? Is it easy to see that such activations fail similarly to piecewise linear activations?\n\n2) I think it worth replacing \"up to a multiplicative factor of 2\" everywhere in the paper with \"up to a multiplicative factor of 2+o(1)\" to be mathematically precise. If one is pedantic, 2nd+1 is not within a factor of 2 from nd.\n\nI believe the paper contains several typos:\nL47: a Euclidean -> an Euclidean\nL129: this paper -> that paper (?)\nL456 quality -> qualify.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors study the injectivity of functions of multisets, which has garnered a lot of attention recently in the machine learning community following work on point clouds and graphs. Unlike earlier work, which prove results for generic continuous functions or polynomials, then may resort to the universality of MLPs but with unbounded width, here the authors directly focus on MLPs with finite width and prove injectivity with near-optimal width, for almost all parameters. This holds with analytic non-linearity, and on the contrary negative results are given for piecewise linear functions like ReLU. Their result relies mostly on a new \"finite witness\" theorem, which extend previous results known for semi-algebraic sets and functions to sub-analytic sets and functions. Several corollaries are presented, as well as some illustrative numerical experiments.\n\nSTRENGTHS:\nThis is honestly a fantastic paper, I very much enjoy the read, it is clear, pedagogical, and refreshingly honest. The results presented are of great interest for the community. Although it is an extension of a known results on semi-algebraic sets, as acknowledged by the authors, the extension seems anything but trivial. The large part dedicated to negative results and/or limitations is very much appreciated.\n\nWEAKNESSES:\nMinor weaknesses; maybe a tad more explanation on the difference between semi-algebraic and sub-analytic proof (I understand both lies on the same underlying proof technique initiated with works on phase retrieval, but with different tools, how would you describe the main difference if possible?), and a bit more outlooks: do you have any more insight what would be a desirable property to show on ReLU networks, since they are definitely not injective, but are used in practice?\n\nQUESTIONS:\nSome questions/minor typos:\n- l45: nessecary\n- equation (***): missing indices $i$ on the $x$'s\n- same equations: might be clearer if the $a$ and $b$ were indexed by $i$ and the $x$'s by $j$ (or vice versa)\n- thm 3.4: co-existence of $D$ and $D_\\theta$\n- eq (6) and lines below: in your definition of Wasserstein, do you mean \"uniform\" weights instead of \"unit\"? $S_1$ and $S_2$ may have different cardinality, but the associated measure must be normalized to compute (the classical) Wasserstein\n- l 288: seleceted\n- l 298: Lipshcitz\n- the experiment on bi-Lipschitz is not really illustrative of the theory, since functions are not bi-Lipschitz. It is not uninteresting in describing another phenomenon (the \"almost\" bi-Lipschitzness on finite data, etc), but I wouldn't say that it \"corroborates Thm 5.1\"\n\nLIMITATIONS:\nAuthors have focused extensively on negative results and limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nInjective functions on multisets are commonly employed in the literature for universality results on (multi)set architectures and graph neural network separation results. Usually, the assumption is that MLPs can implement moment injective functions. The paper studies whether MLP architectures are moment-injective in the space of multisets and the conditions that are required to ensure that. \n\nThe paper argues that for a (shallow) MLP to be moment injective, its activations have to be analytic and discriminatory. The paper considers a more generalized treatment that views multisets from the perspective of signed discrete measures. The goal of the proof for moment injectivity is to show that a shallow MLP separates all pairs of distinct measures in the space of measure parameters. There are two key conditions for the main theorem : \n\n- the activation of the MLP needs to be analytic and discriminatory, which are classic conditions for MLP results\n- the finite witness theorem, which is used to reduce an infinite number of equalities down to a finite one.\n\nThe finite witness theorem (and its generalized version) can be leveraged to show the moment-injectivity of various functions.\n\nThe paper then examines cases (alphabets) where moment injectivity doesn't hold for piece-wise linear activations. The stability properties (stability is studied through the lens of bi-lipschitzness) of injective multiset functions induced by moment injective MLPs are then studied and it is shown that as long as any moment injective function is differentiable at some point, then the induced multiset function won't be stable (bi-lipschitz).\n\nFinally, the paper provides a couple of applications of the technical results to function approximation and graph separation and provides some experimental evidence to back some of the technical claims up.\n\nSTRENGTHS:\n- The paper contains several interesting mathematical results which could be useful to the broader graph/set NN community.\n- The paper is well written and generally provides enough context and guidance for the results to be understandable.\n- The theoretical results are focused on architectures that are used in practice.\n- The experiments provide additional context and intuition for the applicability of the results.\n- The paper is upfront about its scope, limitations, and practical applicability.\n\nWEAKNESSES:\n- The initial motivation of the paper is the moment injectivity of 'practical' MLPs. While several results are ultimately established, we can see from the experiment on graph separation (and it is clearly stated in the conclusion) that even non-analytic functions just barely fail in a few cases to match 1-WL. This seems to suggest that the practical relevance of the results is a bit questionable, or maybe there are other factors that could be mitigating the non-analyticity of the activations. \n\n- A few extra explanatory lines could be included in the section of the finite witness theorem (or perhaps in the introduction), that provide context about its exact role in the proof. Currently, the theorem shows up a bit abruptly when theorem 3.3 is presented. OK, it can be inferred from the proof of 3.3 that infinitely many equalities are reduced down to a finite number, but I believe the readability of that section could improve if some of the context that is provided in the supplementary material was moved in the main text to make sure that the role of the theorem is very clearly explained and emphasized. Specifically, when explaining proof 3.3, around line 154 or perhaps right below the statement of 3.4, in subsection 3.1. Those could be good places to provide a few more sentences that restate the purpose of the theorem and explain the context around it (papers by Balan et al., and Dym and Gortler).\n\nI am skeptical about the significance of the results when it comes to practical considerations but the paper is well written and provides mathematical insights that can be relevant to modern deep learning architectures so I lean towards accepting.\n\nQUESTIONS:\nWe see in the second experiment a somewhat significant difference between leaky ReLU and ReLU. Could that be a matter of randomness from different initialization, or could the properties of the function that could lead to worse outcomes when it comes to matching the 1-WL? I don't see any obvious reason why the leaky activation would be much different.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}