SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tal Amir<sup>1</sup> Steven J. Gortler<sup>2</sup> Ilai Avni<sup>1</sup> Ravina Ravina<sup>1</sup> Nadav Dym<sup>1,3</sup>
$^{1}$  Faculty of Mathematics, Technion – Israel Institute of Technology, Haifa, Israel  
 $^{2}$  School of Engineering and Applied Sciences, Harvard University, Cambridge, USA  
 $^{3}$  Faculty of Computer Science, Technion – Israel Institute of Technology, Haifa, Israel.
Injective multiset functions have a key role in the theoretical study of machine learning on multisets and graphs. Yet, there remains a gap between the provably injective multiset functions considered in theory, which typically rely on polynomial moments, and the multiset functions used in practice, which rely on neural moments — whose injectivity on multisets has not been studied to date.
In this paper, we bridge this gap by showing that moments of neural networks do define injective multiset functions, provided that an analytic non-polynomial activation is used. The number of moments required by our theory is optimal essentially up to a multiplicative factor of two. To prove this result, we state and prove a finite witness theorem, which is of independent interest.
As a corollary to our main theorem, we derive new approximation results for functions on multisets and measures, and new separation results for graph neural networks. We also provide two negative results: (1) moments of piecewise-linear neural networks cannot be injective multiset functions; and (2) even when moment-based multiset functions are injective, they can never be bi-Lipschitz.
Multisets are a slight generalization of sets: like sets, they are an unordered collection of elements  $\{\pmb{x}_1,\dots ,\pmb{x}_k\}$ , but unlike sets, repetitions are allowed. Multisets arise naturally in many machine-learning tasks. They are the natural way to represent point clouds in  $\mathbb{R}^3$ , neighborhoods of vertices in graphs, and any other data that has an intrinsic order that is immaterial to the task at hand.
We refer to functions and architectures whose inputs are multisets in  $\mathbb{R}^d$  as multiset functions and multiset architectures. By definition, these functions do not depend on the order in which the multiset elements are given. This is important not only because the order is irrelevant and thus should not affect the output, but also because otherwise a model may overfit the training data by making predictions based on its intrinsic order.
Multiset architectures are typically constructed using a combination of permutation-invariant operations such as sum- and max-pooling [34], attention mechanisms [23] and sorting [46]. One simple and popular approach, pioneered in the seminal Deep-Sets paper [45], employs multiset functions based solely on sum-pooling. Namely, if the elements of all multisets come from some fixed alphabet  $\Omega$ , any function  $f:\Omega \to \mathbb{R}^m$  induces a multiset function  $\hat{f}$ , to which we refer as the moment of  $f$ :
$$
\hat {f} \left(\left\{\left. \left\{\boldsymbol {x} _ {1}, \dots , \boldsymbol {x} _ {k} \right. \right\} \right\}\right) = \sum_ {i = 1} ^ {k} f \left(\boldsymbol {x} _ {i}\right). \tag {1}
$$
While moment functions of the form (1) are simple, they are quite powerful. For example, in [45] it was shown that if  $\Omega$  is countable, and the multiset sets have no repetitions (so they are just sets), then for an appropriate  $f:\Omega \to \mathbb{R}$ , the induced function  $\hat{f}$  maps the input sets injectively to  $\mathbb{R}$ .
Injectivity is indeed a desired property for multiset functions. The search for such functions stems from the quest to find an architecture that can approximate all multiset functions. Clearly, if  $\hat{f}$  assigns the same value  $\hat{f}(S_1) = \hat{f}(S_2)$  to two different multisets  $S_1 \neq S_2$ , then any architecture based on  $\hat{f}$  will yield a poor approximation of a multiset function that assigns different values to  $S_1$  and  $S_2$ .
The authors of [45] showed that injectivity is not only necessary for approximation, but also sufficient: Under the assumption that the alphabet  $\Omega$  is countable, if the moment  $\hat{f}$  of  $f$  maps multisets injectively to  $\mathbb{R}^m$ , then any multiset function  $F$  can be written as a composition of the form  $F(\{\{\pmb{x}_1,\dots ,\pmb{x}_k\}\}) = g\left(\sum_{i = 1}^{k}f(\pmb{x}_i)\right)$ . Motivated by this observation, the authors proposed a neural architecture of this form, with the functions  $f$  and  $g$  replaced by Multi-Layer Perceptrons (MLPs). This step was justified by the universal approximation power of MLPs.
These intriguing results inspired further research, mainly focusing on seeking injective multiset functions of the form (1) for continuous alphabets such as  $\Omega = \mathbb{R}^d$ . Preferably, such functions should (a) have a minimal embedding dimension  $m$  while ensuring that  $f:\Omega \to \mathbb{R}^{m}$  induces an injective  $\hat{f}$ ; and (b) be practical to compute. We next summarize some of these results:
For a countable  $\Omega$ , the Deep-Sets paper as well as [44] showed that an embedding dimension  $m = 1$  is sufficient. For  $\Omega = \mathbb{R}$ , if the multisets contain at most  $n$  elements, and  $f$  is continuous, then an embedding dimension of  $m \geq n$  is necessary and sufficient for injectivity [41, 6, 42].
For  $\Omega = \mathbb{R}^d$  and multisets of size at most  $n$ , it was shown in [16] that  $m \geq nd$  is necessary for injectivity. As for an upper bound on the required  $m$ , while some polynomials discussed in the literature achieve injectivity with a rather high exponential [3, 26, 37] or polynomial [43] dimension  $m$ , recent work [9] achieved injectivity with  $m = 2nd + 1$ , using a polynomial  $f$  with randomly chosen coefficients — thus achieving the lower bound essentially up to a multiplicative factor of 2.
While the above works provide injective multiset functions with optimal or near-optimal embedding dimension, these functions are typically polynomials, and not the MLPs used in practice. As mentioned above, many papers [45, 44, 26] justify this by the fact that MLPs can approximate any function, and thus any polynomial. However, using this argument, we have no control on the number of neurons required for injectivity — which in some cases may be infinite, as we show in Section 4. In this paper, we address this limitation by providing a practical and efficient method to construct functions of the form (1) that are provably injective while having a near-optimal number of neurons. We now state this formally.
Let  $\Omega \subseteq \mathbb{R}^d$  be a set, to which we refer as an alphabet. Denote by  $S_{\leq n}(\Omega)$  the collection of all multisets  $\{\{\pmb{x}_1,\dots ,\pmb{x}_k\}\}$  with  $\pmb {x}_1,\ldots ,\pmb {x}_k\in \Omega$  and  $k\leq n$ . Any function  $f:\Omega \to \mathbb{R}^{m}$  induces a moment function  $\hat{f}:\mathcal{S}_{\leq n}(\Omega)\to \mathbb{R}^m$  as in (1). If  $\hat{f}$  is injective, we say that  $f$  is moment injective on  $S_{<  n}(\Omega)$ .
We also consider a natural generalization from multiset to measures, by identifying each multiset  $\{\pmb{x}_1,\dots ,\pmb{x}_k\}$  with the measure  $\mu = \sum_{i = 1}^{k}\delta_{\pmb{x}_i}$ , where  $\delta_{\pmb{x}}$  is the Dirac measure that assigns a unit weight to  $\pmb{x}$ . In this generalized setting, the induced multiset function  $\hat{f}$  of (1) is just the integral of  $f$  with respect to the measure  $\mu$ . More generally, we consider signed measures  $\mu = \sum_{i = 1}^{n}w_{i}\delta_{\pmb{x}_{i}}$ , with weights  $w_{i}\in \mathbb{R}$  that can be negative, and points  $\pmb{x}_i$  that belong to an alphabet  $\Omega \subseteq \mathbb{R}^d$ . We denote the space<sup>1</sup> of all such measures by  $\mathcal{M}_{\leq n}(\Omega)$ . A function  $f:\Omega \to \mathbb{R}^m$  induces a moment function  $\hat{f}:\mathcal{M}_{\leq n}(\Omega)\to \mathbb{R}^m$  defined by
$$
\hat {f} (\mu) = \int_ {\Omega} f (\boldsymbol {x}) d \mu (\boldsymbol {x}) = \sum_ {i = 1} ^ {n} w _ {i} f \left(\boldsymbol {x} _ {i}\right), \quad \text {w h e r e} \quad \mu = \sum_ {i = 1} ^ {n} w _ {i} \delta_ {\boldsymbol {x} _ {i}}. \tag {2}
$$
<sup>1</sup>While we use the term space for  $S_{\leq n}(\Omega)$  and  $\mathcal{M}_{\leq n}(\Omega)$ , note that these are not vector spaces, since the sum of two measures in these spaces might be supported on more than  $n$  points.
Domain	M≤n(Rd)	S≤n(Rd)	M≤n(Σ)	S≤n(Zd)	S≤n(Σα)
Analytic activation	2n(d+1)+1	2nd+1	2n+1	1	1
Piecewise-linear activation	∞	∞	∞	∞	1
Lower bound	n(d+1)	nd	n	1	1
Table 1: The embedding dimension required for constructing injective functions of measures and multiset.  $\Sigma \subset \mathbb{R}^d$  is any infinite countable alphabet. First row: dimensions for which our theorems guarantee injectivity when using analytic non-polynomial activations. Second row: with infinite alphabets, moments of a neural network of any finite size with a piecewise-linear activation cannot be injective, except in the multiset case, with some special countable alphabets such as  $\Sigma_{\alpha}$ , defined in Appendix B.1. Third row: lower bounds on the embedding dimension required for injectivity. These bounds show that our results from the first row of the table are optimal essentially up to a factor of two.
If  $\hat{f}$  is injective, we say that  $f$  is moment-injective on  $\mathcal{M}_{\leq n}(\Omega)$ . Naturally, injectivity on  $\mathcal{M}_{\leq n}(\Omega)$  implies injectivity on subsets of this space, such as the space of measures in  $\mathcal{M}_{\leq n}(\Omega)$  that are probability measures, or that have only positive weights. In particular, if  $f$  is moment-injective on  $\mathcal{M}_{\leq n}(\Omega)$ , then it is moment-injective on  $S_{\leq n}(\Omega)$ .
To summarize, the main questions we focus on in this paper are:
Main Questions: (a) Under what conditions is an MLP  $f$  moment-injective on spaces of multiset  $S_{\leq n}(\Omega)$  or measures  $\mathcal{M}_{\leq n}(\Omega)$ ? (b) How many neurons are needed to achieve this injectivity?
Interestingly, we find that the answers to these two questions largely depend on the activation function. Consider shallow neural networks  $f: \mathbb{R}^d \to \mathbb{R}^m$  of the form
$$
f (\boldsymbol {x}; \boldsymbol {A}, \boldsymbol {b}) = \sigma (\boldsymbol {A} \boldsymbol {x} + \boldsymbol {b}), \quad \boldsymbol {A} \in \mathbb {R} ^ {m \times d}, \boldsymbol {b} \in \mathbb {R} ^ {m}, \tag {3}
$$
with the activation function  $\sigma: \mathbb{R} \to \mathbb{R}$  applied entrywise to  $A\pmb{x} + \pmb{b}$ . Suppose that  $\sigma$  is analytic and non-polynomial; such activations include the sigmoid, softplus, tanh, swish and sin. In Section 3 we show that for a large enough  $m$ , such networks  $f(\pmb{x}; \pmb{A}, \pmb{b})$  with random parameters  $\pmb{A}, \pmb{b}$  are moment-injective on  $\mathcal{M}_{\leq n}(\Omega)$  and on  $S_{\leq n}(\Omega)$ ; namely, their induced moment functions  $\hat{f}$  of (2) are injective. This holds for various natural choices of  $\Omega$ .
The embedding dimension  $m$  required in (3) depends on the dimension  $d$  of  $\Omega$ : For  $\Omega = \mathbb{R}^d$ , to achieve injectivity on  $S_{\leq n}(\Omega)$  or  $\mathcal{M}_{\leq n}(\Omega)$ , it suffices to take  $m = 2nd + 1$  or  $m = 2n(d + 1) + 1$  respectively. When  $\Omega$  is countable,  $m = 1$  or  $m = 2n + 1$  are sufficient (corresponding to  $d = 0$ ). In Appendix C we show that in all these cases, these embedding dimensions are optimal essentially up to a multiplicative factor of two. These results are summarized in Table 1. In Appendix C we also discuss examples where the optimal embedding cardinality for  $\mathcal{M}_{\leq n}(\mathbb{R})$  is obtained.
At the core of our proof of moment injectivity is a theorem which we name the finite witness theorem. This theorem enables reducing an infinite family of analytic equality constraints  $\{F(\pmb{x};\pmb{\theta}) = 0 \mid \pmb{\theta} \in \mathbb{W}\}$  to a finite subset  $\{F(\pmb{x};\pmb{\theta}_i) = 0 \mid i = 1,\dots,m\}$ . This theorem generalizes the results in [9], where a special case of it was proved for semialgebraic domains and functions. The theorem we prove here (see Appendix A) applies to a much wider class of domains and functions, among which are analytic functions. In addition to our main result, we use the finite witness theorem to prove moment injectivity of Gaussian functions (Proposition 3.5) and deep MLPs (Proposition 3.6), and we believe it shall find additional applications beyond those discussed in this work.
Negative Results We also prove two negative results for moment-based multiset functions: We show that in contrast to analytic activations, with piecewise linear (PwL) activations, such as ReLU, leaky ReLU and hard arctan, moment injectivity on spaces of measures  $\mathcal{M}_{\leq n}(\Omega)$  with infinite  $\Omega$  is impossible. On multiset spaces  $S_{\leq n}(\Omega)$ , moment injectivity with PwL activations can be obtained for some irregular, countable  $\Omega$ , such as the alphabet  $\Omega = \Sigma_{\alpha}$  defined in Appendix B, but not for infinite alphabets that arise naturally, like  $\Omega = \mathbb{R}^d$ ,  $\mathbb{Z}^d$  or  $(0,1)^d$ . These results are summarized in the bottom row of Table 1. The second negative result is that while moments of MLPs with analytic activations can be injective, they can never be stable in the bi-Lipschitz sense. This points to a possible advantage
Analytic	PwL
Dim	Tanh	SiLU	Sin	HardTanh	ReLU	Leaky
1	0	0	0	7	17	7
10	0	0	0	3	7	7
50	0	0	0	4	5	5
100	0	0	0	1	0	0
Figure 1: (a) The number of failures of graph neural networks, with varying hidden dimension and activation, to achieve WL separation on the 600 graphs from the TUDataset [30]. Analytic activations succeed on all graphs, as Theorem 6.3 predicts. (b) The normalized smallest singular value of multiset functions induced by piecewise-linear ReLU-networks and analytic SiLU-networks. Piecewise-linear networks have singularities on squares intersecting the diagonal, leading to non-injectivity. Analytic networks are moment injective, but have singularities on the diagonal, which leads to a non-Lipschitz inverse. See the end of Section 5 for more details.
of injective multiset functions that are not based on moments, but rather on sorting [3] or max-filters [5]. These multiset functions are not only injective but also bi-Lipschitz.
Implications for learning on multisets and graphs The result on moment injectivity of MLPs with analytic non-polynomial activation enables us to improve upon two seminal theoretical results in the study of functions on multisets and graphs:
(a) Universality for multisets. In Corollaries 6.1 and 6.2, we show that any continuous function on a space of multisets or measures respectively can be presented as a continuous vector-to-vector function composed with a moment function  $\hat{f}$  of an MLP of the form (3). The MLP has the same embedding dimension  $m$  as in Table 1. Essentially, this result replaces the moment-injective polynomials traditionally used in the characterization of multiset functions [45, 42] by MLPs.
(b) Separation power of Graph Neural Networks. Famously, the ability of Message-Passing Neural Networks (MPNNs) to separate distinct graphs is at most that of the Weisfeiler-Leman (WL) graph isomorphism test, with equivalence taking place if the multiset functions used in the MPNN are injective [44]. Injective multiset functions are also used in generalizations of this result, such as the equivalence of high-order Graph Neural Networks (GNNs) to high-order WL tests [31, 26], and recent results on geometric GNNs and their corresponding WL tests [15, 16, 25, 33, 8].
Using the fact that an embedding dimension of one is sufficient to achieve injectivity on  $S_{\leq n}(\Omega)$  with countable  $\Omega$ , we show in Theorem 6.3 that standard MPNNs with analytic non-polynomial activations and random parameters have the separation power of WL, even when their architecture only uses a single feature per node. This can be compared on the one hand with the construction in [44], which also requires a single node feature but uses multiset aggregators that are not MLPs, and on the other hand with works that do consider MLPs with ReLU activations [31, 1], but require a number of node features and parameters that depends polylogarithmically on the number of nodes. In contrast, our construction requires a single node feature and a fixed number of parameters (though we have no bound on the number of bits required for achieving separation using floating-point arithmetic). A numerical verification of these results is shown in Figure 1(a), where we show that, on the 600 graphs in the TUDataset [30], MPNNs with three different analytic activations were equivalent to the WL-test even with a single node feature, whereas three different PwL activations were in some cases weaker than WL when a small number of node features was used.
Independently of this work, it was recently proved in [17] that MPNNs with certain analytic activations can separate all trees of depth two, while separation fails with PwL and even piecewise-polynomial activations. Our results here are stronger in that we show separation for all graphs separable by WL, and all analytic non-polynomial activations.
We denote vectors by boldface letters, e.g.  $x$ ,  $y$ , and scalars by plain letters  $x$ ,  $y$ . The inner product of  $a$ ,  $x$  is denoted by  $a \cdot x$ . Throughout this work, the term measure always refers to signed measures.
3 Moment injectivity with analytic activations
In this section, we prove moment injectivity for MLPs with analytic non-polynomial activations. We begin by showing that for any non-polynomial function  $\sigma : \mathbb{R} \to \mathbb{R}$ , a measure  $\mu \in \mathcal{M}_{\leq n}(\Omega)$  is uniquely determined by the integrals of all functions  $\{\sigma (\boldsymbol{a} \cdot \boldsymbol{x} + b) \mid \boldsymbol{a} \in \mathbb{R}^d, b \in \mathbb{R}\}$ . When this holds, we say that  $\sigma$  is discriminatory:
Definition 3.1. Let  $\sigma : \mathbb{R} \to \mathbb{R}$  be a continuous function. We say that  $\sigma$  is discriminatory if for any two signed Borel measures  $\mu, \mu'$  on  $\mathbb{R}^d$  that are distinct (i.e.  $\mu \neq \mu'$ ), finite (i.e.  $|\mu(A)|, |\mu'(A)| < \infty$  for all Borel  $A \subseteq \mathbb{R}^d$ ) and compactly supported, there exist  $a \in \mathbb{R}^d$ ,  $b \in \mathbb{R}$  such that
$$
\int_ {\mathbb {R} ^ {d}} \sigma (\boldsymbol {a} \cdot \boldsymbol {x} + b) d \mu (\boldsymbol {x}) \neq \int_ {\mathbb {R} ^ {d}} \sigma (\boldsymbol {a} \cdot \boldsymbol {x} + b) d \mu^ {\prime} (\boldsymbol {x}). \tag {4}
$$
The definition of discriminatory activation functions comes from Cybenko's celebrated paper on the universality of MLPs [7], where it was proved that sigmoid-like activations are discriminatory. This, in turn, was used to prove the universality of MLPs with such activations. In later papers [24, 32] it was shown that universality can be achieved by all continuous non-polynomial activations. In the following simple proposition, we use a reverse argument to that used by Cybenko, and show that activations that allow for universality are automatically discriminatory:
Proposition 3.2. Let  $\sigma : \mathbb{R} \to \mathbb{R}$  be a continuous function that is not a polynomial; then  $\sigma$  is discriminatory.
Proof idea. Suppose that  $\int \sigma (\pmb {a}\cdot \pmb {x} + b)d\mu = \int \sigma (\pmb {a}\cdot \pmb {x} + b)d\mu^{\prime}$  for all  $\pmb{a},b$  . By the universality theorem for shallow MLPs [32], all continuous functions can be approximated by linear combinations of functions of the form  $\sigma (\pmb {a}\cdot \pmb {x} + b)$  . Thus, for any continuous function  $f,\int f d\mu = \int f d\mu^{\prime}$  . Since a measure is uniquely determined by its integrals of all continuous functions,  $\mu$  is equal to  $\mu^\prime$
Next, we shall prove our main result: If  $\sigma$  is analytic and discriminatory, then shallow MLPs of reasonable width with  $\sigma$  as activation are moment injective.
Theorem 3.3. Let  $\sigma : \mathbb{R} \to \mathbb{R}$  be an analytic non-polynomial function. Let  $n, d \in \mathbb{N}$ , and set  $m = 2n(d + 1) + 1$ . Then for Lebesgue almost any  $\pmb{A} \in \mathbb{R}^{m \times d}$ ,  $\pmb{b} \in \mathbb{R}^m$ , the shallow MLP  $f(\pmb{x}) = \sigma (\pmb{A} \cdot \pmb{x} + \pmb{b})$  is moment injective on  $\mathcal{M}_{\leq n}(\mathbb{R}^d)$ ; namely, the function  $\hat{f} : \mathcal{M}_{\leq n}(\Omega) \to \mathbb{R}^m$  given by
$$
\hat {f} (\mu) = \sum_ {i = 1} ^ {n} w _ {i} \sigma \left(\boldsymbol {A} \boldsymbol {x} _ {i} + \boldsymbol {b}\right) \text {f o r} \mu = \sum_ {i = 1} ^ {n} w _ {i} \delta_ {\boldsymbol {x} _ {i}} \tag {5}
$$
For moment injectivity on  $\mathcal{S}_{\leq n}(\mathbb{R}^d)$ , it suffices to take  $m = 2nd + 1$ . For  $\mathcal{M}_{\leq n}(\Sigma)$  or  $\mathcal{S}_{\leq n}(\Sigma)$  with countable  $\Sigma$ ,  $m = 2n + 1$  and  $m = 1$  respectively are sufficient.
Our proof of Theorem 3.3 is based on a separate theorem, which we name the finite witness theorem. This theorem enables us to show that, since any two measures can be discriminated by an integral  $\int \sigma (\pmb {a}\cdot \pmb {x} + b)d\mu (\pmb {x})$  for some choice of parameters  $a,b$ , there exists a finite number of witness parameters  $(a_{i},b_{i})_{i = 1}^{m}$  that are sufficient for discriminating between any two measures. This holds under the assumption that the number of points in both measures is bounded. We shall now state a simple version of this theorem, which suffices for proving Theorem 3.3.
Theorem 3.4. (Finite Witness Theorem, simple version) Let  $\mathbb{M} \subseteq \mathbb{R}^L$  be a countable union of affine sets, each of which is of dimension  $\leq D$ . Let  $\mathbb{W} \subseteq \mathbb{R}^{D_\theta}$  be open and connected. Let  $F(\boldsymbol{x}; \boldsymbol{\theta}): \mathbb{M} \times \mathbb{W} \to \mathbb{R}$  be an analytic function. Then for almost any  $(\boldsymbol{\theta}^{(1)}, \dots, \boldsymbol{\theta}^{(2D + 1)}) \in \mathbb{W}^{2D + 1}$ , the following set equality holds:
$$
\{(x, y) \in \mathbb {M} \times \mathbb {M} \mid F (x; \boldsymbol {\theta}) = F (y; \boldsymbol {\theta}), \forall \boldsymbol {\theta} \in \mathbb {W} \} =
$$
$$
\left\{\left(\boldsymbol {x}, \boldsymbol {y}\right) \in \mathbb {M} \times \mathbb {M} \mid F \left(\boldsymbol {x}; \boldsymbol {\theta} ^ {(i)}\right) = F \left(\boldsymbol {y}; \boldsymbol {\theta} ^ {(i)}\right), \forall i = 1, \dots 2 D + 1 \right\}.
$$
2with a minor change: we do not require that all measures considered are supported on the same fixed compact set.
Using the finite witness theorem, we are now ready to prove Theorem 3.3.
Proof of Theorem 3.3. Recall that a signed measure  $\mu \in \mathcal{M}_{\leq n}(\mathbb{R}^d)$  can be parameterized, albeit not uniquely, by a matrix  $\pmb{X} = (x_{1},\dots,x_{n})\in \mathbb{R}^{d\times n}$  representing  $n$  points in  $\mathbb{R}^d$ , and a weight vector  $\pmb{w} = (w_{1},\dots,w_{n})$ , such that  $\mu = \sum_{i = 1}^{n}w_{i}\delta_{\pmb{x}_{i}}$ . Let  $\mathbb{M}$  be the space of measure parameters
$$
\mathbb {M} = \left\{\left(\boldsymbol {w}, \boldsymbol {X}\right) \in \mathbb {R} ^ {n} \times \mathbb {R} ^ {d \times n} \right\}.
$$
Similarly, let  $\mathbb{W}$  be the space of parameters
$$
\mathbb {W} = \left\{\left(\boldsymbol {a}, b\right) \in \mathbb {R} ^ {d} \times \mathbb {R} \right\}.
$$
Define  $F:\mathbb{M}\times \mathbb{W}\to \mathbb{R}$  by
$$
F (\boldsymbol {w}, \boldsymbol {X}; \boldsymbol {a}, b) = \sum_ {i = 1} ^ {n} w _ {i} \sigma (\boldsymbol {a} \cdot \boldsymbol {x} _ {i} + b). \tag {6}
$$
We now invoke the finite witness theorem. Set  $m = 2n(d + 1) + 1$ , and note that  $m = 2\dim (\mathbb{M}) + 1$ . Recall that  $F$  is analytic. According to Theorem 3.4, for almost any choice of  $(a_{i},b_{i})_{i = 1}^{m}\in \mathbb{W}$ ,