SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Thus, it remains to show that for Lebesgue almost any  $\mathbf{A} \in \mathbb{R}^{m \times L}$ ,  $\mathbf{b} \in \mathbb{R}^m$ , the function  $g(\mathbf{y}) = \sigma(\mathbf{A}\mathbf{y} + \mathbf{b})$  is moment-injective on  $\mathcal{M}_{\leq n}(\mathcal{V})$ . To prove this, our first step is to bound the dimension of  $\mathcal{V}$ . By Corollary A.19, since  $f$  is a  $\sigma$ -subanalytic function,  $\mathcal{V}$  is a  $\sigma$ -subanalytic set, and
$$
\dim (\mathcal {Y}) \leq \dim \left(\mathbb {R} ^ {d}\right) = d.
$$
Let  $\mathbb{M}$  be the space of measure parameters over  $\mathcal{V}$  with  $n$  points:
$$
\mathbb {M} = \left\{\left(\boldsymbol {w}, \boldsymbol {Y}\right) \in \mathbb {R} ^ {n} \times \mathcal {Y} ^ {n} \right\},
$$
and let  $\mathbb{W}$  be the space of parameters
$$
\mathbb {W} = \left\{\left(\boldsymbol {a}, b\right) \in \mathbb {R} ^ {L} \times \mathbb {R} \right\}.
$$
Then  $\mathbb{M}$  and  $\mathbb{W}$  are  $\sigma$ -subanalytic sets,  $\dim (\mathbb{M}) \leq n(d + 1)$  and thus  $m = 2n(d + 1) + 1$  is larger or equal to  $2\dim (\mathbb{M}) + 1$ . We can now proceed as in the proof of Theorem 3.3:
$$
F (\boldsymbol {w}, \boldsymbol {Y}; \boldsymbol {a}, b) = \sum_ {i = 1} ^ {n} w _ {i} \sigma (\boldsymbol {a} \cdot \boldsymbol {y} _ {i} + b).
$$
The set  $\mathbb{W}$  is open and connected, and  $F(\pmb {w},\pmb {Y};\pmb {a},b)$  is analytic as a function of  $(a,b)$  for any fixed  $(\pmb {w},\pmb {Y})\in \mathbb{M}$ . Therefore, by Corollary A.20, for almost any choice of  $(a_{i},b_{i})_{i = 1}^{m}\in \mathbb{W}$ , the following set equality holds:
$$
\begin{array}{l} \left\{\left(\left(\boldsymbol {w}, \boldsymbol {Y}\right), \left(\boldsymbol {w} ^ {\prime}, \boldsymbol {Y} ^ {\prime}\right)\right) \in \mathbb {M} \times \mathbb {M} \mid F \left(\boldsymbol {w}, \boldsymbol {Y}; \boldsymbol {a}, b\right) = F \left(\boldsymbol {w} ^ {\prime}, \boldsymbol {Y} ^ {\prime}; \boldsymbol {a}, b\right), \forall (\boldsymbol {a}, b) \in \mathbb {W} \right\} = \tag {24} \\ \left\{\left(\left(\boldsymbol {w}, \boldsymbol {Y}\right), \left(\boldsymbol {w} ^ {\prime}, \boldsymbol {Y} ^ {\prime}\right)\right) \in \mathbb {M} \times \mathbb {M} \mid F \left(\boldsymbol {w}, \boldsymbol {Y}; \boldsymbol {a} _ {i}, b _ {i}\right) = F \left(\boldsymbol {w} ^ {\prime}, \boldsymbol {Y} ^ {\prime}; \boldsymbol {a} _ {i}, b _ {i}\right), \forall i = 1, \dots , m \right\}. \\ \end{array}
$$
Let  $\mathbf{A} \in \mathbb{R}^{m \times L}$  with rows  $a_1, \ldots, a_m$ , and  $b = (b_1, \ldots, b_m)$ . Suppose that  $\mathbf{A}, \mathbf{b}$  indeed satisfy (24). Then, using the same argument as in the proof of Theorem 3.3, we see that the function  $g$  is moment-injective on  $\mathcal{M}_{\leq n}(\mathcal{V})$ . This concludes the proof.
B Moment Injectivity of piecewise-linear networks
In this appendix, we describe some additional results on moment injectivity of PwL networks. Our results are summarized as follows:
1. PwL networks are not moment injective on  $\mathcal{M}_{\leq n}(\Sigma)$ , when  $\Sigma$  is an infinite set.
2. PwL networks are not moment injective on  $S_{\leq n}(\Omega)$  when  $\Omega = \mathbb{R}^d$  (shown in the main text) or  $\Omega = \mathbb{Z}^d$ .
3. There exist irregular, countable infinite  $\Sigma$ , for which PwL networks are moment injective on  $S_{<n}(\Sigma)$  (but not on  $\mathcal{M}_{<n}(\Sigma)$  as mentioned above).
4. For finite  $\Sigma$ , PwL networks can be moment-injective on both  $\mathcal{M}_{\leq n}(\Sigma)$  and  $S_{\leq n}(\Sigma)$ . The number of neurons needed for injectivity depends on  $n$  and on the size of  $\Sigma$ .
We now prove these results. We begin with discussing infinite alphabets.
B.1 Failure of moment-injectivity for piecewise-linear functions with infinite alphabets
We begin by showing that PwL moment injectivity is never possible on spaces of measures, when the alphabet is infinite:
Proposition B.1. For all natural  $m, d$  and  $n \geq (d + 2)/2$ , and  $\Omega \subseteq \mathbb{R}^d$  that is not finite, if  $\psi: \mathbb{R}^d \to \mathbb{R}^m$  is piecewise linear, then it is not moment injective on  $\mathcal{M}_{\leq n}(\Omega)$ .
Proof. The proof is based on the fact that if  $\mu = \sum_{i=1}^{n} w_i \delta_{\pmb{x}_i}$  is supported on a single linear region  $L$  of  $\pmb{\psi}$ , then, denoting the parameters of the affine function corresponding to the region  $L$  by  $\pmb{A} \in \mathbb{R}^{d \times m}$ ,  $\pmb{b} \in \mathbb{R}^m$ , we have that
$$
\sum_ {i = 1} ^ {n} w _ {i} \boldsymbol {\psi} \left(\boldsymbol {x} _ {i}\right) = \boldsymbol {A} \left(\sum_ {i = 1} ^ {n} w _ {i} \boldsymbol {x} _ {i}\right) + \boldsymbol {b} \left(\sum_ {i = 1} ^ {n} w _ {i}\right). \tag {25}
$$
Thus it is sufficient to find two distinct measures in  $\mathcal{M}_{<n}(\Omega)$  which are supported in the same linear region, and for which the two sums  $\sum_{i=1}^{n} w_i$  and  $\sum_{i=1}^{n} w_i x_i$  give the same value. Since  $\psi$  has a finite number of linear regions while  $\Omega$  is infinite, there is some linear region which contains an infinite number of points in  $\Omega$ . Thus, we can choose  $d+2$  distinct points  $x_1, \ldots, x_{d+2}$  in this region. Since all these points are in  $\mathbb{R}^d$ , they must be affinely dependent, which means there exist weights  $w_1, \ldots, w_{d+2}$ , not all of which are zero, such that
$$
\sum_ {j = 1} ^ {d + 2} w _ {j} \boldsymbol {x} _ {j} = 0 \text {a n d} \sum_ {j = 1} ^ {d + 2} w _ {j} = 0.
$$
We can now define  $k = \lceil (d + 2) / 2 \rceil$ , and set  $\mu = \sum_{i=1}^{k} w_i \delta_{\pmb{x}_i}$  and  $\mu' = \sum_{j=k+1}^{d+2} (-w_j) \delta_{\pmb{x}_j}$ . According to (25), integration of  $\psi$  over  $\mu$  and over  $\mu'$  gives the same value, while  $\mu \neq \mu'$ , and so  $\psi$  is not moment injective.
If we change the setup of Proposition B.1 and consider spaces of multiset s rather than spaces of measures, we get a somewhat more complicate picture: In the following, we give an example of an irregular, infinite, countable alphabet  $\Sigma_{\alpha}$ , for which even the simple identity function  $x\mapsto x$  is moment injective. We shall then show that for the more natural alphabet  $\Sigma = \mathbb{Z}^d$ , PwL moment injectivity is not possible on  $S_{\leq n}(\Sigma)$ . We define  $\Sigma_{\alpha} = \{\alpha ,\alpha^{2},\alpha^{3},\ldots \}$ , where  $\alpha \in \mathbb{R}$  is a transcendental number, which means that it is not the root of any polynomial with rational coefficients. Examples of such numbers include  $\pi$  and  $e$ . In this case, we see that the integral  $\int_{\mathbb{R}}xd\mu (x)$  over any non-zero measure  $\mu = \sum_{i = 1}^{n}w_{i}\delta_{\alpha^{i}}$  where  $w_{i}$  are rational numbers, will not be zero. This implies that the identity function is moment injective on  $S_{\leq n}(\Sigma_{\alpha})$  for any natural  $n$ . Note also that if  $n$  is fixed, we can just take  $\alpha = (n + 1)^{-1}$ , as suggested in [44]. Finally, note that by choosing  $\alpha$  to be positive, we have that the simple piecewise linear MLP ReLU(x) is moment injective on  $S_{\leq n}(\Sigma_{\alpha})$  as well, as  $\mathrm{ReLU}(x) = x$  on the domain  $\Sigma_{\alpha}$ .
We now consider the case  $\Sigma = \mathbb{Z}^d$ :
Proposition B.2. Let  $d, n \geq 2$  be natural numbers and  $\Sigma = \mathbb{Z}^d$ . If  $\psi : \mathbb{R}^d \to \mathbb{R}^m$  is piecewise linear, then it is not moment injective on  $S_{\leq n}(\Sigma)$ .
Proof. Let us first consider the case where  $d = 1$ . Since the number of linear regions is finite, there exists a linear region  $L$  that contains an infinite number of natural numbers. From the convexity of  $L$ , it follows that  $L$  contains an interval of the form  $[N, \infty)$  where  $N \in \mathbb{N}$ . Using the same argument as in Proposition 4.1, we conclude that the moments of the multisets  $\{(2N, 2N)\}$  and  $\{(N, 3N)\}$  are the same, and thus  $\psi$  is not moment injective. The same argument can be applied in the case  $d > 1$  using the identification of  $\mathbb{Z}$  with the elements in  $\mathbb{Z}^d$  whose first  $d - 1$  coordinates are zero.
B.2 Piecewise linear network moment injectivity for sets with finite alphabets
We now consider moment injectivity of PwL networks when the alphabet is finite. Recall that when the activations are analytic, moment injectivity is achievable on  $S_{\leq n}(\Sigma)$  with a single output neuron. For PwL activations, moment injectivity is also possible, but the required number of neurons depends on the cardinality of the alphabet.
Definition B.3. For given  $W, L, d \in \mathbb{N}$ , we denote by  $M(W, L, d)$  the maximal number of linear regions that a ReLU network  $f: \mathbb{R}^d \to \mathbb{R}$  with maximal width  $W$  and depth  $L$  can have.
Proposition B.4. If  $n, W, L, d$  are natural numbers with  $n \geq (d + 2)/2$ , and  $\Sigma \subseteq \mathbb{R}^d$  is a finite alphabet satisfying  $|\Sigma| > M(W, L, d) \cdot (d + 1)$ , then any ReLU neural network  $\psi : \mathbb{R}^d \to \mathbb{R}^m$  of depth  $L$  and maximal width  $W$  will not be injective on  $\mathcal{M}_{\leq n}(\Sigma)$ . Moreover, for large enough  $n$ , assuming that all points in  $\Sigma$  have rational coordinates, such ReLU networks will also not be moment injective on  $S_{\leq n}(\Sigma)$ .
Proof. Since  $M(W, L, d) < \frac{|\Sigma|}{d + 1}$ , by the pigeonhole principle, we know that there must exist at least one linear region that contains at least  $d + 2$  points in  $\Sigma$ . Denote these points  $x_1, x_2, \ldots, x_{d + 2}$ . These points are affinely dependent, and so there exist weights  $w_1, \ldots, w_{d + 2}$ , not all of which are zero, such that
$$
\sum_ {j = 1} ^ {d + 2} w _ {j} \boldsymbol {x} _ {j} = 0 \text {a n d} \sum_ {j = 1} ^ {d + 2} w _ {j} = 0. \tag {26}
$$
We can now define  $k = \lceil (d + 2) / 2 \rceil$ , and set  $\mu = \sum_{i=1}^{k} w_i \delta_{\pmb{x}_i}$  and  $\mu' = \sum_{j=k+1}^{d+2} (-w_j) \delta_{\pmb{x}_j}$ . According to (25), integration of  $\psi$  over  $\mu$  and over  $\mu'$  gives the same value, while  $\mu \neq \mu'$ , and so  $\psi$  is not moment injective on  $\mathcal{M}_{\leq n}(\Sigma)$ .
We now show that moment injectivity fails also on  $S_{\leq n}(\Sigma)$  when  $n$  is large enough, under the assumption that  $\Sigma \subset \mathbb{Q}^d$ . Under this assumption, the above-mentioned points  $\pmb{x}_1, \pmb{x}_2, \dots, \pmb{x}_{d+2}$  are affinely dependent over  $\mathbb{Q}$ , and there exist rational weights  $w_1, \dots, w_{d+2}$  for which (26) holds.
Now we can multiply (26) by an appropriate integer, so that the  $w_{i}$ s are all integers. Using these new weights, suppose that  $n \geq \sum_{i} |w_{i}|$ . Let  $\mu = \sum_{i: w_{i} \geq 0} w_{i} \delta_{\mathbf{x}_{i}}$  and  $\mu' = \sum_{j: w_{j} < 0} (-w_{j}) \delta_{\mathbf{x}_{j}}$ . Then  $\mu$  and  $\mu'$  are two distinct measures that assign the same moment to  $f$ . Since the weights of  $\mu$  and  $\mu'$  are all natural numbers, these measures correspond to multisets, where the weights denote the number of repetitions of each element.
Number of linear regions Proposition B.4 gives a lower bound on the number of linear regions needed for moment injectivity. The relationship between a neural network's size and the number of its linear regions is well studied [29, 35]. In particular, it is known that the number of linear regions can be exponentially larger than the number of the parameters. As shown in ([1], Theorem D.6), the maximal number of linear regions  $M(W,L,d)$  of a ReLU network with maximal width  $W$ , depth  $L$ , and input dimension  $d$ , is bounded from above by  $CW^{d\cdot L}$ , where  $C > 0$  is a constant. Joining this together with Proposition B.4, we see that we cannot have moment injectivity if
$$
C W ^ {d \cdot L} (d + 1) <   | \Sigma |. \tag {27}
$$
This implies that the number of neurons required for injectivity is at least logarithmic in the cardinality of the alphabet. Although this is only a lower bound, we believe that this bound can be achieved using the techniques developed in [1].
We conclude this discussion by providing an upper bound for the number of linear regions needed in the simple case that  $\Sigma = \{\ell_1 < \ell_2 < \ldots < \ell_S\}$  is a subset of  $\mathbb{R}$ , and the depth  $L$  is one. In this case, our bound (27) states that one cannot attain moment injectivity on  $\mathcal{M}_{\leq n}(\Sigma)$  when  $2C \cdot W < S$ . In the other direction, we show that moment injectivity can be obtained when  $W = S$ .
Proposition B.5. Let  $\Sigma = \{\ell_1 < \ell_2 < \ldots < \ell_S\}$ , and let  $n \leq S$  be some natural number. Then there exists a shallow ReLU network of width  $W = S$  that is moment injective on  $\mathcal{M}_{\leq n}(\Sigma)$ .
Proof. Choose some  $t_1, \ldots, t_S$  such that
$$
t _ {1} <   \ell_ {1} <   t _ {2} <   \ell_ {2} <   \dots <   t _ {S} <   \ell_ {S}.
$$
Consider the shallow ReLU network
$$
x \mapsto \boldsymbol {\psi} (x) = \left( \begin{array}{c} \operatorname {R e L U} (x - t _ {1}) \\ \operatorname {R e L U} (x - t _ {2}) \\ \vdots \\ \operatorname {R e L U} (x - t _ {S}) \end{array} \right)
$$
Suppose that  $\mu = \sum_{i=1}^{S} w_i \delta_{\ell_i}$  and  $\mu' = \sum_{i=1}^{S} w_i' \delta_{\ell_i}$  are two measures such that
$$
\int \psi (x) d \mu (x) = \int \psi (x) d \mu^ {\prime} (x).
$$
The equality of the last coordinate implies
$$
w _ {S} ^ {\prime} \left(\ell_ {S} - t _ {S}\right) = \int \operatorname {R e L U} \left(x - t _ {S}\right) d \mu^ {\prime} (x) = \int \operatorname {R e L U} \left(x - t _ {S}\right) d \mu (x) = w _ {S} \left(\ell_ {S} - t _ {S}\right)
$$
and so  $w_{S} = w_{S}^{\prime}$ . Next, we can consider the equality of the  $(S - 1)$ th coordinate and show that this implies that  $w_{S - 1} = w_{S - 1}^{\prime}$ . Continuing with this argument recursively, we see that the measures must agree.
C Optimal embedding dimension of moment-injective functions
In Appendix C.1, we prove the lower bounds on the embedding dimensions showed in Table 1 for  $\mathcal{M}_{\leq n}(\mathbb{R}^d), S_{\leq n}(\mathbb{R}^d)$  and  $\mathcal{M}_{\leq n}(\Sigma)$  for countable  $\Sigma$ . The remaining lower bounds in the table are equal to one, and thus do not require a proof. For  $S_{\leq n}(\mathbb{R}^d)$ , these bounds are already known [16, 6, 42], but here we present a slightly different proof, which also applies to spaces of measures that were not considered previously. As a rule, for all the spaces of measures/multisets considered, the lower bound equals to the intrinsic dimension of the space, i.e., the number of continuous parameters required to describe it. For example,  $S_{\leq n}(\mathbb{R}^d)$  is parameterized by  $n$  continuous vectors in  $\mathbb{R}^d$ , with a total dimension of  $n \cdot d$ , and by a discrete weight vector that has no influence on the dimension, and thus the lower bound in this case is  $n \cdot d$ .
An interesting question that remains open is whether the gap between the embedding dimension achieved here with shallow neural networks, and the best lower bound, can be closed completely. For multisets with features in  $[0,1]$ , [42] showed that the lower bound of  $n$  moments can be attained using  $n$  polynomial functions. In Appendix C.2, we show that for spaces of measures with features in  $\mathbb{R}$ , the lower bound  $2n$  in Table 1 can be attained by  $2n$  functions that form a  $T$ -system (see discussion below). Examples of T-systems include the functions  $x, x^2, \ldots, x^{2n}$ , as well as  $2n$  univariate sigmoid neural networks with distinct parameters. Thus, for  $d = 1$ , we know that the lower bounds on the embedding dimension are optimal. For  $d > 1$ , it seems that this question is still open.
C.1 Lower bounds
Our lower bounds are based on the following intuitive fact, which is a simple corollary to Brouwer's invariance of domain theorem:
Proposition C.1. If  $U \subseteq \mathbb{R}^D$  is an open set and  $F: U \to \mathbb{R}^M$  is continuous and injective, then  $M \geq D$ .
Proof. Suppose by contradiction that  $M < D$ , and let  $\mathbf{0} \in \mathbb{R}^{D - M}$  be a vector of all zeros. Then the function  $\tilde{F}(\boldsymbol{x}) = (\mathbf{0}, F(\boldsymbol{x}))$  is a continuous injective function of  $U \subseteq \mathbb{R}^D$  into  $\mathbb{R}^D$  and by the invariance of domain theorem ([13], Theorem 2B.3)  $\tilde{F}(U)$  should be open. This leads to a contradiction, since arbitrarily small perturbations of the first coordinate of a point  $(\mathbf{0}, F(\boldsymbol{x}))$  in the image of  $\tilde{F}$  will no longer be in the image.
Based on this proposition, we can prove the lower bounds in Table 1. The following lower bound holds for any alphabet  $\Sigma$ .
Theorem C.2. Let  $\Sigma \subseteq \mathbb{R}^d$  with  $|\Sigma| \geq n$ . Suppose that  $f: \Sigma \to \mathbb{R}^m$  is moment injective on  $\mathcal{M}_{\leq n}(\Sigma)$ . Then  $m \geq n$ .
Proof. Fix  $n$  distinct points  $\pmb{x}_1, \ldots, \pmb{x}_n$  in  $\Sigma$ . The function
$$
\left(w _ {1}, \dots , w _ {n}\right) \mapsto \sum_ {i = 1} ^ {n} w _ {i} \delta_ {\boldsymbol {x} _ {i}}
$$
maps  $\mathbb{R}^n$  injectively into the space of measures  $\mathcal{M}_{\leq n}(\Sigma)$ . Since  $f$  is moment injective, it follows that the continuous (in fact, linear) function
$$
\mathbb {R} ^ {n} \ni (w _ {1}, \dots , w _ {n}) \mapsto \sum_ {i = 1} ^ {n} w _ {i} f (\boldsymbol {x} _ {i}) \in \mathbb {R} ^ {m}
$$
is injective. Thus, by Proposition C.1 we have that  $m \geq n$ .
When the alphabet is uncountable, we can obtain stronger lower bounds. We now show two lower bounds on the embedding dimension required for moment injectivity of a continuous function  $f$  with the alphabet  $\Omega = \mathbb{R}^d$ .
Theorem C.3. Let  $f: \mathbb{R}^d \to \mathbb{R}^m$  be a continuous function. Then
1. If  $f$  is moment injective on  $\mathcal{M}_{\leq n}(\mathbb{R}^d)$ , then  $m \geq n(d + 1)$ .
2. If  $f$  is moment injective on  $\mathcal{S}_{\leq n}(\mathbb{R}^d)$ , then  $m \geq nd$ .
Proof. Choose  $n$  distinct points  $\pmb{y}_1, \dots, \pmb{y}_n \in \mathbb{R}^d$  and let  $r > 0$  be small enough so that the distance between any two distinct points  $\pmb{y}_i, \pmb{y}_j$  is larger than  $2r$ . Then the function
$$
\left(w _ {1}, \dots , w _ {n}, \boldsymbol {x} _ {i}, \dots , \boldsymbol {x} _ {n}\right) \mapsto \sum_ {i = 1} ^ {n} w _ {i} \delta_ {\boldsymbol {x} _ {i}}
$$
maps the open set
$$
\{(w _ {1}, \dots , w _ {n}, \boldsymbol {x} _ {i}, \dots , \boldsymbol {x} _ {n}) \mid w _ {i} \neq 0, | \boldsymbol {x} _ {i} - \boldsymbol {y} _ {i} | <   r, i = 1, \dots , n \}
$$
injectively into  $\mathcal{M}_{\leq n}(\mathbb{R}^d)$ . Therefore, the map
$$
\mathbb {R} ^ {n + d n} \ni (w _ {1}, \dots , w _ {n}, \boldsymbol {x} _ {1}, \dots , \boldsymbol {x} _ {n}) \mapsto \sum_ {i = 1} ^ {n} w _ {i} f (\boldsymbol {x} _ {i}) \in \mathbb {R} ^ {m}
$$
is injective, and since it is also continuous, Proposition C.1 implies that  $m \geq n(d + 1)$ .
For the second part of the theorem, where  $f$  is moment injective on  $S_{\leq n}(\mathbb{R}^d)$ , we can use a similar argument, the only difference being that we fix  $w_i = 1$ . Namely, the map
$$
\mathbb {R} ^ {n d} \ni \left(\boldsymbol {x} _ {1}, \dots , \boldsymbol {x} _ {n}\right) \mapsto \sum_ {i = 1} ^ {n} f \left(\boldsymbol {x} _ {i}\right) \in \mathbb {R} ^ {m}
$$
is continuous and injective on the open set of  $(\pmb{x}_1,\dots ,\pmb{x}_n)$  with  $|\pmb {x}_i - \pmb {y}_i| < r$  for all  $i$ , and therefore by Proposition C.1  $m\geq nd$ .
C.2 T-Systems
We now show how to achieve moment injectivity on  $\mathcal{M}_{\leq n}(\mathbb{R})$  with an optimal embedding dimension. Recall from Table 1 that for this space, our lower bound on the embedding dimension of continuous moment-injective functions is  $2n$ , whereas the embedding dimension of the MLPs constructed using our technique is  $4n + 1$ . We shall now show that this gap can be closed, using the moments of  $2n$  functions that form a  $T$ -system, which we now define.
Definition C.4 ([20]). Let  $\Omega \subseteq \mathbb{R}$ . We say that the  $k$  functions  $\tau_{i}:\Omega \to \mathbb{R}$ ,  $i = 1,\dots ,k$ , form a T-system on  $\Omega$ , if for all pairwise-distinct  $x_{1}\ldots x_{k}\in \Omega$ , the square matrix
$$
M _ {\boldsymbol {\tau}} = \left[ \tau_ {i} \left(x _ {j}\right) \right] _ {1 \leq i, j \leq k} \in \mathbb {R} ^ {k \times k} \tag {28}
$$
is invertible.
An example of a T-system on  $\Omega = \mathbb{R}$  is the standard monomial basis  $\tau_{i}(x) = x^{i - 1}, i = 1, \ldots, k$ . With this choice, (28) is the Vandermonde matrix, which is invertible, and hence the monomial basis is a T-system.
We know that the moments of the first  $n$  elements of the standard monomial basis are injective on  $S_{\leq n}(\mathbb{R})$ . The following simple proposition shows that injectivity on all of  $\mathcal{M}_{\leq n}(\mathbb{R})$  can be achieved, at the cost of increasing the number of moments to  $2n$ . Moreover, this is true for all T-systems.
Proposition C.5 (Trivial, based on [20]). If  $\pmb{\tau} = (\tau_{1},\dots,\tau_{2n})$  is a  $\pmb{T}$ -system on  $\Omega \subseteq \mathbb{R}$ , then the induced moment function
$$
\hat {\boldsymbol {\tau}} (\mu) = \left(\int_ {\mathbb {R}} \tau_ {1} (x) d \mu (x), \dots , \int_ {\mathbb {R}} \tau_ {2 n} (x) d \mu (x)\right)
$$
is injective on  $\mathcal{M}_{\leq n}(\Omega)$
Proof. Let  $\mu, \mu' \in \mathcal{M}_{\leq n}(\Omega)$  such that  $\hat{\tau}(\mu) = \hat{\tau}(\mu')$ . Then  $\tau(\mu - \mu') = 0$ , where  $\mu - \mu'$  denotes the difference measure. Since  $\mu - \mu'$  is supported on at most  $2n$  points, we can write
$$
\mu - \mu^ {\prime} = \sum_ {i = 1} ^ {2 n} w _ {i} \delta_ {\boldsymbol {x} _ {i}},
$$
where the points  $\pmb{x}_i$  are pairwise distinct, and some of the  $w_i$  may be zero. Our goal is to show that the vector
$$
\boldsymbol {w} = (w _ {1}, \dots , w _ {2 n})
$$
is all zero. Indeed, the fact that  $\hat{\tau} (\mu -\mu^{\prime}) = 0$  implies that  $M_{\tau}\cdot \boldsymbol {w} = 0$ . And since by assumption  $M_{\tau}$  is full rank, we have that  $\boldsymbol {w} = 0$ .
Example C.6 (Sigmoid T-systems). We've seen that the standard monomial basis forms a T-system. Another well-known example [20] is the family of functions
$$
\tau_ {i} (x) = \frac {1}{x - a _ {i}}, \quad i = 1, \ldots , k.
$$
If the numbers  $a_1, \ldots, a_k$  are pairwise distinct, then  $\pmb{\tau} = (\tau_1, \dots, \tau_k)$  form a T-system on  $\Omega = \mathbb{R} \setminus \{a_1, \dots, a_k\}$ .
We can use these examples to obtain an alternative T-system based on the sigmoid activation  $\sigma(x) = (1 + e^{-x})^{-1}$ . Firstly, using the injectivity of the exponent function, we can deduce from the fact that  $\tau$  is a T-system, that for any distinct  $b_1, \ldots, b_k$  the functions
$$
\hat {\tau} _ {i} (x) = \frac {1}{e ^ {b _ {i}} + e ^ {- x}}, \quad i = 1, \dots , k
$$
form a T-system on  $\Omega = \mathbb{R}$ . It follows that the functions
$$
\sigma (x + b _ {i}) = \frac {1}{1 + e ^ {- x - b _ {i}}} = e ^ {- b _ {i}} \left(\frac {1}{e ^ {b _ {i}} + e ^ {- x}}\right), \quad i = 1, \ldots , k
$$
form a T-system as well. In particular, for all pairwise-distinct  $b_{1}, \ldots, b_{2n}$ , the function
$$
x \mapsto \sigma (x + b _ {i}), \quad i = 1, \dots , 2 n
$$
is moment injective on  $\mathcal{M}_{\leq n}(\mathbb{R})$
Unfortunately, these results cannot be extended to  $\mathcal{M}_{\leq n}(\mathbb{R}^d)$  with  $d > 1$ , as it is known that T-systems can only be defined on subsets of  $\mathbb{R}$  or  $S^1$  [20].
In this section, we provide the proofs omitted from the main text, except for the proofs of Theorem 3.4 and Proposition 3.6, which appear in Appendix A.6.
D.1 Proofs for Section 3