SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Generalising Sussmann's equivalence theorem. The proofs in the main paper rely on the results of Sussmann (1992) on the equivalence between reducibility and non-minimality, and the fact that irreducible functionally equivalent parameters are related by unit negation and exchange transformations. Sussmann (1992) studied a setting with multiple input units but only a single output unit. Lemmas A.1 and A.2 generalise these results to the multi-output setting. $^{1}$  The final necessary change to generalise the results in the main paper is to replace all references to Sussmann's results with references to Lemma A.1 or Lemma A.2.
<sup>1</sup>The proofs reduce the multi-output case to the single-output case, so they still rely on the results of Sussmann (1992). A generalisation similar to Lemma A.1 is given by Fukumizu (1996).
The definitions of unit negation and exchange transformations, reducibility, and non-minimality all generalise to arbitrary  $n$  and  $m$  with the above-mentioned changes. These definitions are repeated here for convenience.
A unit negation transformation is a function  $\sigma_{i}:\mathcal{W}_{h}^{n,m}\to \mathcal{W}_{h}^{n,m}$  for  $i = 1,\ldots ,h$ , where
$$
\sigma_ {i} \left(a _ {1}, b _ {1}, c _ {1}, \dots , a _ {h}, b _ {h}, c _ {h}, d\right) = \left(a _ {1}, b _ {1}, c _ {1}, \dots , - a _ {i}, - b _ {i}, - c _ {i}, \dots , a _ {h}, b _ {h}, c _ {h}, d\right).
$$
A unit exchange transformation is a function  $\tau_{i,j}:\mathcal{W}_h^{n,m}\to \mathcal{W}_h^{n,m}$  for  $i,j = 1,\ldots ,h$ , where
$$
\begin{array}{l} \tau_ {i, j} \left(a _ {1}, b _ {1}, c _ {1}, \dots , a _ {h}, b _ {h}, c _ {h}, d\right) = \left(a _ {1}, b _ {1}, c _ {1}, \dots , c _ {i - 1}, a _ {j}, b _ {j}, c _ {j}, a _ {i + 1}, \right. \\ \ldots , c _ {j - 1}, a _ {i}, b _ {i}, c _ {i}, a _ {j + 1}, \ldots , a _ {h}, b _ {h}, c _ {h}, d). \\ \end{array}
$$
A parameter  $w = (a_{1}, b_{1}, c_{1}, \ldots, a_{h}, b_{h}, c_{h}, d) \in \mathcal{W}_{h}^{n,m}$  is reducible if and only if it satisfies any of the following conditions (otherwise,  $w$  is irreducible):
(i)  $a_{i} = 0$  for some  $i$ ,
(ii)  $b_{i} = 0$  for some  $i$ ,
(iii)  $(b_{i},c_{i}) = (b_{j},c_{j})$  for some  $i\neq j$  ,or
(iv)  $(b_{i},c_{i}) = (-b_{j}, - c_{j})$  for some  $i\neq j$
A parameter  $w \in \mathcal{W}_h^{n,m}$  is non-minimal if and only if  $w$  is functionally equivalent to some  $w' \in \mathcal{W}_{h'}^{n,m}$  with fewer hidden units  $h' < h$ .
Lemma A.1. For  $w \in \mathcal{W}_h^{n,m}$ ,  $w$  is reducible if and only if  $w$  is non-minimal.
Proof.  $(\Rightarrow)$ : A smaller functionally equivalent parameter can be constructed as follows.
(i) If  $a_i = 0$  for some  $i$ , then hidden unit  $i$  fails to contribute to the function. Construct a functionally equivalent parameter  $w' \in \mathcal{W}_{h-1}^{n,m}$  with hidden unit  $i$  omitted:
$$
w ^ {\prime} = \left(a _ {1}, b _ {1}, c _ {1}, \dots , a _ {i - 1}, b _ {i - 1}, c _ {i - 1}, a _ {i + 1}, b _ {i + 1}, c _ {i + 1}, \dots , a _ {h}, b _ {h}, c _ {h}, d\right).
$$
(ii) If  $b_{i} = 0$  for some  $i$ , then hidden unit  $i$  contributes only a constant to the function. Construct a functionally equivalent parameter  $w' \in \mathcal{W}_{h-1}^{n,m}$  with hidden unit  $i$  omitted and the output unit bias vector changed to compensate:
$$
w ^ {\prime} = \left(a _ {1}, b _ {1}, c _ {1}, \dots , a _ {i - 1}, b _ {i - 1}, c _ {i - 1}, a _ {i + 1}, b _ {i + 1}, c _ {i + 1}, \dots , a _ {h}, b _ {h}, c _ {h}, d + a _ {i} \tanh  (c _ {i})\right).
$$
(iii) If  $(b_i, c_i) = (b_j, c_j)$  for some  $i \neq j$ , then hidden units  $i$  and  $j$  contribute proportionately. They can be combined into a single unit (say,  $j$ ) with the same incoming weights and bias, and a combined outgoing weight vector. Construct a functionally equivalent parameter  $w' \in \mathcal{W}_{h-1}^{n,m}$  accordingly:
$$
w ^ {\prime} = \left(a _ {1}, b _ {1}, c _ {1}, \dots , c _ {i - 1}, a _ {i + 1}, \dots , c _ {j - 1}, a _ {j} + a _ {i}, b _ {j}, c _ {j}, a _ {j + 1}, \dots , a _ {h}, b _ {h}, c _ {h}, d\right).
$$
(iv) If  $(b_{i},c_{i}) = -(b_{j},c_{j})$  for some  $i\neq j$ , then hidden units  $i$  and  $j$  contribute in negative proportion. Due to the odd property of tanh they can be combined into a single unit (say,  $j$ ) with incoming weight and bias vectors  $(b_{j},c_{j})$  and a combined outgoing weight vector. Construct a new parameter  $w^{\prime}\in \mathcal{W}_{h - 1}^{n,m}$  accordingly:
$$
w ^ {\prime} = \left(a _ {1}, b _ {1}, c _ {1}, \dots , c _ {i - 1}, a _ {i + 1}, \dots , c _ {j - 1}, a _ {j} - a _ {i}, b _ {j}, c _ {j}, a _ {j + 1}, \dots , a _ {h}, b _ {h}, c _ {h}, d\right).
$$
In all cases, the new parameter  $w' \in \mathcal{W}_{h-1}^{n,m}$  has  $f_{w'} = f_w$ , so  $w$  is non-minimal.
$(\Leftarrow)$ : We reduce to the single-output case and apply the result of Sussmann (1992) to show that  $w$  satisfies at least one of the reducibility conditions.
To reduce to the single-output case, we introduce some notation. From the function  $f_{w}:\mathbb{R}^{n}\to \mathbb{R}^{m}$  define a series of component functions  $f_{w}^{(1)},f_{w}^{(2)},\ldots ,f_{w}^{(m)}:\mathbb{R}^{n}\to \mathbb{R}$  such that for  $x\in \mathbb{R}^n$ ,
$$
f _ {w} (x) = \left(f _ {w} ^ {(1)} (x), f _ {w} ^ {(2)} (x), \dots , f _ {w} ^ {(m)} (x)\right).
$$
$f_{w}^{(1)}$
$f_w^{(2)}$
$f_w^{(m)}$
Figure 3: The connection graphs of the component functions of  $f_{w}$ . Included units and weights are solid. The hidden units of each network share the same incoming weights (and biases, not shown).
Each of these component functions is a simple neural network function in an architecture with  $n$  input units and 1 output unit, corresponding to a subgraph of the connection graph of the original neural network, as illustrated in Figure 3.
Denote the corresponding (overlapping) subvectors of  $w \in \mathcal{W}_h^{n,m}$  as  $w_{(1)},\ldots ,w_{(m)} \in \mathcal{W}_h^{n,1}$ . That is, for  $\mu = 1,\dots ,m$ ,
$$
w _ {(\mu)} = \left(a _ {1, \mu}, b _ {1}, c _ {1}, \dots , a _ {h, \mu}, b _ {h}, c _ {h}, d _ {\mu}\right) \in \mathcal {W} _ {h} ^ {n, 1}.
$$
Now, let  $w' = (a_1', b_1', c_1', \ldots, a_h', b_h', c_h', d') \in \mathcal{W}_{h'}$  such that  $f_{w'} = f_w$  where  $h'$  is the smallest number of hidden units required to implement  $f_w$  ( $h' < h$  by assumption of non-minimality). Apply the same decomposition to  $f_{w'}$  to define  $f_{w'}^{(1)}, \ldots, f_{w'}^{(m)}$ , and to define  $w_{(1)}', \ldots, w_{(m)}' \in \mathcal{W}_{h'}^{n,1}$ .
Apply the results of Sussmann (1992) as follows. Since  $f_{w} = f_{w^{\prime}}$ ,  $f_{w}^{(\mu)} = f_{w^{\prime}}^{(\mu)}$  for  $\mu = 1, \ldots, m$ . It follows that for each  $w_{(\mu)}, w_{(\mu)}^{\prime}$  is a functionally equivalent parameter using fewer units. Therefore, the reducibility conditions (in the special case of  $m = 1$ ) must hold for each  $w_{(\mu)}$  (Sussmann, 1992).
Since conditions (ii-iv) only depend on incoming weights and biases, if any of these conditions hold for any  $w_{(\mu)}$ , then they must also hold for  $w$  itself (which shares the same incoming weights and biases), and the proof is complete. It remains only to consider the case in which conditions (ii-iv) fail to hold for any  $w_{(\mu)}$ , and to show that condition (i) holds for  $w$  itself in this case.
We must introduce yet further notation. For  $i = 1,\dots,h$  denote by  $\varphi_{i}:\mathbb{R}^{n}\to \mathbb{R}$  the function  $\varphi_{i}(x) = \tanh (b_{i}x + c_{i})$ . Similarly for  $j = 1,\ldots ,h^{\prime}$  denote by  $\psi_j:\mathbb{R}^n\rightarrow \mathbb{R}$  the function  $\psi_j(x) = \tanh (b_j'x + c_j')$ . Then, since we have ruled out reducibility conditions (ii-iv) for  $w$ , no  $\varphi_{i}$  is constant (ii) and no two are proportional (iii, iv). The same holds for the  $\psi_{j}$ -conditions (i-iv) do not hold for  $w_{(\mu)}'$ , since  $h^\prime$  was assumed to be minimal. Yet, for  $\mu = 1,\dots,m$ , the linear combination of functions
$$
d _ {\mu} + \sum_ {i = 1} ^ {h} a _ {i, \mu} \varphi_ {i} - d _ {\mu} ^ {\prime} - \sum_ {j = 1} ^ {h ^ {\prime}} a _ {j, \mu} ^ {\prime} \psi_ {j} = f _ {w} ^ {(\mu)} - f _ {w ^ {\prime}} ^ {(\mu)} = 0
$$
yields the zero function. This linear combination remains when excluding those terms with  $a_{i,\mu} = 0$  or  $a_{j,\mu}' = 0$ . Applying the same reasoning as that in Sussmann (1992), due to the independence property of the hyperbolic tangent function (Sussmann, 1992, Lemma 3.1) the remaining terms must be in bijection, such that
$$
\varphi_ {i} = \pm \psi_ {j} \tag {3}
$$
for some  $j$  with  $a_{j,\mu}^{\prime}\neq 0$  for each  $i$  with  $a_{i,\mu}\neq 0$
To complete the proof, note that these relationships (3) between the units of  $w$  and  $w'$  are independent of  $\mu$ . However, the relationships are "exclusive" in the sense that no two  $\varphi_i$  can be proportional to the same  $\psi_j$ , else they would also be proportional to each other (ruled out above). Since there are only  $h'$  units  $\psi_1, \ldots, \psi_{h'}$ , it follows that there must be one hidden unit  $i$  (actually at least  $h - h'$  many units) for which  $a_{i,\mu} = 0$  for all  $\mu = 1, \ldots, m$  (allowing  $\varphi_i$  to avoid any such relationship). That is,  $a_i = (a_{i,1}, \ldots, a_{i,m}) = 0$ , satisfying condition (i) for  $w$  as required.
Lemma A.2. Let  $w \in \mathcal{W}_h^{n,m}$  be irreducible, and let  $w' \in \mathcal{W}_h^{n,m}$ . If  $w$  and  $w'$  are functionally equivalent then there exists a compositional chain of unit negation and exchange transformations, collectively a transformation  $T: \mathcal{W}_h^{n,m} \to \mathcal{W}_h^{n,m}$ , such that  $w' = T(w)$ .
Proof. Once again, we reduce to the case  $m = 1$  and appeal to Sussmann (1992).
Suppose  $w' \in \mathfrak{F}[w]$ . Introduce the same decomposition of the two neural networks as in the proof of Lemma A.1, namely, the component functions  $f_w^{(1)}, \ldots, f_w^{(m)}, f_{w'}^{(1)}, \ldots, f_{w'}^{(m)}$  implemented by the parameter subvectors  $w_{(1)}, \ldots, w_{(m)}, w_{(1)}', \ldots, w_{(m)}' \in \mathcal{W}_h^{n,1}$  (cf. Figure 3).
For  $\mu = 1,\ldots ,m$ , since  $f_{w} = f_{w^{\prime}}$ , we have that  $f_{w}^{(\mu)} = f_{w^{\prime}}^{(\mu)}$ . Now,  $w_{(\mu)}$  and  $w_{(\mu)}^{\prime}$  are not necessarily irreducible, but if they are reducible then it is only by condition (i), since  $w_{(\mu)}$  and  $w_{(\mu)}^{\prime}$  have the incoming weights and biases of  $w$  and  $w^{\prime}$  respectively ( $w$  is irreducible by assumption;  $w^{\prime}$  is irreducible because, with the same number of units as  $w$ , it is necessarily minimal, and irreducibility follows by Lemma A.1). Remove such units with zero outgoing weight from  $w_{(\mu)}$  and  $w_{(\mu)}^{\prime}$  to produce new, functionally equivalent irreducible parameters  $u_{(\mu)}, u_{(\mu)}^{\prime} \in \mathcal{W}_{\mathrm{rank}(w_{(\mu)})}^{n,1}$ . Now by Sussmann (1992, Theorem 2.1) there exists a chain of unit negation and exchange transformations  $T_{\mu}$  such that  $u_{(\mu)} = T_{\mu}(u_{(\mu)}^{\prime})$ .
For each  $\mu$ ,  $T_{\mu}$  implies a relationship between the units of  $w_{(\mu)}$  and  $w_{(\mu)}'$  with nonzero outgoing weights, including possible negations and permutations of these units. This same relationship must hold between those units of  $w$  and  $w'$  since they share incoming weights and biases with  $w_{(\mu)}$  and  $w_{(\mu)}'$ , and (since  $w$  is irreducible, conditions (ii-iv)) these incoming weights are nonzero and the incoming weight and bias vectors are absolutely distinct between units of the same parameter. Moreover, all units are involved in some such relationship because no unit of  $w$  or  $w'$  can have zero outgoing weight vector by reducibility condition (i).
So, one can construct from these implied relationships a composition of unit negation and exchange transformations relating  $w$  and  $w'$  as required.