SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Matthew Farrugia-Roberts  
School of Computing and Information Systems  
The University of Melbourne  
matthew@far.in.net
Understanding the learning process of artificial neural networks requires clarifying the structure of the parameter space within which learning takes place. A neural network parameter's functional equivalence class is the set of parameters implementing the same input-output function. For many architectures, almost all parameters have a simple and well-documented functional equivalence class. However, there is also a vanishing minority of reducible parameters, with richer functional equivalence classes caused by redundancies among the network's units. In this paper, we give an algorithmic characterisation of unit redundancies and reducible functional equivalence classes for a single-hidden-layer hyperbolic tangent architecture. We show that such functional equivalence classes are piecewise-linear path-connected sets, and that for parameters with a majority of redundant units, the sets have a diameter of at most 7 linear segments.
Deep learning algorithms construct neural networks through a local search in a high-dimensional parameter space. This search is guided by the shape of some loss landscape, which is in turn determined by the link between neural network parameters and their input-output functions. Thus, understanding the link between parameters and functions is key to understanding deep learning.
It is well known that neural network parameters often fail to uniquely determine an input-output function. For example, exchanging weights between two adjacent hidden units generally preserves functional equivalence (Hecht-Nielsen, 1990). For many architectures, almost all parameters have a simple class of functionally equivalent parameters. These classes have been characterised for multi-layer feed-forward architectures with various nonlinearities (e.g., Sussmann, 1992; Albertini et al., 1993; Kürkova and Kainen, 1994; Phuong and Lampert, 2020; Vlačić and Bölskei, 2021).
However, all existing work on functional equivalence excludes from consideration certain measure zero sets of parameters, for which the functional equivalence classes may be richer. One such family of parameters is the so-called reducible parameters. These parameters display certain structural redundancies, such that the same function could be implemented with fewer hidden units (Sussmann, 1992; Vlačić and Bölskei, 2021), leading to a richer functional equivalence class.
Despite their atypicality, reducible parameters may play an important role in deep learning. Learning exerts non-random selection pressure, so measure zero sets of parameters may still arise in practice, and indeed reducible parameters are appealing solutions due to parsimony (cf. Farrugia-Roberts, 2023). These parameters are also a source of information singularities (cf. Fukumizu, 1996), relevant to statistical theories of deep learning (Watanabe, 2009; Wei et al., 2022). Moreover, the structure of functional equivalence classes has implications for the local and global shape of the loss landscape, in ways that may influence learning dynamics near reducible parameters.
In this paper, we study functional equivalence classes for single-hidden-layer networks with the hyperbolic tangent nonlinearity, building on the foundational work of Sussmann (1992) on reducibility in this setting. We offer the following theoretical contributions.
1. In Section 4, we give a formal algorithm producing a canonical representative parameter from any functional equivalence class, by systematically eliminating all sources of structural redundancy. This extends prior algorithms that only handle irreducible parameters.
2. In Section 5, we invert this canonicalisation algorithm to characterise the functional equivalence class of any parameter as a union of simple parameter manifolds. This characterisation extends the well-known result for irreducible parameters.
3. We show that in the reducible case, the functional equivalence class is a piecewise-linear path-connected set—that is, any two functionally equivalent reducible parameters are connected by a piecewise linear path comprising only equivalent parameters (Theorem 6.1).
4. We show that if a parameter has a high degree of reducibility (in particular, if the same function can be implemented using half of the available hidden units), then the number of linear segments required to connect any two equivalent parameters is at most 7 (Theorem 6.3).
While the single-hidden-layer hyperbolic tangent architecture is not immediately relevant to modern deep learning, it enables the first comprehensive analysis of neural network structural redundancy. Moreover, feed-forward layers are a fundamental building block of many modern architectures, and so our analysis is partially informative for such extensions. In Section 7 we discuss such extensions and others, as well as connections to other topics in deep learning including loss landscape analysis, model compression, and singular learning theory.
Sussmann (1992) studied functional equivalence in single-hidden-layer hyperbolic tangent networks, showing that two irreducible parameters are functionally equivalent if and only if they are related by simple operations of exchanging and negating the weights of hidden units. This result was later extended to architectures with a broader class of nonlinearities (Albertini et al., 1993; Kürková and Kainen, 1994), to architectures with multiple hidden layers (Fefferman and Markel, 1993; Fefferman, 1994), and to certain recurrent architectures (Albertini and Sontag, 1992, 1993a,b,c). More recently, similar results have been found for ReLU networks (Phuong and Lampert, 2020; Bona-Pellissier et al., 2021; Stock and Gribonval, 2022), and Vlăcic and Bölsceki (2021, 2022) have generalised Sussmann's results to a very general class of architectures and nonlinearities. However, all of these results have come at the expense of excluding from consideration certain measure zero subsets of parameters with richer functional equivalence classes.
A similar line of work has documented the global symmetries of the parameter space—bulk transformations of the entire parameter space that preserve all implemented functions. The search for such symmetries was launched by Hecht-Nielsen (1990). Chen et al. (1993, also Chen and Hecht-Nielsen, 1991) showed that in the case of multi-layer hyperbolic tangent networks, all analytic symmetries are generated by unit exchanges and negations. Rüger and Ossen (1997) extended this result to additional sigmoidal nonlinearities. The analyticity condition excludes discontinuous symmetries acting selectively on, say, reducible parameters with richer equivalence classes (Chen et al., 1993).
Rüger and Ossen (1997) provide a canonicalisation algorithm. Their algorithm negates each hidden unit's weights until the bias is positive, and then sorts each hidden layer's units into non-descending order by bias weight. This algorithm is invariant precisely to the exchanges and negations mentioned above, but fails to properly canonicalise equivalent parameters that differ in more complex ways.
To our knowledge, there is one line of work bearing directly on the topic of the functional equivalence classes of reducible parameters. Fukumizu and Amari (2000) and Fukumizu et al. (2019) have catalogued methods of adding a single hidden unit to a neural network while preserving the network's function, and Şimsek et al. (2021) have extended this work to consider the addition of multiple hidden units. Though derived under a distinct framing, it turns out that the subsets of parameter space accessible by such unit additions correspond to functional equivalence classes, similar to those we study (though in a slightly different architecture). We note these similarities, especially regarding our contributions (2) and (3), in Remarks 5.4 and 5.5 and Remark 6.2.
We consider a family of fully-connected, feed-forward neural network architectures with a single input unit, a single biased output unit, and a single hidden layer of  $h \in \mathbb{N}$  biased hidden units with the hyperbolic tangent nonlinearity  $\tanh(z) = (e^z - e^{-z}) / (e^z + e^{-z})$ . Such an architecture has a parameter space  $\mathcal{W}_h = \mathbb{R}^{3h+1}$ . Our results generalise directly to networks with multi-dimensional inputs and outputs, as detailed in Appendix A.
The weights and biases of the network's units are encoded in the parameter vector in the format  $(a_{1},b_{1},c_{1},\ldots ,a_{h},b_{h},c_{h},d)\in \mathcal{W}_{h}$  where for each hidden unit  $i = 1,\dots ,h$  there is an outgoing weight  $a_i\in \mathbb{R}$ , an incoming weight  $b_{i}\in \mathbb{R}$ , and a bias  $c_{i}\in \mathbb{R}$ , and  $d\in \mathbb{R}$  is an output unit bias. Thus each parameter  $w = (a_{1},b_{1},c_{1},\ldots ,a_{h},b_{h},c_{h},d)\in \mathcal{W}_{h}$  indexes a mathematical function  $f_w:\mathbb{R}\to \mathbb{R}$  defined as follows:
$$
f _ {w} (x) = d + \sum_ {i = 1} ^ {h} a _ {i} \tanh (b _ {i} x + c _ {i}).
$$
Two parameters  $w \in \mathcal{W}_h, w' \in \mathcal{W}_{h'}$  are functionally equivalent if and only if  $f_w = f_{w'}$  as functions on  $\mathbb{R}$  (that is,  $\forall x \in \mathbb{R}, f_w(x) = f_{w'}(x)$ ). Functional equivalence is of course an equivalence relation on  $\mathcal{W}_h$ . Given a parameter  $w \in \mathcal{W}_h$ , the functional equivalence class of  $w$ , denoted  $\mathfrak{F}[w]$ , is the set of all parameters in  $\mathcal{W}_h$  that are functionally equivalent to  $w$ :
$$
\mathfrak {F} [ w ] = \left\{w ^ {\prime} \in \mathcal {W} _ {h} \mid f _ {w} = f _ {w ^ {\prime}} \right\}.
$$
For this family of architectures, the functional equivalence class of almost all parameters is a discrete set fully characterised by simple unit negation and exchange transformations  $\sigma_{i},\tau_{i,j}:\mathcal{W}_{h}\to \mathcal{W}_{h}$  for  $i,j = 1,\ldots ,h$ , where
$$
\begin{array}{l} \sigma_ {i} \left(a _ {1}, b _ {1}, c _ {1}, \dots , a _ {h}, b _ {h}, c _ {h}, d\right) = \left(a _ {1}, b _ {1}, c _ {1}, \dots , - a _ {i}, - b _ {i}, - c _ {i}, \dots , a _ {h}, b _ {h}, c _ {h}, d\right) \\ \tau_ {i, j} \left(a _ {1}, b _ {1}, c _ {1}, \dots , a _ {h}, b _ {h}, c _ {h}, d\right) = \left(a _ {1}, b _ {1}, c _ {1}, \dots , c _ {i - 1}, a _ {j}, b _ {j}, c _ {j}, a _ {i + 1}, \right. \\ \ldots , c _ {j - 1}, a _ {i}, b _ {i}, c _ {i}, a _ {j + 1}, \ldots , a _ {h}, b _ {h}, c _ {h}, d). \\ \end{array}
$$
More formally, these transformations generate the full functional equivalence class for all so-called irreducible parameters (Sussmann, 1992). A parameter  $w = (a_{1}, b_{1}, c_{1}, \ldots, a_{h}, b_{h}, c_{h}, d) \in \mathcal{W}_{h}$  is reducible if and only if it satisfies any of the following conditions (otherwise,  $w$  is irreducible):
(i)  $a_{i} = 0$  for some  $i$ , or
(ii)  $b_{i} = 0$  for some  $i$ , or
(iii)  $(b_{i},c_{i}) = (b_{j},c_{j})$  for some  $i\neq j$  , or
(iv)  $(b_{i},c_{i}) = (-b_{j}, - c_{j})$  for some  $i\neq j$
Sussmann (1992) also showed that in this family of architectures, reducibility corresponds to non-minimality: a parameter  $w \in \mathcal{W}_h$  is reducible if and only if  $w$  is functionally equivalent to some  $w' \in \mathcal{W}_{h'}$  with fewer hidden units  $h' < h$ . We define the rank of  $w$ , denoted  $\mathrm{rank}(w)$ , as the minimum number of hidden units required to implement  $f_w$ :
$$
\operatorname {r a n k} (w) = \min  \left\{h ^ {\prime} \in \mathbb {N} \mid \exists w ^ {\prime} \in \mathcal {W} _ {h ^ {\prime}}; f _ {w} = f _ {w ^ {\prime}} \right\}.
$$
Finally, we make use of the following notions of connectivity for a set of parameters. Given a set  $W \subseteq \mathcal{W}_h$ , define a piecewise linear path in  $W$  as a continuous function  $\rho : [0,1] \to W$  comprising a finite number of linear segments. Two parameters  $w, w' \in \mathcal{W}_h$  are piecewise-linear path-connected in  $W$ , denoted  $w \leftrightarrow w'$  (with  $W$  implicit), if there exists a piecewise linear path in  $W$  such that  $\rho(0) = w$  and  $\rho(1) = w'$ . Note that  $\leftrightarrow$  is an equivalence relation on  $W$ . A set  $W \subseteq \mathcal{W}_h$  is itself piecewise-linear path-connected if and only if  $\leftrightarrow$  is the full relation, that is, all pairs of parameters in  $W$  are piecewise linear path-connected in  $W$ .
The length of a piecewise linear path is the number of maximal linear segments comprising the path. The distance between two piecewise linear path-connected parameters is the length of the shortest path connecting them. The diameter of a piecewise linear path-connected set is the largest distance between any two parameters in the set.
4 Parameter Canonicalisation
A parameter canonicalisation algorithm maps each parameter in a functional equivalence class to a consistent representative parameter within that class. A canonicalisation algorithm serves as a theoretical test of functional equivalence. In Section 5 we invert our canonicalisation algorithm to characterise the functional equivalence class. More practically, canonicalising parameters before measuring the distance between them yields a metric that is independent of functionally irrelevant details such as unit permutations.
Prior work has described canonicalisation algorithms for certain irreducible parameters (Rüger and Ossen, 1997); but when applied to functionally equivalent reducible parameters, such algorithms may fail to produce the same output. We introduce a canonicalisation algorithm that properly canonicalises both reducible and irreducible parameters, based on similar negation and sorting stages, combined with a novel reduction stage. This stage effectively removes or 'zeros out' redundant units through various operations that exploit the reducibility conditions. This process isolates a functionally equivalent but irreducible subparameter.
Algorithm 4.1 (Parameter canonicalisation). Given a parameter space  $\mathcal{W}_h$ , proceed:
1: procedure CANONICALISE  $(w = (a_{1},b_{1},c_{1},\dots ,a_{h},b_{h},c_{h},d)\in \mathcal{W}_{h})$   
2: ▷ Stage 1: Reduce the parameter, zeroing out redundant hidden units  
3:  $Z\gets \{\}$  ▷ keep track of 'zeroed' units  
4: while any of the following four conditions hold do  
5: if for some hidden unit  $i\notin Z,a_i = 0$  then ▷ reducibility condition (i)  
6:  $b_{i},c_{i}\gets 0$   
7:  $Z\gets Z\cup \{i\}$   
8: else if for some hidden unit  $i\notin Z,b_{i} = 0$  then ▷ —— (ii)  
9:  $d\gets d + a_{i}\tanh(c_{i})$   
10:  $a_{i},c_{i}\gets 0$   
11:  $Z\gets Z\cup \{i\}$   
12: else if for some hidden units  $i,j\notin Z,i\neq j,(b_i,c_i) = (b_j,c_j)$  then ▷ —— (iii)  
13:  $a_{j}\gets a_{j} + a_{i}$   
14:  $a_{i},b_{i},c_{i}\gets 0$   
15:  $Z\gets Z\cup \{i\}$   
16: else if for some hidden units  $i,j\notin Z,i\neq j,(b_i,c_i) = (-b_j,-c_j)$  then ▷ —— (iv)  
17:  $a_{j}\gets a_{j} - a_{i}$   
18:  $a_{i},b_{i},c_{i}\gets 0$   
19:  $Z\gets Z\cup \{i\}$   
20: end if  
21: end while  
22: ▷ Stage 2: Negate the nonzero units to have positive incoming weights  
23: for each hidden unit  $i\notin Z$  do  
24:  $a_{i},b_{i},c_{i}\gets \mathrm{sign}(b_{i})\cdot (a_{i},b_{i},c_{i})$   
25: end for  
26: ▷ Stage 3: Sort the units by their incoming weights and biases  
27:  $\pi \gets$  a permutation sorting  $i = 1,\ldots ,h$  by decreasing  $b_{i}$ , breaking ties with decreasing  $c_{i}$   
28:  $w\gets (a_{\pi (1)},b_{\pi (1)},c_{\pi (1)},\ldots ,a_{\pi (h)},b_{\pi (h)},c_{\pi (h)},d)$   
29: ▷ Now,  $w$  has been mutated into the canonical equivalent parameter  
30: return  $w$   
31: end procedure
The following theorem establishes the correctness of Algorithm 4.1.
Theorem 4.2. Let  $w, w' \in \mathcal{W}_h$ . Let  $v = \mathrm{CANONICALISE}(w)$  and  $v' = \mathrm{CANONICALISE}(w')$ . Then (i)  $v$  is functionally equivalent to  $w$ ; and (ii) if  $w$  and  $w'$  are functionally equivalent, then  $v = v'$ .
Proof. For (i), observe that  $f_w$  is maintained by each iteration of the loops in Stages 1 and 2, and by the permutation in Stage 3. For (ii), observe that Stage 1 isolates functionally equivalent and irreducible subparameters  $u \in \mathcal{W}_r$  and  $u' \in \mathcal{W}_{r'}$  of the input parameters  $w$  and  $w'$  (excluding the zeroed units). We have  $f_u = f_w = f_{w'} = f_{u'}$ , so by the results of Sussmann (1992),  $r = r' = \mathrm{rank}(w)$ , and  $u$  and  $u'$  are related by unit negation and exchange transformations. This remains true in the presence of the zero units. Stages 2 and 3 are invariant to precisely such transformations.
5 Full Functional Equivalence Class
Algorithm 4.1 produces a consistent output for all parameters within a given functional equivalence class. It serves as the basis for the following characterisation of the full functional equivalence class.
The idea behind the characterisation is to enumerate the various ways for a parameter's units to be reduced, negated, and sorted throughout Algorithm 4.1. Each such canonicalisation trace corresponds to a simple set of parameters that takes exactly this path through the algorithm, as follows.
Definition 5.1 (Canonicalisation trace). Let  $r, h \in \mathbb{N}, r \leq h$ . A canonicalisation trace of order  $r$  on  $h$  units is a tuple  $(\sigma, \tau)$ , where  $\sigma \in \{-1, +1\}^h$  is a sign vector (interpreted as tracking unit negation throughout the algorithm); and  $\tau : \{1, \dots, h\} \to \{0, 1, \dots, h\}$  is a function with range including  $\{1, \dots, r\}$  (interpreted as tracking unit reduction and permutation throughout the algorithm).
Theorem 5.2. Let  $w \in \mathcal{W}_h$  and  $v = (\alpha_1, \beta_1, \gamma_1, \ldots, \alpha_h, \beta_h, \gamma_h, \delta) = \mathrm{CANONICALISE}(w)$ . Let  $r = \mathrm{rank}(w)$ . Then the functional equivalence class  $\mathfrak{F}[w] \subset \mathcal{W}_h$  is a union of subsets
$$
\mathfrak {F} [ w ] = \bigcup_ {(\sigma , \tau) \in \Gamma (h, r)} \left(X _ {\tau^ {- 1} [ 0 ]} ^ {\delta} \cap \bigcap_ {i = 1} ^ {r} Y _ {\sigma , \tau^ {- 1} [ i ]} ^ {\alpha_ {i}, \beta_ {i}, \gamma_ {i}} \cap \bigcap_ {i = r + 1} ^ {h} Z _ {\sigma , \tau^ {- 1} [ i ]}\right) \tag {1}
$$
where  $\Gamma(h, r)$  denotes the set of all canonicalisation traces of order  $r$  on  $h$  units and
$$
X _ {I} ^ {\delta} = \left\{(a _ {1}, b _ {1}, c _ {1}, \ldots , a _ {h}, b _ {h}, c _ {h}, d) \in \mathcal {W} _ {h} \middle | \begin{array}{c} \forall i \in I, b _ {i} = 0 a n d \\ d + \sum_ {i \in I} a _ {i} \tanh (c _ {i}) = \delta \end{array} \right\};
$$
$$
Y _ {\sigma , I} ^ {\alpha , \beta , \gamma} = \left\{(a _ {1}, b _ {1}, c _ {1}, \ldots , a _ {h}, b _ {h}, c _ {h}, d) \in \mathcal {W} _ {h} \middle | \begin{array}{c} \forall i \in I, \sigma_ {i} \cdot (b _ {i}, c _ {i}) = (\beta , \gamma) \\ \text {a n d} \sum_ {i \in I} \sigma_ {i} a _ {i} = \alpha \end{array} \right\}; a n d
$$
$$
Z _ {\sigma , I} = \left\{(a _ {1}, b _ {1}, c _ {1}, \ldots , a _ {h}, b _ {h}, c _ {h}, d) \in \mathcal {W} _ {h} \middle | \begin{array}{c} \forall i, j \in I, \sigma_ {i} \cdot (b _ {i}, c _ {i}) = \sigma_ {j} \cdot (b _ {j}, c _ {j}) \\ \text {a n d} \sum_ {i \in I} \sigma_ {i} a _ {i} = 0 \end{array} \right\}.
$$
Proof. Suppose  $w' = (a_1', b_1', c_1', \ldots, a_h', b_h', c_h', d') \in \mathcal{W}_h$  is in the union in (1), and therefore in the intersection for some canonicalisation trace  $(\sigma, \tau) \in \Gamma(h, r)$ . Then  $f_{w'} = f_v = f_w$ , as follows:
$$
\begin{array}{l} f _ {w ^ {\prime}} (x) = d ^ {\prime} + \sum_ {i \in \tau^ {- 1} [ 0 ]} a _ {i} ^ {\prime} \tanh (b _ {i} ^ {\prime} x + c _ {i} ^ {\prime}) + \sum_ {j = 1} ^ {r} \sum_ {i \in \tau^ {- 1} [ j ]} a _ {i} ^ {\prime} \tanh (b _ {i} ^ {\prime} x + c _ {i} ^ {\prime}) + \sum_ {j = r + 1} ^ {h} \sum_ {i \in \tau^ {- 1} [ j ]} a _ {i} ^ {\prime} \tanh (b _ {i} ^ {\prime} x + c _ {i} ^ {\prime}) \\ = \delta + \sum_ {j = 1} ^ {r} \alpha_ {j} \tanh  (\beta_ {j} x _ {i} + \gamma_ {j}) \text {s i n c e} w ^ {\prime} \in X _ {\tau^ {- 1} [ 0 ]} ^ {\delta} \cap \bigcap_ {j = 1} ^ {r} Y _ {\sigma , \tau^ {- 1} [ j ]} ^ {\alpha_ {j}, \beta_ {j}, \gamma_ {j}} \cap \bigcap_ {j = r + 1} ^ {h} Z _ {\sigma , \tau^ {- 1} [ j ]}. \\ \end{array}
$$
Now, suppose  $w' \in \mathfrak{F}[w]$ . Construct a canonicalisation trace  $(\sigma, \tau) \in \Gamma(h, r)$  following the execution of Algorithm 4.1 on  $w'$ . Set  $\sigma_i = -1$  where  $\mathrm{sign}(b_i') = -1$ , otherwise  $+1$ . Construct  $\tau$  from identity as follows. In each Stage 1 iteration, if the second branch is chosen, remap  $\tau(i)$  to 0. If the third or fourth branch is chosen, for  $k \in \tau^{-1}[i]$  (including  $i$  itself), remap  $\tau(k)$  to  $j$ . Finally, incorporate the Stage 3 permutation  $\pi$ : simultaneously for  $k \notin \tau^{-1}[0]$ , remap  $\tau(k)$  to  $\pi(\tau(k))$ .
Note CANONICALISE  $(w^{\prime}) = v$  by Theorem 4.2. Then  $w^{\prime}\in X_{\tau^{-1}[0]}^{\delta}$  because  $\tau^{-1}[0]$  contains exactly those units incorporated into  $\delta$ . Moreover, for  $j = 1,\ldots ,r$ ,  $w^{\prime}\in Y_{\sigma ,\tau^{-1}[j]}^{\alpha_{j},\beta_{j},\gamma_{j}}$ , because  $\tau^{-1}[j]$  contains exactly those units incorporated into unit  $j$  of  $v$ , and  $\sigma$  their relative signs ( $\beta_{j} > 0$ ). Likewise, for  $j\in r + 1,\dots ,h$ ,  $w^{\prime}\in Z_{\sigma ,\tau^{-1}[j]}$  (which is vacuous if  $\tau^{-1}[j]$  is empty).
Remark 5.3. If  $w \in \mathcal{W}_h$  is irreducible, then  $\mathrm{rank}(w) = h$ . For  $(\sigma, \tau) \in \Gamma(h, h)$ ,  $\tau$  is a permutation (since the range must include  $\{1, \ldots, h\}$ ). The set of traces therefore corresponds to the set of transformations generated by unit negations and exchanges, as in Sussmann (1992).
Remark 5.4. When  $\mathrm{rank}(w) = h - 1$ , there are, modulo sign vectors and permutations, essentially three canonicalisation traces, corresponding to the three ways of adding an additional unit to a  $(h - 1)$ -unit network discussed by Fukumizu and Amari (2000) and Fukumizu et al. (2019): to introduce a new constant unit or one with zero output, or to split an existing unit in two.
Remark 5.5. Similarly, in Şimsek et al. (2021, Definitions 3.2 and 3.3), an  $(r + j)$ -tuple coupled with a permutation plays the role of  $\tau$  in characterising the expansion manifold, akin to the functional equivalence class but from the dual perspective of adding units to an irreducible parameter. Şimsek et al. (2021) study a setting without a unit negation symmetry, so there is no need for a sign vector.
6 Path Connectivity
In this section, we show that the reducible functional equivalence class is piecewise linear path-connected (Theorem 6.1), and, for parameters with rank at most half of the available number of hidden units, has diameter at most 7 linear segments (Theorem 6.3).
Theorem 6.1. Let  $w \in \mathcal{W}_h$ . If  $w$  is reducible, then  $\mathfrak{F}[w]$  is piecewise linear path-connected.
Proof. It suffices to show that each reducible parameter  $w \in \mathcal{W}_h$  is piecewise linear path-connected in  $\mathfrak{F}[w]$  to its canonical representative  $\mathrm{CANONICALISE}(w)$ . The path construction proceeds by tracing the parameter's mutations in the course of execution of Algorithm 4.1. For each iteration of the loops in Stages 1 and 2, and for each transposition in the permutation in Stage 3, we construct a multi-segment sub-path. To describe these sub-paths, we denote the parameter at the beginning of each sub-path as  $w = (a_1, b_1, c_1, \ldots, a_h, b_h, c_h, d)$ , noting that this parameter is mutated throughout the algorithm, but is functionally equivalent to the original  $w$  at all of these intermediate points.
1. In each iteration of the Stage 1 loop, the construction depends on the chosen branch, as follows. Some examples are illustrated in Figure 1.
(i) A direct path interpolating  $b_{i}$  and  $c_{i}$  to zero.
(ii) A two-segment path, interpolating  $a_{i}$  to zero and  $d$  to  $d + a_{i}\tanh (c_{i})$ , then  $c_{i}$  to zero.
(iii) A two-segment path, interpolating  $a_i$  to zero and  $a_j$  to  $a_j + a_i$ , then  $b_i$  and  $c_i$  to zero.
(iv) A two-segment path, interpolating  $a_i$  to zero and  $a_j$  to  $a_j - a_i$ , then  $b_i$  and  $c_i$  to zero.