SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 1: Example paths constructed for each of the Stage 1 branches. Other dimensions held fixed.
Since (the original)  $w$  is reducible, (the current)  $w$  must have gone through at least one iteration in Stage 1, and must have at least one blank unit  $k$  with  $a_{k}, b_{k}, c_{k} = 0$ . From any such parameter  $w$ , there is a three-segment path in  $\mathfrak{F}[w]$  that implements a blank-exchange manoeuvre transferring the weights of another unit  $i$  to unit  $k$ , and leaving  $a_{i}, b_{i}, c_{i} = 0$ : first interpolate  $b_{k}$  to  $b_{i}$  and  $c_{k}$  to  $c_{i}$ ; then interpolate  $a_{k}$  to  $a_{i}$  and  $a_{i}$  to zero; then interpolate  $b_{i}$  and  $c_{i}$  to zero. Likewise, there is a three-segment path that implements a negative blank-exchange manoeuvre, negating the weights as they are interpolated into the blank unit. With these manoeuvres noted, proceed:
2. In each iteration of the Stage 2 loop for which  $\mathrm{sign}(b_i) = -1$ , let  $k$  be a blank unit, and construct a six-segment path. First, blank-exchange unit  $i$  into unit  $k$ . Then, negative blank-exchange unit  $k$  into unit  $i$ . The net effect is to negate unit  $i$ .
3. In Stage 3, construct a path for each segment in a decomposition of the permutation  $\pi$  as a product of transpositions. Consider the transposition  $(i,j)$ . If  $i$  or  $j$  is blank, simply blank-exchange them. If neither is blank, let  $k$  be a blank unit. Construct a nine-segment path, using three blank-exchange manoeuvres, using  $k$  as 'temporary storage' to implement the transposition: first blank-exchange units  $i$  and  $k$ , then blank-exchange units  $i$  (now blank) and  $j$ , then blank-exchange units  $j$  (now blank) and  $k$  (containing  $i$ 's original weights).
The resulting parameter is the canonical representative and it can be verified that each segment in each sub-path remains in  $\mathfrak{F}[w]$  as required.
Remark 6.2. Şimşek et al. (2021, Theorem B.4) construct similar paths to show the connectivity of their expansion manifold (cf. Remark 5.5). They first connect reduced-form parameters using blank-exchange manoeuvres and then show inductively that each unit addition preserves connectivity.
Theorem 6.3. Let  $w \in \mathcal{W}_h$ . If  $\mathrm{rank}(w) \leq \frac{h}{2}$ , then  $\mathfrak{F}[w]$  has diameter at most 7.
Proof. Let  $w \in \mathcal{W}_h$  with  $\mathrm{rank}(w) = r \leq \frac{h}{2}$ . Let  $w' \in \mathfrak{F}[w]$ . We construct a piecewise linear path from  $w$  to  $w'$  with 7 segments. By Theorem 6.1, a path exists via the canonical representative parameter  $v = \mathrm{CANONICALISE}(w)$ . However, this path has excessive length. We compress the length to 7 by exploiting the following opportunities to parallelise segments and 'cut corners'. These optimisation steps are illustrated in Figure 2.
(a) Let the Stage 1 result from Algorithm 4.1 for  $w$  be denoted  $u$ . Let the Stage 1 result for  $w'$  be denoted  $u'$ . Instead of following the unit negation and exchange transformations from  $u$  to  $v$ , and then back to  $u'$ , we transform  $u$  into  $u'$  directly, not (necessarily) via  $v$ .
(b) We connect  $w$  to  $u$  using two segments, implementing all iterations of Stage 1 in parallel. The first segment shifts the outgoing weights from the blank units to the non-blank units and the output unit bias. The second segment interpolates the blank units' incoming weights and biases to zero. We apply the same optimisation to connect  $w'$  and  $u'$ .
(c) We connect  $u$  and  $u'$  using two blank-exchange manoeuvres (6 segments), exploiting the majority of blank units as 'temporary storage'. First, we blank-exchange the non-blank units of  $u$  into blank units of  $u'$ , resulting in a parameter  $\bar{u}'$  sharing no non-blank units with  $u'$ . Then, we (negative) blank-exchange those weights into the appropriate non-blank units of  $u'$ , implementing the unit negation and exchange transformations relating  $u$ ,  $\bar{u}'$ , and  $u'$ .
(d) The manoeuvres in (b) and (c) begin and/or end by interpolating incoming weights and biases of blank units from and/or to zero, while the outgoing weights are zero. We combine adjacent beginning/end segments together, interpolating directly from the start to the end, without (necessarily) passing through zero. This results in the required seven-segment path, tracing the sequence of parameters  $w, w^1, w^2, \ldots, w^6, w' \in W_h$ .
Figure 2: A conceptual illustration of the four path optimisations, producing a seven-segment piecewise linear path of equivalent parameters in a high-dimensional parameter space. (a) Follow unit negation and exchange transformations directly between reduced parameters, not via the canonical parameter. (b) & (c) Parallelise the reduction steps, and use the majority of blank units to parallelise the transformations. (d) Combine first/last segments of reduction and blank-exchange manoeuvres.
To describe the constructed path in detail, we introduce the following notation for the components of the key parameters  $w, w', u, u', w^1, w^2, \ldots, w^6 \in \mathcal{W}_h$ :
$$
w = \left(a _ {1} ^ {w}, b _ {1} ^ {w}, c _ {1} ^ {w}, \dots , a _ {h} ^ {w}, b _ {h} ^ {w}, c _ {h} ^ {w}, d ^ {w}\right) \quad u = \left(a _ {1} ^ {u}, b _ {1} ^ {u}, c _ {1} ^ {u}, \dots , a _ {h} ^ {u}, b _ {h} ^ {u}, c _ {h} ^ {u}, d ^ {u}\right)
$$
$$
w ^ {\prime} = \left(a _ {1} ^ {w ^ {\prime}}, b _ {1} ^ {w ^ {\prime}}, c _ {1} ^ {w ^ {\prime}}, \dots , a _ {h} ^ {w ^ {\prime}}, b _ {h} ^ {w ^ {\prime}}, c _ {h} ^ {w ^ {\prime}}, d ^ {w ^ {\prime}}\right) \qquad u ^ {\prime} = \left(a _ {1} ^ {u ^ {\prime}}, b _ {1} ^ {u ^ {\prime}}, c _ {1} ^ {u ^ {\prime}}, \dots , a _ {h} ^ {u ^ {\prime}}, b _ {h} ^ {u ^ {\prime}}, c _ {h} ^ {u ^ {\prime}}, d ^ {u ^ {\prime}}\right)
$$
$$
w ^ {k} = \left(a _ {1} ^ {k}, b _ {1} ^ {k}, c _ {1} ^ {k}, \dots , a _ {h} ^ {k}, b _ {h} ^ {k}, c _ {h} ^ {k}, d ^ {k}\right) \quad (k = 1, \dots , 6).
$$
Of the  $h$  units in  $u$ , exactly  $h - r$  are blank—those in the set  $Z$  from  $\mathrm{CANONICALISE}(w)$ . Denote the complement set of  $r$  non-blank units  $U = \{1, \ldots, h\} \setminus Z$ . Likewise, define  $Z'$  and  $U'$  from  $u'$ .
With notation clarified, we can now describe the key points  $w^1, \ldots, w^6$  in detail, while showing that the entire path is contained within the functional equivalence class  $\mathfrak{F}[w]$ .
1. The first segment interpolates each outgoing weight from  $a_i^w$  to  $a_i^u$ , and interpolates the output bias from  $d^w$  to  $d^u$ . That is,  $w^1 = (a_1^u, b_1^w, c_1^w, \ldots, a_h^u, b_h^w, c_h^w, d^u)$ .
To see that this segment is within  $\mathfrak{F}[w]$ , observe that since the incoming weights and biases are unchanged between the two parameters,  $f_{tw^{1} + (1 - t)w}(x) = tf_{w^{1}}(x) + (1 - t)f_{w}(x)$  for  $x\in \mathbb{R}$  and  $t\in [0,1]$ . To show that  $f_{w} = f_{w^{1}}$ , we construct a function  $\tau :\{1,\ldots ,h\} \to$ $\{0,1,\dots,h\}$  from identity following each iteration of Stage 1 of CANONICALISE(w): when the second branch is chosen, remap  $\tau (i)$  to 0; and when the third or fourth branch is chosen, for  $k\in \tau^{-1}[i]$  (including  $i$  itself), remap  $\tau (k)$  to  $j$ . Moreover, we define a sign vector  $\sigma \in \{-1, + 1\} ^h$  where  $\sigma_{i} = -1$  if  $\mathrm{sign}(b_i^w) = -1$ , otherwise  $\sigma_{i} = +1$ . Then:
$$
\begin{array}{l} f _ {w} (x) = d ^ {w} + \sum_ {j = 0} ^ {k} \sum_ {i \in \tau^ {- 1} [ j ]} a _ {i} ^ {w} \tanh  \left(b _ {i} ^ {w} x + c _ {i} ^ {w}\right) \\ = d ^ {w} + \sum_ {i \in \tau^ {- 1} [ 0 ]} a _ {i} ^ {w} \tanh  \left(c _ {i} ^ {w}\right) + \sum_ {j = 1} ^ {h} \left(\sum_ {i \in \tau^ {- 1} [ j ]} \sigma_ {j} \sigma_ {i} a _ {i} ^ {w}\right) \tanh  \left(b _ {j} ^ {w} x + c _ {j} ^ {w}\right) \\ = d ^ {u} + \sum_ {j = 1} ^ {h} a _ {j} ^ {u} \tanh  \left(b _ {j} ^ {w} x + c _ {j} ^ {w}\right) = f _ {w ^ {1}} (x). \\ \end{array}
$$
2. The second segment completes the reduction and begins the first blank-exchange manoeuvre to store the nonzero units in  $Z'$ . For  $i \in U \cap U'$ , pick distinct 'storage' units  $j \in Z \cap Z'$ . There are enough, as  $r \leq \frac{h}{2}$  by assumption thus  $|U \cap U'| = |U| - |Z \cap U'| = r - |Z \cap U'| \leq (h - r) - |Z \cap U'| = |Z'| - |Z \cap U'| = |Z' \cap Z|$ . Interpolate unit  $j$ 's incoming weight from  $b_j^w$  to  $b_i^w$  and interpolate its bias from  $c_j^w$  to  $c_i^w$ . Meanwhile, for all other  $j \in Z$ , interpolate the incoming weight and bias to zero. This segment is within  $\mathfrak{F}[w]$  as for  $j \in Z$ ,  $a_j^1 = a_j^u = 0$  by definition of  $Z$ .
3. The third segment shifts the outgoing weights from the units in  $U \cap U'$  to the units in  $Z \cap Z'$  prepared in step (2). For  $i \in U \cap U'$ , pick the same storage unit  $j$  as in step (2). Interpolate unit  $j$ 's outgoing weight from  $a_j^u = 0$  to  $a_i^u$  and interpolate unit  $i$ 's outgoing weight from  $a_i^u$  to zero. This segment is within  $\mathfrak{F}[w]$  as  $b_i^2 = b_j^2$  and  $c_i^2 = c_j^2$  by step (2).
4. The fourth segment completes the first blank-exchange manoeuvre and begins the second, to form the units of  $u'$ . For  $i \in U'$ , interpolate unit  $i$ 's incoming weight from  $b_i^3$  to  $b_i^{u'}$  and interpolate its bias from  $c_i^3$  to  $c_i^{u'}$ . This segment is within  $\mathfrak{F}[w]$  because for  $i \in U' \cap Z$ ,  $a_i^3 = a_i^u = 0$  by definition of  $Z$ , and for  $i \in U' \cap U$ ,  $a_i^3 = 0$  by step (3).
5. The fifth segment shifts the outgoing weights from the selected units in  $Z'$  to the units in  $U'$  prepared in step (4). We simply interpolate each unit  $i$ 's outgoing weight to  $a_i^{u'}$ .
To see that the segment is within  $\mathfrak{F}[w]$ , note that  $u$  and  $u'$  are related by some unit negation and exchange transformations. Therefore, there is a correspondence between their sets of nonzero units, such that corresponding units have the same (or negated) incoming weights and biases. Due to steps (2)-(4) there are  $r$  'storage' units in  $w^4$  with the weights of the units of  $u$ , and the correspondence extends to these storage units. Since the storage units are disjoint with  $U'$ , this fifth segment has the effect of interpolating the outgoing weight of each of the storage units  $j \in Z'$  in  $w^4$  from  $a_i^u$  to zero (where  $i$  is as in step (3)), while interpolating the outgoing weight of its corresponding unit  $k \in U'$  from zero to  $\pm a_i^u = a_k^{u'}$  (where the sign depends on the unit negation transformations relating  $u$  and  $u'$ ).
6. The sixth segment completes the second blank-exchange manoeuvre and begins to reverse the reduction. For  $i \in Z'$ , interpolate unit  $i$ 's incoming weight from  $b_i^5$  to  $b_i^{w'}$ , and interpolate its bias from  $c_i^5$  to  $c_i^{w'}$ . This segment is within  $\mathfrak{F}[w]$  as for  $i \in Z'$ ,  $a_i^5 = a_i^{u'} = 0$  by definition of  $Z'$ .
7. The seventh segment, of course, interpolates from  $w^6$  to  $w'$ . To see that this segment is within  $\mathfrak{F}[w]$ , note that by steps (5) and (6),  $w^6 = (a_1^{u'}, b_1^{w'}, c_1^{w'}, \ldots, a_h^{u'}, b_h^{w'}, c_h^{w'}, d_u^{u'})$  (noting  $d^u = d^{u'}$  since the output unit's bias is preserved by unit transformations). So the situation is the reverse of step (1), and a similar proof applies.
In this paper, we have investigated functional equivalence classes for reducible single-hidden-layer hyperbolic tangent network parameters, and their connectivity properties. Recall that irreducible parameters have discrete functional equivalence classes described by simple unit negation and exchange transformations. In contrast, reducible functional equivalence classes form a complex union of manifolds, displaying the following rich qualitative structure:
- There is a central discrete constellation of reduced-form parameters, each with maximally many blank units alongside an irreducible subparameter. These reduced-form parameters are related by unit negation and exchange transformations, like for irreducible parameters.
- Unlike in the irreducible case, these reduced-form parameters are connected by a network of piecewise linear paths. Namely, these are (negative) blank-exchange manoeuvres, and, when there are multiple blank units, simultaneous parallel blank-exchange manoeuvres.
- Various manifolds branch away from this central network, tracing in reverse the various reduction operations (optionally in parallel). Dually, these manifolds trace methods for adding units (cf., Fukumizu and Amari, 2000; Fukumizu et al., 2019; Şimşek et al., 2021).
Theorem 6.3 establishes that when there is a majority of blank units, the diameter of the entire union of manifolds becomes a small constant number of linear segments. With fewer blank units it will sometimes require more blank-exchange manoeuvres to traverse the central network of reduced-form parameters. Future work could investigate efficient implementation of various permutations through parallel blank-exchange manoeuvres with varying numbers of blank units available.
Towards modern architectures. Single-hidden-layer hyperbolic tangent networks appear far from relevant to modern deep learning architectures. However, we expect our canonicalisation algorithm to partially generalise and some aspects of our connectivity findings to generalise as follows.
Structural redundancy. Observe that reducibility conditions (i)-(iii) apply to structural redundancies that are generic to every feed-forward layer within any architecture (units with zero, constant, or proportional output). Unit negation symmetries are characteristic of odd nonlinearities only, but other nonlinearities will exhibit their own affine symmetries that would play a similar role. In more sophisticated architectures, these basic sources of structural redundancy will sit alongside other sources such as interactions between layers, attention blocks, layer normalisation, and so on.
Canonicalisation. It follows that Algorithm 4.1 serves as a partial canonicalisation for more sophisticated architectures (with small modifications for alternative nonlinearities). Future works need only find and remove the additional kinds of structural redundancy.
Connectivity. Similarly, additional sources of redundancy expand the functional equivalence class. While global connectivity properties may be lost in this process, individual paths will not be disturbed. We expect that our high-level qualitative finding will hold: that the more reducible a parameter is, the more intricately connected it is to functionally equivalent parameters.
Towards approximate structural redundancy. Our framework is built around the definition of functional equivalence, which requires exact equality of functions for all inputs. A more pragmatic definition would concern approximate equality of functions for relevant inputs. One step in this direction is studying proximity in parameter space to low-rank parameters, though detecting such proximity precisely is formally intractable (Farrugia-Roberts, 2023).
Functional equivalence and the loss landscape. Functionally equivalent parameters have equal loss. Continuous directions and piecewise linear paths within reducible functional equivalence classes therefore imply flat directions and equal-loss paths in the loss landscape. More broadly, the set of low- or zero-loss parameters is a union of functional equivalence classes. If some (very) reducible parameters obtain low loss (such as in the overparameterised setting) then the set of low-loss parameters contains rich functional equivalence classes as subsets.
Of course, having the same loss does not require functional equivalence. Indeed, Garipov et al. (2018) observe functional non-equivalence in low-loss paths. The exact relevance of reducible parameters to these topics remains to be clarified. Of special interest is the connection to theoretical work involving unit pruning (Kuditipudi et al., 2019) and permutation symmetries (Brea et al., 2019).
Canonicalisation and model compression. Our canonicalisation algorithm transforms a neural network parameter into another parameter that has the same input-output behaviour but effectively uses fewer units. This size reduction is not fundamental to the goals of canonicalisation (we could still achieve canonicalisation by producing a standard, maximally dense representation of each equivalent parameter). Nevertheless, Algorithm 4.1 performs (lossless) model compression as a side-effect (cf. Farrugia-Roberts, 2023).
In particular, our canonicalisation algorithm is reminiscent of a unit-based pruning technique (cf., e.g., Hoefler et al., 2021). However, there are a few salient differences. First, pruning algorithms are usually not designed to preserve exact functional equivalence on all inputs, but rather to accept small changes in outputs (on specific inputs). Second, unit-based pruning techniques usually select individual units for removal, and then continue training the network. Our canonicalisation algorithm instead considers operations that simultaneously remove a unit and perform a systematic adjustment to the remaining weights and biases to maintain the network's behaviour without any training.
Of interest, Casper et al. (2021) empirically studied a network pruning that finds units with weak outputs or pairs of units with correlated outputs, and then eliminates or merges these units and makes appropriate adjustments to approximately maintain performance. The elimination and merging operations bear a striking resemblance to the operations in Algorithm 4.1.
Reducible parameters and singular learning theory. When a reducible parameter is a critical point of the loss landscape, it is necessarily a degenerate critical point (due to the continuum of equal-loss equivalent parameters nearby; or cf. Fukumizu, 1996). This places situations where learning encounters (or even approaches) reducible parameters within the domain of singular learning theory (cf. Watanabe, 2009; Wei et al., 2022).
The nature of the degeneracy depends on the exact ways in which it is reducible. For example, is the parameter at the intersection of multiple of the manifolds comprising the functional equivalence class (cf. Theorem 5.2)? The order of variation in directions away from the functional equivalence class also plays a role (cf. Lau et al., 2023). Future work could analyse the structural redundancy to find principled bounds on effective dimensionality.
Reducible parameters exhibit structural redundancy, in that the same input-output function could be implemented with a smaller network. While reducible parameters comprise a measure zero subset of the parameter space, their functional equivalence classes are much richer than those of irreducible parameters. Understanding these rich functional equivalence classes is important to understanding the nature of the loss landscape within which deep learning takes place.
We have taken the first step towards understanding functional equivalence beyond irreducible parameters by accounting for various kinds of structural redundancy in the setting of single-hidden-layer hyperbolic tangent networks. We offer an algorithmic characterisation of reducible functional equivalence classes and an investigation of their piecewise linear connectivity properties. We find in particular that the more redundancy is present in a parameter, the more intricately connected is its functional equivalence class.
We call for future work to seek out, catalogue, and thoroughly investigate sources of structural redundancy in more sophisticated neural network architectures; and to further investigate the role these parameters play in deep learning.
Contributions (1), (2), and (3) also appear in MFR's minor thesis (Farrugia-Roberts, 2022, §5). MFR was affiliated with the School of Computing and Information Systems at the University of Melbourne while completing this research, but is currently affiliated with the University of Cambridge.
We thank Daniel Murfet for helpful feedback during this research and on this manuscript.
MFR received financial support from the Melbourne School of Engineering Foundation Scholarship and the Long-Term Future Fund while completing this research.
A Generalising to multi-dimensional inputs and outputs
In this appendix, we consider a slightly more general family of architectures than that introduced in Section 3. Namely, we consider a family of fully-connected, feed-forward neural network architectures with  $n \in \mathbb{N}^+$  input units,  $m \in \mathbb{N}^+$  biased linear output units, and a single hidden layer of  $h \in \mathbb{N}$  biased hidden units with the hyperbolic tangent nonlinearity. With minor modifications, described in the remainder of this appendix, all definitions, algorithms, theorems, and proofs directly generalise from the case  $n = m = 1$  to arbitrary  $n$  and  $m$ .
Multi-dimensional architecture. Let  $n \in \mathbb{N}^+$ ,  $m \in \mathbb{N}^+$ , and  $h \in \mathbb{N}$ . Define the generalised parameter space  $\mathcal{W}_h^{n,m} = \mathbb{R}^{(n + m + 1)h + m}$ . The weights and biases of the network's units are encoded in the parameter vector in the format  $(a_1, b_1, c_1, \ldots, a_h, b_h, c_h, d) = w \in \mathcal{W}_h^{n,m}$  where for each hidden unit  $i = 1, \ldots, h$  there is an outgoing weight vector  $a_i \in \mathbb{R}^m$ , an incoming weight vector  $b_i \in \mathbb{R}^n$ , and a bias  $c_i \in \mathbb{R}$ ; and  $d \in \mathbb{R}^m$  is an output unit bias vector containing one bias value for each output unit. This time,  $w$  indexes a multi-dimensional mathematical function  $f_w: \mathbb{R}^n \to \mathbb{R}^m$  defined as follows:
$$
f _ {w} (x) = d + \sum_ {i = 1} ^ {h} a _ {i} \tanh  \left(b _ {i} \cdot x + c _ {i}\right). \tag {2}
$$
Note that we use the same tuple notation and ordering  $(a_{1}, b_{1}, c_{1}, \ldots, a_{h}, b_{h}, c_{h}, d)$  but now the  $a_{i}$ , the  $b_{i}$ , and  $d$  all denote multi-component vectors. Accordingly, in Equation (2),  $b_{i}$  and  $x$  are now multiplied using the inner (dot) product, rather than scalar multiplication, since they are both vectors in  $\mathbb{R}^n$ . Moreover,  $a_{i} \in \mathbb{R}^{m}$  as a vector is to be multiplied by the scalar  $\tanh(b_{i} \cdot x + c_{i})$ . That is, the sum is over vectors of contributions to output units from each hidden unit.
To generalise the results of the main paper to this setting the first change necessary is to replace all mentions of scalar weights with these vectors of weights, and other similar changes such as reading the literal zero as vector zero where appropriate.
Signing and sorting incoming weight vectors. The lexicographic order on  $\mathbb{R}^n$ , denoted  $\preceq$ , is a relation such that for  $u, v \in \mathbb{R}^n$ ,  $u \preceq v$  if and only if  $u = v$  or, in the first index  $i = 1, \ldots, n$  where  $u$  and  $v$  differ,  $u_i < v_i$ . From this definition we follow the usual conventions in defining  $\preceq, \succ$ , and  $\succeq$ . Finally, define the lexicographic sign of  $v \in \mathbb{R}^n$ , denoted  $\mathrm{sign}_{\mathrm{lex}}(v)$ , as follows:
$$
\operatorname {s i g n} _ {\mathrm {l e x}} (v) = \left\{ \begin{array}{l l} + 1 & \quad (v \succ 0), \\ 0 & \quad (v = 0), \\ - 1 & \quad (v \prec 0). \end{array} \right.
$$
The parameter canonicalisation algorithm and some of the other theorems and proofs make repeated use of the signs of incoming weight vectors. The lexicographic sign satisfies the requisite properties of the scalar sign function in these uses and so the second change necessary to generalising the results is to replace uses of  $\mathrm{sign}(\cdot)$  with uses of  $\mathrm{sign}_{\mathrm{lex}}(\cdot)$ .
This lexicographic order relation is of course also a total order (see, e.g., Harzheim, 2005, Theorem 4.1.11). Therefore, it allows one to sort a list of vectors. Sorting units by decreasing incoming weights is a key step in Stage 3 of Algorithm 4.1, and so the third change necessary is to use decreasing lexicographic order  $(\succ)$  in this stage.