SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This function is simple enough so that we understand its shape. It has precisely one inflection point, below which it is concave, later it is convex. The derivative at 0 is strictly positive. Judging from this, the function either has 0 as a single global optimum, or it has two global optima, one at 0 and one at a strictly positive local minimum of the strictly convex part of the function, or the global minimum is achieved uniquely at the strictly convex part of the function. To find out, we only need to solve  $f(y) = f(0)$  for  $y$ , where  $f$  stands for the optimized function and determine the number of solutions. The quadratic equation this is equivalent to after excluding the point  $y = 0$  as a solution is  $4y^{2}\sqrt{\lambda_{W_{1}}\lambda_{H_{1}}} - y + 8\lambda_{W_{2}}\sqrt{\lambda_{W_{1}}\lambda_{H_{1}}} = 0$ . This has a unique solution if and only if  $\lambda_{W_2}\lambda_{W_1}\lambda_{H_1} = 1 / 256$ , which is precisely the threshold from the statement of the lemma. The cases outside of this threshold are easily mapped to the system having the global solution 0 or some positive point.
Let us now consider the case  $L > 2$ . First note that, once for any  $l_0$  the optimal  $x_{l_0}^* = 0$ , then at this particular solution all the  $x$ 's must be 0. If  $l_0 = L$ , then this claim is apparent backward-inductively optimizing always over  $x$  with one index lower. If  $2 \leq l_0 < L$ , then the claim is trivial backward-inductively for all  $l \leq l_0$ , while for all  $l > l_0$  this solution is forced by feasibility issues. From now on we will thus work with an implicit strict positiveness assumption. Taking derivatives of this objective w.r.t.  $x_l$  for different  $l$  and putting them equal 0 we arrive at three types of equations depending on  $l$ :
$$
\begin{array}{l} \sqrt {\frac {\lambda_ {W _ {L}}}{\lambda_ {W _ {L - 1}}}} \sqrt {x _ {L - 1}} = x _ {L} + 2 \lambda_ {W _ {L}}, \\ x _ {l} ^ {2} = \frac {\lambda_ {W _ {l}}}{\lambda_ {W _ {l - 1}}} x _ {l + 1} x _ {l - 1}, \quad \text {f o r} 3 \leq l \leq L - 1, \\ x _ {2} ^ {\frac {3}{2}} = \frac {\lambda_ {W _ {2}}}{\sqrt {\lambda_ {W _ {1}} \lambda_ {H _ {1}}}} x _ {3}. \tag {15} \\ \end{array}
$$
Here we can notice two things. First, we have  $L - 1$  equations for  $L - 1$  variables, so we should expect a single solution in most of the cases. Second, the equations are rather simple and recursive.
Proof. The proof follows easily by applying the Lemmas 9, 5 and 4.
Having either a value of  $x_{2}$  or  $x_{L}$ , all other values are uniquely determined. Let us denote  $q := \frac{x_L}{x_{L-1}}$ . Based on this, we can arrive at the following formula:
$$
x _ {L - k} = \frac {\prod_ {j = 1} ^ {k - 1} \lambda_ {W _ {L - j - 1}}}{\lambda_ {W _ {L - 1}} ^ {k - 1}} q ^ {- k} x _ {L}, \tag {16}
$$
for all  $1 \leq k \leq L - 2$ . An empty product is defined as 1. To derive this, for  $L = 3$  the statement is only about  $x_{2}$  and is trivial. For  $L > 3$ , the  $k = 1$  case is trivial, then one can proceed by induction using the recurrent formula  $x_{l}^{2} = \frac{\lambda_{W_{l}}}{\lambda_{W_{l-1}}} x_{l+1} x_{l-1}$ . From formula (16), one can derive another useful equation:
$$
\frac {x _ {k + 1}}{x _ {k}} = \frac {\lambda_ {W _ {L - 1}}}{\lambda_ {W _ {k}}} q, \quad \text {f o r} 2 \leq k \leq L - 1.
$$
Combining this with (15), we also get  $x_{2} = \frac{\lambda_{W_{L} - 1}^{2}}{\lambda_{W_{1}}\lambda_{H_{1}}} q^{2}$ . Combining now this equation with formula (16) for  $x_{2}$  from before one gets:
$$
\frac {\lambda_ {W _ {L} - 1} ^ {2}}{\lambda_ {W _ {1}} \lambda_ {H _ {1}}} q ^ {2} = \frac {\prod_ {j = 1} ^ {L - 3} \lambda_ {W _ {L - j - 1}}}{\lambda_ {W _ {L - 1}} ^ {L - 3}} q ^ {2 - L} x _ {L} \iff x _ {L} = \frac {\lambda_ {W _ {L - 1}} ^ {L - 1}}{\lambda_ {H _ {1}} \prod_ {j = 1} ^ {L - 2} \lambda_ {W _ {j}}} q ^ {L}.
$$
Notice that we managed to express all the terms present in the main objective as simple functions of  $q$  and regularization parameters. Plugging all of these expressions back in the objective, we get:
$$
\frac {\lambda_ {W _ {L}}}{2 \left(\frac {\lambda_ {W _ {L} - 1} ^ {L - 1}}{\lambda_ {H _ {1}} \prod_ {j = 1} ^ {L - 2} \lambda_ {W _ {j}}} q ^ {L} + 2 \lambda_ {W _ {L}}\right)} + \frac {L \lambda_ {W _ {L - 1}}}{2} q. \tag {17}
$$
Since we only care about relative values of this expression, the optimization will be equivalent after multiplying it by  $2 / \lambda_{W_{L - 1}}$  so we get:
$$
\frac {1}{\frac {\lambda_ {W _ {L} - 1} ^ {L}}{\lambda_ {H _ {1}} \lambda_ {W _ {L}} \prod_ {j = 1} ^ {L - 2} \lambda_ {W _ {j}}} q ^ {L} + 2 \lambda_ {W _ {L - 1}}} + L q. \tag {18}
$$
Differentiating this function twice we can see that the second derivative is negative from 0 up to a single threshold at which it is 0, then it is positive. Therefore, based on the value of the first derivative at this threshold, and taking into account that the first derivative at 0 is strictly positive, we can easily see that the function has either a single global minimum at 0 or it has two global minima at 0 and at some positive  $q$  or it has a single global optimum at some positive  $q$ . To find out which of the cases we are at, we can simply solve for
$$
\frac {1}{\frac {\lambda_ {W _ {L} - 1} ^ {L}}{\lambda_ {H _ {1}} \lambda_ {W _ {L}} \prod_ {j = 1} ^ {L - 2} \lambda_ {W _ {j}}} q ^ {L} + 2 \lambda_ {W _ {L - 1}}} + L q = \frac {1}{2 \lambda_ {W _ {L - 1}}}.
$$
After excluding the trivial solution  $q = 0$ , this is equivalent to:
$$
2 \lambda_ {W _ {L - 1}} L q ^ {L} - q ^ {L - 1} + \frac {4 L \lambda_ {H _ {1}} \lambda_ {W _ {L}} \prod_ {j = 1} ^ {L - 2} \lambda_ {W _ {j}}}{\lambda_ {W _ {L - 1}} ^ {L - 2}} = 0.
$$
The case at which we stand w.r.t. the global solutions of the above objective function is equivalent to the number of solutions of this equation. Since the LHS of this equation is a very simple function, it is apparent that the number of solutions to this equation is equivalent to the sign of the LHS evaluated at the single zero-derivative positive  $q$ . The derivative is  $2\lambda_{W_{L - 1}}L^2 q^{L - 1} - (L - 1)q^{L - 2}$  which after solving for  $q > 0$  yields  $q = \frac{L - 1}{2\lambda_{W_{L - 1}}L^2}$ . Plugging this back into the LHS above, we get the expression
$$
2 ^ {1 - L} L ^ {1 - 2 L} \lambda_ {W _ {L - 1}} ^ {1 - L} (L - 1) ^ {L} - (L - 1) ^ {L - 1} 2 ^ {1 - L} \lambda_ {W _ {L - 1}} ^ {1 - L} L ^ {2 - 2 L} + \frac {4 L \lambda_ {H _ {1}} \lambda_ {W _ {L}} \prod_ {j = 1} ^ {L - 2} \lambda_ {W _ {j}}}{\lambda_ {W _ {L - 1}} ^ {L - 2}}.
$$
Thresholding this for 0 and simplifying, we get the final
$$
\lambda_ {H _ {1}} \prod_ {j = 1} ^ {L} \lambda_ {W _ {j}} = \frac {(L - 1) ^ {L - 1}}{2 ^ {L + 1} L ^ {2 L}}.
$$
If the LHS is smaller than this threshold, the function in (18) has a single non-zero global optimum. If the LHS is at the threshold, the optimization problem has both 0 and some single non-zero point for a global solution. If the LHS is above the threshold, the optimization problem has only 0 as a single solution.
With all this, Theorem 3 follows very easily. We only need to conclude that if the optimization problem in Lemma 11 has a single non-zero solution, then both the optimal  $s_{l,1}^{*}$  and  $s_{l,2}^{*}$  are forced to be the same for each  $l$ . From this, we obtain the orthogonality of all the  $\sigma(H_{l}^{*})$  for  $l \geq 2$ . However, from Lemma 6 we know that even the optimal  $H_{l}^{*}$  for  $l \geq 3$  must be orthogonal, since it is non-negative and equal to  $\sigma(H_{l}^{*})$ . From the same lemma, we conclude the statement about  $W_{l}^{*}$  for  $l \geq 2$ . The optimal  $W_{1}^{*}, H_{1}^{*}$  follow easily from Lemma 8.
By now, we only considered the case  $n = 1$ . Now, we proceed to the general  $n$  case, where we need to also prove DNC1. We will proceed by contradiction. Assume there exist  $\tilde{H}_1^*, W_1^*, \ldots, W_L^*$  which optimize  $L$ -DUFM, but  $\tilde{H}_1^*$  is not DNC1-collapsed. Now, since the objective function in 10 is separable in the columns of  $H_1$ , we know
$$
\begin{array}{l} \frac {1}{2 N} \left\| W _ {L} ^ {*} \sigma (\dots W _ {2} ^ {*} \sigma (W _ {1} ^ {*} h _ {c, i} ^ {(1)})) - e _ {c} \right\| _ {F} ^ {2} + \sum_ {l = 1} ^ {L} \frac {\lambda_ {W _ {l}}}{2} \| W _ {l} ^ {*} \| _ {F} ^ {2} + \frac {\lambda_ {H _ {1}}}{2} \left\| h _ {c, i} ^ {(1)} \right\| _ {2} ^ {2} \\ = \frac {1}{2 N} \left\| W _ {L} ^ {*} \sigma \left(\dots W _ {2} ^ {*} \sigma \left(W _ {1} ^ {*} h _ {c, j} ^ {(1)}\right)\right) - e _ {c} \right\| _ {F} ^ {2} + \sum_ {l = 1} ^ {L} \frac {\lambda_ {W _ {l}}}{2} \left\| W _ {l} ^ {*} \right\| _ {F} ^ {2} + \frac {\lambda_ {H _ {1}}}{2} \left\| h _ {c, j} ^ {(1)} \right\| _ {2} ^ {2}, \\ \end{array}
$$
where  $h_{c,i}^{(1)}, h_{c,j}^{(1)}$  are columns of  $\tilde{H}_1^*$ , for all  $c, i, j$  applicable and  $e_c$  is the  $c$ -th standard basis vector. Otherwise, we could have just exchanged all sub-optimal  $h_{c,i}^{(1)}$  with the optimal one and achieve better objective value. However, if the last equality holds, then we can construct an alternative  $\bar{H}_1^*$  where for each class  $c$ , we pick any  $h_{c,i}^{(1)}$  and place it instead of all other  $h_{c,j}^{(1)}$  within the class  $c$ . In this way, the  $\bar{H}_1^*$  will be DNC1-collapsed, while still an optimal solution. However, for  $\bar{H}_1^*$ , we now know that if  $H_1^*$  is constructed by taking just a single sample from both classes (thus forcing  $n = 1$ ), this must be an optimal solution of the  $n = 1$  L-DUFM with  $n\lambda_{H_1}$  as the regularization term for  $H_1$  and same feature vector dimensions and other regularization terms. This is because if there was a better solution for the corresponding problem, then we could construct a counterexample to optimality in the original L-DUFM problem by just simply taking this alternative, better solution and using the feature vectors of this solution for each sample of each individual class.
Therefore, since  $H_1^*$  must be optimal in the new  $n = 1$  problem (with a slight abuse of notation, we re-label the  $n\lambda_{H_1}$  back to  $\lambda_{H_1}$ ), we know that  $\sigma(H_2^*)$  is collapsed in the way described by the theorem statement for  $n = 1$ . By non-negativity of  $\sigma(H_2^*)$ , this means that  $\sigma(H_2^*)$  has two columns, let's call them  $x, y$  for which:  $x, y \geq 0; x^T y = 0, \|x\| = \|y\|$  (by the notation  $x \geq 0$  we mean entry-wise inequality). This also implies that there is no such  $i$  that  $x_i, y_i > 0$ . Moreover, let us assume  $\sigma(H_2^*) \neq 0$ . We can do that, as otherwise simple arguments would lead us into the degenerate solution where all  $W_l^* = 0$ , for which obviously all the columns in the originally optimal  $\tilde{H}_1^*$  would need to be 0 implying simple DNC1 and reducing easily to the case of too big regularization explicitly stated in Theorem 3. We can now omit this case and assume non-triviality of the solution. We will first show that for a given set of optimal weight matrices  $(W_1^*, W_2^*, \ldots, W_L^*)$ , the matrix  $\sigma(H_2^*)$  is uniquely determined regardless of what  $H_1^*$  is. First, by the uniqueness of the solutions in Lemma 11 and by the fact that  $s_{2i}$  depends monotonically on  $\|x\|$ , it follows that for any candidate direction  $z, \|z\| = 1$ , there is a unique multiplier  $\alpha > 0$  for which  $\alpha z$  can be a column of  $\sigma(H_2^*)$ , i.e. there is a uniquely determined norm under which any candidate direction  $z$  must enter  $\sigma(H_2^*)$ . Now, however, by DNC3 for  $l = 2$  we know that rows of  $W_2$  are either zero or aligned with the columns of  $\sigma(H_2^*)$ . Moreover, since  $H_3^*$  (which in case  $L = 2$  shall be considered the output of the network) is orthogonal and thus rank 2, the weight matrix  $W_2^*$  must contain both the columns of  $\sigma(H_2^*)$  as its rows in direction. However, it cannot contain more than 2 different directions, otherwise for any
fixed optimal  $\sigma(H_2^*)$ , there would be an extra direction which would contradict DNC3. Thus,  $W_2^*$  contains exactly two linearly independent (orthogonal) directions. But then, in turn, only these two directions can be columns of  $\sigma(H_2^*)$ . So  $\sigma(H_2^*)$  is unique, possibly up to a switching of its columns  $x, y$ . However, only one permutation of these columns is optimal, because we know that the output of the matrix,  $H_{L+1}$  is orthogonal and only one of its permutations is better aligned with  $I_2$ . Therefore  $\sigma(H_2^*)$  is fully unique.
Invoking Lemma 9, we know that the whole range of matrices  $A_{2}$  are such that  $\sigma(A_{2}) = \sigma(H_{2}^{*})$ , while  $\|A_{2}\|_{*} = \|H_{2}^{*}\|_{*}$ . Namely, by Lemma 9, all the matrices  $A_{2}$  which satisfy that condition are represented by having columns of the form  $x - ty, y - tx$  for  $t \in [0,1]$ . This can be derived as follows. According to the lemma and the description of  $\sigma(H_{2}^{*})$ , this matrix is a good candidate for the non-negative part of the optimal solution of the corresponding optimization problem, and so the optimal solution (in the case of  $L = 2$  a solution for which the nuclear norms of it and its non-negative part are equal) whose non-negative part equals  $\sigma(H_{2}^{*})$  exists. The negative values, according to Lemma 9, must follow a strict, yet still non-unique pattern. The negative values of the (+,0) rows of  $\sigma(H_{2}^{*})$  must be arranged so that they are aligned with the positive values in the corresponding rows, thus being multiple of  $x$ . The negative values of the  $(0, +)$  rows must be, on the other hand, aligned with  $y$ . The coefficients of homothety between  $x$  and the negative part in the (+,0) rows and between  $y$  and the negative part of the  $(0, +)$  rows must be equal due to the condition on equal  $l_{2}$  norms of the negative parts corresponding to  $(+,0), (0, +)$  rows and the fact that  $\|x\| = \|y\|$ . Finally, the coefficients of homothety must be non-positive (yielding non-negative  $t$ ) to retain the negativity and no smaller than  $-1$ , due to the final condition of Lemma 9, stating that the product of any two negative values of counterfactually signed rows must be at most as big as the product of the corresponding positive entries and due to the already established alignment of positive and negative parts. In this way, the only matrices  $A_{2}$  which satisfy these conditions are represented by having columns of the form  $x - ty, y - tx$  for  $t \in [0,1]$ , as stated above (note that  $-ty$  is aligned with  $x$  on  $(+,0)$  rows of  $\sigma(H_{2}^{*})$  and similarly for  $-tx$ ).
Let us denote by  $A_2^t$  a matrix of that form for a particular  $t$ . Now, the SVD of  $A_2^t$  is as follows:  $U^t$  is composed of  $(x - y) / (\sqrt{2} \| x \|)$  and  $(x + y) / (\sqrt{2} \| x \|)$ ,  $\Sigma^t$  is a diagonal matrix with diagonal entries equal to  $\| x \| (1 + t)$  and  $\| x \| (1 - t)$ , and  $V^t$  is composed of the vectors  $(1 / \sqrt{2}, -1 / \sqrt{2})^T$  and  $(1 / \sqrt{2}, 1 / \sqrt{2})^T$ . For  $t \neq 0$ , this is the unique SVD up to  $\pm$  multiplications. For  $t = 0$ ,  $A_2^0$  is orthogonal and the SVD is non-unique. However, as we will see, this will not be relevant in the following argument. Invoking Lemma 8 we see that, for each  $t$ , the set of possible  $W_1^t$  which can be the optimal solution is  $S^t = \{c_\lambda U^t(\Sigma^t)^{1/2} R^T\}$  for all possible orthogonal  $R$  of suitable dimensions. From this representation, we see that the non-uniqueness of SVD is "wrapped" in the freedom of  $R^T$ . If  $t = 0$ ,  $\Sigma^0$  is diagonal and commutative and  $U^0 R^T$  obviously cover all possible representations of  $U$  in the SVD of  $A_2^0$ . If  $t > 0$ , we can easily cover the freedom in signs by  $R$  too. Thus, we can assume a fixed  $U$  and  $V$  for all  $t \in [0,1]$ . The question is, whether for  $0 \leq t_1 < t_2 \leq 1$ ,  $S^{t_1} \cap S^{t_2} = \emptyset$ . To find out, consider some  $R^{t_1}$  and  $R^{t_2}$  orthogonal. We want to know whether it is possible that  $c_\lambda U^{t_1}(\Sigma^{t_1})^{1/2}(R^{t_1})^T = c_\lambda U^{t_2}(\Sigma^{t_2})^{1/2}(R^{t_2})^T$ . First, this obviously does not hold if  $t_2 = 1$ , because of the rank inequality. Assume  $t_2 < 1$  and do the computation:
$$
c _ {\lambda} U ^ {t _ {1}} \left(\Sigma^ {t _ {1}}\right) ^ {1 / 2} \left(R ^ {t _ {1}}\right) ^ {T} = c _ {\lambda} U ^ {t _ {2}} \left(\Sigma^ {t _ {2}}\right) ^ {1 / 2} \left(R ^ {t _ {2}}\right) ^ {T} \Longleftrightarrow
$$
$$
U \left(\Sigma^ {t _ {1}}\right) ^ {1 / 2} \left(R ^ {t _ {1}}\right) ^ {T} = U \left(\Sigma^ {t _ {2}}\right) ^ {1 / 2} \left(R ^ {t _ {2}}\right) ^ {T} \Longleftrightarrow
$$
$$
\left(\Sigma^ {t _ {1}}\right) ^ {1 / 2} \left(R ^ {t _ {1}}\right) ^ {T} = \left(\Sigma^ {t _ {2}}\right) ^ {1 / 2} \left(R ^ {t _ {2}}\right) ^ {T} \Longleftrightarrow
$$
$$
\left(\Sigma^ {t _ {1}}\right) ^ {1 / 2} \left(\Sigma^ {t _ {2}}\right) ^ {- 1 / 2} = \left(R ^ {t _ {2}}\right) ^ {T} \left(R ^ {t _ {1}}\right),
$$
where in the first equivalence we used the established knowledge that we can use the unique representative  $U$  for all  $t$ . However, the diagonal elements of  $(\Sigma^{t_1})^{1/2}(\Sigma^{t_2})^{-1/2}$  are  $\sqrt{(1 + t_1) / (1 + t_2)}$  and  $\sqrt{(1 - t_1) / (1 - t_2)}$ . The first one is strictly greater than 1. On the other hand, all the entries of  $(R^{t_2})^T (R^{t_1})$  are inner products between columns of orthogonal matrices and thus at most 1 in absolute value. This proves  $S^{t_1} \cap S^{t_2} = \emptyset$  for  $0 \leq t_1 < t_2 \leq 1$ .
Thus, there is just a single  $t^*$  for which  $W_1^*$  is optimal and, moreover, there is a single  $A_2^* = A_2^{t^*}$  that can be the output of the first layer. Then, if we analyze the two systems of linear equations  $A_2^* = W_1^* X$ , where  $X$  is the  $d \times 2$  matrix variable (each system for each column of  $X$ ), we know that there is a unique min- $l_2$  norm solution per system. The min- $l_2$  norm solution must be the only optimal solution, because  $H_1$  is regularized using the Frobenius norm. Thus, there is a single  $H_1$
which can be the optimal solution. However, this is a contradiction with the assumption that we had a freedom while constructing  $\bar{H}_1^*$  from  $\tilde{H}_1^*$  - a possibility of choosing different columns from some classes as the representative of that class. Therefore, the  $\tilde{H}_1^*$  must have had a single feature vector per class in the first place, which contradicts the assumption that it is not DNC1 collapsed.
Now that we have the DNC1 collapse, we can obtain the DNC2-3 collapse easily by just again going from the full  $\tilde{H}^*$  to the reduced  $n = 1$  version of it. Moreover, we know what are the forms of  $W_1^*$  and  $H_1^*$  - orthogonal transformations of the SVD of  $H_2^*$ . However, it must be said that these optimal matrices do not need to be orthogonal themselves, due to the  $(\Sigma^t)^{1/2}$  multiplication, which makes the matrix non-orthogonal (consider for instance just taking the identity matrix extended to rectangularity for  $R$ ). This concludes the proof of Theorem 3.
B Extension to multiple classes
Here, we comment shortly on why it is challenging to formulate an equivalent of Theorem 3 for  $K > 2$ . First of all, it is already unclear how to carry out the proof of Lemma 5 for  $K > 2$  and even whether it holds. However, the bigger issue lies in Lemma 9. Note that  $\| H_2\|_* \geq \| \sigma (H_2)\|_*$  only holds for matrices with two rows or columns. A simple counterexample with  $3 \times 3$  matrix goes as follows:
$$
A = \left( \begin{array}{c c c} - 1 & 0 & 1 \\ 0 & 1 & 1 \\ 1 & 1 & 0 \end{array} \right),
$$
for which  $\| A\|_{*} = 3.464$  and  $\| \sigma (A)\|_{*} = 3.494$ . However, this does not mean that Theorem 3 does not hold for  $K > 2$ . For instance, at orthogonal matrices, which are collapsed, one can show that this equation holds by computing the sub-gradient of the nuclear norm and using convexity. It is also hard to come up with counterexamples to the inequality for matrices which are close to orthogonal. Therefore, it is our belief that the cost of losing orthogonality in other loss terms makes the compensation of achieving slightly lower nuclear norm insufficient. However, to make this argument formal, one needs a sophisticated control over this quantity together with all the other results.
C Further numerical experiments on DUFM and ablations
In this section, we further investigate the effect of significant parameters on the training dynamics of DUFM. We focus on the effect of depth, regularization strength and width on the outcome of DUFM optimization. All the plots (except the one in Appendix C.5) are averaged over 10 runs with one standard deviation to both sides displayed as a confidence band. First, however, we supplement the numerical results from Section 4.2 with the promised plots of the losses.
C.1 Extra material to the numerical results in Section 4.2
In Figure 3, we plot the loss functions and theoretical optimum, together with the disentangled loss functions computed separately on the fit term and the regularization terms for  $H_{1}$ ,  $W_{l}$ , with  $1 \leq l \leq L$ . As already discussed in Section 4.2, the DUFM finds the global optimum perfectly and exhibits superb DNC.
C.2 Effect of depth on DUFM training
As already seen by showing 3- and 6-layered DUFM, the depth does not play a significant role on the training of DUFM. However, an implicit bias of GD can be observed for very deep DUFMs. To demonstrate this, we plot the 10-DUFM training results in Figure 4.
Though not well visible on the plots, the DNC2 is sometimes achieved also on  $H_{1}$ ,  $H_{2}$ , which in the shallower DUFMs was an exception. This shows that with increasing depth, GD is more and more implicitly biased towards finding DNC2 representations for these feature matrices too.
Figure 3: Losses and partial losses for 3-DUFM (left) and 6-DUFM (right) as described in Section 4.2. The total training loss is compared to the numerically computed optimum (based on (17)). The unlabeled loss curves correspond to all the regularization losses.
Figure 4: Losses and DNC metric progressions as a function of training time. The unlabeled loss curves correspond to all the regularization losses. Here, the 10-DUFM is trained. The variance in some of the DNC metrics is due to the non-zero chance of qualitatively different behaviors for the first and second layer.
C.3 Effect of width on DUFM training
Unlike depth, width has a significant effect on the DUFM training, especially at small widths. The reason is that the model parameters often get stuck in sub-optimal configurations due to the difficult optimization landscape. To compare, we ran 4-DUFM training with the same hyperparameters except the width. We repeated the experiments for widths of 2, 5, 10, 20, 64, 1000. We plot all the results in Figure 5.
The results suggest that the risk of DUFM not reaching the global optimum and thus DNC drastically decreases with the width. Already for width 20, the DUFM did not fail to reach the global optimum once in ten runs. On the other hand, for width of only 2, the DUFM did not reach the global optimum in any run. Interestingly, the smaller widths yielded less expected behaviour. For instance for width 10, we encountered two runs both reaching a perfect optimum and DNC, of which one exhibited DNC2 also for  $H_{1}$ ,  $H_{2}$ , while the other did not. Noticeably, for width 1000 we see higher variance in some DNC metrics. This is because some of these specific metrics converged to very low values for some runs and for some other stayed comparably higher. This effect appears to be an implicit bias of SGD.
C.4 The effect of weight decay on DUFM training