SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Now, let us consider the special case when  $a^T b = 0$ ,  $a^T a = b^T b$  and  $s_1 = s_2$  for which the LICQ criterion is not satisfied. A very simple computation then yields that any feasible solution achieves the same objective value as the one above. This means that if the problem does have a global solution, it must be this one, otherwise the optimal solution would neither satisfy a KKT condition, nor be in the region where LICQ does not hold. Therefore, we must, finally, reason that this is indeed a globally optimal solution, which is not guaranteed by KKT conditions themselves. For this, it is only necessary to note that the feasible set is obviously compact. Since the objective value is continuous, we can use the well-known fact that such an optimization problem indeed has a global solution.
Now we are ready to finish the proof of Lemma 5. Let us first still assume  $\tilde{s}_1, \tilde{s}_2 > 0$ . Then, combining Lemma 6 with Lemma 7 we get:
$$
\begin{array}{l} \min  _ {W} \| V \\ \text {s . t .} \mathcal {S} (\sigma (W X)) = \left\{s _ {1}, s _ {2} \right\} \\ \end{array}
$$
$$
\frac {(s _ {1} ^ {2} + s _ {2} ^ {2}) (\tilde {s} _ {1} ^ {2} + \tilde {s} _ {2} ^ {2}) - (s _ {1} ^ {2} - s _ {2} ^ {2}) (\tilde {s} _ {1} ^ {2} - \tilde {s} _ {2} ^ {2})}{2 \tilde {s} _ {1} ^ {2} \tilde {s} _ {2} ^ {2}} = \frac {s _ {1} ^ {2}}{\tilde {s} _ {1} ^ {2}} + \frac {s _ {2} ^ {2}}{\tilde {s} _ {2} ^ {2}}.
$$
Now consider the case  $\tilde{s}_2 = 0$ . According to Lemma 6 this implies  $s_2 = 0$ . When optimizing over the matrix  $A$  we are facing the following optimization problem:
$$
\begin{array}{l} \min  _ {A} \frac {b ^ {T} b}{y ^ {T} y} \\ \begin{array}{l} \text {s . t .} \mathcal {S} (A) = \left\{s _ {1}, 0 \right\}, \end{array} \\ \frac {a ^ {T} a}{b ^ {T} b} = \frac {x ^ {T} x}{y ^ {T} y}, \\ \end{array}
$$
where the second constraint is clear from the considerations done in Lemma 6. Combining these two constraints, one simply gets
$$
b ^ {T} b = \frac {y ^ {T} y}{x ^ {T} x + y ^ {T} y} s _ {1} ^ {2}
$$
and plugging this in the objective value we have  $\frac{s_1^2}{s_1^2}$ , which is consistent with the statement of Lemma 7. The case when  $X$  is 0 is trivial. This proves Lemma 5.
Notice that, for all indices but  $l = 1$ , we managed to reduce the optimization over full matrices  $W_{l}$  to the optimization over only the singular values of  $\sigma(H_{k})$  matrices. We will do the same with the first layer's matrices  $W_{1}, H_{1}$ . For this, we again split the optimization of  $\frac{\lambda_{W_1}}{2}\|W_1\|_F^2 + \frac{\lambda_{H_1}}{2}\|H_1\|_F^2$  given  $\{s_1, s_2\} = S(\sigma(W_1H_1))$  into two nested sub-problems, the inner one optimizing the quantity only for a fixed output  $H_{2} := W_{1}H_{1}$  and the outer one optimizing over the output  $H_{2}$  given the optimal value of the inner problem. We can solve the inner subproblem by a direct use of the following lemma, which describes a variational form of a nuclear norm.
is minimized at value  $\sqrt{\lambda_A\lambda_B}\| C\| _*$ , and the minimizers are of the form  $A^{*} = \gamma_{A}U\Sigma^{1 / 2}R^{T},B^{*} = \gamma_{B}R\Sigma^{1 / 2}V^{T}$ . Here, the constants  $\gamma_{A},\gamma_{B}$  only depend on  $\lambda_{A},\lambda_{B}$ ;  $U\Sigma V^T$  is the SVD of  $C$ ; and  $R$  is an orthogonal matrix.
Proof. See Lemma C.1 of [28].
Using this Lemma 8, we know that the optimal value of the inner optimization problem is  $\sqrt{\lambda_{W_1}\lambda_{H_1}}\| H_2\|_*$ . Therefore we only need to solve the outer problem:
$$
\min  _ {H _ {2}} \| H _ {2} \| _ {*}
$$
$$
\begin{array}{l} \text {s . t .} \mathcal {S} (\sigma (H _ {2})) = \left\{s _ {1}, s _ {2} \right\}. \end{array}
$$
To solve this problem, we will state a much more general statement, which can be of independent interest, in the following lemma.
$$
\min  _ {H} \| H \| _ {S _ {\frac {L}{L}} ^ {\frac {2}{L}}} ^ {s. t.}, \quad s. t. \mathcal {S} (\sigma (H)) = \left\{s _ {1}, s _ {2} \right\} \tag {7}
$$
Proof. First note that this optimization problem has two free components:  $(i)\sigma (H)$  so that it satisfies the constraint, and  $(ii)H$  with  $\sigma (H)$  being fixed. This enables us to split the problem into inner and outer minimization. However, the complexity of the problem will prevent us from doing this conditioning in a straightforward way. Further, let us also implicitly note that within the proof we implicitly assume the matrix in question has at least three rows. The case with two rows is very simple and follows the proof with more rows.
We start by explicitly computing the objective value as a function of important statistics of  $H$ . Let  $\sigma(H)$  have columns  $r, s$ . Then,
$$
\sigma (H) ^ {T} \sigma (H) = \left( \begin{array}{c c} r ^ {T} r & r ^ {T} s \\ r ^ {T} s & s ^ {T} s \end{array} \right).
$$
Using the well-known formula on the eigenvalues of such matrix, we know  $\lambda_1 + \lambda_2 = r^T r + s^T s$  and  $\lambda_1\lambda_2 = r^T rs^T s - (r^T s)^2$ . Solving this system we get:
$$
s _ {1, 2} ^ {2} = \lambda_ {1, 2} = \frac {r ^ {T} r + s ^ {T} s \pm \sqrt {\left(r ^ {T} r + s ^ {T} s\right) ^ {2} - 4 \left(r ^ {T} r s ^ {T} s - \left(r ^ {T} s\right) ^ {2}\right)}}{2}. \tag {14}
$$
Let us, for now, fix this  $\sigma(H)$ . For a while we will only optimize over  $H$  s.t.  $\sigma(H)$  is fixed. We will think about the optimization problem in the following way: consider any  $\delta \geq 0$ . Assume we are allowed to "inject" negative values in the place of zero entries of a fixed  $\sigma(H)$  so that the sum of squares of those negative entries is exactly  $\delta$ . What is the minimal value of the objective value given this  $\delta$  and what is the optimal injection? Let us denote any quantity which was subject to such injection (or perturbation) with an extra lower-index  $P$ .
To answer this question, first have a closer look at (14). It is clear that after injecting the negative noise of total squared size  $\delta$ , the  $r_P^T r_P + s_P^T s_P$  in a perturbed matrix will increase by exactly  $\delta$  compared to  $r^T r + s^T s$ . Therefore, if we condition on  $\delta$ , the only degree of freedom lies in the  $-4(r_P^T r_P s_P^T s_P - (r_P^T s_P)^2)$  in the perturbed matrix. Crucially, since the objective value is  $\lambda_{1,P}^{1/L} + \lambda_{2,P}^{1/L}$ , by concavity of  $1/L$ -th power, for any fixed sum of  $\lambda_{1,P} + \lambda_{2,P} = r^T r + s^T s + \delta$  we want the  $\lambda_{i,P}$  to be as far apart as possible. This is obtainable by maximizing the quantity  $-4(r_P^T r_P s_P^T s_P - (r_P^T s_P)^2)$  which is equivalent to minimizing  $r_P^T r_P s_P^T s_P - (r_P^T s_P)^2$ . Importantly, as it will be clear from the forthcoming reasoning, the minimal value of  $\lambda_{2,P}$  is decreasing with
$\delta$ . Therefore, it only makes sense to consider all the  $\delta$  values for which the minimal  $\lambda_{2,P}$  is still bigger than 0 up to the  $\delta$  value for which the minimal  $\lambda_{2,P}$  hits 0 for the first time. From that  $\delta$  on, every optimal  $\lambda_{2,P}$  is 0, but the  $\lambda_{1,P}$  increases and thus becomes sub-optimal. Now let us split the  $\delta$  into  $\delta_1$  and  $\delta_2$ , denoting the total squared size of perturbations per column. If we also fix  $\delta_1$  and  $\delta_2$  and condition on them, even the quantity  $r_P^T r_P s_P^T s_P$  is fixed and we are left with maximizing  $(r_P^T s_P)^2$ . Denote  $p = r^T s$ . To make the next argument clear, let us split the matrix  $\sigma(H)$  into four blocks of rows, where in each block there is one of the four following types of rows:  $(+, +), (+, 0), (0, +), (0, 0)$ . We will now define variables  $a, b, c, d$  as sum of squared entries corresponding to one column of one block. Specifically:
$$
a = \sum_ {i; \sigma (H) _ {i 1} > 0, \sigma (H) _ {i 2} > 0} \sigma (H) _ {i 1} ^ {2}; \quad b = \sum_ {i; \sigma (H) _ {i 1} > 0, \sigma (H) _ {i 2} > 0} \sigma (H) _ {i 2} ^ {2}
$$
$$
c = \sum_ {i; \sigma (H) _ {i 1} > 0, \sigma (H) _ {i 2} = 0} \sigma (H) _ {i 1} ^ {2}; \quad d = \sum_ {i; \sigma (H) _ {i 1} = 0, \sigma (H) _ {i 2} > 0} \sigma (H) _ {i 2} ^ {2}.
$$
The value we are trying to minimize equals, for an unperturbed  $\sigma(H)$  written in this way,  $(a + c)(b + d) - p^2$ . Of course,  $\sigma(H)$  might not have all the types of blocks, in which case the empty sum is just defined as 0. First, we argue that whatever is the optimal assignment of the negative values given  $\delta_1, \delta_2$ , they do not appear in both the  $(0,0)$  block and  $(+,0)$ ,  $(0, +)$  blocks. This is because the only variable we are optimizing for in this conditioning is  $(r_P^T s_P)^2$ . Assume that, given the perturbation, the quantity  $r_P^T s_P$  is negative. Then, any negative values present in the  $(0,0)$  block can be moved to  $(+,0)$ ,  $(0, +)$  blocks strictly decreasing  $r_P^T s_P$  and thus increasing  $(r_P^T s_P)^2$ . On the other hand, if  $r_P^T s_P$  happens to be positive, then any negative values present in the  $(+,0)$ ,  $(0, +)$  blocks can be moved to the  $(0,0)$  block, strictly increasing the  $r_P^T s_P$  value and thus  $(r_P^T s_P)^2$  too. Therefore, we only need to consider cases when we inject the negative values to either the  $(0,0)$  block or to  $(+,0)$ ,  $(0, +)$  blocks, but not both at once. Let us start with the second case and analyze it. Clearly, if  $\sigma(H)$  did not have either the  $(0,0)$  block or  $(+,0)$ ,  $(0, +)$  blocks, then we do not even need this reasoning anymore. Also it is clear, as we will see later, that a matrix  $\sigma(H)$  which would completely lack 0 entries is suboptimal and thus there is always space to inject negative entries.
Case 1: Injecting negative values to  $(+,0),(0, + )$  blocks. For the same definition of  $a,b,c,d$  as above, we further define the squared sizes of total injected negative values in the perturbed  $H_{P}$  per block as  $x,y$ , i.e.,
$$
x = \sum_ {i; \sigma (H) _ {i 1} > 0, \sigma (H) _ {i 2} = 0} H _ {P, i 2} ^ {2}; \quad y = \sum_ {i; \sigma (H) _ {i 1} = 0, \sigma (H) _ {i 2} > 0} H _ {P, i 1} ^ {2},
$$
and thus  $x = \delta_2, y = \delta_1, x + y = \delta$ . Trivially, such a construction only makes sense if it is possible to make  $r_P^T s_P$  negative, otherwise it is strictly better to construct an alternative matrix  $\overline{\sigma(H)}$  with  $a + c = \overline{a} + \overline{c}$ ;  $b + d = \overline{b} + \overline{d}$ ;  $p = \overline{p}$ , but which includes the  $(0,0)$  block and put all the negative value there (this is always possible for matrices with at least 3 rows, since we only need two rows to construct  $\overline{\sigma(H)}$  which satisfies the outlined constraints). This alternative matrix together with negative values in the  $(0,0)$  block would be better, because obviously  $(\overline{r_P^T s_P})^2 > (r_P^T s_P)^2$ .
Once  $r_P^T s_P$  is negative, though, it is clear that it is optimal, given the resources  $x, y$ , to produce as negative value as possible so as to maximize  $(r_P^T s_P)^2$ . By the use of Cauchy-Schwartz inequality, this is possible exactly when the injected negative vectors in the blocks are aligned with their positive counterparts of that block. Then, the inner product between the negative and positive part of a block will equal exactly  $\sqrt{cx}$  or  $\sqrt{yd}$ . Furthermore, if it is possible to construct  $\overline{\sigma(H)}$  for which  $\overline{a} + \overline{b} < a + b$ ;  $\overline{a} + \overline{c} = a + c$ ;  $\overline{b} + \overline{d} = b + d$ ;  $\overline{p} = p$  (which is only possible if there are at least two rows in the (+, +) block and if  $p^2 < ab$  - thanks to the fact that then the angle between columns in the (+, +) block is a free variable and we can decrease it while decreasing  $a, b$  to keep  $p$  unchanged using the law of cosine), then our objective value strictly decreases, since  $\sqrt{cx} + \sqrt{yd} > \sqrt{cx} + \sqrt{dy}$ . Therefore, it is always optimal for the first and second column of the (+, +) block to be aligned, which also yields  $ab = p^2$ . This value of  $p$  is necessary also if the (+, +) block has only one row. Thus, we have that the original value of  $r^T rs^T s - (r^T s)^2$  is  $(a + c)(b + d) - ab$ . The value after perturbation, using all the knowledge we acquired so far is  $(a + c + y)(b + d + x) - (\sqrt{ab} - \sqrt{cx} - \sqrt{dy})^2$ . Subtracting, we get a difference:  $by + ax + xy + 2\sqrt{abc}\sqrt{x} + 2\sqrt{abd}\sqrt{y} - 2\sqrt{cd}\sqrt{xy}$ . Consider now an alternative
matrix  $\overline{H_P}$  such that  $x, y$  remain unchanged and moreover  $\overline{a} = \overline{b} = 0$ ,  $\overline{c} + \overline{d} = a + b + c + d$  and  $\overline{cd} = (a + c)(b + d) - ab$ , meaning that the  $\sigma(\overline{H}_P)$  has the same singular values as  $\sigma(H_P)$ . However, computing the same difference of the same quantities one gets:  $xy - 2\sqrt{\overline{cd}}\sqrt{xy}$ . This value is strictly smaller than the previous one for any fixed pair of  $x, y$ . Therefore, a matrix  $\sigma(H)$  which does not contain the  $(+, +)$  block is the only possible optimal solution for the injection which goes into  $(+, 0)$ ,  $(0, +)$  blocks. Thus, we can reduce the previous problem into considering a matrix  $H_P$  with  $a = b = 0$  and minimizing the difference in objective values  $xy - 2\sqrt{cd}\sqrt{xy}$ , given  $x + y = \delta$ . After plugging in  $y = \delta - x$ , differentiating for  $x$  and putting equal to 0, we get  $\delta - 2x - \frac{\sqrt{cd}}{\sqrt{x(\delta - x)}} (\delta - 2x) = 0$ , which is equivalent to  $(\delta - 2x)(1 - \frac{\sqrt{cd}}{\sqrt{x(\delta - x)}}) = 0$ . From here, we see that either  $x = \delta / 2$  or  $cd = x(\delta - x)$ , which after solving yields  $x_{1,2} = \frac{\delta \pm \sqrt{\delta^2 - 4cd}}{2}$ . If  $\delta^2 \leq 4cd$  then the only stationary point is  $\delta / 2$  and this is clearly an optimal solution. For  $\delta^2 = 4cd$ , the optimal solution yields a rank 1 matrix so that  $\lambda_{2,P} = 0$ . This means we do not need to care about  $\delta^2 > 4cd$ , as this would not achieve smaller value of  $\lambda_{2,P}$ , but would increase the value of  $\lambda_{1,P}$ . So, the solution where  $x = y = \delta / 2$  is optimal and the objective value is  $\delta^2 / 4 - s_1s_2\delta$ . This is clearly smaller than 0 so we actually achieved a decrease in objective value.
Case 2: Injecting negative values to  $(0,0)$  block. For this, it is not necessary to distinguish  $(+, +), (+, 0), (0, +))$  blocks, so we treat them as one. We can define:
$$
e = \sum_ {i; \sigma (H) _ {i 1} + \sigma (H) _ {i 2} > 0} \sigma (H) _ {i 1} ^ {2} +; \quad f = \sum_ {i; \sigma (H) _ {i 1} + \sigma (H) _ {i 2} > 0} \sigma (H) _ {i 2} ^ {2}
$$
$$
x = \sum_ {i; \sigma (H) _ {i 1} = 0, \sigma (H) _ {i 2} = 0} \sigma (H) _ {i 1} ^ {2}; \quad y = \sum_ {i; \sigma (H) _ {i 1} = 0, \sigma (H) _ {i 2} = 0} \sigma (H) _ {i 2} ^ {2}.
$$
The original objective value is  $ef - p^2$ , while the perturbed one is  $(e + x)(f + y) - (p + \sqrt{xy})^2$  and so the difference is  $xf + ye - 2p\sqrt{xy}$ . However, using Cauchy-Schwartz inequality, this is bigger or equal to  $xf + ye - 2\sqrt{fexy} \geq 0$ . Thus, it is not possible to improve the objective value in this case, unlike the previous case where it was. Therefore, this case is clearly sub-optimal and we do not need to deal with it.
The only thing that could potentially go wrong with this proof is that there is not enough rows to build our necessary blocks. This can only happen if  $\sigma(H)$  is a  $2 \times 2$  matrix. This case is, however, very simple and we will not explicitly treat it, but the statement and the proof path are the same for it.
We solved our optimization problem for every fixed allowed perturbation size  $\delta$ . Now we just need to optimize over  $0 \leq \delta \leq 2s_1s_2$ . So we face the following optimization problem:
$$
\begin{array}{l} \min  _ {0 \leq \delta \leq 2 s _ {1} s _ {2}} \left(\frac {s _ {1} ^ {2} + s _ {2} ^ {2} + \delta}{2} + \frac {1}{2} \sqrt {(s _ {1} ^ {2} + s _ {2} ^ {2} + \delta) ^ {2} - 4 (s _ {1} ^ {2} s _ {2} ^ {2} + \delta^ {2} / 4 - s _ {1} s _ {2} \delta)}\right) ^ {\frac {1}{L}} \\ + \left(\frac {s _ {1} ^ {2} + s _ {2} ^ {2} + \delta}{2} - \frac {1}{2} \sqrt {(s _ {1} ^ {2} + s _ {2} ^ {2} + \delta) ^ {2} - 4 (s _ {1} ^ {2} s _ {2} ^ {2} + \delta^ {2} / 4 - s _ {1} s _ {2} \delta)}\right) ^ {\frac {1}{L}}. \\ \end{array}
$$
Denote  $\lambda_{\delta,1}$  the value of the first summand of the above objective value (without  $1 / L$ -th power) and similarly  $\lambda_{\delta,2}$  the value of the second summand (so those are the eigenvalues of optimal  $H_P^T H_P$  given exactly  $\delta$  allowed negative injection). The derivative of the objective w.r.t.  $\delta$  equals:
$$
\frac {1}{n \lambda_ {\delta , 1} ^ {\frac {L - 1}{L}}} \left(\frac {1}{2} + \frac {(s _ {1} + s _ {2}) ^ {2}}{2 D _ {\delta}}\right) + \frac {1}{n \lambda_ {\delta , 2} ^ {\frac {L - 1}{L}}} \left(\frac {1}{2} - \frac {(s _ {1} + s _ {2}) ^ {2}}{2 D _ {\delta}}\right),
$$
where  $D_{\delta} \coloneqq \sqrt{(s_1^2 + s_2^2 + \delta)^2 - 4(s_1^2s_2^2 + \delta^2 / 4 - s_1s_2\delta}$  denotes the square root part of the  $\lambda_{\delta;1,2}$  expression. We will prove that this derivative is either 0 for  $L = 2$  or strictly negative on  $[0, 2s_1s_2]$  everywhere except a set of measure 0 for  $L \geq 3$ . This guarantees for  $L = 2$  the optimal value being equal to the unperturbed matrix while also defines the set of optimal solutions. For  $L \geq 3$  the optimal value is attained at  $\delta = 2s_1s_2$ , the objective value is easily expressible and the optimal solution is characterized by a single perturbation size. To test whether this expression is smaller than 0, nothing happens if we multiply it with  $2L\lambda_{\delta,1}^{\frac{L-1}{L}}\lambda_{\delta,2}^{\frac{2L-2}{L}}$ , so we get:
$$
\lambda_ {\delta , 1} ^ {\frac {2 L - 2}{L}} \left(1 + \frac {(s _ {1} + s _ {2}) ^ {2}}{D _ {\delta}}\right) + \lambda_ {\delta , 1} ^ {\frac {L - 1}{L}} \lambda_ {\delta , 2} ^ {\frac {L - 1}{L}} \left(1 - \frac {(s _ {1} + s _ {2}) ^ {2}}{D _ {\delta}}\right).
$$
We can now compute
$$
\lambda_ {\delta , 1} \lambda_ {\delta , 2} = \frac {(s _ {1} ^ {2} + s _ {2} ^ {2} + \delta) ^ {2} - (s _ {1} ^ {2} - s _ {2} ^ {2}) ^ {2} - 2 (s _ {1} + s _ {2}) \delta}{4} = \left(\frac {2 s _ {1} s _ {2} - \delta}{2}\right) ^ {2}
$$
$$
\lambda_ {\delta , 2} ^ {\frac {2 L - 2}{L}} \left(1 + \frac {(s _ {1} + s _ {2}) ^ {2}}{D _ {\delta}}\right) + \left(\frac {2 s _ {1} s _ {2} - \delta}{2}\right) ^ {\frac {2 L - 2}{L}} \left(1 - \frac {(s _ {1} + s _ {2}) ^ {2}}{D _ {\delta}}\right).
$$
This expression resembles a form for which the use of Jensen inequality would be convenient. However, there are two issues. First, the coefficients, though summing up to 1 are not a convex combination since  $\frac{(s_1 + s_2)^2}{D_\delta} \geq 1$ . So this is a combination outside of the convex hull of the inputs. However, Jensen inequality outside the convex hull is applicable with opposite sign. The bigger issue is that the function to use the Jensen inequality on,  $f(x) = x^{\frac{2L - 2}{L}}$  is only easily defined on  $\mathbb{R}_+$ . To make sure that we can safely use Jensen inequality, we first need to check that the linear combination, though outside of the convex hull of the inputs, still lies in  $\mathbb{R}_+$ . We are therefore asking whether
$$
\lambda_ {\delta , 2} \left(1 + \frac {(s _ {1} + s _ {2}) ^ {2}}{D _ {\delta}}\right) + \frac {2 s _ {1} s _ {2} - \delta}{2} \left(1 - \frac {(s _ {1} + s _ {2}) ^ {2}}{D _ {\delta}}\right) \geq 0.
$$
Now, we can multiply out and simplify this expression (after multiplying by 2):
$$
\begin{array}{l} s _ {1} ^ {2} + s _ {2} ^ {2} + \delta - D _ {\delta} + (s _ {1} ^ {2} + s _ {2} ^ {2}) \frac {(s _ {1} + s _ {2}) ^ {2}}{D _ {\delta}} + \delta \frac {(s _ {1} + s _ {2}) ^ {2}}{D _ {\delta}} - (s _ {1} + s _ {2}) ^ {2} + 2 s _ {1} s _ {2} \\ - \delta - \frac {2 s _ {1} s _ {2} (s _ {1} + s _ {2}) ^ {2}}{D _ {\delta}} + \frac {\delta (s _ {1} + s _ {2}) ^ {2}}{D _ {\delta}} \geq 0 \Longleftrightarrow \\ \end{array}
$$
$$
\left(\left(s _ {1} - s _ {2}\right) ^ {2} + 2 \delta\right) \left(s _ {1} + s _ {2}\right) ^ {2} \geq \left(s _ {1} ^ {2} - s _ {2} ^ {2}\right) ^ {2} + 2 \left(s _ {1} + s _ {2}\right) ^ {2} \delta ,
$$
but this holds even with equality. So we proved that the combination of inputs we are considering here is exactly 0 for which  $x^{\frac{2L - 2}{L}}$  is well-defined. So we can freely use Jensen inequality:
$$
\begin{array}{l} \lambda_ {\delta , 2} ^ {\frac {2 L - 2}{L}} \left(1 + \frac {(s _ {1} + s _ {2}) ^ {2}}{D _ {\delta}}\right) + (2 s _ {1} s _ {2} - \delta) ^ {\frac {2 L - 2}{L}} \left(1 - \frac {(s _ {1} + s _ {2}) ^ {2}}{D _ {\delta}}\right) \leq \\ \left(\lambda_ {\delta , 2} \left(1 + \frac {(s _ {1} + s _ {2}) ^ {2}}{D _ {\delta}}\right) + \frac {2 s _ {1} s _ {2} - \delta}{2} \left(1 - \frac {(s _ {1} + s _ {2}) ^ {2}}{D _ {\delta}}\right)\right) ^ {\frac {2 L - 1}{L}} = 0. \\ \end{array}
$$
Note that if  $L = 2$ , the function we applied Jensen inequality on was, in fact, linear and thus we retained equality. If  $L \geq 3$ , then since the function in question is strictly convex, the equality can only hold if  $\lambda_{\delta,2} = 2s_1s_2 - \delta$ . After writing down the expression on  $\lambda_{\delta,2}$  it becomes clear that this is equivalent to a quadratic equation on  $\delta$ , which can have at most two solutions. So we proved that the derivative is smaller than 0 on the whole interval of interest except a set of measures at most 0, where it is 0. Therefore, the only optimum is at  $\delta = 2s_1s_2$ . Plugging this in the objective value one gets  $(s_1 + s_2)^{\frac{2}{L}}$  and from the proof we also get the precise description of the optimizers for both  $L = 2$  and  $L > 2$  cases.
Now we are ready to formally make the reduction of the optimization problem in (10) into the optimization over only singular values of the matrices  $\sigma(H_{l})$  for  $2 \leq l \leq L$ . We formulate this reduction in a separate lemma for clarity.
Lemma 12. The optimal value of the following optimization problem:
$$
\begin{array}{l} \min _ {W _ {L}, \dots , W _ {1}, H _ {1}} \frac {1}{4} \| W _ {L} \sigma (W _ {L - 1} \dots W _ {2} \sigma (W _ {1} H _ {1})) - I _ {2} \| _ {F} ^ {2} + \sum_ {l = 1} ^ {L} \frac {\lambda_ {W _ {l}}}{2} \| W _ {l} \| _ {F} ^ {2} + \frac {\lambda_ {H _ {1}}}{2} \| H _ {1} \| _ {F} ^ {2} \\ s. t. \left\{s _ {l, 1}; s _ {l, 2} \right\} = \mathcal {S} \left(\sigma \left(H _ {l}\right)\right) \forall 2 \leq l \leq L \\ \end{array}
$$
with  $H_{1}$  having two columns and all the matrices having at least two rows is
$$
\sum_ {i = 1} ^ {2} \left(\frac {\lambda_ {W _ {L}}}{2 (s _ {L , i} ^ {2} + 2 \lambda_ {W _ {L}})} + \sum_ {l = 2} ^ {L - 1} \left(\frac {\lambda_ {W _ {l}}}{2} \frac {s _ {l + 1 , i} ^ {2}}{s _ {l , i} ^ {2}}\right) + \sqrt {\lambda_ {W _ {1}} \lambda_ {H _ {1}}} s _ {2, i}\right),
$$
where the  $\frac{s_{l+1,i}^2}{s_{l,i}^2}$  is defined as 0 if  $s_{l+1,i}^2 = s_{l,i}^2 = 0$ .
Now we are only left with optimizing the objective over all the  $s_{l,i}$ . Crucially, the objective function is separable and symmetric in the  $i$  index. Therefore we can optimize for each index separately. Importantly, if there is only a single solution to this problem, then the solution must be the same for both indices  $i$ , which will yield orthogonality of the  $\sigma(H_l)$  matrices. The ultimate goal is to show that this is indeed the case modulo some special circumstances. The optimal value of this reduced optimization problem is presented in the following, final lemma with abstract notation of the free variables.
Proof. Here we will need to distinguish the cases  $L = 2$  and  $L > 2$ . We start with the case  $L = 2$ . Let us reparametrize the problem so that we will work with  $y = \sqrt{x_2}$ . We get the following expression to optimize the  $y$  over:
$$
\frac {\lambda_ {W _ {2}}}{2 \left(y ^ {2} + 2 \lambda_ {W _ {2}}\right)} + \sqrt {\lambda_ {W _ {1}} \lambda_ {H _ {1}}} y.
$$