SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We will solve this problem using a sequence of  $L$  sub-problems, where in sub-problem  $l$ , we only optimize over matrices with index  $L - l + 1$  and condition on all the other matrices, as well as singular values of all  $\sigma(H_l)$  for  $l \geq 2$  (keeping them fixed). Throughout the process, we will strongly rely on the crucial phenomenon that the optimal solutions of all the sub-problems will only depend on the singular values of the matrices  $\sigma(H_l)$ . We will solve the problem starting from the last weight matrix  $W_L$ , through  $W_{L-1}$  and so on, and finishing with the joint optimization over  $W_1, H_1$ . We divide this procedure into 4 lemmas. The first lemma will only treat  $W_L$ . The second lemma will treat  $W_l$  for  $2 \leq l \leq L-1$ . This lemma will only be used if  $L \geq 3$ . The third lemma will treat  $W_1, H_1$  jointly and the fourth, final lemma will then treat the only free variables left - the singular values of  $\sigma(H_l)$  for all  $l \geq 2$ . Let us denote by  $S(\cdot)$  an operator which returns a set of singular values of any given matrix (in full SVD, i.e. the SVD with number of singular values equal to the smaller dimension of the matrix). Moreover, let  $s_{l,i}$  be the  $i$ -th singular value of  $\sigma(H_l)$ .
Lemma 4. Fix  $H_{1}, W_{1}, \ldots, W_{L-1}$ . Then, the optimal solution of the optimization problem
$$
\min _ {W _ {L}} \frac {1}{4} \| W _ {L} \sigma (W _ {L - 1} (\dots W _ {2} \sigma (W _ {1} H _ {1})) - I _ {2} \| _ {F} ^ {2} + \frac {\lambda_ {W _ {L}}}{2} \| W _ {L} \| _ {F} ^ {2}
$$
Proof. This is a well-known fact from the theory of ridge regression [10]. However, for readers' convenience, we do the proof here too. If we denote the objective of this as  $\mathcal{L}$ , then we get:
$$
\frac {\partial \mathcal {L}}{\partial W _ {L}} = \frac {1}{2} \left(W _ {L} \sigma \left(H _ {L}\right) - I _ {2}\right) \sigma \left(H _ {L}\right) ^ {T} + \lambda_ {W _ {L}} W _ {L}.
$$
Since the objective is strongly convex in  $W_{L}$ , the unique global optimum is found by setting to 0 the derivative. Solving for it, we have
$$
W _ {L} = \sigma (H _ {L}) ^ {T} (\sigma (H _ {L}) \sigma (H _ {L}) ^ {T} + 2 \lambda_ {W _ {L}} I _ {d _ {L}}) ^ {- 1}.
$$
Let us denote  $\sigma(H_L) = U\Sigma V^T$  the rectangular SVD of  $\sigma(H_L)$  where  $U \in \mathbb{R}^{d_L \times d_L}$ ,  $\Sigma \in \mathbb{R}^{d_L \times 2}$ ,  $V \in \mathbb{R}^{2 \times 2}$ . Writing the optimal  $W_L$  in terms of SVD of  $\sigma(H_L)$  we get:
$$
\begin{array}{l} W _ {L} = V \Sigma^ {T} U ^ {T} \left(U \Sigma \Sigma^ {T} U ^ {T} + 2 \lambda_ {W _ {L}} I _ {d _ {L}}\right) ^ {- 1} \\ = V \Sigma^ {T} U ^ {T} U \left(\Sigma \Sigma^ {T} + 2 \lambda_ {W _ {L}} I _ {d _ {L}}\right) ^ {- 1} U ^ {T} \\ = V \Sigma^ {T} \left(\Sigma \Sigma^ {T} + 2 \lambda_ {W _ {L}} I _ {d _ {L}}\right) ^ {- 1} U ^ {T}. \\ \end{array}
$$
Therefore, we obtain the SVD of  $W_{L}$  and that  $(\Sigma \Sigma^{T} + 2\lambda_{W_{L}}I_{d_{L}})^{-1}$  is a matrix of singular values of  $W_{L}$ , from which we obtain:
$$
\| W _ {L} \| _ {F} ^ {2} = \sum_ {i = 1} ^ {2} \frac {s _ {L , i} ^ {2}}{(s _ {L , i} ^ {2} + 2 \lambda_ {W _ {L}}) ^ {2}}.
$$
Furthermore, the first summand of the objective is
$$
\begin{array}{l} \left\| W _ {L} \sigma (H _ {L}) - I _ {2} \right\| _ {F} ^ {2} = \left\| V \Sigma^ {T} (\Sigma \Sigma^ {T} + 2 \lambda_ {W _ {L}} I _ {d _ {L}}) ^ {- 1} U ^ {T} U \Sigma V ^ {T} - I _ {2} \right\| _ {F} ^ {2} \\ = \left\| V \left(\Sigma^ {T} \left(\Sigma \Sigma^ {T} + 2 \lambda_ {W _ {L}} I _ {d _ {L}}\right) ^ {- 1} U ^ {T} U \Sigma\right) V ^ {T} - I _ {2} \right\| _ {F} ^ {2} \\ = \sum_ {i = 1} ^ {2} \left(\frac {s _ {L , i} ^ {2}}{s _ {L , i} ^ {2} + 2 \lambda_ {W _ {L}}} - 1\right) ^ {2}. \\ \end{array}
$$
Putting everything together, we get the following optimal value of the objective:
$$
\begin{array}{l} \frac {1}{4} \sum_ {i = 1} ^ {2} \left(\frac {s _ {L , i} ^ {2}}{s _ {L , i} ^ {2} + 2 \lambda_ {W _ {L}}} - 1\right) ^ {2} + \frac {\lambda_ {W _ {L}}}{2} \sum_ {i = 1} ^ {2} \frac {s _ {L , i} ^ {2}}{(s _ {L , i} ^ {2} + 2 \lambda_ {W _ {L}}) ^ {2}} \\ = \sum_ {i = 1} ^ {2} \frac {2 \lambda_ {W _ {L}} ^ {2}}{2 (s _ {L , i} ^ {2} + 2 \lambda_ {W _ {L}}) ^ {2}} + \frac {\lambda_ {W _ {L}} \sigma_ {1} ^ {2}}{2 (s _ {L , i} ^ {2} + 2 \lambda_ {W _ {L}}) ^ {2}} \\ = \sum_ {i = 1} ^ {2} \frac {\lambda_ {W _ {L}}}{2 \left(s _ {L , i} ^ {2} + 2 \lambda_ {W _ {L}}\right)}, \\ \end{array}
$$
which gives the desired result.
Note that the term we optimized for is the only part of the objective in (10) which depends on  $W_{L}$ . Now we are ready to state the crucial lemma (necessary when  $L \geq 3$ ), which is presented in an abstract way so it can be easily applied outside of the scope of this work.
$$
\min  _ {W} \| W \| _ {F} ^ {2}, \quad \text {s . t .} \left\{s _ {1}, s _ {2} \right\} = \mathcal {S} (\sigma (W X)),
$$
Proof. Notice that, for any fixed value of the matrix  $A \coloneqq \sigma(WX)$ , there is at least one  $W$  which achieves such a value of  $A$ , unless the rank of  $A$  is bigger than the rank of  $X$ , however this is a special case explicitly mentioned in the statement of the lemma for which the problem is non-feasible. Therefore, we can split the optimization into solving two nested sub-problems. The inner sub-problem fixes an output  $A$  for which  $\{s_1, s_2\} = S(A)$  and optimizes only over all  $W$  for which  $A = \sigma(WX)$ . The outer problem then takes the optimal value of this inner problem and optimizes over all the feasible outputs  $A$  satisfying the constraint on the singular values. We will phrase both of the nested sub-problems as separate lemmas:
Proof. Note that this optimization problem is separable in rows of  $W$ . Thus it suffices to solve the problem for a single row  $w_{j}$  and then sum up over all rows. We will, moreover, split the analysis into two cases.
Case 1: Here we will assume that the row of  $A$  we are optimizing for is  $(a_j, b_j)$ , where  $a_j, b_j > 0$ . In this case, we are facing the following (simple) convex optimization problem:
$$
\min  _ {w _ {j}} \| w _ {j} \| _ {2} ^ {2}
$$
$$
\begin{array}{l} \text {s . t .} w _ {j} ^ {T} x = a _ {j} \end{array}
$$
$$
w _ {j} ^ {T} y = b _ {j}
$$
This has a known solution - the pseudoinverse as it is the minimum interpolating solution of a two-datapoint linear regression. However, using the formula would not simplify the expressions in the slightest and so we will solve this problem manually. The Lagrange function for this problem is
$$
\mathcal {L} \left(w _ {j}, u, v\right) = \sum_ {i = 1} ^ {d} w _ {j i} ^ {2} + u \left(\sum_ {i = 1} ^ {d} w _ {j i} x _ {i} - a _ {j}\right) + v \left(\sum_ {i = 1} ^ {d} w _ {j i} y _ {i} - b _ {j}\right),
$$
where  $d$  is the dimension of the columns of  $X$  and rows of  $W$ . We know that solving the KKT conditions for a convex problem with linear constraints yields sufficient and necessary conditions for the optimality. Let us write:
$$
\frac {\partial \mathcal {L}}{\partial w _ {j i}} = 2 w _ {j i} + u x _ {i} + v y _ {i} \stackrel {!} {=} 0.
$$
Together with constraints, these are the only KKT conditions for this problem. Solving for  $w_{ji}$  we get:
$$
w _ {j i} = \frac {- u x _ {i} - v y _ {i}}{2}. \tag {11}
$$
Plugging in the constraints we get:
$$
\sum_ {i = 1} ^ {d} v y _ {i} ^ {2} + u x _ {i} y _ {i} = v y ^ {T} y + u x ^ {T} y = - 2 b _ {j},
$$
$$
\sum_ {i = 1} ^ {d} u x _ {i} ^ {2} + v x _ {i} y _ {i} = u x ^ {T} x + v x ^ {T} y = - 2 a _ {j}.
$$
We can solve for  $v$  from the first equation and plug in to the second equation. Since  $y^{T}y$  must be non-zero, otherwise the constraint  $w_{j}^{T}y = b_{j} > 0$  could not be satisfied, we know  $v$  and  $u$  are well-defined. We get:
$$
u = \frac {2 b _ {j} x ^ {T} y - 2 a _ {j} y ^ {T} y}{x ^ {T} x y ^ {T} y - (x ^ {T} y) ^ {2}}, \tag {12}
$$
unless  $x = \alpha y$ , which we will deal with later. Now we can plug  $u$  in the expression on  $v$  and after some computations we get
$$
v = \frac {2 a _ {j} x ^ {T} y - 2 b _ {j} x ^ {T} x}{x ^ {T} x y ^ {T} y - (x ^ {T} y) ^ {2}}.
$$
Combining, we get the optimal  $w_{j}$ :
$$
\frac {b _ {j} x ^ {T} x - a _ {j} x ^ {T} y}{x ^ {T} x y ^ {T} y - (x ^ {T} y) ^ {2}} y + \frac {a _ {j} y ^ {T} y - b _ {j} x ^ {T} y}{x ^ {T} x y ^ {T} y - (x ^ {T} y) ^ {2}} x.
$$
If  $x = \alpha y$ , then it is clear from (12) that  $b_{j}x^{T}y = a_{j}y^{T}y$ , from which we can deduce  $\alpha = a_{j} / b_{j}$ . From the expression (11) for  $w_{ji}$ , it is clear that the only solution in this case is  $w_{j} = \frac{b_{j}}{y^{T}y} y$  and the same must hold for all other  $j \in \{1, \ldots, \tilde{d}\}$ , where  $\tilde{d}$  is the number of rows of  $W$ .
Case 2: Here we will assume that the row of  $A$  we are optimizing for is, w.l.o.g.  $(a_j, 0)$ , where  $a_j > 0$ . In this case, we are facing the following (simple) convex optimization problem:
$$
w _ {j} ^ {T} y \leq 0
$$
Again, we can write the Lagrange function for this problem and again, solving for KKT conditions will provide full description of the optimal solution:
$$
\mathcal {L} \left(w _ {j}, u, v\right) = \sum_ {i = 1} ^ {d} w _ {j i} ^ {2} + u \sum_ {i = 1} ^ {d} w _ {j i} y _ {i} + v \left(\sum_ {i = 1} ^ {d} w _ {j i} x _ {i} - a _ {j}\right).
$$
We obtain the same condition for  $w_{ji}$  as in case 1. However, since one of our constraints is now an inequality, we will deal with complementarity conditions. Let us first assume that  $u = 0$ . In that case  $w_{ji} = -vx_i / 2$  and from the first constraint
$$
v = \frac {- 2 a _ {j}}{\sum x _ {i} ^ {2}}.
$$
Thus,  $w_{j} = \frac{a_{j}}{x^{T}x} x$ . However judging from this, to satisfy the first constraint, we must have  $x_{i}y_{i} = 0$  for all  $i$  (remember that  $\sum w_{ji}y_{i}$  must be non-positive and that  $x,y$  have non-negative entries). Therefore necessarily  $x^{T}y = 0$ . However, looking at the solution of case 1, we see that after plugging in  $x^{T}y = 0$  and  $b_{j} = 0$ , we get the same expression. Thus, this is a special case of case 1. The second option for complementarity condition is that  $\sum w_{ji}y_{i} = 0$ . Then, we can proceed in the same way as in case 1 to obtain the same formula after plugging in  $b = 0$ . Then again, this is a special case of the formula in case 1.
All in all, we have fully solved the optimization problem in question. Now it remains to compute the optimal value. In the special case that  $x = \alpha y$ , we must have  $bx^{T}y = ay^{T}y$ , otherwise there is no feasible solution. In this case this is equivalent to  $\alpha = a / b$ ,  $w_{j} = b_{j} / (y^{T}y)y$  and  $w_{j}^{T}w_{j} = b_{j}^{2} / (y^{T}y)$ . Therefore, summing over all rows of  $W$  (and keeping in mind that once  $x = \alpha y$ , the same characterization holds for every row) we get  $\| W\| _F^2 = \frac{b^Tb}{y^Ty}$ .
In the general case, let us compute:
$$
\begin{array}{l} w _ {j} ^ {T} w _ {j} = \left\langle \frac {b _ {j} x ^ {T} x - a _ {j} x ^ {T} y}{x ^ {T} x y ^ {T} y - (x ^ {T} y) ^ {2}} y + \frac {a _ {j} y ^ {T} y - b _ {j} x ^ {T} y}{x ^ {T} x y ^ {T} y - (x ^ {T} y) ^ {2}} x, \frac {b _ {j} x ^ {T} x - a _ {j} x ^ {T} y}{x ^ {T} x y ^ {T} y - (x ^ {T} y) ^ {2}} y + \frac {a _ {j} y ^ {T} y - b _ {j} x ^ {T} y}{x ^ {T} x y ^ {T} y - (x ^ {T}) ^ {2}} x \right\rangle \\ = \frac {\left(b _ {j} x ^ {T} x - a _ {j} x ^ {T} y\right) ^ {2} y ^ {T} y + 2 \left(b _ {j} x ^ {T} x - a _ {j} x ^ {T} y\right) \left(a _ {j} y ^ {T} y - b _ {j} x ^ {T} y\right) x ^ {T} y + \left(a _ {j} y ^ {T} y - b _ {j} x ^ {T} y\right) ^ {2} x ^ {T} x}{\left(x ^ {T} x y ^ {T} y - \left(x ^ {T} y\right) ^ {2}\right) ^ {2}} \\ = \frac {b _ {j} ^ {2} \left(x ^ {T} x\right) ^ {2} y ^ {T} y - 2 b _ {j} a _ {j} x ^ {T} x y ^ {T} y x ^ {T} y + a _ {j} ^ {2} \left(x ^ {T} y\right) ^ {2} y ^ {T} y + 2 b _ {j} a _ {j} x ^ {T} x y ^ {T} y x ^ {T} y - 2 b _ {j} ^ {2} x ^ {T} x \left(x ^ {T} y\right) ^ {2}}{\left(x ^ {T} x y ^ {T} y - \left(x ^ {T} y\right) ^ {2}\right) ^ {2}} \\ + \frac {- 2 a _ {j} ^ {2} y ^ {T} y (x ^ {T} y) ^ {2} + 2 b _ {j} a _ {j} (x ^ {T} y) ^ {3} + a _ {j} ^ {2} (y ^ {T} y) ^ {2} x ^ {T} x - 2 b _ {j} a _ {j} x ^ {T} x y ^ {T} y x ^ {T} y + b _ {j} ^ {2} (x ^ {T} y) ^ {2} x ^ {T} x}{(x ^ {T} x y ^ {T} y - (x ^ {T} y) ^ {2}) ^ {2}} \\ = \frac {b _ {j} ^ {2} (x ^ {T} x) ^ {2} y ^ {T} y - b _ {j} ^ {2} x ^ {T} x (x ^ {T} y) ^ {2} + a _ {j} ^ {2} (y ^ {T} y) ^ {2} x ^ {T} x - a _ {j} ^ {2} y ^ {T} y (x ^ {T} y) ^ {2} - 2 a _ {j} b _ {j} x ^ {T} x y ^ {T} y + 2 a _ {j} b _ {j} (x ^ {T} y) ^ {3}}{(x ^ {T} x y ^ {T} y - (x ^ {T} y) ^ {2}) ^ {2}} \\ = \frac {\left(b _ {j} ^ {2} x ^ {T} x + a _ {j} ^ {2} y ^ {T} y - 2 a _ {j} b _ {j} x ^ {T} y\right) \left(x ^ {T} x y ^ {T} y - \left(x ^ {T} y\right) ^ {2}\right)}{\left(x ^ {T} x y ^ {T} y - \left(x ^ {T} y\right) ^ {2}\right) ^ {2}} \\ = \frac {b _ {j} ^ {2} x ^ {T} x - 2 a _ {j} b _ {j} x ^ {T} y + a _ {j} ^ {2} y ^ {T} y}{x ^ {T} x y ^ {T} y - (x ^ {T} y) ^ {2}}. \\ \end{array}
$$
Summing up over rows of  $W$ , we get the desired expression:
$$
\| W \| _ {F} ^ {2} = \frac {b ^ {T} b x ^ {T} x - 2 a ^ {T} b x ^ {T} y + a ^ {T} a y ^ {T} y}{x ^ {T} x y ^ {T} y - (x ^ {T} y) ^ {2}}.
$$
The rest of the statement of the lemma clearly follows from the formula on optimal rows of  $W$ .
Now, as promised, we will optimize over the output matrix  $A$  given that we know what the partial optimal value for each  $A$  is.
$$
\min  _ {A} b ^ {T} b x ^ {T} x - 2 a ^ {T} b x ^ {T} y + a ^ {T} a y ^ {T} y, \quad s. t. \mathcal {S} (A) = \left\{s _ {1}, s _ {2} \right\} \tag {5}
$$
$$
i s \frac {1}{2} (s _ {1} ^ {2} + s _ {2} ^ {2}) (\tilde {s} _ {1} ^ {2} - \tilde {s} _ {2} ^ {2}) + \frac {1}{2} (s _ {1} ^ {2} - s _ {2} ^ {2}) (\tilde {s} _ {1} ^ {2} - \tilde {s} _ {2} ^ {2}), w h e r e \{\tilde {s} _ {1}, \tilde {s} _ {2} \} = \mathcal {S} (X).
$$
Proof. First note that since  $X$  is assumed to be rank 2,  $x \neq \alpha y$  and so the expression in the objective is exactly the numerator of the optimal value of the minimization problem over  $W$  as stated in Lemma 6 (the denominator is constant within the scope of this lemma). Now, using the well-known formulas for the sum and product of squared singular values, we can easily rephrase our constraint in the following way:
$$
a ^ {T} a + b ^ {T} b = s _ {1} ^ {2} + s _ {2} ^ {2} \quad \wedge
$$
$$
a ^ {T} a b ^ {T} b - (a ^ {T} b) ^ {2} = s _ {1} ^ {2} s _ {2} ^ {2}.
$$
First, noticing that both the constraints as well as the objective only depend on  $a^T a, b^T b, a^T b$ , we can reduce the problem to one, where we treat these quantities as free variables that we are optimizing for. The only thing we need to remember will be implicit constraints that  $a^T a, b^T b \geq 0$  and  $(a^T b)^2 \leq a^T ab^T b$ , which is necessary from Cauchy-Schwarz inequality. We will solve this optimization problem using, again, KKT conditions. This time, we will use a stronger rule for deciding on the necessity of KKT conditions – the linear independence constraint qualification (LICQ) rule. According to this rule, any optimal solution has to fulfill the KKT conditions if the gradients of the equality constraints (in this case only two) are linearly independent. In our case, if we order the variables as  $(a^T a, b^T b, a^T b)$  and writing the gradients as:  $(1, 1, 0)$  and  $(b^T b, a^T a, -2a^T b)$  we see that they can only be dependent if  $a^T b = 0$  and  $a^T a = b^T b$ . This is only a feasible solution if  $s_1 = s_2$ . We will return to this case later, showing that it achieves the same objective value as the one coming from KKT conditions.
Now, let us write down the Lagrange function to this problem:
$$
\begin{array}{l} \mathcal {L} (a ^ {T} a, b ^ {T} b, a ^ {T} b, u, v) = a ^ {T} a y ^ {T} y - 2 a ^ {T} b x ^ {T} y + b ^ {T} b x ^ {T} x + u (a ^ {T} a + b ^ {T} b - s _ {1} ^ {2} - s _ {2} ^ {2}) \\ + v (a ^ {T} a b ^ {T} b - (a ^ {T} b) ^ {2} - s _ {1} ^ {2} s _ {2} ^ {2}). \\ \end{array}
$$
Computing the partial derivatives we get:
$$
\frac {\partial \mathcal {L}}{\partial a ^ {T} a} = y ^ {T} y + u + v b ^ {T} b \stackrel {!} {=} 0,
$$
$$
\frac {\partial \mathcal {L}}{\partial b ^ {T} b} = x ^ {T} x + u + v a ^ {T} a \stackrel {!} {=} 0,
$$
$$
\frac {\partial \mathcal {L}}{\partial a ^ {T} b} = - 2 x ^ {T} y - 2 v a ^ {T} b \stackrel {!} {=} 0.
$$
First consider what would happen if  $v = 0$ . In this case,  $x^{T}y = 0$  from the third equation and  $x^{T}x = y^{T}y$  from the first two. Then, however, the objective function is constant on the feasible set and equals  $1 / 2(s_1^2 + s_2^2)(\tilde{s}_1^2 + \tilde{s}_2^2)$ , which agrees with the statement of the Lemma.
Next, assume  $v \neq 0$ . Then we can express all the variables as follows:
$$
a ^ {T} a = \frac {- x ^ {T} x - u}{v},
$$
$$
b ^ {T} b = \frac {- y ^ {T} y - u}{v},
$$
$$
a ^ {T} b = - \frac {x ^ {T} y}{v}.
$$
Plugging this back into the constraints, we can solve for  $v, u$ . We get:
$$
v = - \frac {x ^ {T} x + y ^ {T} y + 2 u}{s _ {1} ^ {2} + s _ {2} ^ {2}}.
$$
Plugging this into the second constraint together with the expressions for  $a^T a, b^T b$  and then simplifying we get:
$$
\begin{array}{l} \frac {(y ^ {T} y + u) (x ^ {T} x + u) (s _ {1} ^ {2} + s _ {2} ^ {2}) ^ {2}}{(x ^ {T} x + y ^ {T} y + 2 u) ^ {2}} - \frac {(x ^ {T} y) ^ {2} (s _ {1} ^ {2} + s _ {2} ^ {2}) ^ {2}}{(x ^ {T} x + y ^ {T} y + 2 u) ^ {2}} = s _ {1} ^ {2} s _ {2} ^ {2} \\ y ^ {T} y x ^ {T} x - (x ^ {T} y) ^ {2} + (y ^ {T} y + x ^ {T} x) u + u ^ {2} = \left(\frac {s _ {1} s _ {2}}{s _ {1} ^ {2} + s _ {2} ^ {2}}\right) ^ {2} (x ^ {T} x + y ^ {T} y) ^ {2} \\ + \left(\frac {2 s _ {1} s _ {2}}{s _ {1} ^ {2} + s _ {2} ^ {2}}\right) ^ {2} (x ^ {T} x + y ^ {T} y) u + \left(\frac {2 s _ {1} s _ {2}}{s _ {1} ^ {2} + s _ {2} ^ {2}}\right) ^ {2} u ^ {2} \\ \end{array}
$$
$$
\frac {\left(s _ {1} ^ {2} - s _ {2} ^ {2}\right) ^ {2}}{\left(s _ {1} ^ {2} + s _ {2} ^ {2}\right) ^ {2}} u ^ {2} + \frac {\left(s _ {1} ^ {2} - s _ {2} ^ {2}\right) ^ {2}}{\left(s _ {1} ^ {2} + s _ {2} ^ {2}\right) ^ {2}} \left(x ^ {T} x + y ^ {T} y\right) u + x ^ {T} x y ^ {T} y - \left(x ^ {T} y\right) ^ {2} - \left(\frac {s _ {1} s _ {2}}{s _ {1} ^ {2} + s _ {2} ^ {2}}\right) ^ {2} \left(x ^ {T} x + y ^ {T} y\right) ^ {2} = 0
$$
$$
u ^ {2} + \left(x ^ {T} x + y ^ {T} y\right) u + \frac {\left(s _ {1} ^ {2} + s _ {2} ^ {2}\right) ^ {2}}{\left(s _ {1} ^ {2} - s _ {2} ^ {2}\right) ^ {2}} \left(x ^ {T} x y ^ {T} y - \left(x ^ {T} y\right) ^ {2}\right) - \left(\frac {s _ {1} s _ {2}}{s _ {1} ^ {2} - s _ {2} ^ {2}}\right) ^ {2} \left(x ^ {T} x + y ^ {T} y\right) ^ {2} = 0. \tag {13}
$$
Recall that the case  $s_1 = s_2$  is treated separately, so we do not divide by zero. Now we can rewrite this expression w.r.t.  $\{\tilde{s}_1,\tilde{s}_2\}$ :
$$
u ^ {2} + (\tilde {s} _ {1} ^ {2} + \tilde {s} _ {2} ^ {2}) u + \frac {(s _ {1} ^ {2} + s _ {2} ^ {2}) ^ {2}}{(s _ {1} ^ {2} - s _ {2} ^ {2}) ^ {2}} \tilde {s} _ {1} ^ {2} \tilde {s} _ {2} ^ {2} - \frac {s _ {1} ^ {2} s _ {2} ^ {2}}{(s _ {1} ^ {2} - s _ {2} ^ {2}) ^ {2}} (\tilde {s} _ {1} ^ {2} + \tilde {s} _ {2} ^ {2}) ^ {2} = 0
$$
We can now simply solve for this to obtain the roots:
$$
u _ {\pm} = \frac {- (\tilde {s} _ {1} ^ {2} + \tilde {s} _ {2} ^ {2}) \pm \sqrt {(\tilde {s} _ {1} ^ {2} + \tilde {s} _ {2} ^ {2}) ^ {2} + \frac {4 s _ {1} ^ {2} s _ {2} ^ {2}}{(s _ {1} ^ {2} - s _ {2} ^ {2}) ^ {2}} (\tilde {s} _ {1} ^ {2} + \tilde {s} _ {2} ^ {2}) ^ {2} - 4 \frac {(s _ {1} ^ {2} + s _ {2} ^ {2}) ^ {2}}{(s _ {1} ^ {2} - s _ {2} ^ {2}) ^ {2}} \tilde {s} _ {1} ^ {2} \tilde {s} _ {2} ^ {2}}}{2},
$$
which after some simplifications yields:
$$
u _ {\pm} = \frac {- (\tilde {s} _ {1} ^ {2} + \tilde {s} _ {2} ^ {2}) \pm \frac {s _ {1} ^ {2} + s _ {2} ^ {2}}{s _ {1} ^ {2} - s _ {2} ^ {2}} (\tilde {s} _ {1} ^ {2} - \tilde {s} _ {2} ^ {2})}{2}.
$$
The formula for  $v$  can now be obtained:
$$
v _ {\pm} = \mp \frac {\tilde {s} _ {1} ^ {2} - \tilde {s} _ {2} ^ {2}}{s _ {1} ^ {2} - s _ {2} ^ {2}}
$$
Having this, we can now compute the objective value using these quantities. We write it down for  $u_{+}, v_{+}$ :
$$
\begin{array}{l} b ^ {T} b x ^ {T} x + a ^ {T} a y ^ {T} y - 2 a ^ {T} b x ^ {T} y = \frac {- (x ^ {T} x + y ^ {T} y) u - 2 (x ^ {T} x y ^ {T} y - (x ^ {T} y) ^ {2})}{v} \\ = \frac {- (\tilde {s} _ {1} ^ {2} + \tilde {s} _ {2} ^ {2}) u - 2 \tilde {s} _ {1} ^ {2} \tilde {s} _ {2} ^ {2}}{v} = \frac {(\tilde {s} _ {1} ^ {2} + \tilde {s} _ {2} ^ {2}) ^ {2} - \frac {s _ {1} ^ {2} + s _ {2} ^ {2}}{s _ {1} ^ {2} - s _ {2} ^ {2}} (\tilde {s} _ {1} ^ {2} - \tilde {s} _ {2} ^ {2}) (\tilde {s} _ {1} ^ {2} + \tilde {s} _ {2} ^ {2}) - 4 \tilde {s} _ {1} ^ {2} \tilde {s} _ {2} ^ {2}}{2 v} \\ = \frac {- \frac {(\tilde {s} _ {1} ^ {2} + \tilde {s} _ {2} ^ {2}) ^ {2}}{\tilde {s} _ {1} ^ {2} - \tilde {s} _ {2} ^ {2}} (s _ {1} ^ {2} - s _ {2} ^ {2}) + (s _ {1} ^ {2} + s _ {2} ^ {2}) (\tilde {s} _ {1} ^ {2} + \tilde {s} _ {2} ^ {2}) + 4 \tilde {s} _ {1} ^ {2} \tilde {s} _ {2} ^ {2} \frac {s _ {1} ^ {2} - s _ {2} ^ {2}}{\tilde {s} _ {1} ^ {2} - \tilde {s} _ {2} ^ {2}}}{2} = \frac {(s _ {1} ^ {2} + s _ {2} ^ {2}) (\tilde {s} _ {1} ^ {2} + \tilde {s} _ {2} ^ {2}) - (s _ {1} ^ {2} - s _ {2} ^ {2}) (\tilde {s} _ {1} ^ {2} - \tilde {s} _ {2} ^ {2})}{2}, \\ \end{array}
$$
which is the expression we wanted to prove. Had we done the same computation, but for  $u_{-}, v_{-}$ , we would get
$$
\frac {(s _ {1} ^ {2} + s _ {2} ^ {2}) (\tilde {s} _ {1} ^ {2} + \tilde {s} _ {2} ^ {2}) + (s _ {1} ^ {2} - s _ {2} ^ {2}) (\tilde {s} _ {1} ^ {2} - \tilde {s} _ {2} ^ {2})}{2},
$$
which is at least as big as the previous expression, so we can drop this one. However, we still need to check, whether this solution of the KKT system is indeed feasible, because we imposed implicit constraints into the problem. Thus, we need to check:  $a^T a, b^T b \geq 0$  and  $a^T ab^T b \geq (a^T b)^2$ . The first two are analogous. We have:  $a^T a = -\frac{x^Tx + u}{v}$ , but since the optimal  $v_+$  is strictly negative, we can only analyze the sign of  $2x^Tx + 2u$  which is:
$$
2 x ^ {T} x + 2 u = x ^ {T} x - y ^ {T} y + \frac {s _ {1} ^ {2} + s _ {2} ^ {2}}{s _ {1} ^ {2} - s _ {2} ^ {2}} (\tilde {s} _ {1} ^ {2} - \tilde {s} _ {2} ^ {2}) \geq \tilde {s} _ {1} ^ {2} - \tilde {s} _ {2} ^ {2} - | x ^ {T} x - y ^ {T} y | \stackrel {?} {\geq} 0.
$$
However, using the same formulas as the constraints in this optimization problem on  $\tilde{s}_1,\tilde{s}_2$  instead, one can, using very simple computations, check that this indeed holds. For the next condition, consider that  $a^T ab^T b - (a^T b)^2$  must have the same sign as  $(x^{T}x + u)(y^{T}y + u) - (x^{T}y)^{2} = x^{T}xy^{T}y - (x^{T}y)^{2} + u^{2} + u(x^{T}x + y^{T}y)$ . This resembles the quadratic function in (13).  $u_{+}$  is a root of that quadratic function. Both this new quadratic function, as well as the one in (13) have the same coefficients in front of  $u^2$  and  $u$ . Therefore, to test for the sign of the expression above, it only suffices to check, whether this alternative quadratic function has at least as big absolute constant:
$$
\begin{array}{l} x ^ {T} x y ^ {T} y - (x ^ {T} y) ^ {2} - \frac {(s _ {1} ^ {2} + s _ {2} ^ {2}) ^ {2}}{(s _ {1} ^ {2} - s _ {2} ^ {2}) ^ {2}} (x ^ {T} x y ^ {T} y - (x ^ {T} y) ^ {2}) + \left(\frac {s _ {1} s _ {2}}{s _ {1} ^ {2} - s _ {2} ^ {2}}\right) ^ {2} (x ^ {T} x + y ^ {T} y) ^ {2} \geq 0 \iff \\ - \frac {s _ {1} ^ {2} s _ {2} ^ {2}}{(s _ {1} ^ {2} - s _ {2} ^ {2}) ^ {2}} 4 \tilde {s} _ {1} ^ {2} \tilde {s} _ {2} ^ {2} + \frac {s _ {1} ^ {2} s _ {2} ^ {2}}{(s _ {1} ^ {2} - s _ {2} ^ {2}) ^ {2}} (\tilde {s} _ {1} ^ {2} + \tilde {s} _ {2} ^ {2}) ^ {2} \geq 0 \iff \frac {s _ {1} ^ {2} s _ {2} ^ {2}}{(s _ {1} ^ {2} - s _ {2} ^ {2}) ^ {2}} (\tilde {s} _ {1} ^ {2} - \tilde {s} _ {2} ^ {2}) ^ {2} \geq 0. \\ \end{array}
$$
The last inequality trivially holds. Therefore, the solution of KKT conditions is feasible.