SlowGuess/NeurIPS_2023_OCR huggingface dataset:
When  $\mu = \tau$ , because  $t(x_{\mathrm{lb}}; \mu) = 0$  and  $x_{\mathrm{ub}} > \sqrt{\mu(a^2 + 1)} > x_{\mathrm{lb}}$ ,  $t(x; \mu)$  is increasing function between  $[x_{\mathrm{lb}}, x_{\mathrm{ub}}]$  then  $t(\sqrt{\mu(a^2 + 1)}; \mu) > t(x_{\mathrm{lb}}; \mu) = 0$ . Moreover,  $t(\sqrt{\mu(a^2 + 1)}, \mu)$ ,  $x_{\mathrm{lb}}$  and  $x_{\mu}^{**}$  are continuous functions w.r.t  $\mu$ ,  $\exists \delta > 0$  which is really small, such that  $\mu = \tau - \delta$  and  $t(\sqrt{\mu(a^2 + 1)}, \mu) > 0$ ,  $t(x_{\mathrm{lb}}, \mu) < 0$  (by Theorem 6(iv)) and  $x_{\mu}^{**} > x_{\mathrm{lb}}$ , hence  $\left.\frac{dx}{d\mu}\right|_{x = x_{\mu}^{**}} < 0$ . It implies when  $\mu$  decreases, then  $x_{\mu}^{**}$  increases. This relation holds until  $x_{\mu}^{**} = \sqrt{\mu(a^2 + 1)}$
$$
\begin{array}{l} t (x _ {\mu} ^ {* *}, \mu) = t (\sqrt {\mu (a ^ {2} + 1)}, \mu) = 0 \\ \Rightarrow \mu = \left(\frac {a (\sqrt {a ^ {2} + 1} - a)}{2 (a ^ {2} + 1)}\right) ^ {2} \\ \end{array}
$$
and  $\sqrt{\mu(a^2 + 1)} = \frac{a(\sqrt{a^2 + 1} - a)}{2\sqrt{a^2 + 1}}$ . Note that when  $\mu < \left(\frac{a(\sqrt{a^2 + 1} - a)}{2(a^2 + 1)}\right)^2$ ,  $t(\sqrt{\mu(a^2 + 1)}, \mu) < 0$ , it implies that  $x_{\mu}^{**} > \sqrt{\mu(a^2 + 1)}$  and  $\left.\frac{dx}{d\mu}\right|_{x = x_{\mu}^{**}} > 0$ , thus decreasing  $\mu$  leads to decreasing  $x_{\mu}^{**}$ . We can conclude
$$
\max  _ {\mu \leq \tau} x _ {\mu} ^ {* *} \leq \frac {a (\sqrt {a ^ {2} + 1} - a)}{2 \sqrt {a ^ {2} + 1}}
$$
Note that  $\forall \mu$  s.t.  $\left(\frac{a(\sqrt{a^2 + 1} - a)}{2(a^2 + 1)}\right)^2 < \mu < \tau, x_\mu^{**} < \left(\frac{a(\sqrt{a^2 + 1} - a)}{2(a^2 + 1)}\right)^2$ , so
$$
t \left(\left(\frac {a (\sqrt {a ^ {2} + 1} - a)}{2 (a ^ {2} + 1)}\right) ^ {2}, \mu\right) \geq 0.
$$
Note that when  $\mu >\frac{a^2}{a^2 + 1}$ , i.e.  $(x_{\mu}^{*})^{2}\geq \mu (a^{2} + 1)$  then
$$
\left.\frac{dx}{d\mu}\right|_{x = x_{\mu}^{*}} > 0
$$
It implies that when  $\mu$  decreases,  $x_{\mu}^{*}$  also decreases. It holds true until  $x_{\mu}^{*} = \sqrt{\mu(a^{2} + 1)}$ . The same analysis can be applied to  $x_{\mu}^{*}$  like above, we can conclude that
$$
\min _ {\tau} x _ {\mu} ^ {*} = \frac {a (\sqrt {a ^ {2} + 1} + a)}{2 \sqrt {a ^ {2} + 1}}
$$
$$
\max  _ {\mu \leq \tau} x _ {\mu} ^ {* *} \leq \frac {a (\sqrt {a ^ {2} + 1} - a)}{2 \sqrt {a ^ {2} + 1}} <   \frac {a (\sqrt {a ^ {2} + 1} + a)}{2 \sqrt {a ^ {2} + 1}} \leq \min  _ {\mu > 0} x _ {\mu} ^ {*}
$$
E.4 Proof of Theorem 7,8 and 9
Proof. The proof is similar to the proof of Theorem 5 and Theorem 6.  $\square$
E.5 Proof of Lemma 1
$$
\nabla^ {2} g _ {\mu} (x, y) = \left( \begin{array}{c c} \mu + y ^ {2} & 2 x y \\ 2 x y & \mu (a ^ {2} + 1) + x ^ {2} \end{array} \right)
$$
Let  $\lambda_1(\nabla^2 g_\mu (x,y)),\lambda_2(\nabla^2 g_\mu (x,y))$  be the eigenvalue of matrix  $\nabla^2 g_\mu (x,y)$ , then
$$
\begin{array}{l} \lambda_ {1} \left(\nabla^ {2} g _ {\mu} (x, y)\right) + \lambda_ {2} \left(\nabla^ {2} g _ {\mu} (x, y)\right) \\ = \operatorname {T r} \left(\nabla^ {2} g _ {\mu} (x, y)\right) = \mu + y ^ {2} + \mu \left(a ^ {2} + 1\right) + x ^ {2} > 0 \\ \end{array}
$$
Now we calculate the product of eigenvalue
$$
\begin{array}{l} \lambda_ {1} \left(\nabla^ {2} g _ {\mu} (x, y)\right) \cdot \lambda_ {2} \left(\nabla^ {2} g _ {\mu} (W)\right) \\ = \det  (\nabla^ {2} g _ {\mu} (W)) \\ = (\mu + y ^ {2}) \left(\mu \left(a ^ {2} + 1\right) + x ^ {2}\right) - 4 x ^ {2} y ^ {2} \\ = \frac {\mu a}{x} \frac {\mu a}{y} - 4 x ^ {2} y ^ {2} > 0 \\ \Leftrightarrow \left(\frac {a \mu}{2}\right) ^ {2 / 3} > x y \\ \Leftrightarrow \left(\frac {a \mu}{2}\right) ^ {2 / 3} > \frac {a \mu}{y ^ {2} + \mu} y \\ \Leftrightarrow y + \frac {\mu}{y} > (4 a \mu) ^ {1 / 3} \\ \end{array}
$$
Note that for  $(x_{\mu}^{*},y_{\mu}^{*}),(x_{\mu}^{***},y_{\mu}^{***})$ , they satisfy (11a) and (11b), this fact is used in third equality and second “ $\Leftrightarrow$ ”. By (32b), we know  $\lambda_1(\nabla^2 g_\mu (x,y))\cdot \lambda_2(\nabla^2 g_\mu (x,y)) > 0$  for  $(x_{\mu}^{*},y_{\mu}^{*}),(x_{\mu}^{***},y_{\mu}^{***})$ , and  $\lambda_1(\nabla^2 g_\mu (x,y))\cdot \lambda_2(\nabla^2 g_\mu (x,y)) < 0$  for  $(x_{\mu}^{**},y_{\mu}^{**})$ , then
$$
\lambda_ {1} \left(\nabla^ {2} g _ {\mu} (x, y)\right) > 0, \lambda_ {2} \left(\nabla^ {2} g _ {\mu} (x, y)\right) > 0 \quad \text {f o r} \left(x _ {\mu} ^ {*}, y _ {\mu} ^ {*}\right), \left(x _ {\mu} ^ {* * *}, y _ {\mu} ^ {* * *}\right)
$$
$$
\lambda_ {1} (\nabla^ {2} g _ {\mu} (x, y)) <   0 \text {o r} \lambda_ {2} (\nabla^ {2} g _ {\mu} (x, y)) <   0 \qquad \text {f o r} (x _ {\mu} ^ {* *}, y _ {\mu} ^ {* *})
$$
$$
\nabla g _ {\mu} (x, y) = 0
$$
Then  $(x_{\mu}^{*},y_{\mu}^{*}),(x_{\mu}^{**},y_{\mu}^{**})$  are locally minima,  $(x_{\mu}^{**},y_{\mu}^{**})$  is saddle point for  $g_{\mu}(W)$ .
E.6 Proof of Lemma 2
Figure 8: Stationary points when  $\mu < \tau$
Proof. Let us define the functions as below
$$
\left\{ \begin{array}{l l} y _ {\mu 1} (x) = \sqrt {\mu (\frac {a - x}{x})} & 0 <   x \leq a \\ y _ {\mu 2} (x) = \frac {\mu a}{\mu \left(a ^ {2} + 1\right) + x ^ {2}} & 0 <   x \leq a \end{array} \right. \tag {37b}
$$
$$
\left\{ \begin{array}{l l} x _ {\mu 1} (y) = \frac {\mu a}{y ^ {2} + \mu} & 0 <   y <   \frac {a}{a ^ {2} + 1} \\ x _ {\mu 2} (y) = \sqrt {\mu \left(\frac {a}{y} - \left(a ^ {2} + 1\right)\right)} & 0 <   y <   \frac {a}{a ^ {2} + 1} \end{array} \right. \tag {38b}
$$
with simple calculations,
$$
y _ {\mu 1} \geq y _ {\mu 2} \Leftrightarrow t (x; \mu) \geq 0 \Leftrightarrow x \in (0, x _ {\mu} ^ {* * *} ] \cup [ x _ {\mu} ^ {* *}, x _ {\mu} ^ {*} ]
$$
$$
x _ {\mu 1} \geq x _ {\mu 2} \Leftrightarrow r (y; \mu) \leq 0 \Leftrightarrow y \in [ y _ {\mu} ^ {*}, y _ {\mu} ^ {* *} ] \cup [ y _ {\mu} ^ {* * *}, \frac {a}{a ^ {2} + 1})
$$
Here we divide  $B_{\mu}$  into three parts,  $C_{\mu 1}, C_{\mu 2}, C_{\mu 3}$
$$
C _ {\mu 1} = \{(x, y) | x _ {\mu} ^ {* *} <   x \leq x _ {\mu} ^ {*}, y _ {\mu 1} <   y <   y _ {\mu} ^ {* *} \} \cup \{(x, y) | x _ {\mu} ^ {*} <   x \leq a, y _ {\mu 2} <   y <   y _ {\mu} ^ {* *} \} \tag {39}
$$
$$
C _ {\mu 2} = \{(x, y) | x _ {\mu} ^ {* *} <   x \leq x _ {\mu} ^ {*}, 0 \leq y <   y _ {\mu 2} \} \cup \{(x, y) | x _ {\mu} ^ {*} <   x \leq a, 0 \leq y <   y _ {\mu 1} \} \tag {40}
$$
$$
C _ {\mu 3} = \left\{(x, y) \mid x _ {\mu} ^ {* *} <   x \leq x _ {\mu} ^ {*}, y _ {\mu 2} \leq y \leq y _ {\mu 1} \right\} \cup \left\{(x, y) \mid x _ {\mu} ^ {*} <   x \leq a, y _ {\mu 1} \leq y \leq y _ {\mu 2} \right\} \tag {41}
$$
$$
\forall (x, y) \in C _ {\mu 1} \Rightarrow \frac {\partial g _ {\mu} (x , y)}{\partial x} > 0, \frac {\partial g _ {\mu} (x , y)}{\partial y} > 0
$$
$$
\forall (x, y) \in C _ {\mu 2} \Rightarrow \frac {\partial g _ {\mu} (x , y)}{\partial x} <   0, \frac {\partial g _ {\mu} (x , y)}{\partial y} <   0
$$
The gradient flow follows
$$
\left( \begin{array}{c} x ^ {\prime} (t) \\ y ^ {\prime} (t) \end{array} \right) = - \left(\frac {\frac {\partial g _ {\mu} (x (t) , y (t))}{\partial x}}{\frac {\partial g _ {\mu} (x (t) , y (t))}{\partial y}}\right) = - \nabla g _ {\mu} (x (t), y (t))
$$
$$
\forall (x, y) \in C _ {\mu 1} \Rightarrow \binom {x ^ {\prime} (t)} {y ^ {\prime} (t)} <   0, \| \nabla g _ {\mu} \| > 0 \tag {42}
$$
$$
\forall (x, y) \in C _ {\mu 2} \Rightarrow \binom {x ^ {\prime} (t)} {y ^ {\prime} (t)} > 0, \| \nabla g _ {\mu} \| > 0 \tag {43}
$$
Note that  $\| \nabla g_{\mu}\|$  is not diminishing and bounded away from 0. Let us consider the  $(x(0),y(0))\in C_{\mu 1}$ , since  $\nabla g_{\mu}(x,y)\neq 0, - \nabla g_{\mu}(x,y) < 0$  in (42) and boundness of  $C_{\mu 1}$ , it implies there exists a finite  $t_0 > 0$  such that
$$
\left(x \left(t _ {0}\right), y \left(t _ {0}\right)\right) \in \partial C _ {\mu 1}, \left(x (t), y (t)\right) \in C _ {\mu 1} \text {f o r} 0 \leq t <   t _ {0}
$$
where  $\partial C_{\mu 1}$  is defined as
$$
\partial C _ {\mu 1} = \left\{(x, y) \mid x _ {\mu} ^ {* *} <   x \leq x _ {\mu} ^ {*}, y = y _ {\mu 1} \right\} \cup \left\{(x, y) \mid x _ {\mu} ^ {*} <   x \leq a, y = y _ {\mu 2} \right\} \subseteq C _ {\mu 3}
$$
For the same reason, if  $(x(0),y(0))\in C_{\mu 2}$  , there exists a finite time  $t_1 > 0$
$$
\left(x \left(t _ {0}\right), y \left(t _ {0}\right)\right) \in \partial C _ {\mu 2}, \left(x (t), y (t)\right) \in C _ {\mu 2} \text {f o r} 0 \leq t <   t _ {1}
$$
where  $\partial C_{\mu 2}$  is defined as
$$
\partial C _ {\mu 2} = \{(x, y) | x _ {\mu} ^ {* *} <   x \leq x _ {\mu} ^ {*}, y = y _ {\mu 2} \} \cup \{(x, y) | x _ {\mu} ^ {*} <   x \leq a, y = y _ {\mu 1} \} \subseteq C _ {\mu 3}
$$
then by lemma 7,  $\lim_{t\to \infty}(x(t),y(t)) = (x_{\mu}^{*},y_{\mu}^{*})$
E.7 Proof of Lemma 3
Proof. This is just a result of the Theorem 5.
E.8 Proof of Lemma 5
$$
\nabla^ {2} g _ {\mu} (W) = \left( \begin{array}{c c} \mu + y ^ {2} & 2 x y \\ 2 x y & \mu (a ^ {2} + 1) + x ^ {2} \end{array} \right) = \left( \begin{array}{c c} \mu & 0 \\ 0 & \mu (a ^ {2} + 1) \end{array} \right) + \left( \begin{array}{c c} y ^ {2} & 2 x y \\ 2 x y & x ^ {2} \end{array} \right)
$$
Let  $\| \cdot \|_{\mathrm{op}}$  is the spectral norm, and it satisfies triangle inequality
$$
\begin{array}{l} \left\| \nabla^ {2} g _ {\mu} (W) \right\| _ {\mathrm {o p}} \leq \left\| \left( \begin{array}{c c} \mu & 0 \\ 0 & \mu (a ^ {2} + 1) \end{array} \right) \right\| _ {\mathrm {o p}} + \left\| \left( \begin{array}{c c} y ^ {2} & 2 x y \\ 2 x y & x ^ {2} \end{array} \right) \right\| _ {\mathrm {o p}} \\ = \mu (a ^ {2} + 1) + \left\| \left( \begin{array}{c c} y ^ {2} & 2 x y \\ 2 x y & x ^ {2} \end{array} \right) \right\| _ {\mathrm {o p}} \\ \end{array}
$$
The spectral norm of the second term in area A is bounded by
$$
\max  _ {(x, y) \in A} \frac {\left(x ^ {2} + y ^ {2}\right) + \sqrt {\left(x ^ {2} + y ^ {2}\right) ^ {2} + 1 2 x ^ {2} y ^ {2}}}{2} \leq \frac {2 a ^ {2} + \sqrt {4 a ^ {4} + 1 2 a ^ {4}}}{2} = 3 a ^ {2}
$$
We use  $x^{2}\leq a^{2},y^{2}\leq a^{2}$  in the inequality. Therefore,
$$
\left\| \nabla^ {2} g _ {\mu} (W) \right\| _ {\mathrm {o p}} \leq 3 a ^ {2} + \mu \left(a ^ {2} + 1\right)
$$
Also, according to [5, 33], for any  $f$ , if  $\nabla^2 f$  exists, then  $f$  is  $L$  smooth if and only if  $|\nabla^2 f|_{\mathrm{op}} \leq L$ . With this, we conclude the proof.
E.9 Proof of Lemma 7
Proof. First we prove  $\forall t \geq 0, (x(t), y(t)) \in C_{\mu 3}$ , because if  $(x(t), y(t)) \notin C_{\mu 3}$ , then there exists a finite  $t$  such that
$$
(x (t), y (t)) \in \partial C _ {\mu 3}
$$
where  $\partial C_{\mu 3}$  is the boundary of  $C_{\mu 3}$ , defined as
$$
\partial C _ {\mu 3} = \{(x, y) | y = y _ {\mu 1} (x) \text {o r} y = y _ {\mu 2} (x), x _ {\mu} ^ {* *} <   x \leq a \}
$$
W.L.O.G, let us assume  $(x(0),y(0))\in \partial C_{\mu 3}$  and  $(x(0),y(0))\neq (x_{\mu}^{*},y_{\mu}^{*})$ . Here are four different cases,
$$
\nabla g _ {\mu} (x (t), y (t)) = \left\{ \begin{array}{l l} \binom {= 0} {> 0} & \text {i f} y (0) = y _ {\mu 1} (x (0)), x _ {\mu} ^ {* *} <   x (0) <   x _ {\mu} ^ {*} \\ \binom {= 0} {<   0} & \text {i f} y (0) = y _ {\mu 1} (x (0)), x _ {\mu} ^ {*} <   x (0) \leq a \\ \binom {<   0} {= 0} & \text {i f} y (0) = y _ {\mu 2} (x (0)), x _ {\mu} ^ {* *} <   x (0) <   x _ {\mu} ^ {*} \\ \binom {> 0} {= 0} & \text {i f} y (0) = y _ {\mu 2} (x (0)), x _ {\mu} ^ {*} <   x (0) \leq a \end{array} \right.
$$
This indicates that  $-\nabla g_{\mu}(x(t),y(t))$  are pointing to the interior of  $C_{\mu 3}$ , then  $(x(t),y(t))$  can not escape  $C_{\mu 3}$ . Here we can focus our attention in  $C_{\mu 3}$ , because  $\forall t\geq 0,(x(t),y(t))\in C_{\mu 3}$ . For Algorithm 1,
$$
\frac {d f (\boldsymbol {z} _ {t})}{d t} = \nabla f (\boldsymbol {z} _ {t}) \dot {\boldsymbol {z}} _ {t} = - \| \nabla f (\boldsymbol {z} _ {t}) \| _ {2} ^ {2}
$$
In our setting,  $\forall (x,y)\in C_{\mu 3}$
$$
\left\{ \begin{array}{l l} \nabla g _ {\mu} (x, y) \neq 0 & (x, y) \neq (x _ {\mu} ^ {*}, y _ {\mu} ^ {*}) \\ \nabla g _ {\mu} (x, y) = 0 & (x, y) = (x _ {\mu} ^ {*}, y _ {\mu} ^ {*}) \end{array} \right.
$$
$$
\frac {d g _ {\mu} (x (t) , y (t))}{d t} = \left\{ \begin{array}{l l} - \| \nabla g _ {\mu} \| _ {2} ^ {2} <   0 & (x, y) \neq (x _ {\mu} ^ {*}, y _ {\mu} ^ {*}) \\ - \| \nabla g _ {\mu} \| _ {2} ^ {2} = 0 & (x, y) = (x _ {\mu} ^ {*}, y _ {\mu} ^ {*}) \end{array} \right.
$$
Plus,  $(x_{\mu}^{*},y_{\mu}^{*})$  is the unique stationary point of  $g_{\mu}(W)$  in  $C_{\mu 3}$ . By lemma 8
$$
g _ {\mu} (x, y) > g _ {\mu} \left(x _ {\mu} ^ {*}, y _ {\mu} ^ {*}\right) \quad (x, y) \neq \left(x _ {\mu} ^ {*}, y _ {\mu} ^ {*}\right)
$$
By Lyapunov asymptotic stability theorem [28], and applying it to gradient flow for  $g_{\mu}(x,y)$  in  $C_{\mu 3}$ , we can conclude  $\lim_{t\to \infty}(x(t),y(t)) = (x_{\mu}^{*},y_{\mu}^{*})$ .
E.10 Proof of Lemma 8
Proof. For any  $(x,y) \in C_{\mu 3}$  in 41, and  $(x,y) \neq (x_{\mu}^{*},y_{\mu}^{*})$ , in Algorithm 7. W.L.O.G, we can assume  $x \in (x_{\mu}^{**},x_{\mu}^{*})$ , the analysis details can also be applied to  $x \in (x_{\mu}^{*},a)$ . It is obvious that  $\tilde{x}_j < \tilde{x}_{j+1}$  and  $\tilde{y}_{j+1} < \tilde{y}_j$ . Also,  $\lim_{j \to \infty} (\tilde{x}_j, \tilde{y}_j) = (x_{\mu}^{*},y_{\mu}^{*})$ . Otherwise either  $\tilde{x}_j \neq x_{\mu}^{*}$  or  $\tilde{y}_j \neq y_{\mu}^{*}$  hold, Algorithm 7 continues until  $\lim_{j \to \infty} (\tilde{x}_j, \tilde{y}_j) = \lim_{j \to \infty} (y_{\mu 2}(\tilde{y}_j), x_{\mu 1}(\tilde{x}_j))$ , i.e.  $(\tilde{x}_j, \tilde{y}_j)$  converges to  $(x_{\mu}^{*},y_{\mu}^{*})$ .
Moreover, note that for any  $j = 0,1,\ldots$
$$
g _ {\mu} \left(\tilde {x} _ {j - 1}, \tilde {y} _ {j - 1}\right) > g _ {\mu} \left(\tilde {x} _ {j - 1}, \tilde {y} _ {j}\right) > g _ {\mu} \left(\tilde {x} _ {j}, \tilde {y} _ {j}\right)
$$
$$
g _ {\mu} \left(\tilde {x} _ {j - 1}, \tilde {y} _ {j - 1}\right) - g _ {\mu} \left(\tilde {x} _ {j - 1}, \tilde {y} _ {j}\right) = \frac {\partial g _ {\mu} \left(\tilde {x} _ {j - 1} , \tilde {y}\right)}{\partial y} \left(\tilde {y} _ {j - 1} - \tilde {y} _ {j}\right) \quad \text {w h e r e} \tilde {y} \in \left(\tilde {y} _ {j}, \tilde {y} _ {j - 1}\right)
$$
$$
\frac {\partial g _ {\mu} (\tilde {x} _ {j - 1} , \tilde {y})}{\partial y} > 0 \Rightarrow g _ {\mu} (\tilde {x} _ {j - 1}, \tilde {y} _ {j - 1}) > g _ {\mu} (\tilde {x} _ {j - 1}, \tilde {y} _ {j})
$$
By the same reason,
$$
g _ {\mu} \left(\tilde {x} _ {j - 1}, \tilde {y} _ {j}\right) > g _ {\mu} \left(\tilde {x} _ {j}, \tilde {y} _ {j}\right)
$$
By Lemma 1,  $(x_{\mu}^{*},y_{\mu}^{*})$  is local minima, and there exists a  $r_\mu >0$  and any  $\{(x,y)\mid \| (x,y) - (x_{\mu}^{*},y_{\mu}^{*})\|_{2}\leq r_{\mu}\} ,g_{\mu}(x,y) > g_{\mu}(x_{\mu}^{*},y_{\mu}^{*})$  Since  $\lim_{j\to \infty}(\tilde{x}_j,\tilde{y}_j) = (x_\mu^*,y_\mu^*)$  , there exists a  $J > 0$  such that  $\forall j > J,\| (\tilde{x}_j,\tilde{y}_j) - (x_\mu^*,y_\mu^*)\|_2\leq r_\mu$  , combining them all
$$
g _ {\mu} (x, y) > g _ {\mu} (\tilde {x} _ {j}, \tilde {y} _ {j}) > g _ {\mu} \left(x _ {\mu} ^ {*}, y _ {\mu} ^ {*}\right)
$$
Algorithm 7: Path goes to  $(x_{\mu}^{*},y_{\mu}^{*})$
Input:  $(x,y)\in C_{\mu 3},x_{\mu 1}(y),y_{\mu 2}(x)$  as (38a),(37b) Output:  $\{(\tilde{x}_j,\tilde{y}_j)\}_{j = 0}^{\infty}$    
1  $(\tilde{x}_0,\tilde{y}_0)\gets (x,y)$    
2 for  $j = 1,2,\ldots$  do   
3  $\tilde{y}_j\gets y_{\mu 2}(\tilde{x}_{j - 1})$    
4  $\tilde{x}_j\gets x_{\mu 1}(\tilde{y}_{j - 1})$    
5 end
E.11 Proof of Lemma 4
Proof. From the proof of Theorem 1, any any scheduling for  $\mu_{k}$  satisfies following will do the job
$$
(2 / a) ^ {2 / 3} \mu_ {k - 1} ^ {4 / 3} \leq \mu_ {k} <   \mu_ {k - 1}
$$
Note that in Algorithm 4, we have  $\widehat{a} = \sqrt{4(\mu_0 + \varepsilon)} < a$ , then it is obvious
$$
(2 / a) ^ {2 / 3} \mu_ {k - 1} ^ {4 / 3} <   (2 / \widehat {a}) ^ {2 / 3} \mu_ {k - 1} ^ {4 / 3}
$$
The same analysis for Theorem 1 can be applied here.
E.12 Proof of Lemma 6
Proof. By the Theorem 3 and Lemma 5 and the fact that  $A_{\mu, \epsilon}^{1}$  is  $\mu$ -stationary point region, we use the same argument as proof of Lemma 7 to demonstrate the gradient descent will never go to  $A_{\mu, \epsilon}^{2}$ .
E.13 Proof of Lemma 9
Proof. By Theorem 9(iv)
$$
\max_{\mu \leq \tau_{\beta}}x_{\mu ,\beta}^{**}\leq \min_{\mu >0}x_{\mu ,\beta}^{*}
$$
We also know from the proof of Corollary 3,  $x_{\mu ,\epsilon}^{**} < x_{\mu ,\beta}^{**}$  and  $x_{\mu ,\beta}^{*} < x_{\mu ,\epsilon}^{*}$ . Consequently,
$$
\max_{\mu \leq \tau_{\beta}}x_{\mu ,\epsilon}^{**}\leq \min_{\mu >0}x_{\mu ,\epsilon}^{*}
$$
Because  $\tau_{\beta} > \tau$  so
$$
\max  _ {\mu \leq \tau} x _ {\mu , \epsilon} ^ {* *} \leq \max  _ {\mu \leq \tau_ {\beta}} x _ {\mu , \epsilon} ^ {* *} \leq \min  _ {\mu > 0} x _ {\mu , \epsilon} ^ {*}
$$
E.14 Proof of Corollary 1
$$
\frac {a ^ {2}}{4 (a ^ {2} + 1) ^ {3}} \leq \frac {1}{2 7} a > 0
$$
when  $a > \sqrt{\frac{5}{27}}$ , then  $\frac{a^2}{4} > \mu_0 = \frac{1}{27} \geq \frac{a^2}{4(a^2 + 1)^3}$ , it satisfies condition in Lemma 4, we obtain the same result.
E.15 Proof of Corollary 2
Proof. Use Theorem 5(vi) and Theorem 6(vi).
E.16 Proof of Corollary 3
Proof: It is easy to know that
$$
r _ {\beta} (y; \mu) > r _ {\epsilon} (y; \mu) > r (y; \mu)
$$
$$
t _ {\beta} (x; \mu) <   t _ {\epsilon} (x; \mu) <   t (x; \mu)
$$
and when  $\mu < \tau$ , there are three solutions to  $r(y; \mu) = 0$  by Theorem 5. Also, we know from Theorem 7, 8
$$
\lim _ {y \to 0 ^ {+}} r _ {\epsilon} (y; \mu) = \infty \quad \lim _ {y \to 0 ^ {+}} r _ {\beta} (y; \mu) = \infty
$$
Note that when  $\left(\frac{1 + \beta}{1 - \beta}\right)^2 \leq a^2 + 1$
$$
r _ {\beta} (\sqrt {\mu}; \mu) = \frac {a (1 + \beta)}{\sqrt {\mu}} - (a ^ {2} + 1) - \frac {a ^ {2} (1 - \beta) ^ {2}}{4 \mu} \leq 0 \quad \forall \mu > 0
$$
$$
0 \geq r _ {\beta} (\sqrt {\mu}; \mu) > r _ {\epsilon} (\sqrt {\mu}; \mu) > r (\sqrt {\mu}; \mu)
$$
Also, we know that for  $y_{\mathrm{ub}}$  defined in Theorem 5(iii), we know  $r(y_{\mathrm{ub}}; \mu) > 0$  from Theorem 5(iv). Therefore,
$$
r _ {\beta} (y _ {\mathrm {u b}}; \mu) > r _ {\epsilon} (y _ {\mathrm {u b}}; \mu) > r (y _ {\mathrm {u b}}; \mu) > 0
$$
Besides,  $\sqrt{\mu} < y_{\mathrm{ub}}$ . By monotonicity of  $r_{\beta}(y;\mu)$  and  $r_{\epsilon}(y;\mu)$  from the Theorem 7(ii) and Theorem 8(ii), it implies that there are at least two solutions to  $r_{\beta}(y;\mu)$  and  $r_{\epsilon}(y;\mu)$ . From the geometry of  $r_{\beta}(y;\mu), r_{\epsilon}(y;\mu), r(y;\mu)$  and  $t_{\beta}(x;\mu), t_{\epsilon}(x;\mu), t(x;\mu)$ , it is trivial to know that  $x_{\mu,\epsilon}^{*} \leq x_{\mu}^{*}$ ,  $y_{\mu,\epsilon}^{*} \geq y_{\mu}^{*}$ ,  $x_{\mu,\epsilon}^{**} \geq x_{\mu}^{**}$ ,  $y_{\mu,\epsilon}^{*} \leq y_{\mu}^{**}$ .
Finally, for every point  $(x,y)\in A_{\mu ,\epsilon}^{1}$ , there exists a pair  $\epsilon_1,\epsilon_2$ , each satisfying  $|\epsilon_1|\leq \epsilon$  and  $|\epsilon_2|\leq \epsilon$ , such that  $(x,y)$  is the solution to
$$
x = \frac {\mu a + \epsilon_ {1}}{\mu + y ^ {2}} \qquad y = \frac {\mu a + \epsilon_ {2}}{x ^ {2} + \mu (a ^ {2} + 1)}
$$
We can repeat the same analysis above to show that  $x_{\mu, \epsilon}^{*} \leq x$ ,  $y_{\mu, \epsilon}^{*} \geq y$ . Applying the same logic to  $\forall (x, y) \in A_{\mu, \epsilon}^{2}$ , we find  $x_{\mu, \epsilon}^{**} \geq x$ ,  $y_{\mu, \epsilon}^{*} \leq y$ . Thus,  $(x_{\mu}^{*}, y_{\mu}^{*})$  is the extreme point of  $A_{\mu, \epsilon}^{1}$  and  $(x_{\mu}^{**}, y_{\mu}^{**})$  is the extreme point of  $A_{\mu, \epsilon}^{2}$ , we get the results.
F Experiments Details
In this section, we present experiments to validate the global convergence of Algorithm 6. Our goal is twofold: First, we aim to demonstrate that irrespective of the starting point, Algorithm 6 using gradient descent consistently returns the global minimum. Second, we contrast our updating scheme for  $\mu_k,\epsilon_k$  as prescribed in Algorithm 6 with an arbitrary updating scheme for  $\mu_k,\epsilon_k$ . This comparison illustrates how inappropriate setting of parameters in gradient descent could lead to incorrect solutions.
(c) Random Initialization 3
(d) Random Initialization 4
Figure 9: Trajectory of the gradient descent path with the different initializations for  $a = 2$ . We observe that regardless of the initialization, Algorithm 6 always converges to the global minimum. Initial  $\mu_0 = \frac{a^2}{4} \frac{(1 - \delta)^3(1 - \beta)^4}{(1 + \beta)^2}$
F.1 Random Initialization Converges to Global Optimum
(d) Random Initialization 4
Figure 10: Trajectory of the gradient descent path with the different initializations for  $a = 0.5$ . We observe that regardless of the initialization, Algorithm 6 always converges to the global minimum. Initial  $\mu_0 = \frac{a^2}{4} \frac{(1 - \delta)^3(1 - \beta)^4}{(1 + \beta)^2}$
F.2 Wrong Specification of  $\delta$  Leads to Spurious Local Optimal
(a)  $\delta = 0.4$
(b)  $\delta = 0.1$
Figure 11: Trajectory of the gradient descent path for two difference  $\delta$ . Left:  $\beta$  violates requirement  $\left(\frac{1 + \beta}{1 - \beta}\right)^2 \leq (1 - \delta)(a^2 + 1)$  in Theorem 4, leading to spurious local minimum. Right:  $\beta$  follows requirement  $\left(\frac{1 + \beta}{1 - \beta}\right)^2 \leq (1 - \delta)(a^2 + 1)$  in Theorem 4, leading to global minimum. Initial  $\mu_0 = \frac{a^2}{4} \frac{(1 - \delta)^3(1 - \beta)^4}{(1 + \beta)^2}$
F.3 Wrong Specification of  $\beta$  Leads to Incorrect Solution
(a)  $\beta = 0.6$
Figure 12: Trajectory of the gradient descent path for two difference  $\beta$ . Left:  $\beta$  violates requirement  $\left(\frac{1 + \beta}{1 - \beta}\right)^2 \leq (1 - \delta)(a^2 + 1)$  in Theorem 4, leading to incorrect solution. Right:  $\beta$  follows requirement  $\left(\frac{1 + \beta}{1 - \beta}\right)^2 \leq (1 - \delta)(a^2 + 1)$  in Theorem 4, leading to global minimum. Initial  $\mu_0 = \frac{a^2}{4} \frac{(1 - \delta)^3(1 - \beta)^4}{(1 + \beta)^2}$
(b)  $\delta = 0.01$
F.4 Faster decrease of  $\mu_{k}$  Leads to Incorrect Solution
(a) Bad scheduling
Figure 13: Trajectory of the gradient descent path for two difference update rules for  $\mu_{k}$  with the same initialization. Left: "Bad scheduling" uses a faster-decreasing scheme for  $\mu_{k}$ , leading to an incorrect solution, even a non-local optimal solution. Right: "Good scheduling" follows updating rule for  $\mu_{k}$  in Algorithm 6, leading to the global minimum. Initial  $\mu_0 = \frac{a^2}{4} \frac{(1 - \delta)^3(1 - \beta)^4}{(1 + \beta)^2}$
(b) Good scheduling