SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Following the same procedure when we deal with  $\epsilon = 0$ . Let us substitute (12a) into (12b), then we obtain an equation that only involves the variable  $y$
$$
r _ {\epsilon} (y; \mu) = \frac {a + \epsilon / \mu}{y} - (a ^ {2} + 1) - \frac {(\mu a - \epsilon) ^ {2} / \mu}{(y ^ {2} + \mu) ^ {2}} \tag {15}
$$
Let us substitute (12b) into (12a), then we obtain an equation that only involves the variable  $x$
$$
t _ {\epsilon} (x; \mu) = \frac {a - \epsilon / \mu}{x} - 1 - \frac {\left(\mu a + \epsilon\right) ^ {2} / \mu}{\left(\mu \left(a ^ {2} + 1\right) + x ^ {2}\right) ^ {2}} \tag {16}
$$
Proceed similarly for equations (13a) and (13b).
$$
r _ {\epsilon_ {-}} (y; \mu) = \frac {a - \epsilon / \mu}{y} - (a ^ {2} + 1) - \frac {(\mu a + \epsilon) ^ {2} / \mu}{(y ^ {2} + \mu) ^ {2}} \tag {17}
$$
$$
t _ {\epsilon_ {-}} (x; \mu) = \frac {a + \epsilon / \mu}{x} - 1 - \frac {(\mu a - \epsilon) ^ {2} / \mu}{(\mu (a ^ {2} + 1) + x ^ {2}) ^ {2}} \tag {18}
$$
Given the substantial role that the system of equations 12a and 12b play in our analysis, the existence of  $\epsilon$  in these equations complicates the analysis, this can be avoided by considering the worst-case scenario, i.e., when  $\epsilon = \beta a\mu$ . With this particular choice of  $\epsilon$ , we can reformulate (15) and (16) as follows, denoting them as  $r_{\beta}(y;\epsilon)$  and  $r_{\beta}(x;\epsilon)$  respectively.
$$
r _ {\beta} (y; \mu) = \frac {a (1 + \beta)}{y} - (a ^ {2} + 1) - \frac {\mu a ^ {2} (1 - \beta) ^ {2}}{(y ^ {2} + \mu) ^ {2}} \tag {19}
$$
$$
t _ {\beta} (x; \mu) = \frac {a (1 - \beta)}{x} - 1 - \frac {\mu a ^ {2} (1 + \beta) ^ {2}}{\left(\mu \left(a ^ {2} + 1\right) + x ^ {2}\right) ^ {2}} \tag {20}
$$
The functions  $r_{\epsilon}(y;\mu)$ ,  $r_{\epsilon_+}(y;\mu)$ , and  $r_{\beta}(y;\mu)$  possess similar properties to  $r(y;\mu)$  as defined in Equation (8), with more details available in Theorem 7 and 8. Additionally, the functions  $t_{\epsilon}(x;\mu)$ ,  $t_{\epsilon_+}(x;\mu)$ , and  $t_{\beta}(x;\mu)$  share similar characteristics with  $t(x;\mu)$  as defined in Equation (9), with more details provided in Theorem 9.
As illustrated in Figure 6, the  $\epsilon$ -stationary point region  $A_{\mu,\epsilon}$  can be partitioned into two distinct areas, of which only the lower-right one contains  $(x_{\mu}^{*}, y_{\mu}^{*})$  and it is of interest to our analysis. Moreover,  $(x_{\mu,\epsilon}^{*}, y_{\mu,\epsilon}^{*})$  and  $(x_{\mu,\epsilon}^{**}, y_{\mu,\epsilon}^{**})$  are extremal points of two distinct regions. The upcoming corollary substantiates this intuition.
Corollary 3. If  $\mu < \tau$  ( $\tau$  is defined in Theorem 5(v)), assume  $\epsilon$  satisfies (14),  $\beta$  satisfies  $\left(\frac{1 + \beta}{1 - \beta}\right)^2 \leq a^2 + 1$ , systems of equations (12a),(12b) at least have two solutions. Moreover,  $A_{\mu,\epsilon} = A_{\mu,\epsilon}^1 \cup A_{\mu,\epsilon}^2$
$$
A _ {\mu , \epsilon} ^ {1} = A _ {\mu , \epsilon} \cap \left\{\left(x, y\right) \mid x \geq x _ {\mu , \epsilon} ^ {*}, y \leq y _ {\mu , \epsilon} ^ {*} \right\} \tag {21}
$$
$$
A _ {\mu , \epsilon} ^ {2} = A _ {\mu , \epsilon} \cap \left\{\left(x, y\right) \mid x \leq x _ {\mu , \epsilon} ^ {* *}, y \geq y _ {\mu , \epsilon} ^ {* *} \right\} \tag {22}
$$
Corollary 3 suggests that  $A_{\mu, \epsilon}$  can be partitioned into two distinct regions, namely  $A_{\mu, \epsilon}^{1}$  and  $A_{\mu, \epsilon}^{2}$ . Furthermore, for every  $(x, y)$  belonging to  $A_{\mu, \epsilon}^{1}$ , it follows that  $x \geq x_{\mu, \epsilon}^{*}$  and  $y \leq y_{\mu, \epsilon}^{*}$ . Similarly, for every  $(x, y)$  that lies within  $A_{\mu, \epsilon}^{2}$ , the condition  $x \leq x_{\mu, \epsilon}^{**}$  and  $y \geq y_{\mu, \epsilon}^{**}$  holds. The region  $A_{\mu, \epsilon}^{1}$  represents the "correct" region that gradient descent should identify. In this context, identifying the region equates to pinpointing the extremal points of the region. As a result, our focus should be on the extremal points of  $A_{\mu, \epsilon}^{1}$  and  $A_{\mu, \epsilon}^{2}$ , specifically at  $(x_{\mu, \epsilon}^{*}, y_{\mu, \epsilon}^{*})$  and  $(x_{\mu, \epsilon}^{**}, y_{\mu, \epsilon}^{**})$ . Furthermore, the key to ensuring the convergence of the gradient descent to the  $A_{\mu, \epsilon}^{1}$  is to accurately identify the "basin of attraction" of the region  $A_{\mu, \epsilon}^{1}$ . The following lemma provides a region within which, regardless of the initialization point of the gradient descent, it converges inside  $A_{\mu, \epsilon}^{1}$ .
Lemma 6. Assume  $\mu < \tau$  ( $\tau$  is defined in Theorem 5(v)),  $\left(\frac{1 + \beta}{1 - \beta}\right)^2 \leq a^2 + 1$ . Define  $B_{\mu, \epsilon} = \{(x, y) | x_{\mu, \epsilon}^{**} < x \leq a, 0 \leq y < y_{\mu, \epsilon}^{**}\}$ . Run Algorithm 5 with input  $f = g_{\mu}(x, y)$ ,  $\eta = \frac{1}{\mu(a^2 + 1) + 3a^2}$ ,  $W_0 = (x(0), y(0))$ , where  $(x(0), y(0)) \in B_{\mu, \epsilon}$ , then after at most  $\frac{2(\mu(a^2 + 1) + 3a^2)(g_{\mu}(x(0), y(0)) - g_{\mu}(x_{\mu}^*, y_{\mu}^*))}{\epsilon^2}$  iterations,  $(x_t, y_t) \in A_{\mu, \epsilon}^1$ .
Lemma 6 can be considered the gradient descent analogue of Lemma 2. It plays a pivotal role in the proof of Theorem 4. In Figure 6, the lower-right rectangle corresponds to  $B_{\mu,\epsilon}$ . Lemma 6 implies that the gradient descent with any initialization inside  $B_{\mu_{k+1},\epsilon_{k+1}}$  will converge to  $A_{\mu_{k+1},\epsilon_{k+1}}^{1}$  at last. Then, by utilizing the previous solution  $W_{\mu_k,\epsilon_k}$  as the initial point, as long as it lies within region  $B_{\mu_{k+1},\epsilon_{k+1}}$ , the gradient descent can converge to  $A_{\mu_{k+1},\epsilon_{k+1}}^{1}$  which is  $\epsilon$  stationary points region that contains  $W_{\mu_{k+1}}^{*}$ , thereby achieving the goal of tracking  $W_{\mu_{k+1}}^{*}$ . Following the scheduling for  $\mu_k$  prescribed in Algorithm 6 provides a sufficient condition to ensure that will happen.
We now proceed to present the theorem which guarantees the global convergence of Algorithm 6.
Theorem 4. If  $\delta \in (0,1)$ ,  $\beta \in (0,1)$ ,  $\left(\frac{1 + \beta}{1 - \beta}\right)^2 \leq (1 - \delta)(a^2 + 1)$ , and  $\mu_0$  satisfies
$$
\frac {a ^ {2}}{4 (a ^ {2} + 1) ^ {3}} \leq \frac {a ^ {2}}{4 (a ^ {2} + 1) ^ {3}} \frac {(1 + \beta) ^ {4}}{(1 - \beta) ^ {2}} \leq \mu_ {0} \leq \frac {a ^ {2}}{4} \frac {(1 - \delta) ^ {3} (1 - \beta) ^ {4}}{(1 + \beta) ^ {2}} \leq \frac {a ^ {2}}{4}
$$
Set the updating rule
$$
\epsilon_ {k} = \min  \left\{\beta a \mu_ {k}, \mu_ {k} ^ {3 / 2} \right\}
$$
$$
\mu_ {k + 1} = (2 \mu_ {k} ^ {2}) ^ {2 / 3} \frac {\left(a + \epsilon_ {k} / \mu_ {k}\right) ^ {2 / 3}}{\left(a - \epsilon_ {k} / \mu_ {k}\right) ^ {4 / 3}}
$$
Then  $\mu_{k + 1}\leq (1 - \delta)\mu_k$ . Moreover, for any  $\varepsilon_{\mathrm{dist}} > 0$ , running Algorithm 6 after  $K(\mu_0,a,\delta ,\varepsilon_{\mathrm{dist}})$  outer iteration
$$
\left\| W _ {\mu_ {k}, \epsilon_ {k}} - W _ {\mathrm {G}} \right\| _ {2} \leq \varepsilon_ {\text {d i s t}} \tag {23}
$$
$$
K(\mu_{0},a,\delta ,\varepsilon_{\mathrm{dist}})\geq \frac{1}{\ln(1 / (1 - \delta))}\max \left\{\ln \frac{\mu_{0}}{\beta^{2}a^{2}},\ln \frac{72\mu_{0}}{a^{2}(1 - (1 / 2)^{1 / 4})},\ln (\frac{3(4 - \delta)\mu_{0}}{\varepsilon_{\mathrm{dist}}{}^{2}}),\frac{1}{2}\ln (\frac{46656\mu_{0}^{2}}{a^{2}\varepsilon_{\mathrm{dist}}{}^{2}}),\frac{1}{3}\ln (\frac{46656\mu_{0}^{3}}{a^{4}\varepsilon_{\mathrm{dist}}{}^{2}})\right\}
$$
The total gradient descent steps are
$$
\sum_ {k = 0} ^ {K (\mu_ {0}, a, \delta , \varepsilon_ {\mathrm {d i s t}})} \frac {2 (\mu_ {k} (a ^ {2} + 1) + 3 a ^ {2}) (g _ {\mu_ {k + 1}} (W _ {\mu_ {k} , \epsilon_ {k}}) - g _ {\mu_ {k + 1}} (W _ {\mu_ {k + 1} , \epsilon_ {k + 1}}))}{\epsilon_ {k} ^ {2}}
$$
$$
\leq 2 \left(\mu_ {0} \left(a ^ {2} + 1\right) + 3 a ^ {2}\right) \left(\frac {1}{\beta^ {6} a ^ {6}} + \left(\max  \left\{\frac {3 (4 - \delta)}{\varepsilon_ {\text {d i s t}} {} ^ {2}}, \frac {2 1 6}{a \varepsilon_ {\text {d i s t}}}, \left(\frac {2 1 6}{a \varepsilon_ {\text {d i s t}}}\right) ^ {2 / 3}, \frac {1}{\beta^ {2} a ^ {2}}, \frac {7 2}{(1 - (1 / 2) ^ {1 / 4}) a ^ {2}} \right\}\right) ^ {3}\right) g _ {\mu_ {0}} \left(W _ {\mu_ {0}} ^ {\epsilon_ {0}}\right)
$$
$$
\lesssim O \left(\mu_ {0} a ^ {2} + a ^ {2} + \mu_ {0}\right) \left(\frac {1}{\beta^ {6} a ^ {6}} + \frac {1}{\varepsilon_ {\text {d i s t}} ^ {6}} + \frac {1}{a ^ {3} \varepsilon_ {\text {d i s t}} ^ {3}} + \frac {1}{a ^ {2} \varepsilon_ {\text {d i s t}} ^ {2}} + \frac {1}{a ^ {6}}\right)
$$
Proof. Upon substituting gradient flow with gradient descent, it becomes possible to only identify an  $\epsilon$ -stationary point for  $g_{\mu}(W)$ . This modification necessitates specifying the stepsize  $\eta$  for gradient descent, as well as an updating rule for  $\mu$ . The adjustment procedure used can substantially influence the result of Algorithm 6. In this proof, we will impose limitations on the update scheme  $\mu_k$ , the stepsize  $\eta_k$ , and the tolerance  $\epsilon_k$  to ensure their effective operation within Algorithm 6. The approach employed for this proof closely mirrors that of the proof for Theorem 1 albeit with more careful scrutiny. In this proof, we will work out all the requirements for  $\mu, \epsilon, \eta$ . Subsequently, we will verify that our selection in Theorem 4 conforms to these requirements.
In the proof, we occasionally use  $\mu, \epsilon$  or  $\mu_k, \epsilon_k$ . When we employ  $\mu, \epsilon$ , it signifies that the given inequality or equality holds for any  $\mu, \epsilon$ . Conversely, when we use  $\mu_k, \epsilon_k$ , it indicates we are examining how to set these parameters for distinct iterations.
Establish the Bound  $y_{\mu ,\epsilon}^{**}\geq \sqrt{\mu}$  First, let us consider  $r_\epsilon (\sqrt{\mu};\mu)\leq 0$  , i.e.
$$
r _ {\epsilon} (\sqrt {\mu}; \mu) = \frac {a + \epsilon / \mu}{\sqrt {\mu}} - (a ^ {2} + 1) - \frac {\mu (a - \epsilon / \mu) ^ {2}}{4 \mu^ {2}} \leq 0
$$
This is always true when  $\mu > 4 / a^2$ , and we require
$$
\epsilon \leq 2 \mu^ {3 / 2} + a \mu - 2 \sqrt {2 a \mu^ {5 / 2} - \mu^ {3} a ^ {2}} \quad \text {w h e n} \mu \leq \frac {4}{a ^ {2}}
$$
Now we name it condition 1.
Condition 1.
$$
\epsilon \leq 2 \mu^ {3 / 2} + a \mu - 2 \sqrt {2 a \mu^ {5 / 2} - \mu^ {3} a ^ {2}} \quad w h e n \mu \leq \frac {4}{a ^ {2}}
$$
Under the assumption that Condition 1 is satisfied. Since  $r_{\epsilon}(y;\mu)$  is increasing function with interval  $y \in [y_{\mathrm{lb},\epsilon}, y_{\mathrm{ub},\epsilon}]$ , and we know  $y_{\mathrm{lb},\epsilon} \leq \sqrt{\mu} \leq y_{\mathrm{ub},\epsilon}$  and based on Theorem 7(ii), we have  $y_{\mathrm{lb},\epsilon} \leq y_{\mu,\epsilon}^{**} \leq y_{\mathrm{ub},\epsilon}$ ,  $r_{\epsilon}(\sqrt{\mu}; \mu) \leq r_{\epsilon}(y_{\mu,\epsilon}^{**}; \mu) = 0$ . Therefore,  $y_{\mu,\epsilon}^{**} \geq \sqrt{\mu}$ .
Ensuring the Correct Solution Path via Gradient Descent Following the argument when we prove Theorem 1, we strive to ensure that the gradient descent, when initiated at  $(x_{\mu_k,\epsilon_k},y_{\mu_k,\epsilon_k})$ , will converge within the "correct"  $\epsilon_{k + 1}$ -stationary point region (namely,  $\| \nabla g_{\mu_{k + 1}}(W)\| _2 < \epsilon_{k + 1}$ ) which includes  $(x_{\mu_{k + 1}}^*,y_{\mu_{k + 1}}^*)$ . For this to occur, we necessitate that:
$$
y _ {\mu_ {k + 1}, \epsilon_ {k + 1}} \stackrel {(1)} {>} y _ {\mu_ {k + 1}, \epsilon_ {k + 1}} ^ {* *} \stackrel {(2)} {>} \sqrt {\mu_ {k + 1}} \geq (2 \mu_ {k} ^ {2}) ^ {1 / 3} \frac {(a + \epsilon_ {k} / \mu_ {k}) ^ {1 / 3}}{(a - \epsilon_ {k} / \mu_ {k}) ^ {2 / 3}} \stackrel {(4)} {>} y _ {\mu_ {k}, \epsilon_ {k}} ^ {*} \stackrel {(5)} {>} y _ {\mu_ {k}, \epsilon_ {k}} \tag {24}
$$
Here (1), (5) are due to Corollary 3; (2) comes from the boundary we established earlier; (3) is based on the constraints we have placed on  $\mu_{k}$  and  $\mu_{k + 1}$ , which we will present as Condition 2 subsequently; (4) is from the Theorem 7(ii) and relationship  $y_{\mu_k,\epsilon_k}^* < y_{\mathrm{lb},\mu_k,\epsilon_k}$ . Also, from the Lemma 9,  $\max_{\mu \leq \tau}x_{\mu ,\epsilon}^{**}\leq \min_{\mu >0}x_{\mu ,\epsilon}^*$ . Hence, by invoking Lemma 6, we can affirm that our gradient descent consistently traces the correct stationary point. Now we state condition to make it happen,
Condition 2.
$$
(1 - \delta) \mu_ {k} \geq \mu_ {k + 1} \geq (2 \mu_ {k} ^ {2}) ^ {2 / 3} \frac {(a + \epsilon_ {k} / \mu_ {k}) ^ {2 / 3}}{(a - \epsilon_ {k} / \mu_ {k}) ^ {4 / 3}}
$$
In this context, our requirement extends beyond merely ensuring that  $\mu_k$  decreases. We further stipulate that it should decrease by a factor of  $1 - \delta$ . Next, we impose another important constraint
Condition 3.
$$
\epsilon_ {k} \leq \mu_ {k} ^ {3 / 2}
$$
Updating Rules Now we are ready to check our updating rules satisfy the conditions above
$$
\mu_ {k + 1} = (2 \mu_ {k} ^ {2}) ^ {2 / 3} \frac {(a + \epsilon_ {k} / \mu_ {k}) ^ {2 / 3}}{(a - \epsilon_ {k} / \mu_ {k}) ^ {4 / 3}}
$$
Check for Conditions First, we check the condition 2. condition 2 requires
$$
(1 - \delta) \mu_ {k} \geq (2 \mu_ {k} ^ {2}) ^ {2 / 3} \frac {(a + \epsilon_ {k} / \mu_ {k}) ^ {2 / 3}}{(a - \epsilon_ {k} / \mu_ {k}) ^ {4 / 3}} \Rightarrow \mu_ {k} \frac {(a + \epsilon_ {k} / \mu_ {k}) ^ {2}}{(a - \epsilon_ {k} / \mu_ {k}) ^ {4}} \leq \frac {(1 - \delta) ^ {3}}{4}
$$
Note that  $\epsilon_{k}\leq \beta a\mu_{k} <   a\mu_{k}$
$$
\mu_ {k} \frac {\left(a + \epsilon_ {k} / \mu_ {k}\right) ^ {2}}{\left(a - \epsilon_ {k} / \mu_ {k}\right) ^ {4}} \leq \mu_ {k} \frac {(1 + \beta) ^ {2}}{(1 - \beta) ^ {4}} \frac {1}{a ^ {2}}
$$
Therefore, once the following inequality is true, Condition 2 is satisfied.
$$
\mu_ {k} \frac {(1 + \beta) ^ {2}}{(1 - \beta) ^ {4}} \frac {1}{a ^ {2}} \leq \frac {(1 - \delta) ^ {3}}{4} \Rightarrow \mu_ {k} \leq \frac {a ^ {2}}{4} \frac {(1 - \delta) ^ {3} (1 - \beta) ^ {4}}{(1 + \beta) ^ {2}}
$$
Because  $\mu_{k} \leq \mu_{0} \leq \frac{a^{2}}{4} \frac{(1 - \delta)^{3}(1 - \beta)^{4}}{(1 + \beta)^{2}}$  from the condition we impose for  $\mu_{0}$ . Consequently, Condition 2 is satisfied under our choice of  $\epsilon_{k}$ .
Now we focus on the Condition 1. Because  $\epsilon_k \leq a\beta \mu_k$ , if we can ensure  $a\beta \mu_k \leq 2\mu_k^{3/2} + a\mu_k - 2\sqrt{2a\mu_k^{5/2} - \mu_k^3a^2}$  holds, then we can show Condition 1 is always satisfied.
$$
a \beta \mu_ {k} \leq 2 \mu_ {k} ^ {3 / 2} + a \mu_ {k} - 2 \sqrt {2 a \mu_ {k} ^ {5 / 2} - \mu_ {k} ^ {3} a ^ {2}}
$$
$$
\begin{array}{l} 2 \sqrt {2 a \mu_ {k} ^ {5 / 2} - \mu_ {k} ^ {3} a ^ {2}} \leq 2 \mu_ {k} ^ {3 / 2} + (1 - \beta) a \mu_ {k} \\ 4 (2 a \mu_ {k} ^ {5 / 2} - \mu_ {k} ^ {3} a ^ {2}) \leq 4 \mu_ {k} ^ {3} + (1 - \beta) ^ {2} a ^ {2} \mu_ {k} ^ {2} + 4 (1 - \beta) a \mu_ {k} ^ {5 / 2} \\ 0 \leq 4 (a ^ {2} + 1) \mu_ {k} ^ {3} + (1 - \beta) ^ {2} a ^ {2} \mu_ {k} ^ {2} - 4 (1 + \beta) a \mu_ {k} ^ {5 / 2} \\ 0 \leq 4 (a ^ {2} + 1) \mu_ {k} - 4 (1 + \beta) a \mu_ {k} ^ {1 / 2} + (1 - \beta) ^ {2} a ^ {2} \quad \text {w h e n} \quad 0 \leq \mu_ {k} \leq 4 / a ^ {2} \\ 0 \leq \mu_ {k} - \frac {(1 + \beta) a}{(a ^ {2} + 1)} \mu_ {k} ^ {1 / 2} + \frac {(1 - \beta) ^ {2} a ^ {2}}{4 (a ^ {2} + 1)} \\ \end{array}
$$
We also notice that
$$
\frac {(1 + \beta) ^ {2} a ^ {2}}{(a ^ {2} + 1) ^ {2}} - 4 \frac {(1 - \beta) ^ {2} a ^ {2}}{4 (a ^ {2} + 1)} \leq 0 \Leftrightarrow \left(\frac {1 + \beta}{1 - \beta}\right) ^ {2} \leq a ^ {2} + 1
$$
Because  $\left(\frac{1 + \beta}{1 - \beta}\right)^2 \leq (1 - \delta)(a^2 + 1)$ , the inequality above always holds and this inequality implies that for any  $\mu_k \geq 0$
$$
0 \leq \mu_ {k} - \frac {(1 + \beta) a}{(a ^ {2} + 1)} \mu_ {k} ^ {1 / 2} + \frac {(1 - \beta) ^ {2} a ^ {2}}{4 (a ^ {2} + 1)}
$$
Therefore, Condition 2 holds. Condition 3 also holds because of the choice of  $\epsilon_{k}$ .
Bound the Distance Let  $c = 72 / a^2$ , and assume that  $\mu$  satisfies the following
$$
\mu \leq \min  \left\{\frac {1}{c} \left(1 - (1 / 2) ^ {1 / 4}\right), \beta^ {2} a ^ {2} \right\} \tag {25}
$$
Note that when  $\mu$  satisfies (25), then  $\mu^{3/2} \leq \beta a\mu$ , so  $\epsilon = \mu^{3/2}$ .
$$
\mu \leq \frac {1}{c} \left(1 - (1 / 2) ^ {1 / 4}\right) = \frac {a ^ {2}}{7 2} \left(1 - (1 / 2) ^ {1 / 4}\right) \leq \frac {a ^ {2}}{4}
$$
$$
\epsilon / \mu = \sqrt {\mu} \leq \frac {a}{2} \tag {26}
$$
$$
\begin{array}{l} t _ {\epsilon} ((a - \epsilon / \mu) (1 - c \mu); \mu) = \frac {1}{1 - c \mu} - 1 - \frac {\mu (a + \epsilon / \mu) ^ {2}}{(\mu (a ^ {2} + 1) + (a - \epsilon / \mu) ^ {2} (1 - c \mu) ^ {2}) ^ {2}} \\ = \frac {c \mu}{1 - c \mu} - \frac {\mu (a + \epsilon / \mu) ^ {2}}{(\mu (a ^ {2} + 1) + (a - \epsilon / \mu) ^ {2} (1 - c \mu) ^ {2}) ^ {2}} \\ \geq c \mu - \mu \frac {(a + \epsilon / \mu) ^ {2}}{(a - \epsilon / \mu) ^ {4} (1 - c \mu) ^ {4}} \\ \geq c \mu - \mu \frac {(a + a / 2) ^ {2}}{(a - a / 2) ^ {4} (1 - c \mu) ^ {4}} \\ = \mu \left(c - \frac {3 6}{a ^ {2} (1 - c \mu) ^ {4}}\right) \\ = \mu \left(\frac {7 2}{a ^ {2}} - \frac {3 6}{a ^ {2} (1 - c \mu) ^ {4}}\right) > 0 \\ \end{array}
$$
Then we know  $(a - \epsilon / \mu)(1 - c\mu) < x_{\mu, \epsilon}^{*}$ . Now we can bound the distance  $\| W_{\mu_k, \epsilon_k} - W_{\mathbf{G}} \|$ , it is important to note that
$$
\begin{array}{l} \left\| W _ {\mu_ {k}, \epsilon_ {k}} - W _ {\mathsf {G}} \right\| = \sqrt {\left(x _ {\mu_ {k} , \epsilon_ {k}} - a\right) ^ {2} + \left(y _ {\mu_ {k} , \epsilon_ {k}}\right) ^ {2}} \\ \leq \max \left\{\sqrt {(x _ {\mu_ {k} , \epsilon_ {k}} ^ {*} - a) ^ {2} + (y _ {\mu_ {k} , \epsilon_ {k}} ^ {*}) ^ {2}}, \sqrt {(x _ {\mu_ {k} , \epsilon_ {k -}} ^ {*} - a) ^ {2} + (y _ {\mu_ {k} , \epsilon_ {k}} ^ {*}) ^ {2}} \right\} \\ \end{array}
$$
We use the fact that  $x_{\mu_k,\epsilon_k}^* < x_{\mu_k,\epsilon_k} < a, x_{\mu_k,\epsilon_k} < x_{\mu_k,\epsilon_{k-}}^*$  and  $y_{\mu_k,\epsilon_k} < y_{\mu_k,\epsilon_k}^*$ . Next, we can separately establish bounds for these two terms. Due to (24),  $y_{\mu_k,\epsilon_k}^* < (2\mu_k^2)^{1/3} \frac{(a + \epsilon_k / \mu_k)^{1/3}}{(a - \epsilon_k / \mu_k)^{2/3}} = \sqrt{\mu_{k+1}}$  and  $(a - \epsilon_k / \mu_k)(1 - c\mu_k) < x_{\mu_k,\epsilon_k}^*$ .
$$
\sqrt {\left(x _ {\mu_ {k} , \epsilon_ {k}} ^ {*} - a\right) ^ {2} + \left(y _ {\mu_ {k} , \epsilon_ {k}} ^ {*}\right) ^ {2}} \leq \sqrt {\mu_ {k + 1} + \left(a - \left(a - \epsilon_ {k} / \mu_ {k}\right) \left(1 - c \mu_ {k}\right)\right) ^ {2}}
$$
Given that if  $x_{\mu_k,\epsilon_{k-}}^* \leq a$ , then  $\sqrt{(x_{\mu_k,\epsilon_k}^* - a)^2 + (y_{\mu_k,\epsilon_k}^*)^2} \geq \sqrt{(x_{\mu_k,\epsilon_{k-}}^* - a)^2 + (y_{\mu_k,\epsilon_k}^*)^2}$ . Therefore, if  $x_{\mu_k,\epsilon_{k-}}^* \geq a$ , we can use the fact that  $x_{\mu_k,\epsilon_{k-}}^* \leq a + \frac{\epsilon_k}{\mu_k}$ . In this case,
$$
\sqrt {(x _ {\mu_ {k} , \epsilon_ {k -}} ^ {*} - a) ^ {2} + (y _ {\mu_ {k} , \epsilon_ {k}} ^ {*}) ^ {2}} \leq \sqrt {\mu_ {k + 1} + (\epsilon_ {k} / \mu_ {k}) ^ {2}} = \sqrt {\mu_ {k + 1} + \mu_ {k}} \leq \sqrt {(2 - \delta) \mu_ {k}}
$$
$$
\| W _ {\mu_ {k}, \epsilon_ {k}} - W _ {\mathsf {G}} \| \leq \max \{\sqrt {\mu_ {k + 1} + (a - (a - \epsilon_ {k} / \mu_ {k}) (1 - c \mu_ {k})) ^ {2}}, \sqrt {(2 - \delta) \mu_ {k}} \}
$$
$$
\begin{array}{l} \mu_ {k + 1} + \big (a - \big (a - \epsilon_ {k} / \mu_ {k} \big) \big (1 - c \mu_ {k} \big) \big) ^ {2} \leq (1 - \delta) \mu_ {k} + \big (a c \mu_ {k} + \sqrt {\mu_ {k}} - c \mu_ {k} ^ {3 / 2} \big) ^ {2} \\ \leq (1 - \delta) \mu_ {k} + 3 \left(a ^ {2} c ^ {2} \mu_ {k} ^ {2} + \mu_ {k} + c ^ {2} \mu_ {k} ^ {3}\right) \\ = (4 - \delta) \mu_ {k} + 3 a ^ {2} c ^ {2} \mu_ {k} ^ {2} + 3 c ^ {2} \mu_ {k} ^ {3} \\ \end{array}
$$
$$
\begin{array}{l} \left\| W _ {\mu_ {k}, \epsilon_ {k}} - W _ {\mathsf {G}} \right\| \leq \max  \{\sqrt {\mu_ {k + 1} + (a - (a - \epsilon_ {k} / \mu_ {k}) (1 - c \mu_ {k})) ^ {2}}, \sqrt {(2 - \delta) \mu_ {k}} \} \\ \leq \max \{\sqrt {(4 - \delta) \mu_ {k} + 3 a ^ {2} c ^ {2} \mu_ {k} ^ {2} + 3 c ^ {2} \mu_ {k} ^ {3}}, \sqrt {(2 - \delta) \mu_ {k}} \} \\ = \sqrt {(4 - \delta) \mu_ {k} + 3 a ^ {2} c ^ {2} \mu_ {k} ^ {2} + 3 c ^ {2} \mu_ {k} ^ {3}} \\ \end{array}
$$
Just let
$$
(4 - \delta) \mu_ {k} \leq (4 - \delta) (1 - \delta) ^ {k} \mu_ {0} \leq \frac {\varepsilon_ {\text {d i s t}} {} ^ {2}}{3} \Rightarrow k \geq \frac {\ln \left(3 (4 - \delta) \mu_ {0} / \varepsilon_ {\text {d i s t}} {} ^ {2}\right)}{\ln (1 / (1 - \delta))} \tag {27}
$$
$$
3 a ^ {2} c ^ {2} \mu_ {k} ^ {2} \leq 3 a ^ {2} c ^ {2} (1 - \delta) ^ {2 k} \mu_ {0} ^ {2} \leq \frac {\varepsilon_ {\text {d i s t}} {} ^ {2}}{3} \Rightarrow k \geq \frac {\ln \left(4 6 6 5 6 \mu_ {0} ^ {2} / \left(a ^ {2} \varepsilon_ {\text {d i s t}} {} ^ {2}\right)\right)}{2 \ln (1 / (1 - \delta))} \tag {28}
$$
$$
3 c ^ {2} \mu_ {k} ^ {3} \leq 3 c ^ {2} (1 - \delta) ^ {3 k} \mu_ {0} ^ {3} \leq \frac {\varepsilon_ {\mathrm {d i s t}} {} ^ {2}}{3} \Rightarrow k \geq \frac {\ln \left(4 6 6 5 6 \mu_ {0} ^ {3} / \left(a ^ {4} \varepsilon_ {\mathrm {d i s t}} {} ^ {2}\right)\right)}{3 \ln (1 / (1 - \delta))} \tag {29}
$$
We use the fact that  $\mu_k \leq (1 - \delta)^k \mu_0$ . In order to satisfy (25),
$$
\mu_ {k} \leq \mu_ {0} (1 - \delta) ^ {k} \leq \frac {a ^ {2}}{7 2} (1 - (1 / 2) ^ {1 / 4}) \Rightarrow k \geq \frac {\ln \frac {7 2 \mu_ {0}}{a ^ {2} (1 - (1 / 2) ^ {1 / 4})}}{\ln \frac {1}{1 - \delta}} \tag {30}
$$
$$
\mu_ {k} \leq \mu_ {0} (1 - \delta) ^ {k} \leq \beta^ {2} a ^ {2} \Rightarrow k \geq \frac {\ln \left(\mu_ {0} / \left(\beta^ {2} a ^ {2}\right)\right)}{\ln \frac {1}{1 - \delta}} \tag {31}
$$
Consequently, running Algorithm 6 after  $K(\mu_0, a, \delta, \varepsilon_{\mathrm{dist}})$  outer iteration
$$
\| W _ {\mu_ {k}, \epsilon_ {k}} - W _ {G} \| _ {2} \leq \varepsilon_ {\text {d i s t}}
$$
$$
K (\mu_ {0}, a, \delta , \varepsilon_ {\mathrm {d i s t}}) \geq \frac {1}{\ln (1 / (1 - \delta))} \max \left\{\ln \frac {\mu_ {0}}{\beta^ {2} a ^ {2}}, \ln \frac {7 2 \mu_ {0}}{a ^ {2} (1 - (1 / 2) ^ {1 / 4})}, \ln (\frac {3 (4 - \delta) \mu_ {0}}{\varepsilon^ {2}}), \frac {1}{2} \ln (\frac {4 6 6 5 6 \mu_ {0} ^ {2}}{a ^ {2} \varepsilon^ {2}}), \frac {1}{3} \ln (\frac {4 6 6 5 6 \mu_ {0} ^ {3}}{a ^ {4} \varepsilon^ {2}}) \right\}
$$
By Lemma 6,  $k$  iteration of Algorithm 6 need the following step of gradient descent
$$
\frac {2 (\mu_ {k} (a ^ {2} + 1) + 3 a ^ {2}) \left(g _ {\mu_ {k + 1}} \left(W _ {\mu_ {k} , \epsilon_ {k}}\right) - g _ {\mu_ {k + 1}} \left(W _ {\mu_ {k + 1} , \epsilon_ {k + 1}}\right)\right)}{\epsilon_ {k} ^ {2}}
$$
Let  $\widehat{K}(\mu_0, a, \delta, \varepsilon_{\mathrm{dist}})$  satisfy  $\mu_{\widehat{K}(\mu_0, a, \delta, \varepsilon_{\mathrm{dist}})} \leq \beta^2 a^2 < \mu_{\widehat{K}(\mu_0, a, \delta, \varepsilon_{\mathrm{dist}})-1}$ . Hence, the total number of gradient steps required by Algorithm 6 can be expressed as follows:
Note from (27) and (30), the following should hold
$$
\mu_ {K (\mu_ {0}, a, \delta , \varepsilon_ {\mathrm {d i s t}})} = \min \{\frac {\varepsilon_ {\mathrm {d i s t}} {} ^ {2}}{3 (4 - \delta)}, \frac {a \varepsilon_ {\mathrm {d i s t}}}{2 1 6}, \left(\frac {a \varepsilon_ {\mathrm {d i s t}}}{2 1 6}\right) ^ {2 / 3}, \beta^ {2} a ^ {2}, \frac {a ^ {2}}{7 2} (1 - (1 / 2) ^ {1 / 4}) \}
$$
$$
\begin{array}{l} \sum_ {k = 0} ^ {K (\mu_ {0}, a, \delta , \varepsilon_ {\mathrm {d i s t}})} \frac {2 (\mu_ {k} (a ^ {2} + 1) + 3 a ^ {2}) (g _ {\mu_ {k + 1}} (W _ {\mu_ {k} , \epsilon_ {k}}) - g _ {\mu_ {k + 1}} (W _ {\mu_ {k + 1} , \epsilon_ {k + 1}}))}{\epsilon_ {k} ^ {2}} \\ \leq 2 \left(\mu_ {0} \left(a ^ {2} + 1\right) + 3 a ^ {2}\right) \left(\frac {1}{\beta^ {6} a ^ {6}} + \left(\max  \left\{\frac {3 (4 - \delta)}{\varepsilon_ {\text {d i s t}} {} ^ {2}}, \frac {2 1 6}{a \varepsilon_ {\text {d i s t}}}, \left(\frac {2 1 6}{a \varepsilon_ {\text {d i s t}}}\right) ^ {2 / 3}, \frac {1}{\beta^ {2} a ^ {2}}, \frac {7 2}{(1 - (1 / 2) ^ {1 / 4}) a ^ {2}} \right\}\right) ^ {3}\right) g _ {\mu_ {0}} \left(W _ {\mu_ {0}} ^ {\epsilon_ {0}}\right) \\ \end{array}
$$
D Additional Theorems and Lemmas
Theorem 5 (Detailed Property of  $r(y;\mu)$ ). For  $r(y;\mu)$  in (8), then
(i) For  $\mu > 0$ ,  $\lim_{y \to 0^+} r(y; \mu) = \infty$ ,  $r\left(\frac{a}{a^2 + 1}, \mu\right) < 0$
(ii) For  $\mu > 0$ ,  $r(\sqrt{\mu}, \mu) < 0$ .
(iii) For  $\mu >\frac{a^2}{4}$