SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 3: Practical (i.e. independent of  $a$  and  $W_{*}$ ) homotopy algorithm for solving (1).
Input: Initial  $W_{0} = W(x_{0},y_{0})$
Output:  $\{W_{\mu_k}\}_{k = 0}^{\infty}$
1  $\mu_0\gets 1 / 27$
2  $W_{\mu_0}\gets \mathrm{GradientFlow}(g_{\mu_0},W_0)$
3 for  $k = 1,2,\ldots$  do
4 Let  $\mu_{k} = \left(2 / \sqrt{5\mu_{0}}\right)^{2 / 3}\mu_{k - 1}^{4 / 3}$  5  $W_{\mu_k}\gets \mathrm{GradientFlow}(g_{\mu_k},W_{\mu_{k - 1}})$
6 end
(a)  $\mu >\tau$
(b)  $\mu = \tau$
Figure 2: The behavior of  $r(y; \mu)$  for different  $\mu$ . Here, for  $\mu > \tau$ , there exists a single solution to  $r(y; \mu) = 0$ , which implies there is one stationary point in Equation (7). When  $\mu = \tau$ , two solutions are found for  $r(y; \mu) = 0$ , suggesting that there are two stationary points in Equation (7). Conversely, when  $\mu < \tau$ , we observe three solutions for  $r(y; \mu) = 0$ , indicating that there are three stationary points in Equation (7)-a local optimum, a saddle point, and a global optimum.
(c)  $\mu <  \tau$
4 A Detailed Analysis of the Evolution of the Stationary Points
The homotopy approach in Algorithm 2 relies heavily on how the stationary points of (7) behave with respect to  $\mu_{k}$ . In this section, we dive deep into the properties of these critical points.
By analyzing the first-order conditions for  $g_{\mu}$ , we first narrow our attention to the region  $A = \{0 \leq x \leq a, 0 \leq y \leq \frac{a}{a^2 + 1}\}$ . By solving the resulting equations, we obtain an equation that only involves the variable  $y$ :
$$
r (y; \mu) = \frac {a}{y} - \frac {\mu a ^ {2}}{(y ^ {2} + \mu) ^ {2}} - (a ^ {2} + 1). \tag {8}
$$
Likewise, we can find an equation only involving the variable  $x$ :
$$
t (x; \mu) = \frac {a}{x} - \frac {\mu a ^ {2}}{(\mu (a ^ {2} + 1) + x ^ {2}) ^ {2}} - 1. \tag {9}
$$
To understand the behavior of the stationary points of  $g_{\mu}(W)$ , we can examine the characteristics of  $t(x; \mu)$  in the range  $x \in [0, a]$  and the properties of  $r(y; \mu)$  in the interval  $y \in [0, \frac{a}{a^2 + 1}]$ .
In Figures 2 and 3, we show the behavior of  $r(y; \mu)$  and  $t(x; \mu)$  for  $a = 1$ . Theorems 5 and 6 in the appendix establish the existence of a  $\tau > 0$  with the following useful property:
Corollary 2. There exists  $\mu < \tau$  such that the equation  $\nabla g_{\mu}(W) = 0$  has three different solutions, denoted as  $W_{\mu}^{*}, W_{\mu}^{**}, W_{\mu}^{***}$ . Then,
$$
\lim _ {\mu \to 0} W _ {\mu} ^ {*} = \left[ \begin{array}{l l} 0 & a \\ 0 & 0 \end{array} \right], \lim _ {\mu \to 0} W _ {\mu} ^ {* *} = \left[ \begin{array}{l l} 0 & 0 \\ 0 & 0 \end{array} \right], \lim _ {\mu \to 0} W _ {\mu} ^ {* * *} = \left[ \begin{array}{l l} 0 & 0 \\ \frac {a}{a ^ {2} + 1} & 0 \end{array} \right]
$$
Note that the interesting regime takes place when  $\mu < \tau$ . Then, we characterize the stationary points as either local minima or saddle points:
(b)  $\mu = \tau$
Figure 3: The behavior of  $t(x; \mu)$  for different  $\mu$ . Here, for  $\mu > \tau$ , there exists a single solution to  $t(x; \mu) = 0$ , which implies there is one stationary point in Equation (7). When  $\mu = \tau$ , two solutions are found for  $t(x; \mu) = 0$ , suggesting that there are two stationary points in Equation (7). Conversely, when  $\mu < \tau$ , we observe three solutions for  $t(x; \mu) = 0$ , indicating that there are three stationary points in Equation (7)-a local optimum, a saddle point, and a global optimum.
Lemma 1. Let  $\mu < \tau$ , then  $g_{\mu}(W)$  has two local minima at  $W_{\mu}^{*}$ ,  $W_{\mu}^{***}$ , and a saddle point at  $W_{\mu}^{**}$ .
With the above results, it has been established that  $W_{\mu}^{*}$  converges to the global minimum  $W_{\mathrm{G}}$  as  $\mu \to 0$ . In the following section for the proof of Theorem 1, we perform a thorough analysis on how to track  $W_{\mu}^{*}$  and avoid the local minimum at  $W_{\mu}^{**}$  by carefully designing the scheduling for  $\mu_{k}$ .
5 Convergence Analysis: From continuous to discrete
We now discuss the iteration complexity of our method when gradient descent is used in place of gradient flow. We begin with some preliminaries regarding the continuous-time analysis.
5.1 Continuous case: Gradient flow
The key to ensuring the convergence of gradient flow to  $W_{\mu}^{*}$  is to accurately identify the basin of attraction of  $W_{\mu}^{*}$ . The following lemma provides a region that lies within such basin of attraction.
Lemma 2. Define  $B_{\mu} = \{(x,y) \mid x_{\mu}^{**} < x \leq a, 0 \leq y < y_{\mu}^{**}\}$ . Run Algorithm 1 with input  $f = g_{\mu}(x,y), z_0 = (x(0), y(0))$  where  $(x(0), y(0)) \in B_{\mu}$ , then  $\forall t \geq 0$ , we have that  $(x(t), y(t)) \in B_{\mu}$  and  $\lim_{t \to \infty} (x(t), y(t)) = (x_{\mu}^{*}, y_{\mu}^{*})$ .
In Figure 1 (b), the lower-right rectangle corresponds to  $B_{\mu}$ . Lemma 2 implies that the gradient flow with any initialization inside  $B_{\mu_{k + 1}}$  will converge to  $W_{\mu_{k + 1}}^{*}$  at last. Then, by utilizing the previous solution  $W_{\mu_k}^*$  as the initial point, as long as it lies within region  $B_{\mu_{k + 1}}$ , the gradient flow can converge to  $W_{\mu_{k + 1}}^{*}$ , thereby achieving the goal of tracking  $W_{\mu_{k + 1}}^{*}$ . Following the scheduling for  $\mu_{k}$  prescribed in Algorithm 2 provides a sufficient condition to ensure that will happen.
The following lemma, with proof in the appendix, is used for the Proof of Theorem 1. It provides a lower bound for  $y_{\mu}^{**}$  and upper bound for  $y_{\mu}^{*}$ .
Lemma 3. If  $\mu < \tau$ , then  $y_{\mu}^{**} > \sqrt{\mu}$ , and  $\frac{(4\mu)^{1/3}}{2}\left(a^{1/3} - \sqrt{a^{2/3} - (4\mu)^{1/3}}\right) > y_{\mu}^{*}$ .
Proof of Theorem 1. Consider that we are at iteration  $k + 1$  of Algorithm 2, then  $\mu_{k + 1} < \mu_k$ . If  $\mu_k > \tau$  and  $\mu_{k + 1} > \tau$ , then there is only one stationary point for  $g_{\mu_k}(x,y)$  and  $g_{\mu_{k + 1}}(x,y)$ , thus, 1 will converge to such stationary point. Hence, let us assume  $\mu_{k + 1} \leq \tau$ . From Theorem 6 in the appendix, we know that  $x_{\mu_{k + 1}}^{**} < x_{\mu_k}^*$ . Then, the following relations hold:
$$
y _ {\mu_ {k + 1}} ^ {* *} \stackrel {(1)} {>} \sqrt {\mu_ {k + 1}} \geq 2 \left(\frac {\mu_ {k} ^ {2}}{4 a}\right) ^ {1 / 3} \stackrel {(2)} {\geq} \frac {(4 \mu_ {k}) ^ {1 / 3}}{2} \left(a ^ {1 / 3} - \sqrt {a ^ {2 / 3} - (4 \mu_ {k}) ^ {1 / 3}}\right) \stackrel {(3)} {>} y _ {\mu_ {k}} ^ {*}
$$
Here (1) and (3) are due to Lemma 3, and (2) follows from  $\sqrt{1 - x} \geq 1 - x$  for  $0 \leq x \leq 1$ . Then it implies that  $(x_{\mu_k}^*, y_{\mu_k}^*)$  is in the region  $\{(x, y) \mid x_{\mu_{k+1}}^{**} < x \leq a, 0 \leq y < y_{\mu_{k+1}}^{**}\}$ . By Lemma 2, the
1 procedure will converge to  $(x_{\mu_{k + 1}}^{*},y_{\mu_{k + 1}}^{*})$  . Finally, from Theorems 5 and 6, if  $\lim_{k\to \infty}\mu_k = 0$  then  $\lim_{k\to \infty}x_{\mu_k}^* = a,\lim_{k\to \infty}y_{\mu_k}^* = 0$  , thus, converging to the global optimum, i.e.,
$$
\lim  _ {k \to \infty} W _ {\mu_ {k}} = W _ {\mathsf {G}}.
$$
5.2 Discrete case: Gradient Descent
In Algorithms 2 and 4, gradient flow is employed to locate the next stationary points, which is not practically feasible. A viable alternative is to execute Algorithm 2, replacing the gradient flow with gradient descent. Now, at every iteration  $k$ , Algorithm 6 uses gradient descent to output  $W_{\mu_k,\epsilon_k}$ , a  $\epsilon_k$  stationary point of  $g_{\mu_k}$ , initialized at  $W_{\mu_{k - 1},\epsilon_{k - 1}}$ , and a step size of  $\eta_{k} = 1 / (\mu_{k}(a^{2} + 1) + 3a^{2})$ . The tolerance parameter  $\epsilon_k$  can significantly influence the behavior of the algorithm and must be controlled for different iterations. A convergence guarantee is established via a simplified theorem presented here. A more formal version of the theorem and a comprehensive description of the algorithm (i.e., Algorithm 6) can be found in Appendix C.
Theorem 2 (Informal). For any  $\varepsilon_{\mathrm{dist}} > 0$ , set  $\mu_0$  satisfy a mild condition, and use updating rule  $\epsilon_k = \min \{\beta a\mu_k,\mu_k^{3 / 2}\}$ ,  $\mu_{k + 1} = (2\mu_k^2)^{2 / 3}\frac{(a + \epsilon_k / \mu_k)^{2 / 3}}{(a - \epsilon_k / \mu_k)^{4 / 3}}$ , and let  $K\equiv K(\mu_0,a,\varepsilon_{\mathrm{dist}})\in O\left(\ln \frac{\mu_0}{a\varepsilon_{\mathrm{dist}}}\right)$ . Then, for any initialization  $W_{0}$ , following the updated procedure above for  $k = 0,\ldots ,K$ , we have:
$$
\| W _ {\mu_ {k}, \epsilon_ {k}} - W _ {\mathrm {G}} \| _ {2} \leq \varepsilon_ {\text {d i s t}}
$$
that is,  $W_{\mu_k,\epsilon_k}$  is  $\varepsilon_{\mathrm{dist}}$ -close in Frobenius norm to global optimum  $W_{\mathbb{G}}$ . Moreover, the total number of gradient descent steps is upper bounded by  $O\left(\left(\mu_0a^2 + a^2 + \mu_0\right)\left(\frac{1}{a^6} + \frac{1}{\varepsilon_{\mathrm{dist}}^6}\right)\right)$ .
We conducted experiments to verify that Algorithms 2 and 4 both converge to the global minimum of (7). Our purpose is to illustrate two main points: First, we compare our updating scheme as given in Line 3 of Algorithm 2 against a faster-decreasing updating scheme for  $\mu_{k}$ . In Figure 4 we illustrate how a naive faster decrease of  $\mu$  can lead to spurious a local minimum. Second, in Figure 5, we show that regardless of the initialization, Algorithms 2 and 4 always return the global minimum. In the supplementary material, we provide additional experiments where the gradient flow is replaced with gradient descent. For more details, please refer to Appendix F.
Figure 4: Trajectory of the gradient flow path for two different update rules for  $\mu_{k}$  with same initialization and  $\mu_0$ . Here, "good scheduling" uses Line 3 of Algorithm 2, while "bad scheduling" uses a faster decreasing scheme for  $\mu_{k}$  which leads the path to a spurious local minimum.
K. B. was supported by NSF under Grant # 2127309 to the Computing Research Association for the CIFellows 2021 Project. B.A. was supported by NSF IIS-1956330, NIH R01GM140467, and the Robert H. Topel Faculty Research Fund at the University of Chicago Booth School of Business. This work was done in part while B.A. was visiting the Simons Institute for the Theory of Computing. P.R. was supported by ONR via N000141812861, and NSF via IIS-1909816, IIS-1955532, IIS-2211907. We are also grateful for the support of the University of Chicago Research Computing Center for assistance with the calculations carried out in this work.
(a) Random Initialization 1
(b) Random Initialization 2
(c) Random Initialization 3
Figure 5: Trajectory of the gradient flow path with the different initializations. We observe that under a proper scheduling for  $\mu_{k}$ , they all converge to the global minimum.
[1] Bello, K., Aragam, B. and Ravikumar, P. [2022], DAGMA: Learning dags via M-matrices and a log-determinant acyclicity characterization, in 'Advances in Neural Information Processing Systems'.
[2] Bertsekas, D. P. [1997], 'Nonlinear programming', Journal of the Operational Research Society 48(3), 334-334.
[3] Bhojanapalli, S., Neyshabur, B. and Srebro, N. [2016], 'Global optimality of local search for low rank matrix recovery', Advances in Neural Information Processing Systems 29.
[4] Boumal, N., Voroninski, V. and Bandeira, A. [2016], 'The non-convex burer-monteiro approach works on smooth semidefinite programs', Advances in Neural Information Processing Systems 29.
[5] Boyd, S., Boyd, S. P. and Vandenberghe, L. [2004], Convex optimization, Cambridge university press.
[6] Brutzkus, A. and Globerson, A. [2017], Globally optimal gradient descent for a convnet with gaussian inputs, in 'International conference on machine learning', PMLR, pp. 605-614.
[7] Chen, W., Drton, M. and Wang, Y. S. [2019], 'On causal discovery with an equal-variance assumption', Biometrika 106(4), 973-980.
[8] Chickering, D. M. [1996], Learning Bayesian networks is NP-complete, in 'Learning from data', Springer, pp. 121-130.
[9] Chickering, D. M. [2003], 'Optimal structure identification with greedy search', JMLR 3, 507554.
[10] Chickering, D. M., Heckerman, D. and Meek, C. [2004], 'Large-sample learning of Bayesian networks is NP-hard', Journal of Machine Learning Research 5, 1287-1330.
[11] Choromanska, A., Henaff, M., Mathieu, M., Arous, G. B. and LeCun, Y. [2015], The loss surfaces of multilayer networks, in 'Artificial intelligence and statistics', PMLR, pp. 192-204.
[12] Cussens, J. [2012], 'Bayesian network learning with cutting planes', arXiv preprint arXiv:1202.3713.
[13] De Sa, C., Re, C. and Olukotun, K. [2015], Global convergence of stochastic gradient descent for some non-convex matrix problems, in 'International conference on machine learning', PMLR, pp. 2332-2341.
[14] Dontchev, A. L., Rockafellar, R. T. and Rockafellar, R. T. [2009], Implicit functions and solution mappings: A view from variational analysis, Vol. 11, Springer.
Algorithm 4: Find a path  $\{W_{\mu_k}\}$  via a particular scheduling for  $\mu_{k}$  when  $a$  is unknown.
Input:  $\mu_0\in \left[\frac{a^2}{4(a^2 + 1)^3},\frac{a^2}{4}\right),\varepsilon >0$  Output:  $\{W_{\mu_k}\}_{k = 0}^{\infty}$    
1  $\widehat{a}\gets \sqrt{4(\mu_0 + \varepsilon)}$  //  $\forall \varepsilon \geq 0$  s.t.  $\widehat{a} <  a$    
2  $W_{\mu_0}\gets \mathrm{GradientFlow}(g_{\mu_0},0)$    
3 for  $k = 1,2,\ldots$  do   
4 Let  $\mu_{k + 1}\in \left[(2 / \widehat{a})^{2 / 3}\mu_k^{4 / 3},\mu_k\right)$    
5  $W_{\mu_{k + 1}}\gets \mathrm{GradientFlow}(g_{\mu_{k + 1}},W_{\mu_k})$    
6 end   
7 return  $\{W_{\mu_k}\}_{k = 0}^{\infty}$
A Practical Implementation of Algorithm 2
We present a practical implementation of our homotopy algorithm in Algorithm 4. The updating scheme for  $\mu_{k}$  is now independent of the parameter  $a$ , but as presented, the initialization for  $\mu_0$  still depends on  $a$ . This is for the following reason: It is possible to make the updating scheme independent of  $a$  without imposing any additional assumptions on  $a$ , as evidenced by Lemma 4 below. The initialization for  $\mu_0$ , however, is trickier, and we must consider two separate cases:
1. No assumptions on  $a$ . In this case, if  $a$  is too small, then the problem becomes harder and the initial choice of  $\mu_0$  matters.
2. Lower bound on  $a$ . If we are willing to accept a lower bound on  $a$ , then there is an initialization for  $\mu_0$  that does not depend on  $a$ .
In Corollary 1, we illustrate this last point with the additional condition that  $a > \sqrt{5 / 27}$ . This essentially amounts to an assumption on the minimum signal, and is quite standard in the literature on learning SEM.
Lemma 4. Under the assumption  $\frac{a^2}{4(a^2 + 1)^3} \leq \mu_0 < \frac{a^2}{4}$ , the Algorithm 4 outputs the global optimal solution to (6), i.e.
It turns out that the assumption in Lemma 4 is not overly restrictive, as there exist pre-determined sequences of  $\{\mu_k\}_{k=0}^{\infty}$  that can ensure the effectiveness of Algorithm 4 for any values of  $a$  greater than a certain threshold.
B From Population Loss to Empirical Loss
The transformation from population loss to empirical can be thought from two components. First, with a given empirical loss, Algorithms 2 and 3 still achieve the global minimum,  $W_{\mathrm{G}}$ , of problem 6, but now the output from the Algorithm is an empirical estimator  $\hat{a}$ , rather than ground truth  $a$ , Theorem 1 and Corollary 1 would continue to be valid. Second, the global optimum,  $W_{\mathrm{G}}$ , of the empirical loss possess the same DAG structure as the underlying  $W_{*}$ . The finite-sample findings in Section 5 (specifically, Lemmas 18 and 19) of Loh and Buhlmann [31], which offer sufficient conditions on the sample size to ensure that the DAG structures of  $W_{\mathrm{G}}$  and  $W_{*}$  are identical.
C From Continuous to Discrete: Gradient Descent
Previously, gradient flow was employed to address the intermediate problem (7), a method that poses implementation challenges in a computational setting. In this section, we introduce Algorithm 6 that leverages gradient descent to solve (7) in each iteration. This adjustment serves practical considerations. We start with the convergence results of Gradient Descent.
Definition 1.  $f$  is  $L$ -smooth, if  $f$  is differentiable and  $\forall x, y \in \operatorname{dom}(f)$  such that  $\| \nabla f(x) - \nabla f(y) \|_2 \leq L \| x - y \|_2$ .
Algorithm 5: Gradient Descent  $(f,\eta ,W_0,\epsilon)$
Input: function  $f$  , step size  $\eta$  , initial point  $W_{0}$  , tolerance  $\epsilon$  Output:  $W_{t}$    
1  $t\gets 0$    
2 while  $\| \nabla f(W_t)\| _2 > \epsilon$  do   
3  $W_{t + 1}\leftarrow W_t - \eta \nabla f(W_t)$    
4  $t\gets t + 1$    
5 end
Algorithm 6: Homotopy algorithm using gradient descent for solving (1).
Input: Initial  $W_{-1} = W(x_{-1},y_{-1})$ $\mu_0\in \left[\frac{a^2}{4(a^2 + 1)^3}\frac{(1 + \beta)^4}{(1 - \beta)^2},\frac{a^2}{4}\frac{(1 - \delta)^3(1 - \beta)^4}{(1 + \beta)^2}\right),$ $\eta_0 = \frac{1}{\mu_0(a^2 + 1) + 3a^2},\epsilon_0 = \min \{\beta a\mu_0,\mu_0^{3 / 2}\}$  Output:  $\{W_{\mu_k}\}_{k = 0}^{\infty}$    
1  $W_{\mu_0,\epsilon_0}\gets$  Gradient Descent  $(g_{\mu_0},\eta_0,W_{-1},\epsilon_0)$    
for  $k = 1,2,\ldots$  do   
Let  $\mu_{k} = (2\mu_{k - 1}^{2})^{2 / 3}\frac{(a + \epsilon_{k - 1} / \mu_{k - 1})^{2 / 3}}{(a - \epsilon_{k - 1} / \mu_{k - 1})^{4 / 3}}$    
Let  $\eta_{k} = \frac{1}{\mu_{k}(a^{2} + 1) + 3a^{2}}$    
Let  $\epsilon_{k} = \min \{\beta a\mu_{k},\mu_{k}^{3 / 2}\}$ $W_{\mu_k,\epsilon_k}\gets$  Gradient Descent  $(g_{\mu_k},\eta_k,W_{\mu_{k - 1}},\epsilon_k)$    
end
Theorem 3 (Nesterov et al. 33). If function  $f$  is  $L$ -smooth, then Gradient Descent (Algorithm 5) with step size  $\eta = 1 / L$ , finds an  $\epsilon$ -first-order stationary point (i.e.  $\| \nabla f(x) \|_2 \leq \epsilon$ ) in  $2L(f(x^0) - f^*) / \epsilon^2$  iterations.
One of the pivotal factors influencing the convergence of gradient descent is the selection of the step size. Theorem 3 selects a step size  $\eta = \frac{1}{L}$ . Therefore, our initial step is to determine the smoothness of  $g_{\mu}(W)$  within our region of interest,  $A = \{0 \leq x \leq a, 0 \leq y \leq \frac{a}{a^2 + 1}\}$ .
Lemma 5. Consider the function  $g_{\mu}(W)$  as defined in Equation 7 within the region  $A = \{0 \leq x \leq a, 0 \leq y \leq \frac{a}{a^2 + 1}\}$ . It follows that for all  $\mu \geq 0$ , the function  $g_{\mu}(W)$  is  $\mu(a^2 + 1) + 3a^2$ -smooth.
Since gradient descent is limited to identifying the  $\epsilon$  stationary point of the function. Thus, we study the gradient of  $g_{\mu}(W) = \mu f(W) + h(W)$ , i.e.  $\nabla g_{\mu}(W)$  has the following form
$$
\nabla g _ {\mu} (W) = \left( \begin{array}{c} \mu (x - a) + y ^ {2} x \\ \mu (a ^ {2} + 1) y - a \mu + y x ^ {2} \end{array} \right)
$$
As gradient descent is limited to identifying the  $\epsilon$  stationary point of the function, we, therefore, focus on  $\| g_{\mu}(W)\| _2\leq \epsilon$ . This can be expressed in the subsequent manner:
$$
\| \nabla g _ {\mu} (W) \| _ {2} \leq \epsilon \Rightarrow - \epsilon \leq \mu (x - a) + y ^ {2} x <   \epsilon \quad \text {a n d} - \epsilon \leq \mu \left(a ^ {2} + 1\right) y - a \mu + y x ^ {2} \leq \epsilon
$$
$$
\{(x, y) \mid \| \nabla g _ {\mu} (W) \| _ {2} \leq \epsilon \} \subseteq \{(x, y) \mid \frac {\mu a - \epsilon}{\mu + y ^ {2}} \leq x \leq \frac {\mu a + \epsilon}{\mu + y ^ {2}}, \frac {\mu a - \epsilon}{x ^ {2} + \mu (a ^ {2} + 1)} \leq y \leq \frac {\mu a + \epsilon}{x ^ {2} + \mu (a ^ {2} + 1)} \}
$$
Here we denote such region as  $A_{\mu ,\epsilon}$
$$
A _ {\mu , \epsilon} = \{(x, y) \mid \frac {\mu a - \epsilon}{\mu + y ^ {2}} \leq x \leq \frac {\mu a + \epsilon}{\mu + y ^ {2}}, \frac {\mu a - \epsilon}{x ^ {2} + \mu \left(a ^ {2} + 1\right)} \leq y \leq \frac {\mu a + \epsilon}{x ^ {2} + \mu \left(a ^ {2} + 1\right)} \} \tag {10}
$$
Figure 6 and 7 illustrate the region  $A_{\mu ,\epsilon}$
Figure 6: An example of  $A_{\mu, \epsilon}$  is depicted for  $a = 0.6$ ,  $\mu = 0.009$ , and  $\epsilon = 0.00055$ . The yellow region signifies  $\epsilon$  stationary points, denoted as  $A_{\mu, \epsilon}$  and defined by Equation (10).  $A_{\mu, \epsilon}$  is the disjoint union of  $A_{\mu, \epsilon}^{1}$  and  $A_{\mu, \epsilon}^{2}$ , which are defined by Equations (21) and (22), respectively.
Figure 7: Here is a localized illustration of  $A_{\mu, \epsilon}$  that includes the point  $(x_{\mu}^{*}, y_{\mu}^{*})$ . This region, referred to as  $A_{\mu, \epsilon}^{1}$ , is defined in Equation (21).
Given that the gradient descent can only locate  $\epsilon$  stationary points within the region  $A_{\mu, \epsilon}$  during each iteration, the boundary of  $A_{\mu, \epsilon}$  becomes a critical component of our analysis. To facilitate clear presentation, it is essential to establish some pertinent notations.
$$
\left\{ \begin{array}{l} x = \frac {\mu a}{\mu + y ^ {2}} \\ y = \frac {\mu a}{\mu \left(a ^ {2} + 1\right) + x ^ {2}} \end{array} \right. \tag {11b}
$$
If the system of equations yields only a single solution, we denote this solution as  $(x_{\mu}^{*},y_{\mu}^{*})$  If it yields two solutions, these solutions are denoted as  $(x_{\mu}^{*},y_{\mu}^{*}),(x_{\mu}^{**},y_{\mu}^{**})$  , with  $x_{\mu}^{**} <   x_{\mu}^{*}$  In the event that there are three distinct solutions to the system of equations, these solutions are denoted as  $(x_{\mu}^{*},y_{\mu}^{*}),(x_{\mu}^{**},y_{\mu}^{**}),(x_{\mu}^{***},y_{\mu}^{***})$  , where  $x_{\mu}^{**} <   x_{\mu}^{**} <   x_{\mu}^{*}$
$$
\left\{ \begin{array}{l} x = \frac {\mu a - \epsilon}{\mu + y ^ {2}} \\ y = \frac {\mu a + \epsilon}{\mu \left(a ^ {2} + 1\right) + x ^ {2}} \end{array} \right. \tag {12a}
$$
If the system of equations yields only a single solution, we denote this solution as  $(x_{\mu ,\epsilon}^{*},y_{\mu ,\epsilon}^{*})$  If it yields two solutions, these solutions are denoted as  $(x_{\mu ,\epsilon}^{*},y_{\mu ,\epsilon}^{*}),(x_{\mu ,\epsilon}^{**},y_{\mu ,\epsilon}^{**})$  , with  $x_{\mu ,\epsilon}^{**} <   x_{\mu ,\epsilon}^{*}$  . In the event that there are three distinct solutions to the system of equations, these solutions are denoted as  $(x_{\mu ,\epsilon}^{*},y_{\mu ,\epsilon}^{*}),(x_{\mu ,\epsilon}^{**},y_{\mu ,\epsilon}^{**}),(x_{\mu ,\epsilon}^{***},y_{\mu ,\epsilon}^{***})$  , where  $x_{\mu ,\epsilon}^{***} <   x_{\mu ,\epsilon}^{**} <   x_{\mu ,\epsilon}^{*}$
$$
\left\{ \begin{array}{l} x = \frac {\mu a + \epsilon}{\mu + y ^ {2}} \\ y = \frac {\mu a - \epsilon}{\mu \left(a ^ {2} + 1\right) + x ^ {2}} \end{array} \right. \tag {13a}
$$
If the system of equations yields only a single solution, we denote this solution as  $(x_{\mu ,\epsilon_{-}}^{*},y_{\mu ,\epsilon_{-}}^{*})$  . If it yields two solutions, these solutions are denoted as  $(x_{\mu ,\epsilon_{-}}^{*},y_{\mu ,\epsilon_{-}}^{*}),(x_{\mu ,\epsilon_{-}}^{**},y_{\mu ,\epsilon_{-}}^{**})$  , with  $x_{\mu ,\epsilon_{-}}^{**} <   x_{\mu ,\epsilon_{-}}^{*}$  . In the event that there are three distinct solutions to the system of equations, these solutions are denoted as  $(x_{\mu ,\epsilon_{-}}^{*},y_{\mu ,\epsilon_{-}}^{*}),(x_{\mu ,\epsilon_{-}}^{**},y_{\mu ,\epsilon_{-}}^{**}),(x_{\mu ,\epsilon_{-}}^{***},y_{\mu ,\epsilon_{-}}^{***})$  , where  $x_{\mu ,\epsilon_{-}}^{**} <   x_{\mu ,\epsilon_{-}}^{**} <   x_{\mu ,\epsilon_{-}}^{*}$
Remark 4. There always exists at least one solution to the above system of equations. When  $\mu$  is sufficiently small, the above system of equations always yields three solutions, as demonstrated in Theorem 5, and Theorem 9.
The parameter  $\epsilon$  can substantially influence the behavior of the systems of equations (12a),(12b) and (13a),(13b). A crucial consideration is to ensure that  $\epsilon$  remains adequately small. To facilitate this, we introduce a new parameter,  $\beta$ , whose specific value will be determined later. At this stage, we merely require that  $\beta$  should lie within the interval  $(0, 1)$ . We further impose a constraint on  $\epsilon$  to satisfy the following inequality:
$$
\epsilon \leq \beta a \mu \tag {14}
$$