Title: Global Optimality in Bivariate Gradient-based DAG Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: global optimization

nonconvex optimization

graphical models

directed acyclic graphs

structure learning
Abstract: Recently, a new class of non-convex optimization problems motivated by the statistical problem of learning an acyclic directed graphical model from data has attracted significant interest. While existing work uses standard first-order optimization schemes to solve this problem, proving the global optimality of such approaches has proven elusive. The difficulty lies in the fact that unlike other non-convex problems in the literature, this problem is not "benign", and possesses multiple spurious solutions that standard approaches can easily get trapped in. In this paper, we prove that a simple path-following optimization scheme globally converges to the global minimum of the population loss in the bivariate setting.

Global Optimality in Bivariate Gradient-based DAG Learning

Abstract

Recently, a new class of non-convex optimization problems motivated by the statistical problem of learning an acyclic directed graphical model from data has attracted significant interest. While existing work uses standard first-order optimization schemes to solve this problem, proving the global optimality of such approaches has proven elusive. The difficulty lies in the fact that unlike other non-convex problems in the literature, this problem is not 'benign', and possesses multiple spurious solutions that standard approaches can easily get trapped in. In this paper, we prove that a simple path-following optimization scheme globally converges to the global minimum of the population loss in the bivariate setting.

1 Introduction

Over the past decade, non-convex optimization has become a major topic of research within the machine learning community, in part due to the successes of training large-scale models with simple first-order methods such as gradient descent-along with their stochastic and accelerated variantsin spite of the non-convexity of the loss function. A large part of this research has focused on characterizing which problems have benign loss landscapes that are amenable to the use of gradientbased methods, i.e., there are no spurious local minima, or they can be easily avoided. By now, several theoretical results have shown this property for different non-convex problems such as: learning a two hidden unit ReLU network [48], learning (deep) over-parameterized quadratic neural networks [43, 27], low-rank matrix recovery [19, 13, 3], learning a two-layer ReLU network with a single non-overlapping convolutional filter [6], semidefinite matrix completion [4, 20], learning neural networks for binary classification with the addition of a single special neuron [30], and learning deep networks with independent ReLU activations [26, 11], to name a few.

Recently, a new class of non-convex optimization problems due to Zheng et al. [51] have emerged in the context of learning the underlying structure of a structural equation model (SEM) or Bayesian network. This underlying structure is typically represented by a directed acyclic graph (DAG), which makes the learning task highly complex due to its combinatorial nature. In general, learning DAGs is well-known to be NP-complete [8, 10]. The key innovation in Zheng et al. [51] was the introduction of a differentiable function h , whose level set at zero exactly characterizes DAGs. Thus, replacing the challenges of combinatorial optimization by those of non-convex optimization. Mathematically, this class of non-convex problems take the following general form:

where Θ ∈ R l represents the model parameters, f : R l → R is a (possibly non-convex) smooth loss function (sometimes called a score function ) that measures the fitness of Θ , and h : R d × d → [0 , ∞ )

is a smooth non-convex function that takes the value of zero if and only if the induced weighted adjacency matrix of d nodes, W (Θ) , corresponds to a DAG.

Given the smoothness of f and h , problem (1) can be solved using off-the-shelf nonlinear solvers, which has driven a series of remarkable developments in structure learning for DAGs. Multiple empirical studies have demonstrated that global or near-global minimizers for (1) can often be found in a variety of settings, such as linear models with Gaussian and non-Gaussian noises [e.g., 51, 34, 1], and non-linear models, represented by neural networks, with additive Gaussian noises [e.g., 29, 52, 49, 1]. The empirical success for learning DAGs via (1), which started with the NOTEARS method of Zheng et al. [51], bears a resemblance to the advancements in deep learning, where breakthroughs like AlexNet significantly boosted the field's recognition, even though there were notable successes before it.

Importantly, the reader should note that the majority of applications in ML consist of solving a single unconstrained non-convex problem. In contrast, the class of problems (1) contains a non-convex constraint. Thus, researchers have considered some type of penalty method such as the augmented Lagrangian [51, 52], quadratic penalty [35], and a log-barrier [1]. In all cases, the penalty approach consists in solving a sequence of unconstrained non-convex problems, where the constraint is enforced progressively [see e.g. 2, for background]. In this work, we will consider the following form of penalty:

It was shown by Bello et al. [1] that due to the invexity property of h , 2 solutions to (2) will converge to a DAG as µ k → 0 . However, no guarantees on local/global optimality were given.

With the above considerations in hand, one is inevitably led to ask the following questions:

(i) Are the loss landscapes g µ k (Θ) benign for different µ k ?

(ii) Is there a (tractable) solution path { Θ k } that converges to a global minimum of (1) ?

Due to the NP-completeness of learning DAGs, one would expect the answer to (i) to be negative in its most general form. Moreover, it is known from the classical theory of constrained optimization [e.g. 2] that if we can exactly and globally optimize (1) for each µ k , then the answer to (ii) is affirmative. This is not a practical algorithm, however, since the problem (1) is nonconvex. Thus we seek a solution path that can be tractably computed in practice, e.g. by gradient descent.

In this work, we focus on perhaps the simplest setting where interesting phenomena take place. That is, a linear SEM with two nodes (i.e., d = 2 ), f is the population least squared loss (i.e., f is convex), and Θ k is defined via gradient flow with warm starts. More specifically, we consider the case where Θ k is obtained by following the gradient flow of g µ k with initial condition Θ k -1 .

Under this setting, to answer (i), it is easy to see that for a large enough µ k , the convex function f dominates and we can expect a benign landscape, i.e., a (almost) convex landscape. Similarly, when µ k approaches zero, the invexity of h kicks in and we could expect that all stationary points are (near) global minimizers. 3 That is, at the extremes µ k →∞ and µ k → 0 , the landscapes seem well-behaved, and the reader might wonder if it follows that for any µ k ∈ [0 , ∞ ) the landscape is well-behaved. We answer the latter in the negative and show that there always exists a τ > 0 where the landscape of g µ k is non-benign for any µ k < τ , namely, there exist three stationary points: i) A saddle point, ii) A spurious local minimum, and iii) The global minimum. In addition, each of these stationary points have wide basins of attractions, thus making the initialization of the gradient flow for g µ k crucial. Finally, we answer (ii) in the affirmative and provide an explicit scheduling for µ k that guarantees the asymptotic convergence of Θ k to the global minimum of (1). Moreover, we show that this scheduling cannot be arbitrary as there exists a sequence of { µ k } that leads { Θ k } to a spurious local minimum.

Overall, we establish the first set of results that study the optimization landscape and global optimality for the class of problems (1). We believe that this comprehensive analysis in the bivariate case provides a valuable starting point for future research in more complex settings.

(a) Contour plot

0.

0.1

0.2

0.3

0.4

0.5

0.6

x

0.

0. 5

0.1

0.15

0.2

0.25

0.3

0.35

0.4

y

g

(

W

)

(

x

*

,

y

*

)

(

x

* *

,

y

* *

)

(

x

* * *

,

y

* * *

)

g

(

W

)

x

=0

g

(

W

)

y

=0

(b) Stationary points

1.1 Our Contributions

More specifically, we make the following contributions:

We present a homotopy-based optimization scheme (Algorithm 2) to find global minimizers of the program (1) by iteratively decreasing the penalty coefficient according to a given schedule. Gradient flow is used to find the stationary points of (2) at each step, starting from the previous solution.

We prove that Algorithm 2 converges globally (i.e. regardless of initialization for W ) to the global minimum (Theorem 1).

We show that the non-convex program (1) is indeed non-benign, and naïve implementation of black-box solvers are likely to get trapped in a bad local minimum. See Figure 1 (a).

Experimental results verify our theory, consistently recovering the global minimum of (1), regardless of initialization or initial penalty value. We show that our algorithm converges to the global minimum while naïve approaches can get stuck.

The analysis consists of three main parts: First, we explicitly characterize the trajectory of the stationary points of (2). Second, we classify the number and type of all stationary points (Lemma 1) and use this to isolate the desired global minimum. Finally, we apply Lyapunov analysis to identify the basin of attraction for each stationary point, which suggests a schedule for the penalty coefficient that ensures that the gradient flow is initialized within that basin at the previous solution.

1.2 Related Work

The class of problems (1) falls under the umbrella of score-based methods, where given a score function f , the goal is to identify the DAG structure with the lowest score possible [9, 22]. We shall note that learning DAGs is a very popular structure model in a wide range of domains such as biology [40], genetics [50], and causal inference [44, 39], to name a few.

Score-based methods that consider the combinatorial constraint. Given the ample set of scorebased methods in the literature, we briefly mention some classical works that attempt to optimize f by considering the combinatorial DAG constraint. In particular, we have approximate algorithms such as the greedy search method of Chickering et al. [10], order search methods [45, 41, 38], the LP-relaxation method of Jaakkola et al. [24], and the dynamic programming approach of Loh and Bühlmann [31]. There are also exact methods such as GOBNILP [12] and Bene [42], however, these algorithms only scale up to ≈ 30 nodes.

Score-based methods that consider the continuous non-convex constraint h . The following works are the closest to ours since they attempt to solve a problem in the form of (1). Most of these developments either consider optimizing different score functions f such as ordinary least squares [51, 52], the log-likelihood [29, 34], the evidence lower bound [49], a regret function [53]; or consider different differentiable characterizations of acyclicity h [49, 1]. However, none of the aforementioned works provide any type of optimality guarantee. Few studies have examined the optimization intricacies of problem (1). Wei et al. [47] investigated the optimality issues and provided local optimality guarantees under the assumption of convexity in the score f and linear models. On the other hand, Ng et al. [35] analyzed the convergence to (local) DAGs of generic methods for solving nonlinear constrained problems, such as the augmented Lagrangian and quadratic penalty methods. In contrast to both, our work is the first to study global optimality and the loss landscapes of actual methods used in practice for solving (1).

Bivariate causal discovery. Even though in a two-node model the discrete DAG constraint does not pose a major challenge, the bivariate setting has been subject to major research in the area of causal discovery. See for instance [36, 16, 32, 25] and references therein.

Penalty and homotopy methods. There exist classical global optimality guarantees for the penalty method if f and h were convex functions, see for instance [2, 5, 37]. However, to our knowledge, there are no global optimality guarantees for general classes of non-convex constrained problems, let alone for the specific type of non-convex functions h considered in this work. On the other hand, homotopy methods (also referred to as continuation or embedding methods) are in many cases capable of finding better solutions than standard first-order methods for non-convex problems, albeit they typically do not come with global optimality guarantees either. When homotopy methods come with global optimality guarantees, they are commonly computationally more intensive as it involves discarding solutions, thus, closely resembling simulated annealing methods, see for instance [15]. Authors in [21] characterize a family of non-convex functions where a homotopy algorithm provably converges to a global optimum. However, the conditions for such family of non-convex functions are difficult to verify and are very restrictive; moreover, their homotopy algorithm involves Gaussian smoothing, making it also computationally more intensive than the procedure we study here. Other examples of homotopy methods in machine learning include [7, 18, 46, 17, 23], in all these cases, no global optimality guarantees are given.

2 Preliminaries

The objective f we consider can be easily written down as follows:

where X ∈ R 2 is a random vector and W ∈ R 2 × 2 . Although not strictly necessary for the developments that follow, we begin by introducing the necessary background on linear SEM that leads to this objective and the resulting optimization problem of interest.

The bivariate model. Let X = ( X 1 , X 2 ) ∈ R 2 denote the random variables in the model, and let N = ( N 1 , N 2 ) ∈ R 2 denote a vector of independent errors. Then a linear SEM over X is defined as X = W glyph[latticetop] ∗ X + N , where W ∗ ∈ R 2 × 2 is a weighted adjacency matrix encoding the coefficients in the linear model. In order to represent a valid Bayesian network for X [see e.g. 39, 44, for details], the matrix W ∗ must be acyclic: More formally, the weighted graph induced by the adjacency matrix W ∗ must be a DAG. This (non-convex) acyclicity constraint represents the major computational hurdle that must overcome in practice (cf. Remark 1).

The goal is to recover the matrix W ∗ from the random vector X . Since W ∗ is acyclic, we can assume the diagonal of W ∗ is zero (i.e. no self-loops). Thus, under the bivariate linear model, it then suffices to consider two parameters x and y that define the matrix of parameters 4

For notational simplicity, we will use f ( W ) and f ( x, y ) interchangeably, similarly for h ( W ) and h ( x, y ) . Without loss of generality, we write the underlying parameter as

In general, we only require N i to have finite mean and variance, hence we do not assume Gaussianity. We assume that Var[ N 1 ] = Var[ N 2 ] , and for simplicity, we consider E [ N ] = 0 and Cov [ N ] = I , where I denotes the identity matrix. Finally, in the sequel we assume w.l.o.g. that a > 0 .

The population least squares. In this work, we consider the population squared loss defined by (3). If we equivalently write f in terms of x and y , then we have: f ( W ) = ((1 -ay ) 2 + y 2 +( a -x ) 2 +1) / 2 . In fact, the population loss can be substituted with empirical loss. In such a case, our algorithm can still attain the global minimum, W G , of problem (6). However, the output W G will serve as an empirical estimation of W ∗ . An in-depth discussion on this topic can be found in Appendix B

The non-convex function h . We use the continuous acyclicity characterization of Yu et al. [49], i.e., h ( W ) = Tr(( I + 1 d W ◦ W ) d ) -d , where ◦ denotes the Hadamard product. Then, for the bivariate case, we have h ( W ) = x 2 y 2 / 2 . We note that the analysis presented in this work is not tailored to this version of h , that is, we can use the same techniques used throughout this work for other existing formulations of h , such as the trace of the matrix exponential [51], and the log-det formulation [1]. Nonetheless, here we consider that the polynomial formulation of Yu et al. [49] is more amenable for the analysis.

Remark 2. Our restriction to the bivariate case highlights the simplest setting in which this problem exhibits nontrivial behaviour. Extending our analysis to higher dimensions remains a challenging future direction, however, we emphasize that even in two-dimensions this problem is nontrivial. Our approach is similar to that taken in other parts of the literature that started with simple cases (e.g. single-neuron models in deep learning).

Remark 3. It is worth noting that our choice of the population least squares is not arbitrary. Indeed, for linear models with identity error covariance, such as the model considered in this work, it is known that the global minimizer of the population squared loss is unique and corresponds to the underlying matrix W ∗ . See Theorem 7 in [31].

Gluing all the pieces together, we arrive to the following version of (1) for the bivariate case:

Moreover, for any µ ≥ 0 , we have the corresponding version of (2) expressed as:

To conclude this section, we present a visualization of the landscape of g µ ( x, y ) in Figure 1 (a), for a = 0 . 5 and µ = 0 . 005 . We can clearly observe the non-benign landscape of g µ , i.e., there exists a spurious local minimum, a saddle point, and the global minimum. In particular, we can see that the basin of attraction of the spurious local minimum is comparable to that of the global minimum, which is problematic for a local algorithm such as the gradient flow (or gradient descent) as it can easily get trapped in a local minimum if initialized in the wrong basin.

which implies

Algorithm 1: GradientFlow( f, z 0 )

1: set z (0) = z 0

2: d dt z ( t ) = -∇ f ( z ( t ))

3: return lim t →∞ z ( t )

Algorithm 2: Homotopy algorithm for solving (1).

3 A Homotopy-Based Approach and Its Convergence to the Global Optimum

To fix notation, let us write W k := W µ k := ( 0 x µ k y µ k 0 ) . and let W G denote the global minimizer of (6). In this section, we present our main result, which provides conditions under which solving a series of unconstrained problems (7) with first-order methods will converge to the global optimum W G of (6), in spite of facing non-benign landscapes. Recall that from Remark 3, we have that W G = ( 0 a 0 0 ) . Since we use gradient flow path to connect W µ k and W µ k +1 , we specify this path in Procedure 1 for clarity. Although the theory here assumes continuous-time gradient flow with t →∞ , see Section 5 for an iteration complexity analysis for (discrete-time) gradient descent, which is a straightforward consequence of the continuous-time theory.

In Algorithm 2, we provide an explicit regime of initialization for the homotopy parameter µ 0 and a specific scheduling for µ k such that the solution path found by Algorithm 2 will converge to the global optimum of (6). This is formally stated in Theorem 1, whose proof is given in Section 5.

Theorem 1. For any initialization W 0 and a ∈ R , the solution path provided in Algorithm 2 converges to the global optimum of (6) , i.e.,

A few observations regarding Algorithm 2: Observe that when the underlying model parameter a glyph[greatermuch] 0 , the regime of initialization for µ 0 is wider; on the other hand, if a is closer to zero then the interval for µ 0 is much narrower. As a concrete example, if a = 2 then it suffices to have µ 0 ∈ [0 . 008 , 1) ; whereas if a = 0 . 1 then the regime is about µ 0 ∈ [0 . 0089 , 0 . 01) . This matches the intuition that for a 'stronger' value of a it should be easier to detect the right direction of the underlying model. Second, although in Line 3 we set µ k in a specific manner, it actually suffices to have

We simply chose a particular expression from this interval for clarity of presentation; see the proof in Section 5 for details.

As presented, Algorithm 2 is of theoretical nature in the sense that the initialization for µ 0 and the decay rate for µ k in Line 3 depend on the underlying parameter a , which in practice is unknown. In Algorithm 3, we present a modification that is independent of a and W ∗ . By assuming instead a lower bound on a , which is a standard assumption in the literature, we can prove that Algorithm 3 also converges to the global minimum:

Corollary 1. Initialize µ 0 = 1 27 . If a > √ 5 / 27 then for any initialization W 0 , Algorithm 3 outputs the global optimal solution to (6) , i.e.

For more details on this modification, see Appendix A.

Algorithm 3: Practical (i.e. independent of a and W

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

y

-0.5

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

r(y,

7

)

a=1,

7

>

=

(a)

µ > τ

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

y

-6

-5

-4

-3

-2

-1

0

1

2

3

4

5

r(y,

7

)

a=1,

7

=

=

(b)

µ

=

τ

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

y

-16

-14

-12

-10

-8

-6

-4

-2

0

2

4

r(y,

7

)

a=1,

7

<

=

(c)

µ < τ

4 A Detailed Analysis of the Evolution of the Stationary Points

The homotopy approach in Algorithm 2 relies heavily on how the stationary points of (7) behave with respect to µ k . In this section, we dive deep into the properties of these critical points.

By analyzing the first-order conditions for g µ , we first narrow our attention to the region A = { 0 ≤ x ≤ a, 0 ≤ y ≤ a a 2 +1 } . By solving the resulting equations, we obtain an equation that only involves the variable y :

Likewise, we can find an equation only involving the variable x :

To understand the behavior of the stationary points of g µ ( W ) , we can examine the characteristics of t ( x ; µ ) in the range x ∈ [0 , a ] and the properties of r ( y ; µ ) in the interval y ∈ [0 , a a 2 +1 ] .

In Figures 2 and 3, we show the behavior of r ( y ; µ ) and t ( x ; µ ) for a = 1 . Theorems 5 and 6 in the appendix establish the existence of a τ > 0 with the following useful property:

Corollary 2. There exists µ < τ such that the equation ∇ g µ ( W ) = 0 has three different solutions, denoted as W ∗ µ , W ∗∗ µ , W ∗∗∗ µ . Then,

Note that the interesting regime takes place when µ < τ . Then, we characterize the stationary points as either local minima or saddle points:

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

x

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

7

)

a=1,

7

>

=

(a)

µ > τ

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

x

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

7

)

a=1,

7

=

=

(b)

µ

=

τ

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

x

-2

-1

0

1

2

3

4

5

7

)

a=1,

7

<

=

(c)

µ < τ

Lemma 1. Let µ < τ , then g µ ( W ) has two local minima at W ∗ µ , W ∗∗∗ µ , and a saddle point at W ∗∗ µ .

With the above results, it has been established that W ∗ µ converges to the global minimum W G as µ → 0 . In the following section for the proof of Theorem 1, we perform a thorough analysis on how to track W ∗ µ and avoid the local minimum at W ∗∗ µ by carefully designing the scheduling for µ k .

5 Convergence Analysis: From continuous to discrete

We now discuss the iteration complexity of our method when gradient descent is used in place of gradient flow. We begin with some preliminaries regarding the continuous-time analysis.

5.1 Continuous case: Gradient flow

The key to ensuring the convergence of gradient flow to W ∗ µ is to accurately identify the basin of attraction of W ∗ µ . The following lemma provides a region that lies within such basin of attraction.

Lemma 2. Define B µ = { ( x, y ) | x ∗∗ µ < x ≤ a, 0 ≤ y < y ∗∗ µ } . Run Algorithm 1 with input f = g µ ( x, y ) , z 0 = ( x (0) , y (0)) where ( x (0) , y (0)) ∈ B µ , then ∀ t ≥ 0 , we have that ( x ( t ) , y ( t )) ∈ B µ and lim t →∞ ( x ( t ) , y ( t )) = ( x ∗ µ , y ∗ µ ) .

In Figure 1 (b), the lower-right rectangle corresponds to B µ . Lemma 2 implies that the gradient flow with any initialization inside B µ k +1 will converge to W ∗ µ k +1 at last. Then, by utilizing the previous solution W ∗ µ k as the initial point, as long as it lies within region B µ k +1 , the gradient flow can converge to W ∗ µ k +1 , thereby achieving the goal of tracking W ∗ µ k +1 . Following the scheduling for µ k prescribed in Algorithm 2 provides a sufficient condition to ensure that will happen.

The following lemma, with proof in the appendix, is used for the Proof of Theorem 1. It provides a lower bound for y ∗∗ µ and upper bound for y ∗ µ .

Proof of Theorem 1. Consider that we are at iteration k +1 of Algorithm 2, then µ k +1 < µ k . If µ k > τ and µ k +1 > τ , then there is only one stationary point for g µ k ( x, y ) and g µ k +1 ( x, y ) , thus, 1 will converge to such stationary point. Hence, let us assume µ k +1 ≤ τ . From Theorem 6 in the appendix, we know that x ∗∗ µ k +1 < x ∗ µ k . Then, the following relations hold:

Here (1) and (3) are due to Lemma 3, and (2) follows from √ 1 -x ≥ 1 -x for 0 ≤ x ≤ 1 . Then it implies that ( x ∗ µ k , y ∗ µ k ) is in the region { ( x, y ) | x ∗∗ µ k +1 < x ≤ a, 0 ≤ y < y ∗∗ µ k +1 } . By Lemma 2, the

1 procedure will converge to ( x ∗ µ k +1 , y ∗ µ k +1 ) . Finally, from Theorems 5 and 6, if lim k →∞ µ k = 0 , then lim k →∞ x ∗ µ k = a, lim k →∞ y ∗ µ k = 0 , thus, converging to the global optimum, i.e.,

5.2 Discrete case: Gradient Descent

In Algorithms 2 and 4, gradient flow is employed to locate the next stationary points, which is not practically feasible. A viable alternative is to execute Algorithm 2, replacing the gradient flow with gradient descent. Now, at every iteration k , Algorithm 6 uses gradient descent to output W µ k ,glyph[epsilon1] k , a glyph[epsilon1] k stationary point of g µ k , initialized at W µ k -1 ,glyph[epsilon1] k -1 , and a step size of η k = 1 / ( µ k ( a 2 +1) + 3 a 2 ) . The tolerance parameter glyph[epsilon1] k can significantly influence the behavior of the algorithm and must be controlled for different iterations. A convergence guarantee is established via a simplified theorem presented here. A more formal version of the theorem and a comprehensive description of the algorithm (i.e., Algorithm 6) can be found in Appendix C.

Theorem 2 (Informal) . For any ε dist > 0 , set µ 0 satisfy a mild condition, and use updating rule glyph[epsilon1] k = min { βaµ k , µ 3 / 2 k } , µ k +1 = (2 µ 2 k ) 2 / 3 ( a + glyph[epsilon1] k /µ k ) 2 / 3 ( a -glyph[epsilon1] k /µ k ) 4 / 3 , and let K ≡ K ( µ 0 , a, ε dist ) ∈ O ( ln µ 0 aε dist ) . Then, for any initialization W 0 , following the updated procedure above for k = 0 , . . . , K , we have:

that is, W µ k ,glyph[epsilon1] k is ε dist -close in Frobenius norm to global optimum W G . Moreover, the total number of gradient descent steps is upper bounded by O ( ( µ 0 a 2 + a 2 + µ 0 ) ( 1 a 6 + 1 ε dist 6 )) .

6 Experiments

We conducted experiments to verify that Algorithms 2 and 4 both converge to the global minimum of (7). Our purpose is to illustrate two main points: First, we compare our updating scheme as given in Line 3 of Algorithm 2 against a faster-decreasing updating scheme for µ k . In Figure 4 we illustrate how a naive faster decrease of µ can lead to spurious a local minimum. Second, in Figure 5, we show that regardless of the initialization, Algorithms 2 and 4 always return the global minimum. In the supplementary material, we provide additional experiments where the gradient flow is replaced with gradient descent. For more details, please refer to Appendix F.

7 Acknowledgments and Disclosure of Funding

K. B. was supported by NSF under Grant # 2127309 to the Computing Research Association for the CIFellows 2021 Project. B.A. was supported by NSF IIS-1956330, NIH R01GM140467, and the Robert H. Topel Faculty Research Fund at the University of Chicago Booth School of Business. This work was done in part while B.A. was visiting the Simons Institute for the Theory of Computing. P.R. was supported by ONR via N000141812861, and NSF via IIS-1909816, IIS-1955532, IIS-2211907. We are also grateful for the support of the University of Chicago Research Computing Center for assistance with the calculations carried out in this work.

(a) Random Initialization 1

(b) Random Initialization 2

0

2

4

6

8

10

12

-2.5

-2

-1.5

-1

-0.5

0

0.5

a = 1.00

start

end

(c) Random Initialization 3

0.4

a = 1.00

start

0.25

a = 1.00

0.35

0.2

0.15

0.3

0.1

0.25

0.05

0.2

0

end

0.15

-0.05

0.1

-0.1

0.05

-0.15

0

0.8

0.9

1

end

1.1

1.2

1.3

-0.2

0.8

1

1.2

1.4

1.6

1.8

2

start

2.2

References

Bello, K., Aragam, B. and Ravikumar, P. [2022], DAGMA: Learning dags via M-matrices and a log-determinant acyclicity characterization, in 'Advances in Neural Information Processing Systems'.

Bertsekas, D. P. [1997], 'Nonlinear programming', Journal of the Operational Research Society 48 (3), 334-334.

Bhojanapalli, S., Neyshabur, B. and Srebro, N. [2016], 'Global optimality of local search for low rank matrix recovery', Advances in Neural Information Processing Systems 29 .

Boumal, N., Voroninski, V. and Bandeira, A. [2016], 'The non-convex burer-monteiro approach works on smooth semidefinite programs', Advances in Neural Information Processing Systems 29 .

Boyd, S., Boyd, S. P. and Vandenberghe, L. [2004], Convex optimization , Cambridge university press.

Brutzkus, A. and Globerson, A. [2017], Globally optimal gradient descent for a convnet with gaussian inputs, in 'International conference on machine learning', PMLR, pp. 605-614.

Chen, W., Drton, M. and Wang, Y. S. [2019], 'On causal discovery with an equal-variance assumption', Biometrika 106 (4), 973-980.

Chickering, D. M. [1996], Learning Bayesian networks is NP-complete, in 'Learning from data', Springer, pp. 121-130.

Chickering, D. M. [2003], 'Optimal structure identification with greedy search', JMLR 3 , 507554.

Chickering, D. M., Heckerman, D. and Meek, C. [2004], 'Large-sample learning of Bayesian networks is NP-hard', Journal of Machine Learning Research 5 , 1287-1330.

Choromanska, A., Henaff, M., Mathieu, M., Arous, G. B. and LeCun, Y. [2015], The loss surfaces of multilayer networks, in 'Artificial intelligence and statistics', PMLR, pp. 192-204.

Cussens, J. [2012], 'Bayesian network learning with cutting planes', arXiv preprint arXiv:1202.3713 .

De Sa, C., Re, C. and Olukotun, K. [2015], Global convergence of stochastic gradient descent for some non-convex matrix problems, in 'International conference on machine learning', PMLR, pp. 2332-2341.

Dontchev, A. L., Rockafellar, R. T. and Rockafellar, R. T. [2009], Implicit functions and solution mappings: A view from variational analysis , Vol. 11, Springer.

Dunlavy, D. M. and O'Leary, D. P. [2005], Homotopy optimization methods for global optimization, Technical report, Sandia National Laboratories (SNL), Albuquerque, NM, and Livermore, CA .

Duong, B. and Nguyen, T. [2022], Bivariate causal discovery via conditional divergence, in 'Conference on Causal Learning and Reasoning'.

Gargiani, M., Zanelli, A., Tran-Dinh, Q., Diehl, M. and Hutter, F. [2020], 'Convergence analysis of homotopy-sgd for non-convex optimization', arXiv preprint arXiv:2011.10298 .

Garrigues, P. and Ghaoui, L. [2008], 'An homotopy algorithm for the lasso with online observations', Advances in neural information processing systems 21 .

Ge, R., Jin, C. and Zheng, Y. [2017], No spurious local minima in nonconvex low rank problems: A unified geometric analysis, in 'International Conference on Machine Learning', PMLR, pp. 1233-1242.

Ge, R., Lee, J. D. and Ma, T. [2016], 'Matrix completion has no spurious local minimum', Advances in neural information processing systems 29 .

Hazan, E., Levy, K. Y. and Shalev-Shwartz, S. [2016], On graduated optimization for stochastic non-convex problems, in 'International conference on machine learning', PMLR, pp. 1833-1841.

Heckerman, D., Geiger, D. and Chickering, D. M. [1995], 'Learning bayesian networks: The combination of knowledge and statistical data', Machine learning 20 (3), 197-243.

Iwakiri, H., Wang, Y., Ito, S. and Takeda, A. [2022], 'Single loop gaussian homotopy method for non-convex optimization', arXiv preprint arXiv:2203.05717 .

Jaakkola, T., Sontag, D., Globerson, A. and Meila, M. [2010], Learning bayesian network structure using lp relaxations, in 'Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics', V ol. 9 of Proceedings of Machine Learning Research , pp. 358-365.

Jiao, R., Lin, N., Hu, Z., Bennett, D. A., Jin, L. and Xiong, M. [2018], 'Bivariate causal discovery and its applications to gene expression and imaging data analysis', Frontiers Genetics 9 , 347.

Kawaguchi, K. [2016], 'Deep learning without poor local minima', Advances in neural information processing systems 29 .

Kazemipour, A., Larsen, B. W. and Druckmann, S. [2019], 'Avoiding spurious local minima in deep quadratic networks', arXiv preprint arXiv:2001.00098 .

Khalil, H. K. [2002], 'Nonlinear systems third edition', Patience Hall 115 .

Lachapelle, S., Brouillard, P., Deleu, T. and Lacoste-Julien, S. [2020], Gradient-based neural dag learning, in 'International Conference on Learning Representations'.

Liang, S., Sun, R., Lee, J. D. and Srikant, R. [2018], 'Adding one neuron can eliminate all bad local minima', Advances in Neural Information Processing Systems 31 .

Loh, P.-L. and Bühlmann, P. [2014], 'High-dimensional learning of linear causal networks via inverse covariance estimation', The Journal of Machine Learning Research 15 (1), 3065-3105.

Mooij, J. M., Peters, J., Janzing, D., Zscheischler, J. and Schölkopf, B. [2014], 'Distinguishing cause from effect using observational data: methods and benchmarks', Arxiv .

Nesterov, Y. et al. [2018], Lectures on convex optimization , Vol. 137, Springer.

Ng, I., Ghassami, A. and Zhang, K. [2020], On the role of sparsity and dag constraints for learning linear DAGs, in 'Advances in Neural Information Processing Systems'.

Ng, I., Lachapelle, S., Ke, N. R., Lacoste-Julien, S. and Zhang, K. [2022], On the convergence of continuous constrained optimization for structure learning, in 'International Conference on Artificial Intelligence and Statistics', PMLR, pp. 8176-8198.

Ni, Y. [2022], 'Bivariate causal discovery for categorical data via classification with optimal label permutation', Arxiv .

Nocedal, J. and Wright, S. J. [1999], Numerical optimization , Springer.

Park, Y. W. and Klabjan, D. [2017], 'Bayesian network learning via topological order', The Journal of Machine Learning Research 18 (1), 3451-3482.

Pearl, J. [2009], Causality: Models, Reasoning, and Inference , 2nd edn, Cambridge University Press.

Sachs, K., Perez, O., Pe'er, D., Lauffenburger, D. A. and Nolan, G. P. [2005], 'Causal proteinsignaling networks derived from multiparameter single-cell data', Science 308 (5721), 523-529.

Scanagatta, M., de Campos, C. P., Corani, G. and Zaffalon, M. [2015], Learning bayesian networks with thousands of variables., in 'NIPS', pp. 1864-1872.

Silander, T. and Myllymaki, P. [2006], A simple approach for finding the globally optimal bayesian network structure, in 'Proceedings of the 22nd Conference on Uncertainty in Artificial Intelligence'.

Soltanolkotabi, M., Javanmard, A. and Lee, J. D. [2018], 'Theoretical insights into the optimization landscape of over-parameterized shallow neural networks', IEEE Transactions on Information Theory 65 (2), 742-769.

Spirtes, P., Glymour, C. N., Scheines, R. and Heckerman, D. [2000], Causation, prediction, and search , MIT press.

Teyssier, M. and Koller, D. [2005], Ordering-based search: A simple and effective algorithm for learning bayesian networks, in 'Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence'.

Wauthier, F. and Donnelly, P. [2015], A greedy homotopy method for regression with nonconvex constraints, in 'Artificial Intelligence and Statistics', PMLR, pp. 1051-1060.

Wei, D., Gao, T. and Yu, Y. [2020], DAGs with no fears: A closer look at continuous optimization for learning bayesian networks, in 'Advances in Neural Information Processing Systems'.

Wu, C., Luo, J. and Lee, J. D. [2018], No spurious local minima in a two hidden unit relu network, in 'International Conference on Learning Representations'.

Yu, Y., Chen, J., Gao, T. and Yu, M. [2019], Dag-gnn: Dag structure learning with graph neural networks, in 'International Conference on Machine Learning', PMLR, pp. 7154-7163.

Zhang, B., Gaiteri, C., Bodea, L.-G., Wang, Z., McElwee, J., Podtelezhnikov, A. A., Zhang, C., Xie, T., Tran, L., Dobrin, R. et al. [2013], 'Integrated systems approach identifies genetic nodes and networks in late-onset alzheimers disease', Cell 153 (3), 707-720.

Zheng, X., Aragam, B., Ravikumar, P. K. and Xing, E. P. [2018], DAGs with NO TEARS: Continuous optimization for structure learning, in 'Advances in Neural Information Processing Systems'.

Zheng, X., Dan, C., Aragam, B., Ravikumar, P. and Xing, E. [2020], Learning sparse nonparametric DAGs, in 'International Conference on Artificial Intelligence and Statistics', PMLR, pp. 3414-3425.

Zhu, S., Ng, I. and Chen, Z. [2020], Causal discovery with reinforcement learning, in 'International Conference on Learning Representations'.

Algorithm 4: Find a path { W µ } via a particular scheduling for µ k when a is unknown.

A Practical Implementation of Algorithm 2

We present a practical implementation of our homotopy algorithm in Algorithm 4. The updating scheme for µ k is now independent of the parameter a , but as presented, the initialization for µ 0 still depends on a . This is for the following reason: It is possible to make the updating scheme independent of a without imposing any additional assumptions on a , as evidenced by Lemma 4 below. The initialization for µ 0 , however, is trickier, and we must consider two separate cases:

No assumptions on a . In this case, if a is too small, then the problem becomes harder and the initial choice of µ 0 matters.

Lower bound on a . If we are willing to accept a lower bound on a , then there is an initialization for µ 0 that does not depend on a .

In Corollary 1, we illustrate this last point with the additional condition that a > √ 5 / 27 . This essentially amounts to an assumption on the minimum signal, and is quite standard in the literature on learning SEM.

Lemma 4. Under the assumption a 2 4( a 2 +1) 3 ≤ µ 0 < a 2 4 , the Algorithm 4 outputs the global optimal solution to (6) , i.e.

It turns out that the assumption in Lemma 4 is not overly restrictive, as there exist pre-determined sequences of { µ k } ∞ k =0 that can ensure the effectiveness of Algorithm 4 for any values of a greater than a certain threshold.

B From Population Loss to Empirical Loss

The transformation from population loss to empirical can be thought from two components. First, with a given empirical loss, Algorithms 2 and 3 still achieve the global minimum, W G , of problem 6, but now the output from the Algorithm is an empirical estimator ˆ a , rather than ground truth a , Theorem 1 and Corollary 1 would continue to be valid. Second, the global optimum, W G , of the empirical loss possess the same DAG structure as the underlying W ∗ . The finite-sample findings in Section 5 (specifically, Lemmas 18 and 19) of Loh and Bühlmann [31], which offer sufficient conditions on the sample size to ensure that the DAG structures of W G and W ∗ are identical.

C From Continuous to Discrete: Gradient Descent

Previously, gradient flow was employed to address the intermediate problem (7), a method that poses implementation challenges in a computational setting. In this section, we introduce Algorithm 6 that leverages gradient descent to solve (7) in each iteration. This adjustment serves practical considerations. We start with the convergence results of Gradient Descent.

Definition 1. f is L -smooth, if f is differentiable and ∀ x, y ∈ dom ( f ) such that ‖∇ f ( x ) -∇ f ( y ) ‖ 2 ≤ L ‖ x -y ‖ 2 .

Algorithm 5: Gradient Descent( f, η, W 0 , glyph[epsilon1] )

Input: function f , step size η , initial point W 0 , tolerance glyph[epsilon1] W

Algorithm 6:

Theorem 3 (Nesterov et al. 33) . If function f is L -smooth, then Gradient Descent (Algorithm 5) with step size η = 1 /L , finds an glyph[epsilon1] -first-order stationary point (i.e. ‖∇ f ( x ) ‖ 2 ≤ glyph[epsilon1] ) in 2 L ( f ( x 0 ) -f ∗ ) /glyph[epsilon1] 2 iterations.

One of the pivotal factors influencing the convergence of gradient descent is the selection of the step size. Theorem 3 select a step size η = 1 L . Therefore, our initial step is to determine the smoothness of g µ ( W ) within our region of interest, A = { 0 ≤ x ≤ a, 0 ≤ y ≤ a a 2 +1 } .

Lemma 5. Consider the function g µ ( W ) as defined in Equation 7 within the region A = { 0 ≤ x ≤ a, 0 ≤ y ≤ a a 2 +1 } . It follows that for all µ ≥ 0 , the function g µ ( W ) is µ ( a 2 +1) + 3 a 2 -smooth.

Since gradient descent is limited to identifying the glyph[epsilon1] stationary point of the function. Thus, we study the gradient of g µ ( W ) = µf ( W ) + h ( W ) , i.e. ∇ g µ ( W ) has the following form

As gradient descent is limited to identifying the glyph[epsilon1] stationary point of the function, we, therefore, focus on ‖ g µ ( W ) ‖ 2 ≤ glyph[epsilon1] . This can be expressed in the subsequent manner:

As a result,

Here we denote such region as A µ,glyph[epsilon1]

Figure 6 and 7 illustrate the region A µ,glyph[epsilon1] .

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.0

0.1

0.2

0.3

0.4

0.5

y

Plot

a

=0.6,

= 0.009,   =0.00055

(

x

*

,

y

*

)

(

x

* *

,

y

* *

)

(

x

* * *

,

y

* * *

)

(

x

*

,

,

y

*

,

)

(

x

* *

,

,

y

* *

,

)

(

x

* * *

,

,

y

* * *

,

)

(

x

*

,

_,

,

y

*

,

_

)

y

=

(

a

x

1) +

x

y

=

(

a

x

1)

x

y

=

a

+

x

2 + (

a

2 +1)

y

=

a

x

2 + (

a

2 +1)

x

0.45

0.50

0.55

0.60

0.65

0.70

0.00

0.02

0.04

0.06

0.08

0.10

y

Plot

a

=0.6,

= 0.009,   =0.00055

(

x

*

,

y

*

)

(

x

*

,

,

y

*

,

)

(

x

*

,

_,

,

y

*

,

_

)

y

=

(

a

x

1)

y

=

a

x

2 + (

a

2 +1)

y

=

(

a

x

1) +

x

y

=

(

a

x

1)

x

y

=

a

+

x

2 + (

a

2 +1)

y

=

a

x

2 + (

a

2 +1)

x

Given that the gradient descent can only locate glyph[epsilon1] stationary points within the region A µ,glyph[epsilon1] during each iteration, the boundary of A µ,glyph[epsilon1] becomes a critical component of our analysis. To facilitate clear presentation, it is essential to establish some pertinent notations.

•

•

•

If the system of equations yields only a single solution, we denote this solution as ( x ∗ µ , y ∗ µ ) . If it yields two solutions, these solutions are denoted as ( x ∗ µ , y ∗ µ ) , ( x ∗∗ µ , y ∗∗ µ ) , with x ∗∗ µ < x ∗ µ . In the event that there are three distinct solutions to the system of equations, these solutions are denoted as ( x ∗ µ , y ∗ µ ) , ( x ∗∗ µ , y ∗∗ µ ) , ( x ∗∗∗ µ , y ∗∗∗ µ ) , where x ∗∗∗ µ < x ∗∗ µ < x ∗ µ .

If the system of equations yields only a single solution, we denote this solution as ( x ∗ µ,glyph[epsilon1] , y ∗ µ,glyph[epsilon1] ) . If it yields two solutions, these solutions are denoted as ( x ∗ µ,glyph[epsilon1] , y ∗ µ,glyph[epsilon1] ) , ( x ∗∗ µ,glyph[epsilon1] , y ∗∗ µ,glyph[epsilon1] ) , with x ∗∗ µ,glyph[epsilon1] < x ∗ µ,glyph[epsilon1] . In the event that there are three distinct solutions to the system of equations, these solutions are denoted as ( x ∗ µ,glyph[epsilon1] , y ∗ µ,glyph[epsilon1] ) , ( x ∗∗ µ,glyph[epsilon1] , y ∗∗ µ,glyph[epsilon1] ) , ( x ∗∗∗ µ,glyph[epsilon1] , y ∗∗∗ µ,glyph[epsilon1] ) , where x ∗∗∗ µ,glyph[epsilon1] < x ∗∗ µ,glyph[epsilon1] < x ∗ µ,glyph[epsilon1] .

If the system of equations yields only a single solution, we denote this solution as ( x ∗ µ,glyph[epsilon1] _ , y ∗ µ,glyph[epsilon1] _ ) . If it yields two solutions, these solutions are denoted as ( x ∗ µ,glyph[epsilon1] _ , y ∗ µ,glyph[epsilon1] _ ) , ( x ∗∗ µ,glyph[epsilon1] _ , y ∗∗ µ,glyph[epsilon1] _ ) , with x ∗∗ µ,glyph[epsilon1] _ < x ∗ µ,glyph[epsilon1] _ . In the event that there are three distinct solutions to the system of equations, these solutions are denoted as ( x ∗ µ,glyph[epsilon1] _ , y ∗ µ,glyph[epsilon1] _ ) , ( x ∗∗ µ,glyph[epsilon1] _ , y ∗∗ µ,glyph[epsilon1] _ ) , ( x ∗∗∗ µ,glyph[epsilon1] _ , y ∗∗∗ µ,glyph[epsilon1] _ ) , where x ∗∗∗ µ,glyph[epsilon1] _ < x ∗∗ µ,glyph[epsilon1] _ < x ∗ µ,glyph[epsilon1] _ .

Remark 4. There always exists at least one solution to the above system of equations. When µ is sufficiently small, the above system of equations always yields three solutions, as demonstrated in Theorem 5, and Theorem 9.

The parameter glyph[epsilon1] can substantially influence the behavior of the systems of equations (12a),(12b) and (13a),(13b). A crucial consideration is to ensure that glyph[epsilon1] remains adequately small. To facilitate this, we introduce a new parameter, β , whose specific value will be determined later. At this stage, we merely require that β should lie within the interval (0 , 1) . We further impose a constraint on glyph[epsilon1] to satisfy the following inequality:

Following the same procedure when we deal with glyph[epsilon1] = 0 . Let us substitute (12a) into (12b), then we obtain an equation that only involves the variable y

Let us substitute (12b) into (12a), then we obtain an equation that only involves the variable x

Proceed similarly for equations (13a) and (13b).

Given the substantial role that the system of equations 12a and 12b play in our analysis, the existence of glyph[epsilon1] in these equations complicates the analysis, this can be avoided by considering the worst-case scenario, i.e., when glyph[epsilon1] = βaµ . With this particular choice of glyph[epsilon1] , we can reformulate (15) and (16) as follows, denoting them as r β ( y ; glyph[epsilon1] ) and r β ( x ; glyph[epsilon1] ) respectively.

The functions r glyph[epsilon1] ( y ; µ ) , r glyph[epsilon1] _ ( y ; µ ) , and r β ( y ; µ ) possess similar properties to r ( y ; µ ) as defined in Equation (8), with more details available in Theorem 7 and 8. Additionally, the functions t glyph[epsilon1] ( x ; µ ) , t glyph[epsilon1] _ ( x ; µ ) , and t β ( x ; µ ) share similar characteristics with t ( x ; µ ) as defined in Equation (9), with more details provided in Theorem 9.

As illustrated in Figure 6, the glyph[epsilon1] -stationary point region A µ,glyph[epsilon1] can be partitioned into two distinct areas, of which only the lower-right one contains ( x ∗ µ , y ∗ µ ) and it is of interest to our analysis. Moreover, ( x ∗ µ,glyph[epsilon1] , y ∗ µ,glyph[epsilon1] ) and ( x ∗∗ µ,glyph[epsilon1] , y ∗∗ µ,glyph[epsilon1] ) are extremal point of two distinct regions. The upcoming corollary substantiates this intuition.

Corollary 3. If µ < τ ( τ is defined in Theorem 5(v)), assume glyph[epsilon1] satisfies (14) , β satisfies ( 1+ β 1 -β ) 2 ≤ a 2 +1 , systems of equations (12a) , (12b) at least have two solutions. Moreover, A µ,glyph[epsilon1] = A 1 µ,glyph[epsilon1] ∪ A 2 µ,glyph[epsilon1]

Corollary 3 suggests that A µ,glyph[epsilon1] can be partitioned into two distinct regions, namely A 1 µ,glyph[epsilon1] and A 2 µ,glyph[epsilon1] . Furthermore, for every ( x, y ) belonging to A 1 µ,glyph[epsilon1] , it follows that x ≥ x ∗ µ,glyph[epsilon1] and y ≤ y ∗ µ,glyph[epsilon1] . Similarly, for every ( x, y ) that lies within A 2 µ,glyph[epsilon1] , the condition x ≤ x ∗∗ µ,glyph[epsilon1] and y ≥ y ∗∗ µ,glyph[epsilon1] holds. The region A 1 µ,glyph[epsilon1] represents the 'correct" region that gradient descent should identify. In this context, identifying the region equates to pinpointing the extremal points of the region. As a result, our focus should be on the extremal points of A 1 µ,glyph[epsilon1] and A 2 µ,glyph[epsilon1] , specifically at ( x ∗ µ,glyph[epsilon1] , y ∗ µ,glyph[epsilon1] ) and ( x ∗∗ µ,glyph[epsilon1] , y ∗∗ µ,glyph[epsilon1] ) . Furthermore, the key to ensuring the convergence of the gradient descent to the A 1 µ,glyph[epsilon1] is to accurately identify the 'basin of attraction' of the region A 1 µ,glyph[epsilon1] . The following lemma provides a region within which, regardless of the initialization point of the gradient descent, it converges inside A 1 µ,glyph[epsilon1] .

Lemma 6. Assume µ < τ ( τ is defined in Theorem 5(v)), ( 1+ β 1 -β ) 2 ≤ a 2 +1 . Define B µ,glyph[epsilon1] = { ( x, y ) | x ∗∗ µ,glyph[epsilon1] < x ≤ a, 0 ≤ y < y ∗∗ µ,glyph[epsilon1] } . Run Algorithm 5 with input f = g µ ( x, y ) , η = 1 µ ( a 2 +1)+3 a 2 , W 0 = ( x (0) , y (0)) , where ( x (0) , y (0)) ∈ B µ,glyph[epsilon1] , then after at most 2( µ ( a 2 +1)+3 a 2 )( g µ ( x (0) ,y (0)) -g µ ( x ∗ µ ,y ∗ µ )) glyph[epsilon1] 2 iterations, ( x t , y t ) ∈ A 1 µ,glyph[epsilon1] .

Lemma 6 can be considered the gradient descent analogue of Lemma 2. It plays a pivotal role in the proof of Theorem 4. In Figure 6, the lower-right rectangle corresponds to B µ,glyph[epsilon1] . Lemma 6 implies that the gradient descent with any initialization inside B µ k +1 ,glyph[epsilon1] k +1 will converge to A 1 µ k +1 ,glyph[epsilon1] k +1 at last. Then, by utilizing the previous solution W µ k ,glyph[epsilon1] k as the initial point, as long as it lies within region B µ k +1 ,glyph[epsilon1] k +1 , the gradient descent can converge to A 1 µ k +1 ,glyph[epsilon1] k +1 which is glyph[epsilon1] stationary points region that contains W ∗ µ k +1 , thereby achieving the goal of tracking W ∗ µ k +1 . Following the scheduling for µ k prescribed in Algorithm 6 provides a sufficient condition to ensure that will happen.

We now proceed to present the theorem which guarantees the global convergence of Algorithm 6.

Set the updating rule where

The total gradient descent steps are

Proof. Upon substituting gradient flow with gradient descent, it becomes possible to only identify an glyph[epsilon1] -stationary point for g µ ( W ) . This modification necessitates specifying the stepsize η for gradient descent, as well as an updating rule for µ . The adjustment procedure used can substantially influence the result of Algorithm 6 . In this proof, we will impose limitations on the update scheme µ k , the stepsize η k , and the tolerance glyph[epsilon1] k to ensure their effective operation within Algorithm 6 . The approach employed for this proof closely mirrors that of the proof for Theorem 1 albeit with more careful scrutiny. In this proof, we will work out all the requirements for µ, glyph[epsilon1], η . Subsequently, we will verify that our selection in Theorem 4 conforms to these requirements.

In the proof, we occasionally use µ, glyph[epsilon1] or µ k , glyph[epsilon1] k . When we employ µ, glyph[epsilon1] , it signifies that the given inequality or equality holds for any µ, glyph[epsilon1] . Conversely, when we use µ k , glyph[epsilon1] k , it indicates we are examining how to set these parameters for distinct iterations.

Establish the Bound y ∗∗ µ,glyph[epsilon1] ≥ √ µ First, let us consider r glyph[epsilon1] ( √ µ ; µ ) ≤ 0 , i.e.

This is always true when µ > 4 /a 2 , and we require

Now we name it condition 1.

Condition 1.

Under the assumption that Condition 1 is satisfied. Since r glyph[epsilon1] ( y ; µ ) is increasing function with interval y ∈ [ y lb ,glyph[epsilon1] , y ub ,glyph[epsilon1] ] , and we know y lb ,glyph[epsilon1] ≤ √ µ ≤ y ub ,glyph[epsilon1] and based on Theorem 7(ii), we have y lb ,glyph[epsilon1] ≤ y ∗∗ µ,glyph[epsilon1] ≤ y ub ,glyph[epsilon1] , r glyph[epsilon1] ( √ µ ; µ ) ≤ r glyph[epsilon1] ( y ∗∗ µ,glyph[epsilon1] ; µ ) = 0 . Therefore, y ∗∗ µ,glyph[epsilon1] ≥ √ µ .

Then µ k +1 ≤ (1 -δ ) µ k . Moreover, for any ε dist > 0 , running Algorithm 6 after K ( µ 0 , a, δ, ε dist ) outer iteration

Ensuring the Correct Solution Path via Gradient Descent Following the argument when we prove Theorem 1, we strive to ensure that the gradient descent, when initiated at ( x µ k ,glyph[epsilon1] k , y µ k ,glyph[epsilon1] k ) , will converge within the "correct" glyph[epsilon1] k +1 -stationary point region (namely, ‖∇ g µ k +1 ( W ) ‖ 2 < glyph[epsilon1] k +1 ) which includes ( x ∗ µ k +1 , y ∗ µ k +1 ) . For this to occur, we necessitate that:

Here (1), (5) are due to Corollary 3; (2) comes from the boundary we established earlier; (3) is based on the constraints we have placed on µ k and µ k +1 , which we will present as Condition 2 subsequently; (4) is from the Theorem 7(ii) and relationship y ∗ µ k ,glyph[epsilon1] k < y lb ,µ k ,glyph[epsilon1] k . Also, from the Lemma 9, max µ ≤ τ x ∗∗ µ,glyph[epsilon1] ≤ min µ> 0 x ∗ µ,glyph[epsilon1] . Hence, by invoking Lemma 6, we can affirm that our gradient descent consistently traces the correct stationary point. Now we state condition to make it happen,

Condition 2.

In this context, our requirement extends beyond merely ensuring that µ k decreases. We further stipulate that it should decrease by a factor of 1 -δ . Next, we impose another important constraint

Condition 3.

Updating Rules Now we are ready to check our updating rules satisfy the conditions above

Check for Conditions First, we check the condition 2. condition 2 requires

Note that glyph[epsilon1] k ≤ βaµ k < aµ k

Therefore, once the following inequality is true, Condition 2 is satisfied.

Because µ k ≤ µ 0 ≤ a 2 4 (1 -δ ) 3 (1 -β ) 4 (1+ β ) 2 from the condition we impose for µ 0 . Consequently, Condition 2 is satisfied under our choice of glyph[epsilon1] k .

Now we focus on the Condition 1. Because glyph[epsilon1] k ≤ aβµ k , if we can ensure aβµ k ≤ 2 µ 3 / 2 k + aµ k -2 √ 2 aµ 5 / 2 k -µ 3 k a 2 holds, then we can show Condition 1 is always satisfied.

We also notice that

Because ( 1+ β 1 -β ) 2 ≤ (1 -δ )( a 2 +1) , the inequality above always holds and this inequality implies that for any µ k ≥ 0

Therefore, Condition 2 holds. Condition 3 also holds because of the choice of glyph[epsilon1] k .

Bound the Distance Let c = 72 /a 2 , and assume that µ satisfies the following

Note that when µ satisfies (25), then µ 3 / 2 ≤ βaµ , so glyph[epsilon1] = µ 3 / 2 .

Then

Then we know ( a -glyph[epsilon1]/µ )(1 -cµ ) < x ∗ µ,glyph[epsilon1] . Now we can bound the distance ‖ W µ k ,glyph[epsilon1] k -W G ‖ , it is important to note that

We use the fact that x ∗ µ k ,glyph[epsilon1] k < x µ k ,glyph[epsilon1] k < a , x µ k ,glyph[epsilon1] k < x ∗ µ k ,glyph[epsilon1] k _ and y µ k ,glyph[epsilon1] k < y ∗ µ k ,glyph[epsilon1] k . Next, we can separately establish bounds for these two terms. Due to (24), y ∗ µ k ,glyph[epsilon1] k < (2 µ 2 k ) 1 / 3 ( a + glyph[epsilon1] k /µ k ) 1 / 3 ( a -glyph[epsilon1] k /µ k ) 2 / 3 = √ µ k +1 and ( a -glyph[epsilon1] k /µ k )(1 -cµ k ) < x ∗ µ k ,glyph[epsilon1] k

. There-

As a result, we have

We use the fact that µ k ≤ (1 -δ ) k µ 0 . In order to satisfy (25).

Consequently, running Algorithm 6 after K ( µ 0 , a, δ, ε dist ) outer iteration

Just let where

By Lemma 6, k iteration of Algorithm 6 need the following step of gradient descent

Let ̂ K ( µ 0 , a, δ, ε dist ) satisfy µ ̂ K ( µ 0 ,a,δ,ε dist ) ≤ β 2 a 2 < µ ̂ K ( µ 0 ,a,δ,ε dist ) -1 . Hence, the total number of gradient steps required by Algorithm 6 can be expressed as follows:

Note from (27) and (30), the following should holds

Therefore,

D Additional Theorems and Lemmas

Theorem 5 (Detailed Property of r ( y ; µ ) ) . For r ( y ; µ ) in (8) , then

(ii) For µ > 0 , r ( √ µ, µ ) < 0 .

(iii) For µ > a 2 4

For

0

< µ

where

Moreover,

2

a

4

(iv) For 0 < µ < a 2 4 , let p ( µ ) = r ( y ub , µ ) , then p ′ ( µ ) < 0 and there exist a unique solution to p ( µ ) = 0 , denoted as τ . Additionally, τ < a 2 4 .

(v) There exists a τ > 0 such that, ∀ µ > τ , the equation r ( y ; µ ) = 0 has only one solution. At µ = τ , the equation r ( y ; µ ) = 0 has two solutions, and ∀ µ < τ , the equation r ( y ; µ ) = 0 has three solutions. Moreover, µ < a 2 4 .

(vi) ∀ µ < τ , the equation r ( y ; µ ) = 0 has three solution, i.e. y ∗ µ < y ∗∗ µ < y ∗∗∗ µ .

Moreover,

Theorem 6 (Detailed Property of t ( x ; µ ) ) . For t ( x ; µ ) in (9) , then

(i) For µ > 0 , lim x → 0 + t ( x ; µ ) = ∞ , t ( a, µ ) < 0

(iii) For µ > a 2 4( a 2 +1) 3

For 0 < µ ≤ a 2 4( a 2 +1) 3

where

Moreover, where

Also,

(iv) For 0 < µ < a 2 4( a 2 +1) 3 and let q ( µ ) = t ( x lb , µ ) , then q ′ ( µ ) > 0 and there exist a unique solution to q ( µ ) = 0 , denoted as τ and τ < a 2 4( a 2 +1) 3 ≤ 1 27 .

(v) There exists a τ > 0 such that, ∀ µ > τ , the equation t ( x ; µ ) = 0 has only one solution. At µ = τ , the equation t ( x ; µ ) = 0 has two solutions, and ∀ µ < τ , the equation t ( x ; µ ) = 0 has three solutions. Moreover, τ < a 2 4( a 2 +1) 3 ≤ 1 27

(vi) ∀ µ < τ , t ( x ; µ ) = 0 has three stationary points, i.e. x ∗∗∗ µ < x ∗∗ µ < x ∗ µ .

Besides,

It also implies that t ( a ( √ a 2 +1 -a ) 2 √ a 2 +1 ; µ ) ≥ 0 and max µ ≤ µ 0 x ∗∗ µ < min µ> 0 x ∗ µ

Lemma 7. Algorithm 1 with input f = g µ ( x, y ) , z 0 = ( x (0) , y (0)) where ( x (0) , y (0)) ∈ C µ 3 in (41) , then ∀ t ≥ 0 , ( x ( t ) , y ( t )) ∈ C µ 3 . Moreover, lim t →∞ ( x ( t ) , y ( t )) = ( x ∗ µ , y ∗ µ )

glyph[negationslash]

Lemma 8. For any ( x, y ) ∈ C µ 3 in (41) , and ( x, y ) = ( x ∗ µ , y ∗ µ )

Theorem 7 (Detailed Property of r glyph[epsilon1] ( y ; µ ) ) . For r glyph[epsilon1] ( y ; µ ) in (15) , then

(i) For µ > 0 , glyph[epsilon1] > 0 , lim y → 0 + r glyph[epsilon1] ( y ; µ ) = ∞ , y ( a a 2 +1 , µ ) < 0

(ii) For µ > ( a -glyph[epsilon1]/µ ) 4 4( a + glyph[epsilon1]/µ ) 2 , then dr glyph[epsilon1] ( y ; µ ) dy < 0 . For 0 < µ ≤ ( a -glyph[epsilon1]/µ ) 4 4( a + glyph[epsilon1]/µ ) 2

Theorem 8 (Detailed Property of r β ( y ; µ ) ) . For r β ( y ; µ ) in (19) , then

(i) For µ > 0 , glyph[epsilon1] > 0 , lim y → 0 + r β ( y ; µ ) =

(ii) For µ > a 2 (1 -β ) 4 4(1+ β ) 2 , then dr β ( y ; µ ) dy < 0 . For 0 < µ ≤ a 2 (1 -β ) 4 4(1+ β ) 2

Theorem 9 (Detailed Property of t β ( x ; µ ) ) . For t β ( x ; µ ) in (20) , then

(i) For µ > 0 , lim x → 0 + t β ( x ; µ ) = ∞ , t β ( a ; µ ) < 0

(ii) For µ > a 2 4( a 2 +1) 3 ( β +1) 4 ( β -1) 2

For

0

< µ

a

2

4(

a

2

+1)

3

(

(

β

β

+1)

-

4

1)

2

where

Also, where

(iii) If 0 < β < √ ( a 2 +1) -1 √ ( a 2 +1)+1 , then there exists a τ β > 0 such that, ∀ µ > τ β , the equation r β ( x ; µ ) = 0 has only one solution. At µ = τ β , the equation r β ( x ; µ ) = 0 has two solutions, and ∀ µ < τ β , the equation r β ( x ; µ ) = 0 has three solutions. Moreover, µ < a 2 4( a 2 +1) 3 ( β +1) 4 ( β -1) 2 .

(iv) If 0 < β < √ ( a 2 +1) -1 √ ( a 2 +1)+1 , then ∀ µ < τ β , t β ( x ; µ ) = 0 has three stationary points, i.e. x ∗∗∗ µ,β < x ∗∗ µ,β < x ∗ µ,β . Besides,

It implies that

E Technical Proofs

E.1 Proof of Theorem 3

Proof. For the sake of completeness, we have included the proof here. Please note that this proof can also be found in [33].

Proof. We use the fact that f is L -smooth function if and only if for any W,Y ∈ dom ( f )

Let W = W t +1 and Y = W t , then using the updating rule W t +1 = W t -1 L ∇ f ( W t )

Therefore,

E.2 Proof of Theorem 5

Proof. (i) For any µ > 0 ,

Lemma 9. Under the same setting as Corollary 3,

For µ ≥ a 2 4 , ((4 aµ ) 1 / 3 y -y 2 -µ ) < 0 ⇔ dr ( y ; µ ) dy < 0 . For µ < a 2 4 , y lb < y < y ub , ((4 aµ ) 1 / 3 y -y 2 -µ ) > 0 ⇔ dr ( y ; µ ) dy > 0 . For µ < a 2 4 , y < y lb or y ub < y, ((4 aµ ) 1 / 3 y -y 2 -µ ) ≤ 0 ⇔ dr ( y ; µ ) dy ≤ 0 . Note that

The intersection between line (4 aµ ) 1 / 3 and function y + µ y are exactly y lb and y ub , and y lb < √ µ < y ub .

(iv) Note that for 0 < µ < a 2 4 ,

Also note that when µ = a 2 4 , y ub = √ µ , p ( µ ) = r ( y ub , µ ) = r ( √ µ, µ ) < 0 , and also if µ < a 2 4 , then

Thus,

Because a (4 a ) 1 / 3 > a 2 (4 a ) 4 / 3 , it is easy to see when µ → 0 , r ((4 µa ) 1 / 3 , µ ) →∞ . We know r ( y ub , µ ) > r ((4 µa ) 1 / 3 , µ ) → ∞ as µ → 0 because of the monotonicity of r ( y ; µ ) in Theorem 5(iii). Combining all of these, i.e.

There exists a τ < a 2 4 such that p ( τ ) = 0

(v) From Theorem 5(iv), for µ > τ , then p ( µ ) = r ( y ub , µ ) > 0 , and for µ = τ , then p ( µ ) = r ( y ub , µ ) = 0 . For µ < τ , then p ( µ ) = r ( y ub , µ ) < 0 , combining Theorem 5(i),5(iii), we get the conclusions.

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

By implicit function theorem [14], for solution to equation r ( y ; µ ) = 0 , there exists a unique continuously differentiable function such that y = y ( µ ) and satisfies r ( y ( µ ) , µ ) = 0 . Therefore,

Therefore by Theorem 5(iii),

Because lim µ → 0 + y lb = lim µ → 0 + y ub = 0 , then lim µ → 0 + y ∗ µ = lim µ → 0 + y ∗∗ µ = 0 . Let us consider r ( a a 2 +1 (1 -cµ ) , µ ) where c = 32 ( a 2 +1) 3 a 2 and µ < 1 2 c

By Theorem 5(iii), then a a 2 +1 (1 -cµ ) < y ∗∗∗ µ , then

Consequently,

E.3 Proof of Theorem 6

Proof. (i) For µ > 0 ,

We use the same argument as before to show that

(iv) Note that for 0 < µ < a 2 4( a 2 +1) 3

then ∂t ∂µ ∣ ∣ ∣ x = x lb > 0 . Let q ( µ ) = t ( x lb , µ ) , because ∂t ∂x ∣ ∣ x = x lb = 0 , then

Note that µ = a 2 4( a 2 +1) 3 , x ub = x lb = (4 µa ) 1 / 3 2 , t ( (4 µa ) 1 / 3 2 , a 2 4( a 2 +1) 3 ) = a (4 µa ) 1 / 3 -1 > 0 . When µ < ( a ( √ a 2 +1 -a ) 2( a 2 +1) ) 2 , then t ( √ µ ( a 2 +1) , µ ) < 0 by Theorem 6(ii). It implies that q ( µ ) < 0 when µ → 0 + . By Theorem 6(iii), q ( µ ) = t ( x lb , µ ) < t ( √ µ ( a 2 +1) , µ ) < 0 . Combining all of the theses, i.e.

There exists a τ < a 2 4( a 2 +1) 3 , q ( τ ) = 0 . Such τ is the same as in Theorem 5(iv).

(v) We follow the same proof from the proof of Theorem 5(v).

(vi) By Theorem 6(v), ∀ µ < µ 0 , there exists three stationary points such that 0 < x ∗∗∗ µ < x lb <

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

By implicit function theorem [14], for solutions to equation t ( x ; µ ) = 0 , there exists a unique continuously differentiable function such that x = x ( µ ) and satisfies t ( x ( µ ) , µ ) = 0 . Therefore,

Therefore, by Theorem 6(iii)

Hence, lim µ → 0 x ∗ µ = a .

When µ = τ , because t ( x lb ; µ ) = 0 and x ub > √ µ ( a 2 +1) > x lb , t ( x ; µ ) is increasing function between [ x lb , x ub ] then t ( √ µ ( a 2 +1); µ ) > t ( x lb ; µ ) = 0 . Moreover, t ( √ µ ( a 2 +1) , µ ) , x lb and x ∗∗ µ are continuous function w.r.t µ , ∃ δ > 0 which is really small, such that µ = τ -δ and t ( √ µ ( a 2 +1) , µ ) > 0 , t ( x lb , µ ) < 0 (by Theorem 6(iv)) and x ∗∗ µ > x lb , hence dx dµ ∣ ∣ ∣ x = x ∗∗ µ < 0 . It implies when µ decreases, then x ∗∗ µ increases. This relation holds until x ∗∗ µ = √ µ ( a 2 +1)

and √ µ ( a 2 +1) = a ( √ a 2 +1 -a ) 2 √ a 2 +1 . Note that when µ < ( a ( √ a 2 +1 -a ) 2( a 2 +1) ) 2 , t ( √ µ ( a 2 +1) , µ ) < 0 , it implies that x ∗∗ µ > √ µ ( a 2 +1) and dx dµ ∣ ∣ ∣ x = x ∗∗ µ > 0 , thus decreasing µ leads to decreasing x ∗∗ µ . We can conclude

Because 0 < x ∗∗∗ µ < x lb < x ∗∗ µ < x ub and lim µ → 0 + x lb = lim µ → 0 + x ub = 0 .

Let us consider t ( a (1 -cµ ) , µ ) where c = 32 a 2 and µ < 1 2 c

By Theorem 6(iii). It implies taking µ → 0 + on both side,

Note that when µ > a 2 a 2 +1 , i.e. ( x ∗ µ ) 2 ≥ µ ( a 2 +1) then

It implies that when µ decreases, x ∗ µ also decreases. It holds true until x ∗ µ = √ µ ( a 2 +1) . The same analysis can be applied to x ∗ µ like above, we can conclude that

Hence

E.4 Proof of Theorem 7,8 and 9

Proof. The proof is similar to the proof of Theorem 5 and Theorem 6.

E.5 Proof of Lemma 1

Proof.

Let λ 1 ( ∇ 2 g µ ( x, y )) , λ 2 ( ∇ 2 g µ ( x, y )) be the eigenvalue of matrix ∇ 2 g µ ( x, y ) , then

Now we calculate the product of eigenvalue

Note that for ( x ∗ µ , y ∗ µ ) , ( x ∗∗∗ µ , y ∗∗∗ µ ) , they satisfy (11a) and (11b), this fact is used in third equality and second ' ⇔ '. By (32b), we know λ 1 ( ∇ 2 g µ ( x, y )) · λ 2 ( ∇ 2 g µ ( x, y )) > 0 for ( x ∗ µ , y ∗ µ ) , ( x ∗∗∗ µ , y ∗∗∗ µ ) , and λ 1 ( ∇ 2 g µ ( x, y )) · λ 2 ( ∇ 2 g µ ( x, y )) < 0 for ( x ∗∗ µ , y ∗∗ µ ) , then

and

Then ( x ∗ µ , y ∗ µ ) , ( x ∗∗∗ µ , y ∗∗∗ µ ) are locally minima, ( x ∗∗ µ , y ∗∗ µ ) is saddle point for g µ ( W ) .

E.6 Proof of Lemma 2

0.

0.1

0.2

0.3

0.4

0.5

0.6

x

0.

0

0.

5

0.10

0.15

0.20

0.25

0.30

0.35

0.40

y

g

(

W

)

(

x

*

,

y

*

)

(

x

* *

,

y

* *

)

(

x

* * *

,

y

* * *

)

g

(

W

)

x

=0

g

(

W

)

y

=0

Proof. Let us define the functions as below

with simple calculations,

and

Here we divide B µ into three parts, C µ 1 , C µ 2 , C µ 3

Also note that

The gradient flow follows

then glyph[negationslash]

Note that ‖∇ g µ ‖ is not diminishing and bounded away from 0. Let us consider the ( x (0) , y (0)) ∈ C µ 1 , since ∇ g µ ( x, y ) = 0 , -∇ g µ ( x, y ) < 0 in (42) and boundness of C µ 1 , it implies there exists a finite t 0 > 0 such that

where ∂C µ 1 is defined as

For the same reason, if ( x (0) , y (0)) ∈ C µ 2 , there exists a finite time t 1 > 0 ,

where ∂C µ 2 is defined as

then by lemma 7, lim t →∞ ( x ( t ) , y ( t )) = ( x ∗ µ , y ∗ µ )

E.7 Proof of Lemma 3

Proof. This is just a result of the Theorem 5.

E.8 Proof of Lemma 5

Proof. Note that

Let ‖ · ‖ op is the spectral norm, and it satisfies triangle inequality

The spectral norm of the second term in area A is bounded by

We use x 2 ≤ a 2 , y 2 ≤ a 2 in the inequality. Therefore,

Also, according to [5, 33], for any f , if ∇ 2 f exists, then f is L smooth if and only if |∇ 2 f | op ≤ L . With this, we conclude the proof.

E.9 Proof of Lemma 7

Proof. First we prove ∀ t ≥ 0 , ( x ( t ) , y ( t )) ∈ C µ 3 , because if ( x ( t ) , y ( t )) / ∈ C µ 3 , then there exists a finite t such that

where ∂C µ 3 is the boundary of C µ 3 , defined as

In our setting, ∀ ( x, y ) ∈ C µ 3

glyph[negationslash]

W.L.O.G, let us assume ( x (0) , y (0)) ∈ ∂C µ 3 and ( x (0) , y (0)) = ( x ∗ µ , y ∗ µ ) . Here are four different cases, glyph[negationslash]

This indicates that -∇ g µ ( x ( t ) , y ( t )) are pointing to the interior of C µ 3 , then ( x ( t ) , y ( t )) can not escape C µ 3 . Here we can focus our attention in C µ 3 , because ∀ t ≥ 0 , ( x ( t ) , y ( t )) ∈ C µ 3 . For Algorithm 1,

glyph[negationslash]

glyph[negationslash]

Plus, ( x ∗ µ , y ∗ µ ) is the unique stationary point of g µ ( W ) in C µ 3 . By lemma 8

glyph[negationslash]

By Lyapunov asymptotic stability theorem [28], and applying it to gradient flow for g µ ( x, y ) in C µ 3 , we can conclude lim t →∞ ( x ( t ) , y ( t )) = ( x ∗ µ , y ∗ µ ) .

E.10 Proof of Lemma 8

Proof. For any ( x, y ) ∈ C µ 3 in 41, and ( x, y ) = ( x ∗ µ , y ∗ µ ) , in Algorithm 7. W.L.O.G, we can assume x ∈ ( x ∗∗ µ , x ∗ µ ) , the analysis details can also be applied to x ∈ ( x ∗ µ , a ) . It is obvious that ˜ x j < ˜ x j +1 and ˜ y j +1 < ˜ y j . Also, lim j →∞ (˜ x j , ˜ y j ) = ( x ∗ µ , y ∗ µ ) . Otherwise either ˜ x j = x ∗ µ or ˜ y j = y ∗ µ hold, Algorithm 7 continues until lim j →∞ (˜ x j , ˜ y j ) = lim j →∞ ( y µ 2 (˜ y j ) , x µ 1 (˜ x j )) , i.e. (˜ x j , ˜ y j ) converges to ( x ∗ µ , y ∗ µ ) .

Moreover, note that for any j = 0 , 1 , . . .

so

Because

Note that

By the same reason, glyph[negationslash]

By Lemma 1, ( x ∗ µ , y ∗ µ ) is local minima, and there exists a r µ > 0 and any { ( x, y ) | ‖ ( x, y ) -( x ∗ µ , y ∗ µ ) ‖ 2 ≤ r µ } , g µ ( x, y ) > g µ ( x ∗ µ , y ∗ µ ) Since lim j →∞ (˜ x j , ˜ y j ) = ( x ∗ µ , y ∗ µ ) , there exists a J > 0 such that ∀ j > J , ‖ (˜ x j , ˜ y j ) -( x ∗ µ , y ∗ µ ) ‖ 2 ≤ r µ , combining them all

glyph[negationslash]

glyph[negationslash]

Algorithm 7: Path goes to ( x ∗ µ , y ∗ µ )

E.11 Proof of Lemma 4

Proof. From the proof of Theorem 1, any any scheduling for µ k satisfies following will do the job

Note that in Algorithm 4, we have ̂ a = √ 4( µ 0 + ε ) < a , then it is obvious

The same analysis for Theorem 1 can be applied here.

E.12 Proof of Lemma 6

Proof. By the Theorem 3 and Lemma 5 and the fact that A 1 µ,glyph[epsilon1] is µ -stationary point region, we use the same argument as proof of Lemma 7 to demonstrate the gradient descent will never go to A 2 µ,glyph[epsilon1] .

E.13 Proof of Lemma 9

Because τ β > τ , so

E.14 Proof of Corollary 1

Proof. Note that as (38a),(37b)

We also know from the proof of Corollary 3, x ∗∗ µ,glyph[epsilon1] < x ∗∗ µ,β and x ∗ µ,β < x ∗ µ,glyph[epsilon1] . Consequently,

when a > √ 5 27 , then a 2 4 > µ 0 = 1 27 ≥ a 2 4( a 2 +1) 3 , it satisfies condition in Lemma 4, we obtain the same result.

E.15 Proof of Corollary 2

Proof. Use Theorem 5(vi) and Theorem 6(vi).

E.16 Proof of Corollary 3

Proof. It is easy to know that and

and when µ < τ , there are three solutions to r ( y ; µ ) = 0 by Theorem 5. Also, we know from Theorem 7, 8

Note that when ( 1+ β 1 -β ) 2 ≤ a 2 +1

Therefore,

Also, we know that for y ub defined in Theorem 5(iii), we know r ( y ub ; µ ) > 0 from Theorem 5(iv). Therefore,

Besides, √ µ < y ub . By monotonicity of r β ( y ; µ ) and r glyph[epsilon1] ( y ; µ ) from the Theorem 7(ii) and Theorem 8(ii), it implies that there are at least two solutions to r β ( y ; µ ) and r glyph[epsilon1] ( y ; µ ) . From the geometry of r β ( y ; µ ) , r glyph[epsilon1] ( y ; µ ) , r ( y ; µ ) and t β ( x ; µ ) , t glyph[epsilon1] ( x ; µ ) , t ( x ; µ ) , it is trivial to know that x ∗ µ,glyph[epsilon1] ≤ x ∗ µ , y ∗ µ,glyph[epsilon1] ≥ y ∗ µ , x ∗∗ µ,glyph[epsilon1] ≥ x ∗∗ µ , y ∗ µ,glyph[epsilon1] ≤ y ∗∗ µ .

Finally, for every point ( x, y ) ∈ A 1 µ,glyph[epsilon1] , there exists a pair glyph[epsilon1] 1 , glyph[epsilon1] 2 , each satisfying | glyph[epsilon1] 1 | ≤ glyph[epsilon1] and | glyph[epsilon1] 2 | ≤ glyph[epsilon1] , such that ( x, y ) is the solution to

We can repeat the same analysis above to show that x ∗ µ,glyph[epsilon1] ≤ x , y ∗ µ,glyph[epsilon1] ≥ y . Applying the same logic to ∀ ( x, y ) ∈ A 2 µ,glyph[epsilon1] , we find x ∗∗ µ,glyph[epsilon1] ≥ x , y ∗ µ,glyph[epsilon1] ≤ y . Thus, ( x ∗ µ , y ∗ µ ) is the extreme point of A 1 µ,glyph[epsilon1] and ( x ∗∗ µ , y ∗∗ µ ) is the extreme point of A 2 µ,glyph[epsilon1] , we get the results.

F Experiments Details

In this section, we present experiments to validate the global convergence of Algorithm 6. Our goal is twofold: First, we aim to demonstrate that irrespective of the starting point, Algorithm 6 using gradient descent consistently returns the global minimum. Second, we contrast our updating scheme for µ k , glyph[epsilon1] k as prescribed in Algorithm 6 with an arbitrary updating scheme for µ k , glyph[epsilon1] k . This comparison illustrates how inappropriate setting of parameters in gradient descent could lead to incorrect solutions.

F.1 Random Initialization Converges to Global Optimum

1.0

0.5

0.0

0.5

1.0

1.5

2.0

x

0.0

0.1

0.2

0.3

0.4

y

start

end

Algorithm 6 for

a

=2

=0.01

=0.4

0 =0.2034

(a) Random Initialization 1

1.0

0.5

0.0

0.5

1.0

1.5

2.0

x

0.4

0.3

0.2

0.1

0.0

0.1

0.2

0.3

0.4

y

start

end

Algorithm 6 for

a

=2

=0.01

=0.4

0 =0.2034

(b) Random Initialization 2

0.0

0.5

1.0

1.5

2.0

x

0.4

0.2

0.0

0.2

0.4

y

start

end

Algorithm 6 for

a

=2

=0.01

=0.4

0 =0.2034

(c) Random Initialization 3

0.0

0.5

1.0

1.5

2.0

x

0.0

0.1

0.2

0.3

0.4

0.5

0.6

y

start

end

Algorithm 6 for

a

=2

=0.01

=0.4

0 =0.2034

(d) Random Initialization 4

0.2

0.0

0.2

0.4

0.6

x

0.0

0.1

0.2

0.3

0.4

0.5

y

start

end

Algorithm 6 for

a

=0.5

=0.01

=0.1

0 =0.0429

(a) Random Initialization 1

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

x

0.4

0.2

0.0

0.2

0.4

y

start

end

Algorithm 6 for

a

=0.5

=0.01

=0.1

0 =0.0429

(b) Random Initialization 2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

x

0.0

0.1

0.2

0.3

0.4

0.5

0.6

y

start

end

Algorithm 6 for

a

=0.5

=0.01

=0.1

0 =0.0429

(c) Random Initialization 3

0.0

0.1

0.2

0.3

0.4

0.5

0.6

x

0.0

0.1

0.2

0.3

0.4

0.5

y

start

end

Algorithm 6 for

a

=0.5

=0.01

=0.1

0 =0.0429

(d) Random Initialization 4

F.2 Wrong Specification of δ Leads to Spurious Local Optimial

0.2

0.0

0.2

0.4

0.6

x

0.0

0.1

0.2

0.3

0.4

0.5

y

start

end

Algorithm 6 for

a

=0.5

=0.01

=0.4

0 =0.0127

(a)

δ

= 0

.

4

0.2

0.0

0.2

0.4

0.6

x

0.0

0.1

0.2

0.3

0.4

0.5

y

start

end

Algorithm 6 for

a

=0.5

=0.01

=0.1

0 =0.0429

(b)

δ

= 0

.

1

F.3 Wrong Specification of β Leads to Incorrect Solution

0.0

0.5

1.0

1.5

2.0

x

0.0

0.1

0.2

0.3

0.4

0.5

y

start

end

Algorithm 6 for

a

=2

=0.6

=0.4

0 =0.0022

(a)

β

= 0

.

6

0.0

0.5

1.0

1.5

2.0

x

0.0

0.1

0.2

0.3

0.4

0.5

y

start

end

Algorithm 6 for

a

=2

=0.01

=0.4

0 =0.2034

(b)

δ

= 0

.

01

F.4 Faster decrease of µ k Leads to Incorrect Solution

0.0

0.1

0.2

0.3

0.4

0.5

0.6

x

0.0

0.1

0.2

0.3

0.4

0.5

y

start

end

Algorithm 6 for

a

=0.5

=0.01

=0.15

0 =0.0361

(a) Bad scheduling

0.0

0.1

0.2

0.3

0.4

0.5

0.6

x

0.0

0.1

0.2

0.3

0.4

0.5

y

start

end

Algorithm 6 for

a

=0.5

=0.01

=0.15

0 =0.0361

(b) Good scheduling

Peer reviews:
comments: SUMMARY:
The authors give a simple optimization algorithm for DAG-learning-inspired optimization problems that avoids the limitations of known techniques.

STRENGTHS:
Originality:
Work is original.
I particularly liked the reduction from a combinatorial problem to a non-convex optimization one.

Quality:
Simple and strong paper.

Clarity:
Clear writing, which can be improved; see questions.

Significance:
Significant topic/contributions.

WEAKNESSES:
No significant weaknesses found.

QUESTIONS:
Discussion around Equation (1) is very clean.

Line 49:
Can you please explain Equation (2) a bit more?

Line 64:
Is it really easy to see? :)

Line 90:
Please explain "homotopy."

Remark 2:
Can you please further explain the difficulties that you mention here?

Can you please elaborate on Equation (7)?

Line 217:
Can you please elaborate on bounding $a$?

Please add more details in the caption of Figure 2.

Line 235:
Why is the interesting regime the one where $\mu < \tau$?

Line 258:
Not clear!

Line 261:
"we know" instead of "we known."

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-22

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of learning the correct Directed Acyclic Graph (DAG) that describes the data using continuous optimization. The connection of this problem with continuous methods comes from a prior work of Zheng et al., where they introduce a differentiable function $h$ whose level set at 0 exactly characterizes DAGs. The problem then becomes to optimize a score function of the data subject to the constraint that h is 0. This is a non-convex problem, since h is non-convex. The standard way of solving it is by converting it to an unconstrained problem and penalizing the solutions with high values of h. This paper provides a way of solving a sequence of these problems with varying penalty parameters, so that the final output provably converges to the true solution, when we have two nodes in the DAG.  The model is analyzed in the population case, i.e. when we have access to infinite data.  The authors first show that for sufficiently small values of the penalty parameter, the loss landscape is not benign, as there exists a saddle point and a spurious local minimum in addition to the global minimum. Using this observation, they show a way to pick a sequence of penalty parameters, which provides a solution path that avoids the local minima and converges provably to the true model asymptotically. Their main result is phrased in terms of gradient flow, but they also provide a finite step analysis for gradient descent.\n\nSTRENGTHS:\nConsider the optimization problem \n\\begin{equation}\\label{eq1}\n\\min_\\Theta g_\\mu(\\Theta) := \\mu f(\\Theta) + h(W(\\Theta))\n\\end{equation}\nwhere $f$ is the least squares loss, $\\mu$ is the penalty parameter, $\\Theta$ represents the model parameters and $W(\\Theta)$ represents the adjacency matrix of the DAG. $f$ is a function that was introduced in prior work, which is $0$ if and only if $W(\\Theta)$ corresponds to a DAG. \nThis paper is the first that provides theoretical guarantees about the optimization landscape of this unconstrained problem for various values of $\\mu$, in the case of $2$ nodes. It is worth noting that this is actually the method that is used in practice to learn DAGs, since enumeration methods quickly become intractable when the number of nodes in the DAG increases.\nThe authors establish interesting properties of the solutions space, namely that if $\\mu$ is sufficiently small, the loss is not benign. \nAlso, the idea of successively solving different optimization problems and using the solution of the last as a starting point for the next is an intriguing one and could have applications beyond this work. The proof strategy is also very intuitive and elegant, by choosing $\\mu$ so that the optimizer of $g_\\mu$ lies in the basin of attraction of the global minimum of the next problem. \nAll arguments are very clearly explained, which makes this paper enjoyable to read.\n\nWEAKNESSES:\n-The result is only proven for $d=2$ nodes in the DAG. This makes most calculations tractable, since there is a closed form for the loss $g_\\mu$ (a quadratic polynomial in two variables). Hence, while the proof strategy is elegant and novel, the technical arguments required to establish the claims do not have significant innovation.\n\nQUESTIONS:\n-It would be interesting to see whether similar results hold for higher values of $d$. Have the authors tried to run simulations for $d=3$ or $4$ and observe similar solution landscapes? What about other choices of loss function or penalty?\n-Is it always true that there is a unique DAG that describes the data? For example, for two nodes, suppose we have $X_2 = X_1 + \\epsilon$, where $\\epsilon\\sim \\mathcal{N}(0,1)$. Then, clearly it is also true that $X_1 = X_2 + \\epsilon'$, where $\\epsilon' \\sim \\mathcal{N}(0,1)$. In that case, it seems that we have two solutions $x=1,y=0$ and $x=0,y=1$. I'm curious where would this come up in the analysis.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-30"}

comments: SUMMARY:
This paper presents a novel approach to the non-convex optimization problems associated with learning the structure of a structural equation model (SEM) or Bayesian network. Considering the equivalent penalty form, the authors propose a homotopy-based optimization scheme that finds global minimizers of the problem by iteratively decreasing the penalty coefficient according to a given schedule. They prove that this algorithm converges globally to the global minimum, regardless of the initialization for W. The authors also demonstrate that the non-convex program is non-benign, meaning that naïve implementation of black-box solvers are likely to get trapped in a bad local minimum.

STRENGTHS:
I am not an expert in this field. But the paper's findings seems to have significant implications in the context of learning the structure of SEMs or Bayesian networks.  The paper is well-structured and the authors clearly explain their methodology and findings. They also provide a clear visualization of the non-convex landscape and the solution trajectory.

WEAKNESSES:
The authors' approach is primarily focused on the bivariate case, which may limit its applicability in more complex settings. Some numerical study on large-scale problems is desirable.

QUESTIONS:
How does the landscape look like in high dimensional setting?  Can you provide any experiments to more than two variables?

LIMITATIONS:
Does not apply
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nthe paper provides a theoretical study on the loss landscape and convergence in gradient-based DAG learning framework. By focusing on linear functions with the number of variable d = 2, They provide a homotopy-based optimization scheme to guarantee the global optimality. Some numerical validations are provided.\n\nSTRENGTHS:\nthe paper focuses on the theoretical study on the global optimality and convergence. This is an important but difficult questions. The theoretical results mainly shows the initialization regime as requirements for convergence, which makes sense given the nonconvex nature, although I did not check the proof in details (given its length).\n\nIllustrative examples and results are provided.\n\nWEAKNESSES:\nUnfortunately, the current results are only applicable to d=2 and linear functions (understandably, of course). No discussions on how nonlinear or  a large number of variables would affect the loss landscape, and/or how the homotopy algorithm would be affected.\n\nWould be more interesting if the insights of the theoretical results can be used in practical algorithms.\n\nQUESTIONS:\n- \"...bears a resemblance to the success of training deep models, which started with AlexNet for image classification\": First of all, modern deep models have success before alexnet on imagenet, so this is not accurate. Second, such a statement is over the top and should be removed.\n- L82: indeed only two DAGs are in the entire search space yet the analysis is quite complex. There is no indication on how such analysis and approach would scale or behave with an increasing number of variables, hence it is hard to judge the usefulness of these analysis yet. In comparison, although the combinatorial search is not used, its expected performance is the same with larger d. \n- Eq 4: x is already used to represent data, and should avoid using it as parameter (lack of notation clarity).\n- Eq 5: while re-ordering variable to obtain such a upper triangular structure is not an issue here yet, in datasets W* would generally not be so. How does a non-upper triangular structure impact analysis, for example a's usage afterward?\n- Some derivation are not given fully. For example, f(W) and h(W) skips many steps in between.\n- L199: what is k here?\n- the regime of initialization for \\mu: very interesting observation.\n- optimization landscape with nonlinear functions: current analysis focuses on linear function. How would it change for nonlinear functions (such neural networks), even for d=2? Given the nonlinear nature, one can imagine it would make the analysis even hard, given two layers of neural networks are the most complex cases that has some theoretical understanding. \n- can the homotopy-based optimization scheme be used to improve convergence of NOTEARS related algorithms?\n\nLIMITATIONS:\nnot discussed but not needed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-31"}