SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chang Deng†* Kevin Bello†,‡ Pradeep Ravikumar‡ Bryon Aragam†
$^{\dagger}$ Booth School of Business, University of Chicago, Chicago, IL 60637  
 $^{\ddagger}$ Machine Learning Department, Carnegie Mellon University, Pittsburgh, PA 15213
Recently, a new class of non-convex optimization problems motivated by the statistical problem of learning an acyclic directed graphical model from data has attracted significant interest. While existing work uses standard first-order optimization schemes to solve this problem, proving the global optimality of such approaches has proven elusive. The difficulty lies in the fact that unlike other non-convex problems in the literature, this problem is not "benign", and possesses multiple spurious solutions that standard approaches can easily get trapped in. In this paper, we prove that a simple path-following optimization scheme globally converges to the global minimum of the population loss in the bivariate setting.
Over the past decade, non-convex optimization has become a major topic of research within the machine learning community, in part due to the successes of training large-scale models with simple first-order methods such as gradient descent—along with their stochastic and accelerated variants—in spite of the non-convexity of the loss function. A large part of this research has focused on characterizing which problems have benign loss landscapes that are amenable to the use of gradient-based methods, i.e., there are no spurious local minima, or they can be easily avoided. By now, several theoretical results have shown this property for different non-convex problems such as: learning a two hidden unit ReLU network [48], learning (deep) over-parameterized quadratic neural networks [43, 27], low-rank matrix recovery [19, 13, 3], learning a two-layer ReLU network with a single non-overlapping convolutional filter [6], semidefinite matrix completion [4, 20], learning neural networks for binary classification with the addition of a single special neuron [30], and learning deep networks with independent ReLU activations [26, 11], to name a few.
Recently, a new class of non-convex optimization problems due to Zheng et al. [51] have emerged in the context of learning the underlying structure of a structural equation model (SEM) or Bayesian network. This underlying structure is typically represented by a directed acyclic graph (DAG), which makes the learning task highly complex due to its combinatorial nature. In general, learning DAGs is well-known to be NP-complete [8, 10]. The key innovation in Zheng et al. [51] was the introduction of a differentiable function  $h$ , whose level set at zero exactly characterizes DAGs. Thus, replacing the challenges of combinatorial optimization by those of non-convex optimization. Mathematically, this class of non-convex problems take the following general form:
$$
\min  _ {\Theta} f (\Theta) \text {s u b j e c t t o} h (W (\Theta)) = 0, \tag {1}
$$
where  $\Theta \in \mathbb{R}^l$  represents the model parameters,  $f: \mathbb{R}^l \to \mathbb{R}$  is a (possibly non-convex) smooth loss function (sometimes called a score function) that measures the fitness of  $\Theta$ , and  $h: \mathbb{R}^{d \times d} \to [0, \infty)$
*changdeng@chicagobooth.edu
is a smooth non-convex function that takes the value of zero if and only if the induced weighted adjacency matrix of  $d$  nodes,  $W(\Theta)$ , corresponds to a DAG.
Given the smoothness of  $f$  and  $h$ , problem (1) can be solved using off-the-shelf nonlinear solvers, which has driven a series of remarkable developments in structure learning for DAGs. Multiple empirical studies have demonstrated that global or near-global minimizers for (1) can often be found in a variety of settings, such as linear models with Gaussian and non-Gaussian noises [e.g., 51, 34, 1], and non-linear models, represented by neural networks, with additive Gaussian noises [e.g., 29, 52, 49, 1]. The empirical success for learning DAGs via (1), which started with the NOTEARS method of Zheng et al. [51], bears a resemblance to the advancements in deep learning, where breakthroughs like AlexNet significantly boosted the field's recognition, even though there were notable successes before it.
Importantly, the reader should note that the majority of applications in ML consist of solving a single unconstrained non-convex problem. In contrast, the class of problems (1) contains a non-convex constraint. Thus, researchers have considered some type of penalty method such as the augmented Lagrangian [51, 52], quadratic penalty [35], and a log-barrier [1]. In all cases, the penalty approach consists in solving a sequence of unconstrained non-convex problems, where the constraint is enforced progressively [see e.g. 2, for background]. In this work, we will consider the following form of penalty:
$$
\min  _ {\Theta} g _ {\mu_ {k}} (\Theta) := \mu_ {k} f (\Theta) + h (W (\Theta)). \tag {2}
$$
It was shown by Bello et al. [1] that due to the invexity property of  $h$ , solutions to (2) will converge to a DAG as  $\mu_k \to 0$ . However, no guarantees on local/global optimality were given.
With the above considerations in hand, one is inevitably led to ask the following questions:
(i) Are the loss landscapes  $g_{\mu_k}(\Theta)$  benign for different  $\mu_k$  ?
(ii) Is there a (tractable) solution path  $\{\Theta_k\}$  that converges to a global minimum of (1)?
Due to the NP-completeness of learning DAGs, one would expect the answer to (i) to be negative in its most general form. Moreover, it is known from the classical theory of constrained optimization [e.g. 2] that if we can exactly and globally optimize (1) for each  $\mu_{k}$ , then the answer to (ii) is affirmative. This is not a practical algorithm, however, since the problem (1) is nonconvex. Thus we seek a solution path that can be tractably computed in practice, e.g. by gradient descent.
In this work, we focus on perhaps the simplest setting where interesting phenomena take place. That is, a linear SEM with two nodes (i.e.,  $d = 2$ ),  $f$  is the population least squared loss (i.e.,  $f$  is convex), and  $\Theta_{k}$  is defined via gradient flow with warm starts. More specifically, we consider the case where  $\Theta_{k}$  is obtained by following the gradient flow of  $g_{\mu_k}$  with initial condition  $\Theta_{k - 1}$ .
Under this setting, to answer (i), it is easy to see that for a large enough  $\mu_{k}$ , the convex function  $f$  dominates and we can expect a benign landscape, i.e., a (almost) convex landscape. Similarly, when  $\mu_{k}$  approaches zero, the invexity of  $h$  kicks in and we could expect that all stationary points are (near) global minimizers. That is, at the extremes  $\mu_{k} \to \infty$  and  $\mu_{k} \to 0$ , the landscapes seem well-behaved, and the reader might wonder if it follows that for any  $\mu_{k} \in [0, \infty)$  the landscape is well-behaved. We answer the latter in the negative and show that there always exists a  $\tau > 0$  where the landscape of  $g_{\mu_k}$  is non-benign for any  $\mu_{k} < \tau$ , namely, there exist three stationary points: i) A saddle point, ii) A spurious local minimum, and iii) The global minimum. In addition, each of these stationary points have wide basins of attractions, thus making the initialization of the gradient flow for  $g_{\mu_k}$  crucial. Finally, we answer (ii) in the affirmative and provide an explicit scheduling for  $\mu_{k}$  that guarantees the asymptotic convergence of  $\Theta_{k}$  to the global minimum of (1). Moreover, we show that this scheduling cannot be arbitrary as there exists a sequence of  $\{\mu_k\}$  that leads  $\{\Theta_k\}$  to a spurious local minimum.
Overall, we establish the first set of results that study the optimization landscape and global optimality for the class of problems (1). We believe that this comprehensive analysis in the bivariate case provides a valuable starting point for future research in more complex settings.
An invex function is any function where all its stationary points are global minima. It is worth noting that the composite objective in (2) is not necessarily invex, even when  $f$  is convex.
This transition or path, from an optimizer of a simple function to an optimizer of a function that closely resembles the original constrained formulation, is also known as a homotopy.
(a) Contour plot
(b) Stationary points
Figure 1: Visualizing the nonconvex landscape. (a) A contour plot of  $g_{\mu}$  for  $a = 0.5$  and  $\mu = 0.005$  (see Section 2 for definitions). We only show a section of the landscape for better visualization. The solid lines represent the contours, while the dashed lines represent the vector field  $-\nabla g_{\mu}$ . (b) Stationary points of  $g_{\mu}$ ,  $r(y; \mu) = 0$  and  $r(x; \mu) = 0$  (see Section 4 for definitions).
Remark 1. We emphasize that solving (1) in the bivariate case is not an inherently difficult problem. Indeed, when there are only two nodes, there are only two DAGs to distinguish and one can simply fit  $f$  under the only two possible DAGs, and select the model with the lowest value for  $f$ . However, evaluating  $f$  for each possible DAG structure clearly cannot scale beyond 10 or 20 nodes, and is not a standard algorithm for solving (1). Instead, here our focus is on studying how (1) is actually being solved in practice, namely, by solving unconstrained non-convex problems in the form of (2). Previous work suggests that such gradient-based approaches indeed scale well to hundreds and even thousands of nodes [e.g. 51, 34, 1].
More specifically, we make the following contributions:
1. We present a homotopy-based optimization scheme (Algorithm 2) to find global minimizers of the program (1) by iteratively decreasing the penalty coefficient according to a given schedule. Gradient flow is used to find the stationary points of (2) at each step, starting from the previous solution.
2. We prove that Algorithm 2 converges globally (i.e. regardless of initialization for  $W$ ) to the global minimum (Theorem 1).
3. We show that the non-convex program (1) is indeed non-benign, and naive implementation of black-box solvers are likely to get trapped in a bad local minimum. See Figure 1 (a).
4. Experimental results verify our theory, consistently recovering the global minimum of (1), regardless of initialization or initial penalty value. We show that our algorithm converges to the global minimum while naive approaches can get stuck.
The analysis consists of three main parts: First, we explicitly characterize the trajectory of the stationary points of (2). Second, we classify the number and type of all stationary points (Lemma 1) and use this to isolate the desired global minimum. Finally, we apply Lyapunov analysis to identify the basin of attraction for each stationary point, which suggests a schedule for the penalty coefficient that ensures that the gradient flow is initialized within that basin at the previous solution.
The class of problems (1) falls under the umbrella of score-based methods, where given a score function  $f$ , the goal is to identify the DAG structure with the lowest score possible [9, 22]. We shall note that learning DAGs is a very popular structure model in a wide range of domains such as biology [40], genetics [50], and causal inference [44, 39], to name a few.
Score-based methods that consider the combinatorial constraint. Given the ample set of score-based methods in the literature, we briefly mention some classical works that attempt to optimize  $f$  by considering the combinatorial DAG constraint. In particular, we have approximate algorithms such as the greedy search method of Chickering et al. [10], order search methods [45, 41, 38], the LP-relaxation method of Jaakkola et al. [24], and the dynamic programming approach of Loh and Buhlmann [31]. There are also exact methods such as GOBNILP [12] and Bene [42], however, these algorithms only scale up to  $\approx 30$  nodes.
Score-based methods that consider the continuous non-convex constraint  $h$ . The following works are the closest to ours since they attempt to solve a problem in the form of (1). Most of these developments either consider optimizing different score functions  $f$  such as ordinary least squares [51, 52], the log-likelihood [29, 34], the evidence lower bound [49], a regret function [53]; or consider different differentiable characterizations of acyclicity  $h$  [49, 1]. However, none of the aforementioned works provide any type of optimality guarantee. Few studies have examined the optimization intricacies of problem (1). Wei et al. [47] investigated the optimality issues and provided local optimality guarantees under the assumption of convexity in the score  $f$  and linear models. On the other hand, Ng et al. [35] analyzed the convergence to (local) DAGs of generic methods for solving nonlinear constrained problems, such as the augmented Lagrangian and quadratic penalty methods. In contrast to both, our work is the first to study global optimality and the loss landscapes of actual methods used in practice for solving (1).
Bivariate causal discovery. Even though in a two-node model the discrete DAG constraint does not pose a major challenge, the bivariate setting has been subject to major research in the area of causal discovery. See for instance [36, 16, 32, 25] and references therein.
Penalty and homotopy methods. There exist classical global optimality guarantees for the penalty method if  $f$  and  $h$  were convex functions, see for instance [2, 5, 37]. However, to our knowledge, there are no global optimality guarantees for general classes of non-convex constrained problems, let alone for the specific type of non-convex functions  $h$  considered in this work. On the other hand, homotopy methods (also referred to as continuation or embedding methods) are in many cases capable of finding better solutions than standard first-order methods for non-convex problems, albeit they typically do not come with global optimality guarantees either. When homotopy methods come with global optimality guarantees, they are commonly computationally more intensive as it involves discarding solutions, thus, closely resembling simulated annealing methods, see for instance [15]. Authors in [21] characterize a family of non-convex functions where a homotopy algorithm provably converges to a global optimum. However, the conditions for such family of non-convex functions are difficult to verify and are very restrictive; moreover, their homotopy algorithm involves Gaussian smoothing, making it also computationally more intensive than the procedure we study here. Other examples of homotopy methods in machine learning include [7, 18, 46, 17, 23], in all these cases, no global optimality guarantees are given.
The objective  $f$  we consider can be easily written down as follows:
$$
f (W) = \frac {1}{2} \mathbb {E} _ {X} \left[ \| X - W ^ {\top} X \| _ {2} ^ {2} \right], \tag {3}
$$
where  $X \in \mathbb{R}^2$  is a random vector and  $W \in \mathbb{R}^{2 \times 2}$ . Although not strictly necessary for the developments that follow, we begin by introducing the necessary background on linear SEM that leads to this objective and the resulting optimization problem of interest.
The bivariate model. Let  $X = (X_{1},X_{2})\in \mathbb{R}^{2}$  denote the random variables in the model, and let  $N = (N_{1},N_{2})\in \mathbb{R}^{2}$  denote a vector of independent errors. Then a linear SEM over  $X$  is defined as  $X = W_{*}^{\top}X + N$ , where  $W_{*}\in \mathbb{R}^{2\times 2}$  is a weighted adjacency matrix encoding the coefficients in the linear model. In order to represent a valid Bayesian network for  $X$  [see e.g. 39, 44, for details], the matrix  $W_{*}$  must be acyclic: More formally, the weighted graph induced by the adjacency matrix  $W_{*}$  must be a DAG. This (non-convex) acyclicity constraint represents the major computational hurdle that must overcome in practice (cf. Remark 1).
The goal is to recover the matrix  $W_{*}$  from the random vector  $X$ . Since  $W_{*}$  is acyclic, we can assume the diagonal of  $W_{*}$  is zero (i.e. no self-loops). Thus, under the bivariate linear model, it then suffices to consider two parameters  $x$  and  $y$  that define the matrix of parameters<sup>4</sup>
$$
W = W (x, y) = \left( \begin{array}{c c} 0 & x \\ y & 0 \end{array} \right) \tag {4}
$$
For notational simplicity, we will use  $f(W)$  and  $f(x,y)$  interchangeably, similarly for  $h(W)$  and  $h(x,y)$ . Without loss of generality, we write the underlying parameter as
$$
W _ {*} = \left( \begin{array}{c c} 0 & a \\ 0 & 0 \end{array} \right) \tag {5}
$$
$$
X = W _ {*} ^ {\top} X + N \Rightarrow \left\{ \begin{array}{l} X _ {1} = N _ {1}, \\ X _ {2} = a X _ {1} + N _ {2}. \end{array} \right.
$$
In general, we only require  $N_{i}$  to have finite mean and variance, hence we do not assume Gaussianity. We assume that  $\mathrm{Var}[N_1] = \mathrm{Var}[N_2]$ , and for simplicity, we consider  $\mathbb{E}[N] = 0$  and  $\mathrm{Cov}[N] = I$ , where  $I$  denotes the identity matrix. Finally, in the sequel we assume w.l.o.g. that  $a > 0$ .
The population least squares. In this work, we consider the population squared loss defined by (3). If we equivalently write  $f$  in terms of  $x$  and  $y$ , then we have:  $f(W) = ((1 - ay)^2 + y^2 + (a - x)^2 + 1)/2$ . In fact, the population loss can be substituted with empirical loss. In such a case, our algorithm can still attain the global minimum,  $W_{\mathrm{G}}$ , of problem (6). However, the output  $W_{\mathrm{G}}$  will serve as an empirical estimation of  $W_{*}$ . An in-depth discussion on this topic can be found in Appendix B
The non-convex function  $h$ . We use the continuous acyclicity characterization of Yu et al. [49], i.e.,  $h(W) = \mathrm{Tr}((I + \frac{1}{d} W \circ W)^d) - d$ , where  $\circ$  denotes the Hadamard product. Then, for the bivariate case, we have  $h(W) = x^2 y^2 / 2$ . We note that the analysis presented in this work is not tailored to this version of  $h$ , that is, we can use the same techniques used throughout this work for other existing formulations of  $h$ , such as the trace of the matrix exponential [51], and the log-det formulation [1]. Nonetheless, here we consider that the polynomial formulation of Yu et al. [49] is more amenable for the analysis.
Remark 2. Our restriction to the bivariate case highlights the simplest setting in which this problem exhibits nontrivial behaviour. Extending our analysis to higher dimensions remains a challenging future direction, however, we emphasize that even in two-dimensions this problem is nontrivial. Our approach is similar to that taken in other parts of the literature that started with simple cases (e.g. single-neuron models in deep learning).
Remark 3. It is worth noting that our choice of the population least squares is not arbitrary. Indeed, for linear models with identity error covariance, such as the model considered in this work, it is known that the global minimizer of the population squared loss is unique and corresponds to the underlying matrix  $W_{*}$ . See Theorem 7 in [31].
Gluing all the pieces together, we arrive to the following version of (1) for the bivariate case:
$$
\min  _ {x, y} f (x, y) := \frac {1}{2} ((1 - a y) ^ {2} + y ^ {2} + (a - x) ^ {2} + 1) \text {s u b j e c t t o} h (x, y) := \frac {x ^ {2} y ^ {2}}{2} = 0. \tag {6}
$$
Moreover, for any  $\mu \geq 0$ , we have the corresponding version of (2) expressed as:
$$
\min  _ {x, y} g _ {\mu} (x, y) := \mu f (x, y) + h (x, y) = \frac {\mu}{2} ((1 - a y) ^ {2} + y ^ {2} + (a - x) ^ {2} + 1) + \frac {x ^ {2} y ^ {2}}{2}. \tag {7}
$$
To conclude this section, we present a visualization of the landscape of  $g_{\mu}(x,y)$  in Figure 1 (a), for  $a = 0.5$  and  $\mu = 0.005$ . We can clearly observe the non-benign landscape of  $g_{\mu}$ , i.e., there exists a spurious local minimum, a saddle point, and the global minimum. In particular, we can see that the basin of attraction of the spurious local minimum is comparable to that of the global minimum, which is problematic for a local algorithm such as the gradient flow (or gradient descent) as it can easily get trapped in a local minimum if initialized in the wrong basin.
4Following the notation in (1), for the bivariate model we simply have  $\Theta \equiv (x,y)$  and  $W(\Theta)\equiv \left( \begin{array}{ll}0 & x\\ y & 0 \end{array} \right)$ . Although  $x$  is used to represent edge weights and not data as in (3), this distinction should not lead to confusion
Algorithm 1: GradientFlow  $(f, z_0)$
1: set  $z(0) = z_0$   
2:  $\frac{d}{dt} z(t) = -\nabla f(z(t))$   
3: return  $\lim_{t\to \infty}z(t)$   
Algorithm 2: Homotopy algorithm for solving (1).  
Input: Initial  $W_0 = W(x_0,y_0), \mu_0 \in \left[\frac{a^2}{4(a^2 + 1)^3}, \frac{a^2}{4}\right)$   
Output:  $\{W_{\mu_k}\}_{k=0}^{\infty}$ $W_{\mu_0} \gets \text{GradientFlow}(g_{\mu_0}, W_0)$   
for  $k = 1,2,\ldots$  do  
    Let  $\mu_k = (2/a)^{2/3} \mu_{k-1}^{4/3}$ $W_{\mu_k} \gets \text{GradientFlow}(g_{\mu_k}, W_{\mu_{k-1}})$   
end
3 A Homotopy-Based Approach and Its Convergence to the Global Optimum
To fix notation, let us write  $W_{k} \coloneqq W_{\mu_{k}} \coloneqq \left( \begin{array}{cc}0 & x_{\mu_{k}} \\ y_{\mu_{k}} & 0 \end{array} \right)$ . and let  $W_{\mathsf{G}}$  denote the global minimizer of (6). In this section, we present our main result, which provides conditions under which solving a series of unconstrained problems (7) with first-order methods will converge to the global optimum  $W_{\mathsf{G}}$  of (6), in spite of facing non-benign landscapes. Recall that from Remark 3, we have that  $W_{\mathsf{G}} = \left( \begin{array}{cc}0 & a\\ 0 & 0 \end{array} \right)$ . Since we use gradient flow path to connect  $W_{\mu_k}$  and  $W_{\mu_{k + 1}}$ , we specify this path in Procedure 1 for clarity. Although the theory here assumes continuous-time gradient flow with  $t \to \infty$ , see Section 5 for an iteration complexity analysis for (discrete-time) gradient descent, which is a straightforward consequence of the continuous-time theory.
In Algorithm 2, we provide an explicit regime of initialization for the homotopy parameter  $\mu_0$  and a specific scheduling for  $\mu_{k}$  such that the solution path found by Algorithm 2 will converge to the global optimum of (6). This is formally stated in Theorem 1, whose proof is given in Section 5.
Theorem 1. For any initialization  $W_0$  and  $a \in \mathbb{R}$ , the solution path provided in Algorithm 2 converges to the global optimum of (6), i.e.,
$$
\lim  _ {k \rightarrow \infty} W _ {\mu_ {k}} = W _ {\mathsf {G}}.
$$
A few observations regarding Algorithm 2: Observe that when the underlying model parameter  $a \gg 0$ , the regime of initialization for  $\mu_0$  is wider; on the other hand, if  $a$  is closer to zero then the interval for  $\mu_0$  is much narrower. As a concrete example, if  $a = 2$  then it suffices to have  $\mu_0 \in [0.008, 1)$ ; whereas if  $a = 0.1$  then the regime is about  $\mu_0 \in [0.0089, 0.01)$ . This matches the intuition that for a "stronger" value of  $a$  it should be easier to detect the right direction of the underlying model. Second, although in Line 3 we set  $\mu_k$  in a specific manner, it actually suffices to have
$$
\mu_ {k} \in \left[ \left(\frac {\mu_ {k - 1}}{2}\right) ^ {2 / 3} \left(a ^ {1 / 3} - \sqrt {a ^ {2 / 3} - \left(4 \mu_ {k - 1}\right) ^ {1 / 3}}\right) ^ {2}, \mu_ {k - 1}\right).
$$
We simply chose a particular expression from this interval for clarity of presentation; see the proof in Section 5 for details.
As presented, Algorithm 2 is of theoretical nature in the sense that the initialization for  $\mu_0$  and the decay rate for  $\mu_{k}$  in Line 3 depend on the underlying parameter  $a$ , which in practice is unknown. In Algorithm 3, we present a modification that is independent of  $a$  and  $W_{*}$ . By assuming instead a lower bound on  $a$ , which is a standard assumption in the literature, we can prove that Algorithm 3 also converges to the global minimum:
Corollary 1. Initialize  $\mu_0 = \frac{1}{27}$ . If  $a > \sqrt{5 / 27}$  then for any initialization  $W_{0}$ , Algorithm 3 outputs the global optimal solution to (6), i.e.
For more details on this modification, see Appendix A.