SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Arun Jambulapati
Simons Institute* jmblpati@berkeley.edu
Kevin Tian
University of Texas at Austin kjtian@cs.utexas.edu
We investigate different aspects of area convexity [She17], a mysterious tool introduced to tackle optimization problems under the challenging  $\ell_{\infty}$  geometry. We develop a deeper understanding of its relationship with more conventional analyses of extragradient methods [Nem04, Nes07]. We also give improved solvers for the subproblems required by variants of the [She17] algorithm, designed through the lens of relative smoothness [BBT17, LFN18].
Leveraging these new tools, we give a state-of-the-art first-order algorithm for solving box-simplex games (a primal-dual formulation of  $\ell_{\infty}$  regression) in a  $d\times n$  matrix with bounded rows, using  $O(\log d\cdot \epsilon^{-1})$  matrix-vector queries. Our solver yields improved runtimes for approximate maximum flow, optimal transport, min-mean-cycle, and other basic combinatorial optimization problems. We also develop a near-linear time algorithm for a matrix generalization of box-simplex games, capturing a family of problems closely related to semidefinite programs recently used as subroutines in robust statistics and numerical linear algebra.
Box-simplex, or  $\ell_{\infty} - \ell_1$ , games are a family of optimization problems of the form
$$
\min  _ {x \in [ - 1, 1 ] ^ {n}} \max  _ {y \in \Delta^ {d}} x ^ {\top} \mathbf {A} y - b ^ {\top} y + c ^ {\top} x. \tag {1}
$$
These problems are highly expressive, as high-accuracy solutions capture linear programs (LPs) as a special case [LS15].<sup>3</sup> More recently, approximation algorithms for the family (1) have been applied to improve runtimes for applications captured by LPs naturally bounded in the  $\ell_{\infty}$  geometry. The current state-of-the-art first-order method for (1) is by Sherman [She17], who used the resulting solver to obtain a faster algorithm for the maximum flow problem. As further consequences, solvers for (1) have sped up runtimes for optimal transport [JST19], a common task in modern computer vision [KPT+17], and min-mean-cycle [AP20], a basic subroutine in solving Markov decision processes [ZP96], among other combinatorial optimization problems [AJJ+22, JJST22].
The runtimes of first-order methods for approximating (1) are parameterized by an additive accuracy  $\epsilon > 0$  and a Lipschitz constant  $L := \| \mathbf{A} \|_{1 \to 1}$ ; in several prominent applications of (1) where the relevant  $\mathbf{A}$  has favorable structure, e.g. the column-sparse edge-vertex incidence matrix of a graph,  $L$  is naturally small. Standard non-Euclidean variants of gradient descent solve a smoothed variant of (1) using  $\widetilde{O}(L^2 \epsilon^{-2})$  matrix-vector products with  $\mathbf{A}$  and  $\mathbf{A}^\top$  [Nes05, KLOS14]. Until
*Work completed at the University of Washington.
†Work completed at Microsoft Research.
This follows as (1) generalizes  $\ell_{\infty}$  regression  $\min_{x\in [-1,1]^n}\| \mathbf{A}x - b\|_{\infty}$ , see Section 3.1 of [ST18].
4 See Section 1.3 for notation used throughout the paper.  $\| \mathbf{A}\|_{1\to 1}$  is the largest  $\ell_1$  norm of any column of  $\mathbf{A}$ .
recently however, the lack of accelerated algorithms for (1) (using a quadratically smaller  $\widetilde{O}(L\epsilon^{-1})$  matrix-vector products) was a notorious barrier in optimization theory. This barrier arises due to the provable lack of a strongly convex regularizer in the  $\ell_{\infty}$  geometry with dimension-independent additive range [ST18],<sup>5</sup> a necessary component of standard non-Euclidean acceleration schemes.
Sherman's breakthrough work [She17] overcame this obstacle and gave an algorithm for solving (1) using roughly  $O(L\log(d)\log(L\epsilon^{-1}) \cdot \epsilon^{-1})$  matrix-vector products. Roughly speaking, Sherman exploited the primal-dual nature of the problem (1) to design a smaller regularizer over the  $\ell_{\infty}$  ball  $[-1,1]^n$ , which satisfied a weaker condition known as area convexity (rather than strong convexity). The algorithm in [She17] combined an extragradient method "outer loop" requiring  $O(L\log d \cdot \epsilon^{-1})$  iterations, and an alternating minimization subroutine "inner loop" solving the subproblems required by the outer loop in  $O(\log(L\epsilon^{-1}))$  steps. However, Sherman's analysis in [She17] was quite ad hoc, and its relationship to more standard optimization techniques has remained mysterious. These mysteries have made further progress on (1) and related problems challenging in several regards.
1. Sherman's algorithm pays a logarithmic overhead in the complexity of its inner loop to solve subproblems to high accuracy. Does it tolerate more approximate subproblem solutions?
2. The analysis of Sherman's alternating minimization scheme relies on multiplicative stability properties of the dual variable  $y$ . In generalizations of (1) where the dual variable is unstable, how do we design solvers handling instability to maintain fast subproblem solutions?
3. The relationship between the area convexity definition [She17] and more standard Bregman divergence domination conditions used in classical analyses of extragradient methods [Nem04, Nes07] is unclear. Can we unify these extragradient convergence analyses?
A central goal of this work is to answer Question 3 by revisiting the area convexity technique of [She17], and developing a deeper understanding of its relationship to more standard tools in optimization theory such as relative smoothness [BBT17, LFN18] and the classical extragradient methods of [Nem04, Nes07]. We summarize these insights in Section 2 and give self-contained expositions in Appendices A and B of the supplement. As byproducts, our improved understanding of area convexity results in new state-of-the-art runtimes for (1) by affirmatively answering Question 1, and the first accelerated solver for a matrix generalization by affirmatively answering Question 2.
Box-simplex games. We give an algorithm for solving (1) improving the runtime of [She17] by a logarithmic factor, by removing the need for high-accuracy inner loop solutions.<sup>7</sup>
Theorem 1. There is an algorithm (stated in full as Algorithm 4 in the supplement) deterministically computing an  $\epsilon$ -approximate saddle point to (1) in time
$$
O \left(\mathrm {n n z} (\mathbf {A}) \cdot \frac {\| \mathbf {A} \| _ {1 \rightarrow 1} \log d}{\epsilon}\right).
$$
We define  $\mathrm{nnz}$  and  $\| \mathbf{A}\|_{1\to 1}$  in Section 1.3. The most direct comparison to Theorem 1 is the work [She17], which we improve. All other first-order methods for solving (1) we are aware of are slower than [She17] by at least  $\approx \min (\epsilon^{-1},\sqrt{n})$  factor. Higher-order methods for (1) (e.g. interior point methods) obtain improved dependences on  $\epsilon$  at the cost of polynomial overhead in the dimension.
As a byproduct of Theorem 1, we improve all the recent applications of (1), including approximate maximum flow [She17], optimal transport [JST19], min-mean-cycle [AP20], and semi-streaming variants of bipartite matching and transshipment  $\mathrm{[AJ^{+}22]}$ . These are summarized in Section 5 of the supplement. The design of efficient approximate solvers for optimal transport in particular is a problem which has received a substantial amount of attention from the learning theory community
For a function  $f$ , we call max  $f - \min f$  its additive range.
The runtime for solving subproblems required by [She17] was not explicitly bounded in that work, but a simple initialization strategy bounds the runtime by  $O(\log (Le^{-1}))$ , see Proposition 1 of [AJJ+22].
This statement assumes direct access to the entries of  $\mathbf{A}$  (as does [She17]). More generally, letting  $\mathcal{T}_{\mathrm{mv}}$  be the time it takes to multiply a vector by the argument, we may replace  $\mathrm{nnz}(\mathbf{A})$  by  $\max (\mathcal{T}_{\mathrm{mv}}(\mathbf{A}),\mathcal{T}_{\mathrm{mv}}(\mathbf{A}^{\top}),\mathcal{T}_{\mathrm{mv}}(|\mathbf{A}|),\mathcal{T}_{\mathrm{mv}}(|\mathbf{A}^{\top}|))$  where  $|\cdot |$  is entrywise, which may be a significant savings when  $\mathbf{A}$  is implicit and nonnegative. An  $\epsilon$ -approximate saddle point is a point with duality gap  $\epsilon$ ; see Section 1.3.
(see Table 1 and the survey [PC19]). The runtime of Theorem 1 represents a natural conclusion to this line of work. We believe our improvement in Theorem 1 (avoiding a high-accuracy subproblem solve) takes an important step towards bridging the state-of-the-art in the theory and practice of optimal transport, a well-motivated undertaking due to its widespread use in applications.
Table 1: Runtime complexities of first-order algorithms for optimal transport. Stated for a  $d \times d$  cost matrix with unit-bounded costs, additive error tolerance  $\epsilon$ . Results labeled "sequential" require a parallel depth which scales polynomially in the problem dimension  $d$ .
Method	Runtime	Comments
[AWR17]	O(d2 log d · ε-3)
[DGK18]	O(d2 log d · ε-2)
[DGK18, LHJ19]	O(d2.5√log d · ε-1)
[BJKS18, Qua20]	O(d2 log(d) log(ε-1) · ε-1)	sequential, randomized
[LMR19]	O(d2 · ε-1 + d · ε-2)	sequential
[JST19]	O(d2 log(d) log(ε-1) · ε-1)
[LHJ22]	O(d7/3√log d · ε-4/3)
Theorem 1	O(d2 log d · ε-1)
Box-spectraplex games. We initiate the algorithmic study of box-spectraplex games of the form
$$
\min  _ {x \in [ - 1, 1 ] ^ {n}} \max  _ {\mathbf {Y} \in \Delta^ {d \times d}} \left\langle \mathbf {Y}, \sum_ {i \in [ n ]} x _ {i} \mathbf {A} _ {i} \right\rangle - \langle \mathbf {B}, \mathbf {Y} \rangle + c ^ {\top} x, \tag {2}
$$
where  $\{\mathbf{A}_i\}_{i\in [n]}$ ,  $\mathbf{B}$  are  $d\times d$  symmetric matrices and  $\mathbf{Y}\succeq \mathbf{0}$  satisfies  $\mathrm{Tr}(\mathbf{Y}) = 1$  (i.e.  $\mathbf{Y}\in \Delta^{d\times d}$ , cf. Section 1.3). This family of problems is a natural formulation of semidefinite programming (SDP), and generalizes (1), the special case of (2) where all of the matrices  $\{\mathbf{A}_i\}_{i\in [n]},\mathbf{B},\mathbf{Y}$  are diagonal.
Our main result on solving (2), stated below as Theorem 2, has already found use in [JRT23] to obtain faster solvers for spectral sparsification and related discrepancy-theoretic primitives, where existing approximate SDP solvers do not apply. Beyond the fundamental nature of the problem solved by Theorem 2, we are optimistic that it will find use in other applications of structured SDPs.
To motivate our investigation of (2), we first describe some history of the study of the simpler problem (1). Many applications captured by (1) are structured settings where optimization is at a relative scale, i.e. we wish to obtain an  $(1 + \epsilon)$ -multiplicative approximation to the value of a LP, OPT :=  $\min_{\mathbf{A}x\leq b}c^{\top}x$ . We use bipartite matching as an example:  $x$  is a fractional matching and  $\mathbf{A}$  is a nonnegative edge-vertex incidence matrix. For LPs with a nonnegative constraint matrix  $\mathbf{A}$ , custom positive LP solvers achieve  $(1 + \epsilon)$ -multiplicative approximations at accelerated rates scaling as  $\widetilde{O} (\epsilon^{-1})$ . This result then implies a  $\widetilde{O} (\epsilon^{-1})$ -rate algorithm for approximating bipartite matching and its generalization, optimal transport (where the first such algorithm used positive LP solvers [BJKS18, Qua20]). However, existing accelerated positive LP solvers [ZO15] are both sequential and randomized, and whether this is necessary has persisted as a challenging open question.
In several applications with nonnegative constraint matrices [JST19, AP20,  $\mathrm{AJJ}^{+}22$ ], this obstacle was circumvented by recasting the problem as a box-simplex game (1), for which faster, deterministic, and efficiently-parallelizable solvers exist (see e.g. Section 4.1 of  $\mathrm{[AJJ^{+}22]}$  for the bipartite matching reduction). In these cases, careful use of box-simplex game solvers (and binary searching for the problem scale) match the guarantees of positive LP solvers, with improved parallelism or determinism. Notably, simpler primitives such as simplex-simplex game solvers do not apply in these settings.
The current state-of-affairs in applications of fast SDP solvers is very similar. For various problems in robust statistics (where the goal is to approximate a covariance matrix or detect a corruption
Progress on the historically simpler problem of (simplex-simplex) matrix games, a relative of (1) where both players are  $\ell_1$  constrained, has stalled since [Nem04] almost 20 years ago, which attains a deterministic runtime of  $O(\mathrm{nnz}(\mathbf{A})\cdot L\log d\cdot \epsilon^{-1})$  where  $L$  is the appropriate Lipschitz constant, analogously to Theorem 1.
Two notable additional problems captured by box-simplex games, maximum flow [She17] and transshipment  $\mathrm{[AJJ^{+}22]}$ , do not have a nonnegative constraint matrix (as flows are signed), so positive LP solvers do not apply.
spectrally) [CDG19, CMY20, JLT20] and numerical linear algebra (where the goal is to reweight or sparsify a matrix sum) [LS17, CG18,  $\mathrm{JLM}^{+}21$ ], positive SDP solvers have found utility. However, these uses appear quite brittle: current positive SDP solvers [ZLO16, PTZ16, JLT20] only handle the special case of packing SDPs (a special class of positive SDP with one-sided constraints), preventing their application in more challenging settings (including pure covering SDPs).
In an effort to bypass this potential obstacle when relying on positive SDP solvers more broadly, we therefore develop a nearly-linear time solver for box-spectraplex games in Theorem 2 (we define  $\mathcal{T}_{\mathrm{mv}}$ , the time required to perform a matrix-vector product, in Section 1.3).
Theorem 2. There is an algorithm which computes an  $\epsilon$ -approximate saddle point to (2) in time
$$
O \left(\left(\mathcal {T} _ {\mathrm {m v}} (\mathbf {B}) + \sum_ {i \in [ n ]} \left(\mathcal {T} _ {\mathrm {m v}} (\mathbf {A} _ {i}) + \mathcal {T} _ {\mathrm {m v}} (| \mathbf {A} _ {i} |)\right)\right) \cdot \frac {L ^ {3 . 5} \log^ {3} \left(\frac {L n d}{\delta \epsilon}\right)}{\epsilon^ {3 . 5}}\right),
$$
with probability  $\geq 1 - \delta$ , for  $L := \|\sum_{i \in [n]} |\mathbf{A}_i\|_{\mathrm{op}} + \|\mathbf{B}\|_{\mathrm{op}}$ . A deterministic variant uses time
$$
O \left(\left(\left(\mathcal {T} _ {\mathrm {m v}} (\mathbf {B}) + \sum_ {i \in [ n ]} \left(\mathcal {T} _ {\mathrm {m v}} (\mathbf {A} _ {i}) + \mathcal {T} _ {\mathrm {m v}} (| \mathbf {A} _ {i} |)\right)\right) \cdot d + d ^ {\omega}\right) \cdot \frac {L \log \left(\frac {L n d}{\delta \epsilon}\right)}{\epsilon}\right).
$$
Theorem 2 follows as a special case of Theorem 3 in the supplement, which has refined guarantees stated in terms of the cost to perform certain queries of matrix exponentials required by our algorithm, typical of first-order methods over  $\Delta^{d\times d}$ . The first runtime in Theorem 2 implements these queries using randomized sketching tools, and the second uses an exact implementation. Beyond its generic utility, we find Theorem 2 conceptually interesting, as its development used our relative smoothness viewpoint on area convexity in Section 2 (for overcoming instability of dual matrix variables), and new techniques for proving area convexity of a matrix variant of the [She17] regularizer.
To put Theorem 2 in context, we compare it to existing solvers for simplex-spectraplex games, a relative of (2) where both players are  $\ell_1$  constrained. Simplex-spectraplex games are more well-studied [KV05, WK06, AK07, BBN13, AL17, CDST19], and the state-of-the-art algorithms [BBN13, AL17, CDST19] query  $\widetilde{O}(L^{2.5}\epsilon^{-2.5})$  vector products in  $\{\mathbf{A}_i\}_{i\in [n]}$  and  $\mathbf{B}$ . A slight variant of these algorithms (using a separable regularizer in place of area convex techniques) gives a complexity of  $\widetilde{O}(n \cdot L^{2.5}\epsilon^{-2.5})$  such queries for the more challenging box-spectraplex games (2), which is the baseline in the literature. In comparison, Theorem 2 requires  $\widetilde{O}(L^{3.5}\epsilon^{-3.5})$  products, but assumes access to  $\{| \mathbf{A}_i | \}_{i\in [n]}$ . This is not without loss of generality, as  $|\mathbf{A}|$  can be dense even when  $\mathbf{A}$  is sparse, though in important robust statistics or spectral sparsification applications (where  $\{\mathbf{A}_i\}_{i\in [n]}$  are rank-one), it is not restrictive. We find it interesting to understand whether access to  $\{| \mathbf{A}_i | \}_{i\in [n]}$  and the  $\widetilde{O}(L\epsilon^{-1})$  overhead in Theorem 2 (due to higher-rank randomized sketches) can be avoided.
The family of box-simplex games (1) is captured by linear programming [LS15], where state-of-the-art solvers [CLS21, vdBLL $^{+}$ 21] run in time  $\widetilde{O}((n + d)^{\omega})$  or  $\widetilde{O}(nd + \min(n, d)^{2.5})$ , where  $\omega \approx 2.37$  is the current matrix multiplication constant [AW21, DWZ22]. These LP solvers run in superlinear time, and practical implementations do not currently exist; on the other hand, the convergence rates depend polylogarithmically on the accuracy parameter  $L\epsilon^{-1}$ . The state-of-the-art approximate solver for (1) (with runtime depending linearly on the input size  $\mathrm{nnz}(\mathbf{A})$  and polynomially on  $L\epsilon^{-1}$ ) is the accelerated algorithm of [She17], which is improved by our Theorem 1 by a logarithmic factor. Finally, we mention [BSW19] as another recent work which relies on area convexity techniques to design an algorithm in a different problem setting; their inner loop also requires high-precision solves (as in [She17]), and this similarly results in a logarithmic overhead in the runtime.
For the specific application of optimal transport, a problem captured by (1) receiving significant recent attention from the learning theory community, several alternative solvers have been developed beyond Table 1. By exploiting relationships between Sinkhorn regularization and faster solvers for matrix scaling [CMTV17, ALdOW17], [BJKS18] gave an alternative solver obtaining a  $\widetilde{O}(n^2 \cdot \epsilon^{-1})$  rate. These algorithms call specialized graph Laplacian system solvers as a subroutine, which are currently
less practical than our methods based on matrix-vector queries. Finally, the recent breakthrough maximum flow algorithm of  $\mathrm{[CKL^{+}22]}$  extends to solve optimal transport in  $O\big(n^{2 + o(1)}\big)$  time (though its practicality is unclear). For moderate  $\epsilon \geq n^{-o(1)}$ , this is slower than Theorem 1.
To our knowledge, there have been no solvers developed tailoring specifically to the family (2). These problems are solved by general-purpose SDP solvers, where the state-of-the-art runtimes of  $\widetilde{O}(n^{\omega}\sqrt{d} + nd^{2.5})$  or  $\widetilde{O}(n^{\omega} + d^{4.5} + n^{2}\sqrt{d})$  [JKL $^{+}$ 20, HJS $^{+}$ 22] are highly superlinear (though again depend polylogarithmically on the inverse accuracy). We believe our techniques in proving Theorem 2 extend to show that a variant of gradient descent in the  $\ell_{\infty}$  geometry [KLOS14] solves (2) in an unaccelerated  $\widetilde{O}(L^2\epsilon^{-2})$  iterations, with the same per-iteration complexity as Theorem 2.
General notation. We use  $\widetilde{O}$  to suppress polylogarithmic factors in problem parameters for brevity. Throughout  $[n] := \{i \in \mathbb{N} \mid 1 \leq i \leq n\}$  and  $\|\cdot\|_p$  is the  $\ell_p$  norm. We refer to the dual space (bounded linear operators) of a set  $\mathcal{X}$  by  $\mathcal{X}^*$ . The all-zeroes and all-ones vectors in dimension  $d$  are denoted  $\mathbb{0}_d$  and  $\mathbb{1}_d$ . When  $u, v$  are vectors of equal dimension,  $u \circ v$  denotes their coordinatewise multiplication. For  $\mathbf{A} \in \mathbb{R}^{n \times d}$  we denote its  $i^{\text{th}}$  row (for  $i \in [n]$ ) by  $\mathbf{A}_i$ ; and  $j^{\text{th}}$  column (for  $j \in [d]$ ) by  $\mathbf{A}_{j}$ . For  $p, q \geq 1$  we define  $\|\mathbf{A}\|_{p \to q} := \sup_{\|v\|_p = 1} \|\mathbf{A}v\|_q$ . The number of nonzero entries in  $\mathbf{A}$  is denoted  $\mathrm{nnz}(\mathbf{A})$ , and  $\mathcal{T}_{\mathrm{mv}}(\mathbf{A})$  is the time it takes to multiply a vector by  $\mathbf{A}$ . We use  $\mathrm{med}(x, -1, 1)$  to project  $x$  onto  $[-1, 1]$ , where med means median. We define the  $d$ -dimensional simplex and the  $d \times d$  spectraplex  $\Delta^d := \{y \in \mathbb{R}_{\geq 0}^d \mid \|y\|_1 = 1\}$  and  $\Delta^{d \times d} := \{\mathbf{Y} \in \mathbb{S}_{\geq 0}^{d \times d} \mid \operatorname{Tr}\mathbf{Y} = 1\}$ . Throughout we reserve  $h(y) := \sum_{j \in [d]} y_j \log y_j$  (for  $y \in \Delta^d$ ) for negated (vector) entropy.
**Optimization.** For convex function  $f$ ,  $\partial f(x)$  refers to the subgradient set at  $x$ ; we sometimes use  $\partial f$  to denote any (consistently chosen) subgradient. Following [LFN18], we say  $f$  is  $L$ -relatively smooth with respect to convex function  $r$  if  $Lr - f$  is convex, and we say  $f$  is  $m$ -strongly convex with respect to  $r$  if  $f - mr$  is convex. For a differentiable convex function  $f$  we define the associated Bregman divergence  $V_{x}^{f}(x') := f(x') - f(x) - \langle \nabla f(x), x' - x \rangle$ , which satisfies the identity
$$
\langle \nabla V _ {x} ^ {f} \left(x ^ {\prime}\right), u - x ^ {\prime} \rangle = \langle \nabla f \left(x ^ {\prime}\right) - \nabla f (x), u - x ^ {\prime} \rangle = V _ {x} ^ {f} (u) - V _ {x ^ {\prime}} ^ {f} (u) - V _ {x} ^ {f} \left(x ^ {\prime}\right). \tag {3}
$$
For convex function  $f$  on two variables  $(x, y)$ , we use  $\partial_y f(x, y)$  to denote the subgradient set at  $y$  of the restricted function  $f(x, \cdot)$ . We call a function  $f$  of two variables  $(x, y) \in \mathcal{X} \times \mathcal{Y}$  convex-concave if its restrictions to the first and second block are respectively convex and concave. We call  $(x, y) \in \mathcal{X} \times \mathcal{Y}$  an  $\epsilon$ -approximate saddle point if its duality gap,  $\max_{y' \in \mathcal{Y}} f(x, y') - \min_{x' \in \mathcal{X}} f(x', y)$ , is at most  $\epsilon$ . For differentiable convex-concave function  $f$  clear from context, its subgradient operator is  $g(x, y) := (\partial_x f(x, y), -\partial_y f(x, y))$ ; when  $f$  is differentiable, we will call this a gradient operator. We say operator  $g$  is monotone if for all  $w, z$  in its domain,  $\langle g(w) - g(z), w - z \rangle \geq 0$ : examples are the subgradient of a convex function or subgradient operator of a convex-concave function.
1.4 Organization
We summarize our improved area convexity insights in Section 2 (with extended expositions in the supplement), and showcase how they prove Theorem 1 in Section 3. Due to space constraints, we defer proving Theorem 2 and our applications to the supplement (Sections 4 and 5, respectively).
2 Revisiting area convexity
In this section and Section 3, we consider box-simplex games of the form (1), for  $\mathbf{A} \in \mathbb{R}^{n \times d}$ ,  $b \in \mathbb{R}^d$ , and  $c \in \mathbb{R}^n$ . For simplicity we assume  $\| \mathbf{A} \|_{1 \to 1} \leq 1$ , lifting this assumption (by scale invariance) in the proof of Theorem 1. We approximate saddle points to (1) by using the family of regularizers:
$$
r _ {\mathbf {A}} ^ {(\alpha)} (x, y) := \left\langle | \mathbf {A} | y, x ^ {2} \right\rangle + \alpha h (y), \text {w h e r e} h (y) := \sum_ {j \in [ d ]} y _ {j} \log y _ {j} \text {i s n e g a t i v e e n t r o p y}, \tag {4}
$$
where the absolute value is applied to  $\mathbf{A}$  entrywise, and squaring is applied to  $x$  entrywise. This family was introduced by [JST19], a minor modification to a similar family given by [She17], and has favorable properties for the geometry present in the problem (1). In the remainder of the paper,