Title: Affine-Invariant Global Non-Asymptotic Convergence Analysis of BFGS under Self-Concordance
Venue: NeurIPS 2025
Accepted: yes
Keywords: Convex Optimization

Quasi-Newton Method

BFGS Algorithm

Global Convergence Analysis

Self-Concordance
Abstract: In this paper, we establish global non-asymptotic convergence guarantees for the BFGS quasi-Newton method without requiring strong convexity or the Lipschitz continuity of the gradient or Hessian. Instead, we consider the setting where the objective function is strictly convex and strongly self-concordant. For an arbitrary initial point and any arbitrary positive-definite initial Hessian approximation, we prove global linear and superlinear convergence guarantees for BFGS when the step size is determined using a line search scheme satisfying the weak Wolfe conditions.  Moreover, all our global guarantees are affine-invariant, with the convergence rates depending solely on the initial error and the strongly self-concordant constant. Our results extend the global non-asymptotic convergence theory of BFGS beyond traditional assumptions and, for the first time, establish affine-invariant convergence guarantees—aligning with the inherent affine invariance of the BFGS method.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proves global linear and non-asymptotic local superlinear convergence rates for the BFGS method when applied to strictly convex and strongly self-concordant functions. Unlike prior work, it does not assume strong convexity or Lipschitz continuity of the gradient or Hessian. The results are affine-invariant and hold under weak Wolfe conditions, for any initialization. The analysis is complemented by numerical experiments supporting the theoretical claims.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper addresses an old and complicated question on the rate of convergence of BFGS, by providing explicit rates for the “real” implementation of BFGS (vanilla BFGS with Armijo-Wolfe line search) for self-concordant functions. While the paper has some weaknesses, it is a strong contribution to optimization. \n\n**Strengths**\n\n- Despite the technicality of the paper, I appreciate the author’s effort to make it accessible\n- The rate of convergence of BFGS was unknown for a long time, and this paper addresses an important gap in the theoretical understanding of this algorithm.\n- The rate is derived for a practical implementation of BFGS (i.e., classical BFGS algorithm with a weak Wolfe condition for the stepsize). Moreover, the analysis matches the affine-invariant nature of BFGS, which is often overlooked in previous work.\n- The rates are non-asymptotic, and all constants have been carefully computed.\n\n**Weaknesses**\n\n- The approach and derivation of the rates are specific to BFGS, and not for a broader class/family of quasi-Newton methods.\n- The class of self-concordant functions is relatively narrow compared to broader classes such as convex or strongly convex functions. While the authors argue that prior works required strong convexity/smoothness, relying on self-concordance could be considered an “even stronger” and less practical assumption.\n- The experimental section is relatively weak and is limited to synthetic problems (artificial cubic function and logistic regression); it would be stronger with results on real-world datasets or applications, or to illustrate better the different convergence behavior predicted by the theoretical results from Theorems 3.3 and 4.3\n\nQUESTIONS:\nDo you believe it is possible to have a similar analysis for a family of quasi-Newton methods, rather than only BFGS?\n\nLIMITATIONS:\nSee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-18"}

comments: SUMMARY:
This paper considers the global non-asymptotic convergence guarantee of the BFGS method for solving strictly convex and strongly self-concordant objective functions. The convergence analysis of the BFGS method in this paper does not require the common assumptions, including strong convexity and Lipschitz continuity of the gradient or Hessian. They provide guarantees of global linear convergence and superlinear convergence after certain iterations.

STRENGTHS AND WEAKNESSES:
Strengths;

* The paper is well-written and easy to follow. In several parts of the work, the authors clarify the difference between their analysis and the analysis in existing work [1].
* The convergence guarantee of the BFGS does not require common assumptions such as strong convexity and Lipschitz continuity of the gradient or Hessians. 
* This work provides extensive numerical experiments to demonstrate the effectiveness of the BFGS method for strictly convex functions.

Weakness:

* Although there are minor differences in specific details between this work and the related study [1], the overall proof framework closely follows that of the existing work [1]. The technical contribution is incremental compared with prior work.


References:

[1] Qiujiang Jin, Ruichen Jiang, and Aryan Mokhtari. Non-asymptotic global convergence analysis of BFGS with the Armijo-Wolfe line search. Conference on Neural Information Processing Systems (NeurIPS 2024), 2024.

QUESTIONS:
* Is it possible to extend the global convergence analysis of the BFGS method to the broader restricted Broyden family, including methods such as the DFP method?
* In line 107 and 108, is the term $M |u|^3 x$ a typo?

LIMITATIONS:
Please refer to the weakness section.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-20

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work analyzes the BFGS algorithm with Wolfe line search for minimizing strictly convex and strongly self-concordant functions. Although BFGS is known to be affine-invariant and to converge superlinearly, a global superlinear rate independent of problem conditioning had not been established. The authors close this gap by proving the first global superlinear convergence rate that depends only on the initial suboptimality and the strong self-concordance constant, for arbitrary starting points and arbitrary positive-definite BFGS initializations.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- This work analyzes a highly relevant algorithm in convex optimization—BFGS with Wolfe line search—and establishes a non-asymptotic, affine-invariant superlinear convergence rate. I consider this a significant theoretical contribution that highlights the algorithm’s close connection to Newton’s method, both being affine-invariant with superlinear rates. Given that BFGS is effectively what’s implemented in widely used libraries like scipy, the result is also of high impact for existing practical implementations. The results are very explicit and satisfying, with all constant dependencies clearly stated.\n- The paper is very well organized and clearly written. I particularly appreciate the following aspects:\n\t- The introduction is very concise, and well-structured. It provides an overview of existing literature, clearly stating the limitations of existing works before presenting the authors' own contributions. I appreciate that the authors go beyond simply listing references when discussing references; the discussions of how they relate to each other and connect to the present work provide valuable context for the reader. The coherence also reflects authors' deep understanding of existing results.\n\t- The setup and assumptions are all very standard and clearly stated. The paper is very polished overall, with no major typographical errors. Although the results use some seemingly non-standard functions and quantities (for instance $\\omega$ and $\\Psi$), they are nonetheless commonly used in recent BFGS analyses. It's very helpful that the authors provide intuitions when each of these objects are introduced. The sketch of proof techniques following each major result are also very helpful, especially when emphasizing what's no longer possible with the self-concordance assumption as opposed to the strong-convexity and smoothness assumptions in previous works, and how the difficulties are overcome. \n\t- I also like that the authors give separate statements for when the BFGS initialization is chosen to be a multiple of the identity. These correspond to practical choices and the rates are more parsable. \n\t- It's nice that the authors also included an iterations complexity with line search taking into consideration, especially because the curvature condition is used as well which requires additional gradient (not just function value) evaluations. \n\n\n**Weaknesses:**\n\nMy only major concern is the applicability of strong self-concordance. The authors claim that \n1. The class of strong self-concordant functions is a subclass of \\[standard\\] self-concordant functions.\n2. The logistic regression objective is satisfies strong self-concordance.\nHowever, we know (for instance from [Bach (2010)](https://arxiv.org/pdf/0910.4627)) that the logistic regression objective does not satisfy the standard self-concordance definition. Rather, it satisfies a generalized self-concordance definition (that is different from the strong self-concordance used in the present paper). Therefore logistic regression cannot be strongly self-concordant?\n\nThe authors did provide an argument for logistic regression being strongly self-concordant (in Appendix F) but I find it a bit hand-wavy. If we consider logistic regression with only 1 data point, the objective is just $f(x)=\\ln(1+\\exp(z^\\top x))$ (absorbing $y$ and the negative sign into $z$), we have $\\nabla^2f(x) = \\sigma'(z^\\top x)z^2$ where $\\sigma$ is the sigmoid function. The authors claim that $f$ is strongly convex wrt the matrix $B$, which in this case is just $z^2$, and thus one can use an existing argument from [47] that $f$ is strongly self-concordant. But strong convexity does not hold globally even wrt $B$ since $\\sigma'$ still asymptotes to $0$. Am I missing something here or is logistic regression actually not strongly self-concordant? \n\n- Similarly, I expect a more rigorous proof that the log-sum-exp function is strongly convex (wrt the corresponding $B$) which lets you apply the argument of [47]. \n- It would also be nice to discuss what kind of functions satisfy standard self-concordance but is not strongly self-concordant. This is to give an idea of how strict this assumption is, since for the affine invariant analysis of Newton's method, standard self-concordance suffices.\n\n\nThe other weakness is on the experimental design:\n1. The breakthrough in this paper is an affine invariant superlinear rate of BFGS. However, the experimental design doesn't really reflect this. It's nice to have plots for varying dimensions and to see that BFGS performs much better than GD and accelerated gradient. But these are well-known and not very surprising. My (not very well-thought-through) suggestion is as follows: for each problem (or just take a real logistic regression dataset which I recommend), run GD / AGD in the original coordinate system as well as a linearly transformed coordinate system, where the transformation is chosen such that GD / AGD performs a lot worse. Then run the BFGS variants in both coordinate systems as well, and show that they are unaffected by this change of basis, while still converging blazing fast.\n2. In addition to suboptimality vs. number of iterations, plots with number of gradient evaluations on the x-axis should also be included (especially since backtracking line search for GD / AGD does not involve additional gradient evals). Perhaps also include a set with wall-clock time on the x-axis. \n3. It would also be nice to see a plot of $\\eta_k$ vs. $k$ to see how often is the unit step size accepted/rejected.\n4. Is it possible to estimate the constants in the bound of, say, Corollary 4.4? If yes, it may be worthwhile to plot the corresponding $(C/k)^k$ curve along with the experimental results just to see how well they match. \n\n**Note:** I did not check all the convergence proofs in detail (especially the algebraic manipulations). I only checked major proof steps and they seem to be correct.\n\nQUESTIONS:\n1. Do you think it would be possible to obtain similar results with just an Armijo backtracking line search? The curvature condition makes the line search slightly more complicated to implement, and additional calls to the gradient oracle. Newton's method only requires the sufficient decrease condition in the line search (though the quadratic rate is local). \n2. Can we optimize the bound in, say, Theorem 3.3 or Theorem 4.3, over $\\alpha$ and $\\beta$ to further tighten the bounds?\n\nLIMITATIONS:\n- It's unclear for now whether strong self-concordance indeed holds for the objectives claimed by the authors. (see weaknesses)\n\n**Minor issues:**\n- After Assumption 2.2 (strong self-concordance), the notion of standard self-concordance is brought up for comparison. The definition for the latter should also be given for readers unfamiliar with the concept.\n- Right after that (lines 107-108), I think the two occurrences of $|u|^3x$ should be typeset as $\\Vert u\\Vert^3_x$, and the last $|u|^3_x$ should be $\\Vert u\\Vert^3_x$?\n- Line 109: if we want to be extra precise, Nesterov's Theorem 5.1.2 [46] is for the affine invariance of _standard_ self-concordance, not strong self-concordance. Although it's pretty obvious that affine invariance also holds for strong self-concordance. \n- Line 153's clarity can be improved by referring back to (9), i.e., \"... of $B_0$ correspond to \\[$\\hat B_0$ in (9) with\\] the weight matrices $P=...$\"\n- Line 288: expression for $T_3$ is hard to read. Perhaps put it into display math mode? The tilde and bar over $B_0$ also seem to be off. \n- Line 324: should this cubic function have a citation? \n- The Appendix could use a bit more organization, such has having its table of content, and a bit of discussion before each proof. It's recommended to use the `restatable` package to restate result statements before the proofs (and put proofs into proof environments). \n- Line 1051: $log$ -> $\\log$\n- Line 1053: after \"that greedy quasi-Newton paper\" perhaps cite [17] again. I recommend using the author-year citation style but I understand the space constraint. \n- Plots: the problem / objective function name can go into the title of the figure in large font instead of in the caption which is hard to find.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a new non-asymptotic convergence analysis for the BFGS quasi-Newton method under the assumption of strict convexity and strong self-concordance, which is weaker than the typical strong convexity and Lipschitz Hessian assumptions. The main results are the establishment of global linear and superlinear convergence rates that are, unlike prior analyses, invariant to affine transformations of the variables. The analysis holds for any initial point and positive-definite initial Hessian, with a step size selected by a line search satisfying the weak Wolfe conditions.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths*\n\nThe paper is technically deep and addresses a well-known theoretical challenge in the analysis of quasi-Newton methods. The primary strength is the rigorous mathematical construction of a convergence proof that aligns with the affine-invariant nature of the BFGS algorithm. The authors successfully overcome significant technical hurdles by developing an analysis based on time-dependent bounds derived from the self-concordance property, rather than relying on the uniform bounds available under stronger assumptions. The paper is also well-written, with the introduction and proof sketches clearly laying out the logical flow of the arguments.\n\n*Weaknesses*\n\nWhile the paper is technically sound, its primary weakness is that the overall significance of its main contribution is questionable and not sufficiently justified.\n\n1. Questionable Significance of Affine-Invariant Guarantees: The central claim is providing the \"first affine-invariant global non-asymptotic convergence guarantees\". While this is a theoretically elegant result, the paper fails to make a compelling case for its practical or theoretical impact. The BFGS algorithm's performance is already known to be affine-invariant; achieving a theoretical rate that reflects this property feels more like a \"cleanup\" of the theory rather than a groundbreaking discovery that offers new insights. The authors do not explain how this new analysis could lead to better algorithms, improved practical implementations, or a fundamentally new understanding of BFGS's behavior. Logistic regression, log-sum-exp, and log barrier functions the only examples that satisfy the strong concordance assumptions in this paper, but these are not broad enough to justify the significance of the paper's extension over the existing results. \n\nIt would help to better assess the significance of this work if the authors can clearly state the key technical difficulties for obtaining affine-invariant convergence analysis. Specifically, can the analysis in the recent breakthroughs on BFGS with standard non-affine assumptions [35-38] be modified and adapted to work with the current paper's setting? If not, what were the main technical innovation that the authors made to overcome such difficulties? \n\n2. Limited Scope of Assumptions: The analysis relies on strong self-concordance, a condition that is not universally met. While the authors correctly argue this is weaker than requiring strong convexity and a Lipschitz Hessian, it is still a restrictive structural assumption. This limits the applicability of these \"first-ever\" guarantees to a specific class of problems, somewhat diminishing the claim of broad impact.\n\n3. Inappropriate Citation: The paper relies on an informal source to support a technical claim. Specifically, Footnote 1 on page 16 cites a Math Stack Exchange post. For a formal academic publication, especially one of this theoretical depth, relying on a community forum instead of a textbook or peer-reviewed article is unprofessional and inappropriate.\n\n*Minor comments*\n\nThe usage of = and := sometimes seems off. For example, on line 85, it might be better to use $H_{t+1}=....$ rather than $:=$. And in equation (9), it might be better to use $:=$.\n\nline 164, potential extra punctuation? \"To further simplify the expression\" instead of \"To further, simplify the expression\"\n\nline 227, typo \"admissible\" instead of \"addmissible\"\n\nThe expression inside big O looks weird in Prop.5.1. The first term goes to 0 as t goes to infinity and the second term goes to negative infinity. From the last part of the proof in the appendix, it seems to me the correct big O might be $O(1+\\log(1+\\Gamma/t)+\\log(1+\\log(1+\\frac{\\Psi(B+\\Gamma)}{t})))$\n\nQUESTIONS:\nPlease see above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}