Title: Quasi-Self-Concordant Optimization with $\ell_{\infty}$ Lewis Weights
Venue: NeurIPS 2025
Accepted: yes
Keywords: Convex optimization

Quasi-Self-Concordant Optimization
Abstract: In this paper, we study the problem $\min_{x\in R^{d},Nx=v}\sum\_{i=1}^{n}f((Ax-b)_{i})$
for a quasi-self-concordant function $f: R\to R$, where $A,N$ are
$n\times d$ and $m\times d$ matrices, $b,v$ are vectors of length
$n$ and $m$ with $n\ge d.$ We show an algorithm based on a trust-region
method with an oracle  that can be implemented using $\widetilde{O}(d^{1/3})$
linear system solves, improving the $\widetilde{O}(n^{1/3})$ oracle
by [Adil-Bullins-Sachdeva, NeurIPS 2021]. Our implementation of
the oracle relies on solving the overdetermined $\ell\_{\infty}$-regression
problem $\min\_{x\in R^{d},Nx=v}\|Ax-b\|\_{\infty}$. We provide an
algorithm that finds a $(1+\epsilon)$-approximate solution to this
problem using $O((d^{1/3}/\epsilon+1/\epsilon^{2})\log(n/\epsilon))$
linear system solves. This algorithm leverages $\ell\_{\infty}$ Lewis
weight overestimates and achieves this iteration complexity via a
simple lightweight IRLS approach, inspired by the work of [Ene-Vladu,
ICML 2019]. Experimentally, we demonstrate that our algorithm significantly
improves the runtime of the standard CVX solver.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the quasi-self-concordant (QSC) minimization problem, which includes logistic regression and regularized $\\ell_p$ regression as special cases. Based on the observation that QSC functions are Hessian stable, the authors propose an iterative algorithm that solves a sequence of $\\ell_{\\infty}$-norm constrained quadratic subproblems, reminiscent of trust-region methods. Then the authors connect their subproblem to an overdetermined $\\ell_{\\infty}$-regression problem. As a contribution of independent interest, they design a new $\\ell_{\\infty}$ regression algorithm based on iteratively reweighted least squares (IRLS) insipred by (Ene and Vladu'19). By incorporating $\\ell_{\\infty}$ Lewis weight overestimates, they improving the complexity from $\\tilde{O}(n^{1/3}/\\epsilon^{2/3})$ in (Ene and Vladu'19) to the-state-of-the-art $\\tilde{O}(d^{1/3}/\\epsilon^{2/3})$. This improvement directly translates to a faster algorithm for QSC minimization, particularly when $d \\ll n$.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The paper presents a simpler $\\ell_{\\infty}$ regression algorithm for the overdetermined setting ($d \\ll n$). The proposed algorithm matches the state-of-the-art result of (Jambulapati, Liu, Sidford'22), and it is conceptually more straightforward and potentially more practical. It builds on the IRLS method from (Ene, Vladu'19), with an effective initialization using $\\ell_{\\infty}$ Lewis weight overestimates.\n- The authors show how $\\ell_{\\infty}$ regression can be used as a subroutine in a trust-region framework for QSC optimization, simplifying the approach of (Adil, Bullins, Sachdeva'21) and achieving better complexity in the overdetermined regime. \n\n## Weaknesses\n- While QSC optimization covers important problems such as logistic regression and regularized $\\ell_p$ regression, the paper does not clarify whether the proposed method yields concrete improvements in these specific applications. \n- The exposition could be improved. In particular, compared to prior work, the two key contributions are: (1) showing that the residual subproblems in (5) can be treated as $\\ell_{\\infty}$ regression problems, enabling simpler IRLS-based methods; and (2) using $\\ell_{\\infty}$ Lewis weight overestimates to initialize the resistances to replace the dependence on $n$ with $d$. However, these points are relegated to the appendix, with limited discussion or intuition in the main text.\n\nQUESTIONS:\n## Questions\n- The $\\ell_2$-regularized $\\ell_p$ regression problem is a key application of QSC optimization, and the authors include it in their experiments. How does the complexity result in this paper compare to existing results specifically for this setting?\n- The main algorithm in Algorithm 3 involves two binary search steps over $\\nu$ and $M$. While this only incurs extra logarithmic factor in the final complexity bound, it adds complexity to the implementation. Do the authors think it is possible to remove these binary search steps?\n\n## Suggestions\n- It would help readers if the authors elaborated on why their initialization using $\\ell_{\\infty}$ Lewis weights offers an advantage over uniform initialization. As I understand it, the main insight comes from Lemma A.2 due to (Jambulapati, Liu, Sidford'22), which provides a potentially tighter lower bound on the increase in energy.\n- On a first reading, it was unclear how Section 3 relates to the central QSC optimization problem. I suggest switching the order of Sections 3 and 4 to better motivate the introduction of the $\\ell_{\\infty}$ regression problem.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers quasi-self concordant optimization, i.e., $\\min \\sum_{i= 1}^n f[Ax-b]i$ , $x\\in R^d, Nx = v$ where $f$ is quasi self concordant. Prior state of the art includes work by [Adil-Bullins-Sachdeva, NeurIPS’21] which solves the problem in $n^{1/3}$ iterations, each iteration solving an $n \\times n$ linear system. This paper uses $\\ell_{\\infty}$ lewis weights and gives an algorithm that finds a solution in $d^{1/3}$ linear systems, where each system solves a $d \\times d$ system.\n\nThis is a big improvement for overdetermined systems when $n>>d$. Prior such improvements have been seen for $\\ell_p$ regression as in the work of JLS’22, where they used $\\ell_p$ lewis weights to obtain such improvements. Furthermore, the paper gives an IRLS based algorithm inspired from EV’19 which can be implemented easily.\n\nThe algorithm is similar to the algorithm of EV’19 where the weights are replaced by overestimates of the $\\ell_{\\infty}$ lewis weights.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper gives a significant improvement over prior works in the over-constrained regime, for important regression problems.\n2. The algorithm is fast in practice.\n\nWeaknesses:\n\n1. How much of the analysis is new is unclear as similar ideas have been used before in JLS’22 where they use a similar change in weights to recover a similar improvement to AKPS’19.\n\nQUESTIONS:\nQuestions:\n1. What is new in the analysis that allows for the incorporation of Lewis weights into EV’19? It seems like reducing to solve to $d\\times d$ systems is new but I am not sure.\n2. I am not sure how much easier this is than ABS’21 for simple cases of softmax regression or $\\ell_{\\infty}$ norm regression. Do the authors have a sense of why their algorithm might be better in practice?\n3. I believe prior works on QSC optimization including ABS’21 and CJJ+’20 also work for hessian stable functions. They just seem to not explicitly write it but the proofs are done for hessian stable functions.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of optimizing generalizing linear models, i.e., $\\min_{x} \\sum_{i} f_i(a_i^\\top x – b_i)$ in the case where all the $f_i$ are the same and quasi-self-concordant (though the techniques of the paper may apply more broadly). The paper provides a trust-region based method to reduce the problem to approximately solving variants of $\\ell_\\infty$-regression which they solve in turn with Lewis weight overestimates used with an iteratively reweighted least squares (ILRS) algorithm. Ultimately, this provides an algorithm which solves the problem in $\\tilde{O}(d^{1/3})$ iterations where $\\tilde{O} (\\cdot)$ here hides polylogarithmic factors as well as polynomial dependencies on a quasi-self-concordance parameter and a $\\ell_\\infty$ measure of the size of the effective domain. Additionally, the paper performs experiments comparing their proposed method against certain benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: As the paper well-articulates, the problem considered encompasses a number of natural optimization problems. Consequently, it is a natural and important problem to seek improved optimization methods for. The paper provides a natural straightforward strategy, which they then implement and perform experiments on. I think this paper nicely highlights the importance of a problem which they provides an elegant solution for it while demonstrating its promise empirically. This paper seems to be making progress by promoting a natural optimization problem, providing a key theoretical benchmark for it, studying the problem in practice, seeking simpler / more straightforward algorithms, and building out the theory of ILRS. I think this paper could facilitate further research in theory and in practice on solving algorithms in this space.\n\nWeaknesses: It is unclear how challenging it is to obtain the $\\tilde{O}(d^{1/3})$ rate provided by the paper by using prior tools and the relationship of the approach to prior work could perhaps use further explanation. The paper provides a reduction from the problem to trust region problems related to $\\ell_\\infty$-regression and then provides a solver for both this problem and the more speciialized $\\ell_\\infty$-regression problem. The reduction to trust region problems seems fairly standard in light of prior work related to quasi-self-concordance that the paper cites. Additionally, as the authors note, $\\ell_\\infty$ regression is then solve-able more efficiently than the method the paper proposes for it (at least in terms of not having an extra $1/\\epsilon^2$ factor, though potentially in a more complicated fashion). Consequently, the novelty in the paper seems more in solving the trust-region problems (as opposed to $\\ell_\\infty$ regression) and in providing an arguably simpler approach than what the literature gets for solving $\\ell_\\infty$ regression (at the cost of the $1/\\epsilon^2$ factor). However, these distinctions and comparison could be better clarified and it is unclear how challenging it is to adapt prior work to solve the trust region problem. Additionally, the empirical results only compare against a limited set of methods. \n\nUltimately, the paper provides a new runtime for a natural quasi-self-concordant optimization problem and provides a new algorithm for $\\ell_\\infty$-regression. Consequently, this paper could make a great addition for NeurIPS. However, I think the writing could be improved to more clearly articulate its contributions and its relationship to prior work.\n\nQUESTIONS:\nQuestion / Suggestion: The paper asserts that it is simpler than ABS21 and that JLS22 is complex. In what way is that true? I think more justification of this earlier would be beneficial. Additionally, JlS22 provided two sets of algorithms, some based on accelerated optimization and some for (p \\neq infinity) that looked more like ILRS. I think the paper would benefit by more clearly comparing its algorithm to all of this prior work, e.g., making clear how challenging it is to adapt previous ILRS methods to work with the norms induced by Lewis weight over estimates in the appropriate way and how hard it is to adapt the ILRs style method in JLS22 to $\\ell_\\infty$\n\nSuggestion: It would be good to clarify what QSC means on the first page and how it is used in this paper. A variety of prior work define QSC with respect to a norm (in the non-1d) case and I think it is good to clarify when the paper uses the QSC term it is very clear whether they are discussing the 1d f or the entire objective.\n\nSuggestion: Since the experiments are done for a regularized variant of p-norm regression it might be beneficial to compare against methods tailored for that problem as well or change the problem (especially since some use similar techniques).\n\nHere are additional detailed comments that might be helpful:\n\n* Page 2: Line 57: “Since the linear system solve …” – there is extensive research on decreasing the cost of iterations of related methods below the time to solve a single linear system. I think this might be important to mention here.\n* Page 2: Line 64: I think it might be helpful to somewhere clearly explain why the eps^{-2} in the abstract doesn’t appear in the theorem. \n* Page 3: Line 130 – 131: (CR)^2/3 type rates for quasi-self-concordant functions with respect to quadratic norms were established earlier in [CJJ+20]. It seems odd to single out Doi23 as a reference for this assertion. \n* Page 3: “can handle more general functions” – Is that a new feature of the method in this paper? It might be nice to clarify.\n\nLIMITATIONS:\nYes, with the possible exception of the concerns raised elsewhere in the review.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}