Title: Efficient Quadratic Corrections for Frank-Wolfe Algorithms
Venue: NeurIPS 2025
Accepted: yes
Keywords: Convex optimization

conditional gradient method

Frank-Wolfe method
Abstract: We develop a Frank-Wolfe algorithm with corrective steps, generalizing previous algorithms including Blended Conditional Gradients, Blended Pairwise Conditional Gradients, and Fully-Corrective Frank-Wolfe.
For this, we prove tight convergence guarantees together with an optimal face identification property.
Furthermore, we propose two highly efficient corrective steps for convex quadratic objectives based on linear optimization or linear system solving, akin to Wolfe's Minimum-Norm Point algorithm, and prove finite-time convergence under suitable conditions.
Beyond optimization problems that are directly quadratic, we revisit two algorithms, Split Conditional Gradient and Second-Order Conditional Gradient Sliding, which can leverage quadratic corrections to accelerate the solution of their quadratic subproblems.
We show improved convergence rates for the first and prove broader applicability for the second.
Finally, we demonstrate substantial computational speedups for Frank-Wolfe-based algorithms with quadratic corrections across the considered problem classes.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce Corrective Frank-Wolfe, a framework unifying many frank-wolfe algorithms. They describe two correction schemes for quadratic problems which are implementations of the new framework. They also mention two specific algorithms which should benefit from the proposed quadratic corrections. Finally the authors present some empirical results demonstrating the convergence rates of their corrected algorithms.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The framework of Corrected Frank-Wolfe seems to be a useful generalization of existing algorithms, potentially enriching future work.\n2. The specific implementation of the framework for quadratic optimization is novel and seems useful as well.\n3. The experiments show significant improvement over the baseline.\n\nWeaknesses\n\n1. Remark 1 after proposition 1 should have been elaborated on. Why would choosing the pairwise gap lead to fewer LMO calls than choosing the away gap?\n2. The paper is very dense in details making it hard to follow. Almost every paragraph contains either new ideas or new formalisms, which leaves little room for explanations. For example one of the major claims of the paper is about SCG and SOCGS, but SOCGS only has one small paragraph and SCG has a theorem with no proof. From reading the paper alone I couldn’t figure out your contribution to these algorithms.\n3. The experiments only compare with one baseline algorithm. More should have been measured, there’s no guarantee that BPCG is the fastest algorithm for the two example problems given, making its choice as baseline questionable.\n4. The experimental section contains graphs which aren’t explained well. There are 8 sets of images for each figure but its not clear what’s the differences between them. Are they different experiments? Different seeds?\n5. It would be beneficial to add a direct speed comparison between the algorithms. How much faster is your approach at reaching a specified gap value?\n\nQUESTIONS:\nSee questions under “weaknesses”.\nSome proofing mistakes:\n* In algorithm 1, the global FW vertex uses V(P) which is undefined (I assume you mean V(X)?)\n* In 2.1 you don’t prove your theorems, you only state them, so you should write “in the following we state two theorems”. It’s fine to have proofs in the appendix if they aren’t particularly enlightening.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a Frank-Wolfe algorithm with corrective steps. Their modification relies on a relaxed optimality condition, as detailed in Algorithm 2. They then specialize the proposed algorithm and establish convergence guarantees for quadratic functions minimized over an affine hull.\n\nThe paper also presents results for constraint sets formed by the intersection of two convex sets. Finally, it extends the analysis from optimizing strongly convex functions over polytopes to self-concordant functions.\n\nThe main message of the paper is that the proposed modification accelerates the algorithm, and this claim is supported by numerical experiments presented in the final section.\n\nSTRENGTHS AND WEAKNESSES:\nWeaknesses:\n\nThe main weakness of the paper is that some of its claims feel vague and hand-wavy, lacking concrete justification. I attempt to clarify these points by raising questions in the following sections.\n\nStrengths:\n\nAside from these concerns, the algorithms and technical proofs are presented very clearly and are a pleasure to read. The literature review covers most of the important related works.\n\nQUESTIONS:\n1. Could the authors clarify whether their claim is that they analytically outperform existing algorithms for a specific family of functions or constraint sets? Or is it primarily an intuition they aim to convey by guiding the reader through the steps of the algorithm? In particular, could the authors be more precise about the following statements:\n“It converges within a single step to the optimal value as soon as …” and “This is both computationally more efficient and avoids dropping too many potentially good vertices from the active set.”\n\n2.  In Section 3.1, which algorithm was ultimately used? Is it the one from [1] with a modified step size? If it is exactly the algorithm from [1], what is the relationship between this section and the rest of the paper? If it is not the same as [1], why is this not clearly stated in the theorem’s statement?\n\n3. It would be helpful if the authors could specify what the step size was in [1] and what the main technical challenges are in improving the logarithmic factor beyond simply adjusting the step size.\n\n4. In Section 3.2, the authors state:\n“Unlike regular FW variants, CGS requires only \n$\\mathcal{O}(\\sqrt{\\epsilon}$ gradient calls to achieve  $\\epsilon$-optimality.” It would be useful if the authors could also mention the gradient call complexity required by standard Frank-Wolfe algorithms for comparison.\n\n5. Regarding the result mentioned in Section 3.2, “we extend this rate to generalized self-concordant functions,”\nhave the authors actually extended the algorithm to self-concordant functions? Could they please clarify this part and elaborate on the technical challenges involved?\n\n6. Regarding the results for quadratic functions, could you generalize your analysis to quadratic functions with a positive semi-definite matrix instead of a positive definite one? Additionally, instead of restricting to strongly convex functions, could you provide comments on functions that satisfy the Polyak-Łojasiewicz (PL) condition?\n\n\n\n[1] Zev Woodstock and Sebastian Pokutta. Splitting the conditional gradient algorithm. SIAM Journal on Optimization.\n\nLIMITATIONS:\nI address the limitations that I have in mind in the previous sections..", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work develops a variant of the Frank-Wolfe algorithm, which they refer to as the Corrective Frank-Wolfe (CFW) algorithm. The authors claim it to be a generalization of a previous variant called the Fully Corrective Frank-Wolfe  (FCFW) Algorithm. The authors claim that FCFW is often computationally impractical and their proposed CFW method is a means to overcome this challenge. Their main results derive convergence rates for this algorithm and its lazified variant. As part of their CFW framework, the work also designs two quadratic correction steps: (i) QC-LP which involves a direct linear program and (ii) QC-MNP which is a specialized variant of the minimum-norm point algorithm. With these new quadratic corrections, the authors show that the Split Conditional Gradient (SCG) algorithm enjoys a faster convergence rate of $O(1/\\sqrt{t}),$ improving a $\\log t$ factor from the previously known result. Secondly, for Second-Order Conditional Gradient Sliding (SOCGS), the authors show global convergence for generalized self-concordant functions. Finally, the work empirically demonstrates the computational benefits of their quadratic corrections on four specific problem instances.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n1. Extremely well-written paper.\n2. The proposed algorithm introduces several corrective steps that can potentially improve computational efficiency. \n3. Empirical results demonstrate significant improvement in iteration complexity over BPCG across a range of tasks.\n\n**Weaknesses**:\n\n1. The theoretical results do not quantify the computational benefits of using the new corrective steps. In particular, it is unclear why the corrective steps lead to computational benefits over the traditional FW update rules. \n\n**Minor issues**:\n\n1. The first two sentences in your abstract are too long. Please break them into smaller, easy-to-digest\n\n2. Explicitly define the vertex set of a compact convex set in the preliminaries section.\n\n3. What is $V(P)$ in Step 5 of your algorithm? I presume you meant $V(\\mathcal{X})$.\n\nQUESTIONS:\n1. In Algorithm 2, can you explictly highlight when the drop step would be executed and when the descent step would be executed?\n\n2. It is unclear how the three steps in Proposition 1 are to be utilized. Should they be tried one by one in order until either the drop or descent step criterion is met?\n\n3. I am new to this field. However, in my understanding, the convergence rates in Theorem 2 and 3 do not appear to be new. So, what is the benefit of your proposed algorithm? Shouldn't you be quantifying the reduction in the computational effort to demonstrate the superiority of your algorithm?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces approximate versions of the fully-corrective Frank-Wolfe \n(FCFW) method applied to quadratic objectives.\nFirst, the authors develop a framework for general corrective\nsteps and prove fast convergence rates for their algorithm, which they call\ncorrective Frank-Wolfe (CFW).\nThen they propose two quadratic corrections which specialize their algorithm:\nQuadratic Correction LP, which solves a linear program and then checks if\nit yields a feasible point, and Quadratic Correction MNP, which solves\na linear system and computes the closes feasible point along the chord between\nthe solution and the current iterate.\nExperiments show that these corrective steps outperform significantly outperform\nblended pairwise conditional gradients (BPCG) on synthetic and real-world problems. \nIn addition, the authors develop a faster convergence rate for split\ncondition gradient (shaving off a log term) and also show how to incorporate \nlazy evaluation of the linear minimization oracle into their framework.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The theoretical framework for CFW is general and may encourage other\n  researchers to develop corrective steps for non-quadratic objectives.\n\n- The quadratic corrective steps work very well compared to BPCG, both\n    in terms of iterations and wall-clock time.\n\n- Corrective steps can also be applied to split conditional gradient and\n    and conditional gradient sliding.\n\n- The authors improve the best known convergence rate for split conditional\n    gradient.\n\n- The paper is well-written.\n\n**Weaknesses**\n\n- The corrective steps the authors propose only apply to quadratic objectives.\n    This may not be a strong restriction since Frank-Wolfe requires a linear \n    minimization oracle, which already restricts the function class in practice.\n\n- No evaluation of the success rates (i.e. how often they reduce to FCFW steps) \n    of the quadratic corrections is provided. \n\n- The experiments only compare to one baseline algorithm, BPCG. It's\n    not clear how much CFW under-performs the ideal method (FCFW) and how\n    it compares to similar similar methods, like approximate FCFW or pairwise \n    Frank-Wolfe.\n\n\n### Detailed Comments\n\nI am not an expert in Frank-Wolfe methods, so it is difficult for me to judge\nthe novelty and potential impact of this paper. \nHowever, I found it interesting to read and I think there are several clever\nideas here which will be useful to other researchers.\nI particularly like that CFW is presented in a general framework, which will\nencourage other researchers to develop new corrective steps for more general\nobjectives.\nWith this said, I think there are several important issues (discussed in detail\nbelow) whose resolution will improve the paper.\nThus, I will recommend weak accept for the moment, with the option to raise\nmy score if these issues are resolved satisfactorily.\n\n**Limitation to Quadratics**:\nWhile the theoretical results for CFW apply to any corrective steps which\nsatisfy the structure of Algorithm 2, the two proposed corrections are only\nuseful for quadratic objective functions.\nThe authors are straightforward about this limitation, which is refreshing.\nMoreover, Frank-Wolfe methods already require access to a linear minimization \noracle (which restricts the applicable function class in practice), so this\nmay not be too significant a drawback.\n\n**Limited Experimental Comparison**:\nI list that CFW significantly outperforms the BPCG baseline as a strength,\nwhich it is. \nYet, the main weaknesses of this paper is still the experimental comparison.\nComparing to only one baseline in a crowded area like Frank-Wolfe methods\nis limited at best and misleading at worst.\nCFW is an approximation to FCFW, yet there are no experiments presented in the\nmain paper comparing these two methods.\nThe authors also draw attention to approximate FCFW and pairwise Frank-Wolfe as\nthe two closest methods to theirs, but then don't compare them experimentally.\nI think this is a glaring problem in a paper which advertises \"the excellent\ncomputation benefits\" of their approach.\n\n**Success Rates of Quadratic Corrections**:\nThe authors mention in Section 4.2 that \"the LP in the QC-LP is almost never\nfeasible, leading to pairwise steps and thus a similar trajectory as BPCG\".\nThis raises a key question which isn't addressed theoretically or empirically:\nhow often do the quadratic correction methods succeed and return feasible \nsolutions which correspond to FCFW steps?\nSection 4.2 shows that sometimes they mostly fail, meaning solving the\nLP/linear system is a waste of computation overall.\nWhile theoretical analysis may impractical, the paper would be strengthened by\nincluding ablation studies on the success of proposed quadratic corrections.\n\n### Minor Comments\n\n- Line 65: It would be helpful to define the MNP algorithm. Is this Wolfe's\n    minimum-norm point that was mentioned in the abstract? Edit: I see this\n    is introduced in Line 87 but I think it would better to state it here.\n\n- Line 72: The large number of acronyms in this paragraph make the discussion\n  difficult to follow. Some paragraphs later on also devolve into \"acronym\n  soup\".\n\n- Line 118: It would be useful to define $a_t, s_t$, and $x_t$ outside of \n    Algorithm 1. Otherwise the reader has to leave the first paragraph\n    of Section 2, reader Algorithm 1, and then combine back in order to \n    understand the discussion.\n\n- Algorithm 2: I interpret $S' \\subset S$ as non-strict inclusion equivalent to\n  $S' \\subseteq S$. The drop step seems to imply $S' \\subset S$ only in that\n  case.\n\n- Line 168: I think it's simpler and clearer to say \"writing $x^*$ as an affine\n    combination of vertices.\n\n- Algorithms 3 and 4: I think the presentation would be better if these were\n    top-aligned.\n\n- Line 176: \"The QC-LP approach assumes enforces the\" --- unnecessary word?\n\n- Algorithm 3: $\\tilde{S}$ appears to define the solution $x^*$ in barycentric\n    coordinates. If this is true, how does the local pairwise step\n    reduce the cardinality compared to $S$ as required by Algorithm 2? \n    Or is the local pair-wise step guaranteed to make sufficient progress?\n\n- Algorithms 3, 4: It would be helpful if the return values matched Algorithm 2.\n\n- Line 192: is it ever possible that the number of atoms exceeds the dimension\n    of the atoms? In this case, it will be more efficient to solve directly\n    in terms of $x$, so you can depend on minimum of the dimension and\n    the size of the active set. Is this meaningful at all?\n\n- Theorem 5: This result fits awkwardly into the current story. Maybe\n    you can remind the reader before this paragraph that each\n    sub-step of SCG can be combined with the ideas from CFW to obtain\n    faster convergence. Thus, it is desirable to improve the rates for\n    SCG and bring them into correspondence with the standard CFW framework. \n\n- All Figures: the lines and font-sizes are too small to be legible without\n    unreasonable zooming.\n\n- Figure 1: The scale of the y-axis in the first row prevents these plots \n    from showing any useful information.\n    I suggest subtracting the optimal value (if you aren't already) and \n    changing the y-limit to cut-off the first few iterations.\n\n- Figure 2: Same comments as for Figure 1.\n\n- Line 271: \"Mentioning that quadratic corrections...\"  --- was this sentence\n    supposed to be an internal comment?\n\n- Line 278: \"We remind that each iteration...\" --- Where does the quadratic\n    sub-problem come from? Is this a feature of AFW? I know the authors\n    have limited space, but providing more details would make this experiment\n    easier to understand.\n\n- I think you could cut down on the discussion in Section 1 to get more space. \n    In particular, the contributions subsection is long and could\n    be reduced by summarizing the just key contributions in a bulleted list. \n    Three pages for the introduction is quite a lot for a NeurIPS paper;\n    two pages is perhaps standard.\n\n- It's not necessary to capitalize every word in an acronym when you introduce\n    it. For example, in \"Fully-Corrective Frank-Wolfe\", only Frank-Wolfe \n    should capitalized, since these are proper nouns.\n\nQUESTIONS:\n- Algorithm 1: I assume that every extreme point of $\\mathcal{X}$ is treated as\n  a vertex if $\\mathcal{X}$ is not polyhedral? Or do you require a polyhedral\n  constraint set?\n\n- Algorithm 1, Line 5: What is $P$? Should this be $\\mathcal{X}$?\n\n- Algorithm 1, Line 10: Do you have perform this minimization exactly?\n\n- Algorithm 2: Where do $s, v$ get used? Aren't these equivalent to $u, w$ in\n    the descent step?\n\n- Algorithm 3: How often can we expect Eq. (7) to return an interior point\n    (see also detailed discussion)?\n\nLIMITATIONS:\nLimitations are reasonably addressed with the exception of using only a single\nexperimental baseline. \nThe authors should provide a strong argument justifying this choice in the\npaper or they should extend their comparison to include additional methods.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}