Title: Revisiting Frank-Wolfe for Structured Nonconvex Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Frank-Wolfe

nonconvex optimization

difference of convex

gradient sliding
Abstract: We introduce a new projection-free (Frank-Wolfe) method for optimizing structured nonconvex functions that are expressed as a difference of two convex functions. This problem class subsumes smooth nonconvex minimization, positioning our method as a promising alternative to the classical Frank-Wolfe algorithm. DC decompositions are not unique; by carefully selecting a decomposition, we can better exploit the problem structure, improve computational efficiency, and adapt to the underlying problem geometry to find better local solutions. We prove that the proposed method achieves a first-order stationary point in $\mathcal{O}(1/\epsilon^2)$ iterations,  matching the complexity of the standard Frank-Wolfe algorithm for smooth nonconvex minimization in general. Specific decompositions can, for instance, yield a gradient-efficient variant that requires only $\mathcal{O}(1/\epsilon)$ calls to the gradient oracle by reusing computed gradients over multiple iterations. Finally, we present numerical experiments demonstrating the effectiveness of the proposed method compared to other projection-free algorithms.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DC-FW, a novel projection-free optimization framework for nonconvex problems formulated as the difference of convex (DC) functions. It combines the classical DCA (Difference-of-Convex Algorithm) structure with the Frank-Wolfe method to solve DC problems without projections. The paper provides theoretical convergence guarantees using a new gap measure (gapDC), and demonstrates that in certain cases, the method achieves improved complexity bounds compared to standard FW. Numerical experiments on QAP, embedding alignment, and neural network training show the versatility and effectiveness of DC-FW.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.\tThe paper addresses a relevant and challenging class of nonconvex problems in a principled way. \n2.\tThe theoretical contributions are solid, and the proposed gapDC measure is well-motivated and rigorously analyzed. \n3.\tThe framework is general, supports multiple DC decompositions, and includes complexity analysis under different conditions. \n4.\tNumerical experiments are diverse, covering discrete optimization, matrix problems, and deep learning.\nWeaknesses:\n1.\tThe theoretical innovation primarily lies in combining known components (DCA + FW), and is more of a well-engineered synthesis than a fundamentally new algorithm. \n2.\tThe numerical experiments, while varied, lack strong verification of theoretical claims such as complexity bounds (e.g., no clear log-log error vs iteration plots). \n3.\tThe neural network experiments are weakly motivated and lack comparison with standard optimizers (e.g., Adam or SGD). \n4.\tPresentation of results (especially the QAP table) is dense and difficult to interpret; no standard deviations or statistical significance is reported.\n\nQUESTIONS:\n1.\tWhile Figure 2 presents FW gap plots against iterations, SVD calls, and wall-clock time, it does not reflect the proposed gapDC measure introduced in the paper. Moreover, the plots do not include theoretical reference lines or convergence slopes (e.g., $O(1/t)$), which would help verify the claimed complexity bounds.\n2.\tWhy are there no comparisons with standard optimizers (e.g., Adam) in the neural network experiments? How does DC-FW perform in such settings? \n3.\tCan the authors elaborate or visualize the effect of different DC decompositions on performance and convergence? \n4.\tPlease clarify the computational cost per iteration of DC-FW compared to CGS and FW-M, especially in large-scale settings. \n5.\tCould the authors improve the readability of the QAP results table (e.g., using heatmaps or grouped bar plots with clearer labels)? Additionally, it would be helpful to analyze why DC-FW performs worse than FW on certain QAP instances, to better understand the limitations of the proposed method.\n6.\tIn Figure 4 of the supplementary material, some of the test accuracy curves — particularly for CIFAR-10 under certain constraint values — exhibit sharp downward spikes followed by immediate recovery. Could the authors clarify the cause of this behavior? Is it due to instability in the line search, stochastic subgradient estimation, or numerical issues arising from the projection-free update?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a DC Frank-Wolfe algorithm for DC programming. The proposed method achieves a first-order stationary point in\n$O(1/\\epsilon^2)$ linear minimization oracle calls. When the set is strongly convex, the oracle complexity improves to $O(1/\\epsilon^{1.5})$. The authors also use the proposed method to smooth optimization and specific decompositions can achieve the gradient computation oracle complexity the same as gradient descent.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. For general DC programming, this paper introduces a DC Frank-Wolfe algorithm, which achieves a first-order stationary point in\n$O(1/\\epsilon^2)$ linear minimization oracles. When the set is strongly convex, the calls of linear minimization oracles improve to $O(1/\\epsilon^{1.5})$. \n\n2. For the special cases of smooth nonconvex optimization, the proposed DC-FW algorithm with a specific decomposition can achieve a $\\epsilon$-suboptimal stationary point measured by $|| x_t - proj_D(x_t-\\nabla \\phi(x_t)) ||^2\\leq gap_{DC}\\leq \\epsilon$ after $O(1/\\epsilon)$ gradient evaluations, which is the same with gradient descent. The proposed DC-FW algorithm is projection-free and it needs $O(1/\\epsilon^2)$ linear minimization oracle calls. When the set is strongly convex,  the oracle complexity improves to $O(1/\\epsilon^{1.5})$.\n\nWeaknesses:\n\n1. For general DC programming, after reading Section 3, it seems that the proposed method is a straightforward application of the FW algorithm to DC programming. The enhanced performance on strongly convex sets directly applies the theoretical results established in (Garber & Hazan, 2015) and this paper contributes nothing new. The novelty and originality seem below the bar of NeurIPS.\n\n2. For the special cases of smooth nonconvex optimization, I wonder whether the DC programming framework is necessary. For problem (7), we can apply the general projected gradient mehtod with the following iterations: \n\n$x_{t+1}=argmin_{x\\in D} \\frac{L}{2} || x - (x_t-\\frac{1}{L}\\nabla \\phi(x_t)) ||^2=argmin_{x\\in D} \\frac{L}{2}||x||^2 - <Lx_t-\\nabla \\phi(x_t), x>$\n\nWhen projection is forbidden, we can use the FW algorithm to solve the above problem in the second equation, that is, subproblem (9) in the main paper. Using the standard analysis of inexact projected gradient descent, we can achieve the $O(1/\\epsilon)$ gradient oracle complexity (the same with PGD). Using the standard analysis of FW algorithm and the theory in (Garber & Hazan, 2015), we can achieve the $O(1/\\epsilon^2)$ and $O(1/\\epsilon^{1.5})$ linear minimization oracle complexity, respectively. The DC programming framework seems unnecessary. We can get the same algorithm and achieve the same complexity using existing standard techniques.\n\n3. For the special cases of smooth nonconvex optimization, when using the decomposition in 4.2, the gradient oracle complexity becomes $O(1/\\epsilon^2)$, which is higher than the $O(1/\\epsilon)$ complexity of PGD.\n\n4. The experiment is inefficient. This paper only compares the posed DC-FW with the original FW algorithm. Suggest also comparing DC-FW with ADMM and its linearized variant (important).\n\nIn conclusion, this paper is well-written. However, the theoretical contribution and novelty are limited.\n\nQUESTIONS:\n1. For the general DC functions, what is the relation bewteen gap_{DC} and $||\\hat \\partial \\phi||$ with $\\hat\\partial \\phi\\in \\partial \\phi$? How to translate gap_{DC} to $||\\hat \\partial \\phi||$ ?\n\n2. What is the benifit of DC structure in nonconvex programming? Can we achieve faster convergence rate than the standard GD or PGD by employing the DC structure?\n\n3. What is the advantage of the DC-FW view compared to the standard analysis framework in Section 4.1?\n\n4. What is the advantage of DC-FW compared to ADMM and its linearized variant.\n\nLIMITATIONS:\n1. For general DC programming, the proposed method seems a straightforward application of the FW algorithm to DC programming.\n\n2. For the special cases of smooth nonconvex optimization, the DC programming framework seems unnecessary.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
In this paper, the authors investigate the Frank-Wolfe algorithm for nonconvex constrained optimization problems with the difference-of-convex structure $f(x) – g(x)$. For this type of problems, a new Frank-Wolfe variant called DC-FW is proposed, which matches the best FW step complexity and achieves a better gradient complexity with respect to function $g(x)$. For strongly-convex feasible domain, it achieves better minimization oracle complexity when the objective functions are strongly-convex or Lipschitz smooth.

STRENGTHS AND WEAKNESSES:
Strength

The authors design a novel and efficient variant of Frank-Wolfe algorithm to solve difference-of-convex optimization problems. The new method matches the best FW step complexity and achieves a better gradient complexity with respect to function $g(x)$. For strongly-convex or Lipschitz smooth problems, the minimization oracle complexity can be improved from $O(1/\epsilon^2)$ to $O(1/\epsilon^{\frac{3}{2}})$.

Weakness

Although the generality of the method is clarified in the introduction section, it is not trivial to convert a practical nonconvex loss function to a DC problem, especially for some complicated tasks in CV or NLP.

QUESTIONS:
1. Can we split any nonconvex objective function into the dffference of two convex functions? For example, we have a two-layer neural network, how should we convert it to a DC problem?
2. After converting a nonconvex problem to a DC problem, will DC-FW still suffer from the saddle point issue?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

comments: SUMMARY:
The paper proposes DC-FW, a projection-free optimization framework for nonconvex problems expressible as the difference of convex (DC) functions. By combining the DC Algorithm (DCA) with Frank-Wolfe (FW) iterations to solve the linearized subproblems, the authors derive an algorithm that maintains the scalability of FW while gaining flexibility from DC decomposition.

STRENGTHS AND WEAKNESSES:
**Strengths**

- A unified view of DC and FW with tight non-asymptotic convergence analysis.
- Accommodates multiple DC decompositions for the same objective.
- Avoids projections—a major bottleneck in constrained optimization.
- Via decomposition that reuses gradients.
- Strong results in classical nonconvex domains.

**Weaknesses**

- Missing real-world deep learning examples: Experiments focus on classical problems.
- Quantitative gains from reusing gradients are not isolated in experiments.

QUESTIONS:
Q1. Could the authors clarify how to choose an optimal DC decomposition for a given objective?

Q2. Is gradient reuse always beneficial?

Q3. Can your method be extended to stochastic settings?

Q4. How does the algorithm behave when $g$ is not differentiable and subgradients are hard to compute?

LIMITATIONS:
- Line-search or step-size tuning in practice might be sensitive, especially in large-scale or noisy settings.

- No statistical significance reported for experimental results.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe article presents a Frank-Wolfe variant (also called projection-free method) DC-FW to minimize a difference of convex functions.\nA proof of rate of convergence matching previously known results is presented.\nIn case of strongly convex constraint set, an improvement for the rate of convergence according to linear minimization oracles is proved.\nTwo main experiments are conducted to compare the performances of the DC-FW against algorithms found in the literature.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nThe paper is well organized and well written.\nThe setting and contributions are clearly \npresented in the introduction.\nThe paper provides theoretical contribution on the convergence rate \nfor DC programming using FW over strongly convex set.\nThe efficiency of DC-FW is demonstrated\nthrough two experiments in paper:\n - Quadratic Assignment Problem (QAP) with  comparisons conducted over **134 datasets**\n - Alignment of Partially Observed Embeddings for 300-dimensional English and French word embeddings.\n The appendix also contains additional experiments with other variants of the algorithm for QAP and additional numerical experiments on neural network training.\n\n**Weaknesses:**\n\nThe influence of the choice of the DC decomposition on the convergence rate is not discussed.\nIn section 4., we are shown two special cases of DC decomposition leading respectively to a gradient sliding problem and proximal point problem for the subproblem of DC-FW. Thus, a  comparison of  between gradient sliding algorithms and proximal point algorithms might be a starting point for a discussion.\n\nQUESTIONS:\nHere are some questions and remarks I had while reading the paper:\n- Line 22-24 Please provide a roof or reference that \"any smooth function can be expressed as a DC decomposition\".\n- In Definition 1, the authors are actually defining a generalized Bregman divergence. See  paragraph 2.2 in  K. C. Kiwiel. _Proximal minimization methods with generalized Bregman functions_. SIAM journal on control and optimization, 35(4):1142–1168, 1997.\n-  Line 120 Please add \"under assumption of $g$ differentiability\"\n- Line 126 Maybe write the formula with a $min$\n  $$\\hat \\Phi_t(x_{t+1}) - \\hat \\Phi_t(x) \\leq \\epsilon/2, \\; \\forall x \\in \\mathcal{D}$$ \n- Line 135-136 Isn't the lsc assumption implied by $L$-smoothness?\n- Line 145-146 \"we can always express as a difference of strongly  convex functions by adding the same quadratic term to both f and g. In this case, subproblem \" proof or reference?\n- Line 168-169 \"but rather focus on the classical FW algorithm\" Where in the Khamaru & Wainwright reference do we have a mention to FW?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}