Title: Fast Projection-Free Approach (without Optimization Oracle) for Optimization over Compact Convex Set
Venue: NeurIPS 2025
Accepted: yes
Keywords: projection-free

homeomorphism

gauge mapping

constrained optimization

convex set
Abstract: Projection-free first-order methods, e.g., the celebrated Frank-Wolfe (FW) algorithms, have emerged as powerful tools for optimization over simple convex sets such as polyhedra, because of their scalability, fast convergence, and iteration-wise feasibility without costly projections. 
However, extending these methods effectively to general compact convex sets remains challenging and largely open, as FW methods rely on expensive linear optimization oracles (LOO), while penalty-based methods often struggle with poor feasibility. 
We tackle this open challenge by presenting **Hom-PGD**, a novel projection-free method without expensive (optimization) oracles. 
Our method constructs a homeomorphism between the convex constraint set and a unit ball, transforming the original problem into an equivalent ball-constrained formulation, thus enabling efficient gradient-based optimization while preserving the original problem structure. 
We prove that Hom-PGD attains *optimal* convergence rates matching gradient descent with constant step-size to find an $\epsilon$-approximate (stationary) solution: $\mathcal{O}(\log (1/\epsilon))$ for strongly convex objectives, $\mathcal{O}(\epsilon^{-1})$ for convex objectives, 
and $\mathcal{O}(\epsilon^{-2})$ for non-convex objectives. 
Meanwhile, Hom-PGD enjoys a low per-iteration complexity of $\mathcal{O}(n^2)$, without expensive oracles like LOO or projection, where $n$ is the input size. 
Our framework further extends to certain non-convex sets, broadening its applicability in practical optimization scenarios with complex constraints. Extensive numerical experiments demonstrate that Hom-PGD achieves comparable convergence rates to state-of-the-art projection-free methods, while significantly reducing per-iteration runtime  (up to 5 orders of magnitude faster) and thus the total problem-solving time.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel method for solving constrained convex optimization problems. The main idea is to construct a homeomorphism between the original problem and a transformed problem with a unit ball constraint. The transformed problem can be solved without using expensive LOO step or complicated projection steps in PGD. The paper provides convergence guarantees for the proposed method in both convex and non-convex settings.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- The method is novel and effective. The homeomorphism and Gauge Mapping is well constructed and the connection between the Global Optimality of the original problem and the stationary point of transformed problem is good. (Proposition 5.2)\n\n- The paper presents strong theoretical results to support the proposed method and shows that it can achieve the same optimal convergence rate as raw gradient descent.\n\n- The empirical results are promising, showing faster convergence compared to LOO and projected gradient descent.\n\n- The paper is well-written and well-structured, I do not find any typos, the visualization is good\n\n\n# Weaknesses:\n- Very minor problem: line 34 could be clarified. Specifically, the case “O(1/epsilon) in the convex setting” requires the gradient to be Lipschitz continuous, and the “O(1/epsilon^2) in the non-convex setting” seems to need additional assumption, i.e., gradient Lipschitz continuity.\n-- Section 4.2 is more important than the theoretical analysis in Section 4.5 in my view because it discuss with the construction of the homeomorphism, which is crucial for practical implementation. However, Algorithm 1, Table 2, and some revalant discussion are deferred to the appendix. It would improve the paper to move them into the main body.\n- The paper claims that a key benefit over FW (or LOO) is avoiding expensive linear minimization. It would be useful to include concrete examples of problems where this is indeed costly, to enhance the paper’s self-consistency. Additionally, the limitation section can be improved by considering the following question.: Are there any case where FW (or LOO) is more efficient than the proposed method? For instance, when the algorithm is slow, or when the closed-form expression in Table 2 is hard to derive while LOO is is easily to solve?\n\nQUESTIONS:\n- Why is the proposed method called \"projection-free\"? The transformed problem still involves a constraint (even though it is a unit ball with closed closed-form solution) and requires projection.\n\n- In Figure 1, why is the initial point located outside the constrained set?\n\n- The paper only evaluates toy examples. Could the method also be applied to practical settings such as adversarial attacks in neural networks, especially with some non-ball constraints? Can the author discuss under which setting the proposed method might be useful for deep learning or neural network training?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates projection-free first-order methods for optimization over a compact and convex set. The key contribution of the paper is to propose an approach, Hom-PGD, which contrary to the popular Frank-Wolfe algorithm does not require a linear minimization oracle (LOO). The main insight is to transform the convex set into a ball, and then solve the transformed problem. The approach achieves optimal convergence rates while enjoying low per-iteration complexity. The experiments show a speed up of 3-6 orders of magnitude in the per-iteration runtime for some problem.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is very clear and easy to follow. Abstract and introduction provide compelling problem statement and motivations.\n- The idea is elegant in its simplicity: transform the convex set into a ball and optimize over the unit ball instead!Proposition 5.2 shows that while this transformation might break convexity, stationary points of the nonconvex problem can be mapped to global optima of the original problem.\n- The convergence analysis is comprehensive (e.g., it covers the convex, strongly convex and non-convex case) and reports optimal convergence rates.\n- The experiments show impressive results, with a 4-6 orders of magnitude per-iteration runtime reduction compared to baselines.\n\nWeaknesses:\n- In general, the paper remains vague about which sets are best handled by the proposed approach versus existing algorithms, and which sets admit a closed-form gauge mapping. This prevents the reader from fully understanding the impact of the proposed method. \n- The experiments would benefit from adding a breakdown of the runtime for the initialization and the last-step computation for Hom-PGD.\n- A main concern is that the experiments do not clarify if the considered problem instances admit a closed-form gauge mapping. In other words, does the speed up demonstrated in the experiments hold even for complicated sets that do not admit a closed-form gauge mapping?\n- The behavior of the ALM approach in Figure 4 seems unexpected. Is the large optimality gap due to the suboptimality gap in the SDP relaxation? Otherwise, I would expect Burer Monteiro to converge fairly quickly?\n\nMinor weaknesses:\n- Line 31: “Despite their slower convergence rate, …”: this sentence is not grammatically correct as written.\n- Right after stating problem (P), it would be useful to provide an example of problems where existing methods do not perform well. This would help the reader visualize the type of constraints that make the problem challenging.\n- Line 90: The paper structure feels a bit redundant and could be merged into the overview of contributions at the end of Section 1.\n- Footnote 2 on page 3 should refer to Appendix B.1 instead of B.2 I think.\n\nQUESTIONS:\n- Which sets are best handled by the proposed approach versus existing algorithms, and which sets admit a closed-form gauge mapping? \n- Does the speed up demonstrated in the experiments hold even for complicated sets that do not admit a closed-form gauge mapping?\n- The behavior of the ALM approach in Figure 4 seems unexpected. Is the large optimality gap due to the suboptimality gap in the SDP relaxation? Otherwise, I would expect Burer Monteiro to converge fairly quickly?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose **Hom-PGD**, a new projection-free first-order method for solving convex optimization problems over general compact convex sets. Hom-PGD reparameterizes the original problem into an equivalent optimization problem constrained to a Euclidean ball, applies standard PGD to solve the reparameterized problem, and then maps the solution back to the original domain. The method achieves optimal convergence rates in the unaccelerated setting and maintains a low per-iteration complexity of $\\mathcal{O}(n^2)$.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The paper presents a thorough literature review of projection-free methods and clearly communicates the theoretical results. Overall, it is well-written, demonstrating both quality and clarity.\n\n2. The proposed algorithm is the first projection-free, first-order framework capable of solving optimization problems over general compact convex sets. It achieves the optimal convergence rate in the unaccelerated setting without relying on expensive oracles, highlighting its significance and originality.\n\n**Weaknesses**\n\n1. The LICQ assumption in Proposition 5.2 appears to be somewhat strong and may limit the applicability of the proposed algorithm. Is it possible to relax this assumption by adopting a weaker constraint qualification?\n\n2. Similarly, the strict complementary slackness assumption in Theorem 1 could be restrictive. Can this assumption be relaxed?\n\n3. In line 261, the authors claim that the proposed algorithm has significantly lower complexity compared to second-order methods. However, is this always the case for first-order methods? It may be beneficial to include a section comparing the overall computational cost with both other first-order and second-order methods.\n\nQUESTIONS:\nSee the Weaknesses section.\n\nOne minor question: Hom-PGD still requires projection onto a ball. Given this, it is unclear whether \"projection-free\" is the most appropriate term to describe the method.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new, simple approach to solving a constrained convex optimization problem. The idea is to compute a homeomorphism from the convex constraint set to the unit ball, and solve the “homeomorphic constrained optimization” problem. To do so, they consider standard projected gradient descent (PGD) in the homeomorphic optimization problem. They provide convergence analysis, complexity analysis, and experiments. They suggest a specific example of a homeomorphism (the gauge/Minkowski function), and provide details about how to efficiently compute/approximate it.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths.**\nThis paper was really a pleasure to read, and introduces a very natural approach to solving constrained convex problems. The theory seems pretty complete. The experiments looked convincing, with quite significant speed-ups in terms of wall-clock time. \n\n**Weaknesses.**\nI do not see any significant weaknesses in this paper. Perhaps it would be nice to include a Hom-PGD pseudocode block somewhere, say in the appendix, and also consider having Equation 35 in the main paper. I think perhaps the minor questions I would have are:\n\n1. The theory assumes that the homeomorphism is twice-differentiable, while the gauge function is only twice-differentiable almost everywhere. While the iterates will almost surely never land on a point at which $\\psi$ is not smooth, does this pose a problem in the analysis when specialized to the gauge homeomorphism?\n\n2. The theoretical analysis assumes access to exact gradients and evaluations of the homeomorphism, while in the algorithm, the gradients and gauge map are approximated. It may be reasonable to provide bounds on how accurately these need to be approximated in order to achieve a target approximation error. However, computing such a bound seems like it would be fairly tedious but not necessarily enlightening.\n\nQUESTIONS:\nI do not really have additional questions beyond the ones I listed above. Some questions that I'm curious about, but are not relevant to how I score the paper, nor do they necessarily need a response: \n\n1. If we only care about the convergence rate in terms of iterations of projected gradient descent, do you think there would be any advantage in recomputing the gauge map centered at a point $\\mathbf{x}^\\circ$ close to the iterates? Or, is it better to just choose $\\mathbf{x}^\\circ$ to be some sort of centroid of $\\mathcal{K}$?\n\n2. You mentioned this approach builds upon [LMY23], which focuses on optimization on the probability simplex. In the case the objective is the simplex, another approach to deal with constraints is to run mirror descent (where the dual space is now unconstrained). The cost here is computing the mirror map versus the gauge map. Do you have a sense for which approach would be favorable?\n\n3. How amenable do you think this homeomorphism approach is to second-order methods?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}