Title: Heavy Ball Momentum for Conditional Gradient
Authors: Bingcong Li
Alireza Sadeghi
Georgios B. Giannakis
Venue: NeurIPS 2021
Keywords: Frank Wolfe
conditional gradient
heavy ball momentum

Abstract: Conditional gradient, aka Frank Wolfe (FW) algorithms, have well-documented merits in machine learning and signal processing applications. Unlike projection-based methods, momentum cannot improve the convergence rate of FW, in general. This limitation motivates the present work, which deals with heavy ball momentum, and its impact to FW. Specifically, it is established that heavy ball offers a unifying perspective on the primal-dual (PD) convergence, and enjoys a tighter \textit{per iteration} PD error rate, for multiple choices of step sizes, where PD error can serve as the stopping criterion in practice. In addition, it is asserted that restart, a scheme typically employed jointly with Nesterov's momentum, can further tighten this PD error bound. Numerical results demonstrate the usefulness of heavy ball momentum in FW iterations.

TL;DR: Heavy ball momentum tightens the primal dual error of Frank-Wolfe (aka conditional gradient) method.

Review discussion:
I believe heavy ball adaptation of the FW algorithm is an original idea.
In practice, computing Lipschitz constants may be expensive or even impossible;
but these assumptions are common in the literature and provides helpful tools to understand the performance of the algorithms. In that perspective, I believe the authors did a good work providing convergence analysis for various step-size choices (especially the parameter-free step size which does not require computation of these parameters).
I am concerned about the computation cost of solving two subproblems at Algorithm 3, even though restart improves the PD error. Can authors also add the computation time or memory use comparisons of these algorithms? I suspect that the restart approach may be costly at large problems and the benefit we get from restart may not compensate for the PD error improvement.
I found the discussions in the paper clear;
however, it would be great if the authors can provide the advantage of heavy ball momentum over Nesterov's momentum numerically.
In addition to discussion on the paper, including a comparison of memory use (or Type II PD error) on these algorithms could show the advantage of HFW over Nesterov or make the motivation more clear.
Lastly, best to my knowledge, projected gradient descent methods such as projected GD are commonly used in practice.
Therefore, including projected GD in the numerical experiments (in addition to FW variants) would help to understand the performance of the algorithm.
No
Thanks for the nice suggestions. More numerical tests will be added in the revised version.
Q1: Should one be concerned about the “two subproblems” in Algorithm 3?
A1: Algorithm 3 is mainly of theoretical interest as it offers deeper understanding of the generalized FW gap, along with a tighter bound.
On the other hand, HFW is more practical thanks to its simplicity.
Regarding numerical results, we tested Algorithm 3 on the l1 norm ball using the `mushroom’ dataset. Algorithm 3 slightly improves over HFW(WFW) when plotting iteration versus primal error.
When comparing runtime, HFW performs better since Algorithm 3 roughly doubles the computational burden.
Q2: How about additional numerical tests on Nesterov and projected GD?
A2: We will add more numerical results in the updated version, as suggested.
But for certain constraint sets, such as the nuclear norm balls, projection can be costly.
In addition, there are no efficient methods available for projection on to the $n$-support norm balls yet. This amplifies the merits of FW type algorithms.
Q3: How about social impact?
A3: Thank you for pointing out the potential this work can have on fairness in ML. This is a nice direction to pursue, and we have included it in our future research agenda.
Originality and quality:
The result is interesting to me as the momentum technique has been the "crown jewel" of first order optimization but is rarely studied in the literature of Frank-Wolfe. The slowness of the FW is usually attributed to the zig-zag phenomenon and "smoothing" the update direction seems (and is proved to be in this paper) a plausible strategy to alleviate this issue.
Clarity:
This paper is well written and I enjoy reading it (there could be some bias because I am familiar with the literature already). The idea of generalized FW gap is well explained.
Significance:
This is an important result to the literature of Frank-Wolfe method.
Questions:
In the experiment section, the first row of Figure 1 looks impressive.
However, since the constraint is the `$\ell$2-norm ball, the HFW method is equivalent to the normalized gradient descent method with heavy ball momentum (with an additional moving average on the variable). The zig-zag phenomenon is potentially missing in this setting and such a success may be due to that the HFW in this setting is indeed of the $O(1/k^2)$ accelerated convergence rate. The authors should clarify this point in their revision because this will not be a common observation of FW under a polytope constraint where FW often has a zig-zag trajectory.
typos:
In Table 1, the "computation" in the 'second' column.
No
Thanks for the valuable comments, we find them useful to improve the quality of this work. Thanks also for pointing out the typos. We will correct them in the revised version.
Q: How is NGD related with HFW on the l2 norm ball?
A: Indeed, on l2 norm constraints, HFW is similar to normalized gradient descent (NGD). We will add this discussion in the revised paper.
However, there is a subtle difference between HFW and NGD. Consider $||x || \leq 1$ for example. The update of NGD is $x_{k+1} = Proj( x_k - \eta_k \frac{g_k}{\|g_k \|} )$. This implies that when projection is in effect, $x_{k+1}$ will lie on the boundary of the l2 norm ball.
While for FW, the update is $x_{k+1} = (1-\eta_k) x_k + \eta_k \frac{g_k}{\|g_k \|}$. Due to the convex combination in the HFW update, it is unlikely to have $x_{k+1}$ on the boundary, though it can come arbitrarily close.
From my point of view, this is a very good paper. The presentation is clear and concise and, to the best of my knowledge, the reported results and proof techniques are novel and correct. FW algorithms are relevant for various applications and incorporating heavy ball momentum seems natural. This paper fills a gap in the sense that improved convergence guarantees for this case are shown. The reported numerical results support the theoretical claims. I have no further concerns.
No
Thank you for reviewing this submission, and also for the favorable recommendation.
Q: How about social impact?
A: As the topic is centered around analytical perspective of FW, we believe it has no direct social impact.
In my view, the main and most original contribution of this work is the proof technique used in Theorem 1. The conciseness and simplicity of the proof of Theorem 1 make the technique not only elegant but likely useful in other related Frank-Wolfe type setups. The remaining content of the paper, in my view, is ancillary.
More specifically, I feel that the core proof idea of defining the dual error via a weighted average of supporting hyperplanes and having this weighted average of gradients correspond to momentum in the FW algorithm is a very nice and original idea. In hindsight, this is a simple and perhaps natural idea, since the averaging of gradients is reminiscent of dual averaging, but good ideas are often obvious is hindsight.
Granted, I have not written a Frank-Wolfe paper myself so I do not claim full expertise in this area.
However, I have read many Frank-Wolfe-type papers, and this technique feels original and worthy publication to me.
No
We appreciate the time and effort spent for reviewing this submission, and also for recognizing its merits.
Indeed, the core of this work is Theorem 1, based on which other results are developed. Thanks also for pointing out the relation of HFW with dual averaging. Due to space limits, we in fact discuss this point in Appendix B.8

Meta-review:
role: PC
data: comment: While the paper received a relatively high score by the reviewers and it does contain some interesting ideas, I am not not quite convinced that the merit justifies acceptance at NeurIPS. Not wanting to overrule the reviewers I recommend weak acceptance.  

Decision: Accept (Poster)