Reusing Combinatorial Structure: Faster Iterative Projections over Submodular Base Polytopes

Abstract: Jai Moondra, Hassan Mortagy, Swati Gupta

Reviews and author responses:

This paper considers the problem of Bregman projections on Submodular Base polytopes. The authors present several tools that could speed up computation when this projection is computed for several nearby points, as is the case for example in iterative algorithms such as mirror descent. In particular, the authors first present an algorithm for the special case of cardinality-based submodular functions, that improve over current best known running time. Then they present tools, for general submodular functions, to detect tight inequalities, reuse active sets, restrict linear optimization to the detected tight inequalities, and round approximate projections to exact ones. They show how these tools can be incorporated into away-step Frank-Wolfe (AFW). They also provide some empirical results showing a reduction in running time of online mirror descent (OMD) when their proposed variant of AFW is used for computing projections, compared to using regular AFW.  Originality and significance:
The result presented for the cardinality-based submodular functions is not really new. In particular, Theorem 2 is a special case of Proposition 8.1 in [19]. But to the best of my knowledge, this paper is the first to explicitly state the relation between Bregman projections on cardinality-based submodular polytopes and isotonic optimization, which is useful to highlight. 

The tools presented for general submodular functions to accelerate Bregman projections are novel. Whether or not they indeed lead to a speed up depends on if the conditions assumed hold or not. The empirical results provided show some evidence that these tools could be helpful. Though in certain cases, one of the tools (reusing active sets) doesn't have an effect (see Figure 4, OMD-ASAFW vs OMD-UAFW), and in others some of the tools lead to worst running time (see Figure 6-right, OMD-A^2FW vs OMD-ASAFW). More comprehensive empirical evaluation is needed. 

Quality: I checked most proofs, they are correct. However the experimental results are not convincing enough.
The examples used (both in the main text and appendix) are too small (ground set size n=25). It is also not clear from the reported results how OMD with the proposed variant of AFW compares with simply using online FW. Showing plots of both regret and runtime for all compared methods (and not just variants of AFW) would better highlight this trade-off. It would be good to also compare with other methods of computing projections. I am particularly interested to see a comparison with minimum norm point algorithm, which already includes one of the proposed tools (rounding to optimal face).
Some information about the set-up is missing: what is the submodular function and Bregman divergence in the first example?

Clarity: The paper is written clearly, with only few typos. 

Other comments/suggestions for improvements:

- The proposed rounding to optimal face (T5) is a generalization of rounding done in minimum norm point algorithm. This is stated on lines 117-118, but it would be good to also restate that lemma 2 is a generalization of the same property used in MNP for $h(x) = \tfrac{1}{2} ||x||_2^2$.

- l.68: there's several more recent papers than [34] that provide faster algorithms for submodular function minimization, with the current fastest approximate algorithm having subquadratic time:

"Chakrabarty, D., Lee, Y. T., Sidford, A., and Wong, S. C.-w. Subquadratic submodular function minimization. In
Proceedings of the ACM SIGACT Symposium on Theory of Computing, STOC 2017"  

- is $n = |E|$? if yes, specify that in the preliminaries.

- first footnote on p.4: this is the definition of convex functions not strictly convex

-----After Rebuttal-----
Thanks for the detailed response. 
I want to clarify that by comparison with MNP I did not mean for a single projection, but a comparison of OMD with the proposed variant of AFW used for projection vs OMD with MNP used for projection. Authors adequately addressed the limitations and potential negative societal impact of their work.


 We thank the reviewer for their detailed reviews and comments. We address their main concerns below.


**1. "On the other hand, the second contribution is difficult to evaluate. The authors gave several interesting tools to speed up the Bregman projection. However, these are simply heuristics and do not yield an improved time complexity."**

To compute a projection over the challenging submodular base polytope (with exponential constraints typically), there are mainly two potential ways of doing so: (i) using Frank-Wolfe variants due to attractiveness of linear subproblems, (ii) using combinatorial algorithms such as [Groenevelt, 1991] and [Nagano and Aihara, 2012] (which typically rely on submodular function minimization for detecting tight sets). Our tools for detecting tight sets can be used to speed-up both of these approaches. However, combinatorial algorithms are very expensive in practice for general submodular polytopes (due to solving iterative submodular function minimizations) and hence our focus is on FW approaches in this work. Moreover, these tools can be applied for most projection-based iterative algorithms optimization algorithms such as FISTA, projected Newton's method (Table 1), not just for FW.

The Frank-Wolfe literature has attempted to incorporate the geometry of the problem in various ways, however none of the existing work utilizes combinatorial properties of previous minimizers or detect tight sets with provable guarantees and round to those. To the best of our knowledge, we are the first to adapt AFW to consider improvements over the basic AFW algorithm using our toolkit. Although A$^2$FW is a standalone algorithm for strongly convex and smooth optimization over base polytopes, A$^2$FW further allows to combine these tools that reuse structure from past projections (e.g. **T1** and **T3**) and obtain even more computational improvements over the basic AFW algorithm. As discussed above, if one decided to compute a projection using FW variants, then our A$^2$FW algorithm would outperform the regular AFW variant due to improved convergence rate in practice (Theorem 6), and hence we can also compute an _exact_ projection quicker by invoking rounding. 

This was verified by our computational experiments. In particular, our experiments show that out toolkit results in orders of magnitude speedup in OMD. Having a theoretical result on improved time complexities is open in the worst-case and would require assumptions on the optimal face. We will add a discussion on this in open questions. 


**2. "Also, the authors proposed Adaptive Away-steps Frank-Wolfe (A2FW) and proved its linear convergence. But the linear convergence holds only for strongly convex and smooth mirror maps, which do not include important projection in online learning such as KL-projection. So it is unclear whether these results improve the time complexity of OMD. The present paper provides empirical evaluation only."**

To get linear convergence, we had to assume smoothness and strong convexity. That being said, iterative optimization methods such as mirror descent and away-step Frank-Wolfe have also been analyzed under more relaxed assumptions (e.g. smoothness only). In these relaxed cases though, the convergence rates will not be linear, but rather sub-linear. In any case, since our algorithm is at least as fast as the basic AFW algorithm, we inherit all its rates in those more relaxed assumptions. We will discuss these extensions in the revision. 


**3. "Note that OMD is a highly theoretical algorithm with the optimal regret guarantee. For a practical purpose, one can use Follow-the-Perturbed-Leader (FPL) because each iteration is very cheap for submodular polytopes. Does FPL perform poorly in practice? How does OMD with the proposed heuristics compare with FPL in running time? The lack of theoretical improvements and comparison is a weak point of the paper."**

We thank the reviewer for pointing this out. We do think this is an interesting venue for exploration, and we will add a comparison in the camera ready. The reason why we didn't include FPL is due to its similarity (both algorithmically and in the regret rates) to the OFW variant we used, which was developed by [Hazan and Minasyan, 2020]. As stated in their paper [Hazan and Minasyan, 2020]: 


*"The algorithm we propose is fairly straightforward, and the main hurdle lies in the analysis. The seminal work of [Kalai and Vempala, 2005] introduces the Follow-the-Perturbed-Leader (FPL) online algorithm that obtains optimal $O(\sqrt{T})$ regret for **linear loss functions**. A more general version of FPL  that applies expectations over the perturbations at each iteration extends the result to general **convex functions** [Hazan, 2016]. Our algorithm mimics the expected FPL replacing the computationally expensive expectations with empirical averages of i.i.d. samples. In particular, this restores the original result of the FPL method attaining $O(\sqrt{T})$ regret with $m = 1$ sample per iteration for linear loss functions as shown in [Kalai and Vempala, 2005]."*

The, OFW variant that we use of [Hazan and Minasyan, 2020] is essentially a practical extension of FPL for the case of convex and smooth loss functions and provably achieves a regret of $O(T^{2/3})$ by taking a large enough number of samples $m = O(\sqrt{T})$ every iteration. However, for the case of linear loss (which is our case since we consider linear loss functions in our experiments), their OFW algorithm coincides with FPL. We will add a discussion and an additional experiment in the revision. 

It might also be worth noting that in our experiments, we find that OMD has a regret 3-10 times lower than OFW on average, while OFW was 3-5 times faster than OMD with A$^2$FW algorithm. So there is a real running time v/s regret tradeoff. That being said, we emphasize that the main goal of our work is to not improve online learning, but to show speed ups in iterative optimization methods using past projections (which we do show, with orders of magnitude improvement). In particular, speeding up OMD is an example of the impact of our toolkit, it's not the main focus of our paper.


**References:**

[1.] Groenevelt, Henri. "Two algorithms for maximizing a separable concave function over a polymatroid feasible region." European Journal of Operations Research 54.2 (1991): 227-236.


[2.] Hazan, Elad. "Introduction to online convex optimization." arXiv preprint arXiv:1909.05207 (2019).


[3.] Hazan, Elad, and Edgar Minasyan. "Faster projection-free online learning." Conference on Learning Theory. PMLR, 2020.


[4.] Kalai, Adam, and Santosh Vempala. "Efficient algorithms for online decision problems." Journal of Computer and System Sciences 71.3 (2005): 291-307.


[5.] Nagano, Kiyohito, and Kazuyuki Aihara. "Equivalence of convex minimization problems over base polytopes." Japan journal of industrial and applied mathematics 29.3 (2012): 519-534.


 We thank the reviewer for taking the time to review the paper. We will first address the limitations that they state of our work.

**1. Motivation for Frank-Wolfe v/s Projection-based Methods.**

We are not at all discrediting Frank-Wolfe method and variants. Online Frank Wolfe was 3-5 orders of magnitude faster in speed in our experiments than OMD with A$^2$FW algorithm (we mention this drop in running time in lines 351-352 for OFW, and in the appendix). The high running time of projection-based variants is the reason they have remained of "theoretical" nature, especially for submodular polytopes where projections are especially challenging. We are simply highlighting the fact that there is more that can be done to speed up projection-based methods (such as projected Newton's method, FISTA, mirror descent variants) by learning from past projections in an iterative setting. On the other hand, Frank-Wolfe variants are especially well-suited to machine learning applications given the simplicity of linear subproblem calls, especially for submodular polytopes. We also report that OFW was 6-7 orders of magnitude faster than OMD with unoptimized AFW (lines 351-352), and make a qualitative statement about this for online learning on our non-cardinality submodular polytopes (line 1018, appendix), but we should have better highlighted it generally. We will revise our exposition given the extra space, and make all the comparisons absolutely clear in terms of both the running time and regret. However, we would again like to emphasize: our goal is to speed up iterative projections; online mirror descent is an example.

**2. Motivation for studied framework in the context of submodularity.**

Submodularity has many many applications in machine learning, and these polytopes are particularly challenging from an optimization perspective since they typically have an exponential number of constraints! We will add an extensive discussion on applications of our framework in the revision. One of our experiments deals with learning over the permutahedron motivated by recommendation systems. Another experiment uses neighborhoods of bipartite graphs as the submodular function, from Jegelka et. al in NeurIPS 2011. There are many more applications of submodularity such as structured regression with submodular penalties for background subtraction in video processing and topographic dictionary learning [Mairal et al. 2011 (b)], structured sparse PCA [Jenatton et al. 2011], wavelet denoising [Rao et al. 2011; Mairal et al. 2011 (a)], to name a few. In many such applications, the goal is to solve a convex minimization problem over a base polytope, which could be solved using our A$^2$FW as we discuss in what follows.

**3. Problem setup and notation.**

We understand that our attempts to motivate a broad class of problems may have been confusing, and we apologize for that. We will include examples in the introduction to make the set up more accessible. An example problem we tackle is the following: to speed up projected gradient descent over the permutahedron. Here, our iterative optimization method is the projected gradient. Each iteration of projected gradient descent involves computing a projection onto the permutahedron, which is a submodular polytope. Our main focus is to use projections in earlier iterations of the PGD method to speed up the computation of projections in latter iterations. Any other iterative method, such as projected Newton's method, or FISTA, can be plugged in instead of PGD. Similarly, any submodular polytope such as spanning trees polytope, or k-subsets, can be plugged in into our setup. We provide Table 1 (examples of iterative algorithms), Table 2 (examples of common submodular polytopes), and Table 3 (examples of mirror maps), and  to give examples of the setup. We will improve this exposition in the revision.

**4. Focus on Away-Step Frank-Wolfe algorithm and theoretical guarantees on combinations of these methods.**

Most of the paper is devoted to our main contribution: our toolkit for speeding-up iterative projections over base polytopes. These tools can be applied for most projection-based iterative algorithms optimization algorithms such as FISTA, projected Newton's method (Table 1), not just for Away-Step Frank-Wolfe. Even combinatorial (exact) algorithms for computing the projection under Euclidean distance functions, such as decomposition approaches of Groenevelt [Groenevelt, 1991] and Nagano and Aihara [Nagano and Aihara, 2012] rely on an expensive submodular function minimization to detect tight sets. Our tools, on the other hand, can {\it reuse} this information from past projections. We further show that our adaptive away-step algorithm which uses the toolkit also enjoys linear convergence. Showing a better rate than that would be highly dependent on the optimal face, and we leave this as an open question.

**5. Line 192 “Therefore, for cardinality-based polytopes, using a combinatorial 193 algorithm is better than any iterative optimization method.” → I do not see why this method is "better than any iterative optimization method."**

In this paragraph, we meant to highlight that combinatorial algorithms over cardinality-based polytopes are better than iterative methods due to two reasons: (1) the total runtime of the PAV algorithm is $O(n\log n)$ compared to the same per iteration running time of Frank-Wolfe for linear optimization, (2) PAV algorithm solves the problem {\it exactly} for simple Bregman projections such as Euclidean projection, and therefore is much faster than an iterative continuous optimization method. We will clarify this better in the revision.

**6. Limitation in the experiments.**

We apologize for the short and terse description of the computations. Due to the page limit, we moved much of the discussion to appendix G and H of the paper. The running time for Online Frank Wolfe was 3 to 5 orders of magnitude faster than OMD with A$^2$FW algorithm, and 6-7 orders of magnitude faster than OMD with {\it unoptimized} AFW (reported in lines 351-352) for our online learning experiment on the permutahedron, and make a qualitative statement about this for online learning on our non-cardinality submodular polytopes (in the appendix). We will revise our exposition given the extra space, and make all the comparisons absolutely clear in terms of both the running time and regret.

Our primary goal in this paper was to demonstrate how combinatorial properties can help speed-up iterative continuous optimization algorithms, and our experiments were aimed at giving a first evidence that our techniques lead to orders of magnitude speedup. We did include some experimental results for non-cardinality-based polytopes in Appendix H. We use randomly generated bipartite graphs, and consider the neighborhood function as the general submodular function (e.g., these functions are also use in [Jegelka et. al, NeurIPS 2011]) and show qualitatively similar results on these polytopes.

On the issue of size of the ground sets: our first experiment was in fact performed for $n = 100$. It was aimed at demonstrating that a large fraction of cuts can be inferred by our tool INFER1 once enough points in a neighborhood have been projected. Each of the 500 runs of the experiment had 500 points projected onto the permutahedron with $n=100$.

We acknowledge the reviewer's concern that more evaluation with larger dimensions for the second experiment is needed. We have performed subsequent experiments for $n = 50$ and $n = 100$, we have observed results very similar to those for $n = 25$ reported in the paper with large speed-ups. We will include these experiments in the revision of the paper, and discuss trade-offs with regret and running time (dependent on the desired accuracy for each projection) in the learning framework.

**References:**

[1.] Groenevelt, Henri. "Two algorithms for maximizing a separable concave function over a polymatroid feasible region." European Journal of Operations Research 54.2 (1991): 227-236.

[2.] Jegelka, Stefanie, Hui Lin, and Jeff A. Bilmes. "On fast approximate submodular minimization." Advances in Neural Information Processing Systems 24 (2011): 460-468.

[3.] Jenatton, Rodolphe, Jean-Yves Audibert, and Francis Bach. "Structured variable selection with sparsity-inducing norms." The Journal of Machine Learning Research 12 (2011): 2777-2824.

[4.] Mairal, J., Jenatton, R., Obozinski, G., \& Bach, F. "Convex and Network Flow Optimization for Structured Sparsity." Journal of Machine Learning Research 12.9 (2011).

[5.] Mairal, J., Jenatton, R., Obozinski, G., \& Bach, F. "Learning hierarchical and topographic dictionaries with structured sparsity." Wavelets and Sparsity XIV. Vol. 8138. International Society for Optics and Photonics, 2011.

[6.] Nagano, Kiyohito, and Kazuyuki Aihara. "Equivalence of convex minimization problems over base polytopes." Japan Journal of Industrial and applied mathematics 29.3 (2012): 519-534.

[7.] "A clustering approach to optimize online dictionary learning." 2012 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2012.

 We thank the reviewer for their positive feedback and comments to improve the presentation of the paper. We will be able to move the A$^2$FW algorithm to the main body, given the extra page for the final version. The only other comment that the reviewer has is to use real-life data and include experiments from the submodular function minimization literature. We will expand the experimental examples accordingly. We would like to highlight that our second experiment actually  mimics a recommendation system over the permutahedron (see details in appendix G). We artificially generated loss functions based on an assumption for what the preferences for a population of users might look like, and implemented online mirror descent as our iterative projection-based method, using the Euclidean distance function. We will add more detail on the motivation for our experiments in the camera ready version, given the extra space. We also used a general (non-cardinality based) submodular function from Jegelka et. al (NeurIPS 2011) [Jegelka et. al, 2011], which is obtained from the neighborhood of random bipartite graphs. We included this experiment in Appendix H, due to space constraints. We will expand the discussion on experiments.

**References:**

[1.] Jegelka, Stefanie, Hui Lin, and Jeff A. Bilmes. "On fast approximate submodular minimization." Advances in Neural Information Processing Systems 24 (2011): 460-468.

 We thank the reviewer for their detailed reviews and for checking our proofs for correctness. In what follows, we focus on the most significant comments raised by the reviewer; we will carefully implement all minor suggestions and include useful references.

**1. "Theorem 2 is a special case of Proposition 8.1 in [19]. But to the best of my knowledge, this paper is the first to explicitly state the relation between Bregman projections on cardinality-based submodular polytopes and isotonic optimization, which is useful to highlight."**

To the best of our knowledge, we agree that we are the first to explicitly state the relation between Bregman projections on general cardinality-based submodular polytopes and isotonic optimization. Theorem 2 is indeed a special case of Proposition 8.1 in [Bach, 2013]. This is, however, the starting point of our proof, and further work is needed to simplify the dual problem to isotonic optimization (Lemmas 4 and 5). We will highlight this further in our exposition.

**2. "The experimental results are not convincing enough. The examples used (both in the main text and appendix) are too small (ground set size n=25)."**

Our primary goal in this paper was to demonstrate how combinatorial properties can help speed-up iterative continuous optimization algorithms, and our experiments were aimed at giving a first evidence that our techniques lead to orders of magnitude speedup. On the issue of size of the ground sets: our first experiment was in fact performed for $n = 100$. It was aimed at demonstrating that a large fraction of cuts can be inferred by our tool INFER1 once enough points in a neighborhood have been projected. We used a fairly large dataset in this case - each of the 500 runs of the experiment had 500 points projected onto the permutahedron.

We acknowledge the reviewer's concern that more experiments with larger dimensions for the second experiment are needed. We have performed subsequent experiments for $n = 50$ and $n = 100$, we have observed results very similar to those for $n = 25$ reported in the paper with large speed-ups. We will include experiments with larger $n$ in the camera-ready version of the paper, and discuss trade-offs with regret and running time (dependent on the desired accuracy for each projection) in the learning framework. Again, note that even with a small ground set size $n$, the number of constraints in submodular polytopes is exponential in $n$.

**3. "Some information about the set-up is missing: what is the submodular function and Bregman divergence in the first example?"**

We apologize for the short and terse description of the computations in the main body (we moved some of the discussion of the experiments to Appendix G and H). We will expand this discussion significantly and clarify the setup better in the main body of the paper. We used the permutahedron with Euclidean distance functions in the first experiment and second experiments.

To show the benefit of our approach in an iterative projections setting, we mimicked a recommendation system in our second experiment in the main body of the paper. We artificially generated loss functions based on an assumption for what random preferences for a population of users might look like, and implemented online mirror descent as our iterative projection-based method over the permutahedron with the Euclidean distance. We also included an experiment with non-cardinality based submodular function, based on neighborhoods of randomly generated bipartite graphs (inspired by experiments in [Jegelka et. al, NeurIPS 2011]). We showed qualitatively similar improvements in orders of magnitude running time of OMD for non-cardinality polytopes.

**4. "It is also not clear from the reported results how OMD with the proposed variant of AFW compares with simply using online FW. Showing plots of both regret and runtime for all compared methods (and not just variants of AFW) would better highlight this trade-off."**

In the second experiment, we mimicked an online recommendation engine by learning over the permutahedron. We implemented Online Mirror Descent with variants on the projection subroutine, and Online Frank Wolfe. We reported that Online Mirror Descent has a regret 3-10 times lower than Online Frank Wolfe on average, thus bolstering the claim that we need to invest research to speed-up this optimal learning method and its variants. This drop in regret is _significant_ in terms of revenue for an online retail platform.  In our experiments, the running time for Online Frank Wolfe was 3 to 5 orders of magnitude lower than OMD with A$^2$FW (we report that OFW was 6-7 orders of magnitude faster than OMD with {\it unoptimized} AFW in lines 351-352), and further discuss drop in running time for non-cardinality submodular polytopes in the appendix (line 1018). We will revise our exposition given the extra space, and make all the comparisons absolutely clear in terms of both the running time and regret. In our plots, given the focus of the paper to improve iterated projection time, we plot running time using OMD with various projection subroutines plugged in, including using individual tools such as REUSE (corresponding to OMD-ASAFW variant) and INFER (corresponding to OMD-TSAFW variant). Including the time of OFW moved the focus from the interesting parts of the toolkit related results. Your point is well taken however and we will highlight the OFW performance in regret and running time better. 

**5. "It would be good to also compare with other methods of computing projections. I am particularly interested to see a comparison with minimum norm point algorithm, which already includes one of the proposed tools (rounding to optimal face)."**

Our toolkit has the most significant impact when learning from repeated (general) projections of nearby points, e.g., in a outer learning problem such as online mirror descent. Therefore, comparing our toolkit to Wolfe's minimum norm point may not be a fair comparison. We suspect the comparison between MNP v/s AFW with rounding (which is what our toolkit would reduce to for a single projection) would show better iteration count and slower time to converge for MNP. We will add this experiment and add a comment about this comparison, since it may be interesting to the submodular optimization researchers and practitioners. 

**6. "Whether or not they indeed lead to a speed up depends on if the conditions assumed hold or not. The empirical results provided show some evidence that these tools could be helpful. Though in certain cases, one of the tools (reusing active sets) doesn't have an effect (see Figure 4, OMD-ASAFW vs OMD-UAFW), and in others some of the tools lead to worst running time (see Figure 6-right, OMD-A$^2$FW vs OMD-ASAFW). More comprehensive empirical evaluation is needed.**

We thank the reviewer for noticing this variance in behavior in our experiments. The plot in Figure 6-right shows an {\it improvement} in running time compared to OMD with unoptimized AFW by using {\it any} of our tools. Though our toolkit is useful in general, the magnitude of its usefulness can vary dependent on the problem and data. The key reason behind this is simple: the time to check whether a tool is useful in an iteration can sometimes reduce the benefit of using that tool, this is what leads to differences between A$^2$FW and ASAFW. What would be very interesting going forward is to be able to learn/predict which tools might be useful for which problem types and data variations. Our focus in this work has been to show a significant impact in orders of magnitude improvement in the runtime for iterative projections (using the example of OMD) with our toolkit. Predicting the set of tools which would be most impactful remains open. We will discuss this as an open question in the revision.

**References:**

[1.] Jegelka, Stefanie, Hui Lin, and Jeff A. Bilmes. "On fast approximate submodular minimization." Advances in Neural Information Processing Systems 24 (2011): 460-468.

This paper presents  new techniques to speed up projections to submodular base polytopes.   The techniques are inspired by both discrete and continuous perspectives.  Arranging the proposed tools together, they present the adaptive away-step Frank-Wolfe algorithm.  The runtime of computing certain Bregman projection is improved by a factor of $\Omega(n/log(n))$.  Contributions:  

This paper presents several techniques to speed up projections over submodular polytopes.  They give solid theoretical proofs of the tools and algorithms.  It can be viewed as a combinations of several existing techniques, say,  Edmonds greedy algorithm, Frank-Wolfe et al.  However, the toolkit proposed surely introduces some new techniqes.  

Overall, I think it is a solid work in terms theorecial exploration of fast projecting to submodular base polytopes.  


Presentation & clarity: 

- Very well structured paper, main technical components are very well presented, e.g., the toolkit T1-T6.  I really like the illustrations in Figures 1 -3.   

- Better to put Alg. 5 (A^2FW) in the main text,  (though I know it was defered to appendix due to space limit).  It is a pretty well designed algorithm.   



Experiments:  

Figure 4 clearly show computational advantages of your techniques on synthetic data. 
It would be great to see some real-world experiments, say, some experiments from submodular function minimizaiton literature or others.   
 


Minor typos:  

- line 306:  weather -> whether 
- line  64:  their -> there  Yes

This paper provides a faster algorithm for the Bregman projection onto submodular base polytopes.
- For the case the submodular function is cardinality-based and the mirror map is separable convex, the authors show that the Bregman projection can be done in $O(n \\log n + n EO)$ time, where $n$ is the size of the ground set and $EO$ is the running time of the evaluation oracle of the submodular function. This result improved the previous $O(n^2 EO)$ time.
- Furthermore, the authors provided a toolkit to speed up the Bregman projection in practice. In the numerical experiment, the algorithm with the toolkit runs faster in several orders of magnitudes.  The paper is well-written and easy to read. The improved algorithm for cardinality-based submodular functions seems nice.

On the other hand, the second contribution is difficult to evaluate. The authors gave several interesting tools to speed up the Bregman projection. However, these are simply heuristics and do not yield an improved time complexity. Also, the authors proposed Adaptive Away-steps Frank-Wolfe (A2FW) and proved its linear convergence. But the linear convergence holds only for strongly convex and smooth mirror maps, which do not include important projection in online learning such as KL-projection. So it is unclear whether these results improve the time complexity of OMD. The present paper provides empirical evaluation only.

Note that OMD is a highly theoretical algorithm with the optimal regret guarantee. For a practical purpose, one can use Follow-the-Perturbed-Leader (FPL) because each iteration is very cheap for submodular polytopes. Does FPL perform poorly in practice? How does OMD with the proposed heuristics compare with FPL in running time? The lack of theoretical improvements and comparison is a weak point of the paper.

Overall, the paper has a neat result but is not strong enough from the theoretical perspective. N/A

The paper focuses on projection-base methods such as mirror descent for problems, where constraint sets are defined by submodular polytopes. These problems are challenging as they contain an exponential number of constraints. These projection-based methods have superior convergence rates in comparison with their Frank-Wolfe counterparts. However, the latter optimizes a linear objective in each iteration that is easier than doing the projection required in the former. The paper precisely addresses this issue and proposes an Away Step Frank-Wolfe method that leverages the submodular structure of the problem to solve the projection step efficiently. 
  The paper focuses on providing ways for speeding up the projection based optimization methods such as mirror descent that have superior convergence rates, but suffer from expensive projection steps. They concentrate on constraints defined by submodular polytopes. This is an interesting contribution and they provide theoretical results on their proposed Away Step Frank-Wolfe for preforming projections; however, I am not convinced that these problems are well-motivated. They briefly mention applications such as principal component analysis subject to k-sets. However, I believe these applications should be presented in more details to provide motivations for the studied algorithm. Presentation of the paper is decent; however, some notations are confusing. For instance, the main class of problems that the paper aims to solve are never explicitly presented using math, which makes following the paper hard. Moreover, the experimental results are lacking and the description of the numerical results is extremely sparse and rushed. Finally, the paper focuses all of its main sections on their Away-Step Frank-Wolf algorithm; however, this method is only interesting in conjunction with projection based optimization methods. It would be interesting to see analysis or theoretical guarantees on combinations of these methods. Moreover, the paper needs to provide more motivations for considering this class of projection-based methods; indeed, in experiments , Online Frank-Wolfe algorithm is mentioned as a competitor, but is not reported in Fig.  4.  

Typos and minor issues

(a) Line 64 exit→ existing 
(b) Line 138 put space before "We" 
(c) Line 192 “Therefore, for cardinality-based polytopes, using a combinatorial 193 algorithm is better than any iterative optimization method.” → I do not see why this method is "better than any iterative optimization method." 
 Despite presenting an interesting contribution, I see serious limitations for the current results.

(a) As mentioned, the paper does not provide enough evidence for the interest in the studied class of problems. Moreover, it does not explicitly state the studied class of problems.

(b) The paper focuses on their proposed method for performing projection more precisely; however, it needs to provide stronger  motivations on why the projection based methods are considered in the first place. I understand that the projection based methods come with a superior convergence rate. However, I do not believe that this discredits Frank-Wolfe methods altogether. A more insightful approach would be to state computation complexity along with convergence rates. In this way, we would be able to understand for which problem sizes the projection based methods are more efficient.

(c) The experimental results are quite limited: first, the presentation of data and problems are extremely lacking and confusing. Second, they are reported on one dataset that is not large. Third, the results for Online Frank-Wolfe algorithm is not reported in the figure.   

Meta-review: This paper presents new techniques to speed up projections to submodular base polytopes. The authors gave theoretical results and some experimental evidence regarding the speed-up. I highly appreciated the rebuttal and discussion followed for this paper. 

As reviewers indicated, there is a mismatch between the theory and practice. The experiments are quite small and miss important baselines. So, it is not clear whether the proposed method is really useful. Unless the authors address this issue, the paper may not be accepted in the current format.  

Decision: accept