SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this paper we designed two new parallel algorithms for minimizing submodular functions  $f: 2^{[n]} \to \mathbb{Z} \cap [-M, + M]$  with round complexities  $\widetilde{O}(n^{1/3}M^{2/3})$  and 2, and query complexities  $\widetilde{O}(n^2M^2)$  and  $O(n^{M+1})$ , respectively. These  $M$ -dependent sublinear dependence on  $n$  in the round complexities stand in contrast to the  $\widetilde{\Omega}(n)$ -lower bound on the number of rounds required for SFM when  $M = n^{\Theta(n)}$ . On the way to the first result, we obtain a new efficient parallel algorithm for  $\varepsilon$ -approximate minimization of  $\ell_{\infty}$ -Lipschitz convex functions over  $[0,1]^n$  with round-complexity  $\widetilde{O}(n^{1/3}\varepsilon^{-2/3})$ . Given results of [Nem94, DG19] the dependence on  $n$  is optimal for constant  $\varepsilon$ .
Two related open questions are whether one can obtain  $o(n)$ -round SFM algorithms with polylogarithmic dependence on  $M$ , and whether one can obtain algorithms for  $\varepsilon$ -approximate minimization of  $\ell_{\infty}$ -Lipschitz convex functions over  $[0,1]^n$  in  $\widetilde{O}(n^{1/3}\mathrm{poly}\log(1/\varepsilon))$ -rounds, or can one prove lower bounds ruling them out. Another related open question is whether one can perform SFM in  $O(\mathrm{poly}(M))$ -rounds with query complexity  $\mathrm{poly}(n)$ .
We thank the anonymous reviewers of NeurIPS 2023 for helpful comments. Part of this work was done while Deeparnab Chakrabarty, Haotian Jiang, and Aaron Sidford were attending the Discrete Optimization trimester program at the Hausdorff Research Institute for Mathematics.
[ACJ+21] Hilal Asi, Yair Carmon, Arun Jambulapati, Yujia Jin, and Aaron Sidford. Stochastic bias-reduced gradient methods. Advances in Neural Information Processing Systems, 34:10810-10822, 2021.
[ALS20] Brian Axelrod, Yang P Liu, and Aaron Sidford. Near-optimal approximate discrete and continuous submodular function minimization. In Proceedings of the Fourteenth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 837-853. SIAM, 2020.
[BJL+19] Sébastien Bubeck, Qijia Jiang, Yin Tat Lee, Yuzhhi Li, and Aaron Sidford. Complexity of highly parallel non-smooth convex optimization. Advances in Neural Information Processing Systems, 2019.
[BPC+11] Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine learning, 3(1):1-122, 2011.
[BS18] Eric Balkanski and Yaron Singer. Parallelization does not accelerate convex optimization: Adaptivity lower bounds for non-smooth convex minimization. arXiv preprint arXiv:1808.03880, 2018.
[BS20] Eric Balkanski and Yaron Singer. A lower bound for parallel submodular minimization. In Proceedings of the 52nd annual ACM SIGACT Symposium on Theory of Computing (STOC), pages 130-139, 2020.
[BV04] Dimitris Bertsimas and Santosh Vempala. Solving convex programs by random walks. Journal of the ACM (JACM), 51(4):540-556, 2004.
[BVZ01] Yuri Boykov, Olga Veksler, and Ramin Zabih. Fast approximate energy minimization via graph cuts. IEEE Transactions on pattern analysis and machine intelligence, 23(11):1222-1239, 2001.
[CCK21] Deeparnab Chakrabarty, Yu Chen, and Sanjeev Khanna. A polynomial lower bound on the number of rounds for parallel submodular function minimization and matroid intersection. In 2021 IEEE 62nd Annual Symposium on Foundations of Computer Science (FOCS). IEEE, 2021.
[CGJS22] Deeparnab Chakrabarty, Andrei Graur, Haotian Jiang, and Aaron Sidford. Improved lower bounds for submodular function minimization. In 2022 IEEE 63rd Annual Symposium on Foundations of Computer Science (FOCS), pages 245-254. IEEE, 2022.
[CGT00] AR Conn, NIM Gould, and Ph L Toint. Trust region methods, mps/siam ser. Optim., SIAM, Philadelphia, PA, 2000.
$\left[\mathrm{CJJ}^{+}20\right]$  Yair Carmon, Arun Jambulapati, Qijia Jiang, Yujia Jin, Yin Tat Lee, Aaron Sidford, and Kevin Tian. Acceleration with a ball optimization oracle. Advances in Neural Information Processing Systems, 33:19052-19063, 2020.
[CJJ+23] Yair Carmon, Arun Jambulapati, Yujia Jin, Yin Tat Lee, Daogao Liu, Aaron Sidford, and Kevin Tian. Resqueing parallel and private stochastic convex optimization. arXiv preprint arXiv:2301.00457, 2023.
[CLSW17] Deeparnab Chakrabarty, Yin Tat Lee, Aaron Sidford, and Sam Chiu-wai Wong. Subquadratic submodular function minimization. In Proceedings of the 49th Annual ACM SIGACT Symposium on Theory of Computing (STOC), pages 1220–1231, 2017.
[DBW12] John C Duchi, Peter L Bartlett, and Martin J Wainwright. Randomized smoothing for stochastic optimization. SIAM Journal on Optimization, 22(2):674-701, 2012.
[DG19] Jelena Diakonikolas and Cristóbal Guzmán. Lower bounds for parallel and randomized convex optimization. In Conference on Learning Theory, pages 1132–1157. PMLR, 2019.
[DVZ21] Daniel Dadush, László A. Végh, and Giacomo Zambelli. Geometric rescaling algorithms for submodular function minimization. Mathematics of Operations Research, 46(3):1081-1108, 2021.
[Edm70] Jack Edmonds. Submodular functions, matroids, and certain polyhedra. Edited by G. Goos, J. Hartmanis, and J. van Leeuwen, page 11, 1970.
[FG88] Awi Federgruen and Henry Groenevelt. Characterization and optimization of achievable performance in general queueing systems. Operations Research, 36(5):733-741, 1988.
[FI00] Lisa Fleischer and Satoru Iwata. Improved algorithms for submodular function minimization and submodular flow. In Proceedings of the thirty-second annual ACM symposium on Theory of computing, pages 107-116, 2000.
[GLS81] Martin Grötschel, László Lovász, and Alexander Schrijver. The ellipsoid method and its consequences in combinatorial optimization. Combinatorica, 1(2):169-197, 1981.
[GLS88] Martin Grötschel, László Lovász, and Alexander Schrijver. Geometric algorithms and combinatorial optimization. Springer, 1988.
[GPRW20] Andrei Graur, Tristan Pollner, Vidhya Ramaswamy, and S. Matthew Weinberg. New query lower bounds for submodular function minimization. 11th Innovations in Theoretical Computer Science Conference, ITCS, pages 64:1-64:16, 2020.
[GR18] Pontus Giselsson and Anders Rantzer. Large-scale and distributed optimization, volume 2227. Springer, 2018.
[Har08] Nicholas James Alexander Harvey. Matchings, matroids and submodular functions. PhD thesis, Massachusetts Institute of Technology, 2008.
[IFF01] Satoru Iwata, Lisa Fleischer, and Satoru Fujishige. A combinatorial strongly polynomial algorithm for minimizing submodular functions. Journal of the ACM (JACM), 48(4):761-777, 2001.
[IO09] Satoru Iwata and James B Orlin. A simple combinatorial algorithm for submodular function minimization. In Proceedings of the twentieth annual ACM-SIAM symposium on Discrete algorithms, pages 1230-1237. SIAM, 2009.
[Iwa03] Satoru Iwata. A faster scaling algorithm for minimizing submodular functions. SIAM Journal on Computing, 32(4):833-840, 2003.
[JB11] Stefanie Jegelka and Jeff Bilmes. Submodularity beyond submodular energies: coupling edges in graph cuts. In CVPR 2011, pages 1897-1904. IEEE, 2011.
[JBS13] Stefanie Jegelka, Francis Bach, and Suvrit Sra. Reflection methods for user-friendly submodular optimization. Advances in Neural Information Processing Systems, 26, 2013.
[Jia21] Haotian Jiang. Minimizing convex functions with integral minimizers. In Proceedings of the 2021 ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 976-985. SIAM, 2021.
[Jia22] Haotian Jiang. Minimizing convex functions with rational minimizers. Journal of the ACM, 70(1):1-27, 2022.
[QS95] Maurice Queyranne and Andreas S Schulz. Scheduling unit jobs with compatible release dates on parallel machines with nonstationary speeds. In *Integer Programming and Combinatorial Optimization: 4th International IPCO Conference Copenhagen, Denmark*, May 29–31, 1995 Proceedings 4, pages 307–320. Springer, 1995.
$\left[\mathrm{SBB}^{+}18\right]$  Kevin Scaman, Francis Bach, Sébastien Bubeck, Laurent Massoulie, and Yin Tat Lee. Optimal algorithms for non-smooth distributed optimization in networks. Advances in Neural Information Processing Systems, 31, 2018.
[Sch00] Alexander Schrijver. A combinatorial algorithm minimizing submodular functions in strongly polynomial time. Journal of Combinatorial Theory, Series B, 80(2):346-355, 2000.
[Sho77] Naum Z Shor. Cut-off method with space extension in convex programming problems. Cybernetics, 13(1):94-96, 1977.
[Top98] Donald M Topkis. Supermodularity and complementarity. Princeton university press, 1998.
[Vai89] Pravin M Vaidya. A new algorithm for minimizing convex functions over convex sets. In 30th Annual IEEE Symposium on Foundations of Computer Science (FOCS), pages 338-343, 1989.
[Ver18] Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
[Vyg03] Jens Vygen. A note on Schrijver's submodular function minimization algorithm. Journal of Combinatorial Theory, Series B, 88(2):399-402, 2003.
[YN76] David B Yudin and Arkadii S Nemirovski. Evaluation of the information complexity of mathematical programming problems. *Ekonomika i Matematicheskie Metody*, 12:128-142, 1976.
A Supplement
In this supplement, we provide the complete proofs of Lemma 2.3 and Theorem 2.2.
Proof of Lemma 2.3. First we show that  $f_{\mathrm{reg}}^{c,r}(x)$  is convex and  $3L$ -Lipschitz. Let  $h(x) \coloneqq 2L(\| x - c\| - r)$ . By triangle inequality and homogeneity of norms,  $2L(\| x - c\| - r)$  is convex and  $2L$ -Lipschitz (with respect to  $\| \cdot \|$ ). Further, it is straightforward to check that the maximum of two convex,  $L$ -Lipschitz functions is convex and  $L$ -Lipschitz and that the sum of a  $L$ -Lipschitz and  $2L$ -Lipschitz function is convex and  $3L$ -Lipschitz. Since  $f_{\mathrm{reg}}^{c,r}(x) = f(x) + \max \{0, h(x)\}$  and constant functions are convex and 0-Lipschitz the result follows.
Next, consider  $y \in \mathbb{R}^n$  with  $\| y - c \| \geq r$ . Since  $\| \cdot \|$  is a norm we see that  $\| y^{c,r} - c \| = r$  and therefore  $f_{\mathrm{reg}}^{c,r}(y^{c,r}) = f(y^{c,r})$ . Furthermore,  $\| y^{c,r} - y \| = \| y - c \| - r$  and so,
$$
\begin{array}{l} f _ {\text {r e g}} ^ {c, r} (y) = f (y) + 2 L (\| y - c \| - r) \geq f (y ^ {c, r}) - L \| y ^ {c, r} - y \| + 2 L (\| y - c \| - r) \\ = f (y ^ {c, r}) + L (\| y - c \| - r). \\ \end{array}
$$
where the inequality follows since  $f$  is  $L$ -Lipschitz with respect to  $\| \cdot \|$ . This yields the desired result as it implies that values of  $f_{\mathrm{reg}}^{c,r}$  are all larger than those of points where  $\| x - c \| \leq r$  for which  $f_{\mathrm{reg}}^{c,r}(x) = f(x)$ . Further, since  $\{x \in \mathbb{R}^n | \| x - c \| \leq r\}$  is closed and  $f$  is Lipschitz, there exists a minimizer of  $\min_{x \in \mathbb{R}^n | \| x - c \| \leq r} f(x)$  and the result follows.
Proof of Theorem 2.2. We invoke Lemma 2.3 and apply BallAccel on the function  $F \coloneqq f_{\mathrm{reg}_{\rho}}^{c,R}$  as defined in Lemma 2.3 with respect to  $\| \cdot \|_2$ ,  $c$  being the origin, and  $\rho \coloneqq \frac{\epsilon_{\mathrm{opt}}}{L\sqrt{2\log n}}$ . With this choice of  $\rho$ , by Lemma 2.6, we know  $|F(x) - f_{\mathrm{reg}}^{c,R}(x)| \leq \epsilon_{\mathrm{opt}}$  everywhere. For brevity, we remove the superscript  $(c,R)$  from  $f_{\mathrm{reg}}$  for the remainder of the proof.
We now claim that  $F$  has a minimizer  $x_{*}$  with  $\| x_{*}\|_{2}\leq 3R$ . To see this, let  $y\in \mathbb{R}^n$  be a point with  $\| y\| _2 > 3R$  and consider  $\tilde{y} = \frac{y}{\|y\|_2}\cdot R$ . Therefore,
$$
F (y) \geq f _ {\operatorname {r e g}} (y) - \epsilon_ {\text {o p t}} = f (y) + 2 L \cdot (\| y \| _ {2} - R) - \epsilon_ {\text {o p t}} \geq f _ {\operatorname {r e g}} (\tilde {y}) + L \cdot (\| y \| _ {2} - R) - \epsilon_ {\text {o p t}}
$$
where the first inequality follows from pointwise approximation of  $F$  and  $f_{\mathrm{reg}}$ , and the second follows since  $f$  is also  $L$ -Lipschitz in the  $\| \cdot \|_2$  norm, and so  $f_{\mathrm{reg}}(\tilde{y}) = f(\tilde{y}) \leq f(y) + L \cdot (\| y \|_2 - R)$ . Again using the pointwise approximation of  $F$  and  $f_{\mathrm{reg}}$  we get  $F(y) \geq F(\tilde{y}) - 2\epsilon_{\mathrm{opt}} + L \cdot (\| y \|_2 - R)$ , and if  $\| y \|_2 > 3R$  using that  $\epsilon_{\mathrm{opt}} \in (0, LR]$  we get a contradiction to minimality of  $F(y)$ .
Note that the stochastic subgradient  $g'$  of  $f_{\mathrm{reg}}$  is given by  $g' = g + 2L \cdot v$  for  $v \in \partial h(x)$  where  $h(x) = \max(0, \|x\|_2 - R)$ . Note that  $\|v\|_2^2 \leq 1$ , so the stochastic gradient  $g'$  satisfies  $\mathbb{E}\|g'(x)\|_2^2 \leq 2\sigma^2 + 8L^2$ . It follows that by setting  $r = \rho$ , Proposition 2.8 implies that for any  $\phi, \lambda > 0$ , there exists a  $(\phi, \lambda, \rho)$ -ball optimization oracle for  $F$  which makes  $O\left(\frac{\sigma^2}{\phi\lambda}\right)$  total queries to  $g$  in  $O(1)$  parallel rounds (as  $\sigma^2 \geq L^2$  by definition of  $L$ -Lipschitzness).
Next we apply ball acceleration in Proposition 2.9 to  $F$ . We have already argued above that the minimizer  $x_{*}$  of  $F$  satisfies  $\| x_{*}\|_{2}\leq 3R$ . We set the parameter  $r = \rho = \frac{\epsilon_{\mathrm{opt}}}{L\sqrt{2\log n}}$  and the  $R$  and  $L$  multiplied by factor 3. Using these, the parameters of (6) in Proposition 2.9 become
$$
\kappa = \frac {L R}{\epsilon_ {\mathrm {o p t}}}, K = \left(\frac {L R \sqrt {2 \log n}}{\epsilon_ {\mathrm {o p t}}}\right) ^ {2 / 3}, \text {a n d} \lambda_ {\star} = \frac {\epsilon_ {\mathrm {o p t}} K ^ {2}}{R ^ {2}} \cdot \log^ {2} \kappa .
$$
Using Proposition 2.9 and Proposition 2.8, we get that the number of rounds of queries is  $(CK\log \kappa)$ $(C\log^{2}(R\kappa /\rho))$  which is  $\widetilde{O} ((LR / \epsilon_{\mathrm{opt}})^{2 / 3})$
Next, we bound the query complexity. Adding up the queries made by the ball-optimization oracle calls made in all iterations per part 1 of Proposition 2.9 is
$$
\left(C K \log \kappa\right) \cdot \left(C \log^ {2} \left(R \kappa / \rho\right)\right) \cdot \frac {\sigma^ {2}}{\lambda_ {\star} ^ {2} \rho^ {2}} = \widetilde {O} \left(K \sigma^ {2} \lambda_ {\star} ^ {- 2} \rho^ {- 2}\right).
$$
Adding up the queries made by the ball-optimization oracle calls made in all iterations per part 2 of Proposition 2.9 is
$$
\sum_ {j \in [ \lceil \log_ {2} K + C \rceil ]} \left(C ^ {2} 2 ^ {- j} K \log (R \kappa / \rho)\right) \cdot \frac {2 ^ {j} C \sigma^ {2} \log^ {2} (R \kappa / \rho)}{\lambda_ {\star} ^ {2} \rho^ {2}} \text {w h i c h i s a l s o} \widetilde {O} \left(K \sigma^ {2} \lambda_ {\star} ^ {- 2} \rho^ {- 2}\right).
$$
Substituting the values from above and  $\rho = \frac{\epsilon_{\mathrm{opt}}}{L\sqrt{2\log n}}$ , we get that the total query complexity is  $\widetilde{O}((\sigma R / \epsilon_{\mathrm{opt}})^2)$ .
Finally note that Proposition 2.9 applied to  $F$  computes an  $\epsilon_{\mathrm{opt}}$ -approximate minimizer which is a  $2\epsilon_{\mathrm{opt}}$ -approximate minimizer of  $f_{\mathrm{reg}}$ . By setting  $\epsilon_{\mathrm{opt}} = \varepsilon / 2$ , Lemma 2.3 implies that we can find an  $\varepsilon$ -approximate minimizer  $x$  of  $f$  with  $\| x \|_2 \leq R$ . This completes the proof of the theorem.