SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Taihei Oki
The University of Tokyo Tokyo, Japan
oki@mist.i.u-tokyo.ac.jp
Shinsaku Sakaue
sakaue@mist.i.u-tokyo.ac.jp
Recent years have seen a growing interest in accelerating optimization algorithms with machine-learned predictions. Sakaue and Oki (NeurIPS 2022) have developed a general framework that warm-starts the  $L$ -convex function minimization method with predictions, revealing the idea's usefulness for various discrete optimization problems. In this paper, we present a framework for using predictions to accelerate  $M$ -convex function minimization, thus complementing previous research and extending the range of discrete optimization algorithms that can benefit from predictions. Our framework is particularly effective for an important subclass called laminar convex minimization, which appears in many operations research applications. Our methods can improve time complexity bounds upon the best worst-case results by using predictions and even have potential to go beyond a lower-bound result.
Recent research on algorithms with predictions [29] has demonstrated that we can improve algorithms' performance beyond the limitations of the worst-case analysis using predictions learned from past data. In particular, a surge of interest has been given to research on using predictions to improve the time complexity of algorithms, which we refer to as warm-starts with predictions for convenience. Since Dinitz et al. [11]'s seminal work on speeding up the Hungarian method for weighted bipartite matching with predictions, researchers have extended this idea to algorithms for various problems [7, 35, 10]. Sakaue and Oki [39] have found similarities between the idea and standard warm-starts in continuous convex optimization and extended it to  $L$ -convex function minimization, a broad class of discrete optimization problems studied in discrete convex analysis [31]. They thus have shown that warm-starts with predictions can improve the time complexity of algorithms for various discrete optimization problems, including weighted bipartite matching and weighted matroid intersection.
In this paper, we extend the idea of warm-starts with predictions to a new direction called  $M$ -convex function minimization, another important problem class studied in discrete convex analysis. The M-convexity is known to be in conjugate relation to the L-convexity. Hence, exploring the applicability of warm-starts with predictions to M-convex function minimization is crucial to broaden further the range of algorithms that can benefit from predictions, as is also mentioned in [39]. This paper mainly discusses an important subclass of M-convex function minimization called laminar convex minimization (Laminar), a large problem class widely studied in operations research (see references in Section 1.2). To make it easy to imagine, we describe the most basic form (Box) of Laminar,
$$
\left(\text {B o x}\right) \quad \underset {x \in \mathbb {Z} ^ {n}} {\text {m i n i m i z e}} \sum_ {i = 1} ^ {n} f _ {i} \left(x _ {i}\right) \quad \text {s u b j e c t t o} \sum_ {i = 1} ^ {n} x _ {i} = R, \ell_ {i} \leq x _ {i} \leq u _ {i} (i = 1, \dots , n), \tag {1}
$$
where  $f_1, \ldots, f_n: \mathbb{R} \to \mathbb{R}$  are univariate convex functions,  $R \in \mathbb{Z}, \ell_1, \ldots, \ell_n \in \mathbb{Z} \cup \{-\infty\}$ , and  $u_1, \ldots, u_n \in \mathbb{Z} \cup \{+\infty\}$ . Note that the variable  $x \in \mathbb{Z}^n$  is an integer vector, which is needed when,
Table 1: Our results and the best worst-case bounds for General, Laminar, Nested, and Box, where General refers to general M-convex function minimization discussed in Section 3.1.  $n$  is the number of variables,  $R$  specifies the equality constraint as in (1), and  $m = |\{ Y \in \mathcal{F} : |Y| \geq 2\} | = \mathrm{O}(n)$  is the number of additional constraints needed to convert Box into Nested and Laminar (see Section 4).
PROBLEM	OUR RESULTS	WORST-CASE TIME COMPLEXITY
General	O(nSFM + n2EOf·||x* - x̂||1)	O(n2log(L/n) min{n, n+log2(L/n)/log n}EOf) [43]
Laminar	O(n||x* - x̂||1)	O(n2log n log mR/n) [18, 34]1
Nested	O(n||x* - x̂||1)	O(n log m log R) [46]
Box	O(n + log n · ||x* - x̂||1)	O(n log R/n) [14, 19]
for example, considering allocating  $R$  indivisible resources to  $n$  entities. As detailed later, adding some constraints and objectives to Box yields more general classes, Nested and Laminar, where the level of generality increases in this order. Streamlining repetitive solving of such problems by using predictions can provide substantial benefits of saving computation costs, as we often encounter those problems in, e.g., resource allocation [22], equilibrium analysis [16], and portfolio management [8].
1.1 Our contribution
We give a framework for accelerating M-convex minimization with predictions building on previous research [11, 39] (Section 3). We show that, given a vector  $\hat{x} \in \mathbb{R}^n$  that predicts an optimal solution  $x^* \in \mathbb{Z}^n$ , the greedy algorithm (Algorithm 1) for M-convex function minimization finds an optimal solution in  $\mathrm{O}(T_{\mathrm{init}} + T_{\mathrm{loc}} \| x^* - \hat{x} \|_1)$  time, where  $T_{\mathrm{init}}$  and  $T_{\mathrm{loc}}$  represent the time for converting  $\hat{x}$  into an initial feasible solution and for finding a locally steepest descent direction, respectively. Since we can minimize  $\| x^* - \hat{x} \|_1$  provably and approximately given optimal solutions to past instances [11, 23], this framework can provide better time complexity bounds benefiting from predictions. We also discuss how to apply our framework to general M-convex function minimization in Section 3.1.
Section 4 presents our main technical results. We apply our framework to Laminar, Nested, and Box and obtain time complexity bounds shown in Table 1. Our time complexity bounds can improve the existing worst-case bounds given accurate predictions. In particular, our  $\mathrm{O}(n\| x^{*} - \hat{x}\|_{1})$ -time bound for Laminar improves the existing  $\mathrm{O}(n^2\log n\log \frac{mR}{n})$ -time bound even if prediction error  $\| x^{*} - \hat{x}\|_{1}$  is as large as  $\mathrm{O}(n)$ . Our result for Nested is a corollary of that for Laminar and improves the existing worst-case bound if  $\| x^{*} - \hat{x}\|_{1} = \mathrm{O}(1)$ . In the case of Box, we can further reduce the time complexity to  $\mathrm{O}(n + \log n\cdot \| x^{*} - \hat{x}\|_{1})$  by modifying the algorithm for Laminar. Notably, our algorithm for Box runs in  $\mathrm{O}(n)$  time if  $\| x^{*} - \hat{x}\|_{1} = \mathrm{O}(n / \log n)$ , even though there exists an  $\Omega (n\log \log (R / n^{2}))$ -time lower bound for Box [19]. As far as we know, this is the first result that can go beyond the lower bound on the time complexity in the context of warm-starts with predictions. Experiments in Section 5 confirm that using predictions can improve empirical computation costs.
Few studies in the literature have made explicit improvements upon the theoretically fastest algorithms, even if predictions are accurate enough. The only exception is [7], whose shortest-path algorithm with predictions can improve the best worst-case time complexity by a couple of log factors. By contrast, our methods with accurate predictions can improve the best worst-case bounds by  $\mathrm{O}(n)$  (up to log factors) in the Laminar case and potentially go beyond the lower-bound result in the Box case. Thus, our work not only extends the class of problems that we can efficiently solve using predictions but also represents a crucial step toward revealing the full potential of warm-starts with predictions. In this paper, we do not discuss the worst-case time complexity of our algorithms since we can upper bound it by executing standard algorithms with worst-case guarantees in parallel, as discussed in [39].
Algorithms with predictions [29], improving algorithms' performance by using predictions learned from past data, is a promising subfield in beyond the worst-case analysis of algorithms [38]. While this idea initially gained attention to improve competitive ratios of online algorithms [36, 2, 28, 1],
<sup>1</sup>While the worst-case analysis in [18, 34] focuses on separable objective functions, we can extend it to more general Laminar in (2) by introducing additional variables at the slight cost of setting  $m = \Theta(n)$ .
recent years have seen a surge of interest in improving algorithms' running time [11, 7, 39, 35, 10]. A comprehensive list of papers on algorithms with predictions is available at [27]. The most relevant to our work is [39], in which predictions are used to accelerate  $\mathrm{L} - \mathrm{L}^{\natural}$ -convex function minimization, a large problem class including weighted bipartite matching and weighted matroid intersection. On the other hand, warm-starts with predictions remain to be studied for M-convex function minimization, $^2$  another essential class that is in conjugate relation to L-convex function minimization in discrete convex analysis [31]. Although our basic idea for utilizing predictions is analogous to the previous approach [11, 39], our algorithmic techniques to obtain the time complexity bounds in Table 1 for the specific M-convex function minimization problems are entirely different (see Section 4).
M-convex function minimization includes many important nonlinear integer programming problems, including Laminar, Nested, and Box, which have been extensively studied in the context of resource allocation [22]. A survey of recent results is given in [41]. Table 1 summarizes the worst-case time complexity bounds relevant to ours. Besides, faster algorithms for those problems under additional assumptions have been studied. For example, Schoot Uiterkamp et al. [41] showed that, if an objective function is a sum of  $f(x_{i} + b_{i})$  ( $i = 1, \dots, n$ ) for some identical convex function  $f$  and  $b_{i} \in \mathbb{Z}$ , we can solve Box, Nested, and Laminar with existing algorithms that run in  $O(n)$  [4, 21],  $O(n \log m)$  [46], and  $O(n^{2})$  [30] time, respectively. Hochbaum [19] gave an  $O(n \log n \log \frac{R}{n})$ -time algorithm for Laminar with separable objective functions and no lower bound constraints. Even in those special cases, our results in Table 1 are comparable or better given that prediction errors  $\| x^{*} - \hat{x} \|_{1}$  are small enough. There also exist empirically fast algorithms [42, 47], whose time complexity bounds are generally worse than the best results. Other problem classes with network and submodular constraints have also been studied [20, 30]. Extending our framework to those classes is left for future work.
Resource allocation with continuous variables has also been well-studied. One may think we can immediately obtain faster algorithms for discrete problems by accelerating continuous optimization algorithms for relaxed problems with predictions and using obtained solutions as warm-starts. However, this is not true since there generally exists an  $\mathrm{O}(n)$  gap in the  $\ell_1$ -norm between real and integer optimal solutions [30, Example 2.9], implying that we cannot always obtain faster algorithms for solving a discrete problem even if an optimal solution to its continuous relaxation is available for free.
Let  $N \coloneqq \{1, \ldots, n\}$  be a finite ground set of size  $n$ . For  $i \in N$ , let  $e_i$  be the  $i$ th standard vector, i.e., all zero but the  $i$ th entry set to one. For any  $x \in \mathbb{R}^N$  and  $X \subseteq N$ , let  $x(X) = \sum_{i \in X} x_i$ . Let  $\lfloor \cdot \rfloor$  denote (element-wise) rounding to a closest integer. For a function  $f: \mathbb{Z}^N \to \mathbb{R} \cup \{+\infty\}$  on the integer lattice  $\mathbb{Z}^N$ , its effective domain is defined as  $\operatorname{dom} f \coloneqq \{x \in \mathbb{Z}^N : f(x) < +\infty\}$ . A function  $f$  is called proper if  $\operatorname{dom} f \neq \emptyset$ . For  $Q \subseteq \mathbb{R}^N$ , its indicator function  $\delta_Q: \mathbb{R}^N \to \{0, +\infty\}$  is defined by  $\delta_Q(x) \coloneqq 0$  if  $x \in Q$  and  $+\infty$  otherwise.
2.1 M-convexity and greedy algorithm for M-convex function minimization
We briefly explain M-convex functions and sets; see [31, Sections 4 and 6] for details. We say a proper function  $f: \mathbb{Z}^N \to \mathbb{R} \cup \{+\infty\}$  is M-convex if for every  $x, y \in \operatorname{dom} f$  and  $i \in \{i' \in N : x_{i'} > y_{i'}\}$ , there exists  $j \in \{j' \in N : x_{j'} < y_{j'}\}$  such that  $f(x) + f(y) \geq f(x - e_i + e_j) + f(y + e_i - e_j)$ . A non-empty set  $Q \subseteq \mathbb{Z}^N$  is said to be M-convex if its indicator function  $\delta_Q: \mathbb{Z}^N \to \{0, +\infty\}$  is M-convex. Conversely, if  $f$  is an M-convex function,  $\operatorname{dom} f$  is an M-convex set. An M-convex set always lies in a hyperplane defined by  $\{x \in \mathbb{R}^N : x(N) = R\}$  for some  $R \in \mathbb{Z}$ . It is worth mentioning that the M-convexity is built upon the well-known basis exchange property of matroids, and the base polytope of a matroid is the convex hull of an M-convex set.
The main subject of this paper is M-convex function minimization,  $\min_{x\in \mathbb{Z}^N}f(x)$ , where  $f:\mathbb{Z}^N\to \mathbb{R}\cup \{+\infty \}$  is an M-convex function. Note that  $\mathrm{dom}f\subseteq \mathbb{Z}^N$  represents the feasible region of the problem. For convenience of analysis, we assume the following basic condition.
Assumption 2.1. An  $M$ -convex function  $f:\mathbb{Z}^N\to \mathbb{R}\cup \{+\infty \}$  always has a unique minimizer  $x^{*}$ .
${}^{2}$  Similar to the  $\mathrm{L}/\mathrm{L}^{\natural }$  -convex case,we can deal with  ${M}^{\natural }$  -convex functions by considering corresponding M-convex functions with one additional variable. See [31, Section 6.1] for details.
<sup>3</sup>An  $\mathrm{O}(n\log n)$ -time algorithm for Laminar (and Nested) with quadratic objective functions was also proposed in [20], but later it turned out incorrect, as pointed out in [30].
This uniqueness assumption is common in previous research [39, 10] (and is also needed in [11, 7, 35], although not stated explicitly). In the case of Laminar, it is satisfied for generic, strictly convex objective functions. Even if not, there are natural tie-breaking rules, e.g., choosing the minimizer that attains the lexicographic minimum among all minimizers closest to the origin; we can implement this by adding  $\epsilon \| x\| _2^2 +\sum_{i = 1}^n\epsilon^{i + 1}|x_i|$  for sufficiently small  $\epsilon \in (0,1)$  to  $f$ , preserving its M-convexity. This is in contrast to the L-convex case, where arbitrarily many minimizers always exist (see [40]).
We can solve M-convex function minimization by a simple greedy algorithm shown in Algorithm 1, which iteratively finds a locally steepest direction,  $-e_i + e_j$ , and proceeds along it. If this update does not improve the objective value, the current solution is ensured to be the minimizer  $x^{*} = \arg \min f$  due to the M-convexity [31, Theorem 6.26]. The number of iterations depends on the  $\ell_1$ -distance to  $x^{*}$  as follows.
Algorithm 1 Greedy algorithm
1:	x← x°
2:	while not converged :
3:	Find i, j ∈ N that minimize f(x - ei + ej)
4:	if f(x) ≤ f(x - ei + ej) :
5:	return x
6:	x ← x - ei + ej
Proposition 2.2 ([44, Corollary 4.2]). Algorithm 1 terminates exactly in  $\| x^{*} - x^{\circ}\|_{1} / 2$  iterations.
A similar iterative method is used in the L-convex case [39], whose number of iterations depends on the  $\ell_{\infty}$ -distance and a steepest direction is found by some combinatorial optimization algorithm (e.g., the Hopcroft-Karp algorithm in the bipartite-matching case). On the other hand, in the M-convex case, computing a steepest direction is typically cheap (as we only need to find two elements  $i, j \in N$ ), while the number of iterations depends on the  $\ell_{1}$ -distance, which can occupy a larger fraction of the total time complexity than the  $\ell_{\infty}$ -distance. Hence, reducing the number of iterations with predictions can be more effective in the M-convex case. Section 3 presents a framework for this purpose.
Similar methods to Algorithm 1 are also studied in submodular function maximization [25]. Indeed, M-convex function minimization captures a non-trivial subclass of submodular function maximization that the greedy algorithm can solve (see [31, Note 6.21]), while it is NP-hard in general [32, 13]
3 Warm-start-with-prediction framework M-convex function minimization
We present a framework for warm-starting the greedy algorithm for M-convex function minimization with predictions. Although it resembles those of previous studies [11, 39], it is worth stating explicitly how the time complexity depends on prediction errors for M-convex function minimization.
We consider the following three phases as in previous studies: (i) obtaining an initial feasible solution  $x^{\circ} \in \mathbb{Z}^{N}$  from a prediction  $\hat{x} \in \mathbb{R}^{N}$ , (ii) solving a new instance by warm-starting an algorithm with  $x^{\circ}$ , and (iii) learning predictions  $\hat{x}$ . The following theorem gives a time complexity bound for (i) and (ii), implying that we can quickly solve a new instance if a given prediction  $\hat{x}$  is accurate.
Theorem 3.1. Let  $f: \mathbb{Z}^N \to \mathbb{R} \cup \{+\infty\}$  be an M-convex function that has a unique minimizer  $x^{*} = \arg \min f$  and  $\hat{x} \in \mathbb{R}^N$  be a possibly infeasible prediction. Suppose that Algorithm 1 starts from an initial feasible solution satisfying  $x^{\circ} \in \arg \min \{ \| x - \lfloor \hat{x} \rceil \|_1 : x \in \operatorname{dom} f \}$ . Then, Algorithm 1 terminates in  $O(\| x^{*} - \hat{x} \|_{1})$  iterations. Thus, if we can compute  $x^{\circ}$  in  $T_{\mathrm{init}}$  time and find  $i, j \in N$  that minimize  $f(x - e_i + e_j)$  in Step 3 in  $T_{\mathrm{loc}}$  time, the total time complexity is  $O(T_{\mathrm{init}} + T_{\mathrm{loc}} \| x^{*} - \hat{x} \|_{1})$ .
Proof. Due to Proposition 2.2, the number of iterations is bounded by  $\| x^{*} - x^{\circ}\|_{1} / 2$ . Thus, it suffices to prove  $\| x^{*} - x^{\circ}\|_{1} = \mathrm{O}(\| x^{*} - \hat{x}\|_{1})$ . By using the triangle inequality, we obtain  $\| x^{*} - x^{\circ}\|_{1} \leq \| x^{*} - \hat{x}\|_{1} + \| \hat{x} -\lfloor \hat{x}\rfloor \|_{1} + \| \lfloor \hat{x}\rfloor -x^{\circ}\|_{1}$ . We below show that the right-hand side is  $\mathrm{O}(\| x^{*} - \hat{x}\|_{1})$ . The second term is bounded as  $\| \hat{x} -\lfloor \hat{x}\rfloor \| _1 \leq \| \hat{x} -x^*\| _1$  since  $x^{*} \in \mathbb{Z}^{N}$ . As for the third term, we have  $\| \lfloor \hat{x}\rfloor -x^{\circ}\|_{1} \leq \| \lfloor \hat{x}\rfloor -x^{*}\|_{1}$  since  $x^{\circ} \in \operatorname {dom}f$  is a feasible point closest to  $\lfloor \hat{x}\rfloor$  and  $x^{*} \in \operatorname {dom}f$ , and the right-hand side,  $\| \lfloor \hat{x}\rfloor -x^{*}\|_{1}$ , is further bounded as  $\| \lfloor \hat{x}\rfloor -x^{*}\|_{1} \leq \| \lfloor \hat{x}\rfloor -\hat{x}\|_{1} + \| \hat{x} -x^{*}\|_{1} \leq 2\| \hat{x} -x^{*}\|_{1}$  due to the previous bound on the second term. Thus,  $\| x^{*} - x^{\circ}\|_{1} \leq 4\| \hat{x} -x^{*}\|_{1}$  holds.
Note that we round  $\hat{x}$  to a closest integer point  $\lfloor \hat{x} \rceil$  before projecting it onto  $\operatorname{dom} f$ , while rounding comes after projection in the  $\mathrm{L} / \mathrm{L}^{\sharp}$ -convex case [39]. This subtle difference is critical since rounding after projection may result in an infeasible integer point that is far from the minimizer  $x^{*}$  by  $\mathrm{O}(n)$  in the  $\ell_{1}$ -norm. To avoid this, we swap the order of the operations and modify the analysis accordingly.
Let us turn to phase (iii), learning predictions. This phase can be done in the same way as previous studies [11, 23]. In particular, we can learn predictions in an online fashion with the standard online subgradient descent method (OSD) as in [23], which works as follows in our case. Let  $V \subseteq \mathbb{R}^N$  be a convex set that we expect to contain an optimal prediction. For any sequence of M-convex functions  $f_1, \ldots, f_T$ , we regard  $L_t(\hat{x}) \coloneqq \| x_t^* - \hat{x} \|_1$  for  $t = 1, \ldots, T$  as loss functions, where  $x_t^*$  is the minimizer of  $f_t$ . Fixing  $\hat{x}_1 \in V$  arbitrarily, for  $t = 1, \ldots, T$ , OSD iteratively computes a subgradient  $z_t \in \partial L_t(\hat{x}_t)$  and set  $\hat{x}_{t+1} = \arg \min_{\hat{x} \in V} \| \hat{x}_t - \eta z_t - \hat{x} \|_2$ , where  $\eta > 0$  is the step size. OSD returns predictions  $\hat{x}_1, \ldots, \hat{x}_T$  that enjoy a regret bound (see, e.g., [33]), and a sample complexity bound follows from online-to-batch conversion [5, 9]. Formally, the following proposition guarantees that we can provably learn predictions to decrease the time complexity bound in Theorem 3.1.
Proposition 3.2 ([23]). Let  $f_t: \mathbb{Z}^N \to \mathbb{R} \cup \{+\infty\}$  for  $t = 1, \ldots, T$  be a sequence of  $M$ -convex functions, each of which has a unique minimizer  $x_t^* = \arg \min f_t$ , and  $V \subseteq \mathbb{R}^N$  be a closed convex set whose  $\ell_2$ -diameter is  $D$ . Then, OSD with  $\eta = D / \sqrt{nT}$  returns  $\hat{x}_1, \ldots, \hat{x}_T \in V$  that satisfy
$$
\sum_ {t = 1} ^ {T} \| x _ {t} ^ {*} - \hat {x} _ {t} \| _ {1} \leq \min  _ {\hat {x} ^ {*} \in V} \sum_ {t = 1} ^ {T} \| x _ {t} ^ {*} - \hat {x} ^ {*} \| _ {1} + O (D \sqrt {n T}).
$$
Furthermore, for any distribution  $\mathcal{D}$  over  $M$ -convex functions  $f: \mathbb{Z}^N \to \mathbb{R} \cup \{+\infty\}$ , each of which has a unique minimizer  $x_f^* = \arg \min f$ ,  $\delta \in (0,1]$ , and  $\varepsilon > 0$ , given  $T = \Omega\left(\left(\frac{D}{\varepsilon}\right)^2 n \log \frac{1}{\delta}\right)$  i.i.d. draws,  $f_1, \ldots, f_T$ , from  $\mathcal{D}$ , we can compute  $\hat{x} \in V$  that satisfies
$$
\underset {f \sim \mathcal {D}} {\mathbb {E}} \| x _ {f} ^ {*} - \hat {x} \| _ {1} \leq \min  _ {\hat {x} ^ {*} \in V} \underset {f \sim \mathcal {D}} {\mathbb {E}} \| x _ {f} ^ {*} - \hat {x} ^ {*} \| _ {1} + \varepsilon
$$
with a probability of at least  $1 - \delta$  via online-to-batch conversion (i.e., we apply OSD to  $\| x_{f_t}^* -\cdot \| _1$  for  $t = 1,\ldots ,T$  and average the outputs).
The convex set  $V$  should be designed based on prior knowledge of upcoming instances. For example, previous studies [11, 39] set  $V = [-C, + C]^N$  for some  $C > 0$ , which is expected to contain optimal solutions of all possible instances; then  $D = 2C\sqrt{n}$  holds. In our case, we sometimes know that the total amount of resources is fixed, i.e.,  $x(N) = R$ , and that every  $x_{i}$  is always non-negative. Then, we may set  $V = \{x\in [0,R]^N:x(N) = R\}$ , whose  $\ell_2$ -diameter is  $D = R\sqrt{2}$ .
3.1 Time complexity bound for general M-convex function minimization
We here discuss how to apply the above framework to general M-convex function minimization. The reader interested in the main results in Table 1 can skip this section and go to Section 4.