SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wenjing Chen, Victoria G. Crawford  
Department of Computer Science & Engineering  
Texas A&M University  
jj9754@tamu.edu, vcrawford@tamu.edu
In this paper, we consider the optimization problem Submodular Cover (SCP), which is to find a minimum cardinality subset of a finite universe  $U$  such that the value of a submodular function  $f$  is above an input threshold  $\tau$ . In particular, we consider several variants of SCP including the general case, the case where  $f$  is additionally assumed to be monotone, and finally the case where  $f$  is a regularized monotone submodular function. Our most significant contributions are that: (i) We propose a scalable algorithm for monotone SCP that achieves nearly the same approximation guarantees as the standard greedy algorithm in significantly faster time; (ii) We are the first to develop an algorithm for general SCP that achieves a solution arbitrarily close to being feasible; and finally (iii) we are the first to develop algorithms for regularized SCP. Our algorithms are then demonstrated to be effective in an experimental section on data summarization and graph cut, two applications of SCP.
Submodularity captures a diminishing returns property of set functions: Let  $f: 2^U \to \mathbb{R}$  be defined over subsets of a universe  $U$  of size  $n$ . Then  $f$  is submodular if for all  $A \subseteq B \subseteq U$  and  $x \notin B$ ,  $f(A \cup \{x\}) - f(A) \geq f(B \cup \{x\}) - f(B)$ . Examples of submodular set functions include cut functions in graphs [Balkanski et al., 2018], information-theoretic quantities like entropy and mutual information [Iyer et al., 2021], determinantal point processes [Gillenwater et al., 2012], and coverage functions [Bateni et al., 2017]. Submodular set functions arise in many important real-world applications including active learning [Kothawade et al., 2021, 2022], partial label learning [Bao et al., 2022], structured pruning of neural networks [El Halabi et al., 2022], data summarization [Tschiatschek et al., 2014], and client selection in federated learning [Balakrishnan et al., 2022].
While the majority of existing work has focused on developing approximation algorithms to maximize a submodular function subject to some constraint [Nemhauser et al., 1978a, Mirzasoleiman et al., 2015a, Harshaw et al., 2019, Buchbinder et al., 2014], in this paper we focus on developing algorithms for the related optimization problem of Submodular Cover (SCP), defined as follows.
Problem 1 (Submodular Cover (SCP)). Let  $f:2^{U}\to \mathbb{R}_{\geq 0}$  be a nonnegative submodular set function defined over subsets of the ground set  $U$  of size  $n$ . Given threshold  $\tau \leq \max \{f(X):X\subseteq U\}$ , SCP is to find  $\mathrm{argmin}\{|X|:f(X)\geq \tau \}$ .
SCP captures applications where we seek to achieve a certain value of  $f$  in as few elements as possible. For example, consider data summarization, where a submodular function  $f$  is formulated to measure how effectively a subset  $X$  summarizes the entire dataset  $U$  [Tschiatschek et al., 2014]. Then if we set  $\tau = \max \{f(X):X\subseteq U\}$ , SCP asks to find the set of minimum size in  $U$  that achieves the maximum effectiveness as a summary. Another example is when expected advertising revenue
functions are formulated over subsets of a social network [Hartline et al., 2008], then SCP asks how we can reach a certain amount of revenue while picking as small a subset of users as possible.
In this paper, we propose and analyze algorithms for several variants of SCP including the general case, the case where  $f$  is assumed to be monotone<sup>1</sup>, and finally when  $f$  is a regularized monotone submodular function (and potentially takes on negative values). We now list an overview of the contributions of our paper. In addition, a table summarizing all of our algorithmic contributions can be found in Table 1 in the appendix.
(i) We first address the need for scalable algorithms for SCP where  $f$  is assumed to be monotone (MSCP). While the greedy algorithm finds the best possible approximation guarantee for MSPC [Feige, 1998], it makes  $O(n^{2})$  queries of  $f$  which may be impractical in many applications. We propose and introduce two algorithms for MSPC which achieve nearly the same theoretical guarantee as the greedy algorithm but only make  $O(n\ln (n))$  queries of  $f$ . In addition, we extend the work of Iyer and Bilmes [2013a] to a method of converting fast randomized approximation algorithms for the dual cardinality constrained monotone submodular maximization problem [Nemhauser et al., 1978b] into approximation algorithms for MSPC.
(ii) Next, we address the need for algorithms that can produce nearly feasible solutions to the general SCP problem, where  $f$  can be nonmonotone. In particular, we provide the first algorithm for SCP that returns a solution  $S$  that is guaranteed to satisfy: (i)  $f(S) \geq (1 - \epsilon)\tau$ ; and (ii)  $|S| \leq 2(1 + \alpha)|OPT| / \epsilon$  where  $OPT$  is an optimal solution to the instance and  $\epsilon, \alpha > 0$  are input parameters. A caveat for our algorithm is that it is not necessarily polynomial time and requires an exact solution to the cardinality constrained submodular maximization problem [Buchbinder et al., 2014] on an instance of size  $O(|OPT| / \epsilon^2)$ .
(iii) Third, we are the first to consider regularized monotone SCP (RMSCP). RMSCP is where the objective  $f = g - c$  where  $g$  is a nonnegative, monotone, and submodular function and  $c$  is a modular cost penalty function.  $f$  is not necessarily monotone but potentially takes on negative values, and therefore this new problem doesn't fall under the general SCP problem. We develop a method of converting algorithms for the regularized monotone submodular maximization problem [Harshaw et al., 2019] into ones for RMSCP. We then propose the first algorithm for RMSCP, which is a greedy algorithm using queries to a distorted version of  $f = g - c$ .
(iv) Finally, we conduct an experimental analysis for our algorithms for MSPC and general SCP on instances of data summarization and graph cut. We find that our algorithms for MSPC make a large speedup compared to the standard greedy approach, and we explore the pros and cons of each relative to the other. We also find that our algorithm for general SCP is practical for our instances despite not being guaranteed to run in polynomially many queries of  $f$ .
1.1 Preliminary Definitions and Notation
We first provide a number of preliminary definitions that will be used throughout the paper: (i) The Submodular Maximization Problem (SMP) is the dual optimization problem to SCP defined by, given budget  $\kappa$  and nonnegative submodular function  $f$ , find  $\operatorname{argmax}\{f(X):X\subseteq U,|X|\leq \kappa \}$ ; (ii) Monotone SCP (MSCP) is the special case of SCP where  $f$  is additionally assumed to be monotone; (iii) Regularized MSP (RMSCP) is where  $f = g - c$  and  $g$  is monotone, submodular, and nonnegative, while  $c$  is a modular² nonnegative cost function; (iv) OPT is used to refer to the optimal solution to the instance of SCP that should be clear from the context; (v) OPTSM is used to refer to the optimal solution to the instance of SMP that should be clear from the context; (vi) An  $(\alpha ,\beta)$ -bicriteria approximation algorithm for SCP returns a solution  $X$  such that  $|X|\leq \alpha |OPT|$  and  $f(X)\geq \beta \tau$ . An  $(\alpha ,\beta)$ -bicriteria approximation algorithm for SMP returns a solution  $X$  such that  $f(X)\geq \alpha f(OPT)$  and  $|X|\leq \beta \kappa$ . Notice that in the  $(\alpha ,\beta)$  notation, the approximation on the objective is first, and the approximation on the constraint is second; (vii) The marginal gain of adding an element  $u\in U$  to a set  $S\subseteq U$  is denoted as  $\Delta f(S,u) = f(S\cup u) - f(S)$ ; (viii) The function  $f_{\tau} = \min \{f,\tau \}$ .
$^{1}$ A set function  $f$  is monotone if for all  $A \subseteq B \subseteq U$ ,  $f(A) \leq f(B)$ .
2Every  $x\in U$  is assigned a cost  $c_{x}$  such that  $c(X) = \sum_{x\in X}c_x$
MSCP is the most studied variant of SCP [Wolsey, 1982, Wan et al., 2010, Mirzasoleiman et al., 2015b, 2016, Crawford et al., 2019]. The standard greedy algorithm produces a logarithmic approximation guarantee for MSP in  $O(n^{2})$  queries of  $f$  [Wolsey, 1982], and this is the best approximation guarantee that we can expect unless NP has  $n^{\mathcal{O}(\log (\log (n)))}$ -time deterministic algorithms [Feige, 1998]. One version of the greedy algorithm for MSP works as follows: A set  $S$  is initialized to be  $\emptyset$ . Iteratively, the element  $\mathrm{argmax}\{\Delta f(S,x):x\in U\}$  is added to  $S$  until  $f(S)$  reaches  $(1 - \epsilon)\tau$ . It has previously been shown that this is a  $(\ln (1 / \epsilon),1 - \epsilon)$ -bicriteria approximation algorithm [Krause et al., 2008]. Beyond greedy algorithms, algorithms for the distributed setting [Mirzasoleiman et al., 2015c, 2016], the streaming setting [Norouzi-Fard et al., 2016], as well as the low-adaptivity setting Fahrbach et al. [2019] for MSP have been proposed.
On the other hand, developing algorithms for SCP in full generality is more difficult since the monotonicity of  $f$  is not assumed. It is not even obvious how to find a feasible solution. The standard greedy algorithm does not have any non-trivial approximation guarantee for SCP. In fact, to the best of our knowledge, no greedy-like algorithms have been found to be very useful for SCP. Recently, Crawford [2023] considered SCP and proved that it is not possible to develop an algorithm that guarantees  $f(X) \geq \tau / 2$  for SCP in polynomially many queries of  $f$  assuming the value oracle model. On the other hand, algorithmic techniques that are used for SMP in the streaming setting [Alaluf et al., 2022] proved to be useful for SCP. In particular, Crawford [2023] proposed an algorithm using related techniques to that of Alaluf et al. that achieves a  $(O(1 / \epsilon^2), 1 / 2 - \epsilon)$ -bicriteria approximation guarantee for SCP in polynomially many queries of  $f$ . We also take an approach inspired by the streaming algorithm of Alaluf et al., but sacrifice efficiency in order to find a solution for SCP that is arbitrarily close to being feasible.
SMP has received relatively more attention than SCP [Nemhauser et al., 1978b, Badanidiyuru and Vondrák, 2014, Mirzasoleiman et al., 2015a, Feige et al., 2011, Buchbinder et al., 2014, Alaluf et al., 2022]. Iyer and Bilmes [2013b] proposed a method of converting algorithms for SMP to ones for SCP. In particular, given a deterministic  $(\gamma, \beta)$ -bicriteria approximation algorithm for SMP, the algorithm convert (see pseudocode in Section 4.1 in the supplementary material) proposed by Iyer and Bilmes produces a deterministic  $((1 + \alpha)\beta, \gamma)$ -bicriteria approximation algorithm for SCP. The algorithm works by making  $\log_{1 + \alpha}(n)$  guesses for  $|OPT|$  (which is unknown in SCP), running the SMP algorithm with the budget set to each guess, and returning the smallest solution with  $f$  value above  $\gamma\tau$ . However, this approach is limited by the approximation guarantees of existing algorithms for SMP. The best  $\gamma$  for monotone SMP is  $1 - 1/e$ , and the best for general SMP where  $f$  is not assumed to be monotone is significantly lower [Gharan and Vondrák, 2011]. Several of the algorithms that we propose in this paper do generally follow the model of convert in that they rely on guesses of  $|OPT|$ , but are different because they: (i) Implicitly use bicriteria approximation algorithms for SMP which have better guarantees on the objective  $(\gamma)$  because they do not necessarily return a feasible solution; (ii) Are more efficient with respect to the number of queries of  $f$ , since convert potentially wastes many queries of  $f$  by doing essentially the same behavior for different guesses of  $|OPT|$ .
2 Algorithms and theoretical guarantees
In this section, we present and theoretically analyze our algorithms, with each subsection corresponding to the variant of SCP we consider. In particular, in Section 2.1 we first consider algorithms for MSCP, followed by the general problem of SCP in Section 2.2, and finally in Section 2.3, we consider algorithms for RMSCP.
2.1 Monotone submodular cover
In this section, we develop and analyze approximation algorithms for MSCP. The greedy algorithm is a tight  $(\ln (1 / \epsilon), 1 - \epsilon)$ -bicriteria approximation algorithm for MSCP [Krause et al., 2008]. However, the greedy algorithm makes  $O(n^{2})$  queries of  $f$ , which is impractical in many application settings with large  $U$  and/or when queries of  $f$  are costly [Mirzasoleiman et al., 2015a]. Motivated by this, we propose and analyze the algorithms thresh-greedy-c and stoch-greedy-c for MSCP which give about the same bicriteria approximation guarantees but in many fewer queries of  $f$ .
We first describe thresh-greedy-c. thresh-greedy-c is closely related to the existing threshold greedy algorithm for monotone SMP (MSMP) [Badanidiyuru and Vondrak, 2014], and therefore we relegate the pseudocode of thresh-greedy-c to Section 4.1 in the supplementary material and only include a brief discussion here. At each iteration of thresh-greedy-c, instead of picking the element with the highest marginal gain into  $S$ , it sequentially adds any elements in  $U$  with marginal gain above a threshold,  $w$ , at the time of addition.  $w$  is initialized to  $\max_{u\in U}f(\{u\})$ , and is decreased by a factor of  $(1 - \epsilon /2)$  when the algorithm proceeds to the next iteration. thresh-greedy-c adds elements to a solution  $S$  until  $f(S)$  reaches  $(1 - \epsilon)\tau$ , which is shown to happen in at most  $\ln (2 / \epsilon)|OPT| + 1$  elements in the proof of Theorem 1. We now state the theoretical guarantees of thresh-greedy-c in Theorem 1.
Theorem 1. thresh-greedy-c produces a solution with  $(\ln (2 / \epsilon) + 1,1 - \epsilon)$ -bicriteria approximation guarantee to  $MSCP$ , in  $O\left(\frac{n}{\epsilon}\log \left(\frac{n}{\epsilon}\right)\right)$  number of queries of  $f$ .
Another method of speeding up the standard greedy algorithm is by introducing randomization, as has been done for MSMP in the stochastic greedy algorithm [Mirzasoleiman et al., 2015a]. A natural question is whether a randomized algorithm for MSMP can be converted into an algorithm for MSCP using the algorithm convert of Iyer and Bilmes [2013b]. However, convert relies on a deterministic approximation guarantee. We now introduce a new algorithm called convert-rand that is analogous to convert but runs the MSMP algorithm repeatedly in order to have the approximation guarantee hold with high probability. Pseudocode for convert-rand, as well as a proof of Theorem 2 can be found in Section 4.1 in the supplementary material.
Theorem 2. Any randomized  $(\gamma, \beta)$ -bicriteria approximation algorithm for MSMP that runs in time  $\mathcal{T}(n)$  where  $\gamma$  holds only in expectation can be converted into an approximation algorithm for MSCP that with probability at least  $1 - \delta$  is a  $((1 + \alpha)\beta, \gamma - \epsilon)$ -bicriteria approximation algorithm that runs in time  $O(\log_{1 + \alpha}(|OPT|)) \ln(1/\delta) \mathcal{T}(n) / \ln(\frac{1 - \gamma + \epsilon}{1 - \gamma})$ .
Therefore by applying Theorem 2 to the stochastic greedy algorithm of Mirzasoleiman et al., we have a  $(1 + \alpha, 1 - 1 / e - \epsilon)$ -bicriteria approximation algorithm for MSCP with high probability in  $O(n\log_{1 + \alpha}(|OPT|)\ln (1 / \delta)\ln (1 / \epsilon) / \ln (\frac{1 - \gamma + \epsilon}{1 - \gamma}))$  queries of  $f$ . However, a factor of  $1 - 1 / e - \epsilon$  of  $\tau$  is not very close to feasible, and further the convert-rand method wastes many queries of  $f$  essentially doing the same computations for different guesses of  $|OPT|$ . Therefore we focus the rest of this section on developing an algorithm, stoch-greedy-c, that uses the techniques of the stochastic greedy algorithm more directly for MSCP.
The idea behind the stochastic greedy algorithm for MSMP is that instead of computing the marginal gains of all elements at each iteration, we take a uniformly random sampled subset from  $U$  and pick the element with the highest marginal gain among the sampled subset. If the sampled subset is sufficiently large, in particular of size at least  $(n / \kappa)\ln (1 / \epsilon)$  where  $\kappa$  is the budget for the instance of MSMP and  $\epsilon > 0$  is an input, then with high probability a uniformly random element of  $OPT_{SM}$  will appear in the sampled subset and the marginal gain of adding the element is nearly the same as the standard greedy algorithm in expectation. However, in MSMP we know that  $|OPT_{SM}| = \kappa$ , but in MSP  $|OPT|$  is unknown. Therefore it is not obvious how to apply this technique in a more direct way than convert-rand.
We now introduce our algorithm stoch-greedy-c for MSCP, pseudocode for which is provided in Algorithm 1. stoch-greedy-c takes as input  $\epsilon >0$ ,  $\delta >0$ ,  $\alpha >0$ , and an instance of MSCP. stoch-greedy-c keeps track of  $O(\ln (1 / \delta))$  possibly overlapping solutions  $S_{1}, S_{2}, \ldots$  throughout a sequence of iterations. stoch-greedy-c also keeps track of an estimate of  $|OPT|$ ,  $g$ . During each iteration, for each solution  $S_{i}$ , stoch-greedy-c uniformly randomly and independently samples a set  $R$  of size  $\min \{n, (n / g) \ln (3 / \epsilon)\}$  and adds  $u = \operatorname{argmax} \{\Delta f_{\tau}(S_i, x) : x \in R\}$  to  $S_{i}$ . Every time  $\frac{\alpha}{1 + \alpha} \ln (3 / \epsilon)g$  elements have been added to each  $S_{i}$ ,  $g$  is increased by a factor of  $1 + \alpha$ . stoch-greedy-c stops once there exists an  $S_{i}$  such that  $f(S_{i}) \geq (1 - \epsilon)\tau$ , and returns this solution.
We now state the theoretical results for stoch-greedy-c in Theorem 3.
Theorem 3. Suppose that  $\text{stoch-greedy-c}$  is run for an instance of MSCP. Then with probability at least  $1 - \delta$ ,  $\text{stoch-greedy-c}$  outputs a solution  $S$  that satisfies a  $((1 + \alpha)[\ln (3 / \epsilon)], 1 - \epsilon)$ -bicriteria approximation guarantee in at most  $O\left(\frac{\alpha}{1 + \alpha} n \ln (1 / \delta) \ln^2 (3 / \epsilon) \log_{1 + \alpha}(|OPT|)\right)$  queries of  $f$ .
Algorithm 1 stoch-greedy-c
Input:  $\epsilon, \alpha, \delta$   
Output:  $S \subseteq U$   
1:  $S_i \gets \emptyset \forall i \in \{1, \dots, \ln(1/\delta) / \ln(2)\}$   
2:  $r \gets 1, g \gets 1 + \alpha$   
3: while  $f(S_i) < (1 - \epsilon)\tau \forall i$  do  
4: for  $i \in \{1, \dots, \ln(1/\delta) / \ln(2)\}$  do  
5:  $R \gets$  sample  $\min\{n, n \ln(3/\epsilon)/g\}$  elements from  $U$   
6:  $u \gets \operatorname{argmax}_{x \in R} \Delta f_\tau(S_i, x)$   
7:  $S_i \gets S_i \cup \{u\}$   
8:  $r \gets r + 1$   
9: if  $r > \ln(3/\epsilon)g$  then  $g \gets (1 + \alpha)g$   
10: return  $\operatorname{argmin}\{|S_i| : f(S_i) \geq (1 - \epsilon)\tau\}$
Compared to thresh-greedy-c, stoch-greedy-c has a better dependence on  $\epsilon$  in terms of the number of queries made to  $f$ . In addition, it is possible to extend the stochastic greedy algorithm of Mirzasoleiman et al. to a  $(1 - \epsilon, \ln(1 / \epsilon))$ -bicriteria approximation algorithm for MSMP and then use convert-rand (see Section 4.1 in the supplementary material). However, stoch-greedy-c still would have strictly fewer queries of  $f$  by a factor of  $\frac{\alpha}{1 + \alpha}$  compared to this approach because convert-rand does essentially the same computations for different guesses of  $|OPT|$ .
In order to prove Theorem 3, we first need Lemma 1 below, which states that as long as  $g \leq (1 + \alpha)|OPT|$ , the marginal gain of adding  $u$  in Line 6 is about the same as the standard greedy algorithm in expectation. Next, Lemma 2 below uses Lemma 1 to show that by the time  $g$  reaches  $(1 + \alpha)|OPT|$ ,  $\mathbb{E}[f_{\tau}(S_i)] \geq (1 - \frac{\epsilon}{2})\tau$  for all  $i$ . Finally, because there are  $O(\ln(1/\delta))$  solutions, by the time  $g$  reaches  $(1 + \alpha)|OPT|$ , there exists  $i$  such that  $f(S_i) \geq (1 - \epsilon)\tau$  with probability at least  $1 - \delta$  by using concentration bounds, which is stated in Lemma 3. Because of Lemma 3 we keep increasing  $g$  by a factor of  $(1 + \alpha)$  periodically, because intuitively the longer we keep adding elements, the bigger we know that  $|OPT|$  must be since the algorithm is still running and none of the solution sets has reached  $(1 - \epsilon)\tau$  yet. The proof of Lemmas 1 and 2, and of Theorem 3 can be found in Section 4.1 in the supplementary material.
Lemma 1. Consider any of the sets  $S_{i}$  at the beginning of an iteration on Line 4 where  $g \leq (1 + \alpha)|OPT|$ . Then if  $u_{i}$  is the random element that will be added on Line 6, we have that  $\mathbb{E}[\Delta f_{\tau}(S_i,u_i)] \geq \frac{1 - \epsilon / 3}{(1 + \alpha)|OPT|} (\tau - f_{\tau}(S_i))$ .
Lemma 2. Once  $r$  reaches  $(1 + \alpha)\lceil \ln (3 / \epsilon)|OPT|\rceil$ , we have that  $\mathbb{E}[f_{\tau}(S_i)]\geq \left(1 - \frac{\epsilon}{2}\right)\tau$  for all  $i$ .
Lemma 3. With probability at least  $1 - \delta$ , once  $r$  reaches  $(1 + \alpha)\lceil \ln (3 / \epsilon) | OPT| \rceil$ , we have that  $\max_i f(S_i) \geq (1 - \epsilon)\tau$ .
2.2 Non-monotone submodular cover
In this section, we introduce and theoretically analyze the algorithm stream-c for SCP in the general setting, where  $f$  is not assumed to be monotone. In the general setting, the standard greedy algorithm doesn't have non-trivial approximation guarantee for SCP. In addition, it has previously been shown that it is not possible for an algorithm to guarantee that  $f(X) \geq \tau / 2$  for SCP, where  $X$  is its returned solution, in polynomially many queries of  $f$  assuming the value oracle model [Crawford, 2023]. Our algorithm stream-c does produce a solution  $X$  that is guaranteed to satisfy  $f(X) \geq (1 - \epsilon)\tau$ , but relies on solving an instance of SMP exactly on a set of size  $O(|OPT| / \epsilon^2)$ . Despite not being polynomial time, stream-c is still useful for some instances of SCP because: (i)  $|OPT|$  may be relatively small; and (ii) the instance of SMP may be relatively easy to solve, e.g.  $f$  may be very close to monotone on the instance of SMP even if it was very non-monotone on the original instance of SCP. These aspects of stream-c are further explored in Section 3.
We now describe stream-c, pseudocode for which can be found in Algorithm 2. stream-c takes as input  $\epsilon >0$ ,  $\alpha >0$ , and an instance of SCP. stream-c takes sequential passes through the universe  $U$  (Line 4) with each pass corresponding to a new guess of  $|OPT|$ ,  $g$ .  $g$  is initialized as  $1 + \alpha$ , and at the end of each pass is increased by a factor of  $1 + \alpha$ . Throughout stream-c, a subset
Algorithm 2 stream-c
Input:  $\epsilon, \alpha$   
Output:  $S \subseteq U$   
1:  $S \gets \emptyset, S_1 \gets \emptyset, \dots, S_{2/\epsilon} \gets \emptyset$   
2:  $g \gets 1 + \alpha$   
3: while  $f(S) < (1 - \epsilon)\tau$  do  
4: for  $u \in U$  do  
5: if  $\exists j$  s.t.  $\Delta f(S_j, u) \geq \epsilon \tau/(2g)$  and  $|S_j| < 2g/\epsilon$  then  
6:  $S_j \gets S_j \cup \{u\}$   
7:  $S \gets \operatorname{argmax}\{f(X): X \subseteq \cup_{i=1}^{2/\epsilon} S_i, |X| \leq 2g/\epsilon\}$   
8:  $g = (1 + \alpha)g$   
9: return  $S$