SlowGuess/NeurIPS_2023_OCR huggingface dataset:
of elements of  $U$  are stored into  $2 / \epsilon$  disjoint sets,  $S_{1},\ldots ,S_{2 / \epsilon}$ . An element  $u$  is stored in at most one set  $S_{j}$  if both of the following are true: (i)  $|S_j| < 2g / \epsilon$ ; (ii) adding  $u$  is sufficiently beneficial to increasing the  $f$  value of  $S_{j}$  i.e.  $\Delta f(S_j,u)\geq \epsilon \tau /(2g)$ . If no such  $S_{j}$  exists,  $u$  is discarded. At the end of each pass, stream-c finds  $S = \operatorname{argmax}\{f(X):X\subseteq \cup S_i,|X|\leq 2g / \epsilon \}$  on Line 7. If  $f(S)\geq (1 - \epsilon)\tau$ , then  $S$  is returned and stream-c terminates. We now present the theoretical guarantees of stream-c in Theorem 4.
Theorem 4. Suppose that stream-c is run for an instance of SCP. Then stream-c returns  $S$  such that  $f(S) \geq (1 - \epsilon)\tau$  and  $|S| \leq (1 + \alpha)(2 / \epsilon)|OPT|$  in at most
$$
\log_ {1 + \alpha} (| O P T |) \left(\frac {2 n}{\epsilon} + \mathcal {T} \left((1 + \alpha) \left(\frac {4}{\epsilon^ {2}} | O P T |\right)\right)\right)
$$
queries of  $f$ , where  $T(m)$  is the number of queries to  $f$  of the algorithm for SMP used on Line 7 of Algorithm 2 on an input set of size  $m$ .
The key idea for proving Theorem 4 is that by the time  $g$  is in the region  $[|OPT|, (1 + \alpha)|OPT|]$ , there exists a subset  $X \subseteq \cup S_i$  such that  $|X| \leq 2g / \epsilon$  and  $f(X) \geq (1 - \epsilon)\tau$ . In fact, it is shown in the proof of Lemma 4 in Section 4.2 of the supplementary material that the set  $X$  is  $S_t \cup (\cup_i S_i \cap OPT)$  for a certain one of the sets  $S_t$ . Then when we solve the instance of SMP on Line 7, we find a set that has these same properties as  $X$ , and stream-c returns this set and terminates. Because  $g \leq (1 + \alpha)|OPT|$ , the properties described in Theorem 4 hold. Further notice that  $|\cup_i S_i| \leq 2(1 + \alpha)|OPT| / \epsilon^2$  at all times before stream-c exits, which implies the bounded query complexity in Theorem 4. The key idea for proving Theorem 4 is stated below in Lemma 4 and proven in Section 4.2 in the supplementary material.
Lemma 4. By the time that  $g$  reaches the region  $[|OPT|, (1 + \alpha)|OPT|]$  and the loop on Line 4 of stream- $c$  has completed, there exists a set  $X \subseteq \cup S_i$  of size at most  $2(1 + \alpha)|OPT| / \epsilon$  such that  $f(X) \geq (1 - \epsilon)\tau$ .
2.3 Regularized monotone submodular cover
The final class of submodular functions we consider take the form  $f = g - c$  where  $g$  is monotone, submodular, and nonnegative, while  $c$  is a modular, nonnegative penalty cost function, called the Regularized Monotone Submodular Cover Problem (RMSCP). In this case,  $f$  may take on negative values and therefore this class of submodular functions does not fit into general SCP.  $f$  may also be nonmonotone. Existing theoretical guarantees for the dual problem of Regularized Monotone Submodular Maximization (RMSMP) are in a different form than typical approximation algorithms [Harshaw et al., 2019, Kazemi et al., 2021]. In particular, they are of the following form: Given budget  $\kappa$ , the RMSMP algorithm is guaranteed to return a set  $S$  such that  $|S| \leq \kappa$  and  $g(S) - c(S) \geq \gamma g(OPT_{SM}) - c(OPT_{SM})$  where  $\gamma$  is some value less than 1, e.g.  $1 - 1/e$  for the distorted greedy algorithm of Harshaw et al.. A guarantee of this form means convert cannot be used (the check on Line 2 of the pseudocode for convert in the appendix is the problem). Motivated by this, we first develop an algorithm, convert-reg, that takes algorithms for RMSMP and converts them into an algorithm for RMSCP. Next, we propose a generalization of the distorted greedy algorithm of Harshaw et al. for RMSMP, called distorted-bi, that can be used along with convert-reg to produce an algorithm for RMSCP.
Algorithm 3 convert-reg
Input:  $\alpha >0$    
Output:  $S\subseteq U$    
1:  $\kappa \gets 1 + \alpha ,S\gets \emptyset$    
2: while  $g(S) - \frac{\gamma}{\beta} c(S) <   \gamma \tau$  do   
3:  $S\gets \mathrm{reg}$  run with objective  $g - \frac{\gamma}{\beta} c$  and budget  $\kappa$    
4:  $\kappa \gets (1 + \alpha)\kappa$    
5: return  $S$
We now describe convert-reg, pseudocode for which can be found in Algorithm 3. convert-reg takes as input an algorithm reg for RMSMP with the guarantees described previously, and  $\alpha >0$ . convert-reg repeatedly makes guesses for  $|OPT|$ ,  $\kappa$ . For each guess  $\kappa$ , the algorithm reg is run on an instance of RMSMP with objective  $g - (\gamma /\beta)c$  and budget  $\kappa$ . Once  $g - (\gamma /\beta)c$  reaches  $\gamma \tau$ , convert-reg exits.
The theoretical guarantees of convert-reg are stated below in Theorem 5 and proven in Section 4.3 in the supplementary material. Theorem 5 makes a slightly stronger assumption on reg than its approximation guarantees relative to  $OPT_{SM}$ . In particular, it is assumed that it returns a solution satisfying  $|S| \leq \rho \kappa$  and  $g(S) - c(S) \geq \gamma g(X) - \beta c(X)$  for all  $X \subseteq U$  such that  $|X| \leq \kappa$ , not just for  $OPT_{SM}$ . However, this is true of many algorithms for RMSMP including the distorted greedy algorithm of Harshaw et al..
Theorem 5. Suppose that we have an algorithm  $\text{reg}$  for RMSMP, and given budget  $\kappa$ $\text{reg}$  is guaranteed to return a set  $S$  of cardinality at most  $\rho \kappa$  such that  $g(S) - c(S) \geq \gamma g(X) - \beta c(X)$  for all  $X$  such that  $|X| \leq \kappa$ , in time  $T(n)$ . Then the algorithm convert-reg using  $\text{reg}$  as a subroutine returns a set  $S$  in time  $O(\log_{1 + \alpha}(n)T(n))$  such that  $|S| \leq (1 + \alpha)\rho |OPT|$  and  $g(S) - \frac{\gamma}{\beta} c(S) \geq \gamma \tau$ .
If we use convert-reg on the distorted greedy algorithm of Harshaw et al., we end up with an algorithm for RMSCP that is guaranteed to return a set  $S$  such that  $|S| \leq (1 + \alpha)|OPT|$  and  $g(S) - (1 - 1/e)c(S) \geq (1 - 1/e)\tau$ . If we set  $c = 0$ , then the problem setting reduces to MSPC and the distorted greedy algorithm of Harshaw et al. [2019] is equivalent to the standard greedy algorithm. However, our approximation guarantee does not reduce to the  $(\ln(1/\epsilon), 1 - \epsilon)$ -bicriteria approximation guarantee that would be preferable. A more intuitive result would be one that converges to that of the standard greedy algorithm as  $c$  goes to 0. Motivated by this, we now propose an extension of the distorted greedy algorithm of Harshaw et al. [2019] for RMSMP, distorted-bi, that accomplishes this.
We now describe distorted-bi, pseudocode for which can be found in Section 4.3 in the supplementary material. distorted-bi takes as input an instance of RMSMP and  $\epsilon >0$ . distorted-bi is related to the standard greedy algorithm, but instead of making queries to  $g - c$ , distorted-bi queries a distorted version of  $g - c$  that de-emphasizes  $g$  compared to  $c$ , and evolves over time. In particular, when element  $i$  is being added to the solution set, we choose the element of maximum marginal gain, provided it is positive, to the objective
$$
\Phi_ {i} (X) = \left(1 - \frac {1}{\kappa}\right) ^ {\ln (1 / \epsilon) \kappa - i} g (X) - c (X).
$$
The theoretical guarantees of distorted-bi are now presented in Theorem 6, and the proof of Theorem 6 can be found in Section 4.3 in the supplementary material.
Theorem 6. Suppose that distorted-bi is run for an instance of RMSMP. Then distorted-bi produces a solution  $S$  in  $O(n\kappa \ln(1/\epsilon))$  queries of  $f$  such that  $|S| \leq \ln(1/\epsilon)\kappa$  and for all  $X \subseteq U$  such that  $|X| \leq \kappa$ ,  $g(S) - c(S) \geq (1 - \epsilon)g(X) - \ln(1/\epsilon)c(X)$ .
Therefore by running convert-reg with distorted-bi as a subroutine for RMSMP, we end up with an algorithm for RMSCP that is guaranteed to return a set  $S$  such that  $|S| \leq (1 + \alpha)\ln(1/\epsilon)|OPT|$  and  $g(S) - (1 - \epsilon)c(S)/\ln(1/\epsilon) \geq (1 - \epsilon)\tau$  in  $O((1 + \alpha)n|OPT|\log_{1+\alpha}(|OPT|)\log(1/\epsilon))$  queries of  $f$ .
(a) cover  $f$
(b) cover size
(c) cover queries
(d) cover queries
(e) euall  $f$
(f) euall size
(g) eull queries
Figure 1: The experimental results of running the monotone algorithms on instances of data summarization on the delicious URL dataset ("cover") and running stream-c on the instances of graph cut on the email-EuAll graph ("euall").
(h) eull queries
In this section, we experimentally evaluate the algorithms proposed in Sections 2.1 and 2.2. In particular, the emphasis of Section 3.1 is on evaluation of our algorithm stoch-greedy-c on instances of data summarization, an application of MSPC. Next, we evaluate stream-c on instances of graph cut, an application of SCP that is not monotone, in Section 3.2. Additional details about the applications, setup, and results can be found in Section 5 in the supplementary material.
3.1 Monotone submodular objective
We first compare the solutions returned by stoch-greedy-c ("SG"), greedy-c ("G"), thresh-greedy-c ("TG"), and convert-rand using the bicriteria extension of the stochastic greedy algorithm of Mirzasoleiman et al. (see Section 4.1 in the supplementary material) ("SG2") on instances of data summarization. The data summarization instance featured here in the main paper is the delicious dataset of URLs tagged with topics, and  $f$  takes a subset of URLs to the number of distinct topics represented by those URLs ( $n = 5000$  with 8356 tags) [Soleimani and Miller, 2016]. Additional datasets are explored in Section 5.2 in the supplementary material. We run the algorithms with input  $\epsilon$  in the range (0, 0.15) and threshold values between 0 and  $f(U)$  ( $f(U)$  is the total number of tags). When  $\epsilon$  is varied,  $\tau$  is fixed at  $0.6f(U)$ . When  $\tau$  is varied,  $\epsilon$  is fixed at 0.2. The parameter  $\alpha$  is set to be 0.1 and the initial guess of  $|OPT|$  for stoch-greedy-c and convert-rand is set to be  $\tau / \max_s f(s)$ .
The results in terms of the  $f$  values and size of the solutions are presented in Figure 1(a) and 1(b). From the plots, one can see that the  $f$  values and size of solutions returned by stoch-greedy-c, greedy-c, thresh-greedy-c are nearly the same, and are smaller than the ones returned by convert-rand. This is unsurprising, because the theoretical guarantees on  $f$  and size are about the same for the different algorithms, but convert-rand tends to perform closer to its worst case guarantee on size. The number of queries to  $f$  for different  $\epsilon$  and  $\tau$  are depicted in Figures 1(d) and 1(c). Recall that the theoretical worst case number of queries to  $f$  for stoch-greedy-c, greedy-c, thresh-greedy-c and convert are  $O((\alpha/(1+\alpha))n\ln^2(1/\epsilon)\log_{1+\alpha}(|OPT|)), O(n\ln(1/\epsilon)|OPT|), O(n\log(|OPT|/\epsilon)/\epsilon),$  and  $O(n\ln^2(1/\epsilon)\log_{1+\alpha}(|OPT|))$  respectively. As expected based on these theoretical guarantees, greedy-c does the worst and increases rapidly as  $\tau$  (and therefore  $|OPT|$ ) increases. thresh-greedy-c tends to do worse compared to stoch-greedy-c and convert as  $\epsilon$  gets smaller. stoch-greedy-c consistently performs the fastest out of all of the algorithms.
3.2 Non-Monotone Submodular Objective
We now analyze the performance of stream-c on several instances of graph cut over real social network data. The universe  $U$  is all nodes in the network, and  $f$  is the number of edges between a set and its complement. The network featured in the main paper is the email-EuAll dataset ( $n = 265214$ , 420045 edges) from the SNAP large network collection [Leskovec and Sosic, 2016] and additional datasets can be found in Section 5.1 in the supplementary material. We run stream-c with input  $\epsilon$  in the range  $(0, 0.5)$  and threshold values between 0 and  $f(X)$  where  $X$  is a solution returned by the unconstrained submodular maximization algorithm of Buchbinder et al. [2015] on the instance. When  $\epsilon$  is varied,  $\tau$  is fixed at  $0.9f(X)$ . When  $\tau$  is varied,  $\epsilon$  is fixed at 0.15.
We compare the performance of stream-c using several possible algorithms for the subroutine of SMP over  $\cup S_{i}$  (see line 7 in Algorithm 2), including a polynomial time approximation algorithm and an unconstrained submodular maximization algorithm. In particular, we use the random greedy approximation algorithm for SMP that is proposed in Buchbinder et al. [2014] ("RG"), and the double greedy approximation algorithm for unconstrained submodular maximization proposed in Buchbinder et al. [2015] ("DG"). Random greedy and double greedy are both approximation algorithms  $(1/e$  in expectation and  $1/2$  in expectation respectively), and therefore the stopping conditions are set to be  $\frac{(1 - \epsilon)\tau}{e}$  and  $\frac{(1 - \epsilon)\tau}{2}$  respectively. We also consider an exact algorithm ("EX"), which essentially is a greedy heuristic followed by an exact search of all (exponentially many) possible solutions if the greedy fails. On instances where the exact algorithm was unable to complete in a time period of 5 minutes, we did not include a data point. We further discuss the use of these algorithms in Section 5.1 in the supplementary material.
Before introducing the fourth subroutine, we discuss an interesting pattern that we saw in our instances of graph cut. We noticed that it was often the case that: (i)  $\cup S_i$  tended to be small compared to its upper bound and in fact typically  $|\cup S_i|$  was smaller than the SMP constraint, making the subroutine an instance of unconstrained submodular maximization; (ii) The majority of elements (if not all) were "monotone" in the sense that for many  $x\in \cup S_i$ ,  $\Delta f(\cup S_i / x,x)\geq 0$ . Let  $M\subseteq \cup S_{i}$  be the set of monotone elements. It follows that if (i) holds, then the instance of submodular maximization is equivalent to  $\arg \max_{X\in \cup S_i / M}f(X\cup M)$ . If  $M$  is large in  $\cup S_{i}$ , this new problem instance is relatively easy to solve exactly. This motivates our fourth algorithm, fast-exact ("F-EX"), used on instances where (i) holds, and is to separate  $\cup S_{i}$  into monotone and non-monotone and search for the best subset amongst the non-monotone elements in a similar manner as the plain exact algorithm. We explore to what extent properties (i) and (ii) hold on different instances, as well as give additional details about the fast exact algorithm, in Section 5.2 in the supplementary material.
The results in terms of the  $f$  values and size of the output solutions returned by the four algorithms are plotted in Figure 1(e) and Figure 1(f). From the plots, one can see that the  $f$  values satisfy that  $f(S_{\mathrm{exact}}) \approx f(S_{\mathrm{f - exact}}) > f(S_{\mathrm{DG}}) > f(S_{\mathrm{RG}})$ . This is due to the stopping conditions for each algorithm, which follow from each algorithms approximation guarantee on  $f$  of  $1 - \epsilon$ ,  $1 - \epsilon$ ,  $1/2$ , and  $1/e$  respectively. On the other hand, the size mirrors the  $f$  value, since it tends to be the case that reaching a higher  $f$  value requires more elements from  $U$ . The number of queries made by the algorithms can be seen in Figure 1(h) and 1(g). As expected, the exact algorithms make more queries compared to the approximation algorithms, and in some cases "EX" doesn't even finish. However, by taking advantage of the properties (i) and (ii) discussed above, "F-EX" is able to run even for smaller  $\epsilon$ . Therefore, depending on the application, an exact algorithm on the relatively small set  $\cup S_i$  may be a practical choice in order to achieve a solution that is very close to feasible.
Hossein Soleimani and David J Miller. Semi-supervised multi-label topic models for document classification and sentence labeling. In Proceedings of the 25th ACM international on conference on information and knowledge management, pages 105-114, 2016.
Jure Leskovec and Rok Sosic. Snap: A general-purpose network analysis and graph-mining library. ACM Transactions on Intelligent Systems and Technology (TIST), 8(1):1-20, 2016.
Niv Buchbinder, Moran Feldman, Joseph Seffi, and Roy Schwartz. A tight linear time (1/2)-approximation for unconstrained submodular maximization. SIAM Journal on Computing, 44(5): 1384-1402, 2015.
Pinar Duygulu, Kobus Barnard, Joao FG de Freitas, and David A Forsyth. Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary. In Computer Vision—ECCV 2002: 7th European Conference on Computer Vision Copenhagen, Denmark, May 28–31, 2002 Proceedings, Part IV 7, pages 97–112. Springer, 2002.