Best of Both Worlds: Practical and Theoretically Optimal Submodular Maximization in Parallel

1 Introduction . The cardinality-constrained optimization of a monotone , submodular function f : 2N → R+ , defined on subsets of a ground set N of size n , is a general problem formulation that is ubiquitous in wideranging applications , e.g . video or image summarization [ 30 ] , network monitoring [ 26 ] , information gathering [ 23 ] , and MAP Inference for Determinantal Point Processes [ 20 ] , among many others . The function f : 2N → R+ is submodular iff for all S ⊆ T ⊆ N , x 6∈ T , ∆ ( x |T ) ≤ ∆ ( x |S ) 1 ; and the function f is monotone if f ( S ) ≤ f ( T ) for all S ⊆ T . In this paper , we study the following submodular maximization problem ( SM ) maximizef ( S ) , subject to |S| ≤ k , ( SM ) where f is a monotone , submodular function ; SM is an NP-hard problem . There has been extensive effort into the design of approximation algorithms for SM over the course of more than 45 years , e.g . [ 32 , 12 , 10 , 21 , 24 ] . For SM , the optimal ratio has been shown to be 1− 1/e ≈ 0.63 [ 32 ] . 1∆ ( x |S ) denotes the marginal gain of x to S : f ( S ∪ { x } ) − f ( S ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . As instance sizes have grown very large , there has been much effort into the design of efficient , parallelizable algorithms for SM . Since queries to the objective function can be very expensive , the overall efficiency of an algorithm for SM is typically measured by the query complexity , or number of calls made to the objective function f [ 2 , 9 ] . The degree of parallelizability can be measured by the adaptive complexity of an algorithm , which is the minimum number of rounds into which the queries to f may be organized , such that within each round , the queries are independent and hence may be arbitrariliy parallelized . Observe that the lower the adaptive complexity , the more parallelizable an algorithm is . To obtain a constant approximation factor , a lower bound of Ω ( n ) has been shown on the query complexity [ 24 ] and a lower bound of Ω ( log ( n ) / log log ( n ) ) has been shown on the adaptive complexity [ 3 ] . Several algorithms have been developed recently that are nearly optimal in terms of query and adaptive complexities [ 14 , 11 , 17 , 5 ] ; that is , these algorithms achieve O ( log n ) adaptivity and O ( npolylog ( n ) ) query complexity ( see Table 1 ) . However , these algorithms use sampling techniques that result in very large constant factors that make these algorithms impractical . This fact is discussed in detail in Breuer et al . [ 9 ] ; as an illustration , to obtain ratio 1− 1/e− 0.1 with 95 % confidence , all of these algorithms require more than 106 queries of sets of size k/ log ( n ) in every adaptive round [ 9 ] ; moreover , even if these algorithms are run as heuristics using a single sample , other inefficiencies preclude these algorithms of running even on moderately sized instances [ 9 ] . For this reason , the FAST algorithm of Breuer et al . [ 9 ] has been recently proposed , which uses an entirely different sampling technique called adaptive sequencing . Adaptive sequencing was originally introduced in Balkanski et al . [ 6 ] , but the original version has quadratic query complexity in the size of the ground set and hence is still impractical on large instances . To speed it up , the FAST algorithm sacrifices theoretical guarantees to yield an algorithm that parallelizes well and is faster than all previous algorithms for SM in an extensive experimental evaluation . The theoretical sacrifices of FAST include : the adaptivity of FAST is Ω ( log ( n ) log2 ( log n ) ) , which is higher than the state-of-the-art , and more significantly , the algorithm obtains no approximation ratio for k < 8502 ; since many applications require small choices for k , this limits the practical utility of FAST . A natural question is thus : is it possible to design an algorithm that is both practical and theoretically optimal in terms of adaptivity , ratio , and total queries ? 1.1 Contributions . In this paper , we provide three main contributions . The first contribution is the algorithm LINEARSEQ ( LS , Section 2 ) that achieves with probability 1− 1/n a constant factor ( 4 +O ( ε ) ) −1 in expected linear query complexity and with O ( log n ) adaptive rounds ( Theorem 1 ) . Although the ratio of ≈ 0.25 is smaller than the optimal 1− 1/e ≈ 0.63 , this algorithm can be used to improve the query 2The approximation ratio 1 − 1/e − 4ε of FAST holds with probability 1 − δ for k ≥ θ ( ε , δ , k ) = 2 log ( 2δ−1 log ( 1 ε log ( k ) ) ) /ε2 ( 1− 5ε ) . complexity of many extant algorithms , as we decribe in the related work section below . Interestingly , LINEARSEQ can be modified to have adaptivity O ( log ( n/k ) ) at a small cost to its ratio as discussed in Appendix F. This version of LINEARSEQ is a constant-factor algorithm for SM with smaller adaptivity than any previous algorithm in the literature , especially for values of k that are large relative to n. Our second contribution is an improved parallelizable thresholding procedure THRESHOLDSEQ ( TS , Section 3 ) for a commonly recurring task in submodular optimization : namely , add all elements that have a gain of a specified threshold τ to the solution . This subproblem arises not only in SM , but also e.g . in submodular cover [ 17 ] and non-monotone submodular maximization [ 4 , 18 , 15 , 25 ] . Our TS accomplishes this task with probability 1− 1/n in O ( log n ) adaptive rounds and expected O ( n ) query complexity ( Theorem 2 ) , while previous procedures for this task only add elements with an expected gain of τ and use expensive sampling techniques [ 17 ] ; have Ω ( log2 n ) adaptivity [ 22 ] ; or have Ω ( kn ) query complexity [ 6 ] . Finally , we present in Section 3 the parallelized greedy algorithm PARALLELGREEDYBOOST ( PGB ) , which is used in conjunction with LINEARSEQ and THRESHOLDSEQ to yield the final algorithm LS+PGB , which answers the above question affirmatively : LS+PGB obtains nearly the optimal 1− 1/e ratio with probability 1− 2/n in O ( log n ) adaptive rounds and O ( n ) queries in expectation ; moreover , LS+PGB is faster than FAST in an extensive empirical evaluation ( see Table 2 ) . In addition , LS+PGB improves theoretically on the previous algorithms in query complexity while obtaining nearly optimal adaptivity ( see Table 1 ) . 1.2 Additional Related Work . Adaptive Sequencing . The main inefficiency of the adaptive sequencing method of Balkanski et al . [ 6 ] ( which causes the quadratic query complexity ) is an explicit check that a constant fraction of elements will be filtered from the ground set . In this work , we adopt a similar sampling technique to adaptive sequencing , except that we design the algorithm to filter a constant fraction of elements with only constant probability . This method allows us to reduce the quadratic query complexity of adaptive sequencing to linear query complexity while only increasing the adaptive complexity by a small constant factor . In contrast , FAST of Breuer et al . [ 9 ] speeds up adaptive sequencing by increasing the adaptive complexity of the algorithm through adaptive binary search procedures , which , in addition to the increasing the adaptivity by logarithmic factors , place restrictions on the k values for which the ratio can hold . This improved adaptive sequencing technique is the core of our THRESHOLDSEQ procedure , which has the additional benefit of being relatively simple to analyze . Algorithms with Linear Query Complexity . Our LINEARSEQ algorithm also uses the improved adaptive sequencing technique , but in addition this algorithm integrates ideas from the Ω ( n ) -adaptive linear-time streaming algorithm of Kuhnle [ 24 ] to achieve a constant-factor algorithm with low adaptivity in expected linear time . Integration of the improved adaptive sequencing with the ideas of Kuhnle [ 24 ] is non-trivial , and ultimately this integration enables the theoretical improvement in query complexity over previous algorithms with sublinear adaptivity that obtain a constant ratio with high probability ( see Table 1 ) . In Fahrbach et al . [ 17 ] , a linear-time procedure SUBSAMPLEPREPROCESSING is described ; this procedure is to the best of our knowledge the only algorithm in the literature that obtains a constant ratio with sublinear adaptive rounds and linear query complexity and hence is comparable to LINEARSEQ . However , SUBSAMPLEPREPROCESSING uses entirely different ideas from our LINEARSEQ and has much weaker theoretical guarantees : for input 0 < δ < 1 , it obtains ratio δ 2 2×106 with probability 1 − δ in O ( log ( n ) /δ ) adaptive rounds and O ( n ) queries in expectation – the small ratio renders SUBSAMPLEPREPROCESSING impractical ; also , its ratio holds only with constant probability . By contrast , with ε = 0.1 , our LINEARSEQ obtains ratio ≈ 0.196 with probability 1− 1/n in O ( log ( n ) ) adaptive rounds and O ( n ) queries in expectation . Using LS for Preprocessing : Guesses of OPT . Many algorithms for SM , including FAST and all of the algorithms listed in Table 1 except for SM and our algorithm , use a strategy of guessing logarithmically many values of OPT . Our LINEARSEQ algorithm reduces the interval containing OPT from size k to a small constant size in expected linear time . Thus , LINEARSEQ could be used for preprocessing prior to running FAST or one of the other algorithms in Table 1 , which would improve their query complexity without compromising their adaptive complexity or ratio ; this illustrates the general utility of LINEARSEQ . For example , with this change , the theoretical adaptivity of FAST improves , although it remains worse than LS+PGB : the adaptive complexity of FAST becomes O ( 1 ε2 log ( n ) log ( 1 ε log ( k ) ) ) in contrast to the O ( 1 ε2 log ( n/ε ) ) of LS+PGB . Although SUBSAMPLEPREPROCESSING may be used for the same purpose , its ratio only holds with constant probability which would then limit the probability of success of any following algorithm . Relationship of THRESHOLDSEQ to Existing Methods . The first procedure in the literature to perform the same task is the THRESHOLDSAMPLING procedure of Fahrbach et al . [ 17 ] ; however , THRESHOLDSAMPLING only ensures that the expected marginal gain of each element added is at least τ and has large constants in its runtime that make it impractical [ 9 ] . In contrast , THRESHOLDSEQ ensures that added elements contribute a gain of at least τ with high probability and is highly efficient empirically . A second procedure in the literature to perform the same task is the ADAPTIVESEQUENCING method of Balkanski et al . [ 6 ] , which similarly to THRESHOLDSEQ uses random permutations of the ground set ; however , ADAPTIVE-SEQUENCING focuses on explicitly ensuring a constant fraction of elements will be filtered in the next round , which is expensive to check : the query complexity of ADAPTIVE-SEQUENCING is O ( kn ) . In contrast , our THRESHOLDSEQ algorithm ensures this property with a constant probability , which is sufficient to ensure the adaptivity with the high probability of 1− 1/n in O ( n ) expected queries . Finally , a third related procedure in the literature is THRESHOLDSAMPLING of Kazemi et al . [ 22 ] , which also uses random permutations to sample elements . However , this algorithm has the higher adaptivity of O ( log ( n ) log ( k ) ) , in contrast to the O ( log ( n ) ) of THRESHOLDSEQ . MapReduce Framework . Another line of work studying parallelizable algorithms for SM has focused on the MapReduce framework [ 13 ] in a distributed setting , e.g . [ 7 , 8 , 16 , 28 ] . These algorithms divide the dataset over a large number of machines and are intended for a setting in which the data does not fit on a single machine . None of these algorithms has sublinear adaptivity and hence all have potentially large numbers of sequential function queries on each machine . In this work , our empirical evaluation is on a single machine with a large number of CPU cores ; we do not evaluate our algorithms in a distributed setting . Organization . The constant-factor algorithm LINEARSEQ is described and analyzed in Section 2 ; the details of the analysis are presented in Appendix C. The variant of LINEARSEQ with lower adaptivity is described in Appendix F. The algorithms THRESHOLDSEQ and PARALLELGREEDYBOOST are discussed at a high level in Section 3 , with detailed descriptions of these algorithms and theoretical analysis presented in Appendices D and E. Our empirical evaluation is summarized in Section 4 with more results and discussion in Appendix H .

Summary: Submodular maximization is an important problem with many relevant applications in machine learning. With the huge amount of existing data and increase in the available computational power (increase in the number of machines), it is important to design scalable and parallelizable algorithms with provable guarantees from both practical and theoretical points of view. Many recent works have studied the submodular maximization in the context of adaptivity (“The degree of parallelizability can be measured by the adaptive complexity of an algorithm, which is the minimum number of rounds into which the queries to f may be organized”). In this paper, the authors propose an algorithm that achieves the state-of-the-art in terms of adaptive complexity, query complexity, and approximation ratio.