Constrained Robust Submodular Partitioning

1 Introduction . The problem of partitioning a given set V of items into m blocks , where any two blocks share no items in common , arises in many real-world scenarios and machine learning applications . As an optimization problem , partitioning aims to generate the blocks so that the utilities of the blocks , as measured by a given set function , are good . Submodular functions are a rich family of set functions that naturally captures diversity of a given set of items . They have been applied in many realworld problems [ 26 , 30 , 21 , 17 , 13 , 18 , 28 ] . By maximizing a submodular utility function for each partitioned block , we encourage each block to be representative of the ground set V . Many algorithms have been proposed for various submodular partitioning problems with approximation guarantees . For the submodular welfare problem [ 24 ] , we aim to find a partition such that the sum of the submodular evaluations of every block is maximized . Such an objective promotes the overall utility of the entire partition but some blocks may still have small function values . The robust submodular partitioning problem [ 12 , 29 ] ( often called “ submodular fair allocation with indivisible goods ” ) aims to find the partition such that the minimum-valued block in the partition is maximized according to the submodular function . The robust objective optimizes the worst block in the partition so that all blocks are minimally “ good. ” In the general setting , every block in the partition may have a different submodular function ( the heterogeneous case ) although for this work , we study only the restricted setting where all blocks share the same submodular function ( the homogeneous case ) . The robust submodular partitioning problem has many applications . Given V as the training dataset for a machine learning task , Wei et al . [ 29 ] finds a partition of V for distributed training : every block of partitioned data is sent to a single machine for parallel gradient computations , and the gradients 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . are aggregated over all the blocks in the partition for model updates . Since we enforce each block to be representative of V , the gradients computed across distributed machines are consistent , resulting in reduced variance and improved convergence for the aggregation step . Using a similar idea , Wang et al . [ 25 ] partitions the training data into mini-batches so that every mini-batch is as representative as possible , therefore reducing the variance during mini-batch gradient-based training . In this work , we explore two different algorithmic approaches , Min-Block Greedy and Round-Robin Greedy , for our partitioning problem but under various constraints , newly applied to this problem . For Min-Block Greedy based algorithms , we first show that the 1 m bound for the unconstrained case is tight . We then modify the algorithm to allow a general down-closed constraint C , and prove an approximation bound of ↵ ↵m+1 , where ↵ is the bound for solving the submodular maximization problem under constraint C using a greedy based algorithm . For example , for a cardinality constraint , ↵ = 1 1/e [ 9 ] , and the bound for constrained robust submodular partitioning is ( m+ 11 1/e ) 1 . Similarly , for C as an intersection-of-p-matroids constraint , ↵ = ( 1 + p ) 1 [ 10 ] , and the bound is ( m+ p+ 1 ) 1 ; for C as a knapsack constraint , ↵ = 0.5 ( 1 1/e ) [ 16 ] , and the bound is ( m+ 21 1/e ) 1 . For Round-Robin Greedy based algorithms , when C is a cardinality constraint , we get a bound of ( 1 1/e ) 2 3 , and when C is a matroid constraint , we get a bound of 1 1/e5 . The Min-Block Greedy approach gives a weaker bound , and since the 1 m bound for the unconstrained case is tight , we can not improve upon the 1 m factor for the constrained case . The Round-Robin Greedy approach gives a constant bound , but its running time is worse . The running time for Min-Block Greedy is O ( n2 ) , where n is the ground set size . For RoundRobin Greedy under a matroid constraint , the running time is O ( n2 ( log logm+log 1 ) ) , as it needs to use binary search to find the optimal solution value to the given problem over an exponentially decreasing sequence , with 11+ ( > 0 ) as the multiplicative factor . In all cases , we assume an oracle model , and the running time is in terms of the number of submodular evaluations . An important contribution our work shows is that by utilizing the Min-Block Greedy algorithm result as input , our Round-Robin Greedy algorithm attains strongly polynomial running time — all previous results on the unconstrained case using a Round-Robin-like algorithm have only weakly polynomial running time [ 3 ] . The various constraints ( e.g. , cardinality , matroids , and knapsack ) we study greatly improves the applicability of robust submodular partitioning . Several applications that benefit from the constraints include : ( 1 ) Partition a training data for machine learning models in distributed training or forming deterministic mini-batches [ 29 , 25 ] . The additional constraint can be the number of samples from each class to be no more than a certain value . If there are enough samples in the training data , every resulting block will have the same number of samples for each class , which avoids imbalance , further promoting each block ’ s diversity , and improving the gradients ’ consistency . ( 2 ) Given an undirected graph , we partition the edges into subgraphs so that each subgraph is representative based on the submodular evaluation , and we also constrain each subgraph to have no cycles ( a cycle matroid ) . A practical scenario is that we wish to send information efficiently over a graph of devices . We partition the graph so that information can be sent in parallel , and the constraint to have no cycles enforces that information is not redundantly sent twice to the same device , leading to improved communications efficiency . ( 3 ) Again , for an undirected and connected graph , we partition the edges into subgraph blocks such that if we were to remove any block of the partition from the original graph , the remaining graph remains connected ( which can be done via a bond matroid , where min-cuts are cycles and anything not a cut is independent ) . In practice , this functions as a form of reliability insurance . For a graph of devices , we partition the graph to perform computation in parallel , so that if the connections in one block fail , the other blocks can still operate and communicate since the graph remains connected . 2 Related Work . Golovin [ 12 ] introduces robust submodular partitioning ( i.e. , submodular fair allocation of indivisible goods ) , and proposes a matching-based algorithm with a bound of 1 n m+1 . Khot & Ponnuswami [ 14 ] proposes a binary search based algorithm and gives an improved bound of 12m 1 . Asadpour & Saberi [ 2 ] uses an ellipsoid approximation approach and gives a bound of ⌦ ( 1p nm1/4 logn log3/2 m ) . Wei et al . [ 29 ] gives a simple Min-Block Greedy algorithm and proves a 1 m bound . A Round-Robin Greedy approach is given in [ 3 ] with a bound of 1 e 1 3 . Ghodsi et al . [ 11 ] proposes a local search algorithm with a bound of 13 . Both [ 3 ] and [ 11 ] requires guessing of the optimal solution value from an exponentially decreasing sequence of values , so strictly speaking , they lose an extra ( 1+ ) -factor in the approxima- tion bound where ( 1 + ) is the exponential factor for the guessing sequence . We can set the value small to get close to the constant bounds shown above at the costs of computation . Wang et al . [ 25 ] extends the Min-Block Greedy algorithm with a cardinality constraint , and also shows a hierarchical partitioning framework to reduce the memory costs . We adapt the Min-Block Greedy approach [ 29 ] and the Round-Robin Greedy approach [ 3 ] to the constrained case . To the best of our knowledge , this work is the first ( as far as we know ) to study the robust submodular partitioning problem under all of the various constraints ( cardinality , intersection of matroid , knapsack ) . Wang et al . [ 25 ] is a special case of our work as it only studies the cardinality constraint . Cotter et al . [ 8 ] studies ( as well as allowing multiple blocks to be jointly scored ) a matroid constrained “ groupings ” ( i.e. , coverings , packings , or partitions ) problem but only a fractional subset of groups ( rather than the minimum of the groups ) , is guaranteed to have values larger than the bounded max-min OPT , while our bound compares the min block evaluation to the optimal max-min value . Another line of related research is the submodular load balancing problem , which minimizes the maximum-valued block in the partition according to the submodular evaluations . In contrast to promoting diversity of each block for the robust submodular partition problem , submodular load balancing encourages every block to contain redundant items , similar to standard clustering objectives . Theoretically , this problem has been shown to be much harder as Svitkina & Fleischer [ 22 ] shows a information theoretical lower bound of o ( q n logn ) , and also gives a sampling algorithm to match the lower bound up to constant factors . Similar to the max-min case , Ghodsi et al . [ 11 ] uses the ellipsoidal approximation to get a bound of O ( p n log n ) . Wei et al . [ 29 ] gives a Lovász extension based relaxation algorithm and achieves a bound of m . 3 Preliminaries and Formulation . With a ground set V of n items , a submodular function f is a set function 2V ! R that satisfies the property : f ( A ) + f ( B ) f ( A [ B ) + f ( A \ B ) , where A , B ✓ V . Equivalently , a submodular function is characterized by diminishing returns : f ( v|A ) f ( v|B ) 8v /2 B and A ⇢ B ✓ V , where f ( v|B ) = f ( { v } [ B ) f ( B ) . Submodular functions naturally describe the diversity or representativeness of a given set of items . Many simple greedy-based algorithms have been developed to solve optimization problems involving submodular functions , giving both theoretical approximation guarantees , as well as good empirical performance . We restrict the submodular functions discussed in this paper to be monotone non-decreasing and normalized , i.e. , f ( B ) f ( A ) 8A ✓ B ✓ V , f ( ; ) = 0 . A matroid M = ( V , I ) is a set system that describes the independence relationships among the subsets of the ground set V . I is a set of subsets of V and every S 2 I is considered an independent subset . The matroid rank function is defined as rM ( A ) = max { |S| : S ✓ A , S 2 I } . rM ( V ) indicates the maximum size of a subset that may be independent according to the matroid M. All subsets of cardinality k with some integer k > 0 form a uniform matroid , which we denote by Mu k . A partition matroid is one where we partition the ground set into blocks , and a set is independent if it intersects each block by no more than a block-specific limit . We define a particularly useful partition matroid on an expanded ground set V̄ as follows : We first duplicate the ground set m times , creating V1 = V2 = . . . Vm = V , which are ground set copies . We create an expanded ground set V̄ = ] j=1 : mVj as the disjoint union . A subset S ✓ V̄ is independent in Mpm if for every element v 2 V , let its m copies in V̄ be { v1 , v2 , . . . , vm } , we have |S \ { v1 , v2 , . . . , vm } | 1 , i.e. , S contains at most one copy of element v. Apart from the uniform matroid and this particular partition matroid , there are many other matroids reflecting a natural notion of independence , for example , the linearly-independent set of real vectors and the spanning trees in a graph . In the below , we use both S 2 M and , when clear , S 2 I , to indicate that S is independent in the matroid M = ( V , I ) . Matroids are often used as constraints in submodular optimization problems : maxS2I f ( S ) with a matroid M = ( V , I ) . When M is a uniform matroid Mu k , this reduces to the cardinality submodular max and the greedy algorithm gives a 1 e 1 bound [ 9 ] . For a general constraint with the intersection of p matroids , the simple greedy algorithm gives a 1 p+1 bound [ 10 ] . Suppose we represent a set S as a binary indicator vector xS 2 { 0 , 1 } n , i.e. , 8i 2 [ n ] , xS [ i ] = 1 if vi 2 S or otherwise xS [ i ] = 0 . Then for all the independent sets of a matroid M = ( V , I ) , the convex hull over all the xS , S 2 I forms a polytope , which is called the matroid polytope PM of matroid M [ 9 ] . Based on the convex property of the matroid polytope , algorithms [ 4–6 , 24 ] have been proposed to firstly solve a continuous extension of the submodular optimization problem under the matroid polytope constraint , which generates a fractional solution in [ 0 , 1 ] n , and then round the fractional solution to an integral solution to get the resulting set . The continuous greedy algorithm [ 4 ] gives a 1 e 1 guarantee under a single matroid constraint using pipage rounding [ 1 , 4 ] . Interestingly , running the continuous greedy under a partition matroid constraint ( submodular welfare problem ) gives a uniform fractional solution , i.e. , on the expanded ground set V̄ , the fractional solution x = ( 1 m , 1 m , . . . , 1 m ) ( i.e. , assigning 1 m of every element to each block ) leads to a 1 e 1 bound in expectation for an integral solution that assigns each element in V uniformly to one of the m blocks . This observation also constitutes the basic idea of Round-Robin Greedy for solving the robust submodular partition problem [ 3 ] , which we will discuss in more detail later . For a submodular function f on a ground set V , the robust submodular partition problem ( submodular fair allocation ) [ 12 ] is defined as : max ⇡2⇧ ( V , m ) min A2⇡ f ( A ) , ( 1 ) where m is the number of blocks in a partition , we denote all possible partitions with m blocks of ground set V as ⇧ ( V , m ) , and one partition ⇡ with |⇡| = m is a collection of m disjoint sets . Equivalently , we can represent the partition using a partition matroid constraint on the expanded ground set V̄ : max S✓V̄ , S2Mpm min j2 [ m ] f ( S \ Vj ) . ( 2 ) Intuitively , the above optimization for robust submodular partitioning encourages the minimumvalued block to have a high submodular evaluation . Compared to the submodular welfare problem , the robust submodular partition promotes fairness for every one of the partition blocks . There have been three recent approximation algorithms developed to solve Eq . ( 1 ) . Particularly , Wei et al . [ 29 ] uses a Min-Block Greedy algorithm , which greedily adds the element with the largest gain to the block with the minimum evaluation . Barman & Krishna Murthy [ 3 ] propose a Round-Robin Greedy algorithm , which iteratively traverses all the blocks in a fixed order , and greedily adds an element with the largest gain to each block . Ghodsi et al . [ 11 ] applies a local search approach , which starts with an arbitrary partition and keeps moving an element from a non-minimum block to the minimum block if this relocation improves the objective by certain threshold until no such element can be found . For [ 3 ] and [ 11 ] , they both require guessing the optimal solution ’ s value , and they need to run multiple instances of their algorithms with the guessed optimal values as an exponentially decreasing sequence from the maximal possible value f ( V ) to the optimal solution value µ = max⇡2⇧ ( V , m ) minS2⇡ f ( S ) . With the exponential decreasing factor as 1 + , the running time ( in terms of submodular function calls ) is O ( n2 1 log f ( V ) µ ) for [ 3 ] , and O ( n2m2 1 log f ( V ) µ ) for [ 11 ] . Min-Block Greedy , a much simpler algorithm , has a running time of O ( n2 ) . Note that the settings of [ 3 ] and [ 11 ] are slightly more general than Eq . ( 1 ) as the submodular function for each block can be different . But it ’ s not the heterogeneous case either as they focus on a different notion of optimality .

Summary: Provides constant approximation algorithms for certain "robust" or min-max submodular maximization problems. They consider cardinality and matroid constraints. Also provide good experimental results on partitioning a training dataset.