Submodular + Concave

1 Introduction . Despite their simplicity , first-order optimization methods ( such as gradient descent and its variants , Frank-Wolfe , momentum based methods , and others ) have shown great success in many machine learning applications . A large body of research in the operations research and machine learning communities has fully demystified the convergence rate of such methods in minimizing well behaved convex objectives [ Bubeck , 2015 , Nesterov , 2018 ] . More recently , a new surge of rigorous results has also shown that gradient descent methods can find the global minima of specific non-convex objective functions arisen from non-negative matrix factorization [ Arora et al. , 2012 ] , robust PCA [ Netrapalli et al. , 2014 ] , phase retrieval [ Chen et al. , 2019b ] , matrix completion [ Sun and Luo , 2016 ] , and the training of wide neural networks [ Du et al. , 2019 , Jacot et al. , 2018 , Allen-Zhu et al. , 2019 ] , to name a few . It is also very well known that finding the global minimum of a general non-convex function is indeed computationally intractable [ Murty and Kabadi , 1987 ] . To avoid such impossibility results , simpler goals have been pursued by the community : either developing algorithms that can escape 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . saddle points and reach local minima [ Ge et al. , 2015 ] or describing structural properties under which reaching a local minimizer ensures optimality [ Sun et al. , 2016 , Bian et al. , 2017b , Hazan et al. , 2016 ] . In the same spirit , this paper quantifies a large class of non-convex functions for which first-order optimization methods provably achieve near optimal solutions . More specifically , we consider objective functions that are formed by the sum of a continuous DRsubmodular function G ( x ) and a concave function C ( x ) . Recent research in non-convex optimization has shown that first-order optimization methods provide constant factor approximation guarantees for maximizing continuous submodular functions Bian et al . [ 2017b ] , Hassani et al . [ 2017 ] , Bian et al . [ 2017a ] , Niazadeh et al . [ 2018 ] , Hassani et al . [ 2020a ] , Mokhtari et al . [ 2018a ] . Similarly , such methods find the global maximizer of concave functions . However , the class of F ( x ) = G ( x ) +C ( x ) functions is strictly larger than both concave and continuous DR-submodular functions . More specifically , F ( x ) is not concave nor continuous DR-submodular in general ( Figure 1 illustrates an example ) . In this paper , we show that first-order methods provably provide strong theoretical guarantees for maximizing such functions . The combinations of continuous submodular and concave functions naturally arise in many ML applications such as maximizing a regularized submodular function [ Kazemi et al. , 2020a ] or finding the mode of distributions [ Kazemi et al. , 2020a , Robinson et al. , 2019 ] . For instance , it is common to add a regularizer to a D-optimal design objective function to increase the stability of the final solution against perturbations [ He , 2010 , Derezinski et al. , 2020 , Lattimore and Szepesvari , 2020 ] . Similarly , many instances of log-submodular distributions , such as determinantal point processes ( DPPs ) , have been studied in depth in order to sample a diverse set of items from a ground set [ Kulesza , 2012 , Rebeschini and Karbasi , 2015 , Anari et al. , 2016 ] . In order to control the level of diversity , one natural recipe is to consider the combination of a log-concave ( e.g. , normal distribution , exponential distribution and Laplace distribution ) [ Dwivedi et al. , 2019 , Robinson et al. , 2019 ] and log-submodular distributions [ Djolonga and Krause , 2014 , Bresler et al. , 2019 ] , i.e. , Pr ( x ) ∝ exp ( λC ( x ) + ( 1− λ ) G ( x ) ) for some λ ∈ [ 0 , 1 ] . In this way , we can obtain a class of distributions that contains log-concave and log-submodular distributions as special cases . However , this class of distributions is strictly more expressive than both log-concave and log-submodular models , e.g. , in contrast to log-concave distributions , they are not uni-modal in general . Finding the mode of such distributions amounts to maximizing a combination of a continuous DR-submodular function and a concave function . The contributions of this paper are as follows . • Assuming first-order oracle access for the function F , we develop the algorithms : GREEDY FRANK-WOLFE ( Algorithm 1 ) and MEASURED GREEDY FRANK-WOLFE ( Algorithm 2 ) which achieve constant factors approximation guarantees between 1 − 1/e and 1/e depending on the setting , i.e . depending on the monotonicity and non-negativity of G and C , and depending on the constraint set having the down-closeness property or not . • Furthermore , if we have access to the individual gradients of G and C , then we are able to make the guarantee with respect to C exact using the algorithms : GRADIENT COMBINING FRANK-WOLFE ( Algorithm 3 ) and NON-OBLIVIOUS FRANK-WOLFE ( Algorithm 4 ) . These results are summarized and made more precise in Table 1 and Section 3 . • We then present experiments designed to use our algorithms to smoothly interpolate between contrasting objectives such as picking a diverse set of elements and picking a clustered set of elements . This smooth interpolation provides a way to control the amount of diversity in the final solution . We also demonstrate the use of our algorithms to maximize a large class of ( non-convex/non-concave ) quadratic programming problems . Related Work . The study of discrete submodular maximization has flourished in the last decade through far reaching applications in machine learning and and artificial intelligence including viral marketing [ Kempe et al. , 2003 ] , dictionary learning [ Krause and Cevher , 2010 ] , sparse regression [ Elenberg et al. , 2016 ] , neural network interoperability [ Elenberg et al. , 2017 ] , crowd teaching [ Singla et al. , 2014 ] , sequential decision making [ Alieva et al. , 2021 ] , active learning [ Wei et al. , 2015 ] , and data summarization [ Mirzasoleiman et al. , 2013 ] . We refer the interested reader to a recent survey by Tohidi et al . [ 2020 ] and the references therein . Recently , Bian et al . [ 2017b ] proposed an extension of discrete submodular functions to the continuous domains that can be of use in machine learning applications . Notably , this class of ( non-convex/nonconcave ) functions , so called continuous DR-submodular , contains the continuous multilinear ex- tension of discrete submodular functions Călinescu et al . [ 2011 ] as a special case . Continuous DR-submodular functions can reliably model revenue maximization [ Bian et al. , 2017b ] , robust budget allocation [ Staib and Jegelka , 2017 ] , experimental design [ Chen et al. , 2018 ] , MAP inference for DPPs [ Gillenwater et al. , 2012 , Hassani et al. , 2020b ] , energy allocation [ Wilder , 2018b ] , classes of zero-sum games [ Wilder , 2018c ] , online welfare maximization and online task assignment [ Sadeghi et al. , 2020 ] , as well as many other settings of interest . The research on maximizing continuous DR-submodular functions in the last few years has established strong theoretical results in different optimization settings including unconstrained [ Niazadeh et al. , 2018 , Bian et al. , 2019 ] , stochastic Mokhtari et al . [ 2018a ] , Hassani et al . [ 2017 ] , online [ Chen et al. , 2018 , Zhang et al. , 2019 , Sadeghi and Fazel , 2019 , Raut et al. , 2021 ] , and parallel models of computation [ Chen et al. , 2019a , Mokhtari et al. , 2018b , Xie et al. , 2019 , Ene and Nguyen , 2019 ] . A different line of works study the maximization of discrete functions that can be represented as the sum of a non-negative monotone submodular function and a linear function . The ability to do so is useful in practice since the linear function can be viewed as a soft constraint , and it also has theoretical applications as is argued by the first work in this line [ Sviridenko et al. , 2017 ] ( for example , the problem of maximization of a monotone submodular function with a bounded curvature can be reduced to the maximization of the sum of a monotone submodular function and a linear function ) . In terms of the approximation guarantee , the algorithms suggested by Sviridenko et al . [ 2017 ] were optimal . However , more recent works improve over the time complexities of these algorithms [ Feldman , 2021 , Harshaw et al. , 2019 , Ene et al. , 2020 ] , generalize them to weakly-submodular functions [ Harshaw et al. , 2019 ] , and adapt them to other computational models such as the data stream and MapReduce models [ Kazemi et al. , 2020b , Ene et al. , 2020 ] . 2 Setting and Notation . Let us now formally define the setting we consider . Fix a subset X of Rn of the form ∏n i=1 Xi , where Xi is a closed range in R. Intuitively , a function G : X → R is called ( continuous ) DR-submodular if it exhibits diminishing returns in the sense that given a vector x ∈ X , the increase in G ( x ) obtained by increasing xi ( for any i ∈ [ n ] ) by ε > 0 is negatively correlated with the original values of the coordinates of x . This intuition is captured by the following definition . In this definition ei denotes the standard basis vector in the ith direction . Definition 2.1 . A function G : X → R is DR-submodular if for every two vectors a , b ∈ X , positive value k and coordinate i ∈ [ n ] we have G ( a+ kei ) −G ( a ) ≥ G ( b+ kei ) −G ( b ) whenever a ≤ b and a+ kei , b+ kei ∈ X . 1 It is well known that when G is continuously differentiable , then it is DR-submodular if and only if∇G ( a ) ≥ ∇G ( b ) for every two vectors a , b ∈ X that obey a ≤ b . Furthermore , for the sake of simplicity we assume in this work that X = [ 0 , 1 ] n. We are interested in the problem of finding the point in some convex body P ⊆ [ 0 , 1 ] n that maximizes a given function F : [ 0 , 1 ] n → R that can be expressed as the sum of a DR-submodular function G : [ 0 , 1 ] n → R and a concave function C : [ 0 , 1 ] n → R. To get meaningful results for this problem , we need to make some assumptions . Here we describe three basic assumptions that we make throughout the paper . The quality of the results that we obtain improves if additional assumptions are made , as is described in Section 3 . 1Throughout the paper , inequalities between vectors should be understood as holding coordinate-wise . Our first basic assumption is that G is non-negative . This assumption is necessary since we obtain multiplicative approximation guarantees with respect to G , and such guarantees do not make sense when G is allowed to take negative values.2 Our second basic assumption is that P is solvable , i.e. , that one can efficiently optimize linear functions subject to it . Intuitively , this assumption makes sense because one should not expect to be able to optimize a complex function such as F subject to P if one can not optimize linear functions subject to it ( nevertheless , it is possible to adapt our algorithms to obtain some guarantee even when linear functions can only be approximately optimized subject to P ) . Our final basic assumption is that both functions G and C are L-smooth , which means that they are differentiable , and moreover , their graidents are L-Lipschitz , i.e. , ‖∇f ( a ) −∇f ( b ) ‖2 ≤ L‖a− b‖2 ∀ a , b ∈ [ 0 , 1 ] n . We conclude this section by introducing some additional notation that we need . We denote by o an arbitrary optimal solution for the problem described above , and define D = maxx∈P ‖x‖2 . Additionally , we denote by 0̄ and 1̄ the all-zeros and all-ones vector , respectively ; and given two vectors a , b ∈ Rn , we denote by a ⊙ b their coordinate-wise multiplication , and by 〈a , b〉 their standard Euclidean inner product .

Summary: This paper studies the maximization of a nonconvex function in the form of $F = G + C$ on a convex set $P \\subseteq [0,1]^n$, where $G$ is DR-submodular and $C$ is concave. For various settings (monotonicity/nonnegativity of $G$ or $C$, $P$ is a down-closed/general convex set, the gradient oracle is for only $F$/both $G$ and $C$, etc.), the authors developed the first approximation algorithms.  The algorithms exploit some ideas from submodular function maximization such as Frank-Wolfe, measured continuous greedy, etc. The algorithms have provable fine approximation guarantees, i.e., they achieve $\\alpha$- and $\\beta$-approximation for $G$ and $C$, respectively. These are stronger guarantees than the standard approximation results in submodular function maximization. Furthermore, the authors proved even tighter approximation ratios (i.e., $\\beta \\approx 1$) if the individual gradients of $G$ and $C$ are available.  The authors conducted the numerical experiments in (nonconvex) quadratic programming and D-optimal design. The proposed algorithms achieved better objective values compared to the baseline methods.