Submodular + Concave

1 Introduction . Despite their simplicity , first-order optimization methods ( such as gradient descent and its variants , Frank-Wolfe , momentum based methods , and others ) have shown great success in many machine learning applications . A large body of research in the operations research and machine learning communities has fully demystified the convergence rate of such methods in minimizing well behaved convex objectives [ Bubeck , 2015 , Nesterov , 2018 ] . More recently , a new surge of rigorous results has also shown that gradient descent methods can find the global minima of specific non-convex objective functions arisen from non-negative matrix factorization [ Arora et al. , 2012 ] , robust PCA [ Netrapalli et al. , 2014 ] , phase retrieval [ Chen et al. , 2019b ] , matrix completion [ Sun and Luo , 2016 ] , and the training of wide neural networks [ Du et al. , 2019 , Jacot et al. , 2018 , Allen-Zhu et al. , 2019 ] , to name a few . It is also very well known that finding the global minimum of a general non-convex function is indeed computationally intractable [ Murty and Kabadi , 1987 ] . To avoid such impossibility results , simpler goals have been pursued by the community : either developing algorithms that can escape 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . saddle points and reach local minima [ Ge et al. , 2015 ] or describing structural properties under which reaching a local minimizer ensures optimality [ Sun et al. , 2016 , Bian et al. , 2017b , Hazan et al. , 2016 ] . In the same spirit , this paper quantifies a large class of non-convex functions for which first-order optimization methods provably achieve near optimal solutions . More specifically , we consider objective functions that are formed by the sum of a continuous DRsubmodular function G ( x ) and a concave function C ( x ) . Recent research in non-convex optimization has shown that first-order optimization methods provide constant factor approximation guarantees for maximizing continuous submodular functions Bian et al . [ 2017b ] , Hassani et al . [ 2017 ] , Bian et al . [ 2017a ] , Niazadeh et al . [ 2018 ] , Hassani et al . [ 2020a ] , Mokhtari et al . [ 2018a ] . Similarly , such methods find the global maximizer of concave functions . However , the class of F ( x ) = G ( x ) +C ( x ) functions is strictly larger than both concave and continuous DR-submodular functions . More specifically , F ( x ) is not concave nor continuous DR-submodular in general ( Figure 1 illustrates an example ) . In this paper , we show that first-order methods provably provide strong theoretical guarantees for maximizing such functions . The combinations of continuous submodular and concave functions naturally arise in many ML applications such as maximizing a regularized submodular function [ Kazemi et al. , 2020a ] or finding the mode of distributions [ Kazemi et al. , 2020a , Robinson et al. , 2019 ] . For instance , it is common to add a regularizer to a D-optimal design objective function to increase the stability of the final solution against perturbations [ He , 2010 , Derezinski et al. , 2020 , Lattimore and Szepesvari , 2020 ] . Similarly , many instances of log-submodular distributions , such as determinantal point processes ( DPPs ) , have been studied in depth in order to sample a diverse set of items from a ground set [ Kulesza , 2012 , Rebeschini and Karbasi , 2015 , Anari et al. , 2016 ] . In order to control the level of diversity , one natural recipe is to consider the combination of a log-concave ( e.g. , normal distribution , exponential distribution and Laplace distribution ) [ Dwivedi et al. , 2019 , Robinson et al. , 2019 ] and log-submodular distributions [ Djolonga and Krause , 2014 , Bresler et al. , 2019 ] , i.e. , Pr ( x ) ∝ exp ( λC ( x ) + ( 1− λ ) G ( x ) ) for some λ ∈ [ 0 , 1 ] . In this way , we can obtain a class of distributions that contains log-concave and log-submodular distributions as special cases . However , this class of distributions is strictly more expressive than both log-concave and log-submodular models , e.g. , in contrast to log-concave distributions , they are not uni-modal in general . Finding the mode of such distributions amounts to maximizing a combination of a continuous DR-submodular function and a concave function . The contributions of this paper are as follows . • Assuming first-order oracle access for the function F , we develop the algorithms : GREEDY FRANK-WOLFE ( Algorithm 1 ) and MEASURED GREEDY FRANK-WOLFE ( Algorithm 2 ) which achieve constant factors approximation guarantees between 1 − 1/e and 1/e depending on the setting , i.e . depending on the monotonicity and non-negativity of G and C , and depending on the constraint set having the down-closeness property or not . • Furthermore , if we have access to the individual gradients of G and C , then we are able to make the guarantee with respect to C exact using the algorithms : GRADIENT COMBINING FRANK-WOLFE ( Algorithm 3 ) and NON-OBLIVIOUS FRANK-WOLFE ( Algorithm 4 ) . These results are summarized and made more precise in Table 1 and Section 3 . • We then present experiments designed to use our algorithms to smoothly interpolate between contrasting objectives such as picking a diverse set of elements and picking a clustered set of elements . This smooth interpolation provides a way to control the amount of diversity in the final solution . We also demonstrate the use of our algorithms to maximize a large class of ( non-convex/non-concave ) quadratic programming problems . Related Work . The study of discrete submodular maximization has flourished in the last decade through far reaching applications in machine learning and and artificial intelligence including viral marketing [ Kempe et al. , 2003 ] , dictionary learning [ Krause and Cevher , 2010 ] , sparse regression [ Elenberg et al. , 2016 ] , neural network interoperability [ Elenberg et al. , 2017 ] , crowd teaching [ Singla et al. , 2014 ] , sequential decision making [ Alieva et al. , 2021 ] , active learning [ Wei et al. , 2015 ] , and data summarization [ Mirzasoleiman et al. , 2013 ] . We refer the interested reader to a recent survey by Tohidi et al . [ 2020 ] and the references therein . Recently , Bian et al . [ 2017b ] proposed an extension of discrete submodular functions to the continuous domains that can be of use in machine learning applications . Notably , this class of ( non-convex/nonconcave ) functions , so called continuous DR-submodular , contains the continuous multilinear ex- tension of discrete submodular functions Călinescu et al . [ 2011 ] as a special case . Continuous DR-submodular functions can reliably model revenue maximization [ Bian et al. , 2017b ] , robust budget allocation [ Staib and Jegelka , 2017 ] , experimental design [ Chen et al. , 2018 ] , MAP inference for DPPs [ Gillenwater et al. , 2012 , Hassani et al. , 2020b ] , energy allocation [ Wilder , 2018b ] , classes of zero-sum games [ Wilder , 2018c ] , online welfare maximization and online task assignment [ Sadeghi et al. , 2020 ] , as well as many other settings of interest . The research on maximizing continuous DR-submodular functions in the last few years has established strong theoretical results in different optimization settings including unconstrained [ Niazadeh et al. , 2018 , Bian et al. , 2019 ] , stochastic Mokhtari et al . [ 2018a ] , Hassani et al . [ 2017 ] , online [ Chen et al. , 2018 , Zhang et al. , 2019 , Sadeghi and Fazel , 2019 , Raut et al. , 2021 ] , and parallel models of computation [ Chen et al. , 2019a , Mokhtari et al. , 2018b , Xie et al. , 2019 , Ene and Nguyen , 2019 ] . A different line of works study the maximization of discrete functions that can be represented as the sum of a non-negative monotone submodular function and a linear function . The ability to do so is useful in practice since the linear function can be viewed as a soft constraint , and it also has theoretical applications as is argued by the first work in this line [ Sviridenko et al. , 2017 ] ( for example , the problem of maximization of a monotone submodular function with a bounded curvature can be reduced to the maximization of the sum of a monotone submodular function and a linear function ) . In terms of the approximation guarantee , the algorithms suggested by Sviridenko et al . [ 2017 ] were optimal . However , more recent works improve over the time complexities of these algorithms [ Feldman , 2021 , Harshaw et al. , 2019 , Ene et al. , 2020 ] , generalize them to weakly-submodular functions [ Harshaw et al. , 2019 ] , and adapt them to other computational models such as the data stream and MapReduce models [ Kazemi et al. , 2020b , Ene et al. , 2020 ] . 2 Setting and Notation . Let us now formally define the setting we consider . Fix a subset X of Rn of the form ∏n i=1 Xi , where Xi is a closed range in R. Intuitively , a function G : X → R is called ( continuous ) DR-submodular if it exhibits diminishing returns in the sense that given a vector x ∈ X , the increase in G ( x ) obtained by increasing xi ( for any i ∈ [ n ] ) by ε > 0 is negatively correlated with the original values of the coordinates of x . This intuition is captured by the following definition . In this definition ei denotes the standard basis vector in the ith direction . Definition 2.1 . A function G : X → R is DR-submodular if for every two vectors a , b ∈ X , positive value k and coordinate i ∈ [ n ] we have G ( a+ kei ) −G ( a ) ≥ G ( b+ kei ) −G ( b ) whenever a ≤ b and a+ kei , b+ kei ∈ X . 1 It is well known that when G is continuously differentiable , then it is DR-submodular if and only if∇G ( a ) ≥ ∇G ( b ) for every two vectors a , b ∈ X that obey a ≤ b . Furthermore , for the sake of simplicity we assume in this work that X = [ 0 , 1 ] n. We are interested in the problem of finding the point in some convex body P ⊆ [ 0 , 1 ] n that maximizes a given function F : [ 0 , 1 ] n → R that can be expressed as the sum of a DR-submodular function G : [ 0 , 1 ] n → R and a concave function C : [ 0 , 1 ] n → R. To get meaningful results for this problem , we need to make some assumptions . Here we describe three basic assumptions that we make throughout the paper . The quality of the results that we obtain improves if additional assumptions are made , as is described in Section 3 . 1Throughout the paper , inequalities between vectors should be understood as holding coordinate-wise . Our first basic assumption is that G is non-negative . This assumption is necessary since we obtain multiplicative approximation guarantees with respect to G , and such guarantees do not make sense when G is allowed to take negative values.2 Our second basic assumption is that P is solvable , i.e. , that one can efficiently optimize linear functions subject to it . Intuitively , this assumption makes sense because one should not expect to be able to optimize a complex function such as F subject to P if one can not optimize linear functions subject to it ( nevertheless , it is possible to adapt our algorithms to obtain some guarantee even when linear functions can only be approximately optimized subject to P ) . Our final basic assumption is that both functions G and C are L-smooth , which means that they are differentiable , and moreover , their graidents are L-Lipschitz , i.e. , ‖∇f ( a ) −∇f ( b ) ‖2 ≤ L‖a− b‖2 ∀ a , b ∈ [ 0 , 1 ] n . We conclude this section by introducing some additional notation that we need . We denote by o an arbitrary optimal solution for the problem described above , and define D = maxx∈P ‖x‖2 . Additionally , we denote by 0̄ and 1̄ the all-zeros and all-ones vector , respectively ; and given two vectors a , b ∈ Rn , we denote by a ⊙ b their coordinate-wise multiplication , and by 〈a , b〉 their standard Euclidean inner product .

Summary: This paper studies maximization of submodular + concave functions over solvable polytopes. This problem forms a new class of non-convex optimization problems, for which no theoretical results have been obtained. Depending on the conditions of the problem, the authors develop several algorithms and prove that they return approximately optimal solutions. Experiments demonstrate the practical utility of the proposed algorithms.   --- After rebuttal ---  I appreciate the authors' kind response. My technical concerns have been resolved. However, I still think the experimental comparison in running time is important to clearly demonstrate the effectiveness of Algorithm 4 since $\nabla \bar{G}$ usually requires more computation costs than $\nabla \bar{G}$, as mentioned in the response. Despite the lack of experiments, the theoretical contributions are great and I think the paper is above the acceptance threshold. Therefore, I keep my score of 6.