Differentiable Optimization of Generalized Nondecomposable Functions using Linear Programs

1 Introduction . Commonly used losses such as cross-entropy used in deep neural network ( DNN ) models can be expressed as a sum over the per-sample losses incurred by the current estimate of the model . This allows the direct use of mature optimization routines , and is sufficient for many use cases . But in applications ranging from ranking/retrieval to class imbalanced learning , the most suitable losses for the task do not admit a “ decompose over samples ” form . Examples include Area under the ROC curve ( AUC ) , multi-class variants of AUC , F -score , Precision at a fixed recall ( P @ R ) and others . Optimizing such measures in a scalable manner can pose challenges even in the shallow setting . For AUC maximization , we now know that convex surrogate losses can be used in a linear model Liu et al . [ 2018 ] , Natole et al . [ 2018 ] in the so-called ERM framework . These ideas have been incorporated within deep neural network models and solved using SGD type schemes in Liu et al . [ 2019 ] . Such results on stochastic and online data models have also been explored in Ataman et al . [ 2006 ] , Cortes and Mohri [ 2004 ] , Gao et al . [ 2013 ] . There are also available strategies for measures other 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . than the AUC : Nan et al . [ 2012 ] , Dembczynski et al . [ 2011 ] give exact algorithms for optimizing F -score and Eban et al . [ 2017 ] , Ravi et al . [ 2020 ] proposes scalable methods for non-decomposable objectives which utilizes Lagrange multipliers to construct the proxy objectives . The authors in Mohapatra et al . [ 2018 ] discuss using a function that upper bounds ( structured ) hinge-loss to optimize average precision . Recently , Fathony and Kolter [ 2020 ] presented an adversarial prediction formulation for such nondecomposable measures , and showed that it is indeed possible to incorporate such measures within differentiable pipelines . Our work utilizes the simple observation that a number of these non-decomposable objectives can be expressed in the form of an integer program that can be relaxed to a linear program ( LP ) . Our approach is based on the premise that tackling the LP form of the non-decomposable objective as a module within the DNN , one which permits forward and reverse mode differentiation and can utilize in-built support for specialized GPU hardware in modern libraries such as PyTorch , is desirable . First , as long as a suitable LP formulation for an objective is available , the module may be directly used . Second , based on which scheme is used to solve the LP , one may be able to provide guarantees for the non-decomposable objective based on simple calculations ( e.g. , number of constraints , primal-dual gap ) . The current tools do not entirely address all these requirements . A characteristic of the LPs that arise from the nondecomposable losses mentioned above is that the constraints ( including the mini-batch of samples themselves ) are modified at each iteration – as a function of the updates to the representations of the data in the upstream layers . In Section 3 , we provide LP formulations of widely used nondecomposable terms , which fall squarely within the capabilities of our solver . In Section 4 , we show that the modified Newton ’ s algorithm in Mangasarian [ 2004 ] can be used for deep neural network ( DNN ) training in an end-to-end manner without requiring external solvers , where support for GPUs currently remains limited . Specifically , by exploiting self-concordance of the objective , we show that the algorithm can converge globally without line search . We then analyze the gradient properties of our approach , and some modifications to improve stability during backpropagation . Our experiments in Section 5 show that this scheme based on Mangasarian ’ s parametric exterior penalty formulation of the primal LP is a computationally effective and scalable strategy to solve LPs with a large number of constraints . On the practical side , we provide two ways to deal with the scaling issue when the constraint matrix is large . On the one hand , we show that sufficient dimension reduction can be used in our solver to solve the problem in a lower dimension space . On the other hand , when the matrix is sparse , with our new sparse implementation , we show that we can train Resnet-18 with 10× mini-batch size ( memory savings ) on a 2080TI Nvidia GPU . Our code is available , see Fig 1 for an overview of our solver for differentiating many popular nondecomposable objectives . 2 Related Works . Differentiable optimization of nondecomposable objectives . One closely related result to ours is Liu et al . [ 2019 ] which explored optimizing a surrogate loss of AUC using SGD type methods that made the stochastic AUC optimization more practical for deep neural networks . Compared with Liu et al . [ 2019 ] , our work enables optimizing many more types of nondecomposable objectives for deep neural networks , and relies on the LP reformulation instead of a surrogate loss . In Fathony and Kolter [ 2020 ] , ( the expected values of ) such metrics are approximated by LPs defined by predictor and adversary marginal distributions ( represented as square matrices ) . The ADMM solver in Fathony and Kolter [ 2020 ] , as we will see , inherits some drawbacks of ADMMs including higher memory footprint and slower convergence . Another body of work is the one involving the use of blackbox solvers Pogančić et al . [ 2019 ] , Berthet et al . [ 2020 ] which can be adapted to solve the nondecomposable objectives ( e.g. , reformulating as LPs ) . For example , one idea is to use appropriate ( infimal or gaussian ) convolution to construct approximations of the gradients through the LP . Song et al . [ 2016 ] proposes a using a task loss to perturb the loss and is similar in spirit to blackbox solvers – they both solve a sequence of perturbed argmax/argmin problems first , and then use carefully designed procedures for the backward gradient that often require at least one more solve of the optimization problem . We will discuss these properties later in the paper , and in our experiments . Differentiable LP solver . In principle , of course , backpropagating through a convex optimization model ( and in particular , LPs ) is quite possible . For LPs , we can take derivatives of the optimal value ( or the optimal solution ) of the model with respect to the LP parameters , and this can be accomplished by calling a powerful external solver . Often , this would involve the overhead of running the solver on the CPU . Solvers within CVXPY , are effective but due to their general-purpose nature , rely on interior point methods . OptNet Amos and Kolter [ 2017 ] is quite efficient but designed for quadratic programs ( QP ) : the theoretical results and its efficiency depends on factorizing a matrix in the quadratic term in the objective ( which is zero/non-invertible for LPs ) . The primal-dual properties and implicit differentiation for QPs do not easily translate to LPs due to the polyhedral geometry in LPs . The ideas in Meng et al . [ 2020 ] are only applicable when the number of constraints is approximately equal to the number of variables – an invalid assumption for the models that we study here . 3 Nondecomposable Functions and corresponding LP models . We first present a standard LP form and then reparameterize several generalized nondecomposable objectives in this way , summarized in a table in the appendix . We start with the binary AUC , extend it to multi-class AUC , and then later , discuss a ratio objective , F -score . The appendix also includes a discussion of other objectives that are amenable to our method . 3.1 Notations and Generalized LP formulation . Notations . We use the following notations : ( i ) n : number of samples used in training . ( ii ) X ∈ Rn×dimx : the explanatory features fed to a classifier ( e.g. , parameterized by w ) ; ( iii ) f ( xi ) ( or f ( i ) ) : a score function for the classifier where xi ∈ X ; ( iv ) Y ∈ { 0 , 1 } : target label and Ŷ ∈ { 0 , 1 } : predicted label for binary classification , both in Rn ; ( v ) A⊗B : Kronecker product of matrices A and B . ( vi ) Ir : Identity matrix of size r and 1 is the indicator function . ( vii ) Bk , p ( and B| , k′ ) gives the k-th row ( and k′-th ) column of B. LP formulation . We consider a general linear program ( LP ) that contains nonnegative variables as well as inequality and equality constraints . The form of the LP is given as max ũ , ṽ gT ũ+ hT ṽ subject to Eũ+ F ṽ ≤ p , Bũ+Gṽ = q ũ , ṽ ≥ 0 We can write it more succinctly as Variable z = [ ũ ṽ ] ; Coefficient c = [ −g − h ] ; Constraints A = [ E B −B F G −G ] T ; Constants b = [ p q − q ] T The corresponding primal LP can be written as , minz cT z subject to Az ≤ b , z ≥ 0 . 3.2 Maximizing AUC . The Area under the ROC Curve ( AUC ) calculates the probability that a classifier f ( · ) will rank a randomly chosen positive sample higher than a randomly chosen negative sample . AUC varies between 0 and 1 , where 1 represents all positives being ranked above the negatives . AUC may be estimated using the Wilcoxon-Mann-Whitney ( WMW ) Statistic Hanley and McNeil [ 1982 ] , as Definition 3.1 ( AUC ) . Let n be the number of samples . Let X+ ( and X− resp . ) be the set of positive ( and negative resp . ) samples such that |X+|+ |X−| = n where | · | is the cardinality of the set . Then , AUC is given as ( ∑|X+| i=1 ∑|X−| i=1 1f ( xi ) > f ( xj ) ) / ( |X+| |X−| ) for xi : i ∈ { 1 , · · · , n } . Here , we follow Ataman et al . [ 2006 ] to calculate the AUC based on the WMW statistic as follows . min zij |X+|∑ i=1 |X−|∑ j=1 zij s.t . f ( xi ) − f ( xj ) ≥ ϵ− zij where xi ∈ X+ , xj ∈ X− ; zij ≥ 0 , ( 1 ) where ϵ is a given constant . Problem ( 1 ) computes AUC indirectly by minimizing the number of pairs ( one each from the positive and negative classes ) where the positive sample is not ranked higher than the negative sample : so , the number of zero entries in z equals the number of pairs where this condition is not true . For a given z , we have that , AUC = ( n− ∥z∥0 ) / ( |X+| |X−| ) = ( n− ∑ i ∑ j ϵ−1relu ( 0 , −zij + ϵ ) ) / ( |X+| |X−| ) . If zij is 0 , then ϵ−1relu ( 0 , −zij + ϵ ) equals 1 . Otherwise zij > 0 , it follows from the first constraint in ( 1 ) , that zij ≥ ϵ , so ϵ−1relu ( 0 , −zij + ϵ ) equals 0 . Observe that in ( 1 ) , the number of constraints is |X||X−| , which is O ( n2 ) .

Summary: This paper present a novel approach to perform direct optimization of non-decomposable objective functions (such as AUC or F-measure) in the context of Deep Learning. They propose a general LP-based framework describing these objective functions and provide a general algorithm to solve this problem (and get the corresponding gradients). Finally, this new approach is compared to other methods trying to optimize these objectives directly and surrogate loss functions that are commonly used in practice.