Differentiable Optimization of Generalized Nondecomposable Functions using Linear Programs

1 Introduction . Commonly used losses such as cross-entropy used in deep neural network ( DNN ) models can be expressed as a sum over the per-sample losses incurred by the current estimate of the model . This allows the direct use of mature optimization routines , and is sufficient for many use cases . But in applications ranging from ranking/retrieval to class imbalanced learning , the most suitable losses for the task do not admit a “ decompose over samples ” form . Examples include Area under the ROC curve ( AUC ) , multi-class variants of AUC , F -score , Precision at a fixed recall ( P @ R ) and others . Optimizing such measures in a scalable manner can pose challenges even in the shallow setting . For AUC maximization , we now know that convex surrogate losses can be used in a linear model Liu et al . [ 2018 ] , Natole et al . [ 2018 ] in the so-called ERM framework . These ideas have been incorporated within deep neural network models and solved using SGD type schemes in Liu et al . [ 2019 ] . Such results on stochastic and online data models have also been explored in Ataman et al . [ 2006 ] , Cortes and Mohri [ 2004 ] , Gao et al . [ 2013 ] . There are also available strategies for measures other 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . than the AUC : Nan et al . [ 2012 ] , Dembczynski et al . [ 2011 ] give exact algorithms for optimizing F -score and Eban et al . [ 2017 ] , Ravi et al . [ 2020 ] proposes scalable methods for non-decomposable objectives which utilizes Lagrange multipliers to construct the proxy objectives . The authors in Mohapatra et al . [ 2018 ] discuss using a function that upper bounds ( structured ) hinge-loss to optimize average precision . Recently , Fathony and Kolter [ 2020 ] presented an adversarial prediction formulation for such nondecomposable measures , and showed that it is indeed possible to incorporate such measures within differentiable pipelines . Our work utilizes the simple observation that a number of these non-decomposable objectives can be expressed in the form of an integer program that can be relaxed to a linear program ( LP ) . Our approach is based on the premise that tackling the LP form of the non-decomposable objective as a module within the DNN , one which permits forward and reverse mode differentiation and can utilize in-built support for specialized GPU hardware in modern libraries such as PyTorch , is desirable . First , as long as a suitable LP formulation for an objective is available , the module may be directly used . Second , based on which scheme is used to solve the LP , one may be able to provide guarantees for the non-decomposable objective based on simple calculations ( e.g. , number of constraints , primal-dual gap ) . The current tools do not entirely address all these requirements . A characteristic of the LPs that arise from the nondecomposable losses mentioned above is that the constraints ( including the mini-batch of samples themselves ) are modified at each iteration – as a function of the updates to the representations of the data in the upstream layers . In Section 3 , we provide LP formulations of widely used nondecomposable terms , which fall squarely within the capabilities of our solver . In Section 4 , we show that the modified Newton ’ s algorithm in Mangasarian [ 2004 ] can be used for deep neural network ( DNN ) training in an end-to-end manner without requiring external solvers , where support for GPUs currently remains limited . Specifically , by exploiting self-concordance of the objective , we show that the algorithm can converge globally without line search . We then analyze the gradient properties of our approach , and some modifications to improve stability during backpropagation . Our experiments in Section 5 show that this scheme based on Mangasarian ’ s parametric exterior penalty formulation of the primal LP is a computationally effective and scalable strategy to solve LPs with a large number of constraints . On the practical side , we provide two ways to deal with the scaling issue when the constraint matrix is large . On the one hand , we show that sufficient dimension reduction can be used in our solver to solve the problem in a lower dimension space . On the other hand , when the matrix is sparse , with our new sparse implementation , we show that we can train Resnet-18 with 10× mini-batch size ( memory savings ) on a 2080TI Nvidia GPU . Our code is available , see Fig 1 for an overview of our solver for differentiating many popular nondecomposable objectives . 2 Related Works . Differentiable optimization of nondecomposable objectives . One closely related result to ours is Liu et al . [ 2019 ] which explored optimizing a surrogate loss of AUC using SGD type methods that made the stochastic AUC optimization more practical for deep neural networks . Compared with Liu et al . [ 2019 ] , our work enables optimizing many more types of nondecomposable objectives for deep neural networks , and relies on the LP reformulation instead of a surrogate loss . In Fathony and Kolter [ 2020 ] , ( the expected values of ) such metrics are approximated by LPs defined by predictor and adversary marginal distributions ( represented as square matrices ) . The ADMM solver in Fathony and Kolter [ 2020 ] , as we will see , inherits some drawbacks of ADMMs including higher memory footprint and slower convergence . Another body of work is the one involving the use of blackbox solvers Pogančić et al . [ 2019 ] , Berthet et al . [ 2020 ] which can be adapted to solve the nondecomposable objectives ( e.g. , reformulating as LPs ) . For example , one idea is to use appropriate ( infimal or gaussian ) convolution to construct approximations of the gradients through the LP . Song et al . [ 2016 ] proposes a using a task loss to perturb the loss and is similar in spirit to blackbox solvers – they both solve a sequence of perturbed argmax/argmin problems first , and then use carefully designed procedures for the backward gradient that often require at least one more solve of the optimization problem . We will discuss these properties later in the paper , and in our experiments . Differentiable LP solver . In principle , of course , backpropagating through a convex optimization model ( and in particular , LPs ) is quite possible . For LPs , we can take derivatives of the optimal value ( or the optimal solution ) of the model with respect to the LP parameters , and this can be accomplished by calling a powerful external solver . Often , this would involve the overhead of running the solver on the CPU . Solvers within CVXPY , are effective but due to their general-purpose nature , rely on interior point methods . OptNet Amos and Kolter [ 2017 ] is quite efficient but designed for quadratic programs ( QP ) : the theoretical results and its efficiency depends on factorizing a matrix in the quadratic term in the objective ( which is zero/non-invertible for LPs ) . The primal-dual properties and implicit differentiation for QPs do not easily translate to LPs due to the polyhedral geometry in LPs . The ideas in Meng et al . [ 2020 ] are only applicable when the number of constraints is approximately equal to the number of variables – an invalid assumption for the models that we study here . 3 Nondecomposable Functions and corresponding LP models . We first present a standard LP form and then reparameterize several generalized nondecomposable objectives in this way , summarized in a table in the appendix . We start with the binary AUC , extend it to multi-class AUC , and then later , discuss a ratio objective , F -score . The appendix also includes a discussion of other objectives that are amenable to our method . 3.1 Notations and Generalized LP formulation . Notations . We use the following notations : ( i ) n : number of samples used in training . ( ii ) X ∈ Rn×dimx : the explanatory features fed to a classifier ( e.g. , parameterized by w ) ; ( iii ) f ( xi ) ( or f ( i ) ) : a score function for the classifier where xi ∈ X ; ( iv ) Y ∈ { 0 , 1 } : target label and Ŷ ∈ { 0 , 1 } : predicted label for binary classification , both in Rn ; ( v ) A⊗B : Kronecker product of matrices A and B . ( vi ) Ir : Identity matrix of size r and 1 is the indicator function . ( vii ) Bk , p ( and B| , k′ ) gives the k-th row ( and k′-th ) column of B. LP formulation . We consider a general linear program ( LP ) that contains nonnegative variables as well as inequality and equality constraints . The form of the LP is given as max ũ , ṽ gT ũ+ hT ṽ subject to Eũ+ F ṽ ≤ p , Bũ+Gṽ = q ũ , ṽ ≥ 0 We can write it more succinctly as Variable z = [ ũ ṽ ] ; Coefficient c = [ −g − h ] ; Constraints A = [ E B −B F G −G ] T ; Constants b = [ p q − q ] T The corresponding primal LP can be written as , minz cT z subject to Az ≤ b , z ≥ 0 . 3.2 Maximizing AUC . The Area under the ROC Curve ( AUC ) calculates the probability that a classifier f ( · ) will rank a randomly chosen positive sample higher than a randomly chosen negative sample . AUC varies between 0 and 1 , where 1 represents all positives being ranked above the negatives . AUC may be estimated using the Wilcoxon-Mann-Whitney ( WMW ) Statistic Hanley and McNeil [ 1982 ] , as Definition 3.1 ( AUC ) . Let n be the number of samples . Let X+ ( and X− resp . ) be the set of positive ( and negative resp . ) samples such that |X+|+ |X−| = n where | · | is the cardinality of the set . Then , AUC is given as ( ∑|X+| i=1 ∑|X−| i=1 1f ( xi ) > f ( xj ) ) / ( |X+| |X−| ) for xi : i ∈ { 1 , · · · , n } . Here , we follow Ataman et al . [ 2006 ] to calculate the AUC based on the WMW statistic as follows . min zij |X+|∑ i=1 |X−|∑ j=1 zij s.t . f ( xi ) − f ( xj ) ≥ ϵ− zij where xi ∈ X+ , xj ∈ X− ; zij ≥ 0 , ( 1 ) where ϵ is a given constant . Problem ( 1 ) computes AUC indirectly by minimizing the number of pairs ( one each from the positive and negative classes ) where the positive sample is not ranked higher than the negative sample : so , the number of zero entries in z equals the number of pairs where this condition is not true . For a given z , we have that , AUC = ( n− ∥z∥0 ) / ( |X+| |X−| ) = ( n− ∑ i ∑ j ϵ−1relu ( 0 , −zij + ϵ ) ) / ( |X+| |X−| ) . If zij is 0 , then ϵ−1relu ( 0 , −zij + ϵ ) equals 1 . Otherwise zij > 0 , it follows from the first constraint in ( 1 ) , that zij ≥ ϵ , so ϵ−1relu ( 0 , −zij + ϵ ) equals 0 . Observe that in ( 1 ) , the number of constraints is |X||X−| , which is O ( n2 ) .

Summary: This work proposes to relax some commonly used discrete metrics into a Linear Program (LP). By using the Newton LP method, the algorithm is able to backpropagate through the LP and thus to train a neural network end-to-end while directly optimizing for the metric. In particular, this submission describes how the AUC, multi-class AUC and F-score can be casted as LPs, and how the resulting problems can be solved with a Newton LP method that can be differentiated through. Experiments are presented on Cat&Dog, CIFAR-10, CIFAR-100 and STL.