Differentiable Combinatorial Losses through Generalized Gradients of Linear Programs

1 INTRODUCTION . Combinatorial optimization problems , such as shortest path in a weighted directed graph , minimum spanning tree in a weighted undirected graph , or optimal assignment of tasks to workers , play a central role in many computer science applications . We have highly refined , efficient algorithms for solving these fundamental problems ( Cormen et al. , 2009 ; Schrijver , 2003 ) . However , while we can easily find , for example , the minimal spanning tree in a graph , the total weight of the tree as function of graph edge weights is not differentiable . This problem hinders using solutions to combinatorial problems as criteria in training models that rely on differentiability of the objective function with respect to the model parameters . Losses that are defined by objective value of some feasible solution to a combinatorial problem , not the optimal one , have been recently proposed for image segmentation using deep models ( Zheng et al. , 2015 ; Lin et al. , 2016 ) . These focus on a problem where some pixels in the image have segmentation labels , and the goal is to train a convolutional network that predicts segmentation labels for all pixels . For pixels with labels , a classification loss can be used . For the remaining pixels , a criterion based on a combinatorial problem – for example the maximum flow / minimal cut problem in a regular , lattice graph connecting all pixels ( Boykov et al. , 2001 ) or derived , higher-level super-pixels ( Lin et al. , 2016 ) – is often used as a loss , in an iterative process of improving discrete segmentation labels ( Zheng et al. , 2015 ; Marin et al. , 2019 ) . In this approach , the instance of the combinatorial problem is either fixed , or depends only on the input to the network ; for example , similarity of neighboring pixel colors defines edge weights . The output of the neural network gives rise to a feasible , but rarely optimal , solution to that fixed instance a combinatorial problem , and its quality is used as a loss . For example , pixel labeling proposed by the network is interpreted as a cut in a pre-defined graph connecting then pixels . Training the network should result in improved cuts , but no attempt to use a solver to find an optimal cut is made . Here , we are considering a different setup , in which each new output of the neural network gives rise to a new instance of a combinatorial problem . A combinatorial algorithm is then used to find the optimal solution to the problem defined by the output , and the value of the objective function of the optimal solution is used as a loss . After each gradient update , the network will produce a new combinatorial problem instance , even for the same input sample . Iteratively , the network is expected to learn to produce combinatorial problem instances that have low optimal objective function value . For example , in sequence-to-sequence modeling , the network will output a new sentence that is supposed to closely match the desired sentence , leading to a new optimal sequence alignment problem to be solved . Initially , the optimal alignment will be poor , but as the network improves and the quality of the output sentences get higher , the optimal alignment scores will be lower . Recently , progress in integrating combinatorial problems into differentiable models have been made by modifying combinatorial algorithms to use only differentiable elements ( Tschiatschek et al. , 2018 ; Mensch & Blondel , 2018 ; Chang et al. , 2019 ) , for example smoothed max instead of max in dynamic programming . Another approach involves executing two runs of a non-differentiable , black-box combinatorial algorithm and uses the two solutions to define a differentiable interpolation ( Vlastelica Pogančić et al. , 2020 ; Rolı́nek et al. , 2020 ) . Finally , differentiable linear programming and quadratic programming layers , which can be used to model many combinatorial problems , have been proposed recently ( Amos & Kolter , 2017 ; Agrawal et al. , 2019 ; Wilder et al. , 2019 ; Ferber et al. , 2019 ) . The approaches above allow for differentiating through optimal solution vectors . In many cases , we are interested only in the optimal objective value , not the solution vector , and the approaches above introduce unnecessary overhead . We propose an approach for gradient-descent based training of a network f ( x ; β ) for supervised learning problems involving samples ( x , y ) with the objective criterion involving a loss term of the form L ( β ) = h ( OptSolutionObjectiveValue ( Π ( F ( x ; β ) , y ) ) , where h : R → R is some differentiable function , and Π is a combinatorial solver for a problem instance defined by the output of the β-parameterized network F for feature vector x and by the true label y . We show that a broad class of combinatorial problems can be integrated into models trained using variants of gradient descent . Specifically , we show that for an efficiently solvable combinatorial problem that can be efficiently expressed as an integer linear program , generalized gradients of the problem ’ s objective value with respect to real-valued parameters defining the problem exist and can be efficiently computed from a single run of a black-box combinatorial algorithm . Using the above result , we show how generalized gradients of combinatorial problems can provide sentence-level loss for text summarization using differentiable encoder-decoder models that involve softmax or Gumbel softmax ( Jang et al. , 2016 ) , and a multi-element loss for training classification models when only weakly supervised , bagged training data is available . 2 DIFFERENTIABLE COMBINATORIAL LOSSES . 2.1 BACKGROUND ON GENERALIZED GRADIENTS . A function f : X → R defined over a convex , bounded open set X ∈ Rp is Lipschitz continuous on an open set B ∈ X if there is a finite K ∈ R such that ∀x , y ∈ B |f ( x ) − f ( y ) | ≤ K||x− y|| . A function is locally Lipschitz-continuous if for every point x0 in its domain , there is a neighborhood B0 , an open ball centered at x0 , on which the function is Lipschitz-continuous . For such functions , a generalized gradient can be defined . Definition 1 . ( Clarke , 1975 ) Let f : X → R be Lipschitz-continuous in the neighborhood of x ∈ X . Then , the Clarke subdifferential ∂f ( x ) of f at x is defined as ∂f ( x ) = conv { lim xk→x ∇f ( xk ) } , where the limit is over all convergent sequences involving those xk for which gradient exists , and conv denotes convex hull , that is , the smallest polyhedron that contains all vectors from a given set . Each element of the set ∂f ( x ) is called a generalized gradient of f at x . The Rademacher theorem ( see e.g . ( Evans , 1992 ) ) states that for any locally Lipschitz-continuous function the gradient exists almost everywhere ; convergent sequences can be found . In optimization algorithms , generalized gradients can be used in the same way as subgradients ( Redding & Downs , 1992 ) , that is , nondifferentiability may affect convergence in certain cases . 2.2 GRADIENT DESCENT OVER COMBINATORIAL OPTIMIZATION . Many combinatorial problems have linear objective function and can be intuitively expressed as integer linear programs ( ILP ) , that is , linear programs with additional constraint that the solution vector involves only integers . Any ILP can be reduced to a linear program . Consider an ILP z∗ = ILP ( c , A′ , b′ ) : = minu c Tu s.t . A′u = b′ , u ≥ 0 , u ∈ Zp , with an optimal solution vector u∗ and optimal objective value z∗ . Then , there exists a corresponding linear program LP ( c , A , b ) z∗ = LP ( c , A , b ) : = minu c Tu s.t . Au = b , u ≥ 0 , called ideal formulation ( Wolsey , 1989 ) , for which u∗ is also an optimal solution vector , with the same objective value z∗ . For a feasible , bounded p-dimensional integer program , we can view the pair ( A′ , b′ ) as a convex polyhedron A′ , the set of all feasible solutions . Then , the pair ( A , b ) in the ideal formulation LP is defined as the set of constraints specifying the feasible set A = conv { A′ ∩ Zp } . Convex hull of a subset of a convex set A′ can not extend beyond A′ , thus , A is convex , contains all integer solutions from A′ , and no other integer solutions . The number of linear constraints in the ideal formulation may be exponential in p , and/or in m , the number of the original constraints in A′ . Thus , the existence of the ideal formulation LP for an ILP may not have practical utility for solving the ILP . For a combinatorial problem and its corresponding ILP , we use the ideal formulation of the ILP as a conceptual tool to define generalized gradient of the objective value of the optimal solution to the combinatorial problem with respect to the parameters defining the combinatorial problem . Specifically , our approach first uses a single run of an efficient , black-box combinatorial algorithm to produce the optimal solution vector and the associated objective value . Then , the combinatorial problem is conceptually viewed as an instance of an ILP . A possibly exponentially large linear program ( LP ) equivalent to the ILP is then used , without actually being spelled out or solved , to derive generalized gradients based on the solution vector returned by the combinatorial algorithm . First , we introduce several notions of efficiency of transforming a combinatorial problem into a linear integer program that will be convenient in defining the generalized gradients of combinatorial problems . Definition 2 . Let P ( w ) be a combinatorial problem that is parameterized by a continuous vector w ∈ W ⊆ Rn , whereW is simply connected and n is the problem size , and let k ∈ Z be a constant that may depend on the problem type but not on its size . Then , a combinatorial problem is • primal-dual ∂-efficient if it can be phrased as an integer linear program involving n variables , with kn constraints in an LP formulation equivalent to the ILP , and the parameters ( A , b , c ) of the LP formulation depend on w through ( sub ) differentiable functions , c = c ( w ) , A = A ( w ) , b = b ( w ) . • primal ∂-efficient if it can be phrased as an integer linear program involving n variables , the parameters w of the problem influence the cost vector c through a ( sub ) differentiable function c = c ( w ) , and do not influence the constraints A , b . • dual ∂-efficient if it can be phrased as an integer linear program in which the number of constraints in the equivalent LP formulation is kn , the parametersw of the problem influence b through a ( sub ) differentiable function b = b ( w ) , and do no influence the constraint matrix A nor the cost vector c. The class of ∂-efficient problems includes polynomially solvable combinatorial problems with objective function that is linear in terms of problem parameters . Typically , the functions c = c ( w ) , b = b ( w ) and A = A ( w ) are either identity mapping or are constant ; for example , in the LP for maximum network flow , the cost vector c is composed directly of edge capacities , and A an b are constant for a given flow network topology , and do not depend on capacities . For any polynomially solvable combinatorial problem , we can construct a poly ( n ) -sized Boolean circuit for the algorithm solving it . For each poly ( n ) -sized circuit , there is a linear program with poly ( n ) variables and constraints that gives the same solution ( see ( Dasgupta et al. , 2008 ) , Chap . 7 ) . For example , for MST in a graph with V vertices and E edges , the Martin ’ s ILP formulation ( Martin , 1991 ) has only poly ( V + E ) constraints , but it is an extended formulation that involves V E additional variables on top of the typical E variables used in the standard ILP formulations for MST . Thus , we can not use it to construct an ILP formulation that would make MST primal-dual ∂-efficient . Alternatively , there is an ILP for MST with one binary variable per edge , and the weight of the edge only influences the cost vector c , but to prohibit cycles in the solution there is a constraint for each cycle in the graph , thus the number of constraints is not poly ( n ) for arbitrary graphs . These constraints are specified fully by the topology of the graph , not by the edge weights , so w does not influence A nor b , meeting the conditions for primal ∂-efficiency . The MST example shows that there are problems that are primal ∂-efficient and not primal-dual ∂-efficient . Some polynomially solvable combinatorial problems are not ∂-efficient in any of the above sense . For example , fixed-rank combinatorial problems with interaction costs ( Lendl et al. , 2019 ) can be phrased succinctly as a bilinear program , but lead to prohibitively large linear programs both in terms of the number of variables and the number of constraints . For ∂-efficient problems , we can efficiently obtain generalized gradients of the objective value . Theorem 1 . Consider a combinatorial problem P ( w ) of size n , a parameter vector w from the interior of the parameter domainW , and an algorithm Π ( w ) for solving it in time poly ( n ) . Let z∗ be the optimal objective value returned by Π . Then , • if P is primal ∂-efficient , then the generalized gradients ∂z∗ ( w ) exist , and can be efficiently computed from U∗ , the set of primal solution of the ideal formulation of integer program corresponding to P ; • if P is dual ∂-efficient , then the generalized gradients of ∂z∗ ( w ) exist , and can be efficiently computed from V ∗ , the set of all dual solution to the ideal formulation of the integer program corresponding to P ; • if P is primal-dual ∂-efficient , then the generalized gradients of A over w exist , and can be efficiently computed from U∗ and V ∗ , as defined above . Proof . A series of results ( Gal , 1975 ; Freund , 1985 ; De Wolf & Smeers , 2000 ) shows that if the optimal objective value z∗ = LP ( c , A , b ) for a linear program is finite at ( c , A , b ) and in some neighborhood of ( c , A , b ) , then generalized gradients of z∗ with respect to c , b , and A exist and are ∂z∗ ( c ) = U∗ , ∂z∗ ( b ) = V ∗ , ∂z∗ ( A ) = { −vuT : ( u , v ) ∈ V ∗ × U∗ } . We build on these results to obtain generalized gradients of the linear program corresponding to the combinatorial problem . For the first case in the theorem , definition 2 states that in the linear program corresponding to P , only the cost vector c depends on w , through a ( sub ) differentiable function c = c ( w ) . Since w is in the interior of the parameter domainW , the objective value is finite over some neighborhood of w. Then , ∂z∗ ( w ) = ∂z∗ ( c ) ∂c ∂w = ∂c ∂w U∗ , where the generalized gradient z∗ ( c ) exists and is equal to U∗ . For the second case , the ideal formulation LP exists . Then , from definition 2 we have that ∂z∗ ( w ) = ∂z∗ ( b ) ∂b ∂w = ∂b ∂w V ∗ . The third case is a direct extension of the first two cases . Theorem 1 indicates that black-box combinatorial algorithms can be used to expand the range of transformations that can be efficiently utilized in neural networks . One immediate area of application is using them to specify a loss function . Consider a network F ( x ; β ) parameterized by a vector of tunable parameters β . The network transforms a batch of input samples x into a batch of outputs χ = F ( x ; β ) . Then , in the broadest primal-dual ∂-efficient case , χ is used , possibly with the true classes y , to formulate parameters ( c , A , b ) = g ( χ , y ) of a linear program corresponding to the combinatorial problem , through some ( sub ) differentiable function g. For Algorithm 1 Minimization of a combinatorial loss Input : batch x ⊂ X , y ⊂ Y , network F ( x ; β ) , functions g , h , combinatorial algorithm Π Output : Loss and its generalized gradient , L ( β ) , ∂L ( β ) 1 : procedure COMBLOSSMIN ( x , y , β , F , g , h , Π ) 2 : forward pass χ = F ( x ; β ) 3 : forward pass ( c , A , b ) = g ( χ , y ) 4 : run combinatorial solver to find optimal objective value z∗ = Π ( c , A , b ) and optimal primal and/or dual solution vectors u∗ , v∗ 5 : forward pass L ( β ) = h ( z∗ ) 6 : backward pass through h : ∂L/∂z∗ 7 : backward pass through Π : ∂z∗ ( c ) = u∗ , ∂z∗ ( b ) = v∗ , ∂z∗ ( A ) = −v∗u∗T 8 : backward pass through g and F 9 : ∂L ( β ) = ∂L∂z ( u∗ ∂c∂β − v ∗u∗T ∂A∂β + v ∗ ∂b ∂β ) 10 : return L ( β ) , ∂L ( β ) 11 : end procedure a given β and given batch samples ( x , y ) , we can then define loss as a function of the optimal objective value of the linear program corresponding to the combinatorial problem resulting from g ( F ( x ; β ) , y ) , L ( β ) = h ( z∗ ( c , A , b ) ) . This approach , summarized in Algorithm 1 , allows us to obtain the generalized gradient of the loss with respect to β as long as functions g and h are differentiable . For clarity , in Algorithm 1 , we did not consider functions h depending not just on z but also on x or y , but the extension is straightforward .

Summary: The value of the optimal objective as a function of the cost vector $c$ can be written as $z^*(c) = c^T u^*(c)$ where the optimal solution $u^*$ also depends on $c$. The function $u^*(c)$ is piecewise constant -- there are finitely (resp. countably) many feasible solutions; candidates for $u^*$ -- and so the function $z^*(c)$ is a piecewise linear function of $c$, with gradient $u^*(c)$, wherever it exists (otherwise there is analogous subgradient). Obviously, all it takes for computing $u^*(c)$ is solving -- anyhow -- the combinatorial problem. This is all trivial and well-known, yet the authors do precisely that.