A Surrogate Objective Framework for Prediction+Programming with Soft Constraints

1 Introduction . Mathematical optimization ( a.k.a . mathematical programming ) , e.g. , linear and quadratic programming , has been widely applied in decision-making processes , such as resource scheduling [ 1 ] , goods production planning [ 2 ] , portfolio optimization [ 3 ] , and power scheduling [ 4 ] . In practice , problem parameters ( e.g. , goods demands , and equity returns ) are often contextual and predicted by models with observed features ( e.g. , history time series ) . With the popularity of machine learning techniques and increasing available data , prediction+optimization has become a normal paradigm [ 5 ] . ∗Contributed during internship at Microsoft Research . † Corresponding Author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Prediction becomes critical to the performance of the full prediction+optimization workflow since modern optimization solvers ( e.g. , Gurobi [ 6 ] and CPLEX [ 7 ] ) can already efficiently find optimal solutions for most large scale optimization problems . Traditionally , prediction is treated separately as a general supervised learning problem and learned through minimizing a generic loss function ( e.g. , mean squared error for regression ) . However , studies have shown that minimization of the fitting errors does not necessarily lead to better final decision performance [ 8 , 9 , 10 , 11 ] . Recently , a lot of efforts on using optimization objective to guide the learning of prediction models have been made , which are decision-focused when training prediction models instead of using traditional prediction metrics , e.g . mean squared error losses . For linear objectives , the ‘ Smart Predict then Optimize ’ ( [ 10 ] ) proposes the SPO+ loss function to measure the prediction errors against optimization objectives , while direct optimization ( [ 12 ] ) updates the prediction model ’ s parameters by perturbation . For quadratic objectives , OptNet [ 13 , 14 ] implements the optimization as an implicit layer whose gradients can be computed by differentiating the KKT conditions and then back propagates to prediction neural network . CVXPY [ 15 ] uses similar technologies with OptNet but extends to more general cases of convex optimization . However , all the above state-of-the-art approaches do not contain the soft constraints in their objectives . In this paper , We consider linear ‘ soft-constraints ’ , a penalty in the form of max ( z , 0 ) , where z = Cx− d is a projection of decision variable x ∈ Rn and context variables C ∈ Rm×n , d ∈ Rm . Such soft constraints are often required in practice : for example , they could be the waste of provisioned resources over demands or the extra tax paid when violating regulatory rules . Unfortunately , the max ( · , 0 ) operator is not differentiable and thus can not be directly handled by these existing approaches . To differentiate soft constraints is a primary motivation of this paper . In this paper , we derive a surrogate objective framework for a broad set of real-world linear and quadratic programming problems with linear soft constraints and implement decision-focused differentiable predictions , with the assumption of non-negativity of hard constraint parameters . The framework consists of three steps : 1 ) rewriting all hard constraints into piece-wise linear soft constraints with a bounded penalty multiplier ; 2 ) using a differentiable element-wise surrogate to substitute the piece-wise objective , and solving the original function numerically to decide which segment the optimal point is on ; 3 ) analytically solving the local surrogate and obtaining the gradient ; the gradient is identical to that of the piecewise surrogate since the surrogate is convex/concave such that the optimal point is unique . Our main contributions are summarized as follows . First , we propose a differentiable surrogate objective function that incorporates both soft and hard constraints . As the foundation of our methodology , in Section 3 , we prove that , with reasonable assumptions generally satisfied in the real world , for linear and semi-definite negative quadratic programming problems , the constraints can be transformed into soft constraints ; then we propose an analytically differentiable surrogate function for the soft constraints max ( · , 0 ) . Second , we present the derived analytical and closed-form solutions for three representative optimization problems extended with soft constraints in Section 4 – linear programming with soft constraints , quadratic programming with soft constraints , and asymmetric soft constraint minimization . Unlike KKT-based differentiation methods , our method makes the calculation of gradients straightforward for predicting context parameters in any part of the problem . Finally , we apply with theoretical derivations and evaluate our approach in three scenarios , including synthetic linear programming , portfolio optimization , and resource provisioning in Section 4 , empirically demonstrate that our method outperforms two-stage and other predict+optimization approaches . 2 Preliminaries . 2.1 Real-world Optimization Problems with Soft Constraints . Our target is to solve the broad set of real-world mathematical optimization problems extended with a max ( z , 0 ) term in their objectives where z depends on decision variables and predicted context parameters . In practice , max ( z , 0 ) is very common ; for example , it may model overhead of underprovisioning , over-provisioning of goods , and penalty of soft regulation violations in investment portfolios . We call the above max ( z , 0 ) term in an objective as soft constraints , where z ≤ 0 is allowed to violate as long as the objective improves . The general problem formulation is max x g ( θ , x ) − αT max ( z , 0 ) , z = Cx− d , s.t . Ax ≤ b , Bx = c , x ≥ 0 ( 1 ) where g is a utility function , x ∈ Rn the decision variable , and θ ∈ Rn the predicted parameters . Based on observations on a broad set of practical problem settings , we impose two assumptions on the formulation , which serves as the basis of following derivations in this paper . First , we assume A ∈ Rm1×n ≥ 0 , b ∈ Rm1 ≥ 0 , B ∈ Rm2×n ≥ 0 , and c ∈ Rm2 ≥ 0 hold . This is because for problems with constraint on weights , quantities or their thresholds , these parameters are naturally non-negative . Second , we assume the linearity of soft constraints , that is z = Cx − d , where z ∈ Rm3 , C ∈ Rm3×n , and d ∈ Rm3 . This form of soft constraints make sense in wide application situations when describing the penalty of goods under-provisioning or over-provisioning , a vanish in marginal profits , or running out of current materials . Now we look into three representative instances of Eq.1 , extracted from real-world applications . Linear programming with soft constraints , where g ( x , θ ) = θTx . The problem formulation is max x θTx− αT max ( Cx− d , 0 ) , s.t . Ax ≤ b , Bx = c , x ≥ 0 ( 2 ) where α ≥ 0 . Consider the application of logistics where θ represents goods ’ prices , { A , B , C } are the capability of transportation tools , and { b , c } are the thresholds . Obviously , A , b , B , c ≥ 0 hold . Quadratic programming with soft constraints , where g ( θ , x ) = θTx − xTQx . One example is the classic minimum variance portfolio problem [ 16 ] with semi-definite positive covariance matrix Q and expected return θ to be predicted . We extend it with soft constraints which , for example , may represent regulations on portions of equities for some fund types . Formally with α ≥ 0 , we have : max x θTx− xTQx− αT max ( Cx− d , 0 ) , s.t . Bx = c , x ≥ 0 . ( 3 ) Optimization of asymmetric soft constraints . This set of optimization problems have the objective to match some expected quantities by penalizing excess and deficiency with probably different weights . Such formulation represents widespread resource provisioning problems , e.g. , power [ 4 ] and cloud resources [ 1 ] , where we minimize the cost of under-provisioning and over-provisioning against demands . Formally with α1 , α2 > 0 , we have : max x − ( αT1 max ( Cx− d , 0 ) + αT2 max ( d− Cx , 0 ) ) , s.t . Bx = c , x ≥ 0 . ( 4 ) In this paper , we consider a challenging task where C is a matrix to be predicted with known constants d. In reality , the Cx− d term may represent the `` wasted '' part when satisfying the actual need of d . 2.2 Prediction+Optimization . For compactness we write Eq.1 as maxx∈X f ( x , θ ) , where f is the objective function and X is the feasible domain . The solver for f is to solve x∗ = argmaxx∈X f ( x , θ ) . With parameters θ known , Eq.2–4 can be solved by mature solvers like Gurobi [ 6 ] and CPLEX [ 7 ] . In prediction+optimization , θ is unknown and needs to be predicted from some observed features ξ ∈ Ξ . The prediction model is trained on a dataset D = { ( ξi , θi ) } Ni=1 . In this paper , we consider the prediction model , namely Φ , as a neural network parameterized with ψ . In traditional supervised learning , Φψ is learned by minimizing a generic loss , e.g. , L1 ( Mean Absolute Error ) or L2 ( Mean Squared Error ) , which measures the expected distance between predictions and real values . However , such loss minimization is often inconsistent with the optimization objective f , especially when the prediction model is biased [ 11 ] . Instead , decision-focused learning directly optimizes ψ with respect to the optimization objective f , that is , maxψ E ( ξ , θ ) ∼D [ f ( x̂∗ ( Φψ ( ξ ) ) , θ ) ] . The full computational flow is illustrated in Fig.1 . In the gradient-based learning , update of ψ ’ s gradient is ∂r∂ψ = ∂θ̂ ∂ψ ∂x̂∗ ∂θ̂ ∂r ∂x̂∗ , where utility r = f ( x̂∗ , θ ) . The Jacobian ∂θ̂/∂ψ is computed implicitly by auto-differentiation of deep learning frameworks ( e.g , PyTorch [ 17 ] ) , and ∂r/∂x̂∗ is analytical . The main challenge is to compute ∂x̂∗/∂θ̂ , which depends on differentiating the argmax operation . One recent approach is to rewrite the objective to be convex ( by adding a quadratic regularizer if necessary ) , build and differentiate the optimality conditions ( e.g. , KKT conditions ) [ 18 ] which map θ̂ to the solution x̂∗ , and then apply implicit function theorem to obtain ∂x̂∗/∂θ̂ . Alternatively , in this paper we propose a novel approach that rewrites the problem as an unconstrained problem with soft constraints and derives analytical solutions , based on our observations on the real-world problem structures and coefficient properties . 3 Methodology . Our main idea is to derive a surrogate function for f with a closed-form solution such that the Jacobian ∂x∂θ is analytical , making the computation of gradient straightforward . Unlike other recent work [ 13 , 14 , 15 , 3 ] , our method does not need to solve KKT optimality condition system . Instead , by adding reasonable costs for infeasibility , we convert the constrained problem into an unconstrained one . With the assumption of concavity , we prove that there exist constant vectors β1 , β2 , β3 , such that Eq.1 can be equivalently transformed into an unconstrained problem : max x L ( x ) = max x g ( x , θ ) −αT max ( Cx−d , 0 ) −βT1 max ( Ax−b , 0 ) −βT2 |Bx−c|−βT3 max ( −x , 0 ) ( 5 ) The structure of this section is as follows . Section 3.1 proves that the three types of hard constraints can be softened by deriving bounds of β1 , β2 , β3 ; for this paper , we will assign each entry of the three vectors the equal value ( with a slight abuse of notation , we denote β1 = β2 = β3 = β for the proofs of bounds ; we align with the worst bound applicable to the problem formulation . ) Section 3.2 proposes a novel surrogate function of max ( · , 0 ) , such that the analytical form of ∂x∂θ can be easily derived via techniques of implicit differentiation [ 18 ] and matrix differential calculus [ 19 ] on equations derived by convexity [ 20 ] . Based on such derived ∂x∂θ , we develop our end-to-end learning algorithm of prediction+optimization whose detailed procedure is described in Appendix C .

Summary: The paper proposes a new method to integrate soft constraints in the predict+programming paradigm that focuses on mathematical programming models with a prediction component that needs to be considered during optimization/solving. The new method works by reformulating the problem into piecewise linear constraints and then optimizing a surrogate for the original problem. A theoretical analysis and motivation is provided as well as an experimental evaluation that underlines the effectiveness of the method.