Title: Two-Stage Predict+Optimize for MILPs with Unknown Parameters in Constraints
Venue: NeurIPS 2023
Accepted: yes
Keywords: Constraint optimization

Predict+Optimize
Abstract: Consider the setting of constrained optimization, with some parameters unknown at solving time and requiring prediction from relevant features. Predict+Optimize is a recent framework for end-to-end training supervised learning models for such predictions, incorporating information about the optimization problem in the training process in order to yield better predictions in terms of the quality of the predicted solution under the true parameters. Almost all prior works have focused on the special case where the unknowns appear only in the optimization objective and not the constraints. Hu et al. proposed the first adaptation of Predict+Optimize to handle unknowns appearing in constraints, but the framework has somewhat ad-hoc elements, and they provided a training algorithm only for covering and packing linear programs. In this work, we give a new simpler and more powerful framework called Two-Stage Predict+Optimize, which we believe should be the canonical framework for the Predict+Optimize setting. We also give a training algorithm usable for all mixed integer linear programs, vastly generalizing the applicability of the framework. Experimental results demonstrate the superior prediction performance of our training framework over all classical and state-of-the-art methods.

Two-Stage Predict+Optimize for Mixed Integer Linear Programs with Unknown Parameters in Constraints

Abstract

Consider the setting of constrained optimization, with some parameters unknown at solving time and requiring prediction from relevant features. Predict+Optimize is a recent framework for end-to-end training supervised learning models for such predictions, incorporating information about the optimization problem in the training process in order to yield better predictions in terms of the quality of the predicted solution under the true parameters. Almost all prior works have focused on the special case where the unknowns appear only in the optimization objective and not the constraints. Hu et al. proposed the first adaptation of Predict+Optimize to handle unknowns appearing in constraints, but the framework has somewhat ad-hoc elements, and they provided a training algorithm only for covering and packing linear programs. In this work, we give a new simpler and more powerful framework called Two-Stage Predict+Optimize , which we believe should be the canonical framework for the Predict+Optimize setting. We also give a training algorithm usable for all mixed integer linear programs, vastly generalizing the applicability of the framework. Experimental results demonstrate the superior prediction performance of our training framework over all classical and state-ofthe-art methods.

1 Introduction

Optimization problems are prevalent in modern society, and yet the problem parameters are not always available at the time of solving. For example, consider the real-world application scenario of stocking a store: as store managers, we need to place monthly orders for products to stock in the store. We want to stock products that sell fast and yield high profits, as much of them as possible, subject to the hard constraint of limited storage space. However, orders need to be placed two weeks in advance of the monthly delivery, and the customer demand next month cannot be known exactly at the time of order placement. In this paper, we consider the supervised learning setting, where the unknown parameters can be predicted from relevant features, and there are sufficient historical (features, parameters) pairs as training data for a prediction model. The goal, then, is to learn a prediction model from the training data such that, if we plug in the estimated parameters into the optimization problem and solve for an estimated solution , the estimated solution remains a good solution even after the true parameters are revealed.

The classic approach to the problem would be to train a simple regression model, based on standard losses such as (regularized) ℓ 2 loss, to predict parameters from the features. It is shown, however, that having a small prediction error in the parameter space does not necessarily mean that the estimated solution performs well under the true parameters. The recent framework of Predict+Optimize, by

Elmachtoub and Grigas [7], instead proposes the more effective regret loss for training, which compares the solution qualities of the true optimal solution and the estimated solution under the true parameters. Subsequent works [6, 8, 10, 13, 17, 19, 27] have since appeared in the literature, applying the framework to more and wider classes of optimization problems as well as focusing on speed-vs-prediction accuracy tradeoffs.

However, all these prior works focus only on the case where the unknown parameters appear in the optimization objective, and not in the constraints. The technical challenge for the generalization is immediate: if there were unknown parameters in the constraints, the estimated solution might not even be feasible under the true parameters revealed afterwards! Thus, in order to tackle the Predict+Optimize setting with unknowns in constraints, the recent work of Hu et al. [12] presents the first such adaptation on the framework: they view the estimated solution as representing a soft commitment . Once the true parameters are revealed, corrective action can be taken to ensure feasibility, potentially at a penalty corresponding to the real-life cost of (partially) reneging on a soft commitment. Their framework captures application scenarios whenever such correction is possible, and requires the practitioner to specify both the correction mechanism and the penalty function. These data can be determined and derived from the specific application scenario. As an example, in the product-stocking problem, an additional unknown parameter is the storage space, because it depends on how the current products in the store sell before the new order arrives. We need to place orders two weeks ahead based on predicted storage space. The night before the order arrives, we know the precise available space, meaning that the unknown parameter is revealed. A possible correction mechanism then is to throw away excess products that the store cannot keep, while incurring the penalty that is the retail price of the products, as well as disposal fees.

While the Hu et al. [12] framework does capture many application scenarios, there are important shortcomings. In their framework, they require the practitioner to specify a correction function that amends an infeasible solution into a feasible solution. However, the derivation of a correction function can be rather ad-hoc in nature. In particular, given an infeasible estimated solution, there may be many ways to transform the solution into a feasible one, and yet their framework requires the practitioner to pick one particular way. This leads to the second downside: it is difficult to give a general algorithmic framework that applies to a wide variety of optimization problems. Hu et al. had to restrict their attention only to packing and covering linear programs, for which they could propose a generic correction function. In this work, we aim to vastly generalize the kinds of optimization problems that Predict+Optimize can tackle under uncertainty in the constraints. In addition, the approach of Hu et al. fails to handle the interesting situation in which post-hoc correction is still desirable when the estimated solution is feasible but not good under the true parameters.

Our contributions are three-fold:

· To mitigate the shortcomings of the prior work, we propose and advocate a new framework, which we call Two-Stage Predict+Optimize 1 , that is both conceptually simpler and more expressive in terms of the class of optimization problems it can tackle. The key idea for the new framework is that the correction function is unnecessary. All that is required is a penalty function that captures the cost of modifying one solution to another. A penalty function is sufficient for defining a correction process: we formulate the correction process itself as a 'Stage 2' optimization problem, taking the originally estimated solution as well as the penalty function into account.

· Under this framework, we further propose a general end-to-end training algorithm that applies not only to packing and covering linear programs, but also to all mixed integer linear programs (MILPs). We adapt the approach of Mandi and Guns [18] to give a gradient method for training neural networks to predict parameters from features.

· We apply the proposed method to three benchmarks to demonstrate the superior empirical performance over classical and state-of-the-art training methods.

2 Background

In this section, we give basic definitions for optimization problems and the Predict+Optimize setting [7], and describe the state-of-the-art framework [12] for Predict+Optimize with unknown parameters

in constraints. The theory is stated in terms of minimization but applies of course also to maximization, upon appropriate negation. Without loss of generality, an optimization problem (OP) P can be defined as finding:

where x ∈ R d is a vector of decision variables, obj : R d → R is a function mapping x to a real objective value that is to be minimized, and C is a set of constraints that must be satisfied over x . We call x ∗ an optimal solution and obj ( x ∗ ) the optimal value . A parameterized optimization problem (Para-OP) P ( θ ) is an extension of an OP P :

where θ ∈ R t is a vector of parameters. The objective obj ( x, θ ) and constraints C ( x, θ ) can both depend on θ . When the parameters are known, a Para-OP is just an OP.

In the Predict+Optimize setting [7], the true parameters θ ∈ R t for a Para-OP are not known at solving time, and estimated parameters ˆ θ are used instead. Suppose each parameter is estimated by m features. The estimation will rely on a machine learning model trained over n observations of a training data set { ( A 1 , θ 1 ) , . . . , ( A n , θ n ) } where A i ∈ R t × m is a feature matrix for θ i , in order to yield a prediction function f : R t × m → R t predicting parameters ˆ θ = f ( A ) .

Solving the Para-OP using the estimated parameters, we obtain an estimated solution x ∗ ( ˆ θ ) . When the unknown parameters appear in constraints, one major challenge is that the feasible region is only approximated at solving time, and hence the estimated solution may be infeasible under the true parameters. Fortunately, in certain applications, the estimated solution is not a hard commitment, but only represents a soft commitment that can be modified once the true parameters are revealed. Hu et al. [12] propose a Predict+Optimize framework for such applications. The framework is as follows: i) the unknown parameters are estimated as ˆ θ , and an estimated solution x ∗ ( ˆ θ ) is solved using the estimated parameters, ii) the true parameters θ are revealed, and if x ∗ ( ˆ θ ) is infeasible under θ , it is amended into a corrected solution x ∗ corr ( ˆ θ, θ ) while potentially incurring some penalty , and finally iii) the solution x ∗ corr ( ˆ θ, θ ) is evaluated according to the sum of both the objective, under the true parameters θ , and the incurred penalty from correction.

More formally, a correction function takes an estimated solution x ∗ ( ˆ θ ) and true parameters θ and returns a corrected solution x ∗ corr ( ˆ θ, θ ) that is feasible under θ . A penalty function Pen ( x ∗ ( ˆ θ ) → x ∗ corr ( ˆ θ, θ )) takes an estimated solution x ∗ ( ˆ θ ) and the corrected solution x ∗ corr ( ˆ θ, θ ) and returns a nonnegative penalty. Both the correction function and the penalty function should be chosen according to the precise application scenario at hand. The final corrected solution x ∗ corr ( ˆ θ, θ ) is evaluated using the post-hoc regret , which is defined with respect to the corrected solution x ∗ corr ( ˆ θ, θ ) and the penalty function Pen ( x ∗ ( ˆ θ ) → x ∗ corr ( ˆ θ, θ )) . The post-hoc regret is the sum of two terms: (a) the difference in objective between the true optimal solution x ∗ ( θ ) and the corrected solution x ∗ corr ( ˆ θ, θ ) under the true parameters θ , and (b) the penalty that the correction process incurs. Mathematically, the post-hoc regret function PReg ( ˆ θ, θ ) : R t × R t → R ≥ 0 (for minimization problems) is:

where obj ( x ∗ corr ( ˆ θ, θ ) , θ ) is the corrected optimal value and obj ( x ∗ ( θ ) , θ ) is the true optimal value .

Given the post-hoc regret as a loss function, the empirical risk minimization principle dictates that we choose the prediction function to be the function f from the set of models F attaining the smallest average post-hoc regret over the training data:

3 Two-stage Predict+Optimize Framework

While the prior work by Hu et al. [12] is the first Predict+Optimize framework for unknowns in constraints, and is indeed applicable to a good range of applications, it has several shortcomings.

First, the framework requires mathematically formalizing both a penalty function and a correction function from the application scenario, and essentially imposes differentiability assumptions on the correction function for the framework to be usable. The ad-hoc nature of writing down a correction function limits the practical applicability of the framework. Second, as a result of needing a single (differentiable) correction function, Hu et al. [12] needed to restrict their attention to only packing and covering linear programs, in order to derive a general correction function that is applicable to all the instances. This also significantly limits the immediate applicability of their framework. Third, their framework only corrects an estimated solution when it is infeasible under the true parameters. Yet, there are applications where corrections are possible even when the estimated solution were feasible, but just not very good under the true parameters.

In this paper, we advocate using a simpler yet more powerful framework, which we call Two-Stage Predict+Optimize , addressing all of the above shortcomings. The simplified perspective will allow us to discuss more easily how to handle the entire class of mixed integer linear programs (MILPs) instead of being restricted to just packing and covering linear programs. Since MILPs include all optimization problems in NP (under a reasonable definition of NP for optimization problems), our framework is significantly more applicable in practice. We will describe the Two-Stage Predict+Optimize framework below, and discuss its application to MILPs in the next section.

Our framework is simple: we forgo the idea of a correction function and treat correction itself as an optimization problem, based on the penalty function, the estimated solution and the revealed true parameters. Recall the Hu et al. view of Predict+Optimize under uncertainties in constraints: the estimated solution is a form of soft commitment, which can be modified at a cost once the true parameters are revealed. The penalty function describes the cost of changing from an estimated solution to a final solution. The main observation is that, given an estimated solution and the revealed parameters, we should in fact solve a new optimization problem, formed by applying the true parameters to the original optimization, and adding the penalty function to the objective. The final solution from this new optimization thus takes the penalty of correction into account. This approach yields three immediate advantages. First, the practitioner no longer needs to specify a correction function, thus reducing the ad-hoc nature of the framework. Second, even feasible solutions are allowed to be modified after the true parameters are revealed if the penalty of doing so is not infinity. Third, conditioned on the same penalty function, the solution quality from our two-stage optimization approach is always at least as good as that from using any correction function. The last advantage is presented as Proposition A.1.

Now we formally define the Two-Stage Predict+Optimize framework.

I. In Stage 1, the unknown parameters are estimated as ˆ θ from features. The practitioner then solves the Stage 1 optimization, which is the Para-OP using the estimated parameters, to obtain the Stage 1 solution x ∗ 1 . The Stage 1 solution should be interpreted as some form of soft commitment, that we get to modify in Stage 2 at extra cost/penalty. Assuming the notation of the Para-OP in Section 2, the Stage 1 OP can be formulated as:

II. At the beginning of Stage 2, the true parameters θ are revealed. The Stage 2 optimization problem augments the original Stage 1 problem by adding a penalty term Pen ( x ∗ 1 → x ∗ 2 , θ ) to the objective, which accounts for the penalty (modelled from the application scenario) for changing from the softly-committed Stage 1 solution x ∗ 1 to the new Stage 2 and final solution x ∗ 2 . The Stage 2 OP can then be formulated as:

Solving the Stage 2 problem yields the final Stage 2 'corrected' solution x ∗ 2 .

III. The Stage 2 solution x ∗ 2 is evaluated according to the analogous post-hoc regret, as follows:

where again, x ∗ ( θ ) is an optimal solution of the Para-OP under the true parameters θ . Note that the post-hoc regret depends on all of a) the predicted parameters, b) the induced Stage 1 solution, c) the true parameters and d) the final Stage 2 solution.

To see this new framework applies in practice, the following example expands on the product-stocking problem in the introduction.

Example 1. Consider the product-stocking problem again, where regular orders have to be placed two weeks ahead of monthly deliveries. Since the available space at the time of delivery is unknown when we place the regular orders, depending on the sales over the next two weeks, we need to make a prediction on the available space to make a corresponding order. We learn the predictor using historical sales records from features such as time-of-year and price. Then, we use the predicted available space to optimize for the regular order we place. This is the Stage 1 solution.

The night before the order arrives, the unknown constraint parameter, i.e. the precise available space, is revealed. We can then check if we have over-ordered or under-ordered. In the case of over-ordering, we would have to call and ask the wholesale company to drop some items from the order. The company would perhaps allow taking the items off the final bill, but naturally they have a surcharge for last-minute changes. Similarly, if we under-ordered, we might request the wholesale company to send us more products, again naturally with a surcharge for last-minute ordering. The updated order is the Stage 2 decision. The incurred wholesaler surcharges induce the penalty function.

A reader who is familiar with the literature on two-stage optimization problems may note that the above framework is phrased slightly differently from some other two-stage problem formulations. In particular, some two-stage frameworks phrase Stage 1 solutions as hard commitments, and include recourse variables in both stages of optimization to model what changes are made in Stage 2. We show in Appendix A.1 how our framework can capture this other perspective, and in general discuss how problem modelling can be done in our new framework.

The reader may also wonder: what about application scenarios where the (Stage 1) estimated solution is a hard commitment, and there is absolutely no correction/recourse available? In Appendix A.2, we discuss how our framework is still useful and applicable for learning in these situations.

We also give a more detailed comparison, in Appendix A.3, between our new Two-Stage Predict+Optimize framework and the prior Hu et al. framework. Technically, if we ignored differentiability issues, the two frameworks are mathematically equivalent in expressiveness. However, we stress that our new framework is both conceptually simpler and easier to apply to a far wider class of optimization problems. We show concretely in the next section how to end-to-end train a neural network for this framework for all MILPs, vastly generalizing the method of Hu et al. which is restricted to packing and covering (non-integer) linear programs. In addition, Appendix A.3 also states and proves Proposition A.1, that if we fix an optimization problem, a prediction model and a penalty function, then the solution quality from our two-stage approach is always at least as good as using the correction function approach.

4 Two-Stage Predict+Optimize on MILPs

In this section, we describe how to give an end-to-end training method for neural networks to predict unknown parameters from features, under the Two-Stage Predict+Optimize framework. The following algorithmic method is applicable whenever both stages of optimization are expressible as MILPs. Due to the page limit, the discussion in this section is high-level and brief, with all the calculation details deferred to Appendix B.

The standard way to train a neural network is to use a gradient-based method. In the Two-Stage Predict+Optimize framework, we use the post-hoc regret PReg as the loss function. Therefore, for each edge weight w e in the neural network, we need to compute the derivative d PReg d w e . Using the law of total derivative, we get

As such, we wish to calculate each term on the right hand side.

The easiest term to handle is ∂ ˆ θ ∂w e , since ˆ θ is the neural network output, and so the derivatives can be directly calculated by standard backpropagation [25]. As for the terms ∂PReg ( ˆ θ,θ ) ∂x ∗ 2 ∣ ∣ ∣ x ∗ 1 and ∂PReg ( ˆ θ,θ ) ∂x ∗ 1 ∣ ∣ ∣ x ∗ 2 , they are easily calculable whenever both the optimization objective and penalty function are smooth, and in fact linear as in the case of MILPs. What remains are the terms ∂x ∗ 2 ∂x ∗ and

1

∂x ∗ 1 ∂ ˆ θ . The challenge is that x ∗ 2 is the solution of a MILP optimization (Stage 2) that uses x ∗ 1 as its parameters, i.e., differentiate through a MILP. Similarly, x ∗ 1 depends on ˆ θ through a MILP (Stage 1). Since MILP optima may not change under minor parameter perturbations, the gradients can be either 0 or non-existent, which are uninformative. We thus need to compute some approximation in order to get useful training signals.

Our approach, inspired by the work of Mandi and Guns [18], is to define a new surrogate loss function ˜ PReg that is differentiable and produces informative gradients. Prior works related to learning unknowns in constraints [1, 2, 27] give ways of differentiating through LPs or LPs with regularizations. These works can be used in place of the proposed approach. However, experiments in Appendix E demonstrate that the proposed approach performs at least as well in post-hoc regret performance as the others, while being faster. We show the construction of the proposed approach below, and note that it does not have a simple closed form. Nonetheless, we can compute its gradients.

The rest of the section assumes that both stages of optimization are expressible as a MILP in the following standard form:

with decision variables x ∈ R d and problem parameters c ∈ R d , A ∈ R p × d , b ∈ R p , G ∈ R q × d , h ∈ R q . The subset of indices S denotes the set of variables that are under integrality constraints. Since the unknown parameters may appear in any combination of c, A, b, G and h in the Stage 1 optimization for x ∗ 1 , the surrogate loss function construction needs computable and informative gradients for all of ∂x ∗ ∂c , ∂x ∗ ∂A , ∂x ∗ ∂b , ∂x ∗ ∂G and ∂x ∗ ∂h .

We follow the interior-point based approach of Mandi and Guns [18], used also by Hu et al. [12]. Consider the following convex relaxation of (4), for a fixed value of µ ≥ 0 :

This is a relaxation of (4) by i) dropping all integrality constraints, ii) introducing slack variables s ≥ 0 to turn Gx ≥ h into Gx -s = h and iii) replacing both the x ≥ 0 and s ≥ 0 constraints with the logarithm barrier terms in the objective, with multiplier µ ≥ 0 . The observation is that the gradients ∂x ∂c , ∂x ∂A , ∂x ∂b , ∂x ∂G and ∂x ∂h for (5) are all well-defined, computable and informative for a fixed value of µ ≥ 0 : Slater's condition holds for (5), and so the KKT conditions must be satisfied at the optimum ( x ∗ , s ∗ ) of (5). We can thus compute all the relevant gradients via differentiating the KKT conditions, using the implicit function theorem. We give all the calculation details in Appendix B.

Given the above observation, we then aim to construct the surrogate loss function by replacing the x ∗ 1 and x ∗ 2 , which are supposed to solved using MILP (4), with a) ˜ x 1 that is solved from program (5) relaxation of the Stage 1 optimization problem, using the predicted parameters ˆ θ and b) ˜ x 2 that is solved from the program (5) relaxed version of Stage 2 optimization, using ˜ x 1 and the true parameters θ . The only remaining question then, is, which values of µ do we use for the two relaxed problems?

Given a MILP in the form of (4), the interior-point based solver of Mandi and Guns [18] generates and solves (5) for a sequence of decreasing non-negative µ , with a termination condition that µ cannot be smaller than some cutoff value. Thus, we simply choose the cutoff value to use as ' µ ' in (5), which then completes the definition of the surrogate loss ˜ PReg .

Algorithmically, we train the neural network on the surrogate loss ˜ PReg as follows: given predicted parameters, we run the Mandi and Guns solver to get the optimal solution ( x ∗ , s ∗ ) for the final value of µ . We can then compute the gradient of the output solution with respect to any of the problem parameters using the calculations in Appendix B, combined with backpropagation, to yield d ˜ PReg d w e according to Equation (3).

In Appendix C, we give three example application scenarios, along with their penalty functions, that our training approach can handle. These problems are: a) an alloy production problem, for factory trying to source ores under uncertainty in chemical compositions in the raw materials, b) a variant of the classic 0-1 knapsack with unknown weights and rewards, and c) a nurse roster scheduling problem with unknown patient load. We show explicitly in Appendix C how both stages of optimization

can be formulated as MILPs for these applications, and apply the Appendix B calculations to yield gradient computation formulas for the surrogate loss ˜ PReg for these problems.

A limitation of our approach is the requirement that both stages must be expressible as MILPs, constraining the optimization objectives to be linear in the MILP decision variables. This contrasts the Hu et al. framework [12] which handles non-linear penalties. We point out that even MILPs can handle some non-linearity by using extra decision variables: for example, the absolute-value function. Moreover, the Appendix B gradient calculations can be adapted to handle general differentiable non-linear objectives. We present only MILPs as a main overarching application for this paper because of their widespread use in discrete optimization, with readily available solvers.

5 Experimental Evaluation

We evaluate the proposed method 2 on three benchmarks described in Section 4 and Appendix C. Table 1 reports the relevant benchmark problem sizes. We compare our method (2S) with the state of the art Predict+Optimize method, IntOpt-C [12], and 5 classical regression methods [9]: ridge regression (Ridge), k -nearest neighbors ( k -NN), classification and regression tree (CART), random forest (RF), and neural network (NN). All of these methods use their classic loss function to train the prediction models. At test time, to ensure the feasibility of the solutions when computing the post-hoc regret, we perform Stage 2 optimization on the estimated solutions for these classical regression methods before evaluating the final solution. Additionally, CombOptNet [23] is a different method focusing on learning unknowns in constraints, but with a different goal and loss function. We experimentally compare our proposed method with CombOptNet on the 0-1 knapsack benchmark-the only with available CombOptNet code. We also present a qualitative comparison in Section 6.

In the following experiments, we will need to take care to distinguish two-stage optimization as a training technique (Section 4) and as an evaluation framework (Section 3). We will denote our training method as '2S' in the experiments, and when we say 'Two-Stage Predict+Optimize' framework, we always mean it as an evaluation framework. 2S is always evaluated according to the Two-Stage Predict+Optimize framework. As explained above, we will also evaluate all the classical training methods using the Two-Stage Predict+Optimize framework. For our comparison with the prior work of Hu et al. [12], we will also distinguish their training method and evaluation framework. The name 'IntOpt-C' always refers to their training method using their correction function. We will simply call their evaluation framework the 'Hu et al. framework' or with similar phrasing (see Section 2 to recall details). IntOpt-C will sometimes be evaluated using our new Two-Stage Predict+Optimize framework, and sometimes the prior framework of Hu et al. [12] using their correction function.

The methods of k -NN, RF, NN, and IntOpt-C as well as 2S have hyperparameters, which we tune via cross-validation. We include the hyperparameter types and chosen values in Appendix D. In the main paper we only report the prediction performances. See Appendix H for runtime comparisons.

Alloy Production Problem The alloy production problem is a covering LP, see Appendix C.1 for the practical motivation and LP model. Since Hu et al. [12] also experimented on this problem, we use it to compare our 2S method with IntOpt-C [12], using the same dataset and experimental setting.

We conduct experiments on the production of two real alloys: brass and an alloy blend for strengthening Titanium. For brass, 2 kinds of metal materials, Cu and Zn, are required [14]. The blend of the two materials are, proportionally, req = [627 . 54 , 369 . 72] . For the titanium-strengthening alloy, 4 kinds of metal materials, C, Al, V, and Fe, are required [15]. The blend of the four materials are proportional to req = [0 . 8 , 60 , 40 , 2 . 5] . We use the same real data as that used in IntOpt-C [12] as numerical values in our experiment instances. In this dataset [23], each unknown metal concentration

is related to 4096 features. For experiments on both alloys, 350 instances are used for training and 150 instances for testing the model performance. For NN, IntOpt-C, and 2S, we use a 5-layer fully connected network with 512 neurons per hidden layer.

In the penalty function described in Appendix C.1, we need to choose a penalty factor/multiplier for each supplier. We conduct experiments on 6 types of penalty factor ( σ ) settings: 6 vectors where each entry is i.i.d. uniformly sampled from [0 . 25 ± 0 . 015] , [0 . 5 ± 0 . 015] , [1 . 0 ± 0 . 015] , [2 . 0 ± 0 . 015] , [4 . 0 ± 0 . 015] , and [8 . 0 ± 0 . 015] respectively. This random sampling of σ ensures that the penalty factor for each supplier is different, but remains roughly on the same scale.

The first experiment we run compares 2S+Two-Stage Predict+Optimize framework with IntOpt-C+Hu et al. framework. Specifically, we compare a) using 2S for training and evaluating using the Two-Stage Predict+Optimize framework in Section 3, versus b) using IntOpt-C for training and evaluating using the same correction function from training, according to the Hu et al. framework described in Section 2. Table 2 compares the mean post-hoc regret and standard deviations for the alloy production problem for the two different frameworks. The table shows that Two-Stage Predict+Optimize framework always achieves smaller mean post-hoc regret than the Hu et al. framework. Compared with the Hu et al. framework, our framework obtains 6.18%-35.63% smaller mean post-hoc regret in brass production, and 6.59%-29.89% smaller mean post-hoc regret in titanium-alloy production.

We present a further comparison in Appendix F with a variant of the Hu et al. framework-the ℓ 2 projection idea in [3], which performs even worse than the Hu et al. framework.

The second experiment compares various training approaches all evaluated under the Two-Stage Predict+Optimize framework. That is, the models are trained differently, but at test time, we always use Stage 2 optimization to give a final solution and evaluate post-hoc regret on it. Table 3 reports the mean post-hoc regrets and standard deviations across 10 runs for each training method on the alloy production problem. The table shows that our method, 2S, achieves the best performance, compared with IntOpt-C achieving the second best performance, beating all the classical training approaches. Compared with IntOpt-C, 2S obtains 1.20%-3.18% smaller mean post-hoc regrets in brass production, and 1.18%-5.33% smaller mean post-hoc regret in titanium-alloy production. Compared with the classical approaches, the improvements are much more significant. 2S obtains at least 2.44%-45.48% smaller mean post-hoc regrets in brass production, and at least 1.39%-38.78% smaller mean post-hoc regret in titanium-alloy production. The average True Optimal Values (TOV) are reported in the last column of Table 3 for reference, although the reader should take care to not over-interpret the ratio

between the post-hoc regret and the true optimal value, since the post-hoc regret also includes the penalty term which increases with the penalty factors.

0-1 knapsack In the second example, we showcase our framework on a packing integer programming problem, a variant of the 0-1 knapsack problem, with unknown item prices p i and sizes s i . See Appendix C.2 for details of an application in running a 'proxy buyer' business. Here, the unknown parameters appear in both the objective and constraints. The proposed 2S method can handle this MILP straightforwardly, but the IntOpt-C method cannot be applied. Thus, we only experiment with the Two-Stage Predict+Optimize framework for evaluation, and compare the proposed 2S method with classical approaches and CombOptNet. Again, all approaches are evaluated at test time using the Stage 2 optimization to yield the final solution, on which the post-hoc regret is computed.

The MILP formulation of the two stages and the penalty function are described in Appendix C.2. We use the dataset of Paulus et al. [23], in which each 0-1 knapsack instance consists of 10 items and each item has 4096 features related to its price and size. For both NN and our method, we use a 5-layer fully-connected network with 512 neurons per hidden layer. We conduct experiments on 4 different knapsack capacities: 100, 150, 200, and 250. We use 700 instances for training and 300 instances for testing the model performance. Considering the real-life setting, we use 3 scales of the penalty factor for the penalty function in Appendix C.2: σ = 0 . 05 , 0 . 25 , or 0 . 5 .

Table 4 reports the mean post-hoc regrets and standard deviations across 10 runs for each approach on this 0-1 knapsack problem. Due to the space limitation and the fact that larger penalty factors are unrealistic in this problem setting, we present penalty factors ≥ 1 in Appendix G. The average True Optimal Values (TOV) are reported in the last column, again for reference. As shown in the table, our proposed 2S method has significantly better results. In addition, we observe that across all approaches, the post-hoc regrets decrease as the knapsack capacity increases: this is due to the fact that as the capacity increases, more and more items can be selected, and so minor inaccuracies in predicted values/weights do not affect the selected set of items as much. On the other hand, the advantage of our 2S method over other approaches actually becomes more significant as the capacity increases, demonstrating the superior accuracy of our approach.

Nurse Scheduling Problem Our last experiment is on the nurse scheduling problem (NSP) with unknown patients needs, with the goal of scheduling a nurse roster satisfying unknown patient load demands while minimizing mismatched nurse-shift preferences as the objective. See Appendix C.3 for a description of the application scenario, the MILP formulations of the two stages, as well as the associated penalty function. Given that NSP is not an LP, IntOpt-C again does not apply, and so we

only compare the proposed 2S training method with the classical approaches, using the Two-Stage Predict+Optimize framework for evaluation. Each NSP instance consists of 15 nurses, 7 days, and 3 shifts per day. The nurse preferences are obtained from the NSPLib dataset [26], which is widely used for NSP [16, 20]. The number of patients that each nurse can serve in one shift is randomly generated from [10,20], representing the fact that each nurse has different capabilities. Given that we are unable to find datasets specifically for the patient load demands and relevant prediction features, we follow the experimental approach of Demirovic et al. [4, 5, 6] and use real data from a different problem (the ICON scheduling competition) as the numerical values required for our experiment instances. In this dataset, the unknown number of patients per shift is predicted by 8 features.

Since there are far fewer features than the previous experiments, for both NN and 2S we use a smaller network structure: a 4-layer fully-connected network with 16 neurons per hidden layer. We use 210 instances for training and 90 instances for testing. Just like the first experiment, we use 6 scales of penalty factors (see Appendix C.3 for the penalty function): γ with i.i.d. entries drawn uniformly from [0 . 25 ± 0 . 015] , [0 . 5 ± 0 . 015] , [1 . 0 ± 0 . 015] , [2 . 0 ± 0 . 015] , [4 . 0 ± 0 . 015] , and [8 . 0 ± 0 . 015] .

Table 5 reports the mean post-hoc regrets and standard deviations across 10 runs for each approach on the NSP. The table shows that the proposed 2S method again has the best performance among all the training approaches. Our 2S method obtains at least 7.61%, 15.65%, 17.99%, 20.51%, 46.76%, and 62.49% smaller post-hoc regret than other classical methods when the penalty factor is [0 . 25 ± 0 . 015] , [0 . 5 ± 0 . 015] , [1 . 0 ± 0 . 015] , [2 . 0 ± 0 . 015] , [4 . 0 ± 0 . 015] , and [8 . 0 ± 0 . 015] respectively.

Runtime Analysis Appendix H gives the training times for each method. Most classical approaches are faster than our 2S method, although as shown their post-hoc regrets are much worse. In alloy production, the only setting where IntOpt-C applies, its running time is shorter but comparable with 2S. In 0-1 knapsack, the only problem with public CombOptNet code, the 2S method is much faster.

6 Literature Review

Section 1 already summarized prior works in Predict+Optimize, most of which focus on learning unknowns only in the objective. Only the Hu et al. [12] framework considers unknowns in constraints.

Here we summarize other works related to learning unknowns in optimization problem constraints, particularly those outside of Predict+Optimize. These works can be placed into two categories.

One category also considers learning unknowns in constraints, but with very different goals and measures of loss. For example, CombOptNet [23] and Nandwani et al. [21] focus on learning parameters so as to make the predicted optimal solution (first-stage solution in our proposed framework) as close to the true optimal solution x ∗ as possible in the solution space/metric. By contrast, our proposed framework explicitly formulates the two-stage framework and post-hoc regret in order to directly capture rewards and costs in application scenarios. Experiments on 0-1 knapsack in Section 5 show that these other methods yield worse predictive performance when evaluated on the post-hoc regret, under the proposed two-stage framework.

Another category gives ways to differentiate through LPs or LPs with regularizations, as a technical component in a gradient-based training algorithm. As mentioned in Section 4, these works can indeed be used in place of our proposed approach in Section 4/Appendix B. However, we point out that: (i) these other technical tools are essentially orthogonal to our primary contribution, which is the two-stage framework (Section 3), and (ii) nonetheless, experiments on the 0-1 knapsack in Appendix E demonstrate that our gradient calculation approach performs at least as well in post-hoc regret performance as other works, while being faster.

7 Summary

Weproposed Two-Stage Predict+Optimize: a new, conceptually simpler and more powerful framework for the Predict+Optimize setting where unknown parameters can appear both in the objective and in constraints. We showed how the simpler perspective offered by the framework allows us to give a general training framework for all MILPs, contrasting prior work which apply only to covering and packing LPs. Experimental results demonstrate that our training method offers significantly better prediction performance over other classical and state-of-the-art approaches.

Acknowledgments

We thank the anonymous referees for their constructive comments. In addition, Xinyi Hu and Jimmy H.M. Lee acknowledge the financial support of a General Research Fund (RGC Ref. No. CUHK 14206321) by the University Grants Committee, Hong Kong. Jasper C.H. Lee was supported in part by the generous funding of a Croucher Fellowship for Postdoctoral Research, NSF award DMS-2023239, NSF Medium Award CCF-2107079 and NSF AiTF Award CCF-2006206.

References

A. Agrawal, B. Amos, S. Barratt, S. Boyd, S. Diamond, and J. Z. Kolter. Differentiable convex optimization layers. Advances in neural information processing systems , 32, 2019.

B. Amos and J. Z. Kolter. Optnet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning , pages 136-145. PMLR, 2017.

B. Chen, P. L. Donti, K. Baker, J. Z. Kolter, and M. Bergés. Enforcing policy feasibility constraints through differentiable projection for energy optimization. In Proceedings of the Twelfth ACM International Conference on Future Energy Systems , pages 199-210, 2021.

E. Demirovi´ c, P. J. Stuckey, J. Bailey, J. Chan, C. Leckie, K. Ramamohanarao, and T. Guns. An investigation into Prediction+Optimisation for the knapsack problem. In International Conference on Integration of Constraint Programming, Artificial Intelligence, and Operations Research , pages 241-257. Springer, 2019.

E. Demirovi´ c, P. J. Stuckey, J. Bailey, J. Chan, C. Leckie, K. Ramamohanarao, and T. Guns. Predict+Optimise with ranking objectives: Exhaustively learning linear functions. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence , pages 1078-1085, 2019.

E. Demirovi´ c, P. J. Stuckey, T. Guns, J. Bailey, C. Leckie, K. Ramamohanarao, and J. Chan. Dynamic programming for Predict+Optimise. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence , pages 1444-1451, 2020.

A. N. Elmachtoub and P. Grigas. Smart 'Predict, then Optimize'. Management Science , 68(1):9-26, 2022.

A. N. Elmachtoub, J. C. N. Liang, and R. McNellis. Decision trees for decision-making under the predict-then-optimize framework. In Proceedings of the 37th International Conference on Machine Learning , pages 2858-2867, 2020.

J. Friedman, T. Hastie, and R. Tibshirani. The elements of statistical learning . Springer series in statistics New York, 2001. Volume 1, Number 10.

A. U. Guler, E. Demirovi´ c, J. Chan, J. Bailey, C. Leckie, and P. J. Stuckey. A divide and conquer algorithm for Predict+Optimize with non-convex problems. In Proceedings of the Thirty-Sixth AAAI Conference on Artificial Intelligence , 2022.

Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2023.

X. Hu, J. C. H. Lee, and J. H. M. Lee. Predict+Optimize for packing and covering LPs with unknown parameters in constraints. In Proceedings of the AAAI Conference on Artificial Intelligence , 2022.

X. Hu, J. C. H. Lee, J. H. M. Lee, and A. Z. Zhong. Branch & Learn for recursively and iteratively solvable problems in Predict+Optimize. In Advances in Neural Information Processing Systems , 2022.

K. B. Kabir and I. Mahmud. Study of erosion-corrosion of stainless steel, brass and aluminum by open circuit potential measurements. Journal of Chemical Engineering , pages 13-17, 2010.

N. Kahraman, B. Gülenç, and F. Findik. Joining of titanium/stainless steel by explosive welding and effect on interface. Journal of Materials Processing Technology , 169(2):127-133, 2005.

B. Maenhout and M. Vanhoucke. Branching strategies in a Branch-and-Price approach for a multiple objective nurse scheduling problem. Journal of scheduling , 13(1):77-93, 2010.

J. Mandi, V. Bucarey, M. M. K. Tchomba, and T. Guns. Decision-focused learning: Through the lens of learning to rank. In International Conference on Machine Learning , pages 14935-14947. PMLR, 2022.

J. Mandi and T. Guns. Interior point solving for LP-based Prediction+Optimisation. Advances in Neural Information Processing Systems , 33:7272-7282, 2020.

M. Mulamba, J. Mandi, M. Diligenti, M. Lombardi, V. Bucarey, and T. Guns. Contrastive losses and solution caching for Predict-and-Optimize. arXiv preprint arXiv:2011.05354 , 2020.

R. Muniyan, R. Ramalingam, S. S. Alshamrani, D. Gangodkar, A. Dumka, R. Singh, A. Gehlot, and M. Rashid. Artificial bee colony algorithm with Nelder-Mead method to solve nurse scheduling problem. Mathematics , 10(15):2576, 2022.

Y. Nandwani, R. Ranjan, P. Singla, et al. A solver-free framework for scalable learning in neural ilp architectures. Advances in Neural Information Processing Systems , 35:7972-7986, 2022.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 , pages 8024-8035. 2019.

A. Paulus, M. Rolínek, V. Musil, B. Amos, and G. Martius. Comboptnet: Fit the right NP-hard problem by learning integer programming constraints. In International Conference on Machine Learning , pages 8443-8453. PMLR, 2021.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back-propagating errors. nature , 323(6088):533-536, 1986.

M. Vanhoucke and B. Maenhout. Nsplib-a nurse scheduling problem library: A tool to evaluate (meta-) heuristic procedures. In Operational research for health policy: making better decisions, proceedings of the 31st annual meeting of the working group on operations research applied to health services , pages 151-165, 2007.

B. Wilder, B. Dilkina, and M. Tambe. Melding the data-decisions pipeline: Decision-focused learning for combinatorial optimization. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence , pages 1658-1665, 2019.

A Detailed discussion on the Two-Stage Predict+Optimize framework

A.1 Problem modelling using the framework

As mentioned in Section 3, the proposed Two-Stage Prediction+Optimize framework is phrased differently from some other two-stage problem formulations. The proposed framework phrases Stage 1 solutions as soft commitments, and corrects Stage 1 solutions with penalty in Stage 2. On the other hand, some two-stage frameworks phrase Stage 1 solutions as hard commitments, and include explicit recourse variables in both stages of OP to model the correction in Stage 2. Some optimization problems are more natural to express according to one perspective than the other, while some problems might be straightforward to express in either. This section aims to show that our framework, while explicitly stated and motivated according to the first perspective, is in fact general enough to also easily model the second perspective of hard commitments and recourse actions. In what follows, we first describe different types of variables and how our framework can capture them. Then, we give two example problems that respectively use the soft/hard commitment perspectives, and we detail how the problem can be modelled.

Soft commitment variables: These are variables which represent decisions that correspond to soft commitments made in Stage 1 in an application, namely decisions that may be modified once the true parameters are revealed, but at a cost or penalty. The discussion in Section 3 is tailored for this kind of variables-simply define such a variable in Stage 1 and use a finite penalty function to model the cost of changing this soft commitment in Stage 2.

̸

Recourse/other variables: These are variables which represent explicit actions/decisions taken only in Stage 2, once the true parameters are revealed. These variables are necessary, for example, when Stage 1 actions are all hard commitment variables, to ensure that we have a mechanism for corrective action if the hard Stage 1 decisions are in any way 'incompatible" with the revealed parameters. These corrective actions also typically come at a cost. Thus, to model these variables, simply include them in both Stages 1 and 2, and incorporate their cost into the objective of the optimization problem. There should also be 0 penalty for modifying these variables between the stages.

Hard commitment variables: These are variables x ∗ hard which represent hard commitments made in Stage 1, meaning that after commitment, they absolutely cannot change in Stage 2. To model these variables in our framework, simply write a penalty function that is infinite whenever Stage 1 and Stage 2 solutions for these variables are different. Explicitly, add a term ∞· ✶ [ x ∗ hard , 1 = x ∗ hard , 2 ] . This way, no Stage 2 solution will change these variables from what they were committed to in Stage 1.

To summarize, Stage 1 actions can be classified as either soft or hard commitments, depending on whether they can be changed in Stage 2 (at a finite penalty). Stage 2 actions are classified as 'recourse" variables, which are simply variables that have no penalty from changing between Stage 1 to Stage 2. The above discussion shows how our framework captures all these possibilities. We now give two example applications: the first one is more naturally expressed via the soft commitment perspective, and the second one is more natural to phrase using hard commitments+recourse. We give also their explicit formulations to demonstrate how the modelling is done in our framework.

We first show an example problem which is naturally modelled using soft commitment variables and penalty functions. Consider the product-stocking problem in Example 1 again, where regular orders have to be placed two weeks ahead of monthly deliveries. We aim to maximize the net profit by selling stocked products, under the constraint that the available storage space is limited. Each product i has a purchase price p u i (the price of purchasing the product from the wholesale company) and a selling price p s i (the price of selling the product to customers), and needs s i space to be stocked. Let x i denote whether the product i is ordered. In Stage 1, i.e., two weeks before the delivery, the available storage space Sp at the time of delivery is unknown, and we place the order x based on estimated space. In Stage 2, i.e., the night before the delivery, the precise available space is revealed, and we ask the wholesale company to change the order but need to pay a surcharge for last-minute changes. Assume the surcharge for the last-minute change in the order of product i is c i . In this example, x i is thus a soft commitment variable, and we model the surcharge c i using the penalty function of the framework.

The proposed framework can naturally model this problem. The Stage 1 OP can be formulated as:

In Stage 2, the order x ∗ 1 can be changed with surcharges, which can be modelled as a penalty function:

Then the Stage 2 OP can be formulated as:

Next, we give an example problem which is more naturally modelled using hard commitment variables and recourse variables. Consider a production-planning problem: a company owns a set of facilities and provides services to a set of customers. Each facility i can provide a fixed amount of services m i and has a fixed operating cost f i in the standard working mode. The company aims to meet customer demands d at the minimum operating costs. In Stage 1, the company decides which facilities to open for production based on the estimated demands ˆ d . This is a binary decision variable x i for each facility i . In Stage 2, the orders from customers arrive and the demands d are revealed. If the services provided by the operating facilities in the standard mode cannot meet demands, the company will ask some facilities that are already operating (i.e. x i = 1 ) to work overtime, but naturally need to pay high overtime fees. Let o i denote the unit overtime fee for producing service in facility i , and σ i denote the amount of service provided by overtime working in facility i .

This example is naturally modelled using hard commitment variables and recourse variables. Which facilities to operate, x , is a vector of 0/1 hard commitment variables. The amount of service, σ , provided by the overtime working mode of operating facilities can be modeled by recourse variables, and the recourse costs are the overtime fees o . Using hard commitment variables and recourse variables, the Stage 1 OP can be formulated as:

̸

In Stage 2, we include a term ∞· ✶ [ x ∗ 1 = x ] in the penalty function part of the Stage 2 objective to make sure that x cannot be changed, while the penalty for changing σ is zero since it is a recourse variable. The Stage 2 OP is formulated as:

̸

In summary, we discussed how to model in our framework soft and hard commitment actions in Stage 1, as well as recourse/other actions in Stage 2. We gave two concrete examples to demonstrate how such modelling can be done.

A.2 What if correction/recourse is not possible in the application?

The motivating premise of this paper is that the application scenario at hand allows for some posthoc corrective action once the true parameters are revealed. One natural question is: what if such

̸

corrective action (Stage 2 actions) is not actually possible in the application? For example, in our running example of the product-stocking problem, we considered a wholesale company that allows for order changes the night before. Other wholesalers may not allow such a correction/modification. Our framework can essentially still model these scenarios: just set the penalty of modification to infinity (or at least, very large numbers for practice). Concretely, use the penalty function ∞· ✶ [ x ∗ 2 = x ∗ 1 ] (or replace ∞ with a very large number). This penalty function encourages the learning algorithm to learn conservative predictions that maximize the chances of yielding Stage 1 decisions that remain feasible in Stage 2.

To show this, we ran another quick experiment, using the 0-1 knapsack problem setting in the paper (with knapsack capacity = 100). This time, as we varied the magnitude of the penalty function, we measure at test time the empirical fraction of Stage 1 solutions that remain feasible under the true parameters. The results in Table 6 demonstrate our claim that as the penalty term increases, the predictions get more and more likely to remain feasible, making it a reasonable way to train a predictor even when Stage 2 correction mechanisms do not actually exist in the application.

A.3 Two-Stage Predict+Optimize vs Prior Hu et al. Framework

As mentioned earlier in Section 3, Two-Stage Predict+Optimize is technically mathematically equivalent to the prior framework of Hu et al. [12], in the sense of expressiveness, ignoring differentiability issues. On the one hand, we can regard the Stage 2 optimization as a form of correction function, and hence Two-Stage Predict+Optimize can be considered as a special case of the Hu et al. [12] framework. On the other hand, given a correction function as in the Hu et al. [12] framework, we can simply modify the penalty function such that we keep the penalty value of the corrected solution, and make the penalty value infinite for all other potential Stage 2 solutions. This forces the Stage 2 optimization to always emulate the correction function. In this sense, our Two-Stage Predict+Optimize framework can also emulate the Hu et al. [12] framework, meaning that the two frameworks are technically equivalent.

Nevertheless, the Two-Stage Predict+Optimize framework is both conceptually simpler and easier to apply . In the main paper, we showed how to perform end-to-end neural network training within this new framework whenever both stages of optimization can be phrased as MILPs, and also give empirical experimental results. Together, they demonstrate the much more general applicability of the Two-Stage Predict+Optimize framework.

We end this appendix with the statement and short proof that, conditioned on the same penalty function and prediction model, Two-Stage Predict+Optimize always outputs at least as good a final solution as the prior framework using any correction function.

Proposition A.1. Consider an arbitrary minimization Para-OP P , penalty function Pen , correction function x ∗ corr , estimated parameters ˆ θ and true parameters θ . Let x ∗ ( ˆ θ ) and x ∗ 1 ( ˆ θ ) both denote the estimated solution from the estimated parameters ˆ θ , x ∗ 2 ( ˆ θ, θ ) be the output final solution from the Two-Stage Predict+Optimize framework, and x ∗ corr ( ˆ θ, θ ) be the output corrected solution from the prior framework of Hu et al. Then,

Proof. Observe that both sides of the inequality are the objective of the Stage 2 optimization problem, evaluated at x ∗ 2 and x ∗ corr respectively. Since x ∗ 2 is the optimal solution to the minimization problem, the inequality follows directly.

B Gradient Calculations for Problem (5)

Approximating ∂x ∗ 1 ∂ ˆ θ . In the context of the MILP, the unknown parameter ˆ θ may either be c, A, b, G , or h . Using the solution x and the barrier weight µ returned from solving Problem (5), we can compute the relevant derivatives of ∂x ∗ 1 ∂ ˆ c . The case of c has already been derived by Mandi and Guns [18] (see Appendix A.1 and A.2 in their paper). Problem (5) can be rewritten as:

Fact B.1. Consider the LP relaxation (6), defining x ′ as a function of c ′ , A ′ and b ′ . Then, according to Mandi and Guns [18], under this definition of x ∗ ,

where X ′ = diag ( x ′ ) , t = µX ′-1 e, T = diag ( t ) , y ′ is the lagrangian multiplier of Problem (6), and κ and τ are additional variables added by Mandi and Guns [18] to represent the duality gap. The gradient ∂x ∗ 1 ∂ ˆ c can be obtained by solving this system of equalities.

Define the notation f ( x, c, G, h ) = c ⊤ x -µ ∑ d i =1 ln( x i ) -µ ∑ q i =1 ln( G ⊤ i x -h i ) . Then, Problem (5) can be expressed as finding:

Using this notation, we write down the following four lemmas on computing ∂x ∗ ∂G , ∂x ∗ ∂h , ∂x ∗ ∂A , and ∂x ∗ ∂b approximately.

Lemma B.2. Consider the LP relaxation (7), defining x ∗ as a function of c, A, b, G and h . Then, under this definition of x ∗ ,

where H = f xx ( x, c, G, h ) denotes the matrix of second derivatives of f with respect to different coordinates of x , and similarly for other subscripts, and explicitly:

̸

and

̸

Note that when there are no equality constraints, i.e., A = 0 , we have

which is the same as the Lemma 3 in [12].

where

Proof. Using the Lagrangian multiplier y , the Lagrangian relaxation of Problem (7) can be written as

▲ ( x, y ; c, G, h ) = f ( x, c, G, h ) + y ⊤ ( b -Ax ) (9) Since x ∗ = arg min x f ( x, c, G, h ) s.t. Ax = b is an optimum, x ∗ must obey the Karush-KuhnTucker (KKT) conditions, obtained by setting the partial derivative of Equation (9) with respect to x and y to 0. Let f x ( x, c, G, h ) denotes the vector of first derivatives of f with respect to different coordinates of x , f xx ( x, c, G, h ) denotes the matrix of second derivatives of f with respect to different coordinates of x , we obtain:

The implicit differentiation of these KKT conditions with respect to G allows us to get the following system of equalities:

By solving this system of equalities, we can obtain

Since f ( x, c, G, h ) = c ⊤ x -µ ∑ d i =1 ln( x i ) -µ ∑ q i =1 ln( G ⊤ i x -h i ) , we have

̸

and

̸

Lemma B.3. Consider the LP relaxation (7), defining x ∗ as a function of c, A, b, G and h . Then, under this definition of x ∗ ,

where H = f xx is defined as in Lemma B.2 and

Note that when there are no equality constraints, i.e., A = 0 , we have

which is the same as the Lemma 2 in [12].

Proof. As stated in the proof of Lemma B.2, using the Lagrangian relaxation and the Karush-KuhnTucker (KKT) conditions, we obtain:

The implicit differentiation of these KKT conditions with respect to h allows us to get the following system of equalities:

By solving this system of equalities, we can obtain

where H = f xx is defined as in Lemma B.2. Since f ( x, c, G, h ) = c ⊤ x -µ ∑ d i =1 ln( x i ) -µ ∑ q i =1 ln( G i x -h i ) , we have

Lemma B.4. Consider the LP relaxation (7), defining x ∗ as a function of c, A, b, G and h . Then, under this definition of x ∗ ,

where I 1 = -∂A ⊤ ∂A ij , I 2 = ∂A ∂A ij , and H = f xx is defined as in Lemma B.2.

Proof. As stated in the proof of Lemma B.2, using the Lagrangian relaxation and the Karush-KuhnTucker (KKT) conditions, we obtain:

Since A ∈ R p × d , fix i ∈ { 1 , . . . , p } , j ∈ { 1 , . . . , d } , the implicit differentiation of these KKT conditions with respect to A ij allows us to get the following system of equalities:

Let I 1 = -∂A ⊤ ∂A ij , I 2 = ∂A ∂A ij . By solving this system of equalities, we can obtain

where H = f xx is defined as in Lemma B.2.

Lemma B.5. Consider the LP relaxation (7), defining x ∗ as a function of c, A, b, G and h . Then, under this definition of x ∗ ,

where H = f xx is defined as in Lemma B.2.

Proof. As stated in the proof of Lemma B.2, using the Lagrangian relaxation and the Karush-KuhnTucker (KKT) conditions, we obtain:

The implicit differentiation of these KKT conditions with respect to b allows us to get the following system of equalities:

By solving this system of equalities, we can obtain

where H = f xx is defined as in Lemma B.2.

C Details for Case Studies

Since the penalty function partly or solely affects the terms ∂PReg ( ˆ θ,θ ) ∂x ∗ 2 ∣ ∣ ∣ x ∗ 1 , ∂PReg ( ˆ θ,θ ) ∂x ∗ 1 ∣ ∣ ∣ x ∗ 2 , and ∂x ∗ 2 ∂x ∗ 1 , we give three case studies for our framework to show how to design the penalty function and compute gradients using the corresponding penalty function.

C.1 Alloy Production Problem

We first demonstrate, using the example of the alloy production problem, how our framework can tackle problems solvable by the prior work of Hu et al. [12]. An alloy production factory needs to produce a certain amount of a particular alloy, requiring a mixture of M kinds of metals. To that end, it must acquire at least req m tons of each of the m ∈ [ M ] metals. The raw materials are to be obtained from K suppliers, each supplying a different type of ore. The factory plans to buy ores from sites and then extract the metals themselves. The ore supplied by site k ∈ [ K ] contains a con km ∈ [0 , 1] fraction of material m at a price of cost k per ton. The goal of the factory is to meet its requirements for each metal at the minimum cost. However, the precise metal concentrations (averaged in a batch) are unknown before the factory actually completes metal extraction. The factory will estimate metal concentrations based on historical buying records, considering features such as the ore type, ore origin, site-reported preliminary samples and so on. Then the factory will decide how much ore to order from each site. This is the Stage 1 solution. The Stage 1 OP is the alloy production problem using the estimated metal concentrations ˆ con , and can be formulated as follows:

After the factory obtains the ores and completes metal extraction, i.e., in Stage 2, the precise metal concentrations/amounts are known. Since the purchased ores are already processed, the factory cannot return ores even if it has bought too much. However, if the obtained metals do not satisfy the requirements, the factory can post-hoc decide to last-minute order more ores at a higher price, for example, (1 + σ k ) cost k per ton from the site k , where σ k ≥ 0 is a non-negative tunable scalar parameter. In this scenario, the penalty function is:

where ◦ is the Hadamard/entrywise product.

With respect to the above penalty function, we are now ready to define the Stage 2 OP:

Note that since the precise metal concentrations con are revealed, the true concentrations are used as the problem parameters instead of the estimated concentrations. The final amount of ores bought from each site, including the ores bought in both Stage 1 and Stage 2, is the Stage 2 solution.

The above formulation is based on the 'soft commitment" modelling approach discussed in Appendix A.1.

The post-hoc regret for the alloy production problem can be explicitly written as:

where x ∗ ( con ) is an optimal solution of the alloy production problem under the true concentrations con . We now show how to compute the relevant gradients as discussed in Section 4 and Appendix B.

Using Equation (21), it is straightforward to compute that the i -th item in vector ∂PReg ( ˆ θ,θ ) ∂x ∗ 2 ∣ ∣ ∣ x ∗

1 and the i -th item in vector ∂PReg ( ˆ θ,θ ) ∂x ∗ 1 ∣ ∣ ∣ x ∗ 2 : ( ∂PReg ( ˆ θ,θ ) ∂x ∗ 2 ∣ ∣ ∣ x ∗ 1 ) i = (1 + σ i ) cost i , ( ∂PReg ( ˆ θ,θ ) ∂x ∗ 1 ∣ ∣ ∣ x ∗ 2 ) i = -σ i cost i .

Now we show how to compute the approximation of the remaining term, ∂x ∗ 2 ∂x ∗ 1 .

Approximation ∂x ∗ 2 ∂x ∗ 1 . We use the same interior-point LP solver to help compute the relevant derivatives. First, the estimated parameters are fed into the LP solver to solve the Stage 1 OP to obtain the Stage 1 optimal solution x ∗ 1 and the corresponding µ , which are used to compute the term ∂x ∗ 1 ∂ ˆ θ . Then the Stage 1 optimal solution x ∗ 1 and the true parameters are fed into the LP solver to solve the Stage 2 OP to obtain the Stage 2 optimal solution x ∗ 2 and the corresponding µ , which are used to compute the term ∂x ∗ 2 ∂x ∗ 1 . Consider the Stage 2 OP in program (20). It is clear that the Stage 2 OP is a MILP, with x ∗ 2 in the objective and x ∗ 1 in h of the constraints. Applying Lemma B.3, we can compute an approximate gradient of the ∂x ∗ 2 ∂x ∗ 1 term.

C.2 Variant of 0-1 Knapsack

The second example, which we call the proxy buyer problem, is a variant of the 0-1 knapsack problem. The unknown parameters appear in both the objective and constraints. This problem, as we shall see, can be handled by our framework, but not by the prior approach by Hu et al. [12], since the problem is inherently discrete and cannot be formulated as LPs.

A proxy buyer is a person who purchases goods for others possibly for a profit. Consider a proxy buyer who is from City A, with a very high cost of living, who regularly travels to City B with a much lower cost of living. Given her regular travels, her friends in City A have asked her to help purchase everyday-life products, which are significantly cheaper in City B, yet the time and transportation cost from City A to City B makes it prohibitive for most people to just go to City B themselves. The traveller commutes between City A and City B once every three months, and has a known and limited capacity cap of goods she could carry and bring back. Before each trip, her friends would make requests for things to buy. For simplicity, one request contains one item. If the buyer brings back the item as requested, her friends will pay her 20% of the price-tag p i of each item i as a courtesy-thankyou. We denote this 'profit' by f i , i.e., f i = 20% p i .

The buyer is popular, and many friends ask her for favours. One day before the buyer leaves for City B, the buyer needs to decide which of her friends' requests to accept, given the limited capacity, and inform them accordingly. The buyer wants to maximize the total amount of courtesy-thankyou money she gets, subject to the hard constraint of the limited suitcase capacity cap . However, the precise price p i of each item i is unknown, due to the uncertainty of the price itself, the volatility of the exchange rate, and the uncertainty of the discount activities of the items. Thus, the 'profit' f i of buying item i is unknown. In addition, the exact size s i of each item i is also estimated. The buyer will estimate the profit, i.e., the prices, and the sizes based on past experiences, considering features such as time-of-year, holiday-or-not, brand and so on. The buyer will decide which requests to accept based on the estimation. This is the Stage 1 solution. The Stage 1 OP is the proxy buyer problem using the estimated sizes ˆ s and estimated profits ˆ f :

After the buyer arrives at City B, the buyer knows the precise price and size of each item. If she cannot carry all the accepted requests, for example, if the packaging for certain items have changed since she last bought them, the buyer will necessarily need to drop some of these requests. The buyer usually feels bad about reneging on a promise to her friends, and treats her friends to a meal as an apology if the request cannot be fulfilled after she promised. For simplicity, we assume that the price of the apology-meal is linear in the profit of the dropped request, since more expensive items are considered 'more important' requests. Here, the linearity factor is independent of the request. That is, if she drops item i , she has to spend σf i amount of money, where σ ≥ 0 is a non-negative tunable scalar parameter. In this scenario, the penalty function is:

We are now ready to define the Stage 2 OP with respect to the above penalty function:

The requests that were finally filled, namely the items that were actually bought by the buyer and brought home to City A, forms the Stage 2 solution.

Then the simplified form of the post-hoc regret for the proxy buyer problem can be written as:

where x ∗ ( f, s ) is an optimal solution of the proxy buyer problem under the true proxy fees f and true sizes s .

Using Equation (24), it is straightforward to compute that the i -th item in vector ∂PReg ( ˆ θ,θ ) ∂x ∗ 2 ∣ ∣ ∣ x ∗ 1 and the i -th item in vector ∂PReg ( ˆ θ,θ ) ∂x ∗ 1 ∣ ∣ ∣ x ∗ 2 : ( ∂PReg ( ˆ θ,θ ) ∂x ∗ 2 ∣ ∣ ∣ x ∗ 1 ) i = ( -1 -σ ) f i , ( ∂PReg ( ˆ θ,θ ) ∂x ∗ 1 ∣ ∣ ∣ x ∗ 2 ) i = σf i

Approximation ∂x ∗ 2 ∂x ∗ 1 . Similar to the computation in Section C.1, we obtain the Stage 1 optimal solution x ∗ 1 , the Stage 2 optimal solution x ∗ 2 , and the corresponding µ from the interior-point LP compute the term ∂x ∗ 2 ∂x ∗ 1 . Consider the Stage 2 OP in program (23), it is clear that the Stage 2 OP is a MILP, with x ∗ 2 in the objective and x ∗ 1 in h of the constraints. Applying Lemma B.3, we can compute an approximate gradient of the ∂x ∗ 2 ∂x ∗ 1 term.

C.3 Nurse Scheduling Problem

Our last example is the nurse scheduling problem (NSP), which can be handled by our framework but not by the prior work of Hu et al. [12] since it is neither a packing LP nor a covering LP.

Consider a large optometry center that needs to assign nurses to shifts per day to meet patients' needs. Every Monday morning, the center collects the nurses' preferences for each shift of the following week. Since nurses may have their own activities and errands during unscheduled shifts, they want to be informed of their schedules as early as possible. After the preferences are collected, on Monday night, the center sets a preliminary shift schedule for the upcoming week based on the estimated number of patients for each shift. Suppose there are n nurses, k days, and s shifts per day, then the number of the total shifts is t = k × s . We formulate the decision variables as a Boolean vector x ∈ { 0 , 1 } d , where d = n × k × s . Let P ∈ { 1 , 2 , 3 , 4 } d represent the value of each nurse's preferences for a particular shift (the higher the number the better), and H ∈ N t represents the number of patients in each shift, which are unknown and need to be predicted. Each nurse i can serve m i patients in one shift. The objective is to maximize the nurses' preferences under a set of constraints: (1) the schedule must satisfy the patient demand, under each shift (2) each nurse must be scheduled for exactly one shift each day (3) no nurse may be scheduled to work a night shift followed immediately by a morning shift. The Stage 1 OP is the NSP using the estimated number of patients ˆ H :

To provide better service to patients, the optometry center has implemented an appointment system that requires patients to schedule an appointment in advance to receive medical care. Reservations for the upcoming week, from Monday to Sunday, close every Sunday evening. At this point, the center knows the precise number of patients for each shift of the next week. The center might adjust the shift schedule to satisfy the actual patient demand or to improve the overall nurse preferences. However, due to the late notice for schedule changes, the nurse's preference may become lower. For example, if a nurse is rescheduled to a shift for which her original preference is 5, now her preference for this shift may become 4 due to the late notice. Besides, a nurse may be more unhappy to be changed to

a low-preference shift. In this scenario, since the nurses' preferences are in { 1 , 2 , 3 , 4 } , the penalty function can be formulated as:

where the i -th item in the penalty function is:

We are now ready to define the Stage 2 OP with respect to the above penalty function:

Then the simplified form of the post-hoc regret for the NSP can be written as:

Using Equation (26), it is straightforward to compute that the i -th item in vector ∂PReg ( ˆ θ,θ ) ∂x ∗ 2 ∣ ∣ ∣ x ∗ 1 and the i -th item in vector ∂PReg ( ˆ θ,θ ) ∂x ∗ 1 ∣ ∣ ∣ x ∗ :

2

Approximation ∂x ∗ 2 ∂x ∗ 1 . Similar to the computation in Section C.1, we obtain the Stage 1 optimal solution x ∗ 1 , the Stage 2 optimal solution x ∗ 2 , and the corresponding µ from the interior-point LP compute the term ∂x ∗ 2 ∂x ∗ 1 . Using the penalty function in Equation (25), the Stage 2 OP can be formulated

as a MILP by adding new variables σ and one more constraint:

Suppose the Stage 2 OP of the NSP can be written as:

Then the standard form of the Stage 2 OP is:

where and b ∈ R nk is an all-ones vector.

It is clear that x ∗ 2 is in the objective and x ∗ 1 is in h of the constraints. Applying Lemma B.3, we can compute an approximate gradient of the ∂x ∗ 2 ∂x ∗ 1 term.

D Hyperparameters for the Experiments

The methods of k -NN, RF, NN, and IntOpt-C as well as 2S have hyperparameters, which we tune via cross-validation: for k -NN, we try k ∈ { 1 , 3 , 5 } ; for RF, we try different numbers of trees in the forest { 10 , 50 , 100 } ; for NN, IntOpt-C, and 2S, we treat the learning rate, epochs and weight decay as hyperparameters.

Tables 7, 8, and 9 show the final hyperparameter choices for the three problems: 1) an alloy production problem, 2) the classic 0-1 knapsack problem, and 3) a nurse roster scheduling problem.

Ridge, k -NN, CART and RF are implemented using scikit-learn [24]. The neural network is implemented using PyTorch [22]. To compute the two stages of optimization at test time for our method, and to compute the optimal solution of an (MI)LP under the true parameters, we use the MILP solver from Gurobi [11].

E Comparisons of the 2S Method and the Prior Differentiation Methods

In this section, we compare the proposed method with prior works [1, 2, 27] that provide ways of differentiating through LPs or LPs with regularization. We conduct comparisons with CvxpyLayer [1] but not OptNet [2] or QPTL [27]. The reason is that the calculation method proposed in QPTL is LP+quadratic regularization using OptNet, and CvxpyLayer is just a conic extension to OptNet. We compared CvxpyLayer [1] with a) no regularization, b) quadratic regularization and c) log-barrier (like our Section 4/Appendix B). The key indicator of its predictive performance is the type of regularization used, with the log-barrier version performing the best, but still slightly worse than our method. We applied CvxpyLayer [1] to the 0-1 knapsack benchmark to compare with our 2S method.

Table 10 reports the mean post-hoc regrets and standard deviations across 10 runs and Table 11 reports the average training times. More precisely, we use it with various regularizations (a. LP with no regularization, b. with quadratic regularization, c. with log-barrier as in our paper) to replace the Section 4/Appendix B gradient calculations. We find that CvxpyLayer [1] never gives better solution quality while 2S is 30%-50% faster.

F Frameworks Comparisons on the Alloy Production Problem

In this section, we further compared the proposed framework with the framework using the differentiable projection idea in [3] on the alloy production benchmark. The idea in [3] is to use the l 2 projection, and we implemented it using CvxpyLayer. The experiment set-up follows that of Table 2: both training and testing use l 2 projection in the second stage, as opposed to solving the second stage optimization problem defined in Section 3. Table 12 shows both the post-hoc regret and training time for l 2 projection. We find that not only is l 2 projection slow, but it has even worse post-hoc regret than the Hu et al. correction [12]. We suspect that this is due to the Hu et al. correction function [12] preserving the direction of the solution vector whereas l 2 projection can change the direction, and that this makes a difference for Alloy Production. In any case, this experiment confirms again that our Two-Stage framework has better post-hoc regret than a framework based on differentiable projections, reinforcing the main message of our paper.

G Experiments on the 0-1 Knapsack Problem with Large Penalty Factors

Table 13 reports the mean post-hoc regrets and standard deviations across 10 runs for each approach on the 0-1 knapsack problem with large penalty factors (penalty factors ≥ 1 ). With more data, we can make further analysis of the performance of the proposed 2S method. Observing Tables 4 and 13, we can see that the trend, in terms of the difference between 2S and other methods, first decreases, then increases, as the penalty factor increases. The trend in Tables 4 and 13 is identical to the trend in Table 3. We can explain this phenomenon as follows.

First, when the penalty factor is small, the rational behavior for the buyer is to just take every order, and only decide which orders to drop when the true parameters are revealed (at close to no cost). 2S identifies and exploits this behavior for small penalties, while classic regression methods are agnostic to this possible tactic. Thus, the advantage of 2S compared to classic regression methods is large in the small penalty case.

Second, when the penalty factor is large, 2S will analogously learn to be conservative, such that the first stage solution likely remains feasible under the true parameters, in order to avoid the necessary (and high) penalty due to having to change to a feasible solution. Again, classic regression methods will be agnostic to this possible tactic, leading to a large advantage of 2S over the classic methods.

Table 5 only has the increasing trend from the large penalty, since it is neither a covering nor a packing program, and so there is no analogous tactic/exploitation for the small penalty.

H Runtimes for the Experiments

In this paper, all models are trained with Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz processors. Table 14 shows the average runtime across 10 simulations for different optimization problems. Since the testing time of different approaches is quite similar, here, the runtime refers to only the training time of the prediction model and does not include the testing time. At training time, only the proposed 2S method and IntOpt-C solve the LP. Training for the usual NN does not involve the LP at all, and so training is much faster (but gives worse results).

Since IntOpt-C cannot handle the variant of the 0-1 knapsack problem and the NSP, we only report the runtime of IntOpt-C for the alloy production problem.

Since the provided code of CombOptNet is only available for the 0-1 knapsack problem, we only report the runtime of CombOptNet for the 0-1 knapsack problem. As Table 14 shows, CombOptNet is drastically slower than the proposed 2S method.

In the alloy production problem, the runtimes of the proposed 2S method are a little larger than that of IntOpt-C. The reason is that 2S needs to solve two LPs when training while IntOpt-C only needs to solve one. But in the alloy production problem, the unknown parameters are on the left hand side of the inequality constraints and the gradient computation includes matrix computation, which is also time-consuming. Thus, the runtimes of the 2S method are larger but not twice as large as that of the IntOpt-C method.

In both the alloy production problem and the variant of the 0-1 knapsack problem, the runtimes of the 2S method are much better than RF.

The runtime of the 2S method is large in the NSP. This is because we use the formulation where each decision variable corresponds to whether a specific nurse is assigned to a specific day and a specific shift. Thus, the number of the decision variable of the relaxed LP is large and the LP takes more time to solve.

Table 14: Average runtime (in seconds) for the alloy production, 0-1 knapsack, and nurse scheduling problems.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPredict+Optimize is an emerging paradigm that lies in the intersection of classical optimization (particularly mixed integer programming) and machine learning. Specifically, it considers the setting where a parameterized optimization problem:\n$$ x^{\\star}(\\theta) = \\operatorname*{argmin}_{x} f(x;\\theta) \\text{ subject to } C(x;\\theta) $$\nmust be solved yet the parameters $\\theta$ are unknown. Here $C(x;\\theta)$ represents constraints that must be respected, which could be equality, inequality, or inclusion. In the last 4--5 years this topic has seen heightened interest. Most approaches, including the one proposed in this paper, assume access to observed features $A$ that correlate with $\\theta$, and aim to learn a predictor $f(A)$ so that $\\hat{\\theta} := f(A) \\approx \\theta$. Then, a problem of the form above is solved with $\\hat{\\theta}$ in place of $\\theta$ to yield a predicted minimizer $x^{\\star}(\\hat{\\theta})$. Hopefully,  $x^{\\star}(\\hat{\\theta}) \\approx x^{\\star}(\\theta)$.\n\n\nThe paper in question is one of the first to actively consider the setting where the constraints depend on $\\theta$; prior work mostly considered constraints of the form $C(x)$. The method proposed is interesting, intuitive and general. They start from the observation that any scheme handling parametrized constraints must be allowed to make post-hoc adjustments to $x^{\\star}(\\hat{\\theta})$  after the true parameters $\\theta$ are revealed, as $x^{\\star}(\\hat{\\theta})$ might not be feasible. Thus, they propose to solve a second, penalized problem:\n\n$$ x_2^{\\star} = \\operatorname{argmin}_x f(x;\\theta) + r(x^{\\star}(\\hat{\\theta}), x) \\text{ subject to } C(x;\\theta) $$\n\nwhere $r(\\bullet, \\bullet)$ penalizes the discrepancy between the stage 1 solution $x^{\\star}(\\hat{\\theta})$ and the stage 2 solution $x_2^{\\star}$.  \n\nThe bulk of the paper is devoted to motivating, introducing and implementing this proposed new method. Numerical experiments illustrate that this new method performs well. Finally, there is a little bit of theory (contained in the appendix) relating this new approach to older approaches to predict+optimize with parametrized constraint sets.\n\nSTRENGTHS:\n- I agree with the assertion in the appendix that the proposed approach is _simple and powerful_. I think this is an elegant solution to an important problem.\n- The paper is well-written; I did not find any typos or unclear sentences. The running example of stocking a store is useful.\n- I appreciated the choice of experiments---they are closer to \"real-world\" problems than many experiments I have encountered in Neurips papers.\n- The Appendices are thoughtfully written, addressing many potential reader's questions and providing several nice extensions of the model at hand.\n- The framework proposed is indeed original, but I think some additional comparisons with existing literature is needed, see \"weaknesses\".\n\nWEAKNESSES:\n- With regard to novelty, I think the authors should compare their work to various differentiable quadratic program (QP) solvers. For example, in [1] formulas for the derivatives of $x^{\\star}$ with respect to constraint (_e.g._ $\\partial x^{\\star}/\\partial A$, to reference the notation of your equation (5)) are given. Note that the MILP in (4) can be turned into a QP by using a quadratic regularizer (as done in [2]) instead of the logarithmic regularizer of Mandi & Guns.\n\n - I found the use of the name \"Two-stage Predict+Optimize\" for your proposed framework a bit confusing. In much of the Predict+Optimize literature, the two-stage approach refers to the approach of training a predictor $f(A)$ to minimize the mean square error $\\|f(A) - \\theta\\|^2$ (see for example [3]) I strongly suggest you add a remark  discussing this towards the beginning of your paper.\n\n- I think the benchmarks considered are a little weak. For example, all 5 classical regression methods are slight variations on the \"two-stage approach\" mentioned above. You should include some SOTA approaches for (one-stage) predict+optimize, e.g. training with the SPO+ loss, Perturbed Optimization [7], blackbox backpropagation [8] etc or justify why such approaches are incompatible with the proposed Two-Stage Predict+Optimize framework. The PyEPO benchmarking suite [4] could be useful.\n\n- I'd like to see more discussion on the computational cost of your proposed approach. Given that it is a tri-level (!) problem, I think this is something to address. For the experiments, could you indicate the dimension of $x$ and the number of constraints, as well as the time required to train using the 2S and IntOpt-C approaches? It might be nice to add a remark about possible approaches to scaling your proposed framework, see [5,6].\n\n[1] _Optnet: Differentiable optimization as a layer in neural networks_ Amos & Kolter, 2017.\n\n[2] _Melding the data-decisions pipeline: Decision focused learning for combinatorial optimization_ Wilder _et al_, 2019\n\n[3] _Interior point solving for LP-based prediction+optimization_ Mandi & Guns, 2020\n\n[4] _PyEPO: A PyTorch-based end-to-end predict-then-optimize library for linear and integer programming_ Tang & Khalil 2022\n\n[5] _Faster predict-and-optimize using three-operator splitting_ McKenzie _et al_, 2023\n\n[6] _Backpropagation through combinatorial solvers: Identity with projection works_, Sahoo _et al_, 2022\n\n[7] _Learning with differentiable perturbed optimizers_ Berthet _et al_, 2020\n\n[8] _Differentiation of blackbox combinatorial solvers_ Vlastelica _et al_, 2019\n\nQUESTIONS:\n- The two-stage nature of your proposed approach reminds me a lot of Model-agnostic Meta-Learning (MAML) [2]. Moreover, I think there is an interesting analogy between the connection between your work and that of Hu et al [3] and the connection between MAML and later implicit MAML (I-MAML) works [4]. I think the appeal of your work could be broadened by adding a discussion of this connection.\n\n- An interesting variant of predict+optimize is the setting where at train time the true cost vectors ($c$ in the notation of your equation (4)) are never revealed, rather only the true solutions $x^{\\star}$ are accessible (see [1] for a discussion on this). Do you have any thoughts on how to extend your approach to this setting?\n\n[1]  _Faster predict-and-optimize using three-operator splitting_ McKenzie _et al_, 2023\n\n[2] _Model-agnostic meta-learning for fast adaptation of deep networks_ Finn _et al_, 2017\n\n[3] _Predict+Optimize for packing and covering LPs with unknown parameters in constraints_ Hu _et al_ 2022\n\n[4] _Meta-learning with implicit gradients_ Rajeswaran _et al_, 2019.\n\n-----\n\nAfter discussion with the authors I've raised my score 6--> 7\n\n-------\n\nLIMITATIONS:\nAs mentioned above, I think the computational complexity of the proposed approach could be a limitation. This should be discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel '2-stage' framework for Predict+Optimize with uncertain parameters in the constraints. In the first \"stage\" a soft commitment is made based on the predictions, and in the second, the commitment is updated based on updated information in such a way that the objective value plus a penalty for deviating from the commitment is minimized. This generalizes the framework of Hu et al. [9]\n\nSTRENGTHS:\n1. Great idea: I find the new framework simple and sensible, I think this is the right framework for Predict+Optimize with unknown constraints. The biggest plus is that it allows for improving soft commitments in which constraints aren't violated, which is not something that Hu et al. allow for, but makes a lot of practical sense. It also removes the need to create a differentiable projection.\n1. Clarity: The paper is well written and covers most bases (the tables could be bigger, though!)\n\nWEAKNESSES:\n1. Requires linear penalties: The penalties have to be linear in the decision variables. Hu et al. [9] doesn't require that.\n1. Is slower than Hu et al.: Because Hu et al. only require running a differentiable projection, which can be quite cheap, it is cheaper than 2S (Appendix E confirms this).\n1. Generalization of Hu et al.'s framework: While Hu et al. propose a specific projection that is limited to packing and constraint problems, there has been work in the literature about differential projections for enforcing feasibility constraints generally [A] and efficiently [B] that could be used instead. It would have been great to compare against those in the experiments.\n1. I understand that the space is limited, but a lot of important information, like the description of the experiments and runtimes are in the Appendix. It would have been useful to have summaries in the main text...\n\n_References:_\n\n[A] Chen, Bingqing, et al. \"Enforcing policy feasibility constraints through differentiable projection for energy optimization.\" Proceedings of the Twelfth ACM International Conference on Future Energy Systems. 2021.\n\n[B] Sanket, Shah, et al. \"Solving online threat screening games using constrained action space reinforcement learning.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 02. 2020.\n\nQUESTIONS:\n1. In Table 3, the difference between 2S and the other methods *decreases* as the penalty factor increases. Why is that? I would have expected the opposite (as in Tables 2 and 4).\n1. On line 355 the paper says, \"On the other hand, the advantage of our 2S method over other approaches actually becomes more significant as the capacity increases, demonstrating the superior accuracy of our approach.\" However, the difference between 2S and other approaches for the lowest penalty factor actually decreases in absolute value?\n\nLIMITATIONS:\nThe limitations are not described, despite having responded 'yes'. The fact that the paper does not acknowledge any of these limitations highlighted in the `weaknesses' section is worrying.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a framework for learning latent variables in optimization problems that appear both in the constraints and objective. In this setting, the user is given features and asked to provide a solution to an optimization problem where the objective and constraints of the optimization problem are partially observed and related to the features. Additionally, the user can optimally modify the solution once the true parameters are known at a cost based on the change. The overall goal is to ensure that the total regret is low where the regret is the total value of the final solution after fixing minus the fixing cost and minus the objective value of the optimal solution in hindsight. The authors propose that the second stage solution should be considered the output of an optimization problem which is given the first stage solution as input and then backpropagates through both optimization problems to update the weights of the predictive model predicting parameters for the first stage optimization problem. The authors propose differentiating through continuous relaxations of these optimization problems using previous work that differentiates through iterates of an interior point method.\n\nThe authors evaluate their approach on several settings to demonstrate improved predictive performance over the investigates baselines.\n\nSTRENGTHS:\nThe main strength of the work is that it considers penalizing the recourse using a flexible optimization problem rather than having a domain-dependent method of doing so as was done in previous work for packing and covering. Additionally, the paper itself is easy to read and\n\nWEAKNESSES:\nGiven that the main contribution of this work is that the framework has good empirical performance, it would be good to strengthen the experiments by evaluating against relevant baselines.\n\nThe work does not compare against relevant baselines and claiming generality to MILP to discount several differentiable continuous optimization baselines when in practice, the proposed approach simply relaxes the integrality constraints to consider differentiating through a continuous LP. Given that this approach considers differentiation of MILP with respect to the constraints as simply differentiating the constraints of the LP relaxation, the authors should evaluate against methods which can differentiate through constraints of an LP which includes cvxpylayers [1], and OptNet with quadratic regularization to differentiate through LP [2,3].\n\nAdditionally, the previous CombOptNet work [19 in the paper] which is formulated explicitly for learning constraints in combinatorial settings, and whose datasets are used for two of the three settings, is not compared against.\n\n\n[1] Agrawal, Akshay, et al. \"Differentiable convex optimization layers.\" Advances in neural information processing systems 32 (2019).\n[2] Amos, Brandon, and J. Zico Kolter. \"Optnet: Differentiable optimization as a layer in neural networks.\" International Conference on Machine Learning. PMLR, 2017.\n[3] Wilder, Bryan, Bistra Dilkina, and Milind Tambe. \"Melding the data-decisions pipeline: Decision-focused learning for combinatorial optimization.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.\n\nQUESTIONS:\nAre there any specific reasons that previous work that learns constraints for LP or comboptnet are not applicable to the investigated settings?\n\nAre there any specific components of this method which are specialized to handle integrality that cannot be handled by simply applying previous approaches for differentiating through continuous problems?\n\nIt might be helpful to compare the gradients of this approach on a subjective level as well. In equation (3) are the gradients from one component relatively large compared to gradients from the other? How do they differ overall from previous work?\n\nLIMITATIONS:\nRelevant limitations are addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors develop a two-stage predict-and-optimize approach. There is a recent paper of Hu et al. [9] that extends the predict-and-optimize framework to having unknown parameters in the constraints. The authors argue that the Hu et al. approach, which requires defining both a \"correction function\" and a \"penalty function\", could be simplified by solving a single optimization problem. The authors show how to run the Mandi and Guns [15] solver to get the answer to this optimization problem. They then demonstrate a wide range of examples for their approach, including some real-world examples.\n\nSTRENGTHS:\nThe authors have written a clear and easy-to-understand paper. I have not checked the proofs with great detail, but the mathematics looks very reasonable. The authors are getting impressive results on the examples that they explore.\n\nThe novelty is that the authors are combining the work of Hu et al. with the work of Mandi and Guns.\n\nThis is a fairly important problem within the Predict+Optimize framework and getting good solutions to these problems is important, so the significance is that this approach could hopefully be used. For the examples that the authors are demonstrating, their approach is clearly better to Hu et al.\n\nWEAKNESSES:\nAs a point of order, it's frustrating that papers sometimes cite references so narrowly. I understand this a bit better when a paper is very mathematical, for instance building a proof based on another proof. But in this paper, the proofs are fairly algorithmic/arithmetic and involve well-known components. I haven't tried to discover who the authors are, but I notice that that 7 of the 23 references [1, 2, 3, 7, 14, 15, 16] are to the same group of close collaborators. Then, another 11 of the 23 references [6, 8, 11, 12, 13, 17, 18, 19, 20, 21, 22] are to software packages, test problems and text books. This leaves only 5 of 23 references which are to papers that are somehow intellectually related but not from a single group of collaborators (of these 5 remaining papers, two are to Elmachtoub and co-workers, two are to Hu and co-workers, and one is to Wilder et al.).\n\nI am anxious about how many of the papers (7) refer to one closely-related group when only 5 of the cited papers are to work outside the group (I discard the 11 software packages / textbooks / test instances because these don't require the same work in linking ideas). Work that singularly focuses on just a single small group of collaborators sometimes neglects to take the broad perspective that often characterizes good science.\n\nThe reason that I'm marking \"Fair\" in the \"Presentation\" tick mark is this very limited interaction with the literature. Other than this problem, the paper is quite clear.\n\n****\n\nAt least as the paper is worded, there is a Hu et al. [9] framework that has been introduced that matches the authors setting, but everything would be better if only we would \"adapt the approach of Mandi and Guns [15]\" [Section 1, line 79]. For example, most of Section 4 (lines 211 - 268) discuss an adaptation of Mandi and Guns [15]. The point of the paper often seems to be that if only Hu et al. had read Mandi and Guns a bit more carefully, Hu et al. would have done things differently.\n\nOverall, the current paper reads like a \"letter to the Editor\" objecting to elements of the work of Hu et al. [9] and trying to correct it. The Hu et al. framework is described in detail (lines 40-82 in the introduction and also Section 3 seem dedicated to Hu et al. and why what they're doing is wrong). Then, Section 4 proposes fixes for the Hu et al. framework using the Mandi and Guns framework.\n\nThe impression this leaves is that the authors' work is a bit incremental. As the authors acknowledge, their framework and the framework of Hu et al. are \"mathematically equivalent in expressiveness\". So the only difference is that the Hu et al. [9] paper develops a \"correction function\" and a \"penalty function\" whereas these authors solve an additional optimization problem (this optimization problem seems very similar to the one already proposed in Mandi and Guns).\n\nThe reason that I'm marking the \"Contribution\" as \"Fair\" is that I can't see anything except modifying Hu et al. to work more like Mandi and Guns.\n\n****\n\nI disagree with the authors that their approach \"should be the canonical framework for the Predict+Optimize setting\". The authors have a lot of impressive results, especially in comparison to Hu et al., but there are times where solving an optimization problem may take too long and it may be better to have a function that is quick-to-evaluate. \n\nThis small note (that the authors are claiming that it's always better to solve an optimization problem) is the only reason that I'm marking the \"Soundness\" as \"Good\" rather than \"Excellent\".\n\nQUESTIONS:\nI've tried to be pretty specific as to what are the weaknesses of the paper so that the authors can correct as relevant. Please take the \"Weaknesses\" section as a set of questions where I'm happy to be corrected.\n\nLIMITATIONS:\nThis part of the paper is fine.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}