SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our approach, inspired by the work of Mandi and Guns [18], is to define a new surrogate loss function  $\widetilde{PReg}$  that is differentiable and produces informative gradients. Prior works related to learning unknowns in constraints [1, 2, 27] give ways of differentiating through LPs or LPs with regularizations. These works can be used in place of the proposed approach. However, experiments in Appendix E demonstrate that the proposed approach performs at least as well in post-hoc regret performance as the others, while being faster. We show the construction of the proposed approach below, and note that it does not have a simple closed form. Nonetheless, we can compute its gradients.
The rest of the section assumes that both stages of optimization are expressible as a MILP in the following standard form:
$$
x ^ {*} = \underset {x} {\arg \min } c ^ {\top} x \text {s . t .} A x = b, G x \geq h, x \geq 0, x _ {S} \in \mathbb {Z} \tag {4}
$$
with decision variables  $x \in \mathbb{R}^d$  and problem parameters  $c \in \mathbb{R}^d$ ,  $A \in \mathbb{R}^{p \times d}$ ,  $b \in \mathbb{R}^p$ ,  $G \in \mathbb{R}^{q \times d}$ ,  $h \in \mathbb{R}^q$ . The subset of indices  $S$  denotes the set of variables that are under integrality constraints. Since the unknown parameters may appear in any combination of  $c, A, b, G$  and  $h$  in the Stage 1 optimization for  $x_1^*$ , the surrogate loss function construction needs computable and informative gradients for all of  $\frac{\partial x^*}{\partial c}, \frac{\partial x^*}{\partial A}, \frac{\partial x^*}{\partial b}, \frac{\partial x^*}{\partial G}$  and  $\frac{\partial x^*}{\partial h}$ .
We follow the interior-point based approach of Mandi and Guns [18], used also by Hu et al. [12]. Consider the following convex relaxation of (4), for a fixed value of  $\mu \geq 0$ :
$$
x ^ {*} = \underset {x, s} {\arg \min } c ^ {\top} x - \mu \sum_ {i = 1} ^ {d} \ln \left(x _ {i}\right) - \mu \sum_ {i = 1} ^ {q} \ln \left(s _ {i}\right) \text {s . t .} A x = b, G x - s = h \tag {5}
$$
This is a relaxation of (4) by i) dropping all integrality constraints, ii) introducing slack variables  $s \geq 0$  to turn  $Gx \geq h$  into  $Gx - s = h$  and iii) replacing both the  $x \geq 0$  and  $s \geq 0$  constraints with the logarithm barrier terms in the objective, with multiplier  $\mu \geq 0$ . The observation is that the gradients  $\frac{\partial x}{\partial c}, \frac{\partial x}{\partial A}, \frac{\partial x}{\partial b}, \frac{\partial x}{\partial G}$  and  $\frac{\partial x}{\partial h}$  for (5) are all well-defined, computable and informative for a fixed value of  $\mu \geq 0$ : Slater's condition holds for (5), and so the KKT conditions must be satisfied at the optimum  $(x^{*}, s^{*})$  of (5). We can thus compute all the relevant gradients via differentiating the KKT conditions, using the implicit function theorem. We give all the calculation details in Appendix B.
Given the above observation, we then aim to construct the surrogate loss function by replacing the  $x_{1}^{*}$  and  $x_{2}^{*}$ , which are supposed to solved using MILP (4), with a)  $\widetilde{x}_{1}$  that is solved from program (5) relaxation of the Stage 1 optimization problem, using the predicted parameters  $\hat{\theta}$  and b)  $\widetilde{x}_{2}$  that is solved from the program (5) relaxed version of Stage 2 optimization, using  $\widetilde{x}_{1}$  and the true parameters  $\theta$ . The only remaining question then, is, which values of  $\mu$  do we use for the two relaxed problems?
Given a MILP in the form of (4), the interior-point based solver of Mandi and Guns [18] generates and solves (5) for a sequence of decreasing non-negative  $\mu$ , with a termination condition that  $\mu$  cannot be smaller than some cutoff value. Thus, we simply choose the cutoff value to use as “ $\mu$ ” in (5), which then completes the definition of the surrogate loss  $PReg$ .
Alge. rically, we train the neural network on the surrogate loss  $\widetilde{PReg}$  as follows: given predicted parameters, we run the Mandi and Guns solver to get the optimal solution  $(x^{*},s^{*})$  for the final value of  $\mu$ . We can then compute the gradient of the output solution with respect to any of the problem parameters using the calculations in Appendix B, combined with backpropagation, to yield  $\frac{\mathrm{d}PReg}{\mathrm{d}w_e}$  according to Equation (3).
In Appendix C, we give three example application scenarios, along with their penalty functions, that our training approach can handle. These problems are: a) an alloy production problem, for factory trying to source ores under uncertainty in chemical compositions in the raw materials, b) a variant of the classic 0-1 knapsack with unknown weights and rewards, and c) a nurse roster scheduling problem with unknown patient load. We show explicitly in Appendix C how both stages of optimization
Table 1: Relevant problem sizes of the three benchmarks.
Problem name	Brass alloy production	Titanium-alloy production	0-1 knapsack	Nurse scheduling problem
Dimension of x	10	10	10	315
Number of constraints	12	14	21	846
Number of unknown parameters	20	40	10	21
Number of features (per parameter)	4096	4096	4096	8
can be formulated as MILPs for these applications, and apply the Appendix B calculations to yield gradient computation formulas for the surrogate loss  $\widetilde{PReg}$  for these problems.
A limitation of our approach is the requirement that both stages must be expressible as MILPs, constraining the optimization objectives to be linear in the MILP decision variables. This contrasts the Hu et al. framework [12] which handles non-linear penalties. We point out that even MILPs can handle some non-linearity by using extra decision variables: for example, the absolute-value function. Moreover, the Appendix B gradient calculations can be adapted to handle general differentiable non-linear objectives. We present only MILPs as a main overarching application for this paper because of their widespread use in discrete optimization, with readily available solvers.
We evaluate the proposed method $^2$  on three benchmarks described in Section 4 and Appendix C. Table 1 reports the relevant benchmark problem sizes. We compare our method (2S) with the state of the art Predict+Optimize method, IntOpt-C [12], and 5 classical regression methods [9]: ridge regression (Ridge),  $k$ -nearest neighbors ( $k$ -NN), classification and regression tree (CART), random forest (RF), and neural network (NN). All of these methods use their classic loss function to train the prediction models. At test time, to ensure the feasibility of the solutions when computing the post-hoc regret, we perform Stage 2 optimization on the estimated solutions for these classical regression methods before evaluating the final solution. Additionally, CombOptNet [23] is a different method focusing on learning unknowns in constraints, but with a different goal and loss function. We experimentally compare our proposed method with CombOptNet on the 0-1 knapsack benchmark—the only with available CombOptNet code. We also present a qualitative comparison in Section 6.
In the following experiments, we will need to take care to distinguish two-stage optimization as a training technique (Section 4) and as an evaluation framework (Section 3). We will denote our training method as "2S" in the experiments, and when we say "Two-Stage Predict+Optimize" framework, we always mean it as an evaluation framework. 2S is always evaluated according to the Two-Stage Predict+Optimize framework. As explained above, we will also evaluate all the classical training methods using the Two-Stage Predict+Optimize framework. For our comparison with the prior work of Hu et al. [12], we will also distinguish their training method and evaluation framework. The name "IntOpt-C" always refers to their training method using their correction function. We will simply call their evaluation framework the "Hu et al. framework" or with similar phrasing (see Section 2 to recall details). IntOpt-C will sometimes be evaluated using our new Two-Stage Predict+Optimize framework, and sometimes the prior framework of Hu et al. [12] using their correction function.
The methods of  $k$ -NN, RF, NN, and IntOpt-C as well as 2S have hyperparameters, which we tune via cross-validation. We include the hyperparameter types and chosen values in Appendix D. In the main paper we only report the prediction performances. See Appendix H for runtime comparisons.
Alloy Production Problem The alloy production problem is a covering LP, see Appendix C.1 for the practical motivation and LP model. Since Hu et al. [12] also experimented on this problem, we use it to compare our 2S method with IntOpt-C [12], using the same dataset and experimental setting.
We conduct experiments on the production of two real alloys: brass and an alloy blend for strengthening Titanium. For brass, 2 kinds of metal materials, Cu and Zn, are required [14]. The blend of the two materials are, proportionally,  $req = [627.54, 369.72]$ . For the titanium-strengthening alloy, 4 kinds of metal materials, C, Al, V, and Fe, are required [15]. The blend of the four materials are proportional to  $req = [0.8, 60, 40, 2.5]$ . We use the same real data as that used in IntOpt-C [12] as numerical values in our experiment instances. In this dataset [23], each unknown metal concentration
$^{2}$ Our implementation is available at https://github.com/Elizabethxhu/NeurIPS_Two_Stage_Predict-Optimize
Table 2: Comparison of the Two-Stage Predict+Optimize framework and the Hu et al. framework on the alloy production problem.
PReg	Two-Stage Predict +Optimize Framework	Hu et al. Framework
Alloy	Penalty factor
Brass	0.25±0.015	43.87±2.73	68.16±6.26
0.5±0.015	65.71±4.81	82.91±5.45
1±0.015	88.75±5.91	107.64±6.85
2±0.015	123.90±6.84	150.47±12.99
4±0.015	161.86±8.49	178.69±10.09
8±0.015	194.06±13.09	206.84±12.51
Titanium-alloy	0.25±0.015	4.52±0.47	6.45±0.81
0.5±0.015	6.03±0.62	7.90±0.56
1±0.015	8.58±0.74	10.73±0.81
2±0.015	12.17±1.24	14.17±1.31
4±0.015	16.10±1.06	17.48±0.99
8±0.015	19.69±0.91	21.08±1.91
Table 3: Mean post-hoc regrets and standard deviations for the alloy production problem using the Two-Stage Predict+Optimize framework.
PReg	2S	IntOpt-C	Ridge	k-NN	CART	RF	NN	TOV
Alloy	Penalty factor
Brass	0.25±0.015	43.87±2.73	45.27±3.35	60.80±2.55	63.32±4.39	77.80±6.37	60.85±2.35	64.96±3.58	312.02±6.94
0.5±0.015	65.71±4.81	67.69±4.25	71.12±3.48	74.36±5.69	93.67±7.03	70.86±3.29	74.32±2.90
1±0.015	88.75±5.91	89.83±4.79	91.82±6.41	96.52±8.90	125.50±9.49	90.97±6.14	93.12±4.24
2±0.015	123.90±6.84	125.46±9.26	133.18±12.98	140.77±16.02	189.12±16.10	131.12±12.48	130.67±10.52
4±0.015	161.86±8.49	164.94±10.33	215.87±26.54	229.22±30.74	316.31±30.95	211.40±25.56	205.76±24.33
8±0.015	194.06±13.09	200.42±8.51	381.30±53.75	406.19±60.42	570.75±61.42	372.01±51.82	355.96±52.25
Titanium-alloy	0.25±0.015	4.52±0.47	4.72±0.58	6.43±0.39	6.13±0.34	7.07±0.45	5.75±0.48	6.56±0.59	30.27±0.54
0.5±0.015	6.03±0.62	6.23±0.64	7.71±0.45	7.27±0.39	8.57±0.45	6.76±0.55	7.38±0.67
1±0.015	8.58±0.74	8.71±0.95	10.26±0.62	9.55±0.52	11.57±0.52	8.76±0.72	9.03±0.84
2±0.015	12.17±1.24	12.31±1.31	15.37±1.03	14.11±0.84	17.57±0.80	12.78±1.11	12.34±1.21
4±0.015	16.10±1.06	16.97±1.70	25.60±1.89	23.24±1.56	29.57±1.53	20.81±1.93	18.95±2.00
8±0.015	19.69±0.91	20.80±1.74	46.04±3.65	41.49±3.03	53.57±3.10	36.88±3.63	32.16±3.60
is related to 4096 features. For experiments on both alloys, 350 instances are used for training and 150 instances for testing the model performance. For NN, IntOpt-C, and 2S, we use a 5-layer fully connected network with 512 neurons per hidden layer.
In the penalty function described in Appendix C.1, we need to choose a penalty factor/multiplier for each supplier. We conduct experiments on 6 types of penalty factor  $(\sigma)$  settings: 6 vectors where each entry is i.i.d. uniformly sampled from  $[0.25 \pm 0.015]$ ,  $[0.5 \pm 0.015]$ ,  $[1.0 \pm 0.015]$ ,  $[2.0 \pm 0.015]$ ,  $[4.0 \pm 0.015]$ , and  $[8.0 \pm 0.015]$  respectively. This random sampling of  $\sigma$  ensures that the penalty factor for each supplier is different, but remains roughly on the same scale.
The first experiment we run compares 2S+Two-Stage Predict+Optimize framework with IntOpt-C+Hu et al. framework. Specifically, we compare a) using 2S for training and evaluating using the Two-Stage Predict+Optimize framework in Section 3, versus b) using IntOpt-C for training and evaluating using the same correction function from training, according to the Hu et al. framework described in Section 2. Table 2 compares the mean post-hoc regret and standard deviations for the alloy production problem for the two different frameworks. The table shows that Two-Stage Predict+Optimize framework always achieves smaller mean post-hoc regret than the Hu et al. framework. Compared with the Hu et al. framework, our framework obtains  $6.18\% -35.63\%$  smaller mean post-hoc regret in brass production, and  $6.59\% -29.89\%$  smaller mean post-hoc regret in titanium-alloy production.
We present a further comparison in Appendix F with a variant of the Hu et al. framework—the  $\ell_2$  projection idea in [3], which performs even worse than the Hu et al. framework.
The second experiment compares various training approaches all evaluated under the Two-Stage Predict+Optimize framework. That is, the models are trained differently, but at test time, we always use Stage 2 optimization to give a final solution and evaluate post-hoc regret on it. Table 3 reports the mean post-hoc regrets and standard deviations across 10 runs for each training method on the alloy production problem. The table shows that our method, 2S, achieves the best performance, compared with IntOpt-C achieving the second best performance, beating all the classical training approaches. Compared with IntOpt-C, 2S obtains  $1.20\% - 3.18\%$  smaller mean post-hoc regrets in brass production, and  $1.18\% - 5.33\%$  smaller mean post-hoc regret in titanium-alloy production. Compared with the classical approaches, the improvements are much more significant. 2S obtains at least  $2.44\% - 45.48\%$  smaller mean post-hoc regrets in brass production, and at least  $1.39\% - 38.78\%$  smaller mean post-hoc regret in titanium-alloy production. The average True Optimal Values (TOV) are reported in the last column of Table 3 for reference, although the reader should take care to not over-interpret the ratio
Table 4: Mean post-hoc regrets and standard deviations for 0-1 knapsack problem using the Two-Stage Predict+Optimize framework.
PReg	Penalty factor	2S	CombOptNet	Ridge	k-NN	CART	RF	NN	TOV
100	0.21	1.26±0.01	9.45±0.19	9.46±0.19	9.38±0.21	8.67±0.13	9.50±0.26	9.81±0.20	29.68±0.14
0.25	6.28±0.05	9.60±0.22	9.77±0.19	9.70±0.19	9.19±0.12	9.82±0.27	10.11±0.20
0.3	9.22±0.10	10.45±0.34	10.16±0.19	10.10±0.18	9.85±0.11	10.22±0.28	10.49±0.21
150	0.21	0.73±0.01	8.90±8.97	9.12±0.22	8.91±0.20	8.46±0.18	9.20±0.27	9.66±0.47	40.23±0.19
0.25	3.64±0.04	9.11±9.41	9.40±0.21	9.19±0.20	8.88±0.17	9.47±0.26	9.92±0.43
0.3	7.27±0.06	9.34±9.38	9.76±0.22	9.53±0.19	9.41±0.17	9.81±0.24	10.23±0.38
200	0.21	0.33±0.01	15.16±0.21	6.57±0.21	6.38±0.29	6.26±0.21	6.59±0.23	7.08±0.95	48.13±0.24
0.25	1.67±0.03	15.20±0.27	6.80±0.20	6.62±0.29	6.57±0.19	6.82±0.21	7.27±0.88
0.3	3.33±0.06	15.25±0.22	7.09±0.19	6.91±0.28	6.95±0.19	7.10±0.18	7.52±0.80
250	0.21	0.07±0.00	20.42±0.25	2.39±0.22	2.18±0.20	2.45±0.20	2.34±0.32	2.70±1.34	53.43±0.26
0.25	0.34±0.02	20.47±0.13	2.53±0.21	2.34±0.19	2.60±0.19	2.49±0.30	2.82±1.26
0.3	0.69±0.04	20.54±0.32	2.71±0.20	2.54±0.18	2.79±0.18	2.67±0.28	2.97±1.16
Table 5: Mean post-hoc regrets and standard deviations for the NSP using the Two-Stage Predict+Optimize framework.
Penalty factor	2S	Ridge	k-NN	CART	RF	NN	TOV
0.25±0.015	3.94±1.91	6.45±4.68	15.20±5.76	26.20±8.96	19.47±7.19	4.27±2.22
0.5±0.015	6.92±2.26	12.68±9.35	30.29±11.53	52.47±17.96	38.93±14.42	8.20±4.40
1.0±0.015	13.12±3.15	25.12±18.71	60.43±23.07	105.01±36.00	77.86±28.99	16.00±8.78	190.21±26.17
2.0±0.015	25.04±9.29	49.95±37.39	120.62±46.08	210.02±72.06	155.64±58.06	31.51±17.40
4.0±0.015	33.29±9.53	99.61±74.78	241.01±92.14	420.04±144.18	311.19±116.23	62.52±34.64
8.0±0.015	46.72±14.80	198.91±149.54	481.79±184.27	840.10±288.45	622.32±232.56	124.54±69.14
between the post-hoc regret and the true optimal value, since the post-hoc regret also includes the penalty term which increases with the penalty factors.
0-1 knapsack In the second example, we showcase our framework on a packing integer programming problem, a variant of the 0-1 knapsack problem, with unknown item prices  $p_i$  and sizes  $s_i$ . See Appendix C.2 for details of an application in running a "proxy buyer" business. Here, the unknown parameters appear in both the objective and constraints. The proposed 2S method can handle this MILP straightforwardly, but the IntOpt-C method cannot be applied. Thus, we only experiment with the Two-Stage Predict+Optimize framework for evaluation, and compare the proposed 2S method with classical approaches and CombOptNet. Again, all approaches are evaluated at test time using the Stage 2 optimization to yield the final solution, on which the post-hoc regret is computed.
The MILP formulation of the two stages and the penalty function are described in Appendix C.2. We use the dataset of Paulus et al. [23], in which each 0-1 knapsack instance consists of 10 items and each item has 4096 features related to its price and size. For both NN and our method, we use a 5-layer fully-connected network with 512 neurons per hidden layer. We conduct experiments on 4 different knapsack capacities: 100, 150, 200, and 250. We use 700 instances for training and 300 instances for testing the model performance. Considering the real-life setting, we use 3 scales of the penalty factor for the penalty function in Appendix C.2:  $\sigma = 0.05$ , 0.25, or 0.5.
Table 4 reports the mean post-hoc regrets and standard deviations across 10 runs for each approach on this 0-1 knapsack problem. Due to the space limitation and the fact that larger penalty factors are unrealistic in this problem setting, we present penalty factors  $\geq 1$  in Appendix G. The average True Optimal Values (TOV) are reported in the last column, again for reference. As shown in the table, our proposed 2S method has significantly better results. In addition, we observe that across all approaches, the post-hoc regrets decrease as the knapsack capacity increases: this is due to the fact that as the capacity increases, more and more items can be selected, and so minor inaccuracies in predicted values/weights do not affect the selected set of items as much. On the other hand, the advantage of our 2S method over other approaches actually becomes more significant as the capacity increases, demonstrating the superior accuracy of our approach.
Nurse Scheduling Problem Our last experiment is on the nurse scheduling problem (NSP) with unknown patients needs, with the goal of scheduling a nurse roster satisfying unknown patient load demands while minimizing mismatched nurse-shift preferences as the objective. See Appendix C.3 for a description of the application scenario, the MILP formulations of the two stages, as well as the associated penalty function. Given that NSP is not an LP, IntOpt-C again does not apply, and so we
only compare the proposed 2S training method with the classical approaches, using the Two-Stage Predict+Optimize framework for evaluation. Each NSP instance consists of 15 nurses, 7 days, and 3 shifts per day. The nurse preferences are obtained from the NSPLib dataset [26], which is widely used for NSP [16, 20]. The number of patients that each nurse can serve in one shift is randomly generated from [10,20], representing the fact that each nurse has different capabilities. Given that we are unable to find datasets specifically for the patient load demands and relevant prediction features, we follow the experimental approach of Demirovic et al. [4, 5, 6] and use real data from a different problem (the ICON scheduling competition) as the numerical values required for our experiment instances. In this dataset, the unknown number of patients per shift is predicted by 8 features.
Since there are far fewer features than the previous experiments, for both NN and 2S we use a smaller network structure: a 4-layer fully-connected network with 16 neurons per hidden layer. We use 210 instances for training and 90 instances for testing. Just like the first experiment, we use 6 scales of penalty factors (see Appendix C.3 for the penalty function):  $\gamma$  with i.i.d. entries drawn uniformly from  $[0.25 \pm 0.015]$ ,  $[0.5 \pm 0.015]$ ,  $[1.0 \pm 0.015]$ ,  $[2.0 \pm 0.015]$ ,  $[4.0 \pm 0.015]$ , and  $[8.0 \pm 0.015]$ .
Table 5 reports the mean post-hoc regrets and standard deviations across 10 runs for each approach on the NSP. The table shows that the proposed 2S method again has the best performance among all the training approaches. Our 2S method obtains at least  $7.61\%$ ,  $15.65\%$ ,  $17.99\%$ ,  $20.51\%$ ,  $46.76\%$ , and  $62.49\%$  smaller post-hoc regret than other classical methods when the penalty factor is  $[0.25 \pm 0.015]$ ,  $[0.5 \pm 0.015]$ ,  $[1.0 \pm 0.015]$ ,  $[2.0 \pm 0.015]$ ,  $[4.0 \pm 0.015]$ , and  $[8.0 \pm 0.015]$  respectively.
Runtime Analysis Appendix H gives the training times for each method. Most classical approaches are faster than our 2S method, although as shown their post-hoc regrets are much worse. In alloy production, the only setting where IntOpt-C applies, its running time is shorter but comparable with 2S. In 0-1 knapsack, the only problem with public CombOptNet code, the 2S method is much faster.
Section 1 already summarized prior works in Predict+Optimize, most of which focus on learning unknowns only in the objective. Only the Hu et al. [12] framework considers unknowns in constraints.
Here we summarize other works related to learning unknowns in optimization problem constraints, particularly those outside of Predict+Optimize. These works can be placed into two categories.
One category also considers learning unknowns in constraints, but with very different goals and measures of loss. For example, CombOptNet [23] and Nandwani et al. [21] focus on learning parameters so as to make the predicted optimal solution (first-stage solution in our proposed framework) as close to the true optimal solution  $x^{*}$  as possible in the solution space/metric. By contrast, our proposed framework explicitly formulates the two-stage framework and post-hoc regret in order to directly capture rewards and costs in application scenarios. Experiments on 0-1 knapsack in Section 5 show that these other methods yield worse predictive performance when evaluated on the post-hoc regret, under the proposed two-stage framework.
Another category gives ways to differentiate through LPs or LPs with regularizations, as a technical component in a gradient-based training algorithm. As mentioned in Section 4, these works can indeed be used in place of our proposed approach in Section 4/Appendix B. However, we point out that: (i) these other technical tools are essentially orthogonal to our primary contribution, which is the two-stage framework (Section 3), and (ii) nonetheless, experiments on the 0-1 knapsack in Appendix E demonstrate that our gradient calculation approach performs at least as well in post-hoc regret performance as other works, while being faster.
We proposed Two-Stage Predict+Optimize: a new, conceptually simpler and more powerful framework for the Predict+Optimize setting where unknown parameters can appear both in the objective and in constraints. We showed how the simpler perspective offered by the framework allows us to give a general training framework for all MILPs, contrasting prior work which apply only to covering and packing LPs. Experimental results demonstrate that our training method offers significantly better prediction performance over other classical and state-of-the-art approaches.