SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 8: Hyperparameters of the experiments on the 0-1 knapsack problem.
Model	Hyperparameters
Proposed	optimizer: optim.Adam; learning rate: 10-7; μ = 10-3; epochs=12
k-NN	k=5
RF	n_estimator=100
NN	optimizer: optim.Adam; learning rate: 10-3; epochs=12
Table 9: Hyperparameters of the experiments on the nurse scheduling problem.
Model	Hyperparameters
Proposed	optimizer: optim.Adam; learning rate: 10-1; μ = 10-3; epochs=8
k-NN	k=5
RF	n_estimator=100
NN	optimizer: optim.Adam; learning rate: 10-2; epochs=8
Ridge,  $k$ -NN, CART and RF are implemented using scikit-learn [24]. The neural network is implemented using PyTorch [22]. To compute the two stages of optimization at test time for our method, and to compute the optimal solution of an (MI)LP under the true parameters, we use the MILP solver from Gurobi [11].
E Comparisons of the 2S Method and the Prior Differentiation Methods
In this section, we compare the proposed method with prior works [1, 2, 27] that provide ways of differentiating through LPs or LPs with regularization. We conduct comparisons with CvxpyLayer [1] but not OptNet [2] or QPTL [27]. The reason is that the calculation method proposed in QPTL is  $\mathrm{LP}+$  quadratic regularization using OptNet, and CvxpyLayer is just a conic extension to OptNet. We compared CvxpyLayer [1] with a) no regularization, b) quadratic regularization and c) log-barrier (like our Section 4/Appendix B). The key indicator of its predictive performance is the type of regularization used, with the log-barrier version performing the best, but still slightly worse than our method. We applied CvxpyLayer [1] to the 0-1 knapsack benchmark to compare with our 2S method.
Table 10 reports the mean post-hoc regrets and standard deviations across 10 runs and Table 11 reports the average training times. More precisely, we use it with various regularizations (a. LP with no regularization, b. with quadratic regularization, c. with log-barrier as in our paper) to replace the Section 4/Appendix B gradient calculations. We find that CvxyLayer [1] never gives better solution quality while 2S is  $30\% -50\%$  faster.
Table 10: Mean post-hoc regrets and standard deviations of the 2S method and CvxpyLayer with different regularization on the 0-1 knapsack problem.
PReg	Penalty factor	2S	CvxpyLayer+log	CvxpyLayer+quad_reg	CvxpyLayer+no_reg
cap=100	0.05	1.26±0.01	1.26±0.01	1.27±0.01	7.70±0.39
0.25	6.28±0.05	6.28±0.05	6.34±0.03	8.87±0.92
0.5	9.22±0.10	9.47±0.31	9.96±0.54	10.13±0.46
cap=150	0.05	0.73±0.01	0.74±0.01	0.75±0.03	6.74±0.58
0.25	3.64±0.04	3.64±0.04	3.70±0.03	7.18±0.77
0.5	7.27±0.06	7.28±0.08	7.39±0.06	8.43±0.58
Table 11: Average runtime (in seconds) of the 2S method and CvxpyLayer with different regularization on the 0-1 knapsack problem.
Runtime	2S	CvxpyLayer+log	CvxpyLayer+quad_reg	CvxpyLayer+no_reg
cap=100	204.76	438.24	571.38	344.50
cap=150	245.61	467.65	662.30	366.83
F Frameworks Comparisons on the Alloy Production Problem
In this section, we further compared the proposed framework with the framework using the differentiable projection idea in [3] on the alloy production benchmark. The idea in [3] is to use the  $l_{2}$  projection, and we implemented it using CvxyLayer. The experiment set-up follows that of Table 2: both training and testing use  $l_{2}$  projection in the second stage, as opposed to solving the second stage optimization problem defined in Section 3. Table 12 shows both the post-hoc regret and training time for  $l_{2}$  projection. We find that not only is  $l_{2}$  projection slow, but it has even worse post-hoc regret than the Hu et al. correction [12]. We suspect that this is due to the Hu et al. correction function [12] preserving the direction of the solution vector whereas  $l_{2}$  projection can change the direction, and that this makes a difference for Alloy Production. In any case, this experiment confirms again that our Two-Stage framework has better post-hoc regret than a framework based on differentiable projections, reinforcing the main message of our paper.
Table 12: Comparison of three frameworks on the alloy production problem.
Penalty factor	PReg	Average runtime
0.25±0.015	0.5±0.015	1±0.015	2±0.015	4±0.015	8±0.015
Two-Stage Predict + Optimize Framework	43.87±2.73	65.71±4.81	88.75±5.91	123.90±6.84	161.86±8.49	194.06±13.09	268.22
Hu et al. Framework	68.16±6.26	82.91±5.45	107.64±6.85	150.47±12.99	178.69±10.09	206.84±12.51	228.00
I2_projection	103.28±4.87	118.90±6.99	150.15±11.45	212.62±20.58	337.59±23.24	562.41±34.29	442.97
G Experiments on the 0-1 Knapsack Problem with Large Penalty Factors
Table 13 reports the mean post-hoc regrets and standard deviations across 10 runs for each approach on the 0-1 knapsack problem with large penalty factors (penalty factors  $\geq 1$ ). With more data, we can make further analysis of the performance of the proposed 2S method. Observing Tables 4 and 13, we can see that the trend, in terms of the difference between 2S and other methods, first decreases, then increases, as the penalty factor increases. The trend in Tables 4 and 13 is identical to the trend in Table 3. We can explain this phenomenon as follows.
First, when the penalty factor is small, the rational behavior for the buyer is to just take every order, and only decide which orders to drop when the true parameters are revealed (at close to no cost). 2S identifies and exploits this behavior for small penalties, while classic regression methods are agnostic to this possible tactic. Thus, the advantage of 2S compared to classic regression methods is large in the small penalty case.
Second, when the penalty factor is large, 2S will analogously learn to be conservative, such that the first stage solution likely remains feasible under the true parameters, in order to avoid the necessary (and high) penalty due to having to change to a feasible solution. Again, classic regression methods will be agnostic to this possible tactic, leading to a large advantage of 2S over the classic methods.
Table 5 only has the increasing trend from the large penalty, since it is neither a covering nor a packing program, and so there is no analogous tactic/exploitation for the small penalty.
Table 13: Mean post-hoc regrets and standard deviations for 0-1 knapsack problem with large penalty factors using the Two-Stage Predict+Optimize framework.
PReg	Penalty factor	2S	CombOptNet	Ridge	k-NN	CART	RF	NN	TOV
cap=100	1	10.90±0.15	10.93±0.17	10.93±0.19	11.11±0.17	11.16±0.14	11.01±0.31	11.26±0.23	29.68±0.14
2	12.31±0.16	12.45±0.25	12.48±0.20	12.49±0.21	13.77±0.26	12.60±0.39	12.78±0.30
4	14.54±0.15	15.66±0.47	15.57±0.25	15.68±0.39	19.01±0.56	15.77±0.62	15.84±0.50
cap=150	1	10.23±0.12	10.22±0.18	10.46±0.23	10.40±0.18	10.46±0.19	10.49±0.21	10.86±0.30	40.23±0.19
2	11.18±0.15	11.74±0.34	11.88±0.30	11.63±0.20	12.56±0.31	11.83±0.19	12.12±0.17
4	13.20±0.16	14.33±0.46	14.71±0.49	14.43±0.33	16.75±0.63	14.53±0.29	14.65±0.41
cap=200	1	6.77±0.36	15.30±0.28	7.67±0.18	7.51±0.27	7.71±0.20	7.67±0.16	8.00±0.65	48.13±0.24
2	8.19±0.12	15.39±0.16	8.84±0.22	8.69±0.26	9.24±0.30	8.80±0.20	8.97±0.37
4	9.71±0.35	15.46±0.22	11.17±0.40	11.06±0.32	12.29±0.59	11.05±0.46	10.91±0.53
cap=250	1	1.37±0.08	20.69±0.20	3.08±0.19	2.94±0.16	3.17±0.17	3.05±0.25	3.28±0.96	53.43±0.26
2	3.34±0.15	20.78±0.20	3.80±0.20	3.73±0.15	3.94±0.20	3.79±0.26	3.89±0.58
4	4.46±0.09	20.93±0.20	5.25±0.35	5.32±0.27	5.47±0.35	5.29±0.48	5.11±0.39
H Runtimes for the Experiments
In this paper, all models are trained with Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz processors. Table 14 shows the average runtime across 10 simulations for different optimization problems. Since the testing time of different approaches is quite similar, here, the runtime refers to only the training time of the prediction model and does not include the testing time. At training time, only the proposed 2S method and IntOpt-C solve the LP. Training for the usual NN does not involve the LP at all, and so training is much faster (but gives worse results).
Since IntOpt-C cannot handle the variant of the 0-1 knapsack problem and the NSP, we only report the runtime of IntOpt-C for the alloy production problem.
Since the provided code of CombOptNet is only available for the 0-1 knapsack problem, we only report the runtime of CombOptNet for the 0-1 knapsack problem. As Table 14 shows, CombOptNet is drastically slower than the proposed 2S method.
In the alloy production problem, the runtimes of the proposed 2S method are a little larger than that of IntOpt-C. The reason is that 2S needs to solve two LPs when training while IntOpt-C only needs to solve one. But in the alloy production problem, the unknown parameters are on the left hand side of the inequality constraints and the gradient computation includes matrix computation, which is also time-consuming. Thus, the runtimes of the 2S method are larger but not twice as large as that of the IntOpt-C method.
In both the alloy production problem and the variant of the 0-1 knapsack problem, the runtimes of the 2S method are much better than RF.
The runtime of the 2S method is large in the NSP. This is because we use the formulation where each decision variable corresponds to whether a specific nurse is assigned to a specific day and a specific shift. Thus, the number of the decision variable of the relaxed LP is large and the LP takes more time to solve.
Table 14: Average runtime (in seconds) for the alloy production, 0-1 knapsack, and nurse scheduling problems.
Runtime(s)	Alloy production	0-1 knapsack	Nurse scheduling
Brass	Titanium-alloy	Capacity=100	Capacity=150	Capacity=200	Capacity=250
2S	268.22	394.53	204.76	245.61	202.65	193.46	537.32
IntOpt-C	228.00	331.38	N/A
CompOptNet	N\A	2341.40	2940.26	2394.05	2383.39	N\A
Ridge	20.22	56.89	22.33	<1
k-NN	25.14	70.22	26.00	<1
CART	30.33	94.89	34.83	<1
RF	959.50	2552.25	1034.07	2.11
NN	212.22	321.11	135.80	11.39