SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xinyi Hu<sup>1</sup>, Jasper C.H. Lee<sup>2</sup>, Jimmy H.M. Lee<sup>1</sup>
$^{1}$ Department of Computer Science and Engineering The Chinese University of Hong Kong, Shatin, N.T., Hong Kong
$^{2}$ Department of Computer Sciences & Institute for Foundations of Data Science
University of Wisconsin-Madison, WI, USA
{xyhu,jlee} @cse.cuhk.edu.hk, jasper.lee@wisc.edu
Consider the setting of constrained optimization, with some parameters unknown at solving time and requiring prediction from relevant features. Predict+Optimize is a recent framework for end-to-end training supervised learning models for such predictions, incorporating information about the optimization problem in the training process in order to yield better predictions in terms of the quality of the predicted solution under the true parameters. Almost all prior works have focused on the special case where the unknowns appear only in the optimization objective and not the constraints. Hu et al. proposed the first adaptation of Predict+Optimize to handle unknowns appearing in constraints, but the framework has somewhat ad-hoc elements, and they provided a training algorithm only for covering and packing linear programs. In this work, we give a new simpler and more powerful framework called Two-Stage Predict+Optimize, which we believe should be the canonical framework for the Predict+Optimize setting. We also give a training algorithm usable for all mixed integer linear programs, vastly generalizing the applicability of the framework. Experimental results demonstrate the superior prediction performance of our training framework over all classical and state-of-the-art methods.
Optimization problems are prevalent in modern society, and yet the problem parameters are not always available at the time of solving. For example, consider the real-world application scenario of stocking a store: as store managers, we need to place monthly orders for products to stock in the store. We want to stock products that sell fast and yield high profits, as much of them as possible, subject to the hard constraint of limited storage space. However, orders need to be placed two weeks in advance of the monthly delivery, and the customer demand next month cannot be known exactly at the time of order placement. In this paper, we consider the supervised learning setting, where the unknown parameters can be predicted from relevant features, and there are sufficient historical (features, parameters) pairs as training data for a prediction model. The goal, then, is to learn a prediction model from the training data such that, if we plug in the estimated parameters into the optimization problem and solve for an estimated solution, the estimated solution remains a good solution even after the true parameters are revealed.
The classic approach to the problem would be to train a simple regression model, based on standard losses such as (regularized)  $\ell_2$  loss, to predict parameters from the features. It is shown, however, that having a small prediction error in the parameter space does not necessarily mean that the estimated solution performs well under the true parameters. The recent framework of Predict+Optimize, by
Elmachtoub and Grigas [7], instead proposes the more effective regret loss for training, which compares the solution qualities of the true optimal solution and the estimated solution under the true parameters. Subsequent works [6, 8, 10, 13, 17, 19, 27] have since appeared in the literature, applying the framework to more and wider classes of optimization problems as well as focusing on speed-vs-prediction accuracy tradeoffs.
However, all these prior works focus only on the case where the unknown parameters appear in the optimization objective, and not in the constraints. The technical challenge for the generalization is immediate: if there were unknown parameters in the constraints, the estimated solution might not even be feasible under the true parameters revealed afterwards! Thus, in order to tackle the Predict+Optimize setting with unknowns in constraints, the recent work of Hu et al. [12] presents the first such adaptation on the framework: they view the estimated solution as representing a soft commitment. Once the true parameters are revealed, corrective action can be taken to ensure feasibility, potentially at a penalty corresponding to the real-life cost of (partially) reneging on a soft commitment. Their framework captures application scenarios whenever such correction is possible, and requires the practitioner to specify both the correction mechanism and the penalty function. These data can be determined and derived from the specific application scenario. As an example, in the product-stocking problem, an additional unknown parameter is the storage space, because it depends on how the current products in the store sell before the new order arrives. We need to place orders two weeks ahead based on predicted storage space. The night before the order arrives, we know the precise available space, meaning that the unknown parameter is revealed. A possible correction mechanism then is to throw away excess products that the store cannot keep, while incurring the penalty that is the retail price of the products, as well as disposal fees.
While the Hu et al. [12] framework does capture many application scenarios, there are important shortcomings. In their framework, they require the practitioner to specify a correction function that amends an infeasible solution into a feasible solution. However, the derivation of a correction function can be rather ad-hoc in nature. In particular, given an infeasible estimated solution, there may be many ways to transform the solution into a feasible one, and yet their framework requires the practitioner to pick one particular way. This leads to the second downside: it is difficult to give a general algorithmic framework that applies to a wide variety of optimization problems. Hu et al. had to restrict their attention only to packing and covering linear programs, for which they could propose a generic correction function. In this work, we aim to vastly generalize the kinds of optimization problems that Predict+Optimize can tackle under uncertainty in the constraints. In addition, the approach of Hu et al. fails to handle the interesting situation in which post-hoc correction is still desirable when the estimated solution is feasible but not good under the true parameters.
- To mitigate the shortcomings of the prior work, we propose and advocate a new framework, which we call Two-Stage Predict+Optimize<sup>1</sup>, that is both conceptually simpler and more expressive in terms of the class of optimization problems it can tackle. The key idea for the new framework is that the correction function is unnecessary. All that is required is a penalty function that captures the cost of modifying one solution to another. A penalty function is sufficient for defining a correction process: we formulate the correction process itself as a "Stage 2" optimization problem, taking the originally estimated solution as well as the penalty function into account.
- Under this framework, we further propose a general end-to-end training algorithm that applies not only to packing and covering linear programs, but also to all mixed integer linear programs (MILPs). We adapt the approach of Mandi and Guns [18] to give a gradient method for training neural networks to predict parameters from features.
- We apply the proposed method to three benchmarks to demonstrate the superior empirical performance over classical and state-of-the-art training methods.
In this section, we give basic definitions for optimization problems and the Predict+Optimize setting [7], and describe the state-of-the-art framework [12] for Predict+Optimize with unknown parameters
<sup>1</sup>The literature sometimes uses "two-stage" to mean approaches where the prediction is agnostic to the optimization problem. Here, "two-stage" refers to the soft commitment and the correction.
in constraints. The theory is stated in terms of minimization but applies of course also to maximization, upon appropriate negation. Without loss of generality, an optimization problem (OP)  $P$  can be defined as finding:
$$
x ^ {*} = \underset {x} {\arg \min } o b j (x) \quad \text {s . t .} C (x)
$$
where  $x \in \mathbb{R}^d$  is a vector of decision variables,  $obj: \mathbb{R}^d \to \mathbb{R}$  is a function mapping  $x$  to a real objective value that is to be minimized, and  $C$  is a set of constraints that must be satisfied over  $x$ . We call  $x^*$  an optimal solution and  $obj(x^*)$  the optimal value. A parameterized optimization problem (Para-OP)  $P(\theta)$  is an extension of an OP  $P$ :
$$
x ^ {*} (\theta) = \underset {x} {\arg \min } o b j (x, \theta) \quad \text {s . t .} C (x, \theta)
$$
where  $\theta \in \mathbb{R}^t$  is a vector of parameters. The objective  $obj(x,\theta)$  and constraints  $C(x,\theta)$  can both depend on  $\theta$ . When the parameters are known, a Para-OP is just an OP.
In the Predict+Optimize setting [7], the true parameters  $\theta \in \mathbb{R}^t$  for a Para-OP are not known at solving time, and estimated parameters  $\hat{\theta}$  are used instead. Suppose each parameter is estimated by  $m$  features. The estimation will rely on a machine learning model trained over  $n$  observations of a training data set  $\{(A^1,\theta^1),\dots ,(A^n,\theta^n)\}$  where  $A^i\in \mathbb{R}^{t\times m}$  is a feature matrix for  $\theta^i$ , in order to yield a prediction function  $f:\mathbb{R}^{t\times m}\to \mathbb{R}^t$  predicting parameters  $\hat{\theta} = f(A)$ .
Solving the Para-OP using the estimated parameters, we obtain an estimated solution  $x^{*}(\hat{\theta})$ . When the unknown parameters appear in constraints, one major challenge is that the feasible region is only approximated at solving time, and hence the estimated solution may be infeasible under the true parameters. Fortunately, in certain applications, the estimated solution is not a hard commitment, but only represents a soft commitment that can be modified once the true parameters are revealed. Hu et al. [12] propose a Predict+Optimize framework for such applications. The framework is as follows: i) the unknown parameters are estimated as  $\hat{\theta}$ , and an estimated solution  $x^{*}(\hat{\theta})$  is solved using the estimated parameters, ii) the true parameters  $\theta$  are revealed, and if  $x^{*}(\hat{\theta})$  is infeasible under  $\theta$ , it is amended into a corrected solution  $x_{\mathrm{corr}}^{*}(\hat{\theta},\theta)$  while potentially incurring some penalty, and finally iii) the solution  $x_{\mathrm{corr}}^{*}(\hat{\theta},\theta)$  is evaluated according to the sum of both the objective, under the true parameters  $\theta$ , and the incurred penalty from correction.
More formally, a correction function takes an estimated solution  $x^{*}(\hat{\theta})$  and true parameters  $\theta$  and returns a corrected solution  $x_{corr}^{*}(\hat{\theta},\theta)$  that is feasible under  $\theta$ . A penalty function  $Pen(x^{*}(\hat{\theta})\to x_{corr}^{*}(\hat{\theta},\theta))$  takes an estimated solution  $x^{*}(\hat{\theta})$  and the corrected solution  $x_{corr}^{*}(\hat{\theta},\theta)$  and returns a non-negative penalty. Both the correction function and the penalty function should be chosen according to the precise application scenario at hand. The final corrected solution  $x_{corr}^{*}(\hat{\theta},\theta)$  is evaluated using the post-hoc regret, which is defined with respect to the corrected solution  $x_{corr}^{*}(\hat{\theta},\theta)$  and the penalty function  $Pen(x^{*}(\hat{\theta})\to x_{corr}^{*}(\hat{\theta},\theta))$ . The post-hoc regret is the sum of two terms: (a) the difference in objective between the true optimal solution  $x^{*}(\theta)$  and the corrected solution  $x_{corr}^{*}(\hat{\theta},\theta)$  under the true parameters  $\theta$ , and (b) the penalty that the correction process incurs. Mathematically, the post-hoc regret function  $PReg(\hat{\theta},\theta):\mathbb{R}^t\times \mathbb{R}^t\rightarrow \mathbb{R}_{\geq 0}$  (for minimization problems) is:
$$
P R e g (\hat {\theta}, \theta) = o b j \left(x _ {c o r r} ^ {*} (\hat {\theta}, \theta), \theta\right) - o b j \left(x ^ {*} (\theta), \theta\right) + P e n \left(x ^ {*} (\hat {\theta}) \rightarrow x _ {c o r r} ^ {*} (\hat {\theta}, \theta)\right) \tag {1}
$$
where  $obj(x_{corr}^* (\hat{\theta},\theta),\theta)$  is the corrected optimal value and  $obj(x^{*}(\theta),\theta)$  is the true optimal value.
Given the post-hoc regret as a loss function, the empirical risk minimization principle dictates that we choose the prediction function to be the function  $f$  from the set of models  $\mathcal{F}$  attaining the smallest average post-hoc regret over the training data:
$$
f ^ {*} = \underset {f \in \mathcal {F}} {\arg \min } \frac {1}{n} \sum_ {i = 1} ^ {n} P R e g \left(f \left(A ^ {i}\right), \theta^ {i}\right) \tag {2}
$$
3 Two-stage Predict+Optimize Framework
While the prior work by Hu et al. [12] is the first Predict+Optimize framework for unknowns in constraints, and is indeed applicable to a good range of applications, it has several shortcomings.
First, the framework requires mathematically formalizing both a penalty function and a correction function from the application scenario, and essentially imposes differentiability assumptions on the correction function for the framework to be usable. The ad-hoc nature of writing down a correction function limits the practical applicability of the framework. Second, as a result of needing a single (differentiable) correction function, Hu et al. [12] needed to restrict their attention to only packing and covering linear programs, in order to derive a general correction function that is applicable to all the instances. This also significantly limits the immediate applicability of their framework. Third, their framework only corrects an estimated solution when it is infeasible under the true parameters. Yet, there are applications where corrections are possible even when the estimated solution were feasible, but just not very good under the true parameters.
In this paper, we advocate using a simpler yet more powerful framework, which we call Two-Stage Predict+Optimize, addressing all of the above shortcomings. The simplified perspective will allow us to discuss more easily how to handle the entire class of mixed integer linear programs (MILPs) instead of being restricted to just packing and covering linear programs. Since MILPs include all optimization problems in NP (under a reasonable definition of NP for optimization problems), our framework is significantly more applicable in practice. We will describe the Two-Stage Predict+Optimize framework below, and discuss its application to MILPs in the next section.
Our framework is simple: we forgo the idea of a correction function and treat correction itself as an optimization problem, based on the penalty function, the estimated solution and the revealed true parameters. Recall the Hu et al. view of Predict+Optimize under uncertainties in constraints: the estimated solution is a form of soft commitment, which can be modified at a cost once the true parameters are revealed. The penalty function describes the cost of changing from an estimated solution to a final solution. The main observation is that, given an estimated solution and the revealed parameters, we should in fact solve a new optimization problem, formed by applying the true parameters to the original optimization, and adding the penalty function to the objective. The final solution from this new optimization thus takes the penalty of correction into account. This approach yields three immediate advantages. First, the practitioner no longer needs to specify a correction function, thus reducing the ad-hoc nature of the framework. Second, even feasible solutions are allowed to be modified after the true parameters are revealed if the penalty of doing so is not infinity. Third, conditioned on the same penalty function, the solution quality from our two-stage optimization approach is always at least as good as that from using any correction function. The last advantage is presented as Proposition A.1.
Now we formally define the Two-Stage Predict+Optimize framework.
I. In Stage 1, the unknown parameters are estimated as  $\hat{\theta}$  from features. The practitioner then solves the Stage 1 optimization, which is the Para-OP using the estimated parameters, to obtain the Stage 1 solution  $x_{1}^{*}$ . The Stage 1 solution should be interpreted as some form of soft commitment, that we get to modify in Stage 2 at extra cost/penalty. Assuming the notation of the Para-OP in Section 2, the Stage 1 OP can be formulated as:
$$
x _ {1} ^ {*} = \underset {x} {\arg \min } o b j (x, \hat {\theta}) \quad \text {s . t .} C (x, \hat {\theta})
$$
II. At the beginning of Stage 2, the true parameters  $\theta$  are revealed. The Stage 2 optimization problem augments the original Stage 1 problem by adding a penalty term  $Pen(x_1^* \rightarrow x_2^*, \theta)$  to the objective, which accounts for the penalty (modelled from the application scenario) for changing from the softly-committed Stage 1 solution  $x_1^*$  to the new Stage 2 and final solution  $x_2^*$ . The Stage 2 OP can then be formulated as:
$$
x _ {2} ^ {*} = \underset {x} {\arg \min } o b j (x, \theta) + P e n (x _ {1} ^ {*} \to x, \theta) \quad \text {s . t .} C (x, \theta)
$$
Solving the Stage 2 problem yields the final Stage 2 "corrected" solution  $x_{2}^{*}$ .
III. The Stage 2 solution  $x_{2}^{*}$  is evaluated according to the analogous post-hoc regret, as follows:
$$
P R e g (\hat {\theta}, \theta) = o b j \left(x _ {2} ^ {*}, \theta\right) + P e n \left(x _ {1} ^ {*} \rightarrow x _ {2} ^ {*}, \theta\right) - o b j \left(x ^ {*} (\theta), \theta\right)
$$
where again,  $x^{*}(\theta)$  is an optimal solution of the Para-OP under the true parameters  $\theta$ . Note that the post-hoc regret depends on all of a) the predicted parameters, b) the induced Stage 1 solution, c) the true parameters and d) the final Stage 2 solution.
To see this new framework applies in practice, the following example expands on the product-stocking problem in the introduction.
Example 1. Consider the product-stocking problem again, where regular orders have to be placed two weeks ahead of monthly deliveries. Since the available space at the time of delivery is unknown when we place the regular orders, depending on the sales over the next two weeks, we need to make a prediction on the available space to make a corresponding order. We learn the predictor using historical sales records from features such as time-of-year and price. Then, we use the predicted available space to optimize for the regular order we place. This is the Stage 1 solution.
The night before the order arrives, the unknown constraint parameter, i.e. the precise available space, is revealed. We can then check if we have over-ordered or under-ordered. In the case of over-ordering, we would have to call and ask the wholesale company to drop some items from the order. The company would perhaps allow taking the items off the final bill, but naturally they have a surcharge for last-minute changes. Similarly, if we under-ordered, we might request the wholesale company to send us more products, again naturally with a surcharge for last-minute ordering. The updated order is the Stage 2 decision. The incurred wholesaler surcharges induce the penalty function.
A reader who is familiar with the literature on two-stage optimization problems may note that the above framework is phrased slightly differently from some other two-stage problem formulations. In particular, some two-stage frameworks phrase Stage 1 solutions as hard commitments, and include recourse variables in both stages of optimization to model what changes are made in Stage 2. We show in Appendix A.1 how our framework can capture this other perspective, and in general discuss how problem modelling can be done in our new framework.
The reader may also wonder: what about application scenarios where the (Stage 1) estimated solution is a hard commitment, and there is absolutely no correction/recourse available? In Appendix A.2, we discuss how our framework is still useful and applicable for learning in these situations.
We also give a more detailed comparison, in Appendix A.3, between our new Two-Stage Predict+Optimize framework and the prior Hu et al. framework. Technically, if we ignored differentiability issues, the two frameworks are mathematically equivalent in expressiveness. However, we stress that our new framework is both conceptually simpler and easier to apply to a far wider class of optimization problems. We show concretely in the next section how to end-to-end train a neural network for this framework for all MILPs, vastly generalizing the method of Hu et al. which is restricted to packing and covering (non-integer) linear programs. In addition, Appendix A.3 also states and proves Proposition A.1, that if we fix an optimization problem, a prediction model and a penalty function, then the solution quality from our two-stage approach is always at least as good as using the correction function approach.
4 Two-Stage Predict+Optimize on MILPs
In this section, we describe how to give an end-to-end training method for neural networks to predict unknown parameters from features, under the Two-Stage Predict+Optimize framework. The following algorithmic method is applicable whenever both stages of optimization are expressible as MILPs. Due to the page limit, the discussion in this section is high-level and brief, with all the calculation details deferred to Appendix B.
The standard way to train a neural network is to use a gradient-based method. In the Two-Stage Predict+Optimize framework, we use the post-hoc regret  $PReg$  as the loss function. Therefore, for each edge weight  $w_{e}$  in the neural network, we need to compute the derivative  $\frac{\mathrm{d}PReg}{\mathrm{d}w_{e}}$ . Using the law of total derivative, we get
$$
\frac {\mathrm {d} P R e g (\hat {\theta} , \theta)}{\mathrm {d} w _ {e}} = \left. \frac {\partial P R e g (\hat {\theta} , \theta)}{\partial x _ {2} ^ {*}} \right| _ {x _ {1} ^ {*}} \frac {\partial x _ {2} ^ {*}}{\partial x _ {1} ^ {*}} \frac {\partial x _ {1} ^ {*}}{\partial \hat {\theta}} \frac {\partial \hat {\theta}}{\partial w _ {e}} + \left. \frac {\partial P R e g (\hat {\theta} , \theta)}{\partial x _ {1} ^ {*}} \right| _ {x _ {2} ^ {*}} \frac {\partial x _ {1} ^ {*}}{\partial \hat {\theta}} \frac {\partial \hat {\theta}}{\partial w _ {e}} \tag {3}
$$
As such, we wish to calculate each term on the right hand side.
The easiest term to handle is  $\frac{\partial\hat{\theta}}{\partial w_e}$ , since  $\hat{\theta}$  is the neural network output, and so the derivatives can be directly calculated by standard backpropagation [25]. As for the terms  $\left.\frac{\partial PReg(\hat{\theta},\theta)}{\partial x_2^*}\right|_{x_1^*}$  and  $\left.\frac{\partial PReg(\hat{\theta},\theta)}{\partial x_1^*}\right|_{x_2^*}$ , they are easily calculable whenever both the optimization objective and penalty function are smooth, and in fact linear as in the case of MILPs. What remains are the terms  $\frac{\partial x_2^*}{\partial x_1^*}$  and
$\frac{\partial x_1^*}{\partial \hat{\theta}}$ . The challenge is that  $x_2^*$  is the solution of a MILP optimization (Stage 2) that uses  $x_1^*$  as its parameters, i.e., differentiate through a MILP. Similarly,  $x_1^*$  depends on  $\hat{\theta}$  through a MILP (Stage 1). Since MILP optima may not change under minor parameter perturbations, the gradients can be either 0 or non-existent, which are uninformative. We thus need to compute some approximation in order to get useful training signals.