SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Two important components in the framework of predict-then-calibrate are (i) the flexibility in choosing the prediction model  $\hat{f}$  and (ii) the contextual uncertainty set  $\mathcal{U}_{\alpha}(z)$ . In the following, we illustrate the importance of these two aspects via simple examples.
The value of better prediction. Consider a single-variable LP
$$
\min  _ {x} c x \text {s . t .} - 1 \leq x \leq 1 \tag {11}
$$
where the objective is determined by  $c = \sum_{i=1}^{d} z_i - d \cdot \epsilon$ . Here  $z_i$  and  $\epsilon$  are sampled from an exponential distribution  $\mathrm{Exp}(1)$  for all  $i = 1, \dots, d$ . We note that  $z_i$  can be interpreted as either the covariate itself or some useful latent factor extracted by the prediction model. Consider a prediction model that can only utilize/extract the first  $k$  covariates/latent factors  $z_{1:k} = (z_1, \dots, z_k)$ . The optimal prediction model would be  $f_k(z_{1:k}) = \sum_{i=1}^{k} z_i$ , and also we assume it has perfect uncertainty quantification denoted by  $\mathcal{U}_{\alpha}(z_{1:k})$ . We denote the accordingly robust solution by  $x_{\alpha}^*(z_{1:k})$ , and thus all these solutions for  $k = 1, \dots, d$  meet the coverage guarantee.
Proposition 2. For any  $\alpha \in (0.5,1)$  and  $k\geq 1$
$$
\mathbb {P} \left(x _ {\alpha} ^ {*} \left(z _ {1: k}\right) = 0\right) =
$$
$$
\left. \right. \frac {1}{\Gamma (k)} \left(\gamma \left(k, \max  \left\{0, - d \log \left((1 - \alpha) \left(1 + \frac {1}{d}\right) ^ {d - k}\right)\right\}\right) - \gamma \left(k, \max  \left\{0, - d \log \left(\alpha \left(1 + \frac {1}{d}\right) ^ {d - k}\right)\right\}\right)\right),
$$
and it decreases monotonously with respect to  $k$ . Here,  $\gamma(\cdot, \cdot)$  denotes the lower incomplete gamma function.
Proposition 2 gives an analytical expression for the probability of the optimal robust solution equal to zero. A zero solution can be viewed as a conservative solution when the prediction model is uncertain about the objective  $c$ . While we assume the prediction model is equipped with perfect uncertainty quantification, the proposition says that more informative covariates and/or more powerful prediction models will lead to less conservative solutions for the downstream robustness task. The reason is that a better prediction model can make the uncertainty quantification task easier by smoothing the conditional distribution of the residual  $r|z$  with respect to  $z$ . For example, when  $k = d$  in Proposition 2, the distributions of residuals are identical and thus very smooth for different covariates, which is easier to quantify than the case with  $k = 0$ . This reason can also be extended to general cases.
The value of better calibration. Consider the LP (11) again but with  $c$  specified by
$$
c = (\operatorname {s i g n} (z) + \epsilon) \sqrt {| z |},
$$
where  $z \in \mathbb{R}$  is sampled uniformly from  $[-1, 1]$ , and  $\epsilon$  is sampled from the uniform distribution on  $[-0.5, 0.5]$ . In this case, the sign of the objective  $c$  can be deterministically determined by the sign of  $z$ . With perfect knowledge of the underlying uncertainty, the following solution is the optimal robust solution for all  $\alpha \in (0, 1)$
$$
x ^ {*} (z) = - \operatorname {s i g n} (z).
$$
The following proposition gives another uncertainty set with the optimal prediction model and a coverage guarantee but a suboptimal robust solution.
Proposition 3. For  $\alpha \geq 1/2$ , let the uncertainty set
$$
\mathcal {U} _ {\alpha} (z) = \left\{ \begin{array}{l l} \left[ \sqrt {z} - \frac {1 - \sqrt {2 - 2 \alpha}}{2}, \infty\right), & i f z \in [ 0, 1 ], \\ (- \infty , - \sqrt {- z} + \frac {1 - \sqrt {2 - 2 \alpha}}{2} ], & i f z \in [ - 1, 0). \end{array} \right.
$$
The uncertainty set has a coverage guarantee in that  $\mathbb{P}(c\in \mathcal{U}_{\alpha}(z)) = \alpha$ . If we solve the optimization problem (5) with the uncertainty set  $\mathcal{U}_{\alpha}(z)$ , the following robust solution is obtained
$$
x (z) = \left\{ \begin{array}{l l} - 1, & i f z \geq \frac {3 - 2 \alpha - 2 \sqrt {2 - 2 \alpha}}{4}, \\ 0, & i f \frac {- 3 + 2 \alpha + 2 \sqrt {2 - 2 \alpha}}{4} \leq z \leq \frac {3 - 2 \alpha - 2 \sqrt {2 - 2 \alpha}}{4}, \\ 1, & i f z \leq \frac {- 3 + 2 \alpha + 2 \sqrt {2 - 2 \alpha}}{4}. \end{array} \right.
$$
The robust solution induced by the uncertainty set  $\mathcal{U}_{\alpha}(z)$  is unnecessarily conservative compared to the optimal robust solution  $x^{*}(z)$ . This highlights the importance of good uncertainty quantification, and in particular, contextualized uncertainty quantification. Specifically, the suboptimality of  $x(z)$  arises from that the uncertainty set  $\mathcal{U}_{\alpha}(z)$  in the proposition is not much contextualized with respect to  $z$ , and this justifies the contextualized uncertainty models  $\hat{h}$  and  $\hat{g}$  for the preliminary calibration step in Algorithm 1 and Algorithm 2.
In addition to the contextualized uncertainty quantification, we remark the flexibility in choosing the calibration model or uncertainty set can bring us two benefits. First, it has the potential to further reduce decision conservatism by collaborating with new designs of small uncertainty sets. Specifically, we can design predict-then-calibrate algorithms with even small uncertainty sets based on many previous works that focus on designing minimum size uncertainty sets to achieve a given coverage guarantee in the context-free robust optimization or distributional robust optimization literature (Bertsimas et al. [2018], Gupta [2019]). Second, this framework is also applicable to more general optimization problems such as convex programs or multi-stage stochastic programs. In
particular, for these general problems, if the robust VaR formulation has a tractable construction of the uncertainty set in the context-free setting, we can then design a predict-then-calibrate algorithm by constructing the corresponding contextual uncertainty set accordingly. To close this section, we emphasize that our predict-then-calibrate framework behaves more like a plug-in module that can fit into the existing robust optimization literature, and thus, it is potentially capable of handling more problems than the contextual LP loss discussed in this paper.
4 Extension to Distributionally Robust Contextual LP
In the previous sections, we study the problem of robust contextual LP and the framework of predict-then-calibrate. Now we turn to a related but different problem – distributionally robust contextual LP – through the lens of predict-then-calibrate. We relate the objectives of robustness and distributional robustness with the two types of uncertainty in the uncertainty quantification literature. We show the paradigm of predict-then-calibrate can also be extended to the distributionally robust contextual LP and obtain a new generalization bound for the problem.
The distributionally robust contextual LP concerns the following problem:
$$
\min  _ {x} \max  _ {\mathcal {P} ^ {\prime} \in \Xi_ {\gamma}} \mathbb {E} _ {\mathcal {P} ^ {\prime}} \left[ c ^ {\top} x \right] = \mathbb {E} _ {\mathcal {P} ^ {\prime}} \left[ c \right] ^ {\top} x
$$
$$
\begin{array}{l} \text {s . t .} A x \leq b, x \geq 0 \end{array}
$$
where  $\Xi_{\gamma}$  denotes a set of distributions that describes the uncertainty on  $c$ . Ideally, one hopes that the uncertainty set covers the true conditional distribution  $c|z$ . Compared to the robust version that concerns the VaR/Quantile of the objective  $c$ , the formulation here still concerns the risk-neutral objective but accounts for the uncertainty in estimating  $\mathbb{E}[c|z]$ . The distributionally robust optimization has two roles: first, it can be used to derive solutions with provable guarantees on generalization, and second, it induces a regularization effect for the underlying prediction model.
Figure 1: Illustration of robust optimization (RO) and distributionally robust optimization (DRO) (here we omit the covariates  $z$  for simplicity). DRO concerns epistemic uncertainty Hüllermeier and Waegeman [2021], the part of the uncertainty that can be reduced by obtaining more information. With more samples, the uncertainty set will shrink. In contrast, aleatoric uncertainty Hüllermeier and Waegeman [2021] refers to the inherent and irreducible randomness of  $c$ . To derive the uncertainty set for RO problem, one needs to account for the summation of epistemic and aleatoric uncertainty.
As in the previous section, we denote the prediction error on the validation data  $\mathcal{D}_{\mathrm{val}}$  by
$$
r _ {t} = c _ {t} - \hat {f} (z _ {t}).
$$
The following proposition states that there exists a distributionally robust algorithm that achieves a generalization bound that is oblivious with respect to the underlying prediction model  $\hat{f}$ .
Proposition 4. Assume there exists a constant  $L$  such that the following condition holds for any  $z, z'$
$$
\left| \mathbb {E} [ r | z ] - \mathbb {E} [ r | z ^ {\prime} ] \right| \leq L \| z - z ^ {\prime} \| _ {2} ^ {s}.
$$
Then under mild boundedness assumptions on the distribution, there exists a distributionally robust optimization algorithm that utilizes the validation data  $\mathcal{D}_{val}$  and outputs a solution  $\hat{x}(z)$  such that the following inequality holds with high probability
$$
\mathbb {E} \left[ c ^ {\top} (\hat {x} (z) - x ^ {*} (z)) \right] \leq O \left(T ^ {- \frac {s}{2 (s + d)}} \log T\right).
$$
We defer the detailed proof and the DRO algorithm to Appendix C.5. Specifically, the DRO algorithm performs a nonparametric uncertainty quantification with respect to  $r_t$ 's and thus it is oblivious to the underlying prediction model  $\hat{f}$ . We remark that such idea of calibrating  $r_t$ 's is not new and has been exploited in Wang et al. [2021], Kannan et al. [2020, 2021]. But notably, our result requires a weaker condition on the underlying  $\hat{f}$ , where the previous works rely on a realizability condition of the prediction model. Moreover, compared with the generalization bound in the contextual LP literature Liu and Grigas [2021], although Liu and Grigas [2021] develops an  $O(T^{-1/4})$  generalization bound, they still require the function class of the prediction model can have bounded Rademacher complexity. If that condition holds here, our generalization bound can also be improved accordingly.
In this section, we illustrate the performance of our proposed algorithms via one simple example and also a shortest path problem considered in Elmachtoub and Grigas [2022] and Hu et al. [2022]. We defer more results including more visualizations, another experiment on the fractional knapsack problem Ho-Nguyen and Kılınc-Karzan [2022], and experiment setups to the Appendix. We implement our Algorithm 1 (PTC-B) and Algorithm 2 (PTC-E) against several benchmark methods including the Ellipsoid method that ignores the covariates,  $k$ -nearest neighbor (kNN) approach in Ohmori [2021], the Deep Cluster-Classify (DCC), and Integrated Deep Cluster-Classify (IDCC) approaches in Chenreddy et al. [2022]. More experiment details are deferred to Appendix A.
(a) Prediction
(b) Uncertainty quantification
Figure 2: Performance under different prediction and quantile regression models in PTC-B. The results corresponding to PTC-E are the same since PTC-B and PTC-E coincide in this one-dimensional case. The scattered black points indicate the training samples and the curves indicate the band for the uncertainty set. For the left panel, we fix the uncertainty quantification model  $\hat{h}$  to be a neural network and implement different prediction models. For the right panel, we fix the prediction model  $\hat{f}$  as a neural network and implement different uncertainty quantification models. KR stands for kernel regression, GBR stands for gradient boosting regression, and NN stands for neural network.
Simple LP Visualization
In the first experiment, we study the LP problem (11) where the optimal solution and the uncertainty set can be visualized. Here the covariates  $z = (z_{1},\dots,z_{d})^{\top}$  where  $z_{i}$  is sampled from  $\mathrm{Unif}[-0.5,0.5]$  independently for  $i = 1,\ldots,d$ ,  $\epsilon \sim \mathrm{Unif}[-0.5,0.5]$ , and  $c = (\mathrm{sign}(z_1) + \epsilon)\cdot \sqrt{|z_1|}$  ( $c$  is independent of  $z_{2},\ldots,z_{d}$ ). We consider a risk level of  $\alpha = 0.8$  and dataset size  $T = 1000$ . For our methods of PTC-B and PTC-E, we use  $60\%$  of the data for training  $\hat{f}$ ,  $20\%$  for preliminary calibration  $(D_{1})$ , and  $20\%$  for final adjustment  $(D_{2})$ . In Figure 2, the robust solutions obtained from different prediction methods and different calibration methods are plotted. The results reinforce the discussions in Section 3.2 that either a better prediction model or a better calibration model will improve the final performance of the robust optimization problem.
Shortest path problem
Now we present numerical experiments for a shortest path problem on a  $5 \times 5$  grid with 40 edges, and the cost of traveling through edge  $i$  is  $c_{i}$  for  $i = 1, \dots, n$ . We follow the experiment setup in
Figure 3: Value of better prediction
Figure 4: Sample efficiency
Table 1: Average VaR and coverage on shortest path problems under different risk level settings. The average is reported based on 500 simulation trials.
(a) Average VaR
α	Ellips.	kNN	DCC	IDCC	PTC-B	PTC-E
0.6	2447	1940	2218	2181	1650	1690
0.7	2488	1973	2273	2168	1683	1717
0.8	2535	2010	2312	2205	1708	1774
0.85	2563	2032	2286	2286	1735	1796
0.9	2598	2059	2392	2291	1769	1843
0.95	2646	2098	2373	2249	1832	1905
(b) Average coverage
α	Ellips.	kNN	DCC	IDCC	PTC-B	PTC-E
0.6	0.63	0.65	0.62	0.63	0.56	0.63
0.7	0.73	0.65	0.7	0.73	0.64	0.71
0.8	0.82	0.65	0.8	0.84	0.76	0.8
0.85	0.87	0.65	0.85	0.89	0.79	0.85
0.9	0.92	0.65	0.89	0.93	0.88	0.91
0.95	0.96	0.65	0.94	0.97	0.93	0.97
Elmachtoub and Grigas [2022] and Hu et al. [2022] where the details are deferred to Appendix A.2. In this experiment, we demonstrate several aspects of our algorithms. First, Figure 3 shows the averaged VaR of PTC-B solutions (under different prediction models  $\hat{f}$ ) versus the Mean Squared Error (MSE) on the test data, which reveals a positive correlation between the predictive performance and the quality of the solution obtained in the downstream robust optimization task. We utilize the Kernel Ridge method with the RBF kernel—identified as the top-performing model in Figure 3—as the predictive model and the Neural Network (NN) model as the preliminary calibration model for both PTC-B and PTC-E in the ensuing experiments (Figure 4 and Table 1) to compare with other algorithms. Second, Figure 4 demonstrates that our algorithms exhibit better sample efficiency than the benchmark algorithms as we increase the number of training samples. Specifically, while the model DCC and IDCC take advantage of the expressiveness of a complicated deep learning architecture, they can be more costly in terms of the required training samples. Furthermore, we compare the performance under varying risk level settings in Table 1. The Ellipsoid method does not utilize the covariates information, so although it has a coverage guarantee, it performs worst on the VaR. The kNN method [Ohmori, 2021] has a good VaR performance, but it lacks confidence adjustment, so it is hard to adapt to different confidence levels. The two deep learning-based approaches and ours give good coverage guarantees, while we attribute the advantage of our algorithms to the flexibility in choosing the prediction model and the calibration model.
Conclusion. In this paper, we study the robust contextual LP problem which lies at the intersection of contextual LP/predict-then-optimize and robust optimization. We develop two algorithms for the problem which output box- and ellipsoid shape uncertainty sets. The algorithms introduce a new perspective for contextualized uncertainty calibration, and it highlights the convenience brought by the disentanglement of the prediction and the uncertainty calibration. Yet, as mentioned earlier, the coverage guarantee in Proposition 1 is with respect to a population/average sense. Developing algorithms with individual-level coverage guarantees deserves more future investigation. Also, we hope our work provides a starting point for future research on risk-sensitive and robust decision-making for the problem of contextual optimization and optimization with machine-learned advice.
Naomi S Altman. An introduction to kernel and nearest-neighbor nonparametric regression. The American Statistician, 46(3):175-185, 1992.
Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021.
Gah-Yi Ban and Cynthia Rudin. The big data newsvendor: Practical insights from machine learning. Operations Research, 67(1):90-108, 2019.
Gah-Yi Ban, Jérémie Gallien, and Adam J Mersereau. Dynamic procurement of new products with covariate information: The residual tree method. Manufacturing & Service Operations Management, 21(4):798-815, 2019.
Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. The Annals of Statistics, 51(2):816-845, 2023.
Aharon Ben-Tal, Laurent El Ghaoui, and Arkadi Nemirovski. Robust optimization, volume 28. Princeton university press, 2009.
Dimitris Bertsimas and Nathan Kallus. From predictive to prescriptive analytics. Management Science, 66(3):1025-1044, 2020.
Dimitris Bertsimas and Nihal Koduri. Data-driven optimization: A reproducing kernel hilbert space approach. Operations Research, 70(1):454-471, 2022.
Dimitris Bertsimas and Bart Van Parys. Bootstrap robust prescriptive analytics. Mathematical Programming, 195(1-2):39-78, 2022.
Dimitris Bertsimas, Vishal Gupta, and Nathan Kallus. Data-driven robust optimization. Mathematical Programming, 167:235-292, 2018.
Dimitris Bertsimas, Christopher McCord, and Bradley Sturt. Dynamic optimization with side information. European Journal of Operational Research, 304(2):634-651, 2023.
Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration inequalities: A nonasymptotic theory of independence. Oxford university press, 2013.
Leo Breiman. Classification and regression trees. Routledge, 2017.
Junyu Cao and Rui Gao. Contextual decision-making under parametric uncertainty and data-driven optimistic optimization. Available at Optimization Online, 2021.
Ningyuan Chen, Saman Lagzi, and Joseph Milner. Using neural networks to guide data-driven operational decisions. Available at SSRN 4217092, 2022.
Xin Chen, Melvyn Sim, and Peng Sun. A robust optimization perspective on stochastic programming. Operations Research, 55(6):1058-1071, 2007.
Abhilash Reddy Chenreddy, Nymisha Bandi, and Erick Delage. Data-driven conditional robust optimization. Advances in Neural Information Processing Systems, 35:9525-9537, 2022.
Youngseog Chung, Willie Neiswanger, Ian Char, and Jeff Schneider. Beyond pinball loss: Quantile methods for calibrated uncertainty quantification. Advances in Neural Information Processing Systems, 34:10971-10984, 2021.
Peng Cui, Wenbo Hu, and Jun Zhu. Calibrated reliable regression using maximum mean discrepancy. Advances in Neural Information Processing Systems, 33:17164-17175, 2020.
Priya Donti, Brandon Amos, and J Zico Kolter. Task-based end-to-end model learning in stochastic optimization. Advances in Neural Information Processing Systems, 30, 2017.
Adam N Elmachtoub and Paul Grigas. Smart "predict, then optimize". Management Science, 68(1): 9-26, 2022.
Heyuan Liu and Paul Grigas. Risk bounds and calibration for a smart predict-then-optimize method. Advances in Neural Information Processing Systems, 34:22083-22094, 2021.
Shang Liu, Zhongze Cai, and Xiaocheng Li. Distribution-free model-agnostic regression calibration via nonparametric methods. arXiv preprint arXiv:2305.12283, 2023.
Gar Goei Loke, Qinshen Tang, and Yangge Xiao. Decision-driven regularization: A blended model for predict-then-optimize. Available at SSRN 3623006, 2022.
Elizbar A Nadaraya. On estimating regression. Theory of Probability & Its Applications, 9(1): 141-142, 1964.
Karthik Natarajan, Dessislava Pachamanova, and Melvyn Sim. Incorporating asymmetric distributional information in robust value-at-risk optimization. Management Science, 54(3):573-585, 2008.
Viet Anh Nguyen, Fan Zhang, Jose Blanchet, Erick Delage, and Yinyu Ye. Distributionally robust local non-parametric conditional estimation. Advances in Neural Information Processing Systems, 33:15232-15242, 2020.
Viet Anh Nguyen, Fan Zhang, Jose Blanchet, Erick Delage, and Yinyu Ye. Robustifying conditional portfolio decisions via optimal transport. arXiv preprint arXiv:2103.16451, 2021.
Shunichi Ohmori. A predictive prescription using minimum volume k-nearest neighbor enclosing ellipsoid and robust optimization. Mathematics, 9(2):119, 2021.
R Tyrrell Rockafellar, Stanislav Uryasev, et al. Optimization of conditional value-at-risk. Journal of risk, 2:21-42, 2000.
Suvrajeet Sen and Yunxiao Deng. Learning enabled optimization: Towards a fusion of statistical learning and stochastic programming. INFORMS Journal on Optimization (submitted), 2018.
Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribution generalization: A survey. arXiv preprint arXiv:2108.13624, 2021.
Vladimir Vovk. Conditional validity of inductive conformal predictors. In *Asian conference on machine learning*, pages 475-490. PMLR, 2012.
Tianyu Wang, Ningyuan Chen, and Chun Wang. Distributionally robust prescriptive analytics with Wasserstein distance. arXiv preprint arXiv:2106.05724, 2021.
Yijie Wang, Grani A Hanasusanto, and Chin Pang Ho. Robust contextual portfolio optimization with gaussian mixture models. 2022.
Geoffrey S Watson. Smooth regression analysis. *Sankhya: The Indian Journal of Statistics, Series A*, pages 359-372, 1964.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:2110.11334, 2021.
Shengjia Zhao, Tengyu Ma, and Stefano Ermon. Individual calibration with randomized forecasting. In International Conference on Machine Learning, pages 11387-11397. PMLR, 2020.
Taozeng Zhu, Jingui Xie, and Melvyn Sim. Joint estimation and robustness optimization. Management Science, 68(3):1659-1677, 2022.
A Supplementary for Experiment
Details of benchmarks. For the DCC and IDCC algorithms proposed in Chenreddy et al. [2022], we set the number of clusters to be 10. This choice of the number of clusters is large enough because there are clusters containing no training samples. The Ellipsoid method, which is also a benchmark used in Chenreddy et al. [2022], fits a Gaussian distribution without contextual information and calibrates the radius to cover a proportion of  $\alpha$  training samples. For the kNN conditional RO method [Ohmori, 2021], the hyperparameter,  $k$ , lacks the flexibility to adjust with respect to the number of training samples  $T$ , the dimensionality of the problem, and the risk level  $\alpha$ , so we empirically choose  $k = \max \{\sqrt{T}, 2 \times n\}$  here. Different algorithms utilize the same dataset to ensure fairness in each round of comparison. We redefine  $T$  as its size in the experiment section. For our proposed Algorithm 1 (PTC-B) and 2 (PTC-E) under the PTC framework, the dataset is randomly split into  $60\%$ ,  $20\%$ , and  $20\%$  to do prediction, preliminary calibration, and final adjustment, respectively.
Performance measurements. In the test phase, we randomly generate  $z$  from its marginal distribution. Then, for each  $z$ , we generate a set with 1000 objective vectors,  $\{c^t\}_{t=1}^{1000}$ , from  $\mathbb{P}_{c|z}$ . Given any robust solution  $x$  conditioned on  $z$ , we estimate the  $\mathrm{VaR}_{\alpha}[c^\top x|z]$  by the  $\alpha$ -quantile of  $\{c^t\}_{t=1}^{1000}$  and take its empirical expectation over  $z$  as the average VaR. Similarly, we estimate the average coverage by the average frequency of  $\{c \in \mathcal{U}_z\}$ .
A.1 Simple LP Visualization