Title: Predict-then-Calibrate: A New Perspective of Robust Contextual LP
Venue: NeurIPS 2023
Accepted: yes
Keywords: Uncertainty Quantification

Contextual LP

Robust Optimization

Distributionally Robust Optimization
Abstract: Contextual optimization, also known as predict-then-optimize or prescriptive analytics, considers an optimization problem with the presence of covariates (context or side information). The goal is to learn a prediction model (from the training data) that predicts the objective function from the covariates, and then in the test phase, solve the optimization problem with the covariates but without the observation of the objective function. In this paper, we consider a risk-sensitive version of the problem and propose a generic algorithm design paradigm called predict-then-calibrate. The idea is to first develop a prediction model without concern for the downstream risk profile or robustness guarantee, and then utilize calibration (or recalibration) methods to quantify the uncertainty of the prediction. While the existing methods suffer from either a restricted choice of the prediction model or strong assumptions on the underlying data, we show the disentangling of the prediction model and the calibration/uncertainty quantification has several advantages. First, it imposes no restriction on the prediction model and thus fully unleashes the potential of off-the-shelf machine learning methods. Second, the derivation of the risk and robustness guarantee can be made independent of the choice of the prediction model through a data-splitting idea. Third, our paradigm of predict-then-calibrate applies to both (risk-sensitive) robust and (risk-neutral) distributionally robust optimization (DRO) formulations. Theoretically, it gives new generalization bounds for the contextual LP problem and sheds light on the existing results of DRO for contextual LP. Numerical experiments further reinforce the advantage of the predict-then-calibrate paradigm in that an improvement on either the prediction model or the calibration model will lead to a better final performance.

Predict-then-Calibrate: A New Perspective of Robust Contextual LP

Abstract

Contextual optimization, also known as predict-then-optimize or prescriptive analytics, considers an optimization problem with the presence of covariates (context or side information). The goal is to learn a prediction model (from the training data) that predicts the objective function from the covariates, and then in the test phase, solve the optimization problem with the covariates but without the observation of the objective function. In this paper, we consider a risk-sensitive version of the problem and propose a generic algorithm design paradigm called predict-thencalibrate. The idea is to first develop a prediction model without concern for the downstream risk profile or robustness guarantee, and then utilize calibration (or recalibration) methods to quantify the uncertainty of the prediction. While the existing methods suffer from either a restricted choice of the prediction model or strong assumptions on the underlying data, we show the disentangling of the prediction model and the calibration/uncertainty quantification has several advantages. First, it imposes no restriction on the prediction model and thus fully unleashes the potential of off-the-shelf machine learning methods. Second, the derivation of the risk and robustness guarantee can be made independent of the choice of the prediction model through a data-splitting idea. Third, our paradigm of predict-thencalibrate applies to both (risk-sensitive) robust and (risk-neutral) distributionally robust optimization (DRO) formulations. Theoretically, it gives new generalization bounds for the contextual LP problem and sheds light on the existing results of DRO for contextual LP. Numerical experiments further reinforce the advantage of the predict-then-calibrate paradigm in that an improvement on either the prediction model or the calibration model will lead to a better final performance.

1 Introduction

Contextual linear programming (LP) considers the linear programming problem with the presence of covariates, and it can be viewed as a prediction problem under a constrained optimization context where the output of the prediction model serves as the objective of a downstream LP. The goal is to develop a model (trained from past data) that prescribes a decision/solution for the downstream LP problem using directly the covariates but without observation of the objective function. A similar problem formulation was also studied as prescriptive analytics [Bertsimas and Kallus, 2020] and predict-then-optimize Elmachtoub and Grigas [2022]. The existing literature mainly develops two approaches for the problem: (i) end-to-end algorithms that map the covariates directly to a recommended decision for the LP or (ii) two-step algorithms that first predict the unobserved objective using the covariates and then solve the LP with the predicted objective. While the literature has been largely focused on the risk-neutral objective, our paper considers the risk-sensitive/robust

formulation of the problem. The robust formulation of the contextual LP can be viewed as both a risk-sensitive version of the risk-neutral objective in contextual LP and an adaptive/contextual version of the standard (context-free) robust optimization problem. In comparison with the robust optimization literature, the contextualized robust optimization problem allows a contextual and data-driven way to design the uncertainty set and ideally can utilize the contextual information to make less conservative decisions. The contextual/data-driven uncertainty set has been considered by several existing works on transportation (Guo et al. [2023]), portfolio management (Wang et al. [2022]), and healthcare (Gupta et al. [2020]). While these works focus on applications with special structures or distributions, our paper considers a generic setup and aims for a general solution to the problem.

In our paper, we make the following contribution:

First, we formulate an algorithm design paradigm of predict-then-calibrate and introduce the notion of uncertainty calibration/quantification to the problem of contextual optimization. The idea is to first develop a prediction model without worrying about the downstream robustness, and then tailor the uncertainty calibration methods to quantify the prediction uncertainty towards the robust objective.

Second, we develop two algorithms for robust contextual LP under the paradigm of predict-thencalibrate. The derivation of theoretical guarantees shows the advantage of the disentanglement of the prediction phase and the uncertainty calibration phase. Moreover, we quantify the advantage of a better prediction and/or a better uncertainty calibration both theoretically and empirically.

Third, we also draw a connection with the related problem of distributionally robust optimization. We utilize the same idea of predict-then-calibrate and derive a new generalization bound for the distributional robust version of contextual LP utilizing tools from the nonparametric regression.

In the following, we review the several streams of literature that are related to our work.

Conditional robust optimization. The problem studied in our paper can be viewed as a conditional version of the classic robust optimization problem. The robust optimization stems from the replacement of the risk-neutral expectation objective with a risk-sensitive robust objective such as Value-at-Risk(VaR)/quantile. The literature on robust optimization develops methods to approximate VaR by specifying the shape and size of an uncertainty set to guarantee that a VaR constraint would be satisfied Ghaoui et al. [2003], Chen et al. [2007], Natarajan et al. [2008]. Bertsimas et al. [2018] designs the uncertainty set with a finite-sample probabilistic guarantee on optimal solutions. Goerigk and Kurtz [2020] constructs the uncertainty set as the union of K convex sets by clustering historical data. While all these works concern the context-free setting, the contextual (or conditional) setting, or the problem of robust contextual LP, considers the presence of covariates in robust optimization. Along this line, Ohmori [2021] proposes a volume-minimized ellipsoid covering k -nearest samples in the contextual space, leading to a nonlinear programming problem. Chenreddy et al. [2022] extends the idea in Goerigk and Kurtz [2020] and novelly adopts a deep learning approach to the problem which identifies which clusters the uncertain parameters belong to based on the covariates. Compared to the end-to-end formulations Ohmori [2021], Chenreddy et al. [2022], we do not impose restrictions on the prediction model and have a larger flexibility in choosing the uncertainty quantification method.

Contextual LP/Predict-then-optimize. Our work complements the existing literature on contextual LP by considering a risk-sensitive objective. The existing works Donti et al. [2017], Elmachtoub and Grigas [2022], Ho-Nguyen and Kılınç-Karzan [2022], Hu et al. [2022] mainly study the risk-neutral objective for the problem and aim to develop prediction methods that account for the downstream optimization tasks. To model the remaining uncertainty after prediction in optimization, Sen and Deng [2018] considers training a parameterized point-wise estimation model first, and then approximating the conditional distribution by imposing randomness to the learned parameters empirically. Ban et al. [2019] proposes to forecast the distribution of new product demand by adopting a linear regression model to fit the demands of similar products with covariates, and then using the empirical distribution of covariate-independent residuals to build a sample average approximation (SAA) model. Some recent works Kannan et al. [2020, 2022, 2021] further generalize it to a residuals-based SAA framework to include more machine learning methods and adopt distributionally robust optimization to mitigate the overfitting under limited data. A key difference between our work and this line of works is that we consider a risk-sensitive objective, while these works consider a risk-neutral objective. In this light, our framework of predict-then-calibrate is compatible with any prediction model developed along this line, and it tailors the prediction model for a robustness task.

2 Problem Setup

Consider a linear program (LP) that takes the following standard form

where c ∈ R n , A ∈ R m × n , and b ∈ R m are the inputs of the LP. In addition, there is an available feature vector z ∈ R d that encodes the covariates (side information) associated with the LP.

The tuple ( c, A, b, z ) is drawn from an unknown probability distribution P . The problem of contextual LP (predict-then-optimize, or prescriptive analytics) usually assumes the availability of a learning (training) dataset sampled from the distribution P ,

During the test phase, one needs to recommend a feasible solution x new to a new LP problem using the observation of ( A new , b new , z new ) but without the observation of the objective vector c new:

The focus of the existing literature has been mainly on the risk-neutral objective

where the recommended decision x can be viewed as a function of ( A,b, z ) .

Alternatively, we consider a risk-sensitive objective

where α ∈ (0 , 1) . Here VaR α ( U ) denotes the α -quantile/value-at-risk (VaR) of a random variable U ; specifically, VaR α ( U ) := F -1 U ( α ) with F -1 U ( · ) be the inverse cumulative distribution function of U.

We can interpret the problem (3) in two ways: First, it can be viewed as a risk-sensitive version of the contextual LP problem (2). From the prediction viewpoint, the standard problem (2) concerns only the prediction of E [ c | z ] , while solving (3) may involve a distributional prediction of the conditional distribution c | z . Second, the classic robust LP problem considers a setting where there is no covariate, and the VaR is taken with respect to the (marginal) distribution of c . Comparatively, the problem (3) can be viewed as a conditionally robust version where the VaR is taken with respect to the conditional distribution c | z . The presence of the covariates z reduces our uncertainty on the objective c and thus will induce less conservative decisions with the same level of risk guarantee.

Throughout the paper, we work with a well-trained ML model ˆ f : R d → R n that predicts the objective with the covariates and is learned from the training data D l . We emphasize that we do not impose any assumption on the model, but assume the availability of a validation set D val := { ( c t , A t , b t , z t ) } T t =1 that can be used to quantify the uncertainty of ˆ f.

3 Robust Contextual LP

The problem (3) has been extensively studied under the context-free setting in the literature of robust optimization [Ben-Tal et al., 2009], and the key challenge is the non-convexity of the objective function. Equivalently, the problem (3) can be written as the following optimization problem

where the decision variables are x , c and a measurable set U . The last constraint is with respect to the set U , and the probability is taken with respect to the conditional distribution c | z. One issue to

solve this problem is the intractability in optimization over the uncertainty set. Particularly, even if the conditional distribution c | z is discrete, (4) is equivalent to a mixed integer linear programming problem, which is generally NP-hard and computationally intensive. Similarly, when solving a sample average approximation form of (4) (Bertsimas and Kallus [2020]), the problem becomes similar to the discrete case, and the intractability issue is still inevitable. To resolve this issue, one can solve the following approximation problem with a fixed uncertainty set U satisfying P c | z ( c ∈ U ) ≥ α ,

A box- or ellipsoid-shaped uncertainty set enables a tractable optimization of the problem (5). The uncertainty set hopes to cover the high-density region so that the approximation to (3) is tighter.

Now we present two algorithms that implement this approximation idea in the contextual setting. The key is to quantify the uncertainty of the prediction model. Both algorithms construct contextual uncertainty sets for the prediction model; that is, the size of the uncertainty set changes with respect to the covariates z . In addition, importantly, the construction of these uncertainty sets accounts for the tractability of the downstream robust optimization problem (5) by outputting the box and ellipsoid shapes, respectively. In both algorithms, we first split the available validation data into two sets D 1 and D 2 , and use the first set D 1 for a preliminary calibration, and the second set D 2 for an additional adjustment. The output from both algorithms gives a contextual uncertainty set U ( z ) that works as the input of (5) and thus solves the downstream robust optimization problem.

Algorithm 1 Box Uncertainty Quantification (BUQ)

1: Input: Dataset D val , ML model ˆ f , parameter α

2: Randomly split the validation data into two index sets D 1 ∪ D 2 = { 1 , ..., T } and D 1 ∩ D 2 = ∅

3: %% Preliminary calibration of the uncertainty sets

4: for t ∈ D 1 do

5: Calculate the residual vector on the t -th validation sample

and denote r t = ( r t 1 , ..., r tn ) ⊤

6: end for

7: Train a quantile regression model ˆ h ( z ) : R d → R n that minimizes

where ρ α ( · ) := α ( · ) + +(1 -α )( · ) + denotes the pinball loss

8: %% Additional adjustment of the size of the uncertainty sets

9: for t ∈ D 2 do

10: Let

11: end for

12: Choose a minimal η > 0 such that

where 1 {·} is the indicator function and the inequality is required to hold element-wise 13: Output: ˆ h , η

In Algorithm 1, we first compute the residual/error vector r t ∈ R n for samples in dataset D 1 . Then for each coordinate i = 1 , ..., n , we fit a quantile regression model ˆ h that predicts the quantile of the absolute error by minimizing (7). Specifically, the model ˆ h aims to predict the quantiles of each coordinate of the residual vector r t . Then, in the second step of Algorithm 1, we pretend the quantile model ˆ h is 'correct' and construct the uncertainty sets accordingly for samples in dataset

D 2 . Importantly, the uncertainty set is parameterized by a scalar η . We use the parameter η to perform an additional adjustment of the uncertainty set so that the coverage probability α is met on D 2 by (8). The algorithm outputs the function ˆ h and η , and it will output a box-shaped uncertainty set U (1) α ( z ) = [ ˆ f ( z ) -η ˆ h ( z ) , ˆ f ( z ) + η ˆ h ( z ) ] for a new sample with covariates z .

Algorithm 2 Ellipsoid Uncertainty Quantification (EUQ)

1: Input: Dataset D val , ML model ˆ f , parameter α

2: Randomly split the validation data into two index sets D 1 ∪ D 2 = { 1 , ..., T } and D 1 ∩ D 2 = ∅

3: %% Preliminary calibration of the uncertainty sets

4: for t ∈ D 1 do

5: Calculate the residual vector on the t -th validation sample

6: end for

7: Use D 1 to fit a model ˆ g : R d → R that predicts the α -quantile of ∥ r t ∥ 2 using the covariates z t

8: Fit a zero-mean normal distribution for the scaled residual vector

and denote the covariance matrix as ˆ Σ

9: %% Additional adjustment of the size of the uncertainty sets

10: Choose a minimal η > 0 such that

where 1 {·} is the indicator function and the inequality is required to hold element-wise.

11: Output: ˆ g , ˆ Σ , η

Algorithm 2 follows a similar spirit as Algorithm 1. It first computes the residual vector r t for samples in dataset D 1 , but differently from Algorithm 1, it fits a regression model for the norm ∥ r t ∥ 2 . Then it fits a zero-mean normal distribution for the scaled residual vector. The motivation is that the residual vector (up to a scale ˆ g ( z ) ) follows a multivariate normal distribution. We remark that although the algorithm design is motivated by such a setting of normal distribution, its theoretical guarantee does not rely on this normal structure. The key is the second step of additional calibration: it pretends the uncertainty set is 'correct' and uses an additional scalar η to make further adjustments such that the coverage probability α is met on D 2 by (9). For a new sample with covariates z , the algorithm outputs

In comparison, for the first step of preliminary calibration, Algorithm 1 quantifies the uncertainty of each coordinate of the residual vector individually, while Algorithm 2 presumes a Gaussian structure. We remark that the two algorithms provide two examples for the design of the preliminary calibration step, but these are not the only two options. As we will see shortly, the second step of additional adjustment is in charge of the coverage guarantee, so the first step can be further modified based on other prior knowledge of the underlying data/optimization problem.

3.1 Algorithm Analysis

The following proposition provides a coverage guarantee for both Algorithm 1 and 2 in that the output uncertainty set will cover a new sample with a probability of approximately α .

Proposition 1. For a new sample ( c, A, b, z ) from distribution P , denote the uncertainty sets output from Algorithm 1 and Algorithm 2 by U (1) α ( z ) and U (2) α ( z ) , respectively. The following inequalities hold for k = 1 , 2 ,

where the probability is with respect to the new sample ( c, A, b, z ) and dataset D 2 .

The proposition gives the following insights into the two algorithms and the general problem of robust contextual LP. First, the second step (in both algorithms) of the adjustment via the scalar η is the key to the coverage guarantee (10), with several advantages. First, as the concentration argument is made with respect to only this second step, the bound is rather tight and free of unnecessary conservation in the uncertainty set construction. Second, it allows full flexibility in the choice of the prediction model ˆ f and the preliminary calibration model ˆ h (Algorithm 1) and ˆ g (Algorithm 2). This reinforces the notion of predict-then-calibrate. The design of the prediction model ˆ f can be solely targeted on the accuracy, and the coverage guarantee can be deferred and taken care of by the calibration steps such as these two algorithms.

Furthermore, we note the probability in (10) is taken with respect to only the validation data and the new sample. It means Proposition 1 holds even if the training data D l (for obtaining ˆ f ) is not sampled from the distribution P . In other words, the result only requires the validation data D val ∼ P and it is robust to a distribution shift of the training distribution when D l ̸∼ P . In a broader context, when the training data, validation data, and test data come from different distributions, this situation is commonly referred to as the out-of-domain (OOD) problem. We refer to the survey papers Shen et al. [2021], Yang et al. [2021] for a more detailed discussion of this problem. Developing more coverage guarantees with OOD data will require future investigations, and we will list this problem as a future direction.

In addition, the coverage guarantee in Proposition 1 can be extended to a robustness guarantee for the contextual LP problem as in the following corollary.

Corollary 1. For a new sample ( c, A, b, z ) from distribution P , denote the uncertainty set output from Algorithm 1 or Algorithm 2 by U α ( z ) . Let x ∗ ( z ) and OPT ( z ) be the optimal solution and the optimal value of LP ( U α ( z )) (5) . Then we have

where the probability is taken with respect to the new sample ( c, A, b, z ) and D 2 .

To end this section, we remark that although the coverage guarantee (10) is in a population sense where the probability is taken with respect to both c and z , our framework of predict-then-calibrate also has the flexibility to achieve an individual coverage by changing the calibration algorithm under some assumptions. Here, the individual coverage means that the conditional probability P ( c ∈ U α ( z ) | z ) = α holds for any covariate z as the constraint in (4), where U α ( z ) denotes any given contextualized uncertainty set. However, the above-mentioned assumption to achieve individual coverage cannot be verified from the data prior, and achieving this coverage is, in general, a hard problem, although there have been many attempts in the literature on conformal prediction and model calibration. In addition, we want to emphasize that, compared to achieving individual coverage, the most significant focus of this paper is to introduce the idea of uncertainty quantification to solve robust optimization, which disentangles the prediction from the calibration. As a result, we delay the discussion of the individual coverage to Appendix B.1.

3.2 Value of Better Prediction and Contextual Uncertainty Set

Two important components in the framework of predict-then-calibrate are (i) the flexibility in choosing the prediction model ˆ f and (ii) the contextual uncertainty set U α ( z ) . In the following, we illustrate the importance of these two aspects via simple examples.

The value of better prediction. Consider a single-variable LP

where the objective is determined by c = ∑ d i =1 z i -d · ϵ. Here z i and ϵ are sampled from an exponential distribution Exp (1) for all i = 1 , .., d . We note that z i can be interpreted as either the covariate itself or some useful latent factor extracted by the prediction model. Consider a prediction model that can only utilize/extract the first k covariates/latent factors z 1: k = ( z 1 , ..., z k ) . The optimal prediction model would be f k ( z 1: k ) = ∑ k i =1 z i , and also we assume it has perfect uncertainty quantification denoted by U α ( z 1: k ) . We denote the accordingly robust solution by x ∗ α ( z 1: k ) , and thus all these solutions for k = 1 , ..., d meet the coverage guarantee.

Proposition 2. For any α ∈ (0 . 5 , 1) and k ≥ 1

and it decreases monotonously with respect to k. Here, γ ( · , · ) denotes the lower incomplete gamma function.

Proposition 2 gives an analytical expression for the probability of the optimal robust solution equal to zero. A zero solution can be viewed as a conservative solution when the prediction model is uncertain about the objective c . While we assume the prediction model is equipped with perfect uncertainty quantification, the proposition says that more informative covariates and/or more powerful prediction models will lead to less conservative solutions for the downstream robustness task. The reason is that a better prediction model can make the uncertainty quantification task easier by smoothing the conditional distribution of the residual r | z with respect to z . For example, when k = d in Proposition 2, the distributions of residuals are identical and thus very smooth for different covariates, which is easier to quantify than the case with k = 0 . This reason can also be extended to general cases.

The value of better calibration. Consider the LP (11) again but with c specified by

where z ∈ R is sampled uniformly from [ -1 , 1] , and ϵ is sampled from the uniform distribution on [ -0 . 5 , 0 . 5] . In this case, the sign of the objective c can be deterministically determined by the sign of z . With perfect knowledge of the underlying uncertainty, the following solution is the optimal robust solution for all α ∈ (0 , 1)

The following proposition gives another uncertainty set with the optimal prediction model and a coverage guarantee but a suboptimal robust solution.

Proposition 3. For α ≥ 1 / 2 , let the uncertainty set

The uncertainty set has a coverage guarantee in that P ( c ∈ U α ( z )) = α. If we solve the optimization problem (5) with the uncertainty set U α ( z ) , the following robust solution is obtained

The robust solution induced by the uncertainty set U α ( z ) is unnecessarily conservative compared to the optimal robust solution x ∗ ( z ) . This highlights the importance of good uncertainty quantification, and in particular, contextualized uncertainty quantification. Specifically, the suboptimality of x ( z ) arises from that the uncertainty set U α ( z ) in the proposition is not much contextualized with respect to z , and this justifies the contextualized uncertainty models ˆ h and ˆ g for the preliminary calibration step in Algorithm 1 and Algorithm 2.

In addition to the contextualized uncertainty quantification, we remark the flexibility in choosing the calibration model or uncertainty set can bring us two benefits. First, it has the potential to further reduce decision conservatism by collaborating with new designs of small uncertainty sets. Specifically, we can design predict-then-calibrate algorithms with even small uncertainty sets based on many previous works that focus on designing minimum size uncertainty sets to achieve a given coverage guarantee in the context-free robust optimization or distributional robust optimization literature (Bertsimas et al. [2018], Gupta [2019]). Second, this framework is also applicable to more general optimization problems such as convex programs or multi-stage stochastic programs. In

particular, for these general problems, if the robust VaR formulation has a tractable construction of the uncertainty set in the context-free setting, we can then design a predict-then-calibrate algorithm by constructing the corresponding contextual uncertainty set accordingly. To close this section, we emphasize that our predict-then-calibrate framework behaves more like a plug-in module that can fit into the existing robust optimization literature, and thus, it is potentially capable of handling more problems than the contextual LP loss discussed in this paper.

4 Extension to Distributionally Robust Contextual LP

In the previous sections, we study the problem of robust contextual LP and the framework of predictthen-calibrate. Now we turn to a related but different problem - distributionally robust contextual LP - through the lens of predict-then-calibrate. We relate the objectives of robustness and distributional robustness with the two types of uncertainty in the uncertainty quantification literature. We show the paradigm of predict-then-calibrate can also be extended to the distributionally robust contextual LP and obtain a new generalization bound for the problem.

The distributionally robust contextual LP concerns the following problem:

where Ξ γ denotes a set of distributions that describes the uncertainty on c . Ideally, one hopes that the uncertainty set covers the true conditional distribution c | z. Compared to the robust version that concerns the VaR/Quantile of the objective c , the formulation here still concerns the risk-neutral objective but accounts for the uncertainty in estimating E [ c | z ] . The distributionally robust optimization has two roles: first, it can be used to derive solutions with provable guarantees on generalization, and second, it induces a regularization effect for the underlying prediction model.

ℚ0.95[GLYPH(cmap:d835)]

ℚ0.05[GLYPH(cmap:d835)]

RO

epistemic+aleatoric

GLYPH(cmap:d835)[GLYPH(cmap:d835)]

DRO

epistemic

ℙ(GLYPH(cmap:d835))

GLYPH(cmap:d835)

As in the previous section, we denote the prediction error on the validation data D val by

The following proposition states that there exists a distributionally robust algorithm that achieves a generalization bound that is oblivious with respect to the underlying prediction model ˆ f.

Proposition 4. Assume there exists a constant L such that the following condition holds for any z, z ′

Then under mild boundedness assumptions on the distribution, there exists a distributionally robust optimization algorithm that utilizes the validation data D val and outputs a solution ˆ x ( z ) such that the following inequality holds with high probability

We defer the detailed proof and the DRO algorithm to Appendix C.5. Specifically, the DRO algorithm performs a nonparametric uncertainty quantification with respect to r t 's and thus it is oblivious to the underlying prediction model ˆ f. We remark that such idea of calibrating r t 's is not new and has been exploited in Wang et al. [2021], Kannan et al. [2020, 2021]. But notably, our result requires a weaker condition on the underlying ˆ f, where the previous works rely on a realizability condition of the prediction model. Moreover, compared with the generalization bound in the contextual LP literature Liu and Grigas [2021], although Liu and Grigas [2021] develops an O ( T -1 / 4 ) generalization bound, they still require the function class of the prediction model can have bounded Rademacher complexity. If that condition holds here, our generalization bound can also be improved accordingly.

5 Experiment

In this section, we illustrate the performance of our proposed algorithms via one simple example and also a shortest path problem considered in Elmachtoub and Grigas [2022] and Hu et al. [2022]. We defer more results including more visualizations, another experiment on the fractional knapsack problem Ho-Nguyen and Kılınç-Karzan [2022], and experiment setups to the Appendix. We implement our Algorithm 1 (PTC-B) and Algorithm 2 (PTC-E) against several benchmark methods including the Ellipsoid method that ignores the covariates, k -nearest neighbor (kNN) approach in Ohmori [2021], the Deep Cluster-Classify (DCC), and Integrated Deep Cluster-Classify (IDCC) approaches in Chenreddy et al. [2022]. More experiment details are deferred to Appendix A.

z

1

1.0

0.5

0.0

0.5

1.0

c

Uncertainty set

KR

NN

0.4

0.2

0.0

0.2

0.4

z

1

NN

KR

Optimal

Robust solution

x

x

=

1

x

=0

x

=1

(a) Prediction

z

1

1.0

0.5

0.0

0.5

1.0

c

Uncertainty set

Linear

GBR

NN

0.4

0.2

0.0

0.2

0.4

z

1

NN

GBR

Linear

Optimal

Robust solution

x

x

=

1

x

=0

x

=1

(b) Uncertainty quantification

Simple LP Visualization

In the first experiment, we study the LP problem (11) where the optimal solution and the uncertainty set can be visualized. Here the covariates z = ( z 1 , ..., z d ) ⊤ where z i is sampled from Unif [ -0 . 5 , 0 . 5] independently for i = 1 , ..., d , ϵ ∼ Unif [ -0 . 5 , 0 . 5] , and c = ( sign ( z 1 ) + ϵ ) · √ | z 1 | (c is independent of z 2 , ..., z d ). We consider a risk level of α = 0 . 8 and dataset size T = 1000 . For our methods of PTC-B and PTC-E, we use 60% of the data for training ˆ f, 20% for preliminary calibration ( D 1 ), and 20% for final adjustment ( D 2 ). In Figure 2, the robust solutions obtained from different prediction methods and different calibration methods are plotted. The results reinforce the discussions in Section 3.2 that either a better prediction model or a better calibration model will improve the final performance of the robust optimization problem.

Shortest path problem

Now we present numerical experiments for a shortest path problem on a 5 × 5 grid with 40 edges, and the cost of traveling through edge i is c i for i = 1 , ..., n . We follow the experiment setup in

5

6

7

8

9

10

MSE (×100)

1.70

1.75

1.80

1.85

1.90

Avg. VaR (×1000)

Random Forest

NN

OLS

KernelRidge-rbf

KernelRidge-poly

2

5

10

20

50

100

T (×100)

1.8

2.0

2.2

2.4

2.6

Avg. VaR (×1000)

Ellipsoid

DCC

IDCC

PTC-B

PTC-E

(a) Average VaR

(b) Average coverage

Elmachtoub and Grigas [2022] and Hu et al. [2022] where the details are deferred to Appendix A.2. In this experiment, we demonstrate several aspects of our algorithms. First, Figure 3 shows the averaged VaR of PTC-B solutions (under different prediction models ˆ f ) versus the Mean Squared Error (MSE) on the test data, which reveals a positive correlation between the predictive performance and the quality of the solution obtained in the downstream robust optimization task. We utilize the Kernel Ridge method with the RBF kernel-identified as the top-performing model in Figure 3-as the predictive model and the Neural Network (NN) model as the preliminary calibration model for both PTC-B and PTC-E in the ensuing experiments (Figure 4 and Table 1) to compare with other algorithms. Second, Figure 4 demonstrates that our algorithms exhibit better sample efficiency than the benchmark algorithms as we increase the number of training samples. Specifically, while the model DCC and IDCC take advantage of the expressiveness of a complicated deep learning architecture, they can be more costly in terms of the required training samples. Furthermore, we compare the performance under varying risk level settings in Table 1. The Ellipsoid method does not utilize the covariates information, so although it has a coverage guarantee, it performs worst on the VaR. The kNN method [Ohmori, 2021] has a good VaR performance, but it lacks confidence adjustment, so it is hard to adapt to different confidence levels. The two deep learning-based approaches and ours give good coverage guarantees, while we attribute the advantage of our algorithms to the flexibility in choosing the prediction model and the calibration model.

Conclusion. In this paper, we study the robust contextual LP problem which lies at the intersection of contextual LP/predict-then-optimize and robust optimization. We develop two algorithms for the problem which output box- and ellipsoid shape uncertainty sets. The algorithms introduce a new perspective for contextualized uncertainty calibration, and it highlights the convenience brought by the disentanglement of the prediction and the uncertainty calibration. Yet, as mentioned earlier, the coverage guarantee in Proposition 1 is with respect to a population/average sense. Developing algorithms with individual-level coverage guarantees deserves more future investigation. Also, we hope our work provides a starting point for future research on risk-sensitive and robust decisionmaking for the problem of contextual optimization and optimization with machine-learned advice.

References

Naomi S Altman. An introduction to kernel and nearest-neighbor nonparametric regression. The American Statistician , 46(3):175-185, 1992.

Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511 , 2021.

Gah-Yi Ban and Cynthia Rudin. The big data newsvendor: Practical insights from machine learning. Operations Research , 67(1):90-108, 2019.

Gah-Yi Ban, Jérémie Gallien, and Adam J Mersereau. Dynamic procurement of new products with covariate information: The residual tree method. Manufacturing & Service Operations Management , 21(4):798-815, 2019.

Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. The Annals of Statistics , 51(2):816-845, 2023.

Aharon Ben-Tal, Laurent El Ghaoui, and Arkadi Nemirovski. Robust optimization , volume 28. Princeton university press, 2009.

Dimitris Bertsimas and Nathan Kallus. From predictive to prescriptive analytics. Management Science , 66(3):1025-1044, 2020.

Dimitris Bertsimas and Nihal Koduri. Data-driven optimization: A reproducing kernel hilbert space approach. Operations Research , 70(1):454-471, 2022.

Dimitris Bertsimas and Bart Van Parys. Bootstrap robust prescriptive analytics. Mathematical Programming , 195(1-2):39-78, 2022.

Dimitris Bertsimas, Vishal Gupta, and Nathan Kallus. Data-driven robust optimization. Mathematical Programming , 167:235-292, 2018.

Dimitris Bertsimas, Christopher McCord, and Bradley Sturt. Dynamic optimization with side information. European Journal of Operational Research , 304(2):634-651, 2023.

Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration inequalities: A nonasymptotic theory of independence . Oxford university press, 2013.

Leo Breiman. Classification and regression trees . Routledge, 2017.

Junyu Cao and Rui Gao. Contextual decision-making under parametric uncertainty and data-driven optimistic optimization. Available at Optimization Online , 2021.

Ningyuan Chen, Saman Lagzi, and Joseph Milner. Using neural networks to guide data-driven operational decisions. Available at SSRN 4217092 , 2022.

Xin Chen, Melvyn Sim, and Peng Sun. A robust optimization perspective on stochastic programming. Operations Research , 55(6):1058-1071, 2007.

Abhilash Reddy Chenreddy, Nymisha Bandi, and Erick Delage. Data-driven conditional robust optimization. Advances in Neural Information Processing Systems , 35:9525-9537, 2022.

Youngseog Chung, Willie Neiswanger, Ian Char, and Jeff Schneider. Beyond pinball loss: Quantile methods for calibrated uncertainty quantification. Advances in Neural Information Processing Systems , 34:10971-10984, 2021.

Peng Cui, Wenbo Hu, and Jun Zhu. Calibrated reliable regression using maximum mean discrepancy. Advances in Neural Information Processing Systems , 33:17164-17175, 2020.

Priya Donti, Brandon Amos, and J Zico Kolter. Task-based end-to-end model learning in stochastic optimization. Advances in Neural Information Processing Systems , 30, 2017.

Adam N Elmachtoub and Paul Grigas. Smart "predict, then optimize". Management Science , 68(1): 9-26, 2022.

Nicolas Fournier and Arnaud Guillin. On the rate of convergence in wasserstein distance of the empirical measure. Probability theory and related fields , 162(3-4):707-738, 2015.

Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. The limits of distribution-free conditional predictive inference. Information and Inference: A Journal of the IMA , 10(2):455-482, 2021.

Laurent El Ghaoui, Maksim Oks, and Francois Oustry. Worst-case value-at-risk and robust portfolio optimization: A conic programming approach. Operations Research , 51(4):543-556, 2003.

Marc Goerigk and Jannis Kurtz. Data-driven robust optimization using unsupervised deep learning. arXiv preprint arXiv:2011.09769 , 2020.

Zhen Guo, Bin Yu, Wenxuan Shan, and Baozhen Yao. Data-driven robust optimization for contextual vehicle rebalancing in on-demand ride services under demand uncertainty. Transportation Research Part C: Emerging Technologies , 154:104244, 2023.

Vishal Gupta. Near-optimal bayesian ambiguity sets for distributionally robust optimization. Management Science , 65(9):4242-4260, 2019.

Vishal Gupta, Brian Rongqing Han, Song-Hee Kim, and Hyung Paek. Maximizing intervention effectiveness. Management Science , 66(12):5576-5598, 2020.

Peter Hall, Rodney CL Wolff, and Qiwei Yao. Methods for estimating a conditional distribution function. Journal of the American Statistical association , 94(445):154-163, 1999.

Lauren Hannah, Warren Powell, and David Blei. Nonparametric density estimation for stochastic optimization with an observable state variable. Advances in Neural Information Processing Systems , 23, 2010.

Chin Pang Ho and Grani A Hanasusanto. On data-driven prescriptive analytics with side information: A regularized nadaraya-watson approach. URL: http://www. optimization-online. org/DB FILE/2019/01/7043. pdf , 2019.

Nam Ho-Nguyen and Fatma Kılınç-Karzan. Risk guarantees for end-to-end prediction and optimization processes. Management Science , 68(12):8680-8698, 2022.

Yichun Hu, Nathan Kallus, and Xiaojie Mao. Fast rates for contextual linear optimization. Management Science , 68(6):4236-4245, 2022.

Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning , 110:457-506, 2021.

Nathan Kallus and Xiaojie Mao. Stochastic optimization forests. Management Science , 2022.

Rohit Kannan, Güzin Bayraksan, and James R Luedtke. Residuals-based distributionally robust optimization with covariate information. arXiv preprint arXiv:2012.01088 , 2020.

Rohit Kannan, Güzin Bayraksan, and James Luedtke. Heteroscedasticity-aware residuals-based contextual stochastic optimization. arXiv preprint arXiv:2101.03139 , 2021.

Rohit Kannan, Güzin Bayraksan, and James R Luedtke. Data-driven sample average approximation with covariate information. arXiv preprint arXiv:2207.13554 , 2022.

Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International conference on machine learning , pages 2796-2804. PMLR, 2018.

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems , 30, 2017.

Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Association , 113(523): 1094-1111, 2018.

Heyuan Liu and Paul Grigas. Risk bounds and calibration for a smart predict-then-optimize method. Advances in Neural Information Processing Systems , 34:22083-22094, 2021.

Shang Liu, Zhongze Cai, and Xiaocheng Li. Distribution-free model-agnostic regression calibration via nonparametric methods. arXiv preprint arXiv:2305.12283 , 2023.

Gar Goei Loke, Qinshen Tang, and Yangge Xiao. Decision-driven regularization: A blended model for predict-then-optimize. Available at SSRN 3623006 , 2022.

Elizbar A Nadaraya. On estimating regression. Theory of Probability & Its Applications , 9(1): 141-142, 1964.

Karthik Natarajan, Dessislava Pachamanova, and Melvyn Sim. Incorporating asymmetric distributional information in robust value-at-risk optimization. Management Science , 54(3):573-585, 2008.

Viet Anh Nguyen, Fan Zhang, Jose Blanchet, Erick Delage, and Yinyu Ye. Distributionally robust local non-parametric conditional estimation. Advances in Neural Information Processing Systems , 33:15232-15242, 2020.

Viet Anh Nguyen, Fan Zhang, Jose Blanchet, Erick Delage, and Yinyu Ye. Robustifying conditional portfolio decisions via optimal transport. arXiv preprint arXiv:2103.16451 , 2021.

Shunichi Ohmori. A predictive prescription using minimum volume k-nearest neighbor enclosing ellipsoid and robust optimization. Mathematics , 9(2):119, 2021.

R Tyrrell Rockafellar, Stanislav Uryasev, et al. Optimization of conditional value-at-risk. Journal of risk , 2:21-42, 2000.

Suvrajeet Sen and Yunxiao Deng. Learning enabled optimization: Towards a fusion of statistical learning and stochastic programming. INFORMS Journal on Optimization (submitted) , 2018.

Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribution generalization: A survey. arXiv preprint arXiv:2108.13624 , 2021.

Vladimir Vovk. Conditional validity of inductive conformal predictors. In Asian conference on machine learning , pages 475-490. PMLR, 2012.

Tianyu Wang, Ningyuan Chen, and Chun Wang. Distributionally robust prescriptive analytics with wasserstein distance. arXiv preprint arXiv:2106.05724 , 2021.

Yijie Wang, Grani A Hanasusanto, and Chin Pang Ho. Robust contextual portfolio optimization with gaussian mixture models. 2022.

Geoffrey S Watson. Smooth regression analysis. Sankhy¯ a: The Indian Journal of Statistics, Series A , pages 359-372, 1964.

Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:2110.11334 , 2021.

Shengjia Zhao, Tengyu Ma, and Stefano Ermon. Individual calibration with randomized forecasting. In International Conference on Machine Learning , pages 11387-11397. PMLR, 2020.

Taozeng Zhu, Jingui Xie, and Melvyn Sim. Joint estimation and robustness optimization. Management Science , 68(3):1659-1677, 2022.

A Supplementary for Experiment

Details of benchmarks. For the DCC and IDCC algorithms proposed in Chenreddy et al. [2022], we set the number of clusters to be 10. This choice of the number of clusters is large enough because there are clusters containing no training samples. The Ellipsoid method, which is also a benchmark used in Chenreddy et al. [2022], fits a Gaussian distribution without contextual information and calibrates the radius to cover a proportion of α training samples. For the k NN conditional RO method [Ohmori, 2021], the hyperparameter, k , lacks the flexibility to adjust with respect to the number of training samples T , the dimensionality of the problem, and the risk level α , so we empirically choose k = max { √ T, 2 × n } here. Different algorithms utilize the same dataset to ensure fairness in each round of comparison. We redefine T as its size in the experiment section. For our proposed Algorithm 1 (PTC-B) and 2 (PTC-E) under the PTC framework, the dataset is randomly split into 60%, 20%, and 20% to do prediction, preliminary calibration, and final adjustment, respectively.

Performance measurements. In the test phase, we randomly generate z from its marginal distribution. Then, for each z , we generate a set with 1000 objective vectors, { c t } 1000 t =1 , from P c | z . Given any robust solution x conditioned on z , we estimate the VaR α [ c ⊤ x | z ] by the α -quantile of { c t ⊤ x } 1000 t =1 and take its empirical expectation over z as the average VaR. Similarly, we estimate the average coverage by the average frequency of { c ∈ U z } .

A.1 Simple LP Visualization

We select NN as the prediction and preliminary calibration models in both PTC-B and PTC-E algorithms and compare them to benchmarks. Here, because c is only 1-dimensional, we replaced the DCC and IDCC algorithms, which are based on deep neural networks, with another clustering method, K-Means. Figure 5 shows the performance of different algorithms when d increases ( d = 1 , 4 , and 16 ), and the plot is shown by fixing z 2 , ..., z d to 0 in the test phase. The Ellipsoid method performs the worst and gives an unchanged uncertainty set for all different z due to its ignorance of the contextual information. All algorithms degrade with an increase in irrelevant information, but localized methods such as kNN and K-Means exhibit more severe deterioration. Our proposed methods, however, maintain a certain degree of stability.

1.0

0.5

0.0

0.5

1.0

c

0.4

0.2

0.0

0.2

0.4

z

1

PTC-E

PTC-B

K-Means

Ellipsoid

Optimal

0.4

0.2

0.0

0.2

0.4

z

1

0.4

0.2

0.0

0.2

0.4

z

1

Ellipsoid

K-Means

PTC-B

PTC-E

x

=

1

x

=0

x

=1

A.2 Shortest Path Problem

In this section, we consider a shortest path problem on a 5 × 5 grid with 40 edges, and the cost of traveling through edge i is c i for i = 1 , ..., 40 . After observing the covariates z (e.g., weather conditions, day of the week), we aim to select a route from the left-top vertex to the right-bottom one to minimize the value-at-risk travel time, i.e., VaR α,z ( c ⊤ x ) . The data generation process is as follows. First, we generate a 0-1 matrix Θ ∈ R 40 × d once with random seed 0 and fix it to encode the parameters of the true model, where each entry is generated from a Bernoulli distribution with probability 0.5. Then, the covariate vector z t ∈ R d is generated from N (0 , I d ) for t = 1 , ..., T .

0.6

0.7

0.8

0.85

0.9

0.95

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Scaled VaR

Ellipsoid

DCC

IDCC

PTC-B

PTC-E

Given z t , the cost on edge i is c t i = [( 1 √ d (Θ z t ) + 3) 5 +1] · ϵ i , where ϵ i ∼ Unif [ 3 4 , 5 4 ] independently for i = 1 , ..., 40 . We implement d = 10 here. To simulate the uneliminated irrelevant information, we set the last two dimensions of z to be independent of c , so their corresponding column vectors in Θ are set to be zero. In the test phase, we evaluate the performance on 500 z 's generated from the marginal distribution of z . We select the Kernel Ridge method with the RBF kernel as our prediction model and the NN model as our preliminary calibration model in both PTC-B and PTC-E.

In Figure 6, we show the box plot version of Table 1 with scaled VaR. Here, to alleviate the impact of randomness on z , we scale the VaR by the optimal value of the corresponding conditional stochastic program with expectation objective, i.e., VaR α / Obj E , where Obj E = max Ax = b,x ≥ 0 E [ c | z ] ⊤ x . Figure 6 once again demonstrates the superior performance of our algorithms in minimizing VaR as that in Table 1.

A.3 Fractional Knapsack Problem

In this section, we consider a fractional knapsack problem and follow the setup in Ho-Nguyen and Kılınç-Karzan [2022]. This problem can be interpreted as a maximization problem, wherein a consumer aims to maximize their utility by selecting items under a budget constraint. Specifically, to fit our formulation, the problem can be written as

Here, c is the utility vector and we use a minimization version to represent a maximization problem. p ∈ R n is some fixed positive vector indicating the price of items, and B > 0 is the budget. The same as the shortest path problem, we randomly generate a 0-1 matrix Θ ∈ B 20 × d once with the last 2 columns entirely filled as zeros to indicate the independence of the last dimensions of z to c . Then, we generate multivariate independent z from Unif [0 , 4] d , and c = (Θ z ) 2 · ϵ , where ϵ is multivariate independent and following Unif [ 4 5 , 6 5 ] n . We conduct experiments on data generated under T = 5000 , d = 10 , and n = 20 , and we choose the Kernel Ridge method with RBF kernel as our prediction model, and the NN as our preliminary calibration model in both the PTC-B and PTC-E.

Given a new covariate z , the conditional RO algorithm will output an uncertainty set. Under different constraints, the worst-case scenario corresponding to the uncertainty set may vary, so we compare the performance under randomly generated constraints. We generate the price vector p with each entry as an integer uniformly sampled from [1 , 1000] . Then, we randomly generated u ∼ Unif [0 , 1] , and the

0.6

0.7

0.8

0.85

0.9

0.95

17.5

15.0

12.5

10.0

7.5

5.0

2.5

0.0

Scaled VaR

Ellipsoid

DCC

IDCC

PTC-B

PTC-E

budget B is sampled from Unif [max j p j , 1 ⊤ p -u · max j p j ] . Here, we randomly generated 10 sets of constraints { ( p i , B i ) } 10 i =1 (with random seed 0 ) and fix them. In the test phase, the average VaR and coverage are taken over 100 z 's and 10 constraints ( 1000 instances in total).

Table 2 associated with Figure 7 displays the comparison results under varying risk levels and demonstrates the advantages of our proposed algorithms again via the lower VaR and proper coverage that closely matches the risk level, where the scaled VaR in Figure 7 is the VaR divided by max x ∈X E [ c | z ] ⊤ x . The contextual-ignorance method, Ellipsoid, performs the worst with the total utilities remaining 0 , which shows its excessive conservatism by specifying the worst-case utility of each item to be non-positive and selecting nothing. The DCC and IDCC, as well as our PTC-B and PTC-E methods, exhibit better performance than the kNN method due to our utilization of expressive NN models, which are not limited to utilizing only local information. By virtue of the flexibility of our algorithms in the model selection, the chosen Kernel Ridge regression model is more suitable than the clustering-based DCC and IDCC methods for this problem with continuous covariates, so our algorithms achieve relatively better performance than theirs.

Table 2: Average VaR and coverage on fractional knapsack problems under different risk level settings. The result is reported based on 500 simulation trials.

(a) Average VaR

(b) Average coverage

B Supplementary for Section 3

This section provides discussions about the individual coverage and proofs for theoretical statements in Section 3. In Section B.1, we will provide a brief literature review about global coverage and individual coverage, and then, provide an algorithm to achieve the individual coverage with

corresponding theoretical results. In Sections B.2, B.3, and B.4, we will show Propositions 1, 2, 3, respectively. The proof will utilize some useful lemmas in Section D.

B.1 Individual Coverage

The uncertainty quantification literature arises from two communities. It is referred to as "conformal prediction" in the field of statistics and "model calibration" in the field of machine learning. In the following, we briefly review the results of global and individual coverage in uncertainty quantification in those two communities, respectively.

Conformal prediction: Milestone works on the topic, Lei et al. [2018], Barber et al. [2023], focus on global coverage. Angelopoulos and Bates [2021], a survey paper on conformal prediction, also focuses on the global guarantee. For conditional/individual coverage, Foygel Barber et al. [2021], Vovk [2012] state an 'impossible triangle' in uncertainty quantification: (i) the coverage result is built for conditional coverage; (ii) it makes no assumptions on the underlying distribution; (iii) it has finite-sample guarantee rather than asymptotic consistency. Essentially, in our paper, Algorithms 1 and 2 basically fail to meet (i), but they barely rely on any assumptions on the underlying distribution and enjoy a finite-sample guarantee, that is, achieving (ii) and (iii). Later in this section, we will introduce a nonparametric algorithm that achieves conditional coverage, and the coverage can be finite-sample, i.e., achieving (i) and (iii), while this result requires additional assumptions on the underlying distribution, which is a violation of (ii).

Model calibration/recalibration: Lakshminarayanan et al. [2017], Cui et al. [2020], Chung et al. [2021], Kuleshov et al. [2018], Zhao et al. [2020] are a few state-of-the-art papers on calibrating the uncertainty of a regression model. We note that none of these papers claim to achieve a conditional coverage guarantee. Rather, they ensure a global guarantee and strive for better empirical conditional coverage. In particular, Zhao et al. [2020] gives a pessimistic result on conditional coverage that it is impossible to verify whether an algorithm achieves individual coverage. However, the proof is based on a very special case, and it is possible to rule it out with some mild conditions.

In the following, we present Algorithm 3 under our predict-then-calibrate framework, which can achieve individual coverage.

Algorithm 3 Uncertainty Quantification with Individual Coverage

1: Input: Validation data D val , ML model ˆ f , kernel choice K h ( · , · ) , bandwidth h , target covariate z 0

2: For each t ∈ D val , let

3: Define the estimation of the conditional distribution r | z = z 0 :

where δ r denotes the delta distribution on r

4: Choose a minimal η > 0 such that

5: Output: Uncertainty set U α ( z 0 ) = { c : ∥ c -ˆ f ( z 0 ) ∥ 2 ≤ η }

Here, this uncertainty quantification algorithm is mainly based on a nonparametric estimation in Hall et al. [1999], Liu et al. [2023], which is also similar to the nonparametric estimation in Algorithm 4. The estimator in Step 3 in Algorithm 3 covers a wide range of non-parametric estimators with different kernels, including k -nearest neighbors, and it can approximate the conditional distributions of the residuals r | z = z 0 with a sublinear convergence rate under some Lipschitz conditions. As a result, by using this estimator, we can directly calculate the size of the uncertainty set so that the estimated conditional distribution attains individual coverage. That is Step 4 in Algorithm 3. In the

following, we will provide a statement without proof since the analysis is almost the same as the proof of Proposition 4. We also refer to Hall et al. [1999], Liu et al. [2023] for a detailed analysis.

Proposition 5. Assume there exists a constant L ′ such that the following Lipschitz condition holds for any z, z ′

where TV ( · , · ) denotes the total variation distance between two distributions, and P r | z , P r | z ′ denotes the conditional distribution of the residuals given covariates z or z ′ , respectively. Then under mild boundedness assumptions on the distribution, the uncertainty set U α ( z 0 ) output from Algorithm 3 satisfies

Moreover, it is worth noting that, through this approximation of the conditional distributions, we can extend our framework in tractable optimizing conditional value-at-risk besides VaR in (3). Specifically, one can generate a number of independent samples { ˜ c k } K k =1 from the approximation of the conditional distribution c | z = z 0 . Then to optimize the following empirical CVaR objective (Rockafellar et al. [2000])

This is a convex program that has an equivalent linear program and is thus computationally tractable.

To close this section, we want to emphasize that the Lipschitz assumption in Proposition 5 cannot be verified from the data prior. Also, we prefer not to claim we have solved the conditional coverage problem for two reasons: (i) We do not want to give an impression to the robust optimization community that conditional coverage is an easy task to achieve. One should proceed with caution when applying the nonparametric algorithm; (ii) We don not want to give an impression to the statistics and ML communities working on uncertainty quantification problems that we underestimate the difficulty of the conditional coverage and lack of understanding of the related literature.

B.2 Proof for Proposition 1 and Corollary 1

In this section, we first show Proposition 1, then show Corollary 1 based on the result of Proposition 1.

Proposition 1. For a new sample ( c, A, b, z ) from distribution P , denote the uncertainty sets output from Algorithm 1 and Algorithm 2 by U (1) α ( z ) and U (2) α ( z ) , respectively. The following inequalities hold for k = 1 , 2

where the probability is with respect to the new sample ( c, A, b, z ) and dataset D 2 .

Proof. Denote n = |D 2 | as the number of samples in the D 2 . In the following, we show for k = 1 , 2 ,

where ⌈·⌉ denotes the ceiling function. If it holds, we can prove this proposition by the following inequalities

To show (12), we will use the exchangeability of the dataset D 2 and new sample ( c, A, b, z ) . Specifically, denote { c t } n t =1 as the samples in D 2 , and denote c n +1 as the new sample. Then, we define η t

for t = 1 , ..., n +1 with respect to Algorithms 1 or 2 as follows

In the proof below, we only show (12) for Algorithm 1, and the proof for Algorithm 2 can be obtained similarly. Based on the choice of the uncertainty set and definition of η in Algorithm 1, we have that η is the ⌈ α ( n +1) ⌉ n -upper quantile in { η t } n t =1 . This choice of η also implies c t +1 ∈ U (1) α if η n +1 ≤ η , which means

Then, it is sufficient to show

To this end, because { c t } n +1 t =1 are i.i.d., we have

for all t = 1 , ..., n, , which is also referred to as the exchangability. Then, we can finish the proof by the following equalities:

where the first equality is obtained by the exchangeability (14), the second equality is obtained by the definition of the expectation, and the last equality is obtained by the fact that

Next, we show Corollary 1:

Corollary 1. For a new sample ( c, A, b, z ) from distribution P , denote the uncertainty set output from Algorithm 1 or Algorithm 2 by U α ( z ) . Let x ∗ ( z ) and OPT ( z ) be the optimal solution and the optimal value of LP ( U α ( z )) (5) . Then we have

where the probability is with respect to the new sample ( c, A, b, z ) and D 2 .

Proof. By definition of the robust optimization problem (5), we have

if c ∈ U ( k ) α ( z ) for k = 1 , 2 . Then, Proposition 1 implies

B.3 Proof for Proposition 2

Proposition 2. For any α ∈ (0 . 5 , 1)

and it decreases monotonously with respect to k.

Proof. Recall that

and z i and ϵ are drawn from an Exponential distribution Exp (1) for all i = 1 , ..., d . For any fixed k = 1 , ..., d , the corresponding prediction model is f k ( z 1: k ) = k ∑ i =1 z i .

We first characterize the distribution of the objective vector c given z 1: k for any k = 1 , ..., d . By the additivity of Gamma distributions and the fact that the exponential distribution is a special case of the Gamma distribution, we have given z 1: k

where the first line comes from the definition of the objective vector (15), the second line comes from changing variables that ˜ z = d ∑ i = k +1 z i and the additivity of Gamma distributions, and others come from the direct calculation. Then, by taking the complement, we have

Next, we compute the probability that x ∗ α ( z 1: k ) = 0 . In the one-dimensional setting with the feasible set { x : -1 ≤ x ≤ 1 } , the optimal solution of (5) can be determined by the sign of the objective vector. Thus, if we are confident that c is non-negative, the optimal solution should be x ∗ α ( z 1: k ) = -1 ; if we are confident that c is non-positive, the optimal solution is x ∗ α ( z 1: k ) = 1 ; if we are not confident about the sign of the objective vector, we will be conservative and choose x ∗ α ( z 1: k ) = 0 . That is,

Then, plugging (16) and (17) into (18), we have P ( c ≤ 0 | z 1; k ) , P ( c ≥ 0 | z 1; k ) ≤ α only when

Integrating the probability on the set (19) with respect to z 1: k , we have

To see it decrease, we can apply Jensen's inequality to see the result.

B.4 Proof for Proposition 3

Proposition 3. For α ≥ 1 / 2 , let the uncertainty set be

The uncertainty set has a coverage guarantee in that P ( c ∈ U α ( z )) = α. If we solve the optimization problem (5) with the uncertainty set U α ( z ) , the following robust solution is obtained

Proof. We first show that P ( c ∈ U α ( z )) = α . Given the prediction function ˆ f ( z ) = sign ( z ) · √ | z | , the residual is r = c -ˆ f ( z ) = ϵ √ | z | . By calculation, we have that the marginal distribution of the objective vector c satisfies

and this distribution is the same as the marginal distribution of the objective vector given z ≥ 0 or z ≤ 0 since the distribution of r only depends on the absolute value of z . Then, we have

where the first line comes from the property of the conditional expectation, the second line comes from the fact that the distribution of the residual is independent of the sign of the covariate z , and the last line comes from (22).

Next, we show that the solution x ( z ) is the optimal solution of problem (5) with the uncertainty set U α ( z ) . By (5) and the feasible set, we have that the optimal solution is 1 or -1 only when all elements in the uncertainty set is non-positive or non-negative, respectively, and the optimal solution is 0 when the uncertainty set contains both positive and negative numbers. Thus, we can find the result by computing the lower and upper bound of the uncertainty set U α ( z ) for z ∈ [0 , 1] and z ∈ [ -1 , 0) , respectively.

C Supplementary for Section 4

This section provides our algorithm and the corresponding analysis for Section 4. We first briefly summarize the problem and then, develop the DRO algorithm raised by our predict-then-calibrate framework and the detailed proof of the algorithm analysis.

As stated in Section 4, we here consider the risk-neutral problem (2) as follows

instead of a risk-sensitive objective in Section 3. As stated in Section 2, we work on a well-trained MLprediction model ˆ f that predicts the objective vector c based on the covariate z . Then, we will apply the idea of distributionally robust optimization to bound the prediction error and develop the generalization bound. We remark again that the prediction model can be any off-the-shelf machine learning method since we have no assumption about it.

C.1 DRO Algorithm

In this part, we derive the DRO algorithm from our predict-them-calibrate framework. As for Algorithms 1 and 2, we work on the prediction error

of all samples in the validation dataset D val = { c t , A t , b t , z t } T t =1 . In the following, by a little abuse of notation, we use t ∈ D val to denote a sample tuple ( c t , A t , b t , z t ) in the validation set.

Algorithm 4 Contextual Distributionally Robust LP

1: Input: Dataset D val , ML model ˆ f , radius ε , bandwidth h , target covariate z 0

2: For each t ∈ D val , let

where

3: Define the estimation of the conditional mean of residuals of covariate z 0 by

and K ( z ) is a kernel function that satisfies Assumption 3

4: Construct the degenerate ambiguity set by

5: Solve the degenerate distributional robust optimization problem

6: Output: ˆ x ( z 0 )

The main idea of our algorithm is similar to distributionally robust optimization that consists of two steps: firstly, construct an ambiguity set Ξ , and secondly, solve the following distributionally robust LP:

and use its optimal solution to solve (2). If the ambiguity set contains the target distribution, say P r | z 0 , we can obtain a good solution to (2) by solving (23) with some generalization bound. Here, P r | z denotes the distribution of the prediction error given the covariate z . In our case, specifically, the ambiguity set can degenerate into a set consisting of the mean of distributions in the ambiguity set since we only focus on the estimation of the conditional mean E [ r | z 0 ] for the target covariate z 0 . Thus, in the following, we will also use Ξ to denote a set of vectors corresponding to the means of

the distributions in Ξ . Then, as the ambiguity set degenerates, the problem (23) also degenerates to the robust optimization problem in Step 5 of Algorithm 4.

Now, we consider the construction of Ξ . We apply a kernel estimation in Steps 3 and 4 of Algorithm 4 to construct Ξ . The following proposition says that if we choose proper ε in Algorithm 4, Ξ contains the conditional mean E [ r | z 0 ] .

Proposition 6. Assume there exists a constant L such that the following condition holds for any z, z ′

Then under mild boundedness assumptions on the distribution, with probability no less than 1 -δ , r 0 defined in Algorithm 4 satisfies

when choosing the bandwidth h = T -1 2 s +2 d .

We defer the proof and assumptions to the next chapter. After developing this estimation error, if the feasible region of (1) is in the unit ball with probability 1, we can directly build the generalization bound for the downstream problem by Proposition 4:

Proposition 4. Under the same assumptions as in Proposition 6, Algorithm 4 outputs a solution ˆ x ( z 0 ) such that

holds with probability no less than 1 -δ .

Here, we emphasize that if the residuals are i.i.d., we can equivalently have L = 0 , s = ∞ . Then, the convergence rate of our Algorithm 4 is O ( T -1 / 2 ) , which matches the rate in Wang et al. [2021], Kannan et al. [2020, 2021]. Moreover, we remark that our results have the potential to be generalized to general cases for distributionally robust optimization if there are more convergence theories developed for the non-i.i.d. samples. Specifically, if the distributions have some smooth properties with respect to the covariate z , the kernel estimation

will also be a good approximation to P r | z 0 , where w t is the weight constructed in Step 3 in Algorithm 4 with some kernel function for all t ∈ D val . The proof will be similar to the proof of Proposition 6. Once being able to utilize samples in D val to approximate P ′ r | z , we can develop a similar DRO algorithm for general cases. However, to our best knowledge, the current state-of-the-art theory can only approximate a distribution by the empirical distribution constructed by i.i.d. samples [Fournier and Guillin, 2015], and its proof cannot be extended to a more general non-i.i.d setting. Thus, one interesting future direction will be developing distributional convergence results for non-i.i.d. settings, and our framework can also have the potential to be applied in general distributionally robust problems.

In the following sections, we will provide detailed proof for Proposition 6 and 4. We first state and interpret our assumptions in Section C.2. Then, in Section C.3, we show two essential lemmas that are helpful for later proof. In Sections C.4 and C.5, we will rigorously restate and show those two propositions, respectively.

C.2 Assumptions for Algorithm 4

Now, we state those assumptions. Most of them are boundedness assumptions. We first state our assumptions about the residual. Recall the residual defined in Algorithm 4 is

for any obejective vector c and covariate z , where ˆ f is the prediction model.

Assumption 1. We assume the residual vector r satisfies the following conditions:

(a) Boundedness: the residual vector is bounded that r ∈ [ -1 , 1] m .

(b) Distribution: the residual vector r given the covariate z follows some unknown continuous distribution P r | z with the density function p r | z .

(c) Smoothness: for any two covariates z, z ′ ∈ [0 , 1] d , the difference between the conditional means of P r | z and P r | z ′ satisfies

for some positive constants L, s > 0 .

Assumptions 1-(a),(b) are just boundedness conditions. Assumption 1-(c) says that the distribution of the residual vector r given the covariate z enjoys some smoothness, and s measures the order of the smoothness. Specifically, if we perturb the covariate z a little bit, the perturbation will only slightly change the corresponding conditional distribution of the residual r given z . This condition is weaker than many recent papers with some true-model assumptions. For example, Wang et al. [2021], Kannan et al. [2020] assume c = f 0 ( z ) + ϵ for some function f 0 ( z ) and random noise ϵ , and Kannan et al. [2021] also assume that all objective vectors for different covariates share a same type of randomness. When f 0 ( z ) can be learned by the prediction model, Assumption 1-(c) holds for L = 0 and s = ∞ .

Assumption 2. We assume the covariate z satisfies the following conditions:

(a) Boundedness: the covariate is bounded that z ∈ [0 , 1] d .

(b) Distribution: the covariate z follows some unknown continuous distribution P z with the density function p ( z ) ∈ C 1 ([0 , 1] d ) .

(c) Smoothness: the density function p ( z ) ∈ C 1 ([0 , 1] d ) satisfies p ( z ) ≤ ¯ p and ∥∇ p ( z ) ∥ 2 ≤ c for some positive constants ¯ p, c ≥ 1 and all z ∈ [0 , 1] d .

The first two parts of Assumption 2 are also boundedness assumptions. Part (c) guarantees that the density function is smooth enough so that it has no steep peak and can not change drastically around any covariate z . We remark that we use this assumption only for simplicity, and it can be replaced by other weaker assumptions that can be satisfied by any density function. The last assumption is about the kernel function we used in Algorithm 4. It basically requires the kernel function to have bounded support and a positive lower bound on the support. This condition can also be satisfied by many kernel functions, such as the uniform kernel and the truncated Gaussian kernel.

Assumption 3. We assume the kernel function satisfies

for some positive constants b R , b r > 0 and R ≥ 1 .

C.3 Essential Lemmas of Proving Propositions 6 and 4

In this section, we let T = |D val | and the bandwidth h = T -γ , where γ is a parameter that we will choose later. Without loss of generality, we let D val = { 1 , ..., T } . For any vector z ∈ R d , we use ( z ) i to denote its i -th entry for i = 1 , ..., d . In addition, denote z 0 as the target covariate, denote K t = K ( z t -z 0 h ) as the value of the kernel function evaluated at z t with bandwidth h , and denote w t ( z t ) = K ( z t -z 0 h ) ∑ T i =1 K ( z i -z 0 h ) as the corresponding weight for any t ∈ D val . Moreover, for simplicity, by a little abuse of notation, we drop the input variables of the kernel and weight functions and use K t and w t to represent the value of the kernel function and the weight when the context is clear.

Before proving Propositions 6 and 4, we first provide two essential lemmas. Lemma 1 shows that, with high probability, there are at least O ( T 1 -dγ ) samples with positive weights. In the proof of the later lemma and Propositions 6 and 4, we will utilize those samples to give an approximation of the conditional mean of the target distribution P r | z .

Lemma 1. Under Assumption 2, for any positive constant δ ∈ (0 , 1) and T ≥ max { 10 , ( δ 8 c ) -1 γ } , with probability no less than 1 -1 T 2 -δ 2 ,

Specifically, if 1 -dγ > 1 2 and T is sufficiently large such that δ 2 (2 R ) d T 1 -dγ ≥ 2 √ T log T ,

Proof. In the following, we show the first inequality (25). The second inequlity (26) can be obtained by plugging the condition δ 8 (2 R ) d T 1 -dγ ≥ 2 √ T log T into (25).

By Hoeffding's inequality, we have for any target covariate z 0 , with probability no less than 1 -1 T 2 ,

where the probability is taken with respect to z t for t = 1 , ..., T . Thus, it is sufficient to show that with probability no less than 1 -δ 2

where the probability is taken with respect to z 0 . To see this, for any z 0 such that p ( z 0 ) > δ/ 4 , by Assumption 2-(c), we have that when T ≥ ( 4 √ dc δ ) 1 /γ ,

for all z satisfying max i =1 ,...,d | ( z ) i -( z 0 ) i | ≤ R · T -γ , where the first line comes from Taylor's expansion and Assumption 2-(c), and the second line comes the condition that T ≥ ( 4 √ dc δ ) 1 /γ . Thus, by integrating the density function on the set { z : max i =1 ,...,d | ( z ) i -( z 0 ) i | ≤ R · T -γ } , we have

Moreover, since P ( { z : p ( z ) < δ/ 2 } ) = ∫ p ( z )1 { ( p ( z ) < δ/ 2) } dz ≤ δ/ 2 , we have P ( { z : p ( z ) ≥ δ/ 2 } ) ≥ 1 -δ/ 2 , and we finish the proof.

The next lemma develops the concentration property for r 0 defined in Step 3 in Algorithm 4.

Lemma 2. Under Assumptions 1 and 3, for any T > 0 , with probability no less than 1 -2 n T 2 , the inequality below holds for all i = 1 , ..., n

where ˜ T = T ∑ t =1 1 { w t > 0 } . Here, the probability is taken with respect to the covariates z 1 , ..., z T and their corresponding residuals.

Proof. This is also an application of Hoeffding's inequality. We first show it for fixed z 1 , ...., z T and fixed target covariate z 0 . In this case, we have ˜ T = T ∑ t =1 1 { w t > 0 } is fixed as well. Moreover, for any w t > 0 , by Assumption 3 and its definition in Algorithm 4 that it is a ratio between corresponding non-zero kernel functions, we have w t ∈ [ b r b R , b R b r ] . In addition, by Assumption 1-(a) that each entry of the residual is in [ -1 , 1] , we have w t · ( r t ) i is bounded by [ -b R b r , b R b r ] . Thus, by Hoeffding's inequality (Lemma 3), we have with probability no less than 1 -2 /T 2 ,

for any i = 1 , ..., d . Then, we can find the result by taking the union bound for all entries i = 1 , ..., n and integrating the probability with respect to z 1 , ..., z T .

Now, we are equipped with all essential lemmas, and in the following, we will show Propositions 6 and 4.

C.4 Proof of Proposition 6

In this section, we will adapt the same notation as in Section C.3.

Proposition 6. Under Assumptions 1, 2 and 3, with probability no less than 1 -δ ,

holds when choosing the bandwidth h = T -1 2 s +2 d .

Proof. To prove it, we first apply the triangle inequality that

where the first and the third line come from the triangle inequality, the second line comes from the definition of r 0 = T ∑ t =1 w t r t , and the last line comes from Assumption 1-(c). In the following, we analyze the first and the second terms in the last line, respectively.

We first analyze the first term in the last line of (27). Denote ˜ T = T ∑ t =1 1 { w t > 0 } . By Hoeffding's inequality, for fixed z 0 , we have with probability no less than 1 -2 T 2 ,

where the first line comes from the definition of w t , the second line comes from Assumption 3, which gives the lower bound of the kernel function, and the third line comes from Hoeffding's inequality. Then, we compute the expectation

Here, the first step is obtained by Assumption 3, the upper bound of the kernel function and the boundedness of its support, the second line comes from Assumption 2, the upper bound of the density function, and the last line is obtained by calculation. Plugging this upper bound into (28), we have the first term in the right-hand side of (27) can be bounded by

Then, for the second term in the right-hand side of (27), under the event of Lemma 2, we have for fixed z 0

Next, we combine (29) and (30) to draw the result. Under the event of Lemma 1 that

the inequality (29) becomes

and the inequality (30) becomes

The event of the intersection of events corresponding to Lemma 1 and inequalities (29) and (30) happends with probability no less than 1 -δ 2 -2 n +3 T 2 . Thus, if T ≥ √ 4 n +6 δ , plugging (31) and (32) into (27), we have with probability no less than 1 -δ ,

Finally, let γ = 1 2 s +2 d , we finish the proof.

C.5 Proof of Proposition 4

In this section, we also use the same notation as in Sections C.2 and C.4. We show that the optimality gap between the solution given by Algorithm 4 and the optimal solution converges to 0 as T →∞ .

Proposition 4. Suppose the feasible region { x : Ax ≤ b, x ≥ 0 } is included in the unit ball { x : ∥ x ∥ ≤ 1 } . and the absolute values of each entry of the prediction function is bounded by ¯ f . Under Assumptions 1, 2 and 3, Algorithm 4 outputs a solution ˆ x ( z 0 ) such that with probability no less than 1 -δ

where x ∗ ( z 0 ) is the optimal solution to LP ( E [ c | z 0 ] , A, b ) , and the probability is taken with respect to z 1 , ..., z T , their corresponding objective vectors, the constraints ( A,b ) and the target covariate z 0 .

Proof. This is a direct application of Proposition 6. We first show the relation between two LPs with different objective vectors. For any two LPs with different objective vectors c 1 , c 2 but with the same constraint ( A,b ) , denote x ∗ 1 and x ∗ 2 as their optimal solutions. Then, if the optimal solutions are contained in the unit ball, we have

where the first step is obtained by rearranging terms, the second step is obtained by Cauchy inequality, the third step is obtained by the optimality of x ∗ 2 with respect to the objective vector c 2 , and the last step is obtained by the condition that x 1 and x 1 are in the unit ball.

Then, since in the event of Proposition 6 that happens with probability no less than 1 -δ ,

and the difference between the objective vectors is the same as the difference between the corresponding residuals, the inequality (33) implies that

and we finish the proof.

D Auxiliary Lemmas

In this section, we state some useful lemmas in the information theory that are helpful in our proof. Lemma 3 (Hoeffding's inequality) . Let X 1 , ..., X T be independent random variables such that X t takes its values in [ u t , v t ] almost surely for all t ≤ T. Then the following inequality holds

for any s > 0 .

Proof. We refer to Chapter 2 of Boucheron et al. [2013].

E Additional Related Work

Due to the space constraints in the main text, we provide additional discussion on related work here.

Contextual RO with a parametrized prediction model. An alternative approach to contextual robust optimization relies on the assumption that the relationship between the outcome variable c and the covariate z is governed by a parameter θ in the true underlying model. In this approach, the estimation/prediction process and the optimization problem are integrated by constructing an uncertainty set for the parameter θ to mitigate the estimation uncertainty. Zhu et al. [2022] constructs the uncertainty set for θ that contains all the parameters with training loss no worse than a threshold, but no coverage guarantee is provided. This can also be viewed as a special case of the decision-driven regularization model introduced in Loke et al. [2022]. Another study by Cao and Gao [2021] extends the consideration of uncertainty to both the parameter and residual. However, the incorporation of estimation and optimization processes in one optimization model can raise tractability issues, restricting the choice of complex yet expressive prediction models.

For the contextual stochastic optimization problem with a risk-neutral objective, below is the other two streams of work in addition to the predict-then-optimize paradigm illustrated in the main text.

Local learning-based conditional stochastic optimization. One stream of work tends to construct stochastic optimization problems with estimated conditional distributions learned by some local

learning methods. Hannah et al. [2010] approximates it by using Nadaraya-Watson kernel regression [Nadaraya, 1964, Watson, 1964] to reweight the historical data. Bertsimas and Kallus [2020] further generalizes it to a framework using a broader range of non-parametric machine learning methods based on local learning, such as k NN [Altman, 1992], classification and regression trees [Breiman, 2017]. To overcome the overfitting effect of these methods when the sample size is small, distributionally robust optimization (DRO) methods are introduced. Bertsimas and Van Parys [2022] enhances the robustness of bootstrap data by constructing the entropy-based DRO model. Nguyen et al. [2020, 2021], Wang et al. [2021], Bertsimas et al. [2023] investigate the Wasserstein-based DRO model. Ho and Hanasusanto [2019] designs a regularized approach and obtains a tractable approximation by leveraging ideas from DRO with a modified χ 2 ambiguity set.

End-to-end prescriptive analytics without prediction. Instead of learning to predict uncertain outcomes, Ban and Rudin [2019] directly learns the function mapping from covariates to decisions by employing the linear decision rule for the newsvendor problem. There are also further studies investigating such integrated approaches without prediction using more complicated models [Bertsimas and Koduri, 2022], such as random forests [Kallus and Mao, 2022] and neural networks [Chen et al., 2022]. Nevertheless, the robustness of the decisions to the epistemic and aleatoric uncertainty is rarely considered in these papers.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study a risk-averse variant of contextual linear optimization, using VaR as the risk measure. The authors develop two heuristic approaches. In both of these approaches, the problem of minimizing the VaR is approximated by a robust optimization problem. The authors propose two different ways of specifying the uncertainty sets, both of which involve building a regression model to predict the conditional mean of the uncertainty, and then calibrating the size of uncertainty with the intent of satisfying a coverage guarantee. The authors provide bounds on the probability that the coverage guarantee will be satisfied, and computational experiments that measure the performance of the proposed method against several baselines. In addition, the authors briefly discuss a distributionally robust variant of contextual linear optimization, and show that there exists a policy that converges to an optimal policy with a specific convergence rate.\n\nSTRENGTHS:\n- The problem selected by the authors, namely risk-sensitive contextual linear optimization, is an interesting and relevant problem.\n- The design of the experiments provided is reasonable and the proposed methods show good performance.\n\nWEAKNESSES:\n- The exact form of the uncertainty set suggested by the authors seems poorly motivated. It would be helpful if the authors could explain why they chose the approach that they did. In the rebuttal, the authors provided an explanation that helped to motivate the form of the uncertainty set.\n\n- The coverage guarantees provided by the authors' methods are not the relevant coverage guarantees for the contextual problem (4). In particular, the conditional probability P(c ϵ U_α(z) | z) should be equal to α (approximately), but the authors instead have designed the methods to guarantee that P(c ϵ U_α(z)) is equal to α (approximately). I view this as a serious flaw in the method, as someone solving a contextual linear optimization problem is presumably specifying their risk for that context, and the authors' approach could in theory severely underestimate the risk in heteroskedastic settings. The authors should redesign the method to ensure that the correct coverage guarantee is met. In the rebuttal, the authors explained how the method could be adapted to provide a conditional coverage guarantee, and explained their rationale for using a global coverage guarantee rather than a conditional one.\n\n- Even if the goal were to produce an unconditional coverage guarantee rather than a conditional one, the theoretical guarantees in Proposition 1 and Corollary 1 are weak. Some very bad ways of choosing uncertainty sets would satisfy these guarantees. For example, you could specify your uncertainty set as a ball with a center of 0 and with radius chosen so that the proportion of samples from the validation set falling in the ball is equal to α, and it would have the same guarantees. It would be better if the authors could show some stronger properties that their choice of uncertainty set satisfies. The authors address this in their rebuttal by acknowledging that the theoretical guarantees could be satisfied by bad choices of uncertainty sets, there is other evidence in the paper that suggests that their choice of uncertainty set is effective. \n\n- In Proposition 3, the authors should ensure that \"P(c ϵ U_α(z) | z) = α\", not that \"P(c ϵ U_α(z)) = α\" since that is the relevant coverage guarantee for the contextual problem. I considered the possibility that this is a typo, but it can be confirmed that the coverage guarantee for the contextual problem is not satisfied. For example, when z=1, the probability P(c ϵ U_α(z) | z) = 1 - sqrt(2-2α)/2, which is not generally equal to α. This example should be reworked. This was also addressed by the author's discussion of the conditional and global coverage guarantees in the rebuttal.\n\n- Section 4 feels like a summary of a different paper that was inserted into this paper. The proposed method for the contextual DRO problem seems only loosely connected to the methods for the risk-sensitive contextual linear optimization. Furthermore, the authors relegate their proposed method for contextual DRO problems to the appendix and provide no computational results for this method. I recommend removing Section 4. The authors addressed this in their rebuttal by clarifying the connections between the contextual DRO problem and the risk-sensitive problem.\n\n- (A minor point) The authors are inconsistent as to whether the values A and b are random or constant. The methodology seems to assume that these are either fixed or, at the very least, independent of the values of c and z, so I would recommend treating these as constant throughout.\n\nQUESTIONS:\n- In the box uncertainty quantification method, why did the authors choose to perform conditional quantile estimation in two steps, where first the mean is estimated and then quantiles are estimated from the residuals? I would note that in existing works where the mean is estimated and then a density is fit on the residuals, there is usually some assumption that justifies this. For example, if it is assumed that c = f(x) + ε, where the errors ε for different observations are i.i.d., then you can perform unconditional density estimation on the residuals (rather than conditional density estimation). However, I don't see any justification in this work. The authors answered this question in their rebuttal.\n\nLIMITATIONS:\n- One limitation (mentioned in the \"Weaknesses\" section) is that the coverage guarantees provided by the method are not the ones that would be desired in the stated problem. This is mentioned in passing, but the authors should investigate this limitation further.\n\n- Even in the absence of uncertainty, the proposed method by the authors is a heuristic in the sense that it does not identify the optimal solution nor are any approximation ratios known. It would be worth exploring how close (or far) the solutions are from optimal solutions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author(s) propose a novel method of robust contextual LP, which extends the conventional contextual LP problem by allowing some uncertainties from the prediction model. It is very well-written, and states very clearly what is the problem setting, and in which direction this work advances. The model properties are discussed in detail and the implication of each theory/statement is explained in a very careful manner.\n\nSTRENGTHS:\n1. Clearly written and well organized\n2. Provides a clear problem statement, and the algorithms are described in good detail\n3.  Strong algorithm analysis with additional detials to provide context for the statements\n4. Good empirical work demonstrates the method\n5. The author proposes two algorithms to construct the contextual uncertainty sets $\\mathcal_{U}$ (which is based on z) for the prediction model. They mention the choice of the prediction model is quite flexible, and there are a couple of advantages by doing so.\n\nWEAKNESSES:\n1. It would be nice if the author could provide the audience with some motivations for this work. For example, under what circumstances would it be beneficial to incorporate a risk-sensitive objective into contextual LP.\n2. All theoretical guarantees mentioned in Section 3.1 require $\\mathcal{D}_2\\sim \\mathcal{P}$, e.g. the validation set is correctly specified.  It would be useful to understand how misspecification impacts these results (either empirically or theoretically).\n\nQUESTIONS:\n1. Proposition 1 provides a coverage guarantee for Algorithm 1 and Algorithm 2. I was a bit confused by the statements. On the left hand side, the inequality goes to 0 and $|\\mathcal{D}_2|$ increases. While on the right handside, the inequality goes to 1 as $|\\mathcal{D}_2|$ increases. Does it mean the more observations in $\\mathcal{D}_2$, the more uncertainty there will be? It would be nice if the author could provide more explanation on it.\n2. As stated above, all theoretical guarantees mentioned in Section 3.1 require $\\mathcal{D}_2\\sim \\mathcal{P}$, e.g. the validation set is correctly specified. For practical application purposes, could the author provide some strategies to make sure this requirement is guaranteed?\n3. In the Experiment Section, Simple LP Visualization paragraph, the author mentions both PTC-B and PTC-E. Maybe I missed it, but it seems in Figure 2, the author only provides visualization for PTC-B. Where is PTC-E?\n\nLIMITATIONS:\nSee above for list of areas for improvement.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
The authors study a risk-sensitive contextual LP setting. They seek to predict the objective function of the LP from a context vector using a generic machine-learning algorithm, and then use this prediction to achieve a low (good) objective value in the LP. Their insight is this can be done cleverly with calibration—instead of changing the ML prediction as others have done, they calibrate the output. In theory and empirical evidence, they show their method has advantages over competitors.

STRENGTHS:
- There is substantial interest in the predict-then-optimize setting for LPs (see the citations on Grigas and Elmachtoub). This setting (while not mentioned by the authors) seems to have clear applications in finance.
- The authors' work is clearly presented and claims are backed up by theory and experiments. The theory and experiments support the claims not only that the approach achieves better objective values, but gives a more detailed view of why (I particularly like 3.2).

WEAKNESSES:
- The method combines tools that we understand pretty well. In particular, it leverages the detailed picture of robust optimization linear programs that we have. The results are naturally limited to LPs.
- There isn't much discussion of limitations.

QUESTIONS:
1. Have the authors' investigated the application to conditional value at risk?

LIMITATIONS:
Not much discussion, a little bit in Future Directions.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the contextual linear optimization problem, where one is given a vector of covariates $z$ that can be used to predict a cost vector $c$, and one wishes to solve the following LP:\n\n$ \\max \\ E[ c \\mid  z]^T x$\n\n$ \\text{subject to}: A x = b, x \\geq 0  $\n\nThis is the risk-neutral version of the contextual LP problem. The risk-sensitive version instead considers the $\\alpha$-value-at-risk with respect to the conditional distribution of $c^T x$ given $z$:\n\n$ \\max \\ VaR_{\\alpha}( c^T x \\mid z )$\n\n$ \\text{subject to}: A x = b, x \\geq 0  $\n\nThe paper proposes an approximate method for solving this problem, where one approximates the VaR with a max over $c$'s in a fixed uncertainty set that depends on the given covariate vector $z$. The uncertainty set itself is obtain from a procedure that in the abstract works like this: take a prediction model $\\hat{f}$ and make predictions of $c$ in a validation set. Calculate the errors/residuals, and build another model, called the calibration model, to predict these residuals. One then constructs an uncertainty set conditional on $z$ where the center comes from the prediction model and the width comes from the calibration model. This approach is shown with high confidence over the validation set to satisfy the $\\alpha$ probability coverage requirement. The paper also tests this approach using synthetic toy instances as well as instances based on the contextual shortest path problem, and shows that it leads to better (lower) VaRs than existing robust-contextual proposals.\n\nSTRENGTHS:\n- From a novelty standpoint, I think this paper does present something new; to my knowledge the combination of value-at-risk within contextual optimization has not been done before.\n\n- The method seems reasonably simple and could be implemented easily. It is also nice to see that there is a high-confidence guarantee on the coverage probability of the set $\\mathcal{U}$ produced by Algorithms 1 and 2. Another nice aspect is that the validation data set does not need to be the same data set used to train the predictive model $\\hat{f}$; this goes along nicely with the goal of robustness in the paper.\n\n- I appreciate that the authors provided the simple examples in Section 3.2 to illustrate how the approach works and the difference between having a good prediction model vs. calibration model.\n\nWEAKNESSES:\n- I think the motivation of the paper is a little weak. While I understand the motivation for predict-then-optimize / contextual optimization, and the motivation for optimizing value-at-risk, it is not clear to me why one would want to couple these two things together. The introduction very quickly brings up the \"risk-sensitive\" aspect of the problem without providing an accompanying problem from the literature or from practice that would need this type of machinery.\n\n- From a practical standpoint, there are a lot of questions that are unanswered, namely how good is the method (can one do better than this method in certain cases? see my suggestion #2 below) and how should one choose the prediction model and the calibration model? \n\n- The numerical evaluation feels incomplete, and the method should be compared to stronger benchmarks; for this type of contextual problem, it feels like there should be lots of ways that one could represent the conditional VaR with respect to a given predictive model (see my suggestion #1 below, on using the conditional distribution outputted by the predictive model).\n\nQUESTIONS:\n_Question/Suggestion 1_: If the goal is to maximize the alpha-VaR conditional on a given covariate vector, could the following simpler strategy not work: in Bertsimas and Kallus (2020), the authors propose solving the contextual problem by using a machine learning model to obtain an estimate of the conditional distribution of $c$ given $z$. (For example, for a CART tree, one would run the given $z$ down the tree to determine the corresponding leaf, look at the historical $(z',c')$ pairs that are in that leaf, and consider the distribution with a weight of $1 / M$ on the $M$ pairs of $(z', c')$ values that are in that leaf, and a weight of zero on all other observations.) Bertsimas and Kallus discusses how this type of strategy can be used in conjunction with other machine learning models, such as random forests and k-nearest neighbors. \n\nMy question here is why could one not take a similar strategy here; specifically, why not output the $x$ that maximizes $\\alpha$-VaR with respect to the estimated conditional distribution? So for example, for CART, this would mean minimizing $\\alpha$-VaR with respect to the same discrete distribution that puts a weight of $1/M$ on the $M$ points that are mapped to the same leaf as the given $z$. (This would likely be some kind of simple MIP problem.)\n\nIn addition to the connection to the existing work of Bertsimas and Kallus, this simpler strategy is also appealing in that one does not need to specify a prediction and a calibration model separately; there is only one machine learning model used that needs to be specified. \n\n\n_Question/Suggestion 2_: Is it possible to say something about how \"optimal\" the proposed predict-then-calibrate approach is? To elaborate, in Bertsimas, Gupta and Kallus (2017), one deals with a robust linear constraint of the form\n\n$\\mathbf{u}^T \\mathbf{x} \\leq b, \\quad \\forall \\ \\textbf{u} \\in \\mathcal{U}$\n\nand one assumes that the actual value of $\\mathbf{u}$ is the random variable $\\mathbf{\\tilde{u}}$. One then seeks to select $\\mathcal{U}$ so that\n\n$ \\text{if} \\ \\mathbf{u}^T \\mathbf{x} \\leq b \\quad \\forall \\mathbf{u} \\in \\mathcal{U}, \\ \\text{then} \\ \\mathbb{P}( \\mathbf{\\tilde{u}} \\mathbf{x} \\leq b ) \\geq 1 - \\epsilon$\n\nBertsimas, Gupta and Kallus (2018) note that this guarantee can be met if $\\mathcal{U}$ is chosen as a $1 - \\epsilon$-confidence region for $\\mathbf{\\tilde{u}}$, i.e., if $\\mathbb{P}( \\mathbf{\\tilde{u}} \\in \\mathcal{U}) \\geq 1 - \\epsilon$, but this is not the only way that the guarantee can be met, and by choosing a smaller $\\mathcal{U}$ one can still have the some probabilistic guarantee and be less conservative. Gupta (2019) shows that a number of uncertainty sets that have been proposed that are based on this confidence region idea are unnecessarily conservative, and that one can obtain much smaller uncertainty sets with the same probabilistic guarantee. The proposed methodology in the paper (replace the intractable problem on lines 106 - 107 with problem (4), and then apply either Algorithm 1 or 2) seems more aligned with this confidence region approach, so it would be great if the authors can discuss more on how tight / optimal this approach is. \n\n_Question/Suggestion 3_: It would be nice to see better motivation for the contextual VaR formulation that the paper studies, both in the introduction and the numerics. \n\n_Other comments/suggestions_:\n\nPage 5, line 143: \"but rather the only two options\" $\\to$ \"but these are not the only two options\". \n\nPage 9, line 264: the word \"better\" is repeated here. \n\nReferences:\nBertsimas, D., Gupta, V., & Kallus, N. (2018). Data-driven robust optimization. Mathematical Programming, 167, 235-292.\nBertsimas, D., & Kallus, N. (2020). From predictive to prescriptive analytics. Management Science, 66(3), 1025-1044.\nGupta, V. (2019). Near-optimal Bayesian ambiguity sets for distributionally robust optimization. Management Science, 65(9), 4242-4260.\n\nLIMITATIONS:\nThe discussion of limitations seems adequate; as noted in my questions above, it would be nice to have some discussion of how optimal this approach is.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers a risk-sensitive version of the contextual LP problem by replacing the original risk-neutral expected cost objective with VaR. The authors propose a new paradiam termed \"predict-then-calibrate\" that first learns a prediction model, and then uses calibration to quantify the uncertainty of the prediction. They present two algorithms that output a box/ellipsoid uncertainty set. The authors then provide a coverage guarantee for both algorithms, and conduct empirical experiments that show favorable properties of the proposed algorithms.\n\nSTRENGTHS:\n- This paper considers contextual linear optimization problems with a risk-sensitive objective, which is a good complement to existing risk-neutral contextual linear optimization literature. It also complements the conditional robust optimization literature by providing a flexible algorithm in terms of the modeling choices. The idea of adding a calibration step in contextual linear optimization is novel.\n- The two algorithms provided are intuitive and flexible to implement. \n- The paper is overall clearly written and well-structured. The authors use simple intuitive examples to illustrate the value of better prediction and calibration, which I find really helpful.\n\nWEAKNESSES:\n- The modeling choice for $\\hat{h}$ seems vague to me. More elaborations on this subject would be helpful.\n- I have some concerns regarding the comparisons in the empirical session. See \"questions\" for details.\n\nQUESTIONS:\n- Can the authors provide more guidelines for the modeling choice of $\\hat{h}$? In practice, how to determine which model is better for $\\hat{h}$?\n- In Figure 4 and Table 1, the authors use the Kernel Ridge method with the RBF kernel as the prediction model and the NN model as the preliminary calibration model in both PTC-B and PTC-E (I found this in the appendix; please state it in the main text). Since RBF kernel works the best in Figure 3, this looks like a somewhat unfair comparison, since we are comparing the best among PTC-B/PTC-E with other algorithms. Can the authors elaborate more on this? Also, how does the run time of difference algorithms compare?\n- Is it possible to empirically demonstrate how good the proposed algorithms do in terms of individual-level coverage?\n\nLIMITATIONS:\nThe authors discussed the limitations in the conclusion session.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-26"}