Title: Conformal Inverse Optimization
Venue: NeurIPS 2024
Accepted: yes
Keywords: inverse optimization

robust optimization

algorithm aversion

data-driven decision making
Abstract: Inverse optimization has been increasingly used to estimate unknown parameters in an optimization model based on decision data. We show that such a point estimation is insufficient in a prescriptive setting where the estimated parameters are used to prescribe new decisions. The prescribed decisions may be low-quality and misaligned with human intuition and thus are unlikely to be adopted. To tackle this challenge, we propose conformal inverse optimization, which seeks to learn an uncertainty set for the unknown parameters and then solve a robust optimization model to prescribe new decisions. Under mild assumptions, we show that our method enjoys provable guarantees on solution quality, as evaluated using both the ground-truth parameters and the decision maker's perception of the unknown parameters. Our method demonstrates strong empirical performance compared to classic inverse optimization.

Conformal Inverse Optimization

Abstract

Inverse optimization has been increasingly used to estimate unknown parameters in an optimization model based on decision data. We show that such a point estimation is insufficient in a prescriptive setting where the estimated parameters are used to prescribe new decisions. The prescribed decisions may be of low-quality and misaligned with human intuition and thus are unlikely to be adopted. To tackle this challenge, we propose conformal inverse optimization, which seeks to learn an uncertainty set for the unknown parameters and then solve a robust optimization model to prescribe new decisions. Under mild assumptions, we show that our method enjoys provable guarantees on solution quality, as evaluated using both the ground-truth parameters and the decision maker's perception of the unknown parameters. Our method demonstrates strong empirical performance compared to classic inverse optimization.

1 Introduction

Inverse optimization (IO) is a supervised learning approach that fits parameters in an optimization model to decision data. The fitted optimization model can then be used to prescribe future decisions. Such problems naturally arise in AI applications where human preferences are not explicitly given, and instead need to be inferred from historical decisions. For this pipeline to succeed in practice, the prescribed decision should not only be of high quality but also align with human intuition (i.e., perceived to be of high-quality). The latter encourages algorithm adoption (Chen et al., 2023; Donahue et al., 2023), which is critical in many real-world settings (Liu et al., 2023; Sun et al., 2022).

As an example, rideshare platforms, e.g., Uber and Lyft, provide a shortest-path to the driver at the start of a trip based on real-time traffic data (Nguyen, 2015). The driver then relies on her perception of the road network formed through past experience to evaluate the path. The driver may deviate from the suggested path if it is perceived to be low-quality. Although seasoned drivers are often capable of identifying a better path due to their tacit knowledge of the road network (Merchán et al., 2022), such deviations impose operational challenges as it may cause rider safety concerns and affect downstream decisions, such as arrival time estimation, trip pricing, and rider-driver matching (Hu et al., 2022). Therefore, the platform may be interested in leveraging historical paths taken by drivers to suggest high-quality paths for future trips, as evaluated using both the travel time and the driver's perception.

In this paper, we first show that the classic IO pipeline may generate decisions that are low-quality and misaligned with human intuition. We next propose conformal IO, which first learns an uncertainty set from decision data and then solves a robust optimization model with the learned uncertainty set to prescribe decisions. Finally, we show that the proposed approach has provable guarantees on the actual and perceived solution quality. Our contributions are as follows.

New framework . We propose a new IO pipeline that integrates i) a novel method to learn uncertainty sets from decision data and ii) a robust optimization model for decision recommendation.

Theoretical guarantees . We prove that, with high probability, the learned uncertainty set contains parameters that make future observed decisions optimal. This coverage guarantee leads to bounds on the optimality gap of the decisions from conformal IO, as evaluated using the ground-truth parameters and the decision maker's (DM's) perceived parameters.

Performance . Through experiments, we demonstrate strong empirical performance of conformal IO compared to classic IO and provide insights into modeling choices.

2 Literature Review

Inverse optimization . IO is a method to estimate unknown parameters in an optimization problem based on decision data (Ahuja and Orlin, 2001; Chan et al., 2014; Chan and Kaw, 2020). Early IO papers focus on deterministic settings where the observed decisions are assumed to be optimal to the specified optimization model. Recently, IO has been extended to stochastic settings where the observed decisions are subject to errors and bounded rationality. Progress has been made to provide estimators that are consistent (Aswani et al., 2018; Dong et al., 2018), tractable (Chan et al., 2019; Tan et al., 2020; Zattoni Scroccaro et al., 2024), and robust to data corruption (Mohajerin Esfahani et al., 2018). Our paper is in the stochastic stream. Unlike existing methods that provide a point estimation of the unknown parameters, we learn an uncertainty set that can be used in a robust optimization model.

Data-driven uncertainty set construction. Recently, data have become a critical ingredient to design the structure (Delage and Ye, 2010; Mohajerin Esfahani et al., 2018; Gao and Kleywegt, 2023) and calibrate the size (Chenreddy et al., 2022; Sun et al., 2023) of an uncertainty set. Our paper is related to the work of Sun et al. (2023) who first use an ML model to predict the unknown parameters and then calibrate an uncertainty set around the prediction. However, this approach does not apply in our setting as it requires observations of the unknown parameters, which we do not have access to. Our paper presents a new approach to calibrating uncertainty sets using decision data.

Estimate, then optimize. Conformal IO belongs to a family of data-driven optimization methods called 'estimate, then optimize' (Elmachtoub et al., 2023). Recent research suggests that even small estimation errors may be amplified in the optimization step, resulting in significant decision errors. This issue can be mitigated by training the estimation model with decision-aware losses (Wilder et al., 2019; Mandi et al., 2022) and robustifying the optimization model (Chan et al., 2023a). We take a similar approach as the second stream, yet deviate from them by i) utilizing decision data instead of observations of the unknown parameters, and ii) focusing on both the ground-truth and perceived solution quality, the latter of which has not been studied in this stream of literature.

Preference learning. Preference learning has been studied in the context of reinforcement learning and has recently attracted significant attention due to its application in AI alignment (Ji et al., 2023). Existing methods focus on learning a reward function/decision policy that is maximally consistent with expert decision trajectories (Ng and Russell, 2000; Wu et al., 2024) or labeled preferences in the form of pair-wise comparison/ranking (Wirth et al., 2017; Christiano et al., 2017; Rafailov et al., 2024). We enrich this stream of literature in two ways. First, we leverage unlabeled decision data that are not state-action trajectories, but solutions to an optimization model. Second, instead of learning a policy that imitates expert behaviors, we aim to extract common wisdom from decision data crowd-sourced from DMs who are not necessarily experts to encourage algorithm adoption.

3 Preliminaries

In this section, we first present the problem setup (Section 3.1) and then describe the challenges with the classic IO pipeline (Section 3.2). Finally, we provide intuition on why robustifying the IO pipeline would help (Section 3.3).

3.1 Problem Setup

Data generation. Consider a forward optimization problem

where x ∈ R n is the decision vector whose feasible region X ( u ) is non-empty and is parameterized by exogenous parameters u ∈ R m , θ ∈ R d is a parameter vector, and f : R n × d → R is the objective function. Suppose u is distributed according to P u supported on U . There exists a ground-truth parameter vector θ ∗ that is unknown to the DM. Instead, the DM obtains a decision ˆ x by solving FO ( ˆ θ , u ) where ˆ θ is a noisy perception of θ ∗ . We assume that, while the distribution P θ of ˆ θ is unknown, it is supported on a known bounded set Θ ⊂ R d and that θ ∗ ∈ Θ . Let P ( θ , u ) denote the joint distribution of ˆ θ and u , ˜ x : Θ ×U → R n be an oracle that returns an optimal solution to FO drawn uniformly at random from X OPT ( θ , u ) := argmin { f ( θ , x ) | x ∈ X ( u ) } .

Objective function. We focus on cases where f is linear in θ and convex in x , i.e., f ( θ , x ) = ∑ i ∈ [ d ] θ i f i ( x ) where f i : R n → R are convex basis functions. This generalizes the linear objective f ( θ , x ) = θ ⊺ x . Moreover, FO with this objective function can be treated as a multi-objective optimization model, which has been used to model routing preferences (Rönnqvist et al., 2017), radiation therapy planning (Chan et al., 2014), and portfolio optimization (Dong and Zeng, 2021). In this setting, the optimal solution to FO is invariant to the scale of θ , i.e., if x ∈ X OPT ( θ , u ) , then x ∈ X OPT ( β θ , u ) for any β ∈ R + . So, we set Θ = { θ ∈ R d ∣ ∣ ∥ θ ∥ 2 = 1 } without loss of generality.

Learning task. Given a dataset of N decision-exogenous parameter pairs D = { ˆ x k , u k } N k =1 , we are interested in finding a decision policy ¯ x : U → R n to suggest decisions for future u . We require ¯ x ( u ) ∈ X ( u ) for any u ∈ U . As discussed later, ¯ x ( u ) is usually generated by solving an optimization model that may have multiple optimal solutions. So we consider randomized policies (e.g., uniformly sample from a set of optimal solutions). This is nonrestrictive because a deterministic policy can be recovered from a randomized policy that samples the deterministic solution with probability one.

As a running example, FOP may represent a shortest path problem, where u k specifies the origin and destination of a trip, θ ∗ indicates the ground-truth travel times on each road segment, while ˆ θ k is a driver's perception of θ ∗ (i.e., perceived travel times). Decision x k corresponds to a path taken by a driver based on her perceived travel times. Given a set of historical trips { u k , ˆ x k } k ∈ [ N ] , our goal is to derive a routing policy ¯ x that can provide path recommendations for future origin-destination pairs.

3.1.1 Assumptions

Assumption 1 (I.I.D. Samples) . The dataset D is generated using ˆ x k := ˜ x ( ˆ θ k , u k ) where ( ˆ θ k , u k ) are i.i.d. samples from P ( θ , u ) for all k ∈ [ N ] .

Assumption 2 (Bounded Inverse Feasible Set) . There exists a constant η ∈ R + such that, for any θ , θ ′ ∈ Θ OPT (ˆ x , u ) , for some ˆ x ∈ X ( u ) and u ∈ U , we have ∥ θ -θ ′ ∥ 2 ≤ η , where

Assumption 3 (Bounded Divergence) . There exists a constant σ ∈ R + such that ∥ E ( ˆ θ ) -θ ∗ ∥ 2 ≤ σ .

Assumption 1 is standard in the ML and IO literature. Assumption 2 is mild because Θ OPT (ˆ x , u ) is by definition bounded and is usually much smaller than Θ . Assumption 3 states that the l 2 distance between the expected perceived parameters and the ground-truth parameters is upper bounded. It is reasonable in many real-world settings. For example, a driver's perceived travel cost ( ˆ θ ) should not be too different from the travel time ( θ ∗ ) as the latter is an important factor that drivers consider.

3.1.2 Evaluation Metrics

Definition 1. The actual optimality gap (AOG) of a decision policy ¯ x is defined as

where the expectation is taken over the joint distribution of the random variable u and the decision sampled using the possibly randomized policy ¯ x .

Definition 2. The perceived optimality gap (POG) of a decision policy ¯ x is defined as

where the expectation is taken with respect to the randomness in ˆ θ , u , and possibly ¯ x .

AOG is an objective performance measure. Achieving a low AOG means that ¯ x can generate highquality decisions. In contrast, POG is a subjective measure that depends on the DM's perception of the problem. Achieving a low POG is critical to mitigate algorithm aversion (Burton et al., 2020).

3.2 An Inverse Optimization Pipeline

Finding ¯ x is challenging for three reasons. First, unlike many ML tasks where the prediction target is unconstrained, we require ¯ x ( u ) to be feasible to FO which may involve a large number of constraints. Supervised learning approaches that predict ˆ x based on u can often fail as they typically do not provide feasibility guarantees. An optimization module is often needed to recover feasibility or produce feasible solutions based on u and some estimated ¯ θ . Second, we do not directly observe θ ∗ or ˆ θ , which precludes using classic ML techniques to estimate them. Finally, AOG and POG may not necessarily align with each other, so we are essentially dealing with a bi-objective problem.

Standard IOPipeline

Conformal Io Pipeline

1.inverse

2.forward

1. data split

3.calibration

problem

problem

K

validation

problem

uncertaintyset

K

4.robust

2.lnverse

forward

decision

problem

decision

point

decision

data

00

o0

problem

data

estimate

prescription

training

point

decision

estimate

prescription

In light of the first two challenges, a classic IO pipeline (visualized in Figure 1) has been proposed to first obtain a point estimation ¯ θ of the unknown parameters and then employ a policy ¯ x IO ( u ) := ˜ x ( ¯ θ , u ) to prescribe decisions for any u ∈ U (Rönnqvist et al., 2017; Babier et al., 2020). Specifically, we can estimate the parameters by solving the following inverse optimization problem

where ℓ is a non-negative loss function that returns 0 when ˆ x k ∈ X OPT ( θ , u k ) . For instance, the following loss function is commonly used in the literature.

Definition 3. The sub-optimality loss of θ is given by

The sub-optimality loss penalizes the optimality gap achieved by the observed decision under the estimated parameters. As remarked by Mohajerin Esfahani et al. (2018), this loss function has better computational properties than its alternatives as it is convex in the unknown parameters. In fact, when the dataset D is large in size and the unknown parameters θ are high-dimensional, the sub-optimality loss is usually the only loss function that leads to a tractable inverse problem, although it does not enjoy properties such as statistical consistency (see Chan et al. (2023b) for detailed discussions). While such a trade-off is acceptable in some applications, we suggest that it is undesirable in our setting because the resulting policy can achieve arbitrarily large AOG and POG. To see this, consider the following example that satisfies Assumptions 1-3. This example is visualized in Figure 2.

Example 1. Let FO ( θ , u ) be the following problem

Let the ground-truth θ ∗ = (cos( π/ 4) , sin( π/ 4)) and U = { u } where u > 1 is a real constant. We are given a dataset D = { ˆ x k , u } N k =1 where ˆ x k = ˜ x ( ˆ θ k , u ) with ˆ θ k uniformly and independently drawn from Θ = { (cos δ, sin δ ) | δ ∈ (0 , π/ 2) } for all k ∈ [ N ] .

Classic IO

Conformal 1O

2

2

x1

x1

?

0

0

0

n

0

u

X1

X1

Lemma 1. In Example 1, let ¯ θ N denote an optimal solution to IO ( D ) with the sub-optimality loss (6) , we have P ( ¯ θ N = θ u ) → 1 as N →∞ , where θ u := ( 1 / √ 1 + u 2 , u/ √ 1 + u 2 ) .

Lemma 1 shows that, when using IO ( D ) with the sub-optimality loss to estimate the unknown parameter in Example 1, the probability of the estimated parameter being θ u converges to one asymptotically. This implies that asymptotically we are almost certain that ¯ x IO ( u ) = ˜ x ( θ u , u ) , i.e. the policy that samples uniformly from the facet corresponding to the constraint x 1 + ux 2 ≥ u . As a result, ¯ x IO can achieve arbitrarily large AOG and POG when u is set to a large enough value.

Proposition 1. In Example 1, let ¯ x IO ( u ) = ˜ x ( θ u , u ) . For any v ∈ R + there exists some ¯ u > 1 such that AOG(¯ x IO ) > v and POG(¯ x IO ) > v for any u > ¯ u .

3.3 Robustifying the Inverse Optimization Pipeline

A natural idea to improve the AOG and POG of ¯ x is to robustify the decision pipeline. Specifically, instead of solving FO with some estimated parameters ¯ θ , we solve the following robust forward optimization problem with an uncertainty set around ¯ θ to prescribe decisions (final steps in Figure 1).

where C is an uncertainty set with ¯ θ being its center and α representing parameters that control its shape/size. Given the support set Θ defined in Section 3.1, we focus on the following uncertainty set.

where α ∈ (0 , π ] represents the max angle between ¯ θ and any vector in the uncertainty set.

Remark 1. An alternative approach to robustify the IO pipeline is to replace IO with a distributionally robust IO for parameter estimation (Mohajerin Esfahani et al., 2018). However, this approach would not help as θ u is still optimal to the distributionally robust IO in Example 1. So, the AOG and POG can still be arbitrarily large. See Appendix A.3 for complete statement and discussions.

Now, in Example 1, we analyze the performance of a policy that utilizes RFO to prescribe decisions.

( ( ¯ ) ¯

Lemma 2. In Example 1, let ¯ x CIO ( u ) be an optimal solution to RFO C θ N , α ) , u where θ N is an optimal solution to IO ( D ) with the sub-optimality loss (6) . When α ∈ (0 , π/ 2) , we have P [AOG( x CIO ) = 0] → 1 and P [ POG(¯ x CIO ) < π/ 2 √ 2 ] → 1 as N →∞ .

Lemmas 2 shows that, when using RFO to prescribe new decisions, the probability of achieving upper-bounded AOG and POG converges to one as N goes to infinity, as long as α ∈ (0 , π/ 2) . These bounds are independent of u , in contrast to the AOG and POG of classic IO that can be arbitrarily large as u changes (Proposition 1). However, the performance of this approach still depends on the choice of α , which is non-trivial when FO is more complex than a two-dimensional linear program. We address this problem next.

4 Conformal Inverse Optimization

In this section, we present a principled approach to learn uncertainty sets that lead to provable performance guarantees. As presented later, the learned uncertainty set contains parameters that make the next DM's decision optimal with a specified probability. We call this approach conformal IO due to its connection to conformal prediction (Vovk et al., 2005), which aims to predict a set that contains the next prediction target with a specified probability. Our approach first converts each context-decision observation ( u k , ˆ x k ) to a parameter set Θ OPT ( u k , ˆ x k ) that contains all the parameters that explain ˆ x k under u k . We then adapt conformal prediction to produce a set that has γ probability of containing at least one member of the next sampled Θ OPT ( u , ˆ x ) . Note that if Θ OPT ( u , ˆ x ) is a singleton almost surely, the approach is equivalent to applying conformal prediction to θ k directly. As illustrated in Figure 1, there are three training steps in conformal IO: i) data split, ii) point estimation, and iii) uncertainty set calibration. We present these steps in Section 4.1 and analyze the properties of conformal IO in Section 4.2.

4.1 Learning an Uncertainty Set

Data split. We first split the dataset D into training and validation sets, namely D train and D val. Let K train and K val index D train and D val, respectively, while N train = |D train | and N val = |D val | .

Point estimation. Given a training set D train, we apply data-driven IO techniques to obtain a point estimation ¯ θ of the unknown parameters. The most straightforward way is to solve IO ( D train ) with any loss function. Alternatively, one may consider using end-to-end learning and optimization methods that do not require observations of the parameter vectors, e.g., the one proposed by Berthet et al. (2020). The point estimation can also come from other sources, e.g., from an ML model that predicts the parameters. Our calibration method is independent of the point estimation method.

Uncertainty set calibration. Given a point estimation ¯ θ , we calibrate an uncertainty set that, with a specified probability, contains parameters that make the next observed decision optimal. This property is critical for the results in Section 4.2 to hold. While we can naively achieve this by setting α = π , the resulting RFO may generate overly conservative decisions. Hence, we are interested in learning the smallest uncertainty set that satisfies this condition. We solve the following calibration problem

where decision α controls the size of the uncertainty set, θ k represent a possible parameter vector associated with data point k ∈ K val , γ ∈ [0 , 1] is a DM-specified confidence level. Constraints (10b) ensure that θ k can make the decision ˆ x k optimal for k ∈ K val. Constraint (10c) ensures that at least γ ( N val +1) of the decisions in D val can find a vector in C that makes it optimal. Constraints (10d) ensure that the parameter vectors are on the unit sphere as defined in Equation (9).

Remark 2 (Optimality Conditions) . The specific form of Constraints (10b) depends on the structure of FO . For example, when FO is a linear program, Constraints (10b) can be replaced with the dual feasibility and strong duality constraints. When the FO is a general convex optimization problem, we can use the KKT conditions. For non-convex forward problems, we can replace Constraints (10b) with f ( θ k , ˆ x k ) ≤ f ( θ k , x ) for all x ∈ X ( u ) , which can be generated in a cutting-plane fashion.

Remark 3 (Feasibility) . For CP to be feasible, we require, for each observed decision, there exists a θ ∈ Θ that make it optimal. This condition holds for a range of problems, e.g., routing problems and the knapsack problem, even if the DM is subject to bounded rationality, i.e., the DM settles for suboptimal solutions due to cognitive/computational limitations. For problems where this condition is violated, we may pre-process D val to project ˆ x to a point in X ( u ) such that the condition is satisfied.

Solving CP is hard. First, CP is non-convex due to Constraints (10d). Second, Constraints (10b) involve the optimality conditions of N val problems, so the size of CP grows quickly as N val in-

creases. Nevertheless, considering a large D val is critical to ensure desirable properties of the learned uncertainty set (Section 4.2). Below we introduce a decomposition method to solve CP efficiently.

Theorem 1. Let D val be a dataset, γ ∈ [0 , 1] , ¯ θ ∈ R d , τ = ⌈ γ ( N val +1) ⌉ and Γ τ be an operator that returns the τ th largest value in a set. The optimal solution to CP ( ¯ θ , D val , γ ) is α γ := arccos (Γ τ ( { c k } k ∈K val )) with c k := max θ k { θ ⊺ k ¯ θ ∣ ∣ ˆ x k ∈ X OPT ( θ k , u k ) , ∥ θ k ∥ 2 = 1 } .

Theorem 1 states that we can solve CP by first solving N val optimization problems whose size is independent of N val and then find a quantile in a set of N val elements. The first step is parallelizable and the second step can be done in O ( N val log( τ )) time. Since the problem required for evaluating c k is a maximization problem, we can replace the constraint ∥ θ k ∥ 2 = 1 with ∥ θ k ∥ 2 ≤ 1 if ¯ θ ∈ R d + , so this problem is convex when the forward problem is a linear program.

4.2 Properties of Conformal IO

Theorem 2 (Uncertainty Set Validity) . Let D val be a dataset that satisfies Assumption 1, ( ˆ θ , u ) be a new i.i.d. sample from P ( θ , u ) , ˆ x = ˜ x ( ˆ θ , u ) , ˆ Θ := Θ OPT (ˆ x , u ) , and α γ be an optimal solution to CP ( ¯ θ , D val , γ ) where ¯ θ ∈ R d . We have, for any γ ∈ [0 , N val / ( N val +1)] , that

̸

For any γ ∈ [0 , 1] , with probability at least 1 -1 /N val ,

̸

̸

Theorem 2 states that our learned uncertainty set is conservatively valid and asymptotically exact (Vovk et al., 2005). More specifically, first, our method will produce a set that contains a θ that makes the next DM's decision optimal no less than γ of the time that it is used (conservatively valid). The probability in Inequality (11) is with respect to the joint distribution over D val and the new sample. Second, once the set is given, we have high confidence that, the probability of the next DM's decision being covered is within ϵ ( N val ) from γ . The probability in Inequality (12) is with respect to the new sample, while the high confidence is with respect to the draw of the validation data set. Overall, we have the almost sure convergence of P ( ˆ Θ ∩ C ( ¯ θ , α γ ) = ∅ ) to γ as N goes to infinity. Finally, we note that in many practical applications, the number of decision observations N can be quite large. For example, in our motivating example, rideshare platforms observe millions of trips on a daily basis, providing ample data for both point estimation and uncertainty set calibration in our pipeline.

Now, we relate the validity results to the performance of conformal IO. The following Lemma is an immediate result of the objective function f being linear in θ .

Lemma 3. For any ˆ x ∈ ˜ x ( ˆ θ , u ) and ( ˆ θ , u ) ∈ Θ ×U , there exists a constant ν (ˆ x ) ∈ R + such that, for any θ , θ ′ ∈ Θ , we have f ( θ , ˆ x ) -f ( θ ′ , ˆ x ) ≤ ν (ˆ x ) ∥ θ -θ ′ ∥ 2 .

̸

Theorem 3 (POG Bound) . Let ¯ x CIO ( u ) be an optimal solution to RFO ( C ( ¯ θ , α 1 ) , u ) for any u ∈ U , where ¯ θ ∈ R d and α 1 are chosen such that, for a new sample ( θ ′ , u ′ ) from P ( θ , u ) and x ′ = ˜ x ( θ ′ , u ′ ) , P ( C ( ¯ θ , α 1 ) ∩ Θ OPT ( u ′ , x ′ ) = ∅ ) = 1 . If Assumptions 2-3 hold, then

and

where µ := E [ ν (˜ x ( ˆ θ , u ))] , µ CIO := E ( ν [¯ x CIO ( u )]) , and µ ∗ := E ( ν [˜ x ( θ ∗ , u )]) .

Theorem 3 states that, when C ( ¯ θ , α ) contains a θ that makes the next DM's decision optimal almost surely, conformal IO achieves upper-bounded POG and AOG. While we can meet this condition by using a large α , the bounds would be large as they increase as α increases, reflecting that the decisions may be overly conservative. Instead, we can use CP to obtain a α that achieves close-to-100% coverage and possibly add a small ∆ α ∈ R + as extra protection. Moreover, we show in Section 5

that, when using γ < 100% , conformal IO still demonstrates favorable performance compared to classic IO. Our bounds have problem-specific constants. To demonstrate tightness, we present their numerical values in Example 1 with α = π/ 4 (Table 1). They closely follow the performance of conformal IO, which outperforms classic IO by a large margin.

5 Numerical Studies

Data generation. We consider two forward problems: The shortest path problem (linear program, d = 120 ) and knapsack problem (integer program, d = 10 ). See Appendix C.2 for their formulations. We use synthetic instances, which is a common practice as there is no well established IO benchmark (Tan et al., 2020; Dong et al., 2018). For both problems, we randomly generate a ground-truth parameters θ ∗ and a dataset of N = 1000 DMs. For each DM k ∈ [ N ] , we generate her perceived parameters as ˆ θ i k = ( θ i ∗ k ∗ p i k + ϵ i k ) + + ϵ 0 for i ∈ [ d ] where p i k is uniformly drawn from [1 / 2 , 2] , ϵ i k is drawn from a normal distribution with mean 0 and standard deviation 1, and ϵ 0 = 0 . 1 . For the shortest path problem, parameters u k represent a random origin-destination pair on the network. For the knapsack problem, u k correspond to the weights of different items and the DM's budget. The item weight w i for i ∈ [ d ] are uniformly drawn from [1 , 10] and are shared among DMs. For each DM k ∈ [ N ] , we generate a budget u k = q k ∑ i w i where q k is uniformly drawn from [1 / 5 , 5] .

Experiment design. Conformal IO is compatible with any approach that can provide a point estimation of unknown parameters using decision data (Step 2 in Section 4.1). To the best of our knowledge, i) IO with the sub-optimality loss and ii) the PFYL approach from Berthet et al. (2020) are the only two methods that can perform this task at scale . We thus implement conformal IO with these two methods. They also serve as our baselines. We call both i) and ii) classic IO to emphasize that they rely on a point estimation for decision prescription, although PFYL is not an IO approach. See Appendix C for implementation details. In all experiments, conformal IO uses the training set for point estimation and the validation set for calibration, while classic IO uses the union of the training and validation sets for point estimation. So, they have access to the same amount of data and are evaluated on the same test set. Unless otherwise noted, experiments are based on a 60/20/20 train-validation-test split and are repeated 10 times with different random seeds.

Uncertainty validity. To verify Theorem 2, we empirically evaluate the out-of-sample coverage achieved by our uncertainty set under different target levels γ and sample sizes N val . The point estimation is generated by IO with the sub-optimality loss. As shown in Figure 3, when the validation set is small ( N val = 10 ), we always achieve the specified target but C ( ¯ θ , α γ ) tends to over-cover (conservatively valid). When using larger validation sets ( N val ∈ { 100 , 200 } ), our coverage level gets closer to the specified γ (asymptotic exact). These empirical findings echo our theoretical analysis.

ShortestPath

(%)

Knapsack

ICoverage

90

I Coverage

90

70

70

Target

Nva/=10

50

50

Empirical

Empirical

30

30

Nval=200

10

10

1030507090

1030507090

Target Coverage  (%)

Target Coverage  (%)

The value of robustness . As shown in Figure 4, solving RFO with an uncertainty set learned by conformal IO leads to decisions of lower AOG and POG, compared to solving FO with a point estimation from classic IO. On average, when varying γ , our approach improves AOG by 20.1-30.4% and POG by 15.0-23.2% for the shortest path problem, and improves AOG by 40.3-57.0% and POG by 13.5-20.1% for the knapsack problem. The improvement is orthogonal to the point estimation method. Our decisions are of higher quality and better align with human intuition than classic IO. Finally, the performance of conformal IO generally improves as the quality of the point estimate increases (see Appendix D for details). Therefore, even though the conformal IO pipeline is robust to estimation errors, using the best available point estimation method is still recommended.

ShortestPath-IO

ShortestPath-IO

Knapsack-10

Knapsack-10

5

3

13

4

POG

G

2

POG

G

11

O

3

9

FORFORFORFORFORFO

FORFORFORFORFORFO

FORFORFORFORFORFO

FORFORFORFORFORFO

(10)(30)(50)(70)(99)

(10)(30)(50)(70)(99)

(10)(30)(50)(70)(99)

(10)(30)(50)(70)(99)

ShortestPath-PFYL

ShortestPath-PFYL

Knapsack-PFYL

Knapsack-PFYL

5

5

3

17

POC

G

AOG

POG

AOG

3

2

13

3

FORFORFORFORFORFO

FORFORFORFORFORFO

FORFORFORFORFORFO

FORFORFORFORFORFO

(10)(30)(50)(70)(99)

(10)(30)(50)(70)(99)

(10)(30)(50)(70)(99)

(10)(30)(50)(70)(99)

Computational efficiency. As shown in Table 2, conformal IO and classic IO require similar training times. When FO is an integer program (knapsack), the training of conformal IO is even faster because it replaces a relatively large inverse integer program (associated with D train ∪ D val), which is notoriously difficult to solve (Bodur et al., 2022), with a smaller inverse integer program (associated with D train) and a set of small calibration problems that are parallelizable (Theorem 1). At the prediction time, our method achieves lower AOG and POG at the cost of solving a more challenging RFO . Nevertheless, the solution time of RFO is within one second in our instances.

Important hyper-parameters. Finally, we provide empirical evidence that sheds light on the choice of two important hyper-parameters in conformal IO: i) confidence level γ , and ii) train-validation split ratio . Regarding γ , as shown in Figure 4, the performance of conformal IO improves quickly as γ increases from 0 to 50% and remains stable and even worsens slightly after that. Hence, it is possible to improve the performance of conformal IO by carefully tuning γ using cross-validation. However, this requires an additional validation dataset. If such a dataset is unavailable, setting γ to a relatively large value (e.g., 0.99) usually yields decent performance, which aligns with our theoretical analysis. Regarding train-validation split ratio , intuitively, both the estimation and calibration can benefit from more data. However, when the dataset is small, we need to strike a balance between these two steps aiming to achieve lower AOG and POG. We implement conformal IO for the shortest path problem under different dataset sizes ( N train + N val) and train-validation split ratios ( N val / ( N train + N val ) ). As shown in Figure 5, when the dataset is small, there is no benefit of using conformal IO because we do not have enough data to obtain good point estimation and uncertainty set simultaneously. However,

the performance of classic IO quickly plateaus as the dataset grows. When given a mid- or large-sized dataset, we can generally benefit from putting more data in D val, echoing our theoretical analysis.

alidation

AOG

ion

POG

20%

-55

-11

2

3

5

latic

20%

-34

-8

1

1

8

三

40%

-16

4

5

22

19

40%

-12

-3

1

14

16

f0

r

60%

-7

13

11

24

25

f0

60%

-4

4

8

14

19

e

S

80%

-13

20

9

23

14

S

80%

-10

8

2

15

11

%

%

160320480640800

160320480640800

# of Observed Routes

# of Observed Routes

6 Conclusion and Future Work

In this paper, we propose conformal IO, a novel IO pipeline to recommend high-quality decisions that align with human intuition. We present a new approach to learning uncertainty sets from decision data, which is then utilized in a robust optimization model to prescribe new decisions. We prove that conformal IO achieves bounded optimality gaps, as measured by the ground-truth parameters and the DM's perceived parameters. We demonstrate the strong empirical performance of conformal IO via extensive numerical studies. Finally, we highlight several challenges that underscore future research directions. First, we focus on objectives that are linear in the unknown parameters. While such objectives are ubiquitous in practice, it is of interest to extend our results for general convex objectives. Second, the robust forward problem, while leading to better decisions, is computationally more costly than the forward problem. Future research can be done to accelerate its solution process. Finally, while Conformal IO consistently outperforms classic IO when the point estimation methods are fixed, our computational experiments suggest that its performance hinges on the quality of the point estimate. Future research could explore point estimation methods that directly optimize the performance of the downstream robust optimization model.

Acknowledgments

The authors are grateful to the anonymous reviewers for their valuable feedback and insightful comments. Erick Delage was partially supported by the Canadian Natural Sciences and Engineering Research Council [Grant RGPIN-2022-05261] and by the Canada Research Chair program [950230057].

References

Ahuja, R. K. and Orlin, J. B. (2001). Inverse optimization. Operations Research, 49(5):771-783.

Aswani, A., Shen, Z.-J., and Siddiq, A. (2018). Inverse optimization with noisy data. Operations Research, 66(3):870-892.

Babier, A., Mahmood, R., McNiven, A. L., Diamant, A., and Chan, T. C. (2020). Knowledge-based automated planning with three-dimensional generative adversarial networks. Medical Physics, 47(2):297-306.

Berthet, Q., Blondel, M., Teboul, O., Cuturi, M., Vert, J.-P., and Bach, F. (2020). Learning with differentiable pertubed optimizers. In Advances in Neural Information Processing Systems, volume 33, pages 9508-9519.

Bodur, M., Chan, T. C., and Zhu, I. Y. (2022). Inverse mixed integer optimization: Polyhedral insights and trust region methods. INFORMS Journal on Computing, 34(3):1471-1488.

Burton, J. W., Stein, M.-K., and Jensen, T. B. (2020). A systematic review of algorithm aversion in augmented decision making. Journal of Behavioral Decision Making, 33(2):220-239.

Chan, T. C. Y., Craig, T., Lee, T., and Sharpe, M. B. (2014). Generalized inverse multiobjective optimization with application to cancer therapy. Operations Research, 62(3):680-695.

Chan, T. C. Y. and Kaw, N. (2020). Inverse optimization for the recovery of constraint parameters. European Journal of Operational Research, 282(2):415-427.

Chan, T. C. Y., Lee, T., and Terekhov, D. (2019). Inverse optimization: Closed-form solutions, geometry, and goodness of fit. Management Science, 65(3):1115-1135.

Chan, T. C. Y., Mahmood, R., O'Connor, D. L., Stone, D., Unger, S., Wong, R. K., and Zhu, I. Y. (2023a). Got (optimal) milk? pooling donations in human milk banks with machine learning and optimization. Manufacturing & Service Operations Management, 0(0).

Chan, T. C. Y., Mahmood, R., and Zhu, I. Y. (2023b). Inverse optimization: Theory and applications. Operations Research, 0(0).

Chen, V., Liao, Q. V., Wortman Vaughan, J., and Bansal, G. (2023). Understanding the role of human intuition on reliance in human-AI decision-making with explanations. In Proceedings of the ACM on Human-Computer Interaction, volume 7, pages 1-32.

Chenreddy, A. R., Bandi, N., and Delage, E. (2022). Data-driven conditional robust optimization. In Advances in Neural Information Processing Systems, volume 35, pages 9525-9537.

Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. (2017). Deep reinforcement learning from human preferences. In Advances in neural information processing systems, volume 30.

Delage, E. and Ye, Y. (2010). Distributionally robust optimization under moment uncertainty with application to data-driven problems. Operations Research, 58(3):595-612.

Donahue, K., Kollias, K., and Gollapudi, S. (2023). When are two lists better than one?: Benefits and harms in joint decision-making. arXiv preprint arXiv:2308.11721.

Dong, C., Chen, Y., and Zeng, B. (2018). Generalized inverse optimization through online learning. In Advances in Neural Information Processing Systems, volume 31.

Dong, C. and Zeng, B. (2021). Wasserstein distributionally robust inverse multiobjective optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, number 7 in 35, pages 59145921.

Elmachtoub, A. N., Lam, H., Zhang, H., and Zhao, Y. (2023). Estimate-then-optimize versus integrated-estimation-optimization: A stochastic dominance perspective. arXiv preprint arXiv:2304.06833.

Gao, R. and Kleywegt, A. (2023). Distributionally robust stochastic optimization with wasserstein distance. Mathematics of Operations Research, 48(2):603-655.

Hu, X., Cirit, O., Binaykiya, T., and Hora, R. (2022). DeepETA: How uber predicts arrival times using deep learning. Uber Engineering Blog. Available at https://www.uber.com/en-CA/blog/deepetahow-uber-predicts-arrival-times/. Accessed: 2024-01-19.

Ji, J., Qiu, T., Chen, B., Zhang, B., Lou, H., Wang, K., Duan, Y., He, Z., Zhou, J., Zhang, Z., et al. (2023). AI alignment: A comprehensive survey. arXiv preprint arXiv:2310.19852.

Liu, M., Tang, X., Xia, S., Zhang, S., Zhu, Y., and Meng, Q. (2023). Algorithm aversion: Evidence from ridesharing drivers. Management Science, 0(0).

Mandi, J., Bucarey, V., Tchomba, M. M. K., and Guns, T. (2022). Decision-focused learning: through the lens of learning to rank. In International Conference on Machine Learning, pages 14935-14947. PMLR.

Merchán, D., Arora, J., Pachon, J., Konduri, K., Winkenbach, M., Parks, S., and Noszek, J. (2022). 2021 Amazon last mile routing research challenge: Data set. Transportation Science, 0(0).

Mohajerin Esfahani, P., Shafieezadeh-Abadeh, S., Hanasusanto, G. A., and Kuhn, D. (2018). Datadriven inverse optimization with imperfect information. Mathematical Programming, 167:191234.

Mohri, M., Rostamizadeh, A., and Talwalkar, A. (2018). Foundations of Machine Learning. MIT press.

Ng, A. Y. and Russell, S. J. (2000). Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning, page 663-670.

Nguyen, T. (2015). ETA phone home: How uber engineers an efficient route. Uber Engineering Blog. Available at https://www.uber.com/en-CA/blog/engineering-routing-engine/. Accessed: 2024-01-19.

Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. (2024). Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36.

Rönnqvist, M., Svenson, G., Flisberg, P., and Jönsson, L.-E. (2017). Calibrated route finder: Improving the safety, environmental consciousness, and cost effectiveness of truck routing in sweden. Interfaces, 47(5):372-395.

Shafer, G. and Vovk, V. (2008). A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3).

Sun, C., Liu, L., and Li, X. (2023). Predict-then-calibrate: A new perspective of robust contextual LP. In Advances in Neural Information Processing Systems.

Sun, J., Zhang, D. J., Hu, H., and Van Mieghem, J. A. (2022). Predicting human discretion to adjust algorithmic prescription: A large-scale field experiment in warehouse operations. Management Science, 68(2):846-865.

Tan, Y., Terekhov, D., and Delong, A. (2020). Learning linear programs from optimal decisions. In Advances in Neural Information Processing Systems, volume 33, pages 19738-19749.

Tang, B. and Khalil, E. B. (2024). PyEPO: A pytorch-based end-to-end predict-then-optimize library for linear and integer programming. Mathematical Programming Computation, 16(3):297-335.

Vovk, V., Gammerman, A., and Shafer, G. (2005). Algorithmic learning in a random world, volume 29. Springer.

Wainwright, M. J. (2019). High-dimensional Statistics: A non-Asymptotic Viewpoint, volume 48. Cambridge University Press.

Wilder, B., Dilkina, B., and Tambe, M. (2019). Melding the data-decisions pipeline: Decision-focused learning for combinatorial optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 1658-1665.

Wirth, C., Akrour, R., Neumann, G., and Fürnkranz, J. (2017). A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1-46.

Wu, F., Ke, J., and Wu, A. (2024). Inverse reinforcement learning with the average reward criterion. In Advances in Neural Information Processing Systems, volume 36.

Zattoni Scroccaro, P., Atasoy, B., and Mohajerin Esfahani, P. (2024). Learning in inverse optimization: Incenter cost, augmented suboptimality loss, and algorithms. Operations Research, 0(0).

A Omitted Statements and Proofs in Section 3

A.1 Poof of Lemma 1

Proof. We first show that ˆ x ∈ { (0 , 1) , ( u, 0) } almost surely. Let δ u := arccos ( 1 / √ 1 + u 2 ) , so cos δ u = 1 / √ 1 + u 2 and sin δ u = u/ √ 1 + u 2 . It is easy to verify that, when ˆ θ k ∈ Θ 1 := { (cos δ, sin δ ) | δ ∈ (0 , δ u ] } , we have ˆ x k = ˜ x ( ˆ θ k , u ) = (0 , 1) almost surely; When ˆ θ k ∈ Θ 2 := { (cos δ, sin δ ) | δ ∈ ( δ u , π/ 2) } , we have ˆ x k = ˜ x ( ˆ θ k , u ) = ( u, 0) almost surely. Since ˆ θ k is uniformly distributed in ˆ θ ∈ Θ = Θ 1 ∪ Θ 2 , the distribution of ˆ x k is

Given a sample set D = { u k , ˆ x k } k ∈ [ N ] , let N 1 and N 2 , respectively, denote the numbers of (0 , 1) and ( u, 0) in D . We next show that when N 1 > 0 and N 2 > 0 , θ u is the unique optimal solution to IO ( D ) . Specifically, in Example 1, IO ( D ) is presented as follows.

A simple calculation gives that when N 1 > 0 and N 2 > 0 , the minimum is 0 which occurs uniquely at θ = (cos δ u , sin δ u ) ; When N 2 = 0 , the minimum is 0 which occurs when θ ∈ Θ 1 ; When N 1 = 0 , the minimum is 0 which occurs when θ ∈ Θ 2 . Therefore, we have

Given the probability distribution given in Equation (15) and that D is generated using i.i.d. samples from P θ , we have

which converges to 1 as N goes to infinity. Therefore, we conclude that P ( ¯ θ N = (cos δ u , sin δ u )) converges to 1 as N goes to infinity.

A.2 Proof of Proposition 1

Proof. We first consider the AOG and POG of the decision policy ¯ x IO ( u ) := ˜ x ( θ u , u ) separately in the following two lemmas.

Lemma 4. In Example 1, let ¯ x IO ( u ) = ˜ x ( θ u , u ) . For any v ∈ R + there exists some ¯ u > 1 such that AOG(¯ x IO ) > v for any u > ¯ u AOG.

Proof. According to the definition of ˜ x , we know that ¯ x IO ( u ) is uniformly drawn from

Since the ground-truth θ ∗ = (cos( π/ 4) , sin( π/ 4)) , the true optimal solution is x ∗ = (0 , 1) with ˜ f ( θ ∗ , u ) = √ 2 / 2 . Hence, we have

Therefore, for any v ∈ R + , there exists ¯ u AOG = 2 2 v + 1 such that AOG(¯ x IO ) > v for any u > ¯ u AOG.

where and

Lemma 5. In Example 1, let ¯ x IO ( u ) = ˜ x ( θ u , u ) . for any v ∈ R + there exists some ¯ u POG > 1 such that POG(¯ x IO ) > v for any u > ¯ u POG.

Proof. According to the definition of ˜ x , ¯ x IO ( u ) is uniformly drawn from

It is easy to verify that, when ˆ θ ∈ Θ 1 := { (cos δ, sin δ ) | δ ∈ (0 , δ u ] } , we have ˆ x k = ˜ x ( ˆ θ , u ) = (0 , 1) with ˜ f ( ˆ θ , u ) = ˆ θ 2 almost surely; When ˆ θ ∈ Θ 2 := { (cos δ, sin δ ) | δ ∈ ( δ u , π/ 2 } , we have ˆ x k = ˜ x ( ˆ θ , u ) = ( u, 0) with ˜ f ( ˆ θ , u ) = u ˆ θ 1 almost surely. Since the optimal solution drawn from X OPT ( θ u , u ) is independent of the DM's perception ˆ θ , we have

The inequality holds because √ 1 + u 2 > u . Therefore, we have, for any v ∈ R + , there exists ¯ u POG = 2 v +1 such that POG(¯ x IO ) > v for any u > ¯ u POG.

Based on Lemmas 4 and 5, we conclude that for any v ∈ R + , there exists ¯ u = max { ¯ u AOG , ¯ u POG } = 2 √ 2 v +1 such that AOG(¯ x IO ) > v and POG(¯ x IO ) > v for any u > ¯ u .

A.3 Robustifying the Inverse Problem

An alternative approach to robustify the classic IO pipeline is to solve a distributionally robust inverse optimization problem. Specifically, consider the following loss function proposed by Mohajerin Esfahani et al. (2018).

Definition 4. The distributionally robust sub-optimality loss of θ is given by

where ˆ P u , ˆ x is the sample distribution of D , B p r ( ˆ P u , ˆ x ) is a p -Wasserstain ball of radius r centered at ˆ P u , ˆ x , and ρ Q is a risk measure, e.g., the value at risk.

The distributionally robust inverse optimization problem is

As shown by Mohajerin Esfahani et al. (2018), the estimated parameters from DRIO achieve bounded out-of-sample sub-optimality loss with a high probability. However, this does not imply bounded AOG and POG for the decision policy.

Lemma 6. In Example 1, θ u is an optimal solution to DRIO ( D ) .

Proof. As shown in the proof of Lemma 7, when α ∈ (0 , π/ 2) , the RFO ( C ( θ u , α ) , u ) has a unique optimal solution (0 , 1) . So ¯ x CIO ( u ) = (0 , 1) almost surely, when ¯ θ N = θ u and α ∈ (0 , π/ 2) . It is easy to verify that, when ˆ θ ∈ Θ 1 := { (cos δ, sin δ ) | δ ∈ (0 , δ u ] } , we have ˆ x k = ˜ x ( ˆ θ , u ) = (0 , 1)

almost surely; When ˆ θ ∈ Θ 2 := { (cos δ, sin δ ) | δ ∈ ( δ u , π/ 2) } , we have ˆ x k = ˜ x ( ˆ θ , u ) = ( u, 0) almost surely. Hence, we have

The inequality holds because u > 1 .

According to Lemma 1, we know that P ( ¯ θ N = θ u ) → 1 as N →∞ . So we conclude that, when α ∈ (0 , π/ 2) , we have P [ POG(¯ x CIO ) < π/ 2 √ 2 ] → 1 as N →∞ .

Lemma 6 shows that, in Example 1, the estimated parameter from DRIO ( D ) may still be θ u . Hence, the decision policy is identical to ¯ x IO whose AOG and POG can be unbounded. The fundamental reason behind these negative results is the misalignment between the sub-optimality loss and the evaluation metrics. Achieving a low sub-optimality loss means that the suggested and observed decisions are of similar quality as evaluated using the estimated parameters. However, this does not speak to the similarity between these two decisions with respect to the DM's perceived parameters (POG) or the ground-truth parameters (AOG). Therefore, the out-of-sample guarantees on the suboptimality loss do not translate into bounded AOG or POG.

A.4 Proof of Lemma 2

We consider the AOG and POG of conformal IO separately in the following two lemmas.

Lemma 7. In Example 1, let ¯ x CIO ( u ) be an optimal solution to RFO ( C ( ¯ θ N , α ) , u ) where ¯ θ N is an optimal solution to IO ( D ) with the sub-optimality loss (6) . When α ∈ (0 , π/ 2) , we have P [AOG( x CIO ) = 0] → 1 as N →∞ .

Proof. We first show that when ¯ θ = θ u and α ∈ (0 , π/ 2) , RFO ( C ( ¯ θ , α ) , u ) has a unique optimal solution (0 , 1) . Let x 1 = (0 , 1) , x 2 = (0 , 2) , x 3 = ( u, 0) and x 4 = ( u, 2) denote the four extreme points of the feasible region X ( u ) , respectively, and

̸

Since FO is a linear program, it suffices to show that, when α ∈ (0 , π/ 2) , R ( x 1 ) < min { R ( x 2 ) , R ( x 3 ) , R ( x 4 ) } because, if there exists an optimal solution that is not an extreme point, then there must exist another extreme point x i such that R ( x 1 ) = R ( x i ) where i = 1 . Next, we compare R ( x 1 ) with R ( x 2 ) , R ( x 3 ) , and R ( x 4 ) .

It is easy to verify that

For x 2 , we have

Hence, we have R ( x 1 ) < R ( x 2 ) when α ∈ (0 , π/ 2) .

For x 3 , we have

Since u > 1 , we have π/ 2 -δ u < π/ 4 < δ u < π/ 2 . We will show that R ( x 1 ) < R ( x 3 ) when α is in (0 , π/ 2 -δ u ) , [ π/ 2 -δ u , δ u ) , and [ δ u , π/ 2) . When α ∈ (0 , π/ 2 -δ u ) , we have

The second line holds due to the sum of angles identity. The third line holds due to the definition of δ u . The fourth line holds because u > 1 . The fifth line is obtained by simple manipulation. The sixth line holds due to the definition of δ u . The seventh line holds due to the sum of angles identity.

When α ∈ [ π/ 2 -δ , δ ) , we have

The second line holds because u > 1 . The third line is obtained through simple manipulation. The forth line holds due to the definition of δ u . For the fifth line, we know that α ∈ [ π/ 2 -δ u , δ u ) ⊆ [ π/ 4 -π/ 2] where cos α is strictly decreasing in α and where sin α is strictly increasing in α . Therefore, cos α < cos δ u = 1 / √ u 2 +1 and sin α ≤ sin( π/ 2 -δ u ) = cos δ u = 1 / √ u 2 +1 . Hence, the fifth line holds. The sixth line holds due to the sum of angles identity.

When α ∈ [ δ u , π/ 2) , we have R ( x 1 ) = 1 < u = R ( x 3 ) .

Hence, R ( x 1 ) < R ( x 3 ) when α ∈ (0 , π/ 2) .

For x 4 , we have

Let δ ∗ 1 denote the optimal solution to the maximization problem for calculating R ( x 1 ) . It is easy to verify that δ ∗ 1 ∈ (0 , π/ 2) when α ∈ (0 , π/ 2) . So cos δ ∗ 1 > 0 and sin δ ∗ 1 > 0 . Hence, we have

The first inequality holds because δ ∗ 1 may not be the maximizer of the problem associated with x 4 . The second inequality holds because u > 1 , cos δ ∗ 1 > 0 , and sin δ ∗ 1 > 0 .

Hence, when α ∈ (0 , π/ 2) , RFO ( C ( δ u , α ) , u ) has a unique optimal solution x 1 , so ¯ x CIO ( u ) = (0 , 1) almost surely. Given that (0 , 1) is also the optimal solution to FO ( δ ∗ , u ) , we have AOG(¯ x CIO ) = 0 when ¯ θ N = θ u and α ∈ (0 , π/ 2) . According to Lemma 1, we know that P ( ¯ θ N = θ u ) → 1 as N →∞ . So we conclude that, when α ∈ (0 , π/ 2) , we have P [AOG(¯ x CIO ) = 0] → 1 as N →∞ .

Lemma 8. In Example 1, let ¯ x CIO ( u ) be an optimal solution to RFO ( C ( ¯ θ N , α ) , u ) where ¯ θ N is an optimal solution to IO ( D ) with the sub-optimality loss (6) . When α ∈ (0 , π/ 2) , we have P [ POG(¯ x CIO ) < π/ 2 √ 2 ] → 1 as N →∞ .

Proof. As shown in the proof of Lemma 7, when α ∈ (0 , π/ 2) , the RFO ( C ( θ u , α ) , u ) has a unique optimal solution (0 , 1) . So ¯ x CIO ( u ) = (0 , 1) almost surely, when ¯ θ N = θ u and α ∈ (0 , π/ 2) . It is easy to verify that, when ˆ θ ∈ Θ 1 := { (cos δ, sin δ ) | δ ∈ (0 , δ u ] } , we have ˆ x k = ˜ x ( ˆ θ , u ) = (0 , 1) almost surely; When ˆ θ ∈ Θ 2 := { (cos δ, sin δ ) | δ ∈ ( δ u , π/ 2) } , we have ˆ x k = ˜ x ( ˆ θ , u ) = ( u, 0) almost surely. Hence, we have

The inequality holds because u > 1 .

According to Lemma 1, we know that P ( ¯ θ N = θ u ) → 1 as N →∞ . So we conclude that, when α ∈ (0 , π/ 2) , we have P [ POG(¯ x CIO ) < π/ 2 √ 2 ] → 1 as N →∞ .

B Proof of Statements in Section 4

B.1 Definitions

Definition 5 (Empirical Rademacher Complexity) . Let F be a class of functions mapping from Z = { Z 1 , Z 2 , . . . , Z m } to [ a, b ] and D be a fixed sample of size N with elements in Z , then the empirical Rademacher Complexity of F with respect to the sample D is defined as

where σ = ( σ 1 , σ 2 , . . . , σ N ) ⊺ with σ i 's being independent uniform random variables taking values in {-1 , 1 } .

Definition 6 (Rademacher Complexity) . Let P denote the distribution according to which samples are drawn. For any integer N ≥ 1 , the Rademacher complexity of a function class F is the expectation of the empirical Rademacher complexity over the samples of size N drawn from P :

Definition 7 (Growth Function) . Let H be a class of functions that take values in {-1 , 1 } . The growth function Π H : N → N for H is defined as

which measures the maximum number of distinct ways in which N data points in Z can be classified using the function class H .

B.2 Useful Lemmas

Lemma 9 (Corollary 3.1 in Mohri et al. (2018)) . Let H be a class of functions taking values in { 1 , -1 } , then, for any integer N ≥ 1 , the following holds

Lemma 10 (Theorem 4.10 in Wainwright (2019)) . For any b -uniformly bounded class of functions F , any positive integer N ≥ 1 , and any scalar δ ≥ 0 , with probability at least 1 -exp ( -Nδ 2 / (2 b 2 ) ) , we have

where R ( F ) denotes the Rademacher complexity of the function class F .

B.3 Proof of Theorem 1

Proof. We first present the extensive formulation of Problem (10). For convenience, we define ˆ Θ k := Θ OPT ( u k , ˆ x k ) for any k ∈ [ N ] . When α ∈ [0 , π ] , cos α is a strictly decreasing in α . Therefore, minimizing α is equivalent to maximizing the value of cos α . We can replace the decision variable α in Problem (10) with a new decision variable c := cos α with an additional constraint t with -1 ≤ c ≤ 1 . In addition, we introduce a new set of decision variables y k ∈ { 0 , 1 } that indicate if ˆ Θ k intersects with the learned uncertainty set ( = 1 ) or not ( = 0 ) for any k ∈ K val. Problem (10) can be presented as follows.

Constraints (39b) ensure that θ k is a member of ˆ Θ k for any k ∈ K val . Constraints (39c) decide if θ k should be taken into account when calculating the maximal cosine value c based on if ˆ Θ k intersects with C . Constraint (39d) ensures that C intersects with at least ⌈ γ ( N val + 1) ⌉ inverse feasible sets. Constraint (39e) enforces θ k to be on the unit sphere as defined in Equation (9). Constraints (39f)-(39g) specify the ranges of the decision variables.

Observing that the objective of Problem (39) is to maximize c and that decision variables θ k of different data points only interact in Constraints (39c). We can re-write Problem (39) as

Note that we replace Constraints (39e) with Constraints (41c) because the objective of Problem (41) is to maximize the inner product of θ k and ¯ θ , so the maximum only occurs when ∥ θ k ∥ 2 = 1 . We further observe that the optimal solution to Problem (40a) is to set y k = 1 for all k such that c k ≥ Γ τ ( { c k } k ∈K val ) and y k = 0 otherwise. Therefore, the optimal objective value of Problem (40a) is c = Γ τ ( { c k } k ∈K val ) corresponding to α γ = arccos Γ τ ( { c k } k ∈K val ) .

B.4 Proof of Lemma 3

Proof. Proof. For any fixed x , we have

where

where ν (ˆ x ) := √∑ i ∈ [ d ] f 2 i (ˆ x ) . The inequality follows the Cauchy-Schwartz inequality.

B.5 Proof of Theorem 2

Proof. We first prove the learned uncertainty set is conservatively valid. Following the conformal prediction language used by Vovk et al. (2005), we define a conformality measure of each data point,i.e. an observed decision and exogenous parameter pair, A ¯ θ : R n ×U → R + as follows

We note that c k = A ¯ θ (ˆ x k , u k ) for any k ∈ K val where c k is defined in Theorem 1. Let τ = ⌈ γ ( N val +1) ⌉ , and A := { A ¯ θ (ˆ x k , u k ) } k ∈K val , or equivalently, A := { c k } k ∈K val . Due to the definition of C ( ¯ θ , α ) and that α is chosen such that cos α = Γ τ ( A ) , the event ' Θ OPT (ˆ x , u ) ∩ C ( ¯ θ , α ) = ∅ ' is equivalent to ' A ¯ θ (ˆ x , u ) ≥ Γ τ ( A ) ', so

̸

̸

Assumption 1 implies that the dataset D ′ = D val ∪ { (ˆ x , u ) } is exchangeable, i.e. the ordering of the data points in D ′ does not affect its joint probability distribution (Shafer and Vovk, 2008). Therefore, the rank (from high to low) of A ¯ θ (ˆ x , u ) in A ′ := A∪{ A ¯ θ (ˆ x , u ) } is uniformly distributed in { 1 , 2 , . . . , N val +1 } . So, we have

̸

The first line holds due to the definition of τ . We obtain the second line by taking the complement of the event in the first line (inside the probability). The third line holds because A ¯ θ (ˆ x , u ) can never be strictly smaller than itself, so any elements in A ′ that are strictly smaller than A ¯ θ (ˆ x , u ) are in A . Note that this line holds only when τ ≤ N val, which occurs when γ ≤ N val / ( N val +1) , because A only has N val elements. We obtain the third line by taking the complement of the event in the second line (inside the probability). The last line holds due to Equation (43). We note that all the probabilities are over the joint distribution of D val and the new sample, i.e. D ′ .

We next prove that the learned uncertainty set is asymptotically exact. Let z k := ( u k , ˆ x k ) , Z := { z k } k ∈K val . We define a function class

Let Π H denote the growth function of H as defined in Definition 7. It is easy to verify that

because the value of h ( z, α ) is monotonically increasing in α for any fixed z ∈ Z , so changing the value of α can only leads to N val +1 different outcomes for a fixed dataset Z .

Therefore, according to Lemma 9, we have

where R N val ( H ) denotes the Rademacher complexity of H when sample size is N val, as defined in Definition 6.

We know that the value of α is chosen such that it is the smallest value that satisfies

so we have

The second inequality holds because

Since D val is i.i.d. sampled, for any fixed α , ∑ k ∈K val h ( z k , α ) /N val provides a sample average approximation to E [ h ( z, α )] , which can be interpreted as P ( Θ OPT (ˆ x , u ) ∩ C ( ¯ θ , α ) ) for any new sample ( ˆ θ , u ) from P ˆ θ , u and ˆ x = ˜ x ( ˆ θ , u ) .

By applying Lemma 10, we have, with probability at least δ = 1 -1 /N val ,

By combing (46)-(50), we have, with probability at least 1 -1 /N val ,

B.6 Proof of Theorem 3

We bound the AOG and POG of conformal IO separately in the following two propositions.

̸

Proposition 2 (Conformal IO Achieves Bounded POG) . Let ¯ x CIO ( u ) be an optimal solution to RFO ( C ( ¯ θ , α 1 ) , u ) for any u ∈ U , where ¯ θ ∈ R d and α 1 are chosen such that, for a new sample ( θ ′ , u ′ ) from P ( θ , u ) and x ′ = ˜ x ( θ ′ , u ′ ) , P ( C ( ¯ θ , α 1 ) ∩ Θ OPT ( u ′ , x ′ ) = ∅ ) = 1 . If Assumptions 2-3 hold, then

̸

Proof. We first bound the perceived optimality gap of a sampled DM. Let ( ˆ θ , u ) be a sample from P ( θ , u ) , ˆ x = ˜ x ( ˆ θ , u ) , ˆ θ CIO ( u ) denote the optimal solution to the inner maximization problem in RFO ( C ( ¯ θ , α 1 ) , u ) when the outer decision variable is set to ˆ x , ¯ θ CIO ( u ) denote the optimal solution to the inner maximization problem in RFO ( C ( ¯ θ , α 1 ) , u ) when the outer decision variable is set to ¯ x CIO ( u ) , If Θ OPT (ˆ x , u ) ∩ C ( ¯ θ , α 1 ) = ∅ , let ˜ θ be an element of Θ OPT (ˆ x , u ) ∩ C ( ¯ θ , α 1 ) , we have

The first line holds due to Lemma 3. The second line holds due to assumption 2. The third line holds due to the definition of ¯ θ CIO ( u ) . The fourth line holds because ( ¯ x CIO ( u ) , ¯ θ CIO ( u ) ) is an optimal solution to RFO ( C ( ¯ θ , α 1 ) , u ) . The fifth line holds due to Lemma 3. The sixth line holds because both ˆ θ CIO ( u ) and ˜ θ are in C ( ¯ θ , α 1 ) so the angle between them is no larger than 2 α 1 . Since both ˆ θ CIO ( u ) and ˜ θ are on the unit sphere, the L 2 distance between them are bounded by 2(1 -cos 2 α 1 ) .

̸

Proposition 3 (Conformal IO Achieves Bounded AOG) . Let ¯ x CIO ( u ) be an optimal solution to RFO ( C ( ¯ θ , α 1 ) , u ) for any u ∈ U , where ¯ θ ∈ R d and α 1 are chosen such that, for a new sample ( θ ′ , u ′ ) from P ( θ , u ) and x ′ = ˜ x ( θ ′ , u ′ ) , P ( C ( ¯ θ , α 1 ) ∩ Θ OPT ( u ′ , x ′ ) = ∅ ) = 1 . If Assumptions 2-3 hold, then

Proof. We first derive an upper bound on the optimality gap of the suggested decision ¯ x CIO ( u ) as evaluated using θ ∗ for any u ∈ U . Let ( ˆ θ , u ) be a sample from P ( θ , u ) , ˆ x = ˜ x ( ˆ θ , u ) , and ˜ θ be an element of Θ OPT (ˆ x , u ) ∩ C ( ¯ θ , α 1 ) , which is non-empty almost surely because α 1 is chosen such that P ( Θ OPT (ˆ x , u ) ∩ C ( ¯ θ , α 1 ) ) = 1 . Let ¯ θ CIO ( u ) denote the optimal solution to the inner maximization problem in RFO ( C ( ¯ θ , α 1 ) , u ) when the outer decision variable is set to ¯ x CIO ( u ) . For any u ∈ U , let x ∗ ( u ) := ˜ x ( θ ∗ , u ) and θ ∗ CIO ( u ) denote the optimal solution to the inner maximization problem in RFO ( C ( ¯ θ , α 1 ) , u ) when the outer decision variable is set to x ∗ ( u ) , we have

The first line holds because of Lemma 3. The second line holds due to Assumption 3. The third line holds because f is linear in θ . The expectation is taken over P θ . The fourth line holds due to Lemma

¯

3. The fifth line holds due to Assumption 2. The sixth line holds because of the definition of θ CIO ( u ) . The seventh line holds because ( ¯ x CIO ( u ) , ¯ θ CIO ( u ) ) is an optimal solution to RFO ( C ( ¯ θ , α 1 ) , u ) . The eigth line holds due to Lemma 3. The ninth line holds since both θ ∗ CIO ( u ) and ˜ θ are on the unit sphere and the angle between them is no greater than 2 α 1 , then the L 2 distance between them is upper bounded by 2(1 -cos 2 α 1 ) .

Next, we bound the AOG of ¯ x CIO. We have

C Numerical Experiment Details

C.1 Computational Setup

All the algorithms are implemented and test using Python 3.9.1 on a MacBook Pro with an Apple M1 Pro processor and 16 GB of RAM. Optimization models are implemented with Gurobi 9.5.2.

C.2 Forward Problems

C.2.1 Shortest-path

We consider the shortest path problem on a 5 × 5 grid network G ( N , E ) where N and E indicate the node and edge sets, respectively. Let E + ( i ) and E -( i ) denote the sets of edges that enter and leave node i ∈ N , respectively. Let u o and u d denote the origin and destination of the trip, respectively. We define x ij ∈ E as binary decision variables that take 1 if road ( i, j ) is traversed for any ( i, j ) ∈ E . The shortest path problem is presented as follows.

The objective function minimizes the total travel cost. The first set of constraints are the flowbalancing constraints that make sure we can find a path from u o to u d . The second set of constraints specify the range of our decision variables. Note that the constriant matrix is totally unimodular, so we can replace the binary constraints with 0 ≤ x ij ≤ 1 for any ( i, j ) ∈ E when implementing this model.

C.2.2 Knapsack

We consider a knapsack problem of d = 10 items. We define binary decision variables x i that indicate if item i ∈ [ d ] is selected ( = 1 ) or not ( = 0 ). The knapsack problem is presented as follows.

The objective maximizes the total value of the selected items. The first constraint enforces a total budget for item selection. The second set of constraints specify the range of our decision variables.

C.3 Obtaining a Point Estimation

We consider two methods to obtain point estimations of the unknown parameters. They are i) datadriven inverse optimization with the sub-optimality loss and ii) the gradient-based method proposed by Berthet et al. (2020). We implement the method from Berthet et al. (2020) with the package provided by Tang and Khalil (2024). Hyper-parameters are tuned using a separate validation set of 200 decision data points. Batch size is set to 64. We use the Adam optimizer with an initial learning rate of 0.1. We train the model for 20 epochs.

We present the implementation details of the data-driven inverse optimization method next. We consider solving

This problem is initialized without Constraints (56b), which were added iteratively using a cuttingplane method. Specifically, in each iteration, after solving Problem (56), let θ ′ and { l ′ k } k ∈K train be the optimal solution. For each data point k ∈ K train, we solve the following sub-problem

Let x ′ k be the optimal solution to the sub-problem. If l ′ k < θ ′ ⊺ ˆ x k -θ ′ ⊺ x ′ k , we add the following cut to Problem (56)

We keep running this procedure until no cut is added to the master Problem (56).

C.4 Solving the Calibration Problem

C.4.1 Shortest Path

For each data point in the validation set, we calculate the value of c k by solving the following problem

where w ∈ R N and v ∈ R |E| + , respectively, denote the dual variables associated with the flowbalancing constraints and the capacity constraints in the primal problem. The first constraint enforces strong duality. The second set of constraints are the dual feasibility constraints. The last constraint ensures the optimal solution is on the unit sphere. Note that we do not need to enforce ∥ θ ∥ 2 = 1 because this is a maximization problem.

C.4.2 Knapsack

For each data point in the validation set, we calculate the value of c k by solving the following calibration problem

We initialize this problem without Constraints (60b). In each iteration, after solving the calibration problem, let θ ′ denote the optimal solution. We solve FO ( θ ′ , u k ) and let x ′ denote the optimal solution. If θ ′ ⊺ x ′ > θ ′ ⊺ ˆ x k , we then add the corresponding cut to the model. We keep running this process until no cut is added.

C.5 Solving the Robust Forward Problem

Let α = cos -1 (Γ k ( { c k } k ∈K val )) . We next solve the following robust model to recommend a new decision to prescribe a decision given a u ∈ U .

We initialize this problem as follows.

We initialize ˜ Θ = ∅ . We first solve Problem (62), let x ′ and Ω ′ denote the optimal solution. Then we solve the following sub-problem

Let θ ′ denote the optimal solution to the sub-problem. If θ ⊺ x ′ > Ω ′ , then we add θ ′ to ˜ Θ and re-solve Problem (62). We keep running this procedure until no new solution is added to ˜ Θ .

D Additional Computational Results

In this section, we conduct additional computational experiments to assess how point estimate quality affects the performance of the Conformal IO pipeline. Specifically, we focus on the knapsack problem, where we randomly generate point estimates with an angular deviation δ from the groundtruth parameter θ ∗ , with δ serving as a proxy for point estimate quality. We then apply our calibration method to the point estimate and subsequently use the robust forward problem to generate decision recommendations. We vary the value of δ and report the average (std) out-of-sample AOG and POG achieved by our pipeline in Table 3.

In general, conformal IO benefits from more accurate point estimates (i.e., when δ is small). However, the trend is not always strictly monotonic. For instance, as δ increases from 0 to π/ 10 , the POG initially decreases before rising again. While beyond the scope of this work, exploring point estimation methods that optimize for the performance of the downstream robust optimization could be a valuable direction for future research.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Theoretical results are proved in Sections 3 and Section 4 (complete proofs are in the appendix). These results are verified numerically in Section 5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations and future research directions are discussed in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Assumptions are stated and justified in Section 3.1.1. Proofs are in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Experiment details are well documented in the paper. We also open source our implementation at https://anonymous.4open.science/r/ConformalIO-B776 .

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Data and source code available at https://anonymous.4open.science/ r/ConformalIO-B776 .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Yes, this information is documented in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report error bars/standard errors for all experimental results in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: This information is disclosed in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code and Ethics and confirm that our paper conform with it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This is a theoretical/methodological paper that presents foundational research in optimization. We do not see a direct path to any negative societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This is a theoretical/methodological paper that presents foundational research in optimization. It does not involve any high-risk data/model.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We properly cite code/papers on which our research is built on.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Our code and data along with documentations are available at https:// anonymous.4open.science/r/ConformalIO-B776 .

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper proposes a robust optimisation method for decision problems that provides not only a point estimate but also an uncertainty set. The methodology is tested on decision making tasks such as shortest path and the knapsack problem and shows favourable results when compared to classical inverse optimisation methods. The theoretical results focus on the validity of the uncertainty set as well as bounds on actual and perceived optimality of the decision policy.

STRENGTHS:
The paper addresses an important problem of inverse optimisation taking advantage of the CP framework. The proposed pipeline is clear and easy to follow. To the best of my knowledge, the work is original. The writing quality and clarity is good.

WEAKNESSES:
My only concern is the practical applicability of the methodology. While I appreciate the motivation for the approach and the extensive discussion of the evaluation metrics, I feel the gains offered by the methodology are minor, particularly given the fact that large datasets will be needed to calibrate the model and choose the hyperparameters.

QUESTIONS:
Can you explain the significance of the $\Delta_\alpha$ term in more detail (line 258)?

While you mention some inverse reinforcement learning literature (e.g. line 72), I wonder if you are aware of any similar probabilistic approaches in that literature?

I appreciate that providing a real-world example may be difficult, but can you comment on the possible issues of using your approach on real-datasets. Are any of the assumptions in your analysis likely to be violated (e.g. due to distribution shift or quality/size of data)?

LIMITATIONS:
The limitations are discussed in Sec. 6 as well as throughout the analysis and experiment sections.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

comments: SUMMARY:
This paper proposes conformal inverse optimization (CIO) which uses conformal prediction to learn uncertainty sets for parameter estimates in IO. These uncertainty sets are then used to formulate a robust version of the forward optimization problem. The authors provide intuition for their method and also provide theory to show that CIO is a principled approach. Experiments on synthetic shortest path and knapsack problems demonstrate the promise of CIO.

STRENGTHS:
* The authors motivate their approach of conformal inverse optimization using examples (e.g., Example 1).

* The figures in section 3 provide helpful intuition for the proposed approach. Nicely done!

* The authors show that their method is theoretically justified — it obtains uncertainty set validity and also allows for a bound on the POG.

* The authors provide a code implementation of their method, which makes their experiments easier to reproduce. Furthermore, the experiments are described in sufficient detail in the main paper.

WEAKNESSES:
* The setup in section 3.1 could benefit from more intuition. I think having a running example (e.g., rider-driver matching) where $x$, $u$, $\theta$, and $\mathcal X(u)$ have a practical meaning would be very helpful.

* Line 85: I think the domain of $f$ should be $\mathbb{R}^{n} \times \mathbb{R}^{d}$, not $\mathbb{R}^{n \times d}$.

* The idea of $\alpha$ (line 202) could be explained better. How exactly does $\alpha$ control the size of the uncertainty set?

QUESTIONS:
* Line 124: What is meant by “algorithm aversion”?

* How is the uncertainty set calibration related to the classic conformal prediction framework?

LIMITATIONS:
Please see “Weaknesses”
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

comments: SUMMARY:
This paper considers a robust inverse optimization problem. The author provides an uncertainty set with probabilistic guarantees for the estimated parameters, illustrates the advantages of robust inverse optimization over deterministic inverse optimization through an example, and demonstrates its superiority in experiments on other problems.

STRENGTHS:
-  This paper considers robust inverse optimization problems, which is a good complement to existing inverse optimization literature.
-  The procedure of efficiently solving CP is described in good detail, which is helpful for implementation.
-  The paper is overall clearly written and well-structured. The authors use a simple intuitive example to illustrate the motivation of robustifying the inverse optimization, which I find really helpful.

WEAKNESSES:
- The present of the uncertainty set in (9) may need justifications.
-  Although the RFO method can work based on any point estimation method, the performance of the solution usually depends on the quality of the point estimation method. I wonder if the authors could discuss this in the experiments.

QUESTIONS:
- The failure of the point estimation method in Example 1 is partially because of the sub-optimality loss. Will the conclusion be changed if using another loss?

LIMITATIONS:
The authors have discussed the limitations in Section 5 & 6.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn some supervised decision making, one usually leverage solution of optimization problem sequentially for future decisions. The authors argue that a single point estimation might not be enough and rather suggest estimating a whole uncertainty set on the model parameter and instead suggest a minimax robust optimization problem for future decisions. A nice contribution is to provide theoretical guarantees on the optimality gap of the proposed solution, with high probability.\n\nSTRENGTHS:\n- The problem considered is quite interesting for safer AI based decision making.\n\n- The derivation of uncertainty sets seems novel and quite intriguing.\n\nWEAKNESSES:\nThe presentation is quite convoluted and hard to follow at several points. See questions below.\n\nQUESTIONS:\n1-) **Presentation** : After introducing the framework, the introduced example 1 seems quite toyish and the presentation here is quite confusing:\n\n- example 1 is followed by lemma 1 that only applies to example 1, which is followed by proposition 1 that only applies to example 1 ... So I am quite curious to see how this generalize to other problems. Otherwise remove the \"lemma\" and \"proposition\" denomination and set it all under the Example 1 umbrella.\n\n- Same for lemma 2.\n\n- I am curious how the main contributions Theorem 2 and 3 applies to the given example 1. Also how does it applies to classical linear model with Gaussian noise? \n\n- In line 86, it seems that the authors assume the knowledge of the distribution $\\mathbb{P}_u$. But it is not clear how is this used in the given examples, and how does it appears in the main results.\n\n\n2) **Related works** : The authors might want to *remove* some overclaim in the paper e.g. in the conclusion *\"We present the first approach to learning uncertainty sets from decision data, which is then utilized in a robust optimization ...\"*, here are some refs of similar previous work, just to name a few.\n\n- (Siddiqui et al 2015) https://link.springer.com/article/10.1057/jors.2014.41\n- (Csaji etal 2015)  https://arxiv.org/abs/1807.08216\n- (Dekker etal 2008) https://hal.science/hal-03046711\n- (Guille-Escuret etal 2024) https://arxiv.org/abs/2401.15254\n\nBeyond the downstream applications that might differ, how the contributions of this paper compare to such previous work?\n\n3-) In the main text, it is unclear how conformal prediction framework is used. As far as I know, uncertainty sets derived from conformal prediction applies to *observations* and not model parameters directly [besides some very recent works that provide early results for such extension]. \n\n4-) How does one obtain a reasonable $\\eta$ in assumption 2. How such a choice affects the presented bound? Typically I am surprised it does not even appears in Equation 12. For example if one consider overparametrized linear model, the sets of solutions are usually unbounded. I also do not understand why we can use scale invariance here without loss of generality. The authors might want to write this part more explicitly\n\n_______ Edit after rebuttal _________\n\nAfter considering authors rebuttal and discussions, I raised my score to support paper acceptance.\n\nLIMITATIONS:\nOK", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-16"}