Title: Contextual Linear Optimization with Bandit Feedback
Venue: NeurIPS 2024
Accepted: yes
Keywords: Contextual stochastic optimization

Personalized decision making

Prescriptive analytics
Abstract: Contextual linear optimization (CLO) uses predictive contextual features to reduce uncertainty in random cost coefficients and thereby improve average-cost performance. An example is the stochastic shortest path problem with random edge costs (e.g., traffic) and contextual features (e.g., lagged traffic, weather). Existing work on CLO assumes the data has fully observed cost coefficient vectors, but in many applications, we can only see the realized cost of a historical decision, that is, just one projection of the random cost coefficient vector, to which we refer as bandit feedback. We study a class of offline learning algorithms for CLO with bandit feedback, which we term induced empirical risk minimization (IERM), where we fit a predictive model to directly optimize the downstream performance of the policy it induces. We show a fast-rate regret bound for IERM that allows for misspecified model classes and flexible choices of the optimization estimate, and we develop computationally tractable surrogate losses. A byproduct of our theory of independent interest is fast-rate regret bound for IERM with full feedback and misspecified policy class. We compare the performance of different modeling choices numerically using a stochastic shortest path example and provide practical insights from the empirical results.

Contextual Linear Optimization with Bandit Feedback

Abstract

Contextual linear optimization (CLO) uses predictive contextual features to reduce uncertainty in random cost coefficients and thereby improve average-cost performance. An example is the stochastic shortest path problem with random edge costs (e.g., traffic) and contextual features (e.g., lagged traffic, weather). Existing work on CLO assumes the data has fully observed cost coefficient vectors, but in many applications, we can only see the realized cost of a historical decision, that is, just one projection of the random cost coefficient vector, to which we refer as bandit feedback. We study a class of offline learning algorithms for CLO with bandit feedback, which we term induced empirical risk minimization (IERM), where we fit a predictive model to directly optimize the downstream performance of the policy it induces. We show a fast-rate regret bound for IERM that allows for misspecified model classes and flexible choices of the optimization estimate, and we develop computationally tractable surrogate losses. A byproduct of our theory of independent interest is fast-rate regret bound for IERM with full feedback and misspecified policy class. We compare the performance of different modeling choices numerically using a stochastic shortest path example and provide practical insights from the empirical results.

1 Introduction

Contextual linear optimization (CLO) models the use of predictive features (context variables) to improve decision making in linear optimization with random coefficients. In CLO, we consider a decision z ∈ Z that incurs an uncertain cost Y ⊤ z determined by a random cost vector Y ∈ R d that is not observed at decision time. We do, however, observe predictive features X ∈ R p prior to decision, which help reduce uncertainty. CLO can be expressed either as a contextual stochastic optimization problem or a linear optimization problem where the cost vector is a conditional expectation:

We assume throughout that Z is a polytope ( Z = Conv( Z ∠ ) for some vertex set ∣ ∣ Z ∠ ∣ ∣ < ∞ with sup z ∈Z ∠ ∥ z ∥ ≤ B ) and Y is bounded (without loss of generality, Y ∈ Y = { y : ∥ y ∥ ≤ 1 } ).

CLO has been the focus of much recent work [e.g., 9, 11, 14, 27, 30], due to its relevance to datadriven operational decision making in many applications such as network optimization, portfolio optimization, product recommendation, etc. The key question in these is how to use data to learn a good policy, π : R p →Z , mapping feature observations to effective decisions. All of this literature studies an offline setting, where a batch of existing data ( X i , Y i ) for i = 1 , . . . , n is observed. These data are viewed as random draws from the joint distribution of ( X,Y ) , and the goal is to learn an effective decision policy from them. Crucially, this assumes that we fully observe the random cost

∗

vector Y i 's, meaning that we fully know the corresponding costs Y ⊤ i z for any potential decision z . This may be unrealistic in many applications.

Consider for example the instantiation of Eq. (1) as a stochastic shortest path problem of transporting one unit from a start node to a desired end node, where Y contains the travel time on each of d edges, z the binary decision whether to traverse each edge, and Z constrains flow conservation. The data we would need in order to apply existing approaches here would consist of simultaneous observations of the travel times on every single edge, encapsulated in the elements of Y i 's. However, such ideal observations may not be available. Indeed, modern travel time prediction models are often based on data from historical individual trips. For example, such is the case with Uber Movement data [26, 33], which is based on the total length of historical rides. Namely, instead of observing the entire Y i , we only observe the total travelling time C i = Y ⊤ i Z i for the path Z i in a historical trip i . We term this observation model bandit feedback as it corresponds to observing the cost only of a given decision and not the counterfactual cost of any alternative decision, as in bandit problems [25].

In this paper, we study the contextual linear optimization problem with bandit feedback and make several contributions. First, we adapt the end-to-end induced empirical risk minimization approach to the bandit feedback setting. This approach allows us to optimize decision policies by directly targeting the decision objective. We provide three different methods to identify the expected cost of any given policy and show how to estimate the expected decision cost from bandit-feedback data. Second, we derive upper bounds on the regret ( i.e. , decision sub-optimality) of the policy that minimizes the estimated objective. Our regret analysis accounts for the misspecification of the policy class and incorporates a margin condition that potentially enables a faster regret rate. This significantly extends existing theory for full-feedback CLO, and as a byproduct, we provide a novel fast rate bound for full-feedback CLO with misspecification. Finally, we demonstrate that existing surrogate losses for full-feedback CLO (such as the well-known SPO+ loss in [11]) can be adapted to our setting, enabling efficient policy optimization. We empirically test this in simulated shortest path problems and provide practical insights.

1.1 Background: The Full Feedback Setting

We first review two major approaches to the CLO problem with full feedback: an estimate-thenoptimize (ETO) approach and an end-to-end induced empirical risk minimization (IERM) approach. To this end, we need to define the generic plug-in policy π f for any given f : R p → R d :

Note that for any given covariate value x , this corresponds to a linear programming problem with coefficients given by f ( x ) . An optimal policy in Eq. (1) corresponds to π f 0 . Without loss of generaility, we restrict the value of π f ( x ) to the set of vertices Z ∠ of the polytope Z with ties broken according to some fixed rule (e.g., a total ordering over Z ∠ such as lexicographic).

The ETO approach starts with estimating f 0 in Eq. (1) by any supervised learning method for predicting Y given X . This can, e.g. , be implemented by minimizing a prediction fitness criterion (e.g., sum of squared errors) over a hypothesis class of functions F ⊆ [ R p → R d ] (e.g., linear functions, decision trees, neural networks). After training an estimator ˆ f of f 0 , the ETO approach then makes decisions according to the induced policy π ˆ f . This approach is a straightforward two-step approach, but it ignores the downstream optimization task in choosing ˆ f ∈ F .

In contrast, some recent literature propose to integrate the estimation and the optimization, directly searching for a decision policy to minimize the decision cost. Following [14], we consider an IERM formulation that minimizes the sample average cost over the class of plug-in policies induced by F :

This approach is end-to-end in that it directly targets the decision-making objective. Recent literature demonstrates that end-to-end approaches like IERM often outperform the ETO approach [12]. The benefits of end-to-end approaches are significant especially in the misspecified setting - that is, when the function class F fails to contain the true expected cost function f 0 , and the policy class Π F does not include the globally optimal decision policy π f 0 [e.g., 11, 14]. While Eq. (3) involves a challenging bi-level optimization due to the definition of Π F , practical computational approximations have been proposed [11, 16].

Nonetheless, the IERM formulation in Eq. (3) requires observing the full feedback Y i 's in the data. In this paper, we study how to extend this approach to the setting with bandit feedback.

1.2 Our Problem: CLO with Bandit Feedback

We now formally define the data generating process in the bandit feedback setting. Assume we have an offline dataset consisting of n data points D = { ( X 1 , Z 1 , C 1 ) , . . . , ( X n , Z n , C n ) } that are independent draws from a distribution P on ( X,Z,C ) generated in the following way. We first draw ( X,Z,Y ) where the ( X,Y ) distribution is as in the full feedback setting and Z is generated according to some historical decision policies for data collection. Then we set C = Z ⊤ Y and omit the full Y . Below we impose some basic assumptions on the generating process of the historical decision Z .

Assumption 1. The data generating process satisfies the following two properties:

(Ignorability) E [ C | Z, X ] = Z ⊤ f 0 ( X ) (which could follow from Z ⊥ ⊥ Y | X ).

(Coverage) inf z ∈ span( Z ): ∥ z ∥ =1 E [( Z ⊤ z ) 2 | X ] > 0 almost surely.

The ignorability condition is a common assumption that plays a fundamental role in learning with partial feedback and causal inference (see the literature in Section 1.3). This condition requires that the assignments of the historical decisions Z do not depend on any unobserved factor potentially dependent on Y . The coverage condition requires that given the covariates X , historical decisions Z can explore all directions of the linear span of the constraint set Z ; otherwise it may be impossible to evaluate certain policies from the observed costs of historical decisions. This is an analogue of the overlap assumption in learning with partial feedback and causal inference [17].

The learning task is to use these data D to come up with a data-driven policy ˆ π ( x ) that has low regret

1.3 Background: Off-Policy Evaluation and Optimization

An important special case is when Z ∠ = { (1 , 0 , . . . , 0) , . . . , (0 , . . . , 0 , 1) } is the canonical basis and Z is the simplex. This corresponds to choosing one of d actions (or randomizing between them). The full-feedback problem in this case is cost-sensitive classification [10]. In the bandit feedback case, with the restriction Z ∈ Z ∠ , it is known as a logged or offline contextual bandit, and a long line of literature studies the problems of estimating and optimizing V ( π ) [1, 8, 18-20, 23, 24, 32, 34-36, 4245]. It is closely related to causal inference, viewing each component of Y as the potential outcome for the corresponding action (or treatment), where we only see the realized outcome C corresponding to the factual action with index 1 in Z ∈ Z ∠ and see nothing about counterfactual actions Z ∠ \{ Z } . Two key differentiating characteristics of our problem with a general polytope constraint Z are the opportunity to leverage the linear cost structure to extrapolate from the costs of historical decisions to costs of other decisions, and the presence of non-trivial constraints on the output of decision policies.

Going beyond just finite discrete arms, [21, 22, 29] consider the logged contextual bandit with a continuum of arms, which is not generally representable in terms of the problem in Section 1.2. These works make no restrictions on the relationship between the expected potential outcome and the corresponding action except generic smoothness, and leverage nonparametric approaches such as kernel weighting. Closest to our work is [6], which imposes a semiparametric assumption on this relationship. This includes our problem (Section 1.2) as a special case under the restriction that, conditional on X , expected outcomes are linear in actions (note that our E [ C | Z, X ] is linear in Z according to ignorability in Section 1.2). Relative to this work, our unique contributions are the use of induced policy classes to naturally deal with the constraints in Z , the adaptation of computational approximation such as SPO+, and obtaining the fast rates for the regret under misspecification.

2 Induced Empirical Risk Minimization with Bandit Feedback

Our IERM approach in the bandit setting follows the same idea of Eq. (3), directly minimizing an estimate of expected costs. However, since we do not observe Y i in our data, it is not as straightforward as a sample average, ∑ n i =1 Y ⊤ i π ( X i ) /n . Instead, we need alternative ways to identify V ( π ) .

Define Σ 0 ( X ) = E [ ZZ ⊤ | X ] and recall f 0 ( X ) = E [ Y | X ] , which we will use to characterize the policy value V ( π ) . We refer to these functions as nuisance functions following the existing literature on off-policy evaluation and learning in contextual bandits or reinforcement learning [39], because they are not directly used for decision-making but serve merely as intermediaries for evaluating the policy value. Let θ ( x, z, c ; f, Σ) be a score function such that

where E P denotes taking expectation over P . Proposition 1 summarizes a few possible choices.

Proposition 1. The following choices of θ ( x, z, c ; f, Σ) all satisfy Eq. (5) :

(Direct Method) θ DM ( x, z, c ; f, Σ) = f ( x ) ;

(Inverse Spectral Weighting) θ ISW ( x, z, c ; f, Σ) = Σ † ( x ) zc ;

(Doubly Robust) θ DR ( x, z, c ; f, Σ) = f ( x ) + Σ † ( x ) z ( c -z ⊤ f ( x )) .

Here Σ † denotes the Moore-Penrose pseudo-inverse of matrix Σ .

The doubly robust identification in Proposition 1 is a generalization of the identification in [6] in that we allow for rank-deficient Σ 0 by using a pseudo-inverse. Otherwise, our work significantly differs from [6] in terms of policy class specification, computation and theoretical analyses (see Section 1.3). Note that identification in Proposition 1 exploits the linearity of the decision cost, which significantly improve upon algorithms that ignore the linear structure and naively extend existing offline bandit learning using discrete actions Z ∠ ; see Section 5 and Appendix C.2 for numerical evidence.

It remains to estimate Eq. (5), since we know neither P nor the nuisance functions f 0 , Σ 0 . Following the approach of [1, 45] for logged contextual bandits, we adapt the cross-fitting procedure advocated in [5]. For simplicity, we focus on the twofold version and assume n is even. The extension to K -fold version is straightforward.

First, we split D into two equal-sized subsamples, D 1 and D 2 , each with n/ 2 data points. We can use D 1 as an auxiliary sample to estimate the nuisance functions f 0 and Σ 0 , denoting the estimates as ˆ f 1 and ˆ Σ 1 . We then use D 2 as a main sample to obtain an estimate of V ( π ) using ˆ f 1 and ˆ Σ 1 :

Of course, we can switch the roles of D 1 and D 2 , i.e. , we use D 2 to get nuisance estimates ˆ f 2 and ˆ Σ 2 , then use D 1 to estimate V ( π ) . Finally, given an induced policy class Π F , the IERM policy ˆ π minimizes the average of the above two V ( π ) estimates over Π F :

Remark 1 (Estimation of f 0 ) . The estimator ˆ f ( x ) can be obtained by minimizing the least square loss over some appropriate function class F N :

Note that two places in Eq. (6) require the specification of a function class for modeling f 0 : the F class used to induce the policy class Π F , and the F N class used to construct nuisance estimators to evaluate the expected cost of induced policies. In practice, we do not need to use the same class for F and F N . In fact, it might be desirable to use a more flexible function class for F N to make sure it is well-specified for accurate policy evaluation, and a simpler class for F to make the policy optimization more tractable. We numerically test out different choices of F and F N in Section 5.

Remark 2 (Estimation of Σ 0 ) . There are multiple ways to estimate Σ 0 . For example, [6] suggest estimating Σ 0 by running a multi-task regression for all ( j, k ) entries to the matrix over some appropriate hypothesis spaces S jk : ˆ Σ = arg min Σ 11 ∈S 11 ,..., Σ dd ∈S dd ∑ i ∈D ∥ Z i Z ⊤ i -Σ( X i ) ∥ 2 Fro . To ensure a positive semi-definite estimator, we may posit each hypothesis Σ to be the outer product of some matrix-valued hypothesis of appropriate dimension. Alternatively, we may only need to consider finitely many feasible decisions z 1 , . . . , z m , such as the feasible paths for stochastic shortest path problems (see experiments in Section 5) or more generally the vertices in Z ∠ . Then we can first estimate the propensity scores e 0 ( z | X ) for z = z 1 , . . . , z m using an suitable estimator ˆ e ( z | X ) and then estimate Σ 0 ( X ) = ∑ m j =1 z j z ⊤ j e 0 ( z j | X ) by ˆ Σ( X ) = ∑ m j =1 z j z ⊤ j ˆ e ( z j | X ) .

3 Theoretical Analysis

In this section, we provide a theoretical regret analysis for the IERM approach with bandit feedback, allowing for model misspecification in the induced policy class Π F . That is, we allow the globally optimal policy π f 0 to be not included in the class Π F . We derive an upper bound on the regret of the IERM approach, in terms of the complexity of the policy class Π F , its misspecification bias, and the estimation errors of nuisance functions.

Before stating the main theorem, we define a few important notations. Let ˜ π ∗ be the best-in-class policy that minimizes expected regret over Π F :

We note that ˜ π ∗ can be different from the global optimal policy π f 0 if π f 0 ̸∈ Π F , and Reg (˜ π ∗ ) characterizes the extent of misspecification in the induced policy class Π F . For j = 1 , 2 , we define the function class

For any function class G : X × Z × C → R , we define the local Rademacher complexity as

where ϵ i are i.i.d. Rademacher variables, and ∥ g ∥ 2 = √ E P [ g 2 ( X,Z,C )] .

Throughout Section 3, we impose two assumptions. Assumption 2 concerns the algorithms we use to estimate the nuisance functions f 0 , Σ 0 . It requires that the nuisance estimates are close enough to their true values, and ∥ θ ( x, z, c ; ˆ f, ˆ Σ) ∥ is bounded.

Assumption 2 (Nuisance Estimation) . The nuisance estimators ˆ f, ˆ Σ trained on a sample of size n satisfy that ∥ θ ( x, z, c ; ˆ f, ˆ Σ) ∥ ≤ Θ for all x, z, c , and for any δ ∈ (0 , 1) and π ∈ Π F , with probability at least 1 -δ ,

In Section 3.1, we will relate Rate N ( n, δ ) to the errors in estimating the individual nuisance functions f 0 and Σ 0 for different choices of score function θ .

Assumption 3, which we term the margin condition, controls the density of the sub-optimality gap near zero in the CLO problem instance and allows us to get faster regret rates. This type of condition was originally considered in the binary classification literature [2, 38]. It is recently extended to contextual linear optimization by [14], which we describe below.

̸

Assumption 3 (Margin Condition) . Let Z ∗ ( x ) = arg min z ∈Z f 0 ( x ) ⊤ z , and ∆( x ) = inf z ∈Z ∠ \Z ∗ ( x ) f 0 ( x ) ⊤ z -inf z ∈Z f 0 ( x ) ⊤ z if Z ∗ ( x ) = Z and ∆( x ) = 0 otherwise. Assume for some α, γ ≥ 0 ,

Lemmas 4 and 5 in [15] show that Assumption 3 generally holds with α = 1 for sufficiently wellbehaved f 0 and continuous X and with α = ∞ for discrete X . Moreover, any CLO instance trivially satisfies α = 0 . Overall, a larger α means that the sub-optimality gap between the best and second-best decisions tends to be large in more contexts, so it is easier to distinguish the optimal decisions from others. We will show that a larger α parameter could lead to faster regret rates.

3.1 Main Theorem

We now provide an upper bound on the regret of the IERM policy ˆ π in Eq. (6).

Theorem 1. Suppose Assumptions 2 and 3 hold, and Z ∗ ( X ) defined in Assumption 3 is a singleton almost surely. Suppose there exists a positive number ˜ r (that depends on n ) that upper bounds the critical radii of the function classes G 1 , G 2 almost surely (i.e., R n/ 2 ( G 1 , ˜ r ) ≤ ˜ r 2 and R n/ 2 ( G 2 , ˜ r ) ≤

˜ r 2 ) and satisfies the inequalities 3 n ˜ r 2 / 128 ≥ log log 2 (1 / ˜ r ) , and 2 exp ( -3 n ˜ r 2 / 128 ) ≤ δ/ 4 . Then, there exists a positive constant ˜ C ( α, γ ) such that with probability at least 1 -δ , we have

The upper bound in Theorem 1 involves several different types of error terms. The first type is the critical radii ˜ r that characterize the complexity of the function classes G 1 , G 2 . This is a common complexity measure for function classes in statistics and machine learning [41]. For example, as we will discuss below, the critical radii scale as ˜ O ( √ η/n ) if G 1 , G 2 are VC subgraph classes of dimension η . The second type is the term Rate N ( n/ 2 , δ/ 4) resulted from the errors in estimating the nuisance functions f 0 , Σ 0 . Similar nuisance estimation errors also appear in the previous literature on offline contextual bandit learning [1, 5, 45] or more general learning problems that involve nuisance functions [13]. Below we will further discuss this term for different choices of score functions θ . The third type is the misspecification error term Reg (˜ π ∗ ) . It is natural to expect a bigger decision regret when the misspecification error Reg (˜ π ∗ ) is higher. In particular, when the policy class Π F is well-specified, i.e., π f 0 ∈ Π F , we would have Reg (˜ π ∗ ) = 0 , and the regret upper bound would scale with the function class complexity through a fast rate O (˜ r 2 α +2 α +2 ) . For VC subgraph classes with ˜ r = ˜ O ( √ η/n ) , the rate can range from ˜ O ( √ η/n ) for α = 0 to ˜ O ( η/n ) for α = ∞ . However, if the policy class is misspecified so that Reg (˜ π ∗ ) > 0 , then the dominating term related to ˜ r would be a slow rate O (˜ r ) . This reveals an interesting phase transition between the correctly specified and misspecified settings, which is not discovered in the previous theory that considers only well-specification [14].

Remark 3. The constant coefficients in the regret upper bound can be improved when 2(1 + α ) /α is an integer (which accommodates the case of α = 1 justified in [15]):

Notably, the constant in front of the misspecification error Reg (˜ π ∗ ) becomes 1 instead of 2 , which we believe is tight. This upper bound follows from nearly the same proof as Theorem 1 except that it handles an inequality slightly differently. Specifically, the proof of Theorem 1 involves a transcendental inequality of the form Reg (ˆ π ) ≤ c 1 Reg (ˆ π ) α 2(1+ α ) ˜ r + c 2 for certain positive terms c 1 , c 2 . This inequality is difficult to solve exactly, so we can only get an upper bound on its solution. It turns out that we can get a better upper bound when 2(1 + α ) /α is an integer.

The nuisance estimation rate. We now show that Rate N ( n, δ ) can be effectively controlled for the DM, ISW and DR score functions. In pariticular, Rate N ( n, δ ) is can be bounded by the estimation errors of the nuisance functions f 0 and Σ 0 .

Proposition 2. For any given δ ∈ (0 , 1) , let χ n,δ be a positive sequence converging to 0 as n →∞ , such that the mean square errors of the nuisance estimates satisfy the following with probability at least 1 -δ :

where Proj span( Z ) ( ˆ f ( X ) -f 0 ( X )) is the projection of ˆ f ( X ) -f 0 ( X ) onto span( Z ) . Then,

If we take θ = θ DM, we have Rate N DM ( n, δ ) = O ( χ n,δ ) ;

If we take θ = θ ISW, we have Rate N ISW ( n, δ ) = O ( χ n,δ ) ;

If we take θ = θ DR, we have Rate N DR ( n, δ ) = O ( χ 2 n,δ ) .

Compared to the DM and ISW scores, the impact of the estimation errors of the nuisances in the DR score is of only second order, i.e. , O ( χ 2 n,δ ) instead of O ( χ n,δ ) . This echos the benefit of DR methods

in causal inference and offline contextual bandit learning [1, 5, 6]. Notably, here we only need to bound the projected error on the nuisance estimator ˆ f , which handles the setting when span ( Z ) does not cover the whole R d space, as is the case with the shortest path problem in Section 5.

Computing the critical radius. The critical radius, ˜ r characterizes the complexity of the function classes G 1 and G 2 defined in Eq. (8). The next proposition shows that ˜ r is of order ˜ O (1 / √ n ) if the function classes have finite VC-subgraph dimensions. For simplicity, we focus on G 1 only.

Proposition 3. Suppose G 1 has VC-subgraph dimension η almost surely. Then for any δ ∈ (0 , 1) , there exists a universal positive constant ˜ C such that

satisfies the inequalities R n ( G 1 , ˜ r ) ≤ ˜ r 2 , 3 n ˜ r 2 / 64 ≥ log log 2 (1 / ˜ r ) , and 2 exp ( -3 n ˜ r 2 / 64 ) ≤ δ/ 4 .

3.2 Byproduct: Fast Rates in the Full Feedback Setting with Misspecification

Although our main theorem is stated for the bandit feedback setting, our regret analysis techniques can be easily adapted to the full feedback setting. The following theorem states a similar regret upper bound. To our best knowledge, this is the first result for CLO that shows a margin-dependent fast rate with potential policy misspecification in the full feedback setting.

Theorem 2. Suppose Z ∗ ( X ) defined in Assumption 3 is a singleton almost surely. Define

and ˜ r F be any solution to the inequality R n ( G F , r ) ≤ r 2 satisfying 3 n (˜ r F ) 2 / 64 ≥ log log 2 (1 / ˜ r F ) and 2 exp ( -3 n (˜ r F ) 2 / 64 ) ≤ δ . If Assumption 3 further holds, then, with probability at least 1 -δ ,

The regret bound is similar to that in Theorem 1 for the bandit setting, except that it does not have the nuisance error term 2 Rate N ( n/ 2 , δ/ 4) . This is because, in the full feedback setting, we observe the entire Y vector, so we do not need to estimate any nuisance functions and can consider the nuisance estimation error term Rate N ( n/ 2 , δ/ 4) as zero. When Π F is a well-specified VC-subgraph class with dimension η , we have Reg (˜ π ∗ ) = 0 , and ˜ r F = ˜ O ( √ η/n ) , so the bound in Theorem 2 reduces to O (( η/n ) ( α +1) / ( α +2) + η/n ) . This bound interpolates between O ( n -1 / 2 ) and O ( n -1 ) according to the margin parameter α , recovering the fast rate in the full-feedback setting without misspecification as given in [14]. In contrast, our bound in Theorem 2 additionally quantifies the impact of policy misspecification, and its generalization Theorem 1 further incorporates the impact of nuisance estimation errors in the bandit-feedback setting.

4 Computationally Tractable Surrogate Loss

The IERM objective is generally nonconvex in f ∈ F , making it computationally intractable to optimize. In the full feedback setting, tractable surrogate losses have been proposed [11, 16]. In this section, we briefly explain the SPO+ loss in [11] and how it can be used in the bandit setting.

The full feedback IERM problem in Eq. (3) can be viewed as minimizing the following loss over F :

This IERM loss is equivalent to the 'smart predict-then-optimize' (SPO) loss in Definition 1 of [11]. Letting z ∗ ( y ) ∈ arg min z ∈Z y ⊤ z with the same tie-breaking rule as in π f , [11] propose the SPO+ surrogate loss:

The SPO+ loss has many desirable properties: given any fixed y , it is an upper bound for the IERM loss, it is convex in f ( x ) , and its subgradient at f ( x ) has a closed form 2( z ∗ ( y ) -z ∗ (2 f ( x ) -y )) .

s

t

Y

j

z

j

In the bandit setting, although Y i is not observed, the score θ ( X i , Z i , C i ; ˆ f, ˆ Σ) plays the same role (see Eqs. (3) and (6)). So it is natural to adapt the SPO+ loss to the bandit setting by replacing the unobserved cost vector Y i by the corresponding score θ ( X i , Z i , C i ; ˆ f, ˆ Σ) :

Then we use the plug-in policy π ˆ f SPO + as the decision policy. This is implemented in the experiments in Section 5. We can similarly adapt any surrogate loss for the full-feedback IERM problem to the bandit feedback setting, simply replacing the cost vector Y i 's by the corresponding scores.

5 Numerical Experiments

We now test the performance of our proposed methods in a simulated stochastic shortest path problem following [11, 14]. Specifically, we aim to go from the start node s to the end node t on a 5 × 5 grid consisting of d = 40 edges, where the costs of traveling on the edges are given by the random vector Y ∈ R 40 (see Fig. 1). We consider covariates X ∈ R 3 and a function f 0 ( x ) = E [ Y | X = x ] whose components are cubic polynomials. The corresponding shortest path problem can be easily formulated into a CLO problem with the constraint set Z given by standard flow preservation constraints. The resulting optimal solution z belongs to { 0 , 1 } 40 , indicating whether passing each edge or not. We note that there are m = 70 feasible paths z 1 , . . . , z m ∈ Z from the source node to the target node, and the feasible paths are linearly dependent with a rank of 18 .

We consider a bandit feedback setting, observing only the total traveling costs C of the historical decisions Z generated by a certain logging policy but not the edge-wise costs Y . Weconsider different types of logging policies that generate the decisions in the observed data. In this section, we report results using a random logging policy that picks a path from all feasible ones at random regardless of the covariate value. In Appendix C, we further study the performance of two different covariatedependent logging policies: one that picks paths according to the sign of the first covariate X 1 , and one that depends on the signs of both X 1 and X 2 . The empirical insights from the covariate-dependent logging policies are qualitatively the same as those for the random logging policy.

We numerically evaluate the performance of the ETO approach and the IERM approach 2 . For both approaches, we use the same class F to construct the decision policies. We consider three different classes for F : a correctly specified polynomial class, a misspecified class that omits two high-order terms (termed degree-2 misspecification), and a misspecified class that omits four high-order terms (termed degree-4 misspecification). For the IERM approach, we consider DM and DR here but defer the results of ISW to Appendix C.2 Table 3 for its significantly worse performance. In both DM and DR, the nuisance f 0 is estimated by a bandit-feedback regression given in Eq. (7) with a ridge penalty, and we test out the three aforementioned function classes for the nuisance class F N as well. In DR, the nuisance Σ 0 ( x ) is estimated by the propensity score approach described in Remark 2,

with the propensity scores estimated by either sample frequencies (for the random logging policy) or suitable decision tree models (for covariate-dependent logging policies). We further consider three variants when plugging ˆ Σ into the doubly robust score: pseudo-inverse (DR PI) as in the original θ DR definition; lambda regularization (DR Lambda), where we replace Σ † with (Σ + λI ) -1 for a positive constant λ ; and clipping (DR Clip), where eigenvalues of Σ below a certain threshold are clipped to the threshold before taking the pseudo-inverse. For all IERM variants, we optimize the SPO+ losses as discussed in Section 4. Further details on the experimentation setup and implementation are summarized in Appendix C. Finally, we consider some naive extensions of the offline contextual bandit learning with finite discrete actions (Naive ETO and Naive SPO+), where we view the feasible paths as separate discrete actions, without considering the linear structure of the decision-making problem. See Appendix A for details and Section 1.3 for background on offline bandits.

We test the performance of different methods on an independent testing sample of size 2000 , and evaluate the ratio of their regrets relative to the expected cost of the global optimal policy π f 0 . Table 1 shows the average relative regret ratios of different methods across 50 replications of the experiment for a random logging policy. Due to space limitations, we include results for the training data size n = 400 , 1000 , 1600 , and defer results for other sizes to Appendix C.2. The relative regret of all methods properly decrease with the training data size n . In particular, the SPO+ approximation for the end-to-end IERM approach perform better than the ETO method. Among the SPO+ methods, the DM score achieves the best performance, while the DR score based pesudo-inverse performs the worst. Through a close inspection, we found that the bias adjustment term that involves the pseudo-inverse in the DR score causes a significant variance inflation. In fact, the ISW score also performs badly due to the high variance (see Appendix C.2). The Lambda regularization and Clip techniques can effectively reduce the variance and result in improved decision-making. Moreover, the naive benchmarks that ignore the linear structure of the decision costs have much worse performance. This shows the importance of leveraging the linear problem structure.

In Table 2, we show the performance of different methods when either the policy-inducing model F or the nuisance model F N or both are misspecified. We observe that when the policy-inducing model is misspecified, the end-to-end SPO+ methods perform much better than the ETO method, provided that the nuisance model for SPO+ is correctly specified. This is consistent with the findings in the previous full-feedback CLO literature [e.g., 11, 12, 14], showing the benefit of integrating

estimation and optimization for misspecified policy models. However, the advantage of the endto-end approaches is weakened dramatically once the nuisance model is misspecified. In this case, the evaluation of decision policies is biased, so the end-to-end approaches also target a 'wrong' objective that may not accurately capture the decision quality. Moreover, we observe that when the nuisance model is misspecified, the DR score can somewhat outperform the DM score, because it can debias the misspecified nuisance to some extent. These results demonstrate new challenges with the bandit-feedback CLO: the end-to-end approaches are sensitive to the misspecification of nuisance models, and the DM and DR scores face different bias-and-variance tradeoffs under nuisance model misspecification. Therefore, in practice we may prefer more flexible models for accurate nuisance modeling, while using simpler policy-inducing models for tractable end-to-end optimization.

6 Conclusions

This paper studies the bandit-feedback setting for contextual linear optimization for the first time. We adapt the induced empirical risk minimization approach to this setting, provide a novel theoretical analysis for the regret of the resulting policies, leverage surrogate losses for efficient optimization, and empirically demonstrate the performance of the proposed methods across different model specifications. Our paper has a few limitations that we aim to address in the future. First, we primarily consider parametric induced policy classes, and it would be interesting to accommodate more general nonparametric classes. Second, we focus mainly on the SPO+ surrogate loss, and investigating other surrogate losses in the bandit feedback setting would also be of great interest.

Acknowledgments and Disclosure of Funding

Nathan Kallus acknowledges that this material is based upon work supported by the National Science Foundation under Grant No. 1846210. Xiaojie Mao is supported in part by National Natural Science Foundation of China (grant numbers 72322001, 72201150, and 72293561) and National Key R&D Program of China (grant number 2022ZD0116700).

References

Susan Athey and Stefan Wager. Policy learning with observational data. Econometrica , 89(1): 133-161, 2021.

Jean-Yves Audibert and Alexandre B Tsybakov. Fast learning rates for plug-in classifiers. The Annals of statistics , 35(2):608-633, 2007.

Stéphane Boucheron, Gábor Lugosi, Pascal Massart, et al. Concentration inequalities using the entropy method. The Annals of Probability , 31(3):1583-1614, 2003.

Olivier Bousquet. Concentration inequalities for sub-additive functions using the entropy method. In Stochastic inequalities and applications , pages 213-247. Springer, 2003.

Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal , 21(1):C1-C68, 2018.

Victor Chernozhukov, Mert Demirer, Greg Lewis, and Vasilis Syrgkanis. Semi-parametric efficient policy learning with continuous actions. Advances in Neural Information Processing Systems , 32, 2019.

Miroslav Dudík, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. arXiv preprint arXiv:1103.4601 , 2011.

Miroslav Dudík, Dumitru Erhan, John Langford, and Lihong Li. Doubly robust policy evaluation and optimization. 2014.

Othman El Balghiti, Adam N Elmachtoub, Paul Grigas, and Ambuj Tewari. Generalization bounds in the predict-then-optimize framework. In NeurIPS , pages 14412-14421, 2019.

Charles Elkan. The foundations of cost-sensitive learning. In International joint conference on artificial intelligence , volume 17, pages 973-978. Lawrence Erlbaum Associates Ltd, 2001.

Adam N Elmachtoub and Paul Grigas. Smart 'predict, then optimize'. Management Science , 68(1):9-26, 2022.

Adam N Elmachtoub, Henry Lam, Haofeng Zhang, and Yunfan Zhao. Estimate-then-optimize versus integrated-estimation-optimization: A stochastic dominance perspective. arXiv preprint arXiv:2304.06833 , 2023.

Dylan J Foster and Vasilis Syrgkanis. Orthogonal statistical learning. The Annals of Statistics , 51(3):879-908, 2023.

Yichun Hu, Nathan Kallus, and Xiaojie Mao. Fast rates for contextual linear optimization. Management Science , 68(6):4236-4245, 2022.

Yichun Hu, Nathan Kallus, and Masatoshi Uehara. Fast rates for the regret of offline reinforcement learning. Mathematics of Operations Research , 2024.

Michael Huang and Vishal Gupta. Learning best-in-class policies for the predict-then-optimize framework. arXiv preprint arXiv:2402.03256 , 2024.

Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences . Cambridge university press, 2015.

Nathan Kallus. Recursive partitioning for personalization using observational data. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 1789-1798. JMLR. org, 2017.

Nathan Kallus. Balanced policy evaluation and learning. In Advances in Neural Information Processing Systems , pages 8895-8906, 2018.

Nathan Kallus. More efficient policy learning via optimal retargeting. Journal of the American Statistical Association , 116(534):646-658, 2021.

Nathan Kallus and Masatoshi Uehara. Doubly robust off-policy value and gradient estimation for deterministic policies. Advances in Neural Information Processing Systems , 33:10420-10430, 2020.

Nathan Kallus and Angela Zhou. Policy evaluation and optimization with continuous treatments. In International conference on artificial intelligence and statistics , pages 1243-1251. PMLR, 2018.

Nathan Kallus, Xiaojie Mao, Kaiwen Wang, and Zhengyuan Zhou. Doubly robust distributionally robust off-policy evaluation and learning. In International Conference on Machine Learning , pages 10598-10632. PMLR, 2022.

Toru Kitagawa and Aleksey Tetenov. Who should be treated? empirical welfare maximization methods for treatment choice. Econometrica , 86(2):591-616, 2018.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Kelsey Maass, Arun V Sathanur, Arif Khan, and Robert Rallo. Street-level travel-time estimation via aggregated uber data. In 2020 Proceedings of the SIAM Workshop on Combinatorial Scientific Computing , pages 76-84. SIAM, 2020.

Jayanta Mandi, James Kotary, Senne Berden, Maxime Mulamba, Victor Bucarey, Tias Guns, and Ferdinando Fioretto. Decision-focused learning: Foundations, state of the art, benchmark and future opportunities. arXiv preprint arXiv:2307.13565 , 2023.

David Pollard. Empirical processes: theory and applications. In NSF-CBMS regional conference series in probability and statistics , pages i-86. JSTOR, 1990.

Noveen Sachdeva, Lequn Wang, Dawen Liang, Nathan Kallus, and Julian McAuley. Off-policy evaluation for large action spaces via policy convolution. In Proceedings of the ACM on Web Conference 2024 , page 3576-3585, 2024.

Utsav Sadana, Abhilash Chenreddy, Erick Delage, Alexandre Forel, Emma Frejinger, and Thibaut Vidal. A survey of contextual optimization methods for decision-making under uncertainty. European Journal of Operational Research , 2024.

Bodhisattva Sen. A gentle introduction to empirical process theory and applications. Lecture Notes, Columbia University , 11:28-29, 2018.

Nian Si, Fan Zhang, Zhengyuan Zhou, and Jose Blanchet. Distributionally robust batch contextual bandits. Management Science , 69(10):5772-5793, 2023.

Yeran Sun, Yinming Ren, and Xuan Sun. Uber movement data: A proxy for average one-way commuting times by car. ISPRS International Journal of Geo-Information , 9(3):184, 2020.

Adith Swaminathan and Thorsten Joachims. Batch learning from logged bandit feedback through counterfactual risk minimization. The Journal of Machine Learning Research , 16(1): 1731-1755, 2015.

Adith Swaminathan and Thorsten Joachims. Counterfactual risk minimization: Learning from logged bandit feedback. In International Conference on Machine Learning , pages 814-823. PMLR, 2015.

Adith Swaminathan and Thorsten Joachims. The self-normalized estimator for counterfactual learning. advances in neural information processing systems , 28, 2015.

Michel Talagrand. New concentration inequalities in product spaces. Inventiones mathematicae , 126(3):505-563, 1996.

Alexander B Tsybakov. Optimal aggregation of classifiers in statistical learning. The Annals of Statistics , 32(1):135-166, 2004.

Masatoshi Uehara, Chengchun Shi, and Nathan Kallus. A review of off-policy evaluation in reinforcement learning. arXiv preprint arXiv:2212.06355 , 2022.

Aad W Van Der Vaart and Jon A Wellner. Weak convergence and empirical processes . Springer, 1996.

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge University Press, 2019.

Baqun Zhang, Anastasios A Tsiatis, Marie Davidian, Min Zhang, and Eric Laber. Estimating optimal treatment regimes from a classification perspective. Stat , 1(1):103-114, 2012.

Yingqi Zhao, Donglin Zeng, A John Rush, and Michael R Kosorok. Estimating individualized treatment rules using outcome weighted learning. Journal of the American Statistical Association , 107(499):1106-1118, 2012.

Xin Zhou, Nicole Mayer-Hamblett, Umer Khan, and Michael R Kosorok. Residual weighted learning for estimating individualized treatment rules. Journal of the American Statistical Association , 112(517):169-187, 2017.

Zhengyuan Zhou, Susan Athey, and Stefan Wager. Offline multi-action policy learning: Generalization and optimization. Operations Research , 71(1):148-183, 2023.

A Naive Extensions of Offline Contextual Bandit Learning

In this section, we show some alternative approaches to solve the stochastic shortest path problem in Section 5. These approaches can be viewed as naive extensions of offline contextual bandit learning with discrete actions.

Specifically, consider the feasible paths z 1 , . . . , z m for m = 70 . We can view them as separate discrete actions, and a feasible decision policy π is a mapping from the covariates X to one of the m = 70 feasible paths. We now adopt one-hot-encoding for the decisions. Consider a simplex Z simplex in R m and zero-one vector ˜ z ∈ { 0 , 1 } m that takes the value 1 on one and only one of its coordinate. Each feasible decision in z 1 , . . . , z m corresponds to one vector ˜ z , e.g., the decision z j corresponds to the ˜ z vector whose j -th entry is 1 and other entries are all 0 .

Our decision-making problem restricted to the feasible decisions z 1 , . . . , z m can be written as follows:

where ˜ f 0 ( x ) = ( E [ C | Z = z 1 , X = x ] , . . . , E [ C | Z = z m , X = x ]) ⊤ . For any given x , the resulting decision will be an one-hot vector that corresponds to an optimal decision at x (which can be equivalently given by π f 0 ( x ) for an plug-in policy in Eq. (2) at the true f 0 ( x ) = E [ Y | X = x ] ). In this formulation, we view the decisions z 1 , . . . , z m as separate discrete actions, and we do not necessarily take into account the linear structure of the decision cost.

We can easily adapt the bandit-feedback ETO to this new formulation. Specifically, we can first construct an estimator ˆ ˜ f for the ˜ f 0 function. For offline bandit learning with discrte actions, this would usually be implemented by regressing the observed total cost C with respect to the covariates X , within each subsample for each of the feasible decisions respectively. Given the estimator ˆ ˜ f , we can then solve the optimization problem min ˜ z ∈Z simplex ˜ z ⊤ ˆ ˜ f 0 ( x ) . We finally inspect which coordinate of the resulting solution is equal to 1 and choose it as the decision.

To adapt the IERM approach, we similarly define the plug-in policy for any given hypothesis ˜ f ( x ) : R p → R m for the function ˜ f 0 ( x ) = ( E [ C | Z = z 1 , X = x ] , . . . , E [ C | Z = z m , X = x ]) ⊤ :

where ties are again broken by some fixed rules. Given a function class ˜ F , we can then consider the induced policy class ˜ Π ˜ F = { ˜ π ˜ f : ˜ f ∈ ˜ F} . For any policy ˜ π ∈ ˜ Π ˜ F , its output is an one-hot vector, whose entry with value 1 corresponds to the chosen decision among z 1 , . . . , z m . For any observed decision Z ∈ { z 1 , . . . , z m } , we denote its one-hot transformation ˜ Z as the zero-one vector whose value-one entry corresponds to the value of Z . For a given observed total cost C , we denote ˜ C as the vector all of whose entries are equal to C . In the lemma below, we show that the value of each policy ˜ π can be also identified by some score funtions.

Lemma 1. For any given policy ˜ π that maps any covariate value x to an m -dimensional one-hot vector, its policy value can be written as follows:

and the score function ˜ θ can take three different forms:

(Direct Method) ˜ θ DM ( x, ˜ z, ˜ c ; ˜ f, ˜ e ) = ˜ f ( x ) ;

(Inverse Propensity Weighting) ˜ θ IPW ( x, ˜ z, ˜ c ; ˜ f, ˜ e ) = ˜ z ˜ e ( x ) ˜ c ;

(Doubly Robust) θ DR ( x, ˜ z, ˜ c ; ˜ f, ˜ e ) = ˜ f ( x ) + ˜ z ˜ e ( x ) (˜ c -˜ f ( x )) .

In the three score functions above, all vector operations are entry-wise operations.

From Lemma 1, the new formulations above have very similar structure as our previous formulation in Section 2 Proposition 1. The major differences are that we restrict to the discrete actions z 1 , . . . , z m , redefine certain variables accordingly, and consider a simplex set as the constraint. We note that the identification in Lemma 1 mimics the DM, IPW and DR identification in offline contextual bandit learning [e.g., 1, 7, 43]. Since the identification formulae are analogous to those in Section 2, we can easily apply the same policy learning methods in Section 2 and the SPO+ relaxation in Section 4.

B Omitted Proofs

B.1 Supporting Lemmas

Lemma2 (Talagrand's inequality, [4, 31, 37]) . Let U i , i = 1 , . . . , n be independent U -valued random variables. Let H be a countable family of measurable real-valued functions on U such that ∥ h ∥ ∞ ≤ v and E [ h ( U 1 )] = · · · = E [ h ( U n )] = 0 , for all h ∈ H . Define

Then, for all t ≥ 0 ,

Lemma 3. Fix functions f, Σ independent of { ( X i , Z i , C i ) } n i =1 such that ∥ θ ( x, z, c ; f, Σ) ∥ ≤ Θ for all x, z, c . Define the function class

Let ˜ r be any solution to the inequality R n ( G , r ) ≤ r 2 satisfying 3 n ˜ r 2 / 64 ≥ log log 2 (1 / ˜ r ) . Then we have

Proof of Lemma 3. When sup g ∈G | ( E n -E P ) g | / ( ∥ g ∥ 2 + ˜ r ) > 6˜ r , one of the following two events must hold true:

Note that ∥ g ∥ 2 ≤ r implies E [( g -E P ( g )) 2 ] ≤ r 2 , and we also have ∥ g -E P ( g ) ∥ ∞ ≤ 2 . By Talagrand's inequality (Lemma 2) over the function class { g -E P ( g ) : g ∈ G} ,

We now bound the expectation E [ Z n ( r )] . Since G is star-shaped 3 , by [41, Lemma 13.6], r → R n ( G , r ) /r is non-increasing. Thus, for any r ≥ ˜ r ,

where

Define

where the first inequality comes from a symmetrization argument, the second inequality uses the fact that r →R n ( G , r ) /r is non-increasing, and the third inequality is by definition of ˜ r .

The Talagrand's then implies

We first bound P ( E 1 ) . Taking r = ˜ r and t = 4˜ r 2 in Eq. (10), we get

We now bound P ( E 2 ) . Note that

Define

Since ∥ g ∥ 2 ≤ 1 , there exists M ≤ log 2 (1 / ˜ r ) such that

Therefore,

We now bound each term in the summation above. If there exists g ∈ G m such that Z n ( ∥ g ∥ 2 ) ≥ 6 ∥ g ∥ 2 ˜ r , then we have

Thus,

Now, taking r = 2 m ˜ r and t = 2 m ˜ r 2 in Eq. (10), we get

Therefore, if 3 64 n ˜ r 2 ≥ log log 2 (1 / ˜ r ) , then

Combining the bounds on P ( E 1 ) and P ( E 2 ) leads to the final conclusion.

Lemma 4. Suppose Assumption 3 holds and P ( |Z ∗ ( X ) | > 1) = 0 . Then there exists a constant ˜ C ( α, γ ) such that for any π ∈ Π F ,

̸

Proof of Lemma 4. This follows directly from [14, Lemma 1].

Lemma 5. Let c 1 , c 2 , r be positive constants. For any α > 0 , if a positive number x satisfies

we have

Proof of Lemma 5. First, note that

The derivative is strictly increasing in y and is eventually positive. Note the function y -c 1 y α/ (2+2 α ) r -c 2 takes a negative value at y = 0 . Then as y increases, the value of y -c 1 y α/ (2+2 α ) r -c 2 first decreases and then increases. Therefore, if y > 0 satisfies the inequality y -c 1 y α/ (2+2 α ) r -c 2 ≥ 0 , then such y also provides an upper bound on x . Hence it is sufficient to show that y = ( c 1 r ) 2 α +2 α +2 +2 c 2 satisfies the inequality, or equivalently,

Suppose α/ (2 + 2 α ) is a rational number. In this case, we can write α/ (2 + 2 α ) = m 1 /m 2 , where m 1 and m 2 are positive integers such that m 2 ≥ 2 m 1 +1 . Eq. (11) is then equivalent to

Using the multinomial theorem, we have

Since m 2 ≥ 2 m 1 +1 , we have for any i = 0 , . . . , m 1 ,

This can be proved by showing the ratio of LHS over RHS is larger than 1 . Hence, Eq. (11) holds true when α/ (2 + 2 α ) is a rational number.

Finally, note that we have

Proof of Lemma 6. Let 2( α +1) /α = m , where m is an integer by assumption. Using similar arguments as in the proof of Lemma 5, it is sufficient to show that for w = c 2 α +2 α +2 1 r 2 α +2 α +2 + 2 α +2 α +2 c 1 y α 2 α +2 r +

is continuous in α . Since any real number α can be viewed as the limit of a sequence of rational numbers and Eq. (11) holds for all rational numbers, it also holds true for all α > 0 .

Lemma 6. Let c 1 , c 2 , r, y, z be positive constants, and let α be a positive constant such that 2(1 + α ) /α is an integer. If a positive number x satisfies

This is equivalent to showing

Since α = 2 / ( m -2) , the above inequality is equivalent to

Using the multinomial theorem, it is easy to see that the expansion of LHS contains all terms on the RHS (plus additional positive terms). This finishes proving our conclusion.

B.2 Proof of Main Theorem

Proof of Theorem 1. To simplify notation, we define

We can decompose the regret as

where the inequality follows from the definition of ˆ π .

By Assumption 2, with probability at least 1 -δ/ 2 ,

We now bound ( E P -E n 1 ) [ θ ( X,Z,C ; ˆ f 2 , ˆ Σ 2 ) ⊤ (ˆ π ( X ) -˜ π ∗ ( X )) ] . By Lemma 3 and the assumption that 2 exp ( -3 n ˜ r 2 / 128 ) ≤ δ/ 4 , we have that with probability at least 1 -δ/ 4 ,

Assuming Eq. (13) holds,

For any π ∈ Π F ,

̸

where the last inequality follows from Lemma 4.

Applying the inequality above for both ˆ π and π f 0 and plug the bounds into Eq. (14), we get

We can similarly bound ( E P -E n 2 ) [ θ ( X,Z,C ; ˆ f 1 , ˆ Σ 1 ) ⊤ (ˆ π ( X ) -˜ π ∗ ( X )) ] .

Combining all pieces together, we get that with probability at least 1 -δ ,

Solving the above inequality with respect to Reg (ˆ π ) /B using Lemma 5, we have

B.3 Proofs of Propositions

Proof of Proposition 1.

For direct method, the conclusion is obvious.

For ISW, we have for any Σ ,

Let M ( x ) be a matrix whose columns include all basis vectors of the span of Z . Then π ( X ) ∈ Range ( M ( x )) . According to the coverage assumption, the column space of Σ 0 ( x ) is identical to the column space of M ( x ) . By the property of pseudo-inverse, the column space of ( I -Σ † 0 Σ 0 ) is orthogonal to the column space of M . Therefore, ( I -Σ + ( X )Σ 0 ( X )) ⊤ π ( X ) = 0 for any π ∈ Z .

For doubly robust score, we have that for any function f, Σ ,

Taking either f = f 0 or Σ = Σ 0 gives 0 .

̸

Proof of Proposition 2. Because π ( x ) -˜ π ∗ ∈ span( Z ) , for the DM score we have

For the ISW score, we have

For the doubly robust score, we can easily get

Here the second equation holds because π ( x ) -˜ π ∗ ( x ) belongs to the linear span of Z , but I -Σ + 0 ( x )Σ 0 is orthogonal to the linear span of Z , as we already argued in the proof of Proposition 1. The third equation holds because the column space of Σ 0 ( X ) is identical to span( Z ) according to the coverage assumption.

Proof of Proposition 3. Define

Note that whenever E Ψ( | W | /w ) ≤ 1 for some random variable W , we have by Markov's inequality that

Throughout the proof, we condition on the event that G 1 has VC-subgraph dimension η . We finish the proof in three steps.

Step I: Critical radius for empirical Rademacher complexity. Define the localized empirical Rademacher complexity

where ϵ 1 , . . . , ϵ n are i.i.d. Rademacher random variables, and ∥ g ∥ n = √∑ n i =1 g 2 ( X i , Z i , C i ) /n . Let ˆ r ∗ n be the smallest positive solution to ˆ R n ( G 1 , r ) ≤ r 2 / 32 . In what follows, we show that there exists a universal constant C such that

For any g ∈ G 1 , define set

Let D ( t, G ) be the t -packing number of G and N ( t, G ) be the t -covering number. Note that ∥ g ∥ ≤ √ nr for all g ∈ G . By [28, Theorem 3.5],

So by Eq. (15),

Consider the function class

Note that √ nr is the envelope of G ′ 1 on ( X 1 , Z 1 , C 1 ) , . . . , ( X n , Z n , C n ) . Applying [40, Theorem 2.6.7] gives

for a universal constant ˜ C . Thus,

where ∫ 1 0 √ 2 log C +15 -3 log tdt < ∞ . We then obtain that for a (different) universal constant ˜ C ,

Therefore, for any samples ( X i , Z i , C i ) n i =1 , any ˆ r n ≥ C √ η log( n +1) /n is a valid solution to ˆ R n ( G 1 , r ) ≤ r 2 / 32 , which implies Eq. (16).

Step II: Critical radius for Rademacher complexity. Let r ∗ n be the smallest positive solution to the inequality R n ( G 1 , r ) ≤ r 2 / 32 . We now bound r ∗ n .

For any t > 0 , define the random variable

so that R n ( G 1 , r ) = E P [ W n ( r )] by construction. Define the events

where

Following the proof of [14, Lemma EC.12],

Assuming Eq. (18) holds,

for some universal constant ˜ c 1 > 0 . Moreover, for any ζ ≥ 1 , we have R n ( G 1 , ζr ∗ n ) ≥ R n ( r ∗ n ) ≥ ( r ∗ n ) 2 / 32 . By [3, Theorem 16],

for some universal constant ˜ c 2 > 0 . Combining all pieces we have

By step I in the proof, P ( ˆ r ∗ n ≤ ˜ C 0 √ η log( n +1) /n ) = 1 for some constant ˜ C 0 . Let ˜ C > 5 ˜ C 0 be a constant such that 2 -˜ C (˜ c 1 ∧ ˜ c 2 ) < 1 / 6 . If r ∗ n > C √ η log( n +1) /n , by Eq. (17) we have P ( ˆ r ∗ n > ˜ C 0 √ η log( n +1) /n ) > 0 , which leads to contradiction. Thus,

Finally, any r ≥ ˜ C √ η log( n +1) /n solves the inequality R n ( G 1 , r ) ≤ r 2 .

Step III: Checking other conditions. The other two inequalities, 3 n ˜ r 2 / 64 ≥ log log 2 (1 / ˜ r ) and 2 exp ( -3 n ˜ r 2 / 64 ) ≤ δ/ 2 , are easily satisfied as long as we take ˜ C big enough.

B.4 Proof for Full Feedback Setting

Proof of Theorem 1. To simplify notation, we write

We can decompose the regret as

where the inequality follows from the definition of ˆ π F .

We now bound ( E P -E n ) [ Y ⊤ ( ˆ π F ( X ) -˜ π ∗ ( X ) )] . Following a similar proof as in the proof of Lemma 3, we have that with probability at least 1 -δ ,

For any π ∈ Π F ,

̸

where the last inequality follows from Lemma 4. Thus,

Combining all pieces together, we get that with probability at least 1 -δ ,

Solving the above inequality with respect to Reg (ˆ π F ) /B using Lemma 5, we have

C Additional Experimental Details

In Section 5, we provide experimental results for different methods under various model specifications and different logging policies. In this section, we further explain the details of experiment setup, implementation, and provide additional experimental results. All experiments in the paper are implemented on a cloud computing platform with 128 CPUs of model Intel(R) Xeon(R) Platinum 8369B CPU @ 2.70GHz, 250GB RAM and 500GB storage. The experiment for each specification of F and F N and each logging policy takes around 2 days of running time based on parallel computing on 100 CPUs.

C.1 Experimental Setup and Implementation Details

Data generating process. We first generate i.i.d draws of the covariates X = ( X 1 , X 2 , X 3 ) ⊤ ∈ R 3 from independent standard normal distribution. Then we simulate the full feedback Y according to the equation Y = f 0 ( X ) + ϵ , where f 0 ( X ) = 3 + W ∗ 1 X 1 + W ∗ 2 X 2 + W ∗ 3 X 3 + W ∗ 4 X 1 X 2 + W ∗ 5 X 2 X 3 + W ∗ 6 X 1 X 3 + W ∗ 7 X 1 X 2 X 3 for coefficient vectors W ∗ 1 , . . . , W ∗ 7 ∈ R d and a random noise ϵ drawn from the Unif [ -0 . 5 , 0 . 5] . To fix the coefficient vectors W ∗ 1 , . . . , W ∗ 7 , we draw their entries independently the from the Unif [0 , 1] distribution once, and then use the resulting fixed coefficient vectors throughout the experiment. We sample observed decisions Z from the set of all feasible path decisions { z 1 , . . . , z m } for m = 70 , according to different logging policies that will be described shortly. Then the total cost C = Y ⊤ Z is recorded in the observed data.

We consider three different logging policies. One is a random logging policy that uniformly samples each decision from the feasible decisions, regardless of the covariate value. The other two are covariate-dependent logging policies. For these two policies, we first remove 20 feasible decisions that correspond to the optimal decisions for some covariate observations in the testing data, and then randomly sample the observed decisions from the rest of 50 feasible decisions according to different rules. This means that many promising decisions are not explored by the logging policies at all. We hope to use these logging policies to demonstrate the value of leveraging the linear structure of the

̸

decision-making problem, since exploiting the linear structure allows to extrapolate the feedback from the logged decisions to decisions never explored by the policies. Specifically, we further divide the remaining 50 feasible decisions into two even groups. Then the two different logging policies sample decisions from the two groups according to different rules depending on the covariate value.

One covariate-dependent logging policy samples the decisions according to the sign of the first covariate X 1 . When X 1 > 0 , the logging policy chooses the first group with probability 2 / 3 and the second group with probability 1 / 3 . When X 1 ≤ 0 , the logging policy chooses the first group with probability 1 / 3 and the second group with probability 2 / 3 . Once deciding the group, the policy then uniformly samples one decision from the chosen group. For concreteness, we will refer to this policy as the X 1 -policy.

The other covariate-dependent policy samples the decisions according to the signs of both X 1 and X 2 . When X 1 > 0 and X 2 > 0 , the logging policy chooses the two groups with probabilities 2 / 3 and 1 / 3 respectively. When X 1 > 0 and X 2 ≤ 0 , the logging policy chooses the two groups with probabilities 1 / 3 and 2 / 3 respectively. When X 1 ≤ 0 and X 2 > 0 , the policy chooses the two groups with probabilities 3 / 4 and 1 / 4 respectively. When X 1 ≤ 0 and X 2 ≤ 0 , the policy chooses the two groups with probabilities 1 / 4 and 3 / 4 respectively. Once deciding the group, the policy again uniformly samples one decision from the chosen group. We will refer to this policy as the X 1 X 2 -policy.

Specification of the policy-inducing model and nuisance model. For the policy-inducing model and nuisance model, we consider three different classes. One is the correctly specified class { ( x 1 , x 3 , x 3 ) ↦→ W 0 + W 1 x 1 + W 2 x 2 + W 3 x 3 + W 4 x 1 x 2 + W 5 x 2 x 3 + W 6 x 1 x 3 + W 7 x 1 x 2 x 3 : W 0 , . . . , W 7 ∈ R } . The second model class { ( x 1 , x 3 , x 3 ) ↦→ W 0 + W 1 x 1 + W 2 x 2 + W 3 x 3 + W 4 x 1 x 2 + W 5 x 2 x 3 : W 0 , . . . , W 5 ∈ R } omits two interaction terms and is thus misspecified (which we refer to as degree-2 misspecification). The third model class { ( x 1 , x 3 , x 3 ) ↦→ W 0 + W 1 x 1 + W 2 x 2 + W 3 x 3 : W 1 , W 2 , W 3 ∈ R } omits all four interaction terms (which we refer to as degree-4 misspecification).

Nuisance estimation for the SPO+ methods. The SPO+ methods involve two different nuisances. One is the function f 0 ( x ) = E [ Y | X = x ] . Weestimate this by the least squares regression in Eq. (7), with the F N class being one of the three classes described above (i.e., correct specification, degree-2 misspecification, degree-4 misspecification). We incorporate an additional ridge penalty with a coefficient 1 . We estimate the the nuisance Σ 0 ( x ) using the propensity score approach described in Remark 2, so we need to estimate the propensity scores P ( Z = z j | X = x ) for j = 1 , . . . , m . For the random logging policy, we simply estimate P ( Z = z j | X = x ) for any x by the empirical frequency of the decision z j in the observed data. For the X 1 -policy and X 1 X 2 -policy, we estimate the propensity scores by classification decision trees of depth 3 trained to classify each instance to one of the observed classes among z 1 , . . . , z m . These nuisances are all estimated using the two-fold cross-fitting described in Section 1.2.

Since the Σ 0 matrix is rank-deficient, we cannot directly invert it. Besides taking the pseudo-inverse, we also implement the Lambda regularization and clipping techniques described in Section 5. For Lambda regularization, we set the regularization parameter λ to 1 . For the clipping technique, we consider the eigen-decomposition of Σ 0 , set all eigenvalues below 1 to 1 , and then take a pseudoinverse of the transformed matrix. These two lead to the SPO+ DR Lambda and SPO+ DR Clip methods in Section 5 and this section.

Naive Benchmarks. We also implemented the benchmarks in Appendix A. For ETO, SPO+ DM, and SPO+ DR, we estimate ˜ f 0 ( x ) by regressing C with respect to X using data for each observed decision respectively. The regression function class uses similar correctly specified class, degree-2 misspecified class, degree-4 misspecified class mentioned above, with only slight difference in the dimension since the output of ˜ f 0 is m -dimensional while the output of f 0 is d -dimensional. For SPO+ DR and SPO+ IPW, we need to estimate the propensity scores. These are again estimated by either sample frequency or decision trees. Note that some feasible decisions are never observed in the training data. For these decisions, the corresponding component of ˜ f 0 is heuristically imputed by a pooled regression of C against X using all observed data. For SPO+ IPW and SPO+ DR, although the propensity scores for the unseen decisions are zero, they do not impact the policy evaluation since they only need the propensity score for decisions observed in the data.

SPO+ Tuning and Optimization. We need to solve the SPO+ optimization problem in Section 4 for the DM and DR scores. Following [11], we incorporate an additional ridge penalty on the coefficients of the hypotheses. We select the penalty coefficient from a grid of 0 , 0 . 001 , 0 . 01 and 10 points distributed uniformly on the logarithmic scale over 0 . 1 to 100 . This is done by minimizing the out-of-sample error on an independent validation dataset with size equal to the corresponding training data. The penalty coefficient is finally set as the half of the value chosen by this validation procedure.

To optimize the SPO+ loss, [11] recommend applying standard convex optimization solvers to a dual reformulation, or by running stochastic subgradient descents. We tried both approaches and found that the stochastic subgradient descent method with the default hyperparameters in [11, 14] (number of iterations, step size, batch size) runs much faster, while performing similarly to the dual reformulation method.

C.2 Additional Experimental Results

In this section, we provide some additional experimental results. Specifically, Tables 3 and 4 are extensions of Tables 1 and 2 for the random logging policy. In particular, Table 3 shows additional results on ISW and IPW methods. They are generally worse than other alternative methods. Table 4 additionally show results for the naive benchmarks under model misspecification, and further confirm their worse performance. Tables 5 to 8 provide results for two covariate-dependent logging policies, the X 1 -policy and X 1 X 2 -policy respectively. The performance of different methods tend to remain similar or slightly degrade under these more complicated logging policies. However, the comparisons of different methods remain qualitatively the same.

Table 3: Mean relative regret ratio of different methods when the nuisance model F N and the policyinducing model F are correctly specified. The logging policy is a random policy.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We provide the new methodology in Section 2, main theoretical results in Section 3, and numerical findings in Section 5.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our paper in Section 6.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Our main theoretical results are in Section 3, and we have a proposition in Section 2. All detailed proofs are provided in Appendix B.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide open access to data and code at https://github.com/ CausalML/CLOBandit .

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide open access to data and code at https://github.com/ CausalML/CLOBandit

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide detailed explanations for the experiment setup in Section 5 and Appendix C.2.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We have statistical significance results in our experiments, and all our results are significant. We did not report them in the paper due to limited space, but can provide them upon request.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Computer resources are reported in Appendix C.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have reviewed the NeurIPS Code of Ethics and the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper is foundational research and not tied to particular applications.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Peer reviews:
comments: SUMMARY:
This paper studies Contextual Linear Optimization (CLO) with bandit feedback and presents a class of algorithms termed Induced Empirical Risk Minimization (IERM). The authors also derive the regret upper bounds. The regret analysis accounts for the misspecification of the policy class and incorporates a margin condition that potentially enables a faster regret rate.

STRENGTHS:
1. The extension of the CLO problem to bandit feedback is an important and novel research direction.
2. The mathematical formulations and proofs are rigorous and well-presented.
3. The analysis allows for model misspecification in the induced policy class.

WEAKNESSES:
Although I believe this paper is technically sound, the readability could be improved. The presentation of the problem and model could be clearer, especially for readers unfamiliar with the CLO problem. It would be better to provide a more detailed explanation of the problem setting before delving into mathematical formulas. For example, in the introduction, the authors could explain the interaction protocol between the learner and the environment, and specify which quantities are known or unknown to the learner/environment. Such elaborations would make the paper more accessible.

QUESTIONS:
N/A

LIMITATIONS:
Limitations are adequately discussed in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses the partial bandit feedback setup for contextual linear optimization. The authors propose an algorithm based on Induced Empirical Risk Minimization (IERM), which incorporates doubly-robust estimation to handle the reward model misspecification. They provide a regret bound for the partial bandit feedback setup and, as a byproduct, offer a regret bound for the full bandit feedback setup under model misspecification. This contribution is significant due to the robustness of the theoretical results and the broad applicability of the findings. The paper is well-written and merits acceptance with only minor revisions.\n\nSTRENGTHS:\nThis work accommodates the practicality such as partial feedback and model misspecification. In addition, the theoretical results are solid enough. The numerical experiments demonstrate the theoretical results well.\n\nWEAKNESSES:\nThis setup is similar to partial monitoring. It would be great if the suggested model is compared with partial monitoring. In addition, it is hard to interpret the result to the reviewer, as the complexities of terms are not stated. The reviewer would like to see a summary of what the order of regret with respect to n is. Further, the model misspecification is not fully explained in the first two sections. The authors should explain the model misspecification before starting a theoretical analysis. Lastly, there are some minor errors in the presentation:\n- In line 108, \"is\" should be corrected.\n- There is no definition of $\\chi^2_{n,\\delta}$.\n- In line 74, \"in that that\" should be corrected.\n\nQUESTIONS:\nBased on my understanding, ETO is a family of algorithms that use estimates directly, such as plug-in policies with estimates. In contrast, IERM-type algorithms include an additional optimization step based on these estimates. This step appears to make IERM-type algorithms function similarly to optimism in the face of uncertainty (OFU)-type algorithms. What aspects of IERM-type algorithms make them more effective than ETO-type algorithms? Is this improvement related to their exploration schemes?\n\nLIMITATIONS:\nIt is not well explained why the suggested algorithm has improved performance as compared to the existing algorithms.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims at the following problem. The learner observes side information $x$ and needs to output $z$. The Nature will then generate $Y$ (which depends on $x$, but may be randomised) and the learner will suffer loss $Y'z$. The key feature of the setup is that $Y$ is not directly observable. The paper describes convincing real-life applications fitting this scenario.\n\nOne is after a policy $z=\\pi(x)$ minimising the loss expectation on the basis of the past data. One can obtain it by estimating the conditional expectations $f_0(x) = E(Y\\mid x)$ first. However, this is not necessarily the best approach. In the style of reinforcement learning, the paper aims to estimate the policy value $V(\\pi)$ directly.\n\nThe paper puts forward an algorithm, or, rather an approach for this called IERM (I understand some steps in it involve optimisation, which is very hard to perform). The performance of the approach is studied in Theorem 1. It bounds the regret of the policy found by IERM w.r.t. the optimal policy in a class. The bound involves Rademacher complexity of the class.\n\nThere is substantial empirical evaluation.\n\nSTRENGTHS:\nI believe the approach of the paper is useful and very practical.\n\nWEAKNESSES:\nThe paper is very hard to read for an uninitiated person like me. I found it very hard to distinguish between given data, estimates, datapoints, random variables etc. \n\nPerhaps it would help if the paper were structured in a more traditional way with definitions and algorithms rather than the free-flowing narrative.\n\nPage 3, line 125: I am not sure I parse\n\nwhich we call as nuisance functions as they will be used to\n\ncorrectly. Did you mean\n\nwhich we call nuisance functions because they will be used to\n\n?\n\n(I should say I do not understand the choice of the word \"nuisance\". The pun is lost on me.)\n\nQUESTIONS:\nNone.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}