Risk Bounds and Calibration for a Smart Predict-then-Optimize Method

The predict-then-optimize framework is fundamental in practical stochastic decision-making problems : first predict unknown parameters of an optimization model , then solve the problem using the predicted values . A natural loss function in this setting is defined by measuring the decision error induced by the predicted parameters , which was named the Smart Predict-then-Optimize ( SPO ) loss by Elmachtoub and Grigas [ 2021 ] . Since the SPO loss is typically nonconvex and possibly discontinuous , Elmachtoub and Grigas [ 2021 ] introduced a convex surrogate , called the SPO+ loss , that importantly accounts for the underlying structure of the optimization model . In this paper , we greatly expand upon the consistency results for the SPO+ loss provided by Elmachtoub and Grigas [ 2021 ] . We develop risk bounds and uniform calibration results for the SPO+ loss relative to the SPO loss , which provide a quantitative way to transfer the excess surrogate risk to excess true risk . By combining our risk bounds with generalization bounds , we show that the empirical minimizer of the SPO+ loss achieves low excess true risk with high probability . We first demonstrate these results in the case when the feasible region of the underlying optimization problem is a polyhedron , and then we show that the results can be strengthened substantially when the feasible region is a level set of a strongly convex function . We perform experiments to empirically demonstrate the strength of the SPO+ surrogate , as compared to standard ` 1 and squared ` 2 prediction error losses , on portfolio allocation and cost-sensitive multi-class classification problems . 1 Introduction . The predict-then-optimize framework , where one predicts the unknown parameters of an optimization model and then plugs in the predictions before solving , is prevalent in applications of machine learning . Some typical examples include predicting future asset returns in portfolio allocation problems and predicting the travel time on each edge of a network in navigation problems . In most cases , there are many contextual features available , such as time of day , weather information , financial and business news headlines , and many others , that can be leveraged to predict the unknown parameters and reduce uncertainty in the decision making problem . Ultimately , the goal is to produce a high quality prediction model that leads to a good decisions when implemented , such as a position that leads to a large return or a route that induces a small realized travel time . There has been a fair amount of recent work examining this paradigm and other closely related problems in data-driven decision making , such as the works of Bertsimas and Kallus [ 2020 ] , Donti et al . [ 2017 ] , Elmachtoub and Grigas [ 2021 ] , Kao et al . [ 2009 ] , Estes and Richard [ 2019 ] , Ho and Hanasusanto [ 2019 ] , Notz and Pibernik [ 2019 ] , Kotary et al . [ 2021 ] , the references therein , and others . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this work , we focus on the particular and important case where the optimization problem of interest has a linear objective with a known convex feasible region and where the contextual features are related to the coefficients of the linear objective . This case includes the aforementioned shortest path and portfolio allocation problems . In this context , Elmachtoub and Grigas [ 2021 ] developed the Smart Predict-then-Optimize ( SPO ) loss function , which directly measures the regret of a prediction against the best decision in hindsight ( rather than just prediction error , such as squared error ) . After the introduction of the SPO loss , recent work has studied its statistical properties , including generalization bounds of the SPO loss function in El Balghiti et al . [ 2019 ] and generalization and regret convergence rates in Hu et al . [ 2020 ] . Moreover , since the SPO loss is not continuous nor convex in general [ Elmachtoub and Grigas , 2021 ] , which makes the training of a prediction model computationally intractable , Elmachtoub and Grigas [ 2021 ] introduced a novel convex surrogate loss , referred to as the SPO+ loss . Elmachtoub and Grigas [ 2021 ] highlight and prove several advantages of the SPO+ surrogate loss : ( i ) it still accounts for the downstream optimization problem when evaluating the quality of a prediction model ( unlike prediction losses such as the squared ` 2 loss ) , ( ii ) it has a desirable Fisher consistency property with respect to the SPO loss under mild conditions , and ( iii ) it often performs better than commonly considered prediction losses in experimental results . Unfortunately , although a desirable property of any surrogate loss in this context , Fisher consistency is not directly applicable when one only has available a finite dataset , which is always the case in practice , because it relies on full knowledge of the underlying distribution . Motivated thusly , it is desirable to develop risk bounds that allow one to translate an approximate guarantee on the risk of a surrogate loss function to a corresponding guarantee on the SPO risk . That is , risk bounds ( and the related notion of calibration functions ) answer the question : to what tolerance δ should the surrogate excess risk be reduced to in order to ensure that the excess SPO risk is at most ? Note that , with enough data , it is possible in practice to ensure a ( high probability ) bound on the excess surrogate risk through generalization and optimization guarantees . The main goal of this work is to provide risk bounds for the SPO+ surrogate loss function . Our results , to the best of our knowledge , are the first risk bounds of the SPO+ loss , besides the analysis of the 1-dimensional scenario in Ho-Nguyen and Kılınç-Karzan [ 2020 ] . Our results consider two cases for the structure of the feasible region of the optimization problem : ( i ) the case of a bounded polyhedron , and ( ii ) the case of a level set of a smooth and strongly convex function . In the polyhedral case , we prove that the risk bound of the SPO+ surrogate is O ( 2 ) , where is the desired accuracy for the excess SPO risk . Our results hold under mild distributional assumptions that extend those considered in Elmachtoub and Grigas [ 2021 ] . In the strongly convex level set case , we improve the risk bound of the SPO+ surrogate to O ( ) by utilizing novel properties of such sets that we develop , namely stronger optimality guarantees and continuity properties . As a consequence of our analysis , we can leverage generalization guarantees for the SPO+ loss to obtain the first sample complexity bounds , with respect to the SPO risk , for the SPO+ surrogate under the two cases we consider . In Section 5 , we present computational results that validate our theoretical findings . In particular we present results on entropy constrained portfolio allocation problems which , to the best of our knowledge , is the first computational study of predict-then-optimize problems for a strongly convex feasible region . Our results on portfolio allocation problems demonstrate the effectiveness of the SPO+ surrogate . We also present results for cost-sensitive multi-class classification that illustrate the benefits of faster convergence of the SPO risk in the case of strongly convex sets as compared to polyhedral ones . Starting with binary classification , risk bounds and calibration have been previously studied in other machine learning settings . Pioneer works studying the properties of convex surrogate loss functions for the 0-1 loss include Zhang et al . [ 2004 ] , Bartlett et al . [ 2006 ] , Massart et al . [ 2006 ] and Steinwart [ 2007 ] . Works including Zhang [ 2004 ] , Tewari and Bartlett [ 2007 ] and Osokin et al . [ 2017 ] have studied the consistency and calibration properties of multi-class classification problems , which can be considered as a special case of the predict-then-optimize framework [ El Balghiti et al. , 2019 ] . Most related to the results presented herein is the work of Ho-Nguyen and Kılınç-Karzan [ 2020 ] , who study the uniform calibration properties of the squared ` 2 loss , and the related work of Hu et al . [ 2020 ] , who also develop fast sample complexity results for the SPO loss when using a squared ` 2 surrogate . Notation . Let represent element-wise multiplication between two vectors . For any positive integer m , let [ m ] denote the set { 1 , . . . , m } . Let Ip denote the p by p identity matrix for any positive integer p. For c̄ ∈ Rd and a positive semi-definite matrix Σ ∈ Rd×d , let N ( c̄ , Σ ) denote the normal distribution P ( c ) = e − 1 2 ( c−c̄ ) TΣ−1 ( c−c̄ ) √ ( 2π ) d det ( Σ ) . We will make use of a generic given norm ‖ · ‖ on w ∈ Rd , as well as its dual norm ‖ · ‖∗ which is defined by ‖c‖∗ = maxw : ‖w‖≤1 cTw . For a positive definite matrix A , we define the A-norm by ‖w‖A : = √ wTAw . Also , we denote the diameter of the set S ⊆ Rd by DS : = supw , w′∈S ‖w − w′‖2 . 2 Predict-then-optimize framework and preliminaries . We now formally describe the predict-then-optimize framework , which is widely prevalent in stochastic decision making problems . We assume that the problem of interest has a linear objective , but the cost vector of the objective , c ∈ C ⊆ Rd , is not observed when the decision is made . Instead , we observe a feature vector x ∈ X ⊆ Rp , which provides contextual information associated with c. Let P denote the underlying joint distribution of the pair ( x , c ) . Let w denote the decision variable and assume that we have full knowledge of the feasible region S ⊆ Rd , which is assumed to be non-empty , compact , and convex . When a feature vector x is provided , the goal of the decision maker is to solve the contextual stochastic optimization problem : min w∈S Ec∼P ( ·|x ) [ cTw ] = min w∈S Ec∼P ( ·|x ) [ c ] Tw . ( 1 ) As demonstrated by ( 1 ) , for linear optimization problems the predict-then-optimize framework relies on a prediction of the conditional expectation of the cost vector , namely E [ c|x ] = Ec∼P ( ·|x ) [ c ] . Let ĉ denote a prediction of the conditional expectation , then the next step in the predict-then-optimize setting is to solve the deterministic optimization problem with the cost vector ĉ , namely P ( ĉ ) : min w∈S ĉTw . ( 2 ) Depending on the structure of the feasible region S , the optimization problem P ( · ) can represent linear programming , conic programming , and even ( mixed ) integer programming , for example . In any case , we assume that we can solve P ( · ) to any desired accuracy via either a closed-form solution or a solver . Let w∗ ( · ) : Rd → S denote a particular optimization oracle for problem ( 2 ) , whereby w∗ ( ĉ ) is an optimal solution of P ( ĉ ) . ( We assume that the oracle is deterministic and ties are broken in an arbitrary pre-specified manner . ) In order to obtain a model for predicting cost vectors , namely a cost vector predictor function g : X → Rd , we may leverage machine learning methods to learn the underlying distribution P from observed data { ( x1 , c1 ) , . . . , ( xn , cn ) } , which are assumed to be independent samples from P. Most importantly , following ( 1 ) , we would like to learn the conditional expectation and thus g ( x ) can be thought of as an estimate of E [ c|x ] . We follow a standard recipe for learning a predictor function g where we specify a loss function to measure the quality of predictions relative to the observed realized cost vectors . In particular , for a loss function ` , the value ` ( ĉ , c ) represents the loss or error incurred when the cost vector prediction is ĉ ( i.e. , ĉ = g ( x ) ) and the realized cost vector is c. Let R ` ( g ; P ) : = E ( x , c ) ∼P [ ` ( g ( x ) , c ) ] denote the the risk of given loss function ` and let R∗ ` ( P ) = infg′ R ` ( g′ ; P ) denote the optimal ` -risk over all measurable functions g′ . Also , let R̂n ` ( g ) : = 1 n ∑n i=1 ` ( g ( xi ) , ci ) denote the empirical ` -risk . Most commonly used loss functions are based on directly measuring the prediction error , including the ( squared ) ` 2 and the ` 1 losses . However , these losses do not take the downstream optimization task nor the structure of the feasible region S into consideration . Motivated thusly , one may consider a loss function that directly measures the decision error with respect to the optimization problem ( 2 ) . Elmachtoub and Grigas [ 2021 ] formalize this notion in our context of linear optimization problems with their introduction of the SPO ( Smart Predict-then-Optimize ) loss function , which is defined by ` SPO ( ĉ , c ) : = c Tw∗ ( ĉ ) − cTw∗ ( c ) , where ĉ ∈ Rd is the predicted cost vector and c ∈ C is the realized cost vector . Due to the possible non-convexity and possible discontinuities of the SPO loss , Elmachtoub and Grigas [ 2021 ] also propose a convex surrogate loss function , the SPO+ loss , which is defined as ` SPO+ ( ĉ , c ) : = max w∈S { ( c− 2ĉ ) Tw } + 2ĉTw∗ ( c ) − cTw∗ ( c ) . Importantly , the SPO+ loss still accounts for the downstream optimization problem ( 2 ) and the structure of the feasible region S , in contrast to losses that focus only on prediction error . As discussed by Elmachtoub and Grigas [ 2021 ] , the SPO+ loss can be efficiently optimized via linear/conic optimization reformulations and with ( stochastic ) gradient methods for large datasets . Elmachtoub and Grigas [ 2021 ] provide theoretical and empirical justification for the use of the SPO+ loss function , including a derivation through duality theory , promising experimental results on shortest path and portfolio optimization instances , and the following theorem which provides the Fisher consistency of the SPO+ loss . Theorem 2.1 ( Elmachtoub and Grigas [ 2021 ] , Theorem 1 ) . Suppose that the feasible region S has a non-empty interior . For fixed x ∈ X , suppose that the conditional distribution P ( ·|x ) is continuous on all of Rd , is centrally symmetric around its mean c̄ : = Ec∼P ( ·|x ) [ c ] , and that there is a unique optimal solution of P ( c̄ ) . Then , for all ∆ ∈ Rp it holds that Ec∼P ( ·|x ) [ ` SPO+ ( c̄+ ∆ , c ) − ` SPO+ ( c̄ , c ) ] = Ec∼P ( ·|x ) [ ( c+ 2∆ ) T ( w∗ ( c ) − w∗ ( c+ 2∆ ) ) ] ≥ 0 . Moreover , if ∆ 6= 0 , then Ec∼P ( ·|x ) [ ` SPO+ ( c̄+ ∆ ) − ` SPO+ ( c̄ ) ] > 0 . Notice that Theorem 2.1 holds for arbitrary x ∈ X , i.e. , it employs a nonparametric analysis as is standard in consistency and calibration results , whereby there are no constraints on the predicted cost vector associated with x . Under the conditions of Theorem 2.1 , given any x ∈ X , we know that the conditional expectation c̄ = Ec∼P ( ·|x ) [ c ] is the unique minimizer of the SPO+ risk . Furthermore , since c̄ is also a minimizer of the SPO risk , it holds that the SPO+ loss function is Fisher consistent with respect to the SPO loss function , i.e. , minimizing the SPO+ risk also minimizes the SPO risk . However , in practice , due to the fact that we have available only a finite dataset and not complete knowledge of the distribution P , we can not directly minimize the true SPO+ risk . Instead , by employing the use of optimization and generalization guarantees , we are able to approximately minimize the SPO+ risk . A natural question is then : does a low excess SPO+ risk guarantee a low excess SPO risk ? More formally , we are primarily interested in the following questions : ( i ) for any > 0 , does there exist δ ( ) > 0 such that RSPO+ ( g ; P ) − R∗SPO+ ( P ) < δ ( ) implies that RSPO ( g ; P ) −R∗SPO ( P ) < ? , and ( ii ) what is the largest such value of δ ( ) that guarantees the above ? Excess risk bounds via calibration . The notions of calibration and calibration functions provide a useful set of tools to answer the previous questions . We now review basic concepts concerning calibration when using a generic surrogate loss function ` , although we are primarily interested in the aforementioned SPO+ surrogate . An excess risk bound allows one to transfer the conditional excess ` -risk , E [ ` ( ĉ , c ) |x ] − infc′ E [ ` ( c′ , c ) |x ] , to the conditional excess ` SPO-risk , E [ ` SPO ( ĉ , c ) |x ] − infc′ E [ ` SPO ( c′ , c ) |x ] . Calibration , which we now briefly review , is a central tool in developing excess risk bounds . We adopt the definition of calibration presented by Steinwart [ 2007 ] and Ho-Nguyen and Kılınç-Karzan [ 2020 ] , which is reviewed in Definition 2.1 below . Definition 2.1 . For a given surrogate loss function ` , we say ` is ` SPO-calibrated with respect to P if there exists a function δ ` ( · ) : R+ → R+ such that for all x ∈ X , ĉ ∈ C , and > 0 , it holds that E [ ` ( ĉ , c ) |x ] − inf c′ E [ ` ( c′ , c ) |x ] < δ ` ( ) ⇒ E [ ` SPO ( ĉ , c ) |x ] − inf c′ E [ ` SPO ( c′ , c ) |x ] < . ( 3 ) Additionally , if ( 3 ) holds for all P ∈ P , where P is a class of distributions on X × C , then we say that ` is uniformly calibrated with respect to the class of distributions P . A direct approach to finding a feasible δ ` ( · ) function and checking for uniform calibration is by computing the infimum of the excess surrogate loss subject to a constraint that the excess SPO loss is at least . This idea leads to the definition of the calibration function , which we review in Definition 2.2 below . Definition 2.2 . For a given surrogate loss function ` and true cost vector distribution Pc , the conditional calibration function δ̂ ` ( · ; Pc ) is defined , for > 0 , by δ̂ ` ( ; Pc ) : = inf ĉ∈Rd { E [ ` ( ĉ , c ) ] − inf c′ E [ ` ( c′ , c ) ] : E [ ` SPO ( ĉ , c ) ] − inf c′ E [ ` SPO ( c′ , c ) ] ≥ } . Moreover , given a class of joint distributions P , with a slight abuse of notation , the calibration function δ̂ ` ( · ; P ) is defined , for > 0 , by δ̂ ` ( ; P ) : = inf x∈X , P∈P δ̂ ` ( ; P ( ·|x ) ) . If the calibration function δ̂ ` ( · ; P ) satisfies δ̂ ` ( ; P ) > 0 for all > 0 , then the loss function ` is uniformly ` SPO-calibrated with respect to the class of distributions P . To obtain an excess risk bound , we let δ∗∗ ` denote the biconjugate , the largest convex lower semi-continuous envelope , of δ ` . Jensen ’ s inequality then readily yields δ∗∗ ` ( RSPO ( g , P ) −R∗SPO ( P ) ) ≤ R ` ( g , P ) −R ` ( P ) , which implies that the excess surrogate risk R ` ( g , P ) −R ` ( P ) of a predictor g can be translated into an upper bound of the excess SPO risk RSPO ( g , P ) −R∗SPO ( P ) . For example , the uniform calibration of the least squares ( squared ` 2 ) loss , namely ` LS ( ĉ , c ) = ‖ĉ − c‖22 , was examined by Ho-Nguyen and Kılınç-Karzan [ 2020 ] . They proved that the calibration function is δ ` LS ( ) = 2/D2S , which implies an upper bound of the excess SPO risk by RSPO ( g , P ) −R∗SPO ( P ) ≤ DS ( RLS ( g , P ) −R∗LS ( P ) ) 1/2 . In this paper , we derive the calibration function of the SPO+ loss and thus reveal the quantitative relationship between the excess SPO risk and the excess surrogate SPO+ risk under different circumstances .

Summary: This paper derives uniform calibration bounds for the SPO+ loss relative to the SPO loss, when the feasible region is polyhedral or a level set of strongly convex functions. This enables us to translate excessive risk bound in terms of SPO+ loss, ie, the convex surrogate loss,  into bounds for the SPO loss. Notably, the authors show that the class of symmetric distributions considered in Elmachtoub and Grigas is not restrictive enough to derive calibration bounds, and instead stronger restrictions must be incorporated.