Diagnostic Tool for Out-of-Sample Model Evaluation
Authors: Ludvig Hult
Dave Zachariah
Peter Stoica
Venue: Accepted by TMLR ()

Abstract: Assessment of model fitness is a key part of machine learning. The standard paradigm of model evaluation is analysis of the average loss over future data. This is often explicit in model fitting, where we select models that minimize the average loss over training data as a surrogate, but comes with limited theoretical guarantees. In this paper, we consider the problem of characterizing a batch of out-of-sample losses of a model using a calibration data set. We provide finite-sample limits on the out-of-sample losses that are statistically valid under quite general conditions and propose a diagonistic tool that is simple to compute and interpret. Several numerical experiments are presented to show how the proposed method quantifies the impact of distribution shifts, aids the analysis of regression, and enables model selection as well as hyperparameter tuning.

Published in Transactions on Machine Learning Research (10/2023)
Diagnostic Tool for Out-of-Sample Model Evaluation
Ludvig Hult Department of Information Technology
budvig-haite Oult su.se
Uppsala Universily
Dave Zachariah
dave. wawe.zachariahe wu.se
Deportment Information Technology
Uppsala University
Petre Stoice
Psoit
Department of Information Technology
Uppsala University
Reviewed on OpenReview: /ttt::://pperevetie net/
forum? id- ULf302G9DC
Abstract
Assessment of model fitness is key part of machine learning. The standard paradigm of model evaluation is analysis of the average loss over future data. This often explicit in model fitting, where we select models that minimize the average loss over training data as a surrogate, but comes with limited theoretical guarantees. In this paper, we consider the problem of characterizing batch of out-of-sampll losses of model using calibration data set. We provide finite-sample limits on the out-of-sample losses that are statistically valid under quite general conditions and propose diagonistic tool that is simple to compule and interpret. Several numerical experiments are presented to show how the proposed method quantifies the impact of distribution shifts, aids the analysis of regression, and enables model selection as well as hyperparameter tuning.
1 Introduction
Fitting a model to data is a central task machine learning, signal processing, statistics and other areas (Bishop 2006 Hastie et al. 2009 Söderström Stoica 2001 Kay 1993 Fitzmaurice al. 2011]. A fitted model can be assessed by considering loss function {(-) that evaluates the model on future data points. This is called out-of-samppl analysis, since it considers data points beyond those in the training data sample.
Classical statistical models are often assessed by verifying assumption validity through statistical or graphical tools, a process called model diagnostics, diagnostic checks, or regression diagnostics (Ruppert et al. 20033 Belsley et al. 1980). The term has also been established a wider sense for general checks to verify a model fitness for given data, see e.g. Casper et al. 2022); Zhang al. (022). We propose a diagnostic tool that helps evaluating the model performance on out-of-sample data.
In this paper, consider the problem of characterizing m out-of-sample losses of model f using calibra- tion data set. The choice of m will depend the application. In online learning, 1 is reasonable since the model may only be valid for a single prediction. In problems with low data-volumes, such as predictive h-aath model to be used on e.g. 30 future patients with a rare desease, m = 30 may be appropriate. For models in high volume applications, such as e-commerce, the limit m + oo may be relevant. We derive finitc-sample limits on the out-of-sample losses that are statistically valid under quite general conditions and for any of the choices of m. An illustration is provided in Fig. | where we fit model for predicting house prices to training data and evaluate its absolute prediction error calibration data set D. The average

---

Published in Transactions on Machine Learning Research (10/2023)
100%
LAL-curve (B= 0.50)
80%
LAL-curve (B 0.75)
Expected loss from CV
60%
40%
20%
0%
$0
$20K
$40K
$60K
$80K
$100K
Figure 1: Model error diagonistic. Consider predictive model f(X) of house price Y in an area described by a feature vector X. How will this model perform in m futuure price predictions? We evaluate the model using the prediction error ((X,Y) Y f(X) the chosen loss function. The pross-validation method (CV) evaluates the model by estimating the expected out-of-sample loss, which in this case is circa $36K (indicated by black vertical line). Being an estimate of an expectation, it provides limited information about what individual loss values might be observed. Suppose the model will be used in m = 100 prediction instances, and we want to bound at least B = 50% of these future losses. Then the LAL-curve infers that with probability at least 80% (level a 20%), this fraction prediction errors will not exceed $25K (see blue solid curve) For a stronger guarantees, e.g. bounding least 75% of the future losses, the LAL-curve indicates that prediction errors of $48K must be tolerated at level 20% (scc orange dashed curve). Full details for this experiment are provided in Sec.
loss on D is form of 'cross-validation and is indicated in the figure. The figure also illustrates the proposed diagnostic statistic: an upper bound f (D) on the 3-fraction of m yet unobserved prediction errors that holds with confidence level 1 - a. By computing this statistical limit (D( as function of a, we quantify how probable different out-of-sample losses are for the model f. Since this quantification size of D, the limit can be employed as diagnostic tool also in cases where calibration data is scarce costly to obtain. Moreover, the validity does not depend the distribution of the data used to train f; the limit also be used to analyze the severity of distribution shifts, we will illustrate in the numerical experiments below.
valid for any
The rest of the paper is organized as follows. We first formalize the general problem of interest, then propose a mcasure (D) which we refer to as the level-a limit (LAL) on the i-fraction of m out-of-sample losses and prove its statistical guarantees. This is followed by a series of numerical experiments that demonstrate the utility of LAL. In the closing discussion, the proposed method is related existing literature.
2 Problem Statement
Let f denote a model fitted to data samples Do drawn from a distribution po. We aim to quantify the performance of f on m out-of-sample data points [Z,...., Zm drawn from distribution p, which may differ from Po- The performance quantified using any real-valued loss function f(z) the user wants. We let upper case letters denote random variables, e.g., Z, and let the lower case version, 2, represent their realization.
Example 1 (Density Estimation using a Gaussian model). A data point vector E R and we consider Gaussian density model f(z) W f. E), with fitted mean jt and covariance matric E. A common loss function is the negative logelikeli
{(x) 21n (2) 7 + 1n
ignoring the constant and scaling by factor 2.
Example 2 (Regression). A data point pair of features x and label y, i.c., 2 2= (x. v). The model f(x) is any estimate of the conditional expectation function Elylar). Example models include Gaussian process
2

---

Published in Transactions on Machine Learning Research (10/2023)
regressors, random forest regressors, and neural networks. A common loss function is the squared-error loss [(r, v) = [(x)
We assumee that have access to colibration data set D .Z...... Zn} with samples drawn randomly from p, such that the combined data set of n-m samples is exchangeable, i.c., the joint density of the random vector (Zi, (Zi....; Zn .i,..,, 2m) is invariant under relabeling of the data points. While the calibration data set {Zi....,, ]n} is available to use in computations, the out-of-sample data {ZI...., Zm3 is future, yet unseen data. Exchangeability includes the common independent and identically distributed (iid) data assumption as special case. An example of exchangeable non-iid data is sampling without replacement from a finite population.
The problem we consider is to characterize unknown out-of-sample losses
(.7))....,,(2...
of the model f. Specifically, we want to bound fraction 3 E (0, 1) of the losses with confidence That is, we want to find a statistical limit la (D) how large future losses are likely to observe for the model:
P at least a fraction 8 of losses f(z)) respects ((Z.) < e (D) 2 a.
(2)
We will call such value (D) the level-a limit for the 3-fraction of m losses, or a LAL for short.
We call the graph of (D) versus a LAL-curve, as illustrated in, e.g., Fig. By plotting the value (D) on the horizontal axis, we can visualize the tail behaviour of the out-of-sample losses for f a transparent manner. Thus given a valid LAL, we propose use the LAL-curve diagnostic tool for model evaluation.
3
Method
For notational convenience, let L; {(Z,) and arrange the m out-of-sample losses in increasing order: La) L(2 .... S. L(m)- The criterion can be expressed compactly as
P [1m01]
F2 (D) < a
(3)
and our objective to find a limit (D) for any given 3 E (0, 1] and a E (0,
For the calibration data D, let L; = {(ff) and define the of losses (Lig...; L and order statistics L(y L
Lnn. We also define the special cases Lo) and (( to be the infimum and supremum of the support of the distribution of {L possibly toc.
3.1 General LAL expression
The following result holds generically; it assumes two or more continuously distributed random variables don't lake exactly the same value. The probability of such an event zero. To simplify the reading, we will skip the assertions of 'almost surely' and 'assuming no ties..
Theorem 1. Let (D) = L(kr) where
k* min {k fl..... 1} a(k)) S a).
(n-j+m [m3]] (i+ im (m
a(k)
(4)
j=k
m
This valid LAL, salisfying 13
Proof. Define the set of calibration losses L (Lf...., ... L and out-of-sample losses = (L1,..., Lm We will first prove the result. in the case of continuous random variables, and then for discrete random variables.
3

---

Published in Transactions on Machine Learning Research (10/2023)
Consider the case when L U C has continuous joint distribution, so there are no ties. We need to show that
IP ((mm))) > Linus] aa
(5)
Because of the decomposition into sum
n
PP ((mm)) Lixis} P Linn) S (((me)) Σ j=k
P
L S ((ma)) < LigtDl.
(6)
closed form expression for IP (ij S L(i (1itt)) may lead us proving 55
The set of random variables -UL-= = (Li...., Ln Li...., .iil...,,nnyy Lmn can be sorted. Denote these sorted variables J(1) J(2) <... < (n+ Every such value J()) come from an origin set, which is either L or L. This assignment to the origin set partitions the set of -aluus into two subsets. There are ("*t))) such partitions, and they equally probable, due to the assumed exchangeability.
We next investigate how many of the partitions fulfil Lig) S (i) < Liftr for 1 i m, 1 i3jS S n. In this situation, L()) = J(+t5)- Of the i + j losses less than J(++)) j has L as the origin set. There are thus ((+j²)) equally probable partitions of the lesser loss values. Similarly, the loss values greater than L(e) be partitioned in (" ("=)tz=") ways. This shows that
n- (n-Stm-) (sti-1) n-j
P (To) S L0s < 18gt1]
(7)
n+ m
By combining with 6 we have found
n jt-j (n-Stmm-mmpp
[m (3+ [mmf]] 11
P ((mmo)) > [i00)
Σ
al (k)
j=k
Any such that dk) $ a would provide us with a valid LAL. However, since the range of d(()) is (,-nn
the case when a < ma define alk) to
means no such & exists. Defining (*) = 0 for all and r < allows us to finally
m- mm]) (++ [mm]] (mm
a(k) = Σ j=b
12
n-j
m
increasing its domain of definition to {1...., n + 1} and its range to 1]. By selecting k = min(k e
[I....,n+ + (H(k) S a), we have proven the theorem for continuous variables.
Next, we prove the result for discrete random variables. Let F be the joint edf for L'UL, where each loss takes values in finite or countable set V {wi of real numbers. Since may gel ties with non-zero probability, the preceding analysis fails. To cireunvent this problem, construct random vector (Ai,..., An+m) with edf F" such that
(4 2 F(4,, ..l..... (ntm) always
F F' bmm) F(I...,,,4 (n+m) if (1,,..,, km+m) E vu+m
Define further a set of random variables 7, = min fo E Vlu 2 Ai) for all i. Now (Ai,..., An+m) is equal to (Li... Ln Li,..,, Lm) in distribution. Also, (A() > A + (M() > (()) for all i,j. Together, this means P [[(Tmm)) Lie-) IP [T([@m]+n) > T(o->] < P [e(om]+n) > >(r-)] = a(k*). where we have used the result on continuous random variables on to compute A*.
Remark The definition of the LAL, 2 only demands that the limit level is at least 1 1- a. However, the more conservative (D) is, the larger the encess coverage. From the proof, we see that if the joint set of losses (Lf Lú, Li,...,Inp has no ties, one may compute the exact coverage 1 1-a(k") Thus when the method is conservative, it is transporently
4

---

Published in Transactions on Machine Learning Research (10/2023)
Remark 2. The proof technique above is inspired by Fligner & Wolfe 1976) which proves the result for the special case of üd dala. It. is noteworthy that when generalizing from idd to exchangeable dala, we keep the same level of precision.
3.2 LAL for single out-of-sample data point
When = the LAL takes very simple closed form. By deriving it from basic principles rather than using Thm. we also obtain a lightness guarantee.
Theorem 2. For single out-of out-of-sample data point (m=1), LAL can be constructed (a Likyy
where K- = [(n 1)(1-a)) For continuous data distributions, the almost sure out-of-sample loss guarantee
is
-
1 SP a
(8)
For discrele data distribulions, only the upper bound in S con be guaranteed.
Proof. When (Lg,..., , L I1) are continuous, the values are almost surely unique, and therefore there are (*t))= = 1 n+I equally likely ways select which one L1. Only one such selection obeys Li) S L1 S L(tt)) Therefore,
A*
P[L9()) - 4 < ( = 1+1 n
and P[L, S L(ke)
1+
Because a1 [(n + 1)(1-)))((( S (1 - a) 1/( + 1) compute
< 1PL > L( aa
1 n
When (Lf,..., "f.....,,,, LG L1) are discrete and ties are possible, we can still prove the upper bound. Consider rank of R of L1, i... put {Li.... L. L1} in nondecreasing order, and let R denote the position of L1. When there are ties, position the tied values in a uniformly random way. By construction R is uniformly distributed over I..... +n++1) 10+11 1) and
k*
P[L S La--) P(R 2 k-
2
Remark For iid dala and m = 1, the LAL-curve approaches the complementary odf of L1, bee, F. This facilitates the interpretation of the LAL-curue as a quantile point estimate. To see this, let F-1 denote the empirical quantile function of L1 based on the losses Li...., L. Then the LAL of Thm. 2 can equivalently
be defined as
V (1 a))
if = a) E (0,1)
e (D)
(9)
((n+1)
else
If L has bounded and connected range, F-11 converges uniformly to F-1 (Bogoya al. 2016), and (D) F-1 (1 -a). Plotting the vertical aris against F-1(1 a) is identical to plotting the complementary
odf 1 1--Fi F(f) on the vertical azis against l, in this case, the LAL-curve converyes to the graph of the complementary edf.
3.3 LAL for large out-of-sample data sets
If the number of out-of-sample data points is very large, may limit argument and let - in Thm. M
Corollary 1. Let BIN- B) denole the quantile function of binomial distribution with paromelers (n, 3). For an infinite sequence of exchangeable losses (LS ... L; Li.... ......,, ), let (D( Lik-) with
k' -= BIN (11 o; m, 3)-
5

---

Published in Transactions on Machine Learning Research (10/2023)
This LAL satisfies limit form of 3
lim P (mm))) > (D)) S a
(10)
m-000
Proof. The result follows by expanding the binomial coefficients with Stirling's formula and taking the limit for m + co. See App. A for details.
]
Starting from Cor. 1 we can find a different interpretation of the LAL when m7 - 00.
Remark 4. Consider the case of iid data, where L and Lj have a common cdf F for all i,j. Let F-1 be the quantile function, and Fim be the empirical quantile function based on ; L.
As m + oc, we have that [((m)))= = Fm + FB) Therefore, (TPp simplifies to (F) (2) S a, which gives another interpretation of the LAL as the boundary of a confidence interval C. = (-oc, (D)). This interval salisfies IP (3) & C (C9) S a and tleus a valid confidence interval for the B-quantile of L, for all i.
This intuition is also useful in the non-iid case as m co. The De Finetti theorem (Kingman, 1978 states that if (Li.... ... Li L1, L2...) forms an infinite sequence of ezchangeable random variables, there is auziliary random variable Ç such that all the conditional variables (LīlÇ) and (Ljls) are iid with edf Fę. Therefore, the interpretation of the LAL as the boundary of a confidence interval is useful in the gechangeable-butnool data setting, even if its interpretation must be handled with more caution.
4 Experiments
This section presents examples of how the LAL can be applied to analyzing the out-of-sampll loss of a model or family of models. Code to reproduce all experiments can be found at httt::://gttuub..co/ll----
The experiments are intended to be illustrative rather than exhaustive. They are limited in two respects. Firstly, to reduce runtime and simplify reproducibility of the experiments, we consider models that have relatively few parameters and are trained using small data sets. Since the theoretical guarantees hold for any fitted model, they also hold for large models and models trained on large data sets. Secondly, the experiments use small to moderate calibration set sizes n. As suggested in Remark 3 increasing n improves the inference of loss distribution, but there practical limitation to be considered: When n large, is advisable to transfer some data to the training set to improve the model performance, rather than improving our inferences about the model performance.
4.1 Study of asymptotics
This experiment illustrates the limit for different corresponding to Thm. 1 Cor. and Thm. 2
The data set consist of California housing prices from the 1990 census (Kelley Pace & Barry 1997|. covering 20 640 housing blocks.
Each data point z = (z, v) represents a single city block. The label y E ER is the median house value in the block, and the feature vector æ E IR consists of continuous variables such as block coordinates, median house ages and tenant median income. The training data set D, has no 15 000 sampled without replacement. The calibration data set D has 150 and is sampled without replacement from the remaining data.
The model is regression for the logarithm of the median house value, operating on standardized features and labels, and uses a random Fourier feature basis (Rahimi & Recht 2008). Let K random Fourier functions with bandwidth b be stacked in a vector d. The model is f(z) explo(x)T0) 0)), where is found by L2- regularized least squares regression. The hyperparameters (number of basis functions K, bandwidth b and regularization strength A) are tuned by five-fold cross-validation on the training data. The loss function is the absolute error expressed in dollars
(xx ly -((x))
6

---

Published in Transactions on Machine Learning Research (10/2023)
100%
B= 1.00
0
B 0.20
80%
==0.40
0
== 0.60
60%
B 0080
B 0.80
40%
20%
0%
50
$25K
$50K
$75K
$100K
50
$25K
$50K
$75K
$100K
50
$25K
$50K
$75K
$100K
(g(D)
Figure/ 2: LAL-curves. From left right, m computed with Thm. m 30 computed with Thm and
computed with Cor. 1 The curves assure that fraction B of out-of-sample losses will not exceed (D), with at least probability 1 a. For example, we see that among the m = 30 next samples, at least 80% of them will have prediction losses less than $70, 000, with confidence of 90%.
We now turn to limiting out-of-sample losses, where E {1,30, oo). LAL-curves were drawn with varying 6-fractions, shown in Fig. [ and Fig. 2 The empirical risk, defined as the average loss on the calibration data, is also calculated, and the calibration losses are presented as a histogram.
Averaging the losses over the calibration data gives an unbiased point estimate of the expected out-of-sample loss for f but it, by itself, lacks statistical guarantees. While estimates the mean out-of-sample prediction error to be around $35 000, the LAL-curve for single new prediction error (m 1) in Fig. shows that it may be close to $80 000 (for = 10%).
If now consider batch of m 30 predictions, the LAI-curve for m 30 in Fig. informs us that 3 = 80% of them will have errors smaller than $70 000 (for a z 10%). The number of data points blocks not analyzed are 5 490, so we may also consider the limiting case m + co. The LAL curve now tells us that
3 = 80% prediction errors will be smaller than $60 000 (for a = 10%).
By comparing the LAL-curves in Fig. we learn how the out-of-sample batch size m affects the tail behavior For m 30, the decay is faster.
of the LAL fixed B. The LAL-curve is slowly decaying for For the decay is even more abrupt. Some intuition can be gained for iid data. As m increases. the variance of [((m)]) decreases (it is asymptotically normal and /m-consistent, see e.g. (Vart 1998.
Cor. 21.5)), and the bound in (3] can be made tighter.
4.2 Distribution shift analysis
This experiment illustrates the analysis of distribution shifts (Qunonero-Caneeee Candela al.) 2009), also known as concept drift, using the LAI-curve. The experiment also verifies Thm. numerically. The availability of calibration data D invites the notion of refitting or fine-tuming, to adapt to potential distribution shift (Lee) let al. 2023). This is not always possible, c.g. when calibration data is limited (too small n). It may still desirable evaluate the model performance on out-of-distribution data. Since the LAL is valid for any n, it will always be an available diagnostic tool to analyze the extent to which distribution shift affects the model performance.
Lu et al. 2018)) proposes taxonomy for distribution shift detection methods. In this taxonomy, the LAL would qualify as an error rate-bosed method, it is based on a loss function. Example methods in this group are the Drift Detection Method (DDM) (Gama et al. 2004) and Black Box Shift Estimation (BBSE) (Lipton et al. 2018) which detects distribution shift by monitoring when classifier performance changes with statistical significance. In contrast, the LAL does focus on distributions of out-of-sample losses, enabling analysis of how/shift severity for distribution shifts (again using the Lu al. (2014) taxonomy). Other methods for how, foeus on statistical distances between feature distributions (Rabanser et al. 2019). While

---

Published in Transactions on Machine Learning Research (10/2023)
8
100%
f(x)
80%
722
4
2
0
20%
-2
-0.5
0.0
0.5
1.0
1.5
2.0
0
3
12
(a) Fitted model and calibration data
(b) LAL-Curves, 1, =1
Figure 3: Quadratic regression model f(x) evaluated using absolute prediction error loss function ((z, y) ly [())]- (a) A fitted model and two different calibration data sets Dj iid P1 (identical to training data
distribution) and D2 sid out-of-sample loss exceeds the limit (D) by probability at most a, as given by the curve. (See also Thm. 2).
~ (shifted distribution).
(b) LAL-curves under distributions P1 and P2- A single
such methods do not need labels, they do not measure shifts in the way that matters for the model, i.c., the loss.
Let Z: = (X,, Yi), with real valued X; and Y,, and generate data according to
X, N (M, x²)
(11)
YI = N (1(s 1) + 1),
(12)
The training data set Do was created with nig = 100, = 0 0.5. quadratic regression model was used, since approximates the conditional expectation function well over the training data; f(x) 0u + 0 ozr2 fitted to Do via least-squares. See Fig. 3a We analyze the case of out-of-sample batch size 1. The performance of the model is evaluated using the absolute error loss:
((x,y) = f())
(13)
The out-of-sample losses are quantified for two different data distributions. In the first case, / = 0.5. there is no distribution shift, and we use a calibration data set D1 for which 30. In the second case, calibration data set D: also having
μ = 0.75, 0.75, resulting in shift in X and LAL-curves in both cases are presented in Fig. 3b which reveal significantly larger out-of-sample losses for the distribution P2 from which D2 was drawn. The 5%-tail losses are nearly twice as large for the shifted distribution.
30. The
Under the same experimental setup, we verified the tightness result of Thm. Using 2 000 Monte Carlo runs, the empirical coverage was computed for different a, and plotted in Fig. 4a Morcover, Rem. 3 states
the convergence of the LAL to the quantile function F-1((1 - a) n increases. This is illustrated in Fig. 4b The quantile function was numerically approximated using 10 samples. The expected LAL is computed using 2 000 Monte Carlo runs with data sampled from P2. We see that the LAL approaches the quantile function as increases.
4.3 Classification error analysis
This experiment shows that the proposed methodology can be applied to classification models as well. We will also consider how adversarial distribution shifts manifest. in the LAL-curve. We use the Palmer Penguin data set, popularized by Horst et al. (2020). The 333 complete record data points are pairs & = (x,, y) of features x; E IR7 and labels y; E 2,3 The labels y; are categorical, encoding the penguin spocies. The features 2; are vectors of (Island, Bill Length, Bill Depth, Flipper Length, Body Mass, Sex, Year), mixture of categorical and integer-valued features.
8

---

Published in Transactions on Machine Learning Research (10/2023)
100%
1.8
30
1 [g(
80%
Expected Ratio
300
1.6
+ n = 3000
60%
1.4
40%
1.2
20%
1.0
0%
20%
40%
60%
80%
100%
0%
20%
40%
60%
80%
100%
(a) Empirical coverage
(b) Convergence of the quantile function
Figure 4: The finite-sample guarantee of Thm. is verified in (a) which illustrates both the upper bound (solid line) and lower bound (dashed line). (b) Ilsstraaton of the connection between LAL and the quantile function (Rem. 3. The expected value of the ratio (D) - a) evaluated for different values of n with data drawn from the shifted distribution P2. As increases, the expected ratio approaches 1, from above.
We use a training data set Do of = 150, leaving 183 samples for calibration. Using Do, we fit f(x) via multinomial logistic regression using L2-regularization and cross-validation. The model output is three- dimensional vector f(x) = (z), fax),, f=() approximating the conditional probabilities, so that fx( approximates PIY x| = xr]. The model to be evaluated on calibration data using the misclassification probability loss
=1 ((..))=1-f((3
(14)
The out-of-sample batch size set to m 1.
Two different calibration data sets D and D2 of sample size 50 constructed. We sampled from the 183 held out data points without replacement. For D,, the initial probability of selecting a sample was uniform over the data. For D2, the initial probability of selecting a sample was proportional to l(z, v), effectively an adversarial roweighting of samples. The LAL-curves of Thm. 2 (sct m = and / arbitrary) are presented Fig. 5 showing that LAL-curves can be applied to classification models. The fact that one curve comes from an adversarial calibration sample is manifest by larger LAL for every confidence a compared the non-adversarial calibration sample.
4.4 Regression error analysis
This experiment shows how alternative loss functions can be used to analyze the asymmetryy errors in regression problems. We use the UCT Airfoil data set (Dua & Graff 2017). The task is to predict a label y R representing the sound measured in dB. The features R Each data point is feature-label pair 2, (z).
The calibration data 'D is constructed by weighted sampling of n 100 samples. The probability to draw a data point (I,, 31) is proportional to exp([1 0 0 -1] x;), making data points with high frequency and small displacement more likely to sample, similar to the distribution shift experiments in Tibshirani el al. 22000 sec. 2.2). The remaining 1 403 data points constitute the training data set Do.
The model fz) uses spline basis that is fit using least-squares with I2-reuularization and volssvalii.tion. To study the asymmetry of prediction errors, compute the LAI-curves for subsequent experiment

---

Published in Transactions on Machine Learning Research (10/2023)
100%
20%
B overshoot
80%
t2(2
15%
lg undershoot
60%
fg abs
10%
40%
5%
20%
0%
0%
0%
20%
40%
60%
80%
100%
7.5
10.0
12.5
15.0
17.5
20.0
1
Figure 5: LAL-Curves (m = 1) for the classification error analysis. The loss function indicates the cer- tainty of misclassification the model assigned 80% probability to the wrong la- bels. Data set D1, is exchangeable with the training data. Data set D2 is adversarially sampled, present- ing notably larger LAL.
Figure 6: LAI-curves (m = 1) for the regression er- ror analysis. Using different loss functions, all units of dB, gives deeper insight in the model fit. Looking at around 2 8%, using loss for under- and overshoot, to overshoot out-of-sample data than it is to under- shoot.
loss of 80% means that
that the model is more likely
(m = 1) using three different loss functions,
overshoot loss
((r, max y)
(15)
undershoot loss f(x,y) max(0, f(x))
(16)
absolute loss f(x,y) ly f(x)
(17)
which are all in units of dB to enable physical interpretation. The results are shown in Fig. 6 where
observe that f produces more severe overshoots than undershoots. The chosen loss functions 15 17 differs
from the loss function used for model fitting (squared-error), illustrating the freedom that an analyst has to characterize the performance of
4.5 Model comparison
This experiment is concerned with model selection. The data used is the monthly number of earthquakes worldwide with magnitude 2 5 between 2012 and 2022 (USGS 2022 There are 120 data points 2120: with 2, E {0, 2..... These are randomly split into 100 and 20 data points, forming Do and D. We learn two models of the number of earthquakes per month 2, P(Z = 2), using the maxinum likelihood method: a Poisson model pPolsson (z) and Negative Binomial model Neggin The models evaluated using the negative log- ligliiel loss
((z) -olg (()
(18)
and Fig. 7 presents their respective LAL-curves for subscquent carthquake, i.c., m 1. From this result we can couclude that the simpler one-parameter Poisson model produces much larger out-of-sample losses than the two- poopaarmeter Negative Binomial model.
We compare the LAL analysis with a common model selection metric, the Akaike Information Criterion (AIC) Ding et al.) 2018). It evaluates the models as PoCPoisson = 1 926.34 and Neg = 1 015.63. One may also compare models by the average loss on the calibration data, yielding 5.69/4.78 nats for the Poisson/NeggBi Both these model selection metrics favor the NegBin model, agrcement with the LAL-curve. Whereas the model selection metries report a single number, the LAL-curve presents more nuanced picture, showing the reduction in tail losses.
10

---

Published in Transactions on Machine Learning Research (10/2023)
B
100%
Poisson
90%
80%
NegBin
to oo (D;A)
95%
99%
60%
40%
20%
0%
10 -6
10-5
10
10
10-2
0.0
2.5
5.0
7.5 10.0
12.5
15.0
(D(
Figure 8: Relation between a regularization param- eter in neural network model (*)x,, and the cor- responding LAL for m - oc, varying B, and (1 (1-a) a) 99% confidence. Appropriate regularization im- proves the performance in the tail of the loss dis- tribution (8 99%), with optimum around x ~ 10-3. The bulk of the losses (B 95%) are not reduced by regularization; their LAI. increase with A.
Comparison of models for earthquake statistics, using LAL curves and loss funetion used is the negalive log-likelihood of data under the fitted model. By considering how it handles the points, we see that the Negative Binomial model provides a better fit. than the Poisson model.
Figure 7:
8 = The
20% most diffieult-to-fit data
4.6 Hyper-parameter tuning
This experiment shows how the LAL-curve can be used for tuning a hyperparameter when training neural network.
The data set is the MNIST handwritten digits (Bottou et al. 1994h. The training data set Do has no 6- 6-10 data points, and the calibration data set 'D has n = 10 data points. The features 2; are images of handwritten digits, and the labels are integers 0-9. Data points are tuples = (x,,
We construct a family of models (x) = NN(x; 0*). The function NN(x; 07) consists a dense feed- forward neural network with 3 hidden layers 250 units cach, paramcters 6*, ReLU activations and softmax output. The parameter 0 is learned by minimizing the cross entropy, using the Adam optimizer with an I2-regularization parameter (aka. 'weight decay" in the deep learning literature). The optimization run for 100 epochs, employing batch size of 1024 and learning rate 0.01. The output of the network (*x is a 10-dimensional vector, where the ith component approximates Ply i|z].
We evaluate the model using the nogative log-likclihood loss
((c, y) = log f2) (a) = - log [N ()),
(19)
Since deep learning methods may be deployed without retraining over large number of predictions, consider the case m - oo (Cor. 1 and study the LAL for 3-fraction of out-of-sample losses. The results in Fig. [ show that small regularization initially reduces losses for outliers without increasing loss on nominal samples.
5
Discussion
This section elaborates on connections related fields.
5.1
Model evaluation and selection techniques
Cross-validatory out-of-sample expected loss estimation arguably the common approach to model
evaluation in machine learning (Arlot & Celisse 2010 Stone
1974
Bishop 2006:
Hastie
al. 2009). The
11

---

Published in Transactions on Machine Learning Research (10/2023)
idea is that model performance is measured by expected loss on out-of-sample data (called the risk), and cross-validation estimates this quantity. In its most common form, fraction of the training data is held out from model fitting. The average loss of the model is computed on the held out data, forming an estimate of the risk. Repeated splitting and refitting of the model (k-fold cross-validation) can be used to estimate the bias and variance of such estimates. Model evaluation with LAL shifts focus to evaluating model performance on probabalistic bounds on out-of-sample losses. This is significant when the out-of-sample loss distribution is multimodal, skewed or otherwise not well described by its mean and variance alone.
Evaluating models with respect to the risk may be done without vrossvalidattin. Statistical learning theories, such as the VC-theory (Vapnik) 1991), J provide asymptotic bounds on the risk under certain assumptions on the models and the data. Similarly, M-estimation (Vaart, 1998) provides another asymptotically valid method of fitting parametric models, quantifying the convergence of the average loss on training data to the risk. If the losses are bounded and the samples are iid or constructed as sampling without replacement, non-parametric results for inferring the risk appear to be promising (Waudby- Smith & Ramdas 2020) improving on both the Hoeffding inequality and the Empirical Bernstein bounds (Audibert et al.) 2009;
Maurer & Pontil 2009). For unbounded losses and non-iid exchangeable data, the inferential problem of estimating the risk remain open. LAL-curves provide a nonparemctric and nonasymptotic way to evaluate model performance by not using the risk as the quantity of comparison.
In this work, we have computed ((, at given confidence levels a. Conversely, one can interpret (D) as the boundary value for rejection of the null hypothesis that model predictions are exchangeable, at level a. Others have used hypothesis testing for model evaluation. Posterior predictive checks (Gelman et al. 2013; Rubin 1984), and the data consistency criterion, (Lindholm et al.] 2019) rely on exchangeability between observed data and data generated by the model to this end. Those methods are used to test whether a model is compatible with data, and reports a p-value for the test. A LAL-based analysis acknowledges that models are always misspecified and instead quantifies how well a model performs.
An analyst may wish to not only diagnose and compare models, but also decide which model in a set of candidates to use. This called model selection (Ding et al. 2018). Two principal goals motivate the model selection algorithm: 1) model selection for inference, or 2) model selection for prediction. The first goal has been used to motivate well known information criteria such as the AIC (Stoica & Sclen 2004). Information critera has also been combined with hypothesis testing to assess if a cerlain model or model class can be said to be significantly closer to the true one than other candidates (Vuong. 1989). Regarding the second goal, Ding et al. (2018) posits that the 'best' model is the one with smallest expected out-of-sample loss, which may be estimated by cross-validatton. By fixing a at any desired level, the LAL can be used to rank models in fashion similar to cross-validatory risk estimation. However, by employing the full LAI-curve as indicated in Sec. 4.5 a better informed decision can be taken by balancing typical versus outlier performance.
The Value-at-Risk (VaR) is a risk measurement used in the finance industry. For a random variable L describing finanical loss, the value-at-risk at level 1 - a can be defined as [Jorion] 2007 Eq 5.1)
VaR = inf (x: IP[L > x] S {}
Note that x is not a random variable in the above expression. Comparing with (3) and fixing m 1, we find that the VaR is the smallest LAL, conditional on the calibration data D. Any VaR constitute valid LAL for m= 1, but the converse is not true. Many VaR estimation methods rely on parametric assumptions or are only asymptotically valid. (Christoffersen) 2009 Alemany et al. 2013) The relaxed definition of L.AL in contrast to VaR enables the distribution-free methodology presented in this article. For its use in e.g. financial risk measurements, VaR has been further developed into several variants such as Conditional VaR, Tilted VaR, Entropic VaR, and more. (Rockafellar & Uryasev 2002: Li et al., 2021) This line of reasearch may be beneficial for the LAL as a diagnostic tool, and constitute direction for further research, but some of the requirements on financial risk metrics may not necessarily carry over to all model performance metrics.
5.2 Non- non-parametric statistics and conformal prediction
For case of exposition, the LAL has primarily been discussed as a point value (D). One can also consider it the boundary point of the interval (-oc, (D)) As such, it can be understood as variant of non-
12

---

Published in Transactions on Machine Learning Research (10/2023)
parametric tolerance interval (Thm. 1. prediction interval (Thm. 2 a confidence interval (Rem. 4 See for instance Vardeman (1922 for comparison between the different statistical intervals.
Fligner & Wolfe (1976) show how to construct non-asymptotic, non-parametric prediction intervals for quantiles of future data, and is a methodological precursor for the results this paper.
Considering LAL a confidence interval of quantile for iid data, there are other results with exact coverage (Zieliáski & Zieliñski 2005), whereas the formula presented in this article sometimes conservative. One could use that method with LAL to get exact guarantee. Such intervals constructed via extra random- ization and become harder to interpret. The LAI-curves in specific have no exact counterpart. We have therefore chosen to avoid this construction.
The theory nonparametric prediction intervals also forms a foundation for conformal prediction. This field focuses on producing prediction sets for the output of any predictive model f, see e-g- Vovk et al. (2005) or Angelopoulos & Bates (2021) for introductions to the field. We will clarify the connection between conformal prediction and the LAL, in the case of split-conformal inference. The general case is essentially identical.
M++1 taking values in W. We wish to produce prediction Consider set of exchangeable random vectors {Wi set C. ((W))M) so that
P[WM+1 C. ((W)) 2
To this end, define a real-valued nonconformilty score A W; + A(W.) - Ad, with the semantic that a large value means W, does not conform to the general data set. Since the score is real valued, one can employ similar principles as in Thm. 2 define a prediction interval Aa such that
P[AM+1 Aa ((A)) 2 -a By letting Ca be the inverse image of A. ({A+)) under A, i.e,
C. ((W.)) = [w[A(w) E Aa ((A.)
ensure the desired coverage. Conformal prediction methodology is thus largely centered on finding suitable nonconformity scores A that are computationally tractable and handle various inference targets and data distributions (Angelopoulos & Bates 2021
More recently Lei et al. 2018 Thm. 2.1), an upper bound on the coverage rate was derived,
1 1-at
2 PW E C. ((W))) 2 -
M +11
that holds whenever the non-conformity scores are almost surely unique. The line of reasoning is similar but not identical to Thm. 2
6 Conclusion
We have proposed the level-a loss (LAL.) curve as a diagonistic tool for out-of-sample analysis of model f. The method requires specifying loss function of interest and the access to calibration data set. In return it provides finite-sample guarantees about the probability of a batch of out-of-sample losses exceeding certain threshold. The LAL is simple to compute and easy to interpret. A series of numerical experiments have been presented to show its uscfulness in regression error analysis, distribution shift analysis, model selection and hyper-parameter luning. We anticipate that there are many other areas of applications for this methodology.
References
Ramon Alemany, Catalina Bolancé, and Montserrat Guillén. A nonparametric approach to calculating value- at-risk. Insurance: Mathematics and Economics, 52(():255-266 262, 2013. ISSN 0167-6687. doi: https://doi. org/ agg/10..116//ismatteen 201...2..20 http://ppwww...eeecccaacecceo//r 10166668713000048
13

---

Published in Transactions on Machine Learning Research (10/2023)
Anastasios N. Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainly quantification, 2021. URL
Sylvain Arlot and Alain Celisse, A survey of cross-validation
procedures for model se- doi: 10.1214/09-S5054.
lection. Statistics Surveys, 4, Jan 2010. ISSN 1935-7516. URL
A
Jean-Yves Audibert, Rémi Munos, and Csaba Szepesvári. Exploration-exxplitatii tradeoff using variance estimates in multi-armed bandits. Theoretical Computer Science, 410(19):1876 1902, April 2009. ISSN 03043975. doi: 10. tt..//..ts..009.0000... S0304997759900667X
URL https://1inkinghubb wwtpe//er...///eeeeeeeeeeee//eee
David A. Belsley, Edwin Kuh, and Roy E. Welsch. Regression Diagnostics: Identifying Influential Data and Sources of Collinearity- Wiley Series in Probability and Statistics. Wiley, 1 edition, Jun 1980. ISBN
978-0-471-05550-4. doi: 10. 10.10///4777755553. URI. 1002/0471725153
cww/coo hoom/diibbok/100/ /book/10..
Christopher M. Bishop.
Pattern recognition and machine learning. Springer, New York, 2006. ISBN 978-0-387-31073--.
Information science and statistics.
J. M. Bogoya, A. Böttcher, and E. A. Maximenko. From convergence in distribution to uniform convergence. Boletín de la Sociedad Matemática Mericana, 22(2):695-710, October 2016. ISSN 1405-213X, 2296-4495. doi: 10.1007 000000//5550000--005555
URL http:///inkk http///iiinkssiinnerr....
springer cww wttp://0..0000//000000000----000
Léon Bottou, Corinna Cortes, John S. Denker, Harris Drucker, Isabelle Guyon, Lawrence D. Jackel, Yann Le Cun, Urs A. Muller, Eduard Säckinger, Patrice Simard, and Vladimir Vapnik. Comparison of classifier methods: a case study in handwritten digit recognition. In Proceedings of the 12th IAPR International Conference on Pattern Recogrition, Conference B: Computer Vision Image Processing, volume 2, pp. 77-82, Jerusalem, October 1994. IEEE. URL
Stephen Casper, Kaivalya Hariharan, and Dylan Hadfield-Menell Diagnostics for deep neural networks with automated ppste attacks. In NeurIPS ML Safety Workshop, 2022. URL https://openreevlev..
Peter Christoffersen. Value at risk models. In Handbook of fenancial lime series, pp. 753 766. Springer, 2009.
Jie Ding, Vahid Tarokh, and Yuhong Yang. d techniques: An overview. IEEE Signal Processing Magazine, (566):66334 2018. doi: 141100/MMP.2018.2220 2867638. URL https://doi. 119///..011/.111 2018.2867633
1109/MSP,
Dheeru Dua and Casey Graff. UCT Machine Learning Repository, 2017. URL http://arccive.ice. uci. edu/ml
Garrett M. Fitzmaurice, Nan M. Laird, and James H. Ware. Applied longitudinal analysis. Wiley series in probability and statistics. Wiley, Hoboken, N.J, 2nd ed edition, 2011. ISBN 978-0-470-33027-7.
Michael A. Fligner and Douglas A. Wolfe. Some Applications of Sample Analogues to the Probability Integral Transformation and Coverage Property. The American Statistician, 30(2):78, May 1976. ISSN 00031305. doi: 00.2397/2683799. URL btetps://wwx.jestt
João Gama, Pedro Medas, Gladys Castillo, and Pedro Rodrigues. Learning with drift detection. In Ana C. Bazzan and Sofiane Labidi (eds.), Advances in Artificial Intelligence SBIA 2004, volume 3171 of Lecture Notes Computer Science, pp. 286-295. Springer Berlin Heidelberg, Berlin, Heidelberg. 2004. ISBN 778-3-540-23237-7 doi: 10.1007/978-3 5-17788-554--264555.1 URL http://link springer. www.l/ngrr.ommo../////o//oo
1007/978- 3- 5-8----6---5--555--- 29
14

---

Published in Transactions on Machine Learning Research (10/2023)
Andrew Gelman, John B Carlin, Hal S Stern, David B Dunson, Aki Vehtari, and Donald B Rubin. Bayesion Data Analysis, Third Edition. CRC Press, Hoboken, 2013. ISBN 978-1-4398-9820--8
Trevor Hastie, Robert Tibshirani, and Jerome H. Friedman. The elements of statistical learning: data mäning, inference, and prediction. Springer series in statistics. Springer, New York, NY, 2nd edition, 2009. ISBN 978-0-387-84857--
Allison Marie Horst, Alison Presmanes Hill, and Kristen B Gorman. palmerpenguins: Palmer Archipelago (Antarctica) penguin data, 2020. URL https: bottps:/aaliisonooree githubgittbb...... iatpaui.ooppllaann wo/palmerreenguins///////////
Philippe Jorion. Value at risk: the new benchmark for managing financial risk. McGraw-Hill, New York, 3rd ed edition, 2007. ISBN 978-0-07-113692-3.
Steven M. Kay. Fundamentals of statisticol signal processing. Prentice Hall signal processing series. Prentice- Hall PTR, Englewood Cliffs, N.J, 1993. ISBN 978-0-13-345711-7 978-0-13-504135- 178-0-13-280803-33
R. Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics & Probability Letters, 33(3): May 1997. ISSN 01677152. doi: 10.1016 1111006//81117771529999990000000 elsevier. ewsevier.com/reeeeeeeeeeeeeeeeee http://wwwo//eeeeeeeeee////////0
URL httpps://linkinghubb
J. F. C. Kingman. Uses of exchangeability. The Annals of Probability, 6(2), Apr 1978. ISSN 0091-1798. doi: 10.1214/20p/11766991 ottp///rrr//pppoooooooooo////oor volume- of httpe///...cccaaaaatt/t/a/////// ottp://uaaaaaaaoaoooooooooooo/o/ probability/
full
Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. Surgical fine-tuning improves adaptation to distribution shifts. In The Eleventh International Conference on Learning Representations, 2023. URL https:// wwwp:rrwwwww...eeeeeeeeeeeee aot.
Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry Wasserman. Distribution- Predictive Inference for Regression. Journal of the American Statisticol Association, 113((23):1094 1111, July 2018. ISSN 0162 1459, 1537-274X. doi: 2017 1307116
URL https:
www..tandfonliae httpee...e..////////00000////000
Tian Li, Ahmad Beirami, Maziar Sanjabi, and Virginia Smith. Tilted empirical risk minimization, Mar 2021. arXiv:2007.0116; [cs, math, stat).
URL
Andreas Lindholm, Dave Zachariah, Petre Stoica, and Thomas B. Schon. Data Consistency Approach to Model Validation. IEEE Access, 7:59788 59796, 2019. ISSN 2169-3536. doi: 10.1109/ACCESS.200 2915109. URL https:://eeeeppli
ieee /
Zachary Lipton, Yu- Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In Jennifer Dy and Andreas Krause (cds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 3122 3130. PMLR, 10-15 Jul 2018. URL https://proceeddngg: mlr htt.:/erreee/////pttoo1.... hhmll
Jie Lu, Anjin Liu, Fan Dong, Feng Gu, Joao Gama, and Guangquan Zhang. Learning under Concept Drift: A Review. IEEE Transactions on Knowledge and Dala Engineering, pp. 1 2018. ISSN 1041-4347, 1558- 2191, 2326-3865. doi: 10.1109/TTDEB 2018 URL http://arxiv org org/abs//004.055 2004.05785.
arXiv:
Ning Lu, Guangquan Zhang, and Jie Lu. Concept drift detection via competence models. doi: https://1inninghubt elsevier. /wtp////rrraeee/////000000000000
Artifi- URL
ciol Intelligence, 209:11-28 28, April 2014. ISSN 00043702,
Andreas Maurer and Massimiliano Pontil. Empirical Bernstein Bounds and Samplo Penaliza- tion. In COLT 2009 Proceedings, Montreal, Quebee, Canada, 2009. URL htttps://ww..s..ccgt
-col1:200//paper//012.2
15

---

Published in Transactions on Machine Learning Research (10/2023)
Joaquin Quiñooero- Candela,, Masashi Sugyiama, Anton Schwaighofer, and Neil D. Lawrence (eds.). Dataset shift in machine learning. Neural information processing series. MIT Press, Cambridge, Mass, 2009. ISBN 978-0-262-17005-5.
Stephan Rabanser, Stephan Günnemann, and Zachary Lipton. Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift. In Н. Wallach, Н. Larochelle, A. Beygelzimer, F. d' Alché- Buc, E. Fox, and R. Garnett (ods.), Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL https://prroceedings neurips. httiipss.c//pape/21////// (1662600/15eb885ffaa667777
Paper. paaper..ff..
Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. In D. Koller, D. Schuurmans, Y. Bengio, and L.. Bottou (eds.), Advances in Newral Information Processing Systems, volume 21. Curran Associates, Inc., 2008 URL https:// .pd Herbert Robbins. A remark on stirling's formula. The American Mathematical Monthly, 62(1):26-29) 1955.
proceedings wwwpeedddngg..euriss.....
ISSN 00029890, 19300972. URL http://wwwz jtr.rjtttorrr... wttp/staaaee//0000111
R.Tyrrell Rockafellar and Stanislav Uryasev. Conditional value-at-risk for general loss distribu- tions. Journal of Banking & Finance, 26(7):1443 1471, 2002. ISSN 0378-4266. doi: htlps://doi. org/
URL
50378426602002716
Donald B. Rubin. Bayesianly Justifiable and Relevant Frequency Calculations for the Applied Statistician.
The Annals of Statistics, 12(4):1151 1172, December 1984, ISSN 0090-5364. doi: 10.
URL sttp:://projecteuclidd otg.:d/..ggu/uccleeeee/ aot.
David Ruppert, M. P. Wand, and R. J. Carroll. Semiparametric Regression. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2003. ISBN 978-0-511-75545-3,
P. Stoica and Y. Selen. Model-order selection: review of information criterion rules. IEEE Signal Processing Magazine, (1(4):36-4) 47,, Jul 2004. ISSN 1558-0792. doi: 101100/MMSP20044.13
M. Stone. Cross Validatory Choice and Assessment of Statistical Predictions. Journal of the Royal Statis- lical Sociely: Series B (Methotologicaa), 36(2):111 133, January 1974. ISSN 00359246. doi: 10.1111/ 199517-6661.1994:46000999 1974. tb00994.9
URL
Torsten Söderström and Petre Stoica. System identification. Prentice Hall international series in systems and control engineering. Prentice-Hall, New York, NY, reprint edition, 2001. ISBN 978-0-13-881236-2.
Ryan J. Tibshirani, Rina Foygel Barber, Emmanuel J. Candès, and Aaditya Ramdas. Conformal Prediction Under Covariate Shift. vrXiv:1904.000 [stat], July 2020. URL http:p//aarriv.rrr//////0.....00
USGS.
Earthquake Catalog, 2022. URL http:///////e/eaooooo//////////r
time.
A. W. van der Vaart. Asymptotic Statistics. Cambridge University Press, edition, October 1998. doi: 10. 10.107/CB0098805180222 18022266. URL 977051180256/tyye/bboo
V. Vapnik. Principles of Risk Minimization for Learning Theory. In J. Moody, S. Han- son, and R. P. Lippmann (eds.), Advances in Neural Information Processing Systems, vol- ume 4. Morgan-Kaufmann, 1991. URL https://proceedingss httpggg.....r.rssepppeee///eee/e
Frfdgfaaaaafffffftteeeeaaeeeeeee pdf
Stephen B. Vardeman. What about the other intervals? The American Statistician, 46(3):193, Aug 1992. ISSN 00031305. doi: 10.2307/2685212. URL http://wwwoooooootttatt/////////
jstor.
crossref
16

---

Published in Transactions on Machine Learning Research (10/2023)
Vladimir Vovk, A. Gammerman, and Glenn Shafer. Algorithmic learning in random world. Springer, New York, 2005. URL
Quang Vuong. Likelihood ratio tests for model selection and non-nested hypotheses. Econometrioa, (7()): 307, Mar 1989. ISSN 00129682. doi: 10.2307/1912550 URL httpp:://ww. www.:jjttorrreoeoe... org/staabee/111557 origin=crossref
Tan Waudby-Smith and Aaditya Ramdas. Estimating mcans of bounded random variables by betting, 2020. URL httpsp///axxx...r//////000....
Yuhui Zhang, Jeff Z HaoChen, Shil-Cheng Huang, Kuan-Chieh Wang, James Zou, and Serena Yeung. Drml: Diagnosing and rectifying vision models using language. In NewrIPS ML Safety Workshop, 2022. URL
Ryszard Zieliíski and Wojciech Zieliáski. Best exact nonparamelric confidence intervals for quantiles. Statis- tics, 39(1):67-71, 2005. URL http://p..//./..oo////////////22
A Details for Proof of Corollary 1
We must first show that
(n-stm-fmo)) [mot] (5+ ( (st(m37-1)
lim m=+00
=())
11
Bd (1 - m3
By definition of binomial coefficient, this is
(n inj++m (m31)..i + mm - 1)!m!!!
lim m-
(n - jn(m) (mm31)11((( 1)((n a + m)!
Rearrangement of factors gives
(n - j + (mB3))(( [mp] 1)1m!
lim 100
(m - [m ) m33 1).(m + m)!
So must now show that
(n - [m !Cj + [m0] - 1)1m!
lim m=+00
80 33
(m
[m/3 () m33 1)((n i m)!
(m)m
By the upper and lower bounds in Stirling's 's formula (Robbins) 1955).
11
2mn
e T2
< nl S / 12n
e Tem
17

---

Published in Transactions on Machine Learning Research (10/2023)
Let 8 = [m,]] m3. We introduce
-=)
n- 11-3 + mm11 - -5)
h(m, Q) = (2 - j + m(1 - B) -6))1/2
6
exp
12(n - + m(1 - 3) 8) + +Q
j + m3 + 54m8+8-1
x (2m(j mB 1))1
e
exp
12(j m3 + 8 1) + +Q
x (2mm)1/2
m
1
exp
12m + Q
m(1-9)-8
-1
(2m(ma(1 - -B) -3)
m(1 3)
exp
12(m 3) - 8) Q)
mB-146
-1
x (27(mB - 1 -1)
mB
1+0
exp
12(m3 - 1 1) +
x (2T(n++ + (m)) 1/22
e
p+m
-1
exp
ni + m
12(n + m) + (1-
allowing us to state succinetly
h(m, (x.
< (1 < h(m,0)
The proof is complete if we can show that limm-400 h(m, Q) 3 (1 3)" To see this we rearrange the factors. We will also use little-oh notation, i.c. f(m) (lg(m)) iff lim (m 0. When taking limits, use that 0 < 8 < for all m.
7' (n - m(1 B) 8)(i m3 + 5 1)m
1/2
h(m,Q) Q)
23 (m(1 - 3) - 6) - 6) + m)
=1+0(1)
x e (1-j+m(1 - 8) ++j + 1+ mm11 m11-8))+ ms+1 -m)
11 j + m(1 3) - m(1-0)-8
j + mB + 8
mB+8-1
771
X
m(1 - 33 -
m3 5 -
m
-exp(n--)++) +0)-((1)
=exp(s)+o(1)
-exp(-n)+o01)
(n - m(1 B) 8) m) (j 3m + 8 11
(n + (m)m) m)-
=1=-)(1-8B) (101111-))-+mmmmm
=m +0(mm)²)))
=m- =m(=+o(m=")
exp
[12(m j + m(1 B) -)) + 1)-1)-11- -
m3 - (1 Q)
- 3) 5) -1- =1+0(1)
x exp
(1- Q1
=1+o(1)
By calculus of little-oh notation we find
h(m, Q) exp(n j) exp(i) exp(- mm m -30 (1- 33 + o())
which in turn means
lim h(m, Q) (1 -3 m=0000
We have thus shown that the limit of is
11 Σ
lim alk)
88 -y
m0000
j=b
18

---

Published in Transactions on Machine Learning Research (10/2023)
By recognizing the binomial cumulative distribution function (and denoting it with the symbol BIN(:; n. we see
lim a(k) = 1 - -BIN 1,n,
Finally
k* = min {N E (1, , lim a(k) S aa)
BIN- 0; n,n,
B Extended Distribution Shift experiment
Continuing on the experiment distribution shift, from Sec. 4.2 We keep 30, and also fix the random
seed across runs to make the comparison clear. We generate calibration data sets D for three scenarios:
Scenario 1, Increasing mean
The mean f shift from 0.5 (no shift) to (large shift), and the standard deviation 0.5 (no shift).
Scenario 2, Incrcasnig variance
The mean = 0.5 (no shift), and the standard deviation shifts from o = 0.5 (no shift) to 2 (large shift)
Scenario 3, Decreasing variance
The mean 0.5 (no shift), and the standard deviation shifts from a = 0.5 (no shift) to 0.05 (large shift)
These three series of calibration-sets have their LAL-curves plotted; see Fig. 9
The results of Scenario 1 show that the mean shift increase, model performance generally deteriorate. One detail should be noted for moderate mean-shifts. Some of the outlier data points with < 0.5 contributed to large LAL when there is no shift, but with increasing shift, these data points are moved towards 0.5, where model performance is generally better. This effect means that the LAL decreases somewhat, can be obscrved for a x 10%. When the mcan-shift increases, outlicrs with x > 0.5 produce larger losses, which is observed through the LAL-curve moving to the right. large mean-shift moves most of the calibration set to x where model performance worse, as indicated by the whole LAI-curve shifting the right, not only for selected a.
The results of Scenario 2 show that the tail of the LAL-curve increase with feature variance, indicating that model performance outliers get worse. Inliers are not much affected.
The results of Scenario 3 show that the LAI-curve moves to the left with decreasing feature variance, indi- cating smaller losses. This means that the data concentrates in regions of feature space where the model performs well. We do observe some distribution shift, but this model still performs well on the data.
Taken together, this indicate that the LAL can indicate distribution shift, that only detects shifts relevant for model performance, and that it indicates whether the shift incurs model degradation generally, only on outliers.
19

---

Published in Transactions on Machine Learning Research (10/2023)
100%
100%
80%
80%
60%
60%
40%
40%
20%
20%
0%
0%
5
10
15
20
0
20
40
60
80
100
(a) LAL curves for distribution shift in Scenario 1, Increasing
(b) LAL curves for distribution shift in Scenario 2, Increasnig variance
100%
80%
60%
40%
20%
0%
18
(c) LAL curves for distribution shift in Scenario 3, Decreasing variance
Figure LAL-curves are drawn for various calibration sets D under distributions shift, as described Appendix Dark blue line no shift, Yellow line large shift.
20


Reviews:
[{"review_id": "U6GpMk7xn0", "reviewer": "Reviewer_bGoR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The proposed bound is very intuitive, and fits well within concerns of reliability in ML.\n- The proposed bound has a clear application to a variety of evaluation settings.\n- Authors investigate a variety of settings (classification / regression, bounded / unbounded loss, in-distribution / out-of-distribution) experimentally.\n\n### Weaknesses\n\nA recurring question I have about this paper is how it fits within the context of the greater evaluation literature. In particular:\n- For the proof of Thm 1, what components of the proof are required to deal with the non-iid data (departing from the Fligner & Wolfe (1976) work)?\n- I'd be curious to see a detailed example comparing LAL to existing methods described in the literature (for example, expanding upon Figure 1 by adding to the cross-validation metric already provided). Even without an experimental analysis, discussing concretely where LAL addresses failure modes of existing methodologies (VC-theory, etc.) would help position this work with respect to the existing literature. For example, by assuming access to the calibration set drawn from the same distribution as the evaluation points, existing tools to deal with distribution shift (e..g, measures of discrepancy) are no longer necessary, and I suspect (?) that tighter bounds are possible.\n\nOn a more minor note, \n- I found the proof of Theorem 1 to be very difficult to parse. Unless I a mistaken, $j$ is not defined. What are you choosing when you state that there are $\\binom{n-j+m-i}{m-i}$ valid choices? It sounds like you are choosing an ordering, but earlier (before Thm 1) you state that the calibration and test sets are ordered.\n- There are repeated results / statements in the experimental section. (\"The label y is the median house value in the block. The training data set D0 has n0 = 15 000 sampled without replacement. The calibration data set D has n = 150 and is sampled without replacement from the remaining data\", also the caption of Figure 2 is repeated in the main text).\n\n### Related questions\n- Many error bounds on models focus on worst-case analysis, which can be recovered by setting $\\beta=1$ for LAL if I understand correctly. Do you recover similar results, or does the existence of the calibration set provide tighter bounds?\n- On the experimental side, I really liked the OOD analysis. How does the bound evolve as you increase the type (mean/variance) and scale of shift?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This papers introduces an evaluation methodology for ML models based upon providing a bound the model's loss on new samples, which may be drawn from out-of-distribution data.\n\nThis \"level-$\\alpha$ limit\" (LAL) provides an upper bound $\\ell_\\alpha^\\beta(\\mathcal D)$  such that with probability $1-\\alpha$, a fraction $\\beta$ of samples has a loss bounded by $\\ell_\\alpha^\\beta(\\mathcal D)$, where $\\mathcal D$ is a calibration dataset drawn from the same distribution as the evaluation points (but not necessarily the training data distribution). \n\nThe authors provide a closed-form expression for a possible LAL in Theorem 1, and reinterpret this bound as a confidence interval boundary. \n\nFinally, the authors evaluate their LAL across a variety of classification and regression tasks, with and without dataset shift.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Clarifying the proof of thm.1, improving the positioning of the paper within existing literature.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cdcVRufyNY", "reviewer": "Reviewer_dgnY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n* It solves a simple but interesting method to evaluate a model.\n* Various experimental circumstances are tested.\n\n## Weaknesses\n\n* No existing methods are compared.\n* The scale of experiments is small.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates a method to assess a model for comparing the models.  Unlike a traditional approach to evaluating a model, the authors try to characterize a batch of out-of-sample losses using a calibration data.  Eventually, finite-sample limits on the out-of-sample losses are studied and its statistical validity under general conditions is provided.  Finally, several numerical experiments, i.e., experiments on distribution shifts, model selection, and hyperparameter tuning, are demonstrated.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please specify why the proposed method is a \"diagnostic\" tool.  I think the term \"diagnostic\" is very ambiguous.\n2. Comparisons to existing methods are missing.\n3. The scale of experiments is small.  I do not think it is necessary for conducting additional large-scale experiments but can you say that the analysis presented in this paper still holds for the large-scale experiments?", "broader_impact_concerns": "I have no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4fLV9QQUI8", "reviewer": "Reviewer_DNsE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths\n- The paper proposes a diagnostic tool, that can be used for various applications.\n- It seems to be simple, easy to compute, and intuitive, such as $k^\\star$ in Theorem 2.\n- Figure 1 is motivating and provides an intuitive explanation to the problem and proposed method.\n- The authors are willing to provide code to reproduce all experiments (while link was not provided in the manuscript to preserve anonymity).\n\nWeaknesses\n- The paper could benefit from more discussions about its advantages over existing metrics or ideas such as Value-at-Risk. For instance, Theorem 2's results seems to be similar to the procedure of deriving the VaR because we are choosing $k^\\star$ to be $\\lceil(n+1)(1-\\alpha)\\rceil$. It would be helpful if there are discussions about the differences and benefits over VaR.\n- The practical benefit of utilizing LAL for distribution shift analysis appears somewhat unclear. Many transfer learning methods and domain adaptation methods study the situation where we have additional test samples during training. The paper's setup seems similar, because we have labelled samples from target distribution $p$. It would be natural to use this for training if we already know that there is a distribution shift. For identifying the potential shift, a more naive approach may be to just compare the two average test losses for $\\mathcal{D}_1$ and $\\mathcal{D}_2$. It would be helpful if the paper can have more motivating discussions for this application.\n- Since the choice of $m$ appears somewhat arbitrary, the inclusion of guidelines for $m$ selection would improve the paper's practical value.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a scalar metric called level-$\\alpha$ limit for the $\\beta$-fraction of $m$ losses, or LAL for short, where $m$ is the number of unknown out-of-sample input-output pairs. The first theorem shows how we can find this limit for a given fraction $\\beta$ and probability $\\alpha$ by using calibration data from the test distribution $p$. Then the paper shows another Theorem for the case with $m=1$.  The paper proves a corollary when $m\\to\\infty$. Various applications of LAL are shown in the experiments. Finally, the paper discusses related work such as model evaluation techniques, non-parametric statistics, and conformal prediction.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would appreciate it if the authors can provide some discussions about the 3 aspects I raised in \"weaknesses\" in the previous section.", "broader_impact_concerns": "There is no \"Broader Impact Statement\" section in this paper. I have no ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Most of the reviewers and myself found the paper well written. The authors motivate well the problem they want to tackle and nicely introduce the notions and results they need to develop their methodology. 

One of the reviewers found that the experiments were limited, a viewpoint with which I agree. However, I believe that as an initial introduction, the experiments conducted by the authors suffice as a proof of concept to justify their new tools. This is why I have decided to accept the paper.

Please take into consideration reviewers' comments before submitting the final version of your work.
In particular, I found the proof of Theorem 1 a little hard to follow. Could the authors formalize their arguments a little more to improve readability? 
Finally, the authors should acknowledge that their experiments are small-scale and discuss how larger-scale experiments could be conducted."
Decision: Accept (poster)