# Cross-Validation, Bootstrap, and Support Vector Machines

Authors: Masaaki Tsujitani; Yusuke Tanaka
Journal: Advances in Artificial Neural Systems
 (2011)

---

## Abstract

This paper considers the applications of resampling methods to support vector machines (SVMs). We take into account the leaving-one-out cross-validation (CV) when determining the optimum tuning parameters and bootstrapping the deviance in order to summarize the measure of goodness-of-fit in SVMs. The leaving-one-out CV is also adapted in order to provide estimates of the bias of the excess error in a prediction rule constructed with training samples. We analyze the data from a mackerel-egg survey and a liver-disease study.


---

## Body

## 1. Introduction

In recent years, support vector machines (SVMs) have been intensively studied and applied to practical problems in many fields of science and engineering [1–3]. SVMs have many merits that distinguish them from many other machine learning algorithms, including the nonexistence of local minima, the speed of calculation, and the use of only two tuning parameters. There are at least two reasons to use a leaving-one-out cross-validation (CV) [4]. First, the criterion based on the method is demonstrated to be favorable when determining the tuning parameters. Second, the method can estimate the bias of the excess error in prediction. No standard procedures exist by which to assess the overall goodness-of-fit of the model based on SVM. By introducing the maximum likelihood principle, the deviance allows us to test the goodness-of-fit of the model. Since no adequate distribution theory exists for the deviance, we provide bootstrapping on the null distribution of the deviance for the model having optimum tuning parameters for SVM with a specified significance level [5–8].The remainder of this paper is organized as follows. In Section2, using the leaving-one-out CV, we focus on the determination of the tuning parameters and the evaluation of the overall goodness-of-fit with the optimum tuning parameters based on bootstrapping. The leaving-one-out CV is also adapted in order to provide estimates of the bias of the excess error in a prediction rule constructed with training samples [9]. In Section 3, the one-against-one method is used to estimate a vector of multiclass probabilities for each pair of classes and then to couple the estimates together [3, 10]. In Section 4, the methods are illustrated using mackerel-egg survey and liver-disease data. We discuss the relative merits and limitations of the methods in Section 5.


## 2. Support Vector Machines and Resampling  Methods

### 2.1. Support Vector Machines

Givenn training pairs (x1,y1),(x2,y2),…,(xn,yn), where xi is an input vector and yi∈{-1,+1}, the SVM solves the following primal problem: (1)min⁡β,b12βTβ+C∑i=1nξis.t.yi{βTϕ(xi)+b}≥1-ξi,ξi≥0,i=1,2,…,n,
where β is a unit vector (i.e., ∥β∥=1), T denotes the transposition of the matrix, K(x,xi)≡ϕ(xi)Tϕ(xi) is the kernel function, C is the tuning parameter denoting the tradeoff between the margin width and the training data error, and ξi≥0 are slack variables. For an unknown input pattern x, we have the decision function (2)f(x)=∑i=1nαiyiK(x,xi)+b,
where {αi,i=1,2,…,n;αi≥0} are the Lagrange multipliers. We employ the Gaussian radial basis function as the kernel function [3, 11, 12] (3)K(x,x′)=exp⁡(-γ‖x-x′‖2),
where γ>0 is a fixed parameter, and (4)‖x-x′‖2=〈x-x′,x-x′〉.Binary classification is performed by using the decision functionf(x): the input X=(x1,x2,…,xn)T is assigned to the positive class if f(x)≥0, and to the negative class otherwise. Platt [13] proposed one method for producing probabilistic outputs from a decision function by using logistic link function (5)p(y=+1∣f)=11+eAf+B,
where fi=f(xi) and yi represent the output of the SVM and the target value for the sample, respectively [14]. This is equivalent to fitting a logistic regression model to the estimated decision values. The unknown parameters A,B in (5) can be estimated by minimizing the cross-entropy (6)Min⁡A,B[-∑i=1n{yiln⁡pi+(1-yi)ln⁡(1-pi)}],
where (7)pi=11+eAfi+B.
Putting (8)ti=yi+12={0:yi=-1,1:yi=+1,
from (6), (7), and (8), we obtain (9)-{tiln⁡pi+(1-yi)ln⁡(1-ti)}=ti(Afi+B)+ln⁡{1+exp⁡(-Afi-B)}.
Lin et al. [15] observed that the problem of  ln(0) never occurs for (9).


### 2.2. Leaving-One-Out Cross-Validation

#### 2.2.1. CV Score

We must determine the optimum values of tuning parametersC and γ in (1) and (3), respectively. This can be done by means of the leaving-one-out CV; a by-product is that the excess error rate of incorrectly predicting the outcome is estimated.Let the initial sampleX={X1,X2,…,Xi-1,Xi,Xi+1,…,Xn} with Xi=(xi,ti) be independently distributed according to an unknown distribution. The leaving-one-out CV algorithm is then given as follows (see, e.g., [5]).Step 1.
From the initial sampleX, Xi are deleted in order to form the training sample X[i]={X1,X2,…,Xi-1,Xi+1,…,Xn}.Step 2.
Using each training sample, fit an SVM and predict the decision valuef̂[i] for Xi.Step 3.
From the decision valuef̂[i], we can predict p̂[i] for the deleted ith sample using (7) and calculate the predicted log-likelihood t[i]lnp̂[i]+(1-t[i])ln(1-p̂[i]).Step 4.
Steps1 to 3 are repeated for i=1,2,…,n.Step 5.
The CV score (i.e., averaged predicted log-likelihood) is given by(10)CV=-2∑i=1n{t[i]ln⁡p̂[i]+(1-t[i])ln⁡(1-p̂[i])}.Step 6.
Carry out a grid search over tuning parametersC and γ, taking the tuning parameters with minimum CV as optimal. It should be noted that the CV score is asymptotically equivalent to AIC (akaike information criterion) and EIC (extended information criterion) [16–18].


#### 2.2.2. Excess Error Estimation

Let theactual error rate be the probability of incorrectly predicting the outcome of a new observation, given a discriminant rule on initial sample X; this is useful for performance assessment of a discriminant rule. Given a discriminant rule based on the initial sample, the error rates of discrimination are also of interest. As the same observations are used for forming and assessing the discriminant rule, this proportion of errors, called the apparent error rate, underestimates the actual error rate. The estimate of the error rate is seriously biased when the initial sample is small. This bias for a given discriminant rule is called the excess error of that rule. To correct this bias and estimate the error rates, we provide the bias correction of the apparent error rate associated with a discriminant rule, which is constructed by fitting to the training sample in the SVM.By applying a discriminant rule to the initial sampleX, we can form the realized discriminant rule ηX. Let ηX[i] be the discrimination rule based on X[i]. Given a subject with xi, we predict the response by ηX[i](xi). The algorithm for leaving-one-out CV that estimates the excess error rate when fitting a SVM is given as follows [9].Step 1.
Generate the training sampleX[i], and construct the realized discrimination rule ηX[i] based on X[i]. Then, define
(11)Q(ti;ηX[i](xi))={1:incorrectdiscrimination,0:otherwise.
Then leaving-one-out CV error rate is given by
(12)1n∑i=1nQ(ti;ηX[i](xi)).Step 2.
Calculate the apparent error(13)1n∑i=1nQ(ti;ηX(xi)).Step 3.
The cross-validation estimator of expected excess error is(14)r̂CV=1n∑i=1nQ(ti;ηX(xi))-1n∑i=1nQ(ti;ηX[i](xi)).


### 2.3. Bootstrapping

Introducing the maximum likelihood principle into the SVM, the deviance allows us to test the goodness-of-fit of the model(15)Dev=2[ln⁡L̂f-ln⁡L̂c]=-2∑i=1n{tiln⁡p̂i+(1-ti)ln⁡(1-p̂i)},
where lnL̂c denotes the maximized log likelihood under some current SVM, and the log likelihood for the saturated model lnL̂f is zero. The deviance given by (15) is, however, not even approximately a χ2 distribution for the case in which ungrouped binary responses are available [19, 20]. The number of degrees of freedom (d.f.) required for the test for significance using the assumed χ2 distribution for the deviance is a contentious issue. No adequate distribution theory exists for the deviance. The reason for this is somewhat technical (for details, see Section  3.8.3 in [19]). Consequently, the deviance on fitting a model to binary response data cannot be used as a summary measure of the goodness-of-fit test of the model.Based on the above discussion, the percentile of deviance for goodness-of-fit test can in principle be calculated. However, the calculations are usually too complicated to perform analytically, so Monte Carlo method can be employed [6, 7].Step 1.
GenerateB bootstrap samples X* from the original sample X. Let Xb* denote the bth bootstrap sample.Step 2.
For the bootstrap sampleXb*, compute the deviance of (15), denoted by Dev*(b).
Steps1 and 2 are repeated independently B times, and the computed values are arranged in ascending order.Step 3.
Take the value of thejth order statistic Dev*(b) of the B replications as an estimate of the quantile of order j/(B+1).Step 4.
The estimate of the100(1-α)th percentile of Dev*(b) is used to test the goodness-of-fit of a model having a specified significance level α=1-j/(B+1). The value of the deviance of (15) being greater than the estimate of the percentile indicates that the model fits poorly. Typically, the number of replication B is in the range of 50≤B≤400.


### 2.4. Influential Analysis

Assessing the discrepancies betweenti and p̂i at the ith observation in (15), the influence measure provides guides and suggestions that may be carefully applied to a SVM [19]. The effect of the ith observation on the deviance can be measured by computing (16)ΔDev[i]=Dev-Dev[i],
where Dev[i] is the deviance with ith observation deleted. The distribution of ΔDev[i] will be approximated by χ2 with d.f. = 1 when the fitted model is correct. An index plot is a reasonable rule of thumb for graphically presenting the information contained in the values of ΔDev[i]. The key idea behind this plot is not to focus on a global measure of goodness-of-fit but rather on local contributions to the fit. An influential observation is one that greatly changes the results of the statistical inference when deleted from the initial sample.Platt [13] proposed the threefold CV for estimating the decision values in (9). However, the value of ΔDev[i] may be negative because three SVMs are trained on splitted three parts of training pairs (x1,y1),(x2,y2),…,(xn,yn). Therefore, in the present paper, we train a single SVM on the training pairs in order to evaluate the decision values fi′s and estimate probabilistic outputs according to [15].


## 2.1. Support Vector Machines

Givenn training pairs (x1,y1),(x2,y2),…,(xn,yn), where xi is an input vector and yi∈{-1,+1}, the SVM solves the following primal problem: (1)min⁡β,b12βTβ+C∑i=1nξis.t.yi{βTϕ(xi)+b}≥1-ξi,ξi≥0,i=1,2,…,n,
where β is a unit vector (i.e., ∥β∥=1), T denotes the transposition of the matrix, K(x,xi)≡ϕ(xi)Tϕ(xi) is the kernel function, C is the tuning parameter denoting the tradeoff between the margin width and the training data error, and ξi≥0 are slack variables. For an unknown input pattern x, we have the decision function (2)f(x)=∑i=1nαiyiK(x,xi)+b,
where {αi,i=1,2,…,n;αi≥0} are the Lagrange multipliers. We employ the Gaussian radial basis function as the kernel function [3, 11, 12] (3)K(x,x′)=exp⁡(-γ‖x-x′‖2),
where γ>0 is a fixed parameter, and (4)‖x-x′‖2=〈x-x′,x-x′〉.Binary classification is performed by using the decision functionf(x): the input X=(x1,x2,…,xn)T is assigned to the positive class if f(x)≥0, and to the negative class otherwise. Platt [13] proposed one method for producing probabilistic outputs from a decision function by using logistic link function (5)p(y=+1∣f)=11+eAf+B,
where fi=f(xi) and yi represent the output of the SVM and the target value for the sample, respectively [14]. This is equivalent to fitting a logistic regression model to the estimated decision values. The unknown parameters A,B in (5) can be estimated by minimizing the cross-entropy (6)Min⁡A,B[-∑i=1n{yiln⁡pi+(1-yi)ln⁡(1-pi)}],
where (7)pi=11+eAfi+B.
Putting (8)ti=yi+12={0:yi=-1,1:yi=+1,
from (6), (7), and (8), we obtain (9)-{tiln⁡pi+(1-yi)ln⁡(1-ti)}=ti(Afi+B)+ln⁡{1+exp⁡(-Afi-B)}.
Lin et al. [15] observed that the problem of  ln(0) never occurs for (9).


## 2.2. Leaving-One-Out Cross-Validation

### 2.2.1. CV Score

We must determine the optimum values of tuning parametersC and γ in (1) and (3), respectively. This can be done by means of the leaving-one-out CV; a by-product is that the excess error rate of incorrectly predicting the outcome is estimated.Let the initial sampleX={X1,X2,…,Xi-1,Xi,Xi+1,…,Xn} with Xi=(xi,ti) be independently distributed according to an unknown distribution. The leaving-one-out CV algorithm is then given as follows (see, e.g., [5]).Step 1.
From the initial sampleX, Xi are deleted in order to form the training sample X[i]={X1,X2,…,Xi-1,Xi+1,…,Xn}.Step 2.
Using each training sample, fit an SVM and predict the decision valuef̂[i] for Xi.Step 3.
From the decision valuef̂[i], we can predict p̂[i] for the deleted ith sample using (7) and calculate the predicted log-likelihood t[i]lnp̂[i]+(1-t[i])ln(1-p̂[i]).Step 4.
Steps1 to 3 are repeated for i=1,2,…,n.Step 5.
The CV score (i.e., averaged predicted log-likelihood) is given by(10)CV=-2∑i=1n{t[i]ln⁡p̂[i]+(1-t[i])ln⁡(1-p̂[i])}.Step 6.
Carry out a grid search over tuning parametersC and γ, taking the tuning parameters with minimum CV as optimal. It should be noted that the CV score is asymptotically equivalent to AIC (akaike information criterion) and EIC (extended information criterion) [16–18].


### 2.2.2. Excess Error Estimation

Let theactual error rate be the probability of incorrectly predicting the outcome of a new observation, given a discriminant rule on initial sample X; this is useful for performance assessment of a discriminant rule. Given a discriminant rule based on the initial sample, the error rates of discrimination are also of interest. As the same observations are used for forming and assessing the discriminant rule, this proportion of errors, called the apparent error rate, underestimates the actual error rate. The estimate of the error rate is seriously biased when the initial sample is small. This bias for a given discriminant rule is called the excess error of that rule. To correct this bias and estimate the error rates, we provide the bias correction of the apparent error rate associated with a discriminant rule, which is constructed by fitting to the training sample in the SVM.By applying a discriminant rule to the initial sampleX, we can form the realized discriminant rule ηX. Let ηX[i] be the discrimination rule based on X[i]. Given a subject with xi, we predict the response by ηX[i](xi). The algorithm for leaving-one-out CV that estimates the excess error rate when fitting a SVM is given as follows [9].Step 1.
Generate the training sampleX[i], and construct the realized discrimination rule ηX[i] based on X[i]. Then, define
(11)Q(ti;ηX[i](xi))={1:incorrectdiscrimination,0:otherwise.
Then leaving-one-out CV error rate is given by
(12)1n∑i=1nQ(ti;ηX[i](xi)).Step 2.
Calculate the apparent error(13)1n∑i=1nQ(ti;ηX(xi)).Step 3.
The cross-validation estimator of expected excess error is(14)r̂CV=1n∑i=1nQ(ti;ηX(xi))-1n∑i=1nQ(ti;ηX[i](xi)).


## 2.2.1. CV Score

We must determine the optimum values of tuning parametersC and γ in (1) and (3), respectively. This can be done by means of the leaving-one-out CV; a by-product is that the excess error rate of incorrectly predicting the outcome is estimated.Let the initial sampleX={X1,X2,…,Xi-1,Xi,Xi+1,…,Xn} with Xi=(xi,ti) be independently distributed according to an unknown distribution. The leaving-one-out CV algorithm is then given as follows (see, e.g., [5]).Step 1.
From the initial sampleX, Xi are deleted in order to form the training sample X[i]={X1,X2,…,Xi-1,Xi+1,…,Xn}.Step 2.
Using each training sample, fit an SVM and predict the decision valuef̂[i] for Xi.Step 3.
From the decision valuef̂[i], we can predict p̂[i] for the deleted ith sample using (7) and calculate the predicted log-likelihood t[i]lnp̂[i]+(1-t[i])ln(1-p̂[i]).Step 4.
Steps1 to 3 are repeated for i=1,2,…,n.Step 5.
The CV score (i.e., averaged predicted log-likelihood) is given by(10)CV=-2∑i=1n{t[i]ln⁡p̂[i]+(1-t[i])ln⁡(1-p̂[i])}.Step 6.
Carry out a grid search over tuning parametersC and γ, taking the tuning parameters with minimum CV as optimal. It should be noted that the CV score is asymptotically equivalent to AIC (akaike information criterion) and EIC (extended information criterion) [16–18].


## 2.2.2. Excess Error Estimation

Let theactual error rate be the probability of incorrectly predicting the outcome of a new observation, given a discriminant rule on initial sample X; this is useful for performance assessment of a discriminant rule. Given a discriminant rule based on the initial sample, the error rates of discrimination are also of interest. As the same observations are used for forming and assessing the discriminant rule, this proportion of errors, called the apparent error rate, underestimates the actual error rate. The estimate of the error rate is seriously biased when the initial sample is small. This bias for a given discriminant rule is called the excess error of that rule. To correct this bias and estimate the error rates, we provide the bias correction of the apparent error rate associated with a discriminant rule, which is constructed by fitting to the training sample in the SVM.By applying a discriminant rule to the initial sampleX, we can form the realized discriminant rule ηX. Let ηX[i] be the discrimination rule based on X[i]. Given a subject with xi, we predict the response by ηX[i](xi). The algorithm for leaving-one-out CV that estimates the excess error rate when fitting a SVM is given as follows [9].Step 1.
Generate the training sampleX[i], and construct the realized discrimination rule ηX[i] based on X[i]. Then, define
(11)Q(ti;ηX[i](xi))={1:incorrectdiscrimination,0:otherwise.
Then leaving-one-out CV error rate is given by
(12)1n∑i=1nQ(ti;ηX[i](xi)).Step 2.
Calculate the apparent error(13)1n∑i=1nQ(ti;ηX(xi)).Step 3.
The cross-validation estimator of expected excess error is(14)r̂CV=1n∑i=1nQ(ti;ηX(xi))-1n∑i=1nQ(ti;ηX[i](xi)).


## 2.3. Bootstrapping

Introducing the maximum likelihood principle into the SVM, the deviance allows us to test the goodness-of-fit of the model(15)Dev=2[ln⁡L̂f-ln⁡L̂c]=-2∑i=1n{tiln⁡p̂i+(1-ti)ln⁡(1-p̂i)},
where lnL̂c denotes the maximized log likelihood under some current SVM, and the log likelihood for the saturated model lnL̂f is zero. The deviance given by (15) is, however, not even approximately a χ2 distribution for the case in which ungrouped binary responses are available [19, 20]. The number of degrees of freedom (d.f.) required for the test for significance using the assumed χ2 distribution for the deviance is a contentious issue. No adequate distribution theory exists for the deviance. The reason for this is somewhat technical (for details, see Section  3.8.3 in [19]). Consequently, the deviance on fitting a model to binary response data cannot be used as a summary measure of the goodness-of-fit test of the model.Based on the above discussion, the percentile of deviance for goodness-of-fit test can in principle be calculated. However, the calculations are usually too complicated to perform analytically, so Monte Carlo method can be employed [6, 7].Step 1.
GenerateB bootstrap samples X* from the original sample X. Let Xb* denote the bth bootstrap sample.Step 2.
For the bootstrap sampleXb*, compute the deviance of (15), denoted by Dev*(b).
Steps1 and 2 are repeated independently B times, and the computed values are arranged in ascending order.Step 3.
Take the value of thejth order statistic Dev*(b) of the B replications as an estimate of the quantile of order j/(B+1).Step 4.
The estimate of the100(1-α)th percentile of Dev*(b) is used to test the goodness-of-fit of a model having a specified significance level α=1-j/(B+1). The value of the deviance of (15) being greater than the estimate of the percentile indicates that the model fits poorly. Typically, the number of replication B is in the range of 50≤B≤400.


## 2.4. Influential Analysis

Assessing the discrepancies betweenti and p̂i at the ith observation in (15), the influence measure provides guides and suggestions that may be carefully applied to a SVM [19]. The effect of the ith observation on the deviance can be measured by computing (16)ΔDev[i]=Dev-Dev[i],
where Dev[i] is the deviance with ith observation deleted. The distribution of ΔDev[i] will be approximated by χ2 with d.f. = 1 when the fitted model is correct. An index plot is a reasonable rule of thumb for graphically presenting the information contained in the values of ΔDev[i]. The key idea behind this plot is not to focus on a global measure of goodness-of-fit but rather on local contributions to the fit. An influential observation is one that greatly changes the results of the statistical inference when deleted from the initial sample.Platt [13] proposed the threefold CV for estimating the decision values in (9). However, the value of ΔDev[i] may be negative because three SVMs are trained on splitted three parts of training pairs (x1,y1),(x2,y2),…,(xn,yn). Therefore, in the present paper, we train a single SVM on the training pairs in order to evaluate the decision values fi′s and estimate probabilistic outputs according to [15].


## 3. Multiclass SVM

We consider the discriminant problem withK classes and n training pairs (x1,t1),(x2,t2),…,(xn,tn), where xi is an input vector and ti=(ti1,ti2,…,tiK) [10, 21, 22]. Let pii,pi2,…,piK denote the response probabilities, with ∑k=1Kpik=1, for multiclass classification with (17)tik={1:inputvectorxiisfromkthclass,0:otherwise.
The log-likelihood is given by (18)ln⁡L=∑i=1n∑k=1K{tikln⁡(pik)}.For multi-class classification, the one-against-one method (also called pairwise classification) is used to produce a vector of multi-class probabilities for each pair of classes, and then to couple the estimates together [10]. The earliest used implementation for multi-class SVM is probably the one-against-one method of [21]. This method constructs K(K-1)/2 classifiers based on the training on data from the kth and lth classes of training set.The SVM solves the primal formulation [3, 10, 23] (19)min⁡βkl,bkl,ξkl12(βkl)Tβkl+C∑tξtkls.t.(βkl)Tϕ(xt)+bkl≥1-ξtkl,ifyt=k,(βkl)Tϕ(xt)+bkl≤-1+ξtkl,ifyt=l,ξtkl≥0.
Given K classes of data for any x, the goal is to estimate (20)pk=Pr{t=k∣x},k=1,2,…,K.
We first estimate pairwise class probabilities (21)rkl≈Pr(t=k∣t=korl,x),
by using (22)rkl≈11+eAf+B,
where A and B are estimated by minimizing the cross entropy using training data and the corresponding decision values f.Hastie and Tibshirani [21] proposed minimizing the Kullback-Leibler (KL) distance between rkl and μkl=E[rkl]=pk/(pk+pl), (23)l(p)=∑k≠lnklrklln⁡(rklμkl)=∑k<lnkl{rklln⁡(rklμkl)+(1-rkl)ln⁡(1-rkl1-μkl)},
where rlk=1-rkl and nkl is the number of training data in the kth and lth classes.Wu et al. [10] propose the second approach to obtain pk from all these rkl's by optimizing (24)min⁡12∑k=1K∑k:k≠l(rklpi-rklpj)2s.t.∑k=1Kpk=0,pk≥0.Thus, we can adopt the leaving-one-out CV similar to the method in Section2.2.Step 1.
From the initial sampleX, Xi are deleted in order to form the training sample X[i]={X1,X2,…,Xi-1,Xi+1,…,Xn}.Step 2.
Using each training sample, fit a SVM in order to estimaterkl[i] by (22), and predict (p̂[i1],p̂[i2],…,p̂[iK]) for the deleted ith sample X[i].Step 3.
Steps1 and 2 are repeated for i=1,2,…,n.Step 4.
The CV score is given by(25)CV=∑i=1n∑k=1K{t[ik]ln⁡(p̂[ik])}.Step 5.
Tuning parameters with minimum CV can be determined as optimal by carrying out a grid search overC and γ.


## 4. Examples

### 4.1. Mackerel-Egg Survey Data

We consider data consisting of 634 observations from a 1992 mackerel egg survey [24]. There are the following predictors of egg abundance: the location (longitude and latitude) at which samples were taken, depth of the ocean, distance from the 200 m seabed contour, and, finally, water temperature at a depth of 20 m. We first fit a SVM. In the same manner as described in [11], we determine tuning parameters C and γ. The optimum values of the tuning parameters are (C,γ)=(28,0.09). The bootstrap estimator of the percentile for the deviance is Dev*(b)=444.31. A comparison with the deviance Dev=443.132 from (15) suggests that the SVM fits the data fairly well. For reference purposes, the histogram of the bootstrapped Dev*(b) for B=400 is provided in Figure 1.Figure 1
Histogram of the bootstrappedDev*(b) for mackerel-egg survey data.We can estimate the apparent errors rate of incorrectly predicting outcome and leaving-one-out CV error rates for several models as shown in Table1. The smoothing parameters in generalized additive models (GAM) [24] and the number of hidden units in a neural network in Table 1 are determined using the leaving-one-out CV. From Table 1, the leaving-one-out CV error rate for the SVM is the smallest among all models, but the apparent error rate is the smallest for the neural network. The CV scores are 477.04, 509.44, and 541.61 for the SVM, logistic discriminant, and neural network, respectively. This implies that the SVM is the best among these three models from the point of view of CV. Figure 2 shows the index plot of ΔDev[i], which indicates that no. 399 and no. 601 are influential observations at the 0.01% level of significance.Table 1
Error rates for mackerel-egg survey data.
ModelApparent error rateLeaving-one-out CV error rateSVM0.1590.167GAM0.1580.175Neural network with three hidden units0.1530.181Fisher's linear discriminant0.1800.185Logistic discriminant0.1920.196Figure 2
DIFDEV for mackerel-egg survey data.


### 4.2. Liver Disease Data

We apply the proposed method to laboratory data collected from 218 patients with liver disorders [25–27]. Four liver diseases were observed: acute viral hepatitis (57 patients), persistent chronic hepatitis (44 patients), aggressive chronic hepatitis (40 patients), and postnecrotic cirrhosis (77 patients). The covariates consist of four liver enzymes: aspartate aminotransferase (AST), alanine aminotransferase (ALT), glutamate dehydrogenase (GIDH), and ornithine carbamyltransferase (OCT). For each (C,γ) pair, the CV performance is measured by training 70% and testing the other 30% of the data. Then, we train the whole training set by using the pair (C,γ)=(93,0.20), which achieves the minimum CV score (=187.93) and predicts the test set. The apparent and leaving-one-out CV error rates for traing and test samples for several models as shown in Table 2. As shown, the apparent error rate for SVM of training sample and the error rate for SVM of test sample are the smallest among all models, but the leaving-one-out CV error rate for SVM of training sample is larger than that of the multinomial logistic discriminant model.Table 2
Error rates for liver disease data.
ModelTraining sampleTest sampleApparent             errorLeaving-one-out             CV errorSVM0.0850.2090.292Fisher's linear discriminant0.3660.3600.340Multinomial logistic discriminant0.2290.1960.323


## 4.1. Mackerel-Egg Survey Data

We consider data consisting of 634 observations from a 1992 mackerel egg survey [24]. There are the following predictors of egg abundance: the location (longitude and latitude) at which samples were taken, depth of the ocean, distance from the 200 m seabed contour, and, finally, water temperature at a depth of 20 m. We first fit a SVM. In the same manner as described in [11], we determine tuning parameters C and γ. The optimum values of the tuning parameters are (C,γ)=(28,0.09). The bootstrap estimator of the percentile for the deviance is Dev*(b)=444.31. A comparison with the deviance Dev=443.132 from (15) suggests that the SVM fits the data fairly well. For reference purposes, the histogram of the bootstrapped Dev*(b) for B=400 is provided in Figure 1.Figure 1
Histogram of the bootstrappedDev*(b) for mackerel-egg survey data.We can estimate the apparent errors rate of incorrectly predicting outcome and leaving-one-out CV error rates for several models as shown in Table1. The smoothing parameters in generalized additive models (GAM) [24] and the number of hidden units in a neural network in Table 1 are determined using the leaving-one-out CV. From Table 1, the leaving-one-out CV error rate for the SVM is the smallest among all models, but the apparent error rate is the smallest for the neural network. The CV scores are 477.04, 509.44, and 541.61 for the SVM, logistic discriminant, and neural network, respectively. This implies that the SVM is the best among these three models from the point of view of CV. Figure 2 shows the index plot of ΔDev[i], which indicates that no. 399 and no. 601 are influential observations at the 0.01% level of significance.Table 1
Error rates for mackerel-egg survey data.
ModelApparent error rateLeaving-one-out CV error rateSVM0.1590.167GAM0.1580.175Neural network with three hidden units0.1530.181Fisher's linear discriminant0.1800.185Logistic discriminant0.1920.196Figure 2
DIFDEV for mackerel-egg survey data.


## 4.2. Liver Disease Data

We apply the proposed method to laboratory data collected from 218 patients with liver disorders [25–27]. Four liver diseases were observed: acute viral hepatitis (57 patients), persistent chronic hepatitis (44 patients), aggressive chronic hepatitis (40 patients), and postnecrotic cirrhosis (77 patients). The covariates consist of four liver enzymes: aspartate aminotransferase (AST), alanine aminotransferase (ALT), glutamate dehydrogenase (GIDH), and ornithine carbamyltransferase (OCT). For each (C,γ) pair, the CV performance is measured by training 70% and testing the other 30% of the data. Then, we train the whole training set by using the pair (C,γ)=(93,0.20), which achieves the minimum CV score (=187.93) and predicts the test set. The apparent and leaving-one-out CV error rates for traing and test samples for several models as shown in Table 2. As shown, the apparent error rate for SVM of training sample and the error rate for SVM of test sample are the smallest among all models, but the leaving-one-out CV error rate for SVM of training sample is larger than that of the multinomial logistic discriminant model.Table 2
Error rates for liver disease data.
ModelTraining sampleTest sampleApparent             errorLeaving-one-out             CV errorSVM0.0850.2090.292Fisher's linear discriminant0.3660.3600.340Multinomial logistic discriminant0.2290.1960.323


## 5. Concluding Remarks

We considered the application of resampling methods to SVMs. Statistical inference based on the likelihood approach for SVMs was discussed, and the leaving-one-out CV was suggested for determining the tuning of parameters and for estimating the bias of the excess error in prediction. Bootstrapping is used to focus on the evaluation of the overall goodness-of-fit with the optimum tuning parameters. Data from a mackerel-egg survey and a liver-disease study are used to evaluate the resampling methods.There is one broad limitation to our approach: the SVM assumed the independence of the predictor variables. More generally, it may be preferable to visualize interactions between predictor variables. The smoothing spline ANOVA models [28] can provide an excellent means for handling data of mutually exclusive groups and a set of predictor variables. We expect that flexible methods for a discriminant model using machine learning theory [1], such as penalized smoothing splines, will be very useful in these real-world contexts.



---

*Source: 302572-2011-07-27.xml*