# Neural Discriminant Models, Bootstrapping, and Simulation

Authors: Masaaki Tsujitani; Katsuhiro Iba; Yusuke Tanaka
Journal: ISRN Artificial Intelligence
 (2012)

---

## Abstract

This paper considers the feed-forward neural network models for data
of mutually exclusive groups and a set of predictor variables. We take
into account the bootstrapping based on information criterion when selecting
the optimum number of hidden units for a neural network model
and the deviance in order to summarize the measure of goodness-of-fit on
fitted neural network models. The bootstrapping is also adapted in order
to provide estimates of the bias of the excess error in a prediction rule
constructed with training samples. Simulated data from known (true)
models are analyzed in order to interpret the results using the neural
network. In addition, the thyroid disease database, which compares estimated
measures of predictive performance, is examined in both a pure
training sample study and in a test sample study, in which the realized
test sample apparent error rates associated with a constructed prediction
rule are reported. Apartment house data of the metropolitan area station
with four-class classification are also analyzed in order to assess the bootstrapping
by comparing leaving-one-out cross-validation (CV).


---

## Body

## 1. Introduction

The neural network model is considered for the multiclass classification problem of assigning each observation into one of multiclass, which is referred to as amultiple-group neural discriminant model. As two-class problems are much easier to solve, we focus on neural networks for multiclass classification with respect to statistical techniques in order to derive the maximum likelihood estimators (MLE) [1–7]. Statistical techniques are formulated in terms of the principle of the likelihood of the neural discriminant model, in which the connection weights of the network are treated as unknown parameters.Besides the theoretical and empirical properties of the bootstrapping [8, 9] in the multiple-group neural discriminant model, there are at least two other reasons to use a bootstrap procedure. First, the criterion based on bootstrapping is demonstrated to be favorable when selecting the optimum number of hidden units. A number of model selection procedures (i.e., methods for the selection of the optimum number of hidden units), such as Akaike information criterion (AIC), BIC (Baysesian information criterion) and cross-validation [10–13] have been proposed. The bootstrap method, however, provides the percentile for the deviance, allowing evaluation of the overall goodness-of-fit and estimation of the bias of the excess error in prediction based on the selected model. Therefore, there is no extra cost for subsequence inference via the bootstrap samples generated for model selection. If a model is selected by a cross-validation method and the bootstrap is used for the subsequence inference, the extra cost of computations is required in resampling for cross-validation. Second, the bootstrap procedures developed in the multiple-group neural discriminant model can be extended, without any theoretical derivation, to more complicated problems such as the generalized additive models (GAM) [14, 15], support vector machines (SVM) [16–19], and vector generalized additive models (VGAM) [20].The remainder of this paper is organized as follows. In Section2 we focus on the selection of the optimum number of hidden units and evaluation of the overall goodness-of-fit with the optimum number of hidden units. A neural network can approximate any reasonable function with arbitrary precision if the number of hidden units tends to infinity [21]. The output of the network fits the training sample too closely if the number of hidden units is increased and the noise is modeled in addition to the desired underlying function. The bootstrapping is also adapted in order to provide estimates of the bias of the excess error in a prediction rule constructed with training samples [22, 23]. Simulated data from known (true) models are used to demonstrate the approximate realization of continuous mapping by neural networks in Section 3. In Section 4 the methods are illustrated using a thyroid disease database in order to show that the overfitting leads poor generalization. Apartment house data of the metropolitan area station with four-class classification are also analyzed in order to assess the bootstrapping by comparing leaving-one-out CV. Finally, in Section 5 we discuss the relative merits and limitations of the methods.


## 2. Materials and Methods

### 2.1. Multiple-Group Neural Discriminant Model

#### 2.1.1. Statistical Inference

The functional representation of the neural network model is considered, as shown in Figure1. The connection weight between the ith unit in the input layer (i=0,…,I) and the jth unit in the hidden layer (j=1,…,H) is αij. Similarly, the weight between the jth unit in the hidden layer (j=0,…,H) and the kth unit in the output layer (k=1,…,K) is βjk. The input to the jth hidden unit is a linear projection of the input vector x=(x1,…,xI), that is,(1)uj=∑i=0Iαijxi,x0≡1,
where α0j is a bias. This is the same idea as incorporating the constant term in the design matrix of a regression by including a column of 1’s [1]. The output of the jth hidden unit is(2)yj=f(uj)=f(∑i=0Iαijxi),
where f(·) is a nonlinear activation function. The most commonly used activation function is the logistic (sigmoid) function:(3)yj=11+exp⁡(-uj).
The input to the kth output unit is(4)vk=∑j=0Hβjkyj,y0≡1,
where β0j is a bias. The activation function of network outputs for the mutually exclusive groups can be achieved using the softmax activation (normalized exponential) function:(5)ok=exp⁡(vk)∑k=1Kexp⁡(vk)=11+exp⁡(-Vk),(6)Vk=vk-ln⁡{∑k′≠kKexp⁡(vk′)},
which can be regarded as a multiclass generalization of logistic function.Figure 1
Single hidden layer neural network model.From (1)–(6), ok can be written in the form(7)ok=exp⁡(vk)∑k′=1Kexp⁡(vk′)=exp⁡(∑j=1Hβjkyj)∑k′=1Kexp⁡(∑j=0Hβjk′yj)=exp⁡{∑j=0Hβjk/(1+exp⁡(-∑i=0Iαijxi))}∑k′=1Kexp⁡{∑j=0Hβjk′/(1+exp⁡(-∑i=0Iαijxi))}.
The output oK to the Kth group can be calculated as oK=1-∑k=1K-1ok. For example, in the case of K=3, it follows that(8)o1=ev1ev1+ev2+ev3,o2=ev2ev1+ev2+ev3.
From o1+o2+o3=1, o3=1-(o1+o2). Thus the number of unit for output layer is 2 (= K-1).By setting the teach value(9)tk〈d〉={1:dthinputvectorx〈d〉=(x1〈d〉,x2〈d〉,…,xI〈d〉),is from thek-group0:o.w.
the log likelihood function for the total sample size D is(10)ln⁡L(θ;x,t)=∑d=1D∑k=1Ktk〈d〉ln⁡ok〈d〉,0≤ok〈d〉≤1,∑k=1Ktk〈d〉=1,
where t〈d〉=(t1〈d〉,t2〈d〉,…,tK〈d〉) and o〈d〉=(o1〈d〉,o2〈d〉,…,oK〈d〉) are the teach and output vectors, respectively, for the dth observation, t=(t〈1〉,t〈2〉,…,t〈D〉), x=(x〈1〉,x〈2〉,…,x〈D〉), α={αij}, and β={βjk}. As usual, the negative log likelihood gives the cross-entropy error function. The unknown parameters θ={α,β} can be estimated by maximizing the log likelihood ((10) with output (7)) by use of batch backpropagation including momentum, in which the training values for unknown parameters are chosen at random. The number of parameters included in the multiple-group neural discriminant model is p=H(I+K-1)+H+K-1.


#### 2.1.2. Determination of the Optimum Number of Hidden Units

The criterion based on bootstrapping is demonstrated to be favorable when selecting the optimum number of hidden units. In conventional statistics, various criteria have been developed for assessing the generalization performance.AIC provides us with a decision as to which of several competing network architectures are best for a given problem. However, the usage of AIC may not be justified theoretically when considering a neural network as an approximation to an underlying model [7, 24]. A bootstrap type nonparametric resampling estimator of Kullback-Leibler information by Ishiguro and Sakamoto [25], Konishi and Kitagawa [26], Ishiguro et al. [27], Kullback and Leibler [28], and Shibata [29] and Shao [30] can provide an alternative to AIC computed from a skewed discrete distribution.Let the training samplesX={X1,X2,…,Xd,…,XD}, Xd={x〈d〉,t〈d〉}, and x〈d〉={x1〈d〉,x2〈d〉,…,xI〈d〉} for d=1,2,…,D be independently distributed in an unknown distribution F. Let F̂ be the empirical distribution function that places a mass equal to 1/D at each point X1,X2,…,Xd,…,XD. We propose the bootstrap sampling algorithm given as follows.Step 1.
GenerateB samples X*, each of size D, drawn with replacement from the training sample X={X1,X2,…,Xd,…,XD}. Denote the bth sample as Xb*,b=1,2,…,B.Step 2.
For each bootstrap sampleXb*,b=1,2,…,B, fit a model to obtain the estimator θ̂(Xb*).Step 3.
The bootstrap estimator of biasC* is given as
(11)C*≅1B∑b=1B[ln⁡L{Xb*;θ̂(Xb*)}-ln⁡L{X;θ̂(Xb*)}],
where C* is the average of differences between log likelihood on the bootstrap sample lnL{Xb*;θ̂(Xb*)} and that on the training sample lnL{X;θ̂(Xb*)}, given θ̂(Xb*).
Thus, Extended Information Criterion (EIC) proposed by Ishiguro et al. [27] is defined as
(12)EIC=-2ln⁡L(X;θ̂(X))+2C*.EIC approach selects the number of hidden units with the minimum value of (12) as Shibata [29] and Shao [30] point out that this method is asymptotically equivalent to leaving-one-out CV and AIC.
Note that the bootstrap algorithm requires refitting of the model (retraining the network)B times [31]. The number of replications B is in the range 20≤B≤200, and so B=200 bootstrap replications are used. The competing networks share the same architecture with the only exception being the number of hidden units.


#### 2.1.3. Bootstrapping the Deviance

No standard procedure by which to assess the overall goodness-of-fit of the multiple-group neural discriminant model has been proposed. By introducing the maximum likelihood principle, the deviance allows us to test the overall goodness-of-fit of the model:(13)Dev=2[ln⁡Lf-ln⁡L(X;θ̂)]=2[∑d=1D∑k=1K{tk〈d〉ln⁡(tk〈d〉ôk〈d〉)}],
where lnL(X;θ̂) denotes the maximized log likelihood under a current neural discriminant model. Since the log likelihood for the full model lnLf=2∑d=1D∑k=1K{tk〈d〉lntk〈d〉} is zero by using the definition 0ln0=0, we have(14)Dev=-2∑d=1D∑k=1K{tk〈d〉ln⁡ôk〈d〉}.
Note that the deviance is two times log likelihood Equation (10). The greater the deviance, the poorer the fit of the model. However, the deviance given in (14) is not even approximately distributed as χ2 for the case in which binary (Bernoulli) responses are available [32–35]. We therefore provide the bootstrap estimator of the percentile (i.e., the critical point) for the deviance given in (14) according to the following algorithm.Step 1.
GenerateB (= 200) bootstrap samples X* drawn with the replacement from the training sample X with the optimum number of hidden units which was determined by the way in Section 2.1.2.Step 2.
For the bootstrap sampleXb*,b=1,2,…,B, the deviance given in (14) is computed as
(15)Dev(b)=-2ln⁡L(Xb*;θ̂(Xb*)).
This process is independently repeated B times, and the computed values are arranged in ascending order.Step 3.
The value of thejth order statistic Dev* of the B replications can be taken as an estimator of the quantile of order j/(B+1).Step 4.
The estimator of the100(1-α)-th percentile (i.e., the 100α% critical point) of Dev* is used to test the goodness-of-fit of the model using a specified significance level α=1-j/(B+1). If the value of the deviance given in (14) is greater than the estimate of the percentile, then the model fits poorly.


#### 2.1.4. Excess Error Estimation

Let error ratee(F;X) be the probability of incorrectly predicting the outcome of a new observation drawn from an unknown distribution F, given a prediction rule on a training sample X. This error rate is defined as the actual error rate, which is of interest in performance assessment of prediction rules. Let F̂ be the empirical distribution function that places a mass equal to 1/D at each point X1,X2,…,Xd,…,XD. We apply a prediction rule η to this training sample X and form the realized prediction rule ηF̂(x〈0〉) for a new observation X0={x〈0〉,t〈0〉}. Let Q(t〈0〉,ηF̂(x〈0〉)) indicate the discrepancy between an observed value t〈0〉 and its predicted value ηF̂(x〈0〉). Let error rate e(F̂;X), referred to as the apparent error rate, be the probability of incorrectly predicting the outcome for the sample drawn from the empirical distribution of the training sample, F̂. Because the training sample is used for both forming and assessing the prediction rule, this proportion (i.e., apparent error rate) underestimates the actual error rate. The difference e(F̂;X)-e(F;X) is the excess error. The expected excess error (i.e., bias) of a given prediction rule [22, 23, 36, 37] is(16)b(F)=E[e(F̂;X)-e(F;X)].When the prediction rule by multiple-group neural discriminant model is allowed to be complicated, overfitting becomes a real danger, and excess error estimation becomes important. Thus we will consider the bootstrapping to estimate the expected excess error when fitting a multiple-group neural discriminant model to the data. The algorithm can be summarized as follows.Step 1.
Generate bootstrap samplesX* from F̂ as described in Section 2.1.2. Let F̂* be the empirical distribution of X1*,X2*,…,Xd*,…,XD*.Step 2.
For each bootstrap sampleX*, fit a model to obtain the estimator θ̂(X*) and construct the realized prediction rule ηF̂* based on X*.Step 3.
The bootstrap estimator of the expected excess error in (16) is given by
(17)R*=1D∑d=1DQ(t〈d〉*,ηF̂*(x〈d〉*))-1D∑d=1DQ(t〈d〉,ηF̂*(x〈d〉)),
where
(18)Q(t〈d〉*,ηF̂*(x〈d〉*))={1:incorrect discriminant.0:o.w.Step 4.
Repeat Step1–Step 3 for bootstrap samples Xb*,b=1,2,…,B(= 200) to get Rb*. The bootstrap estimator of the expected excess error can be obtained as
(19)b(F̂)≅1B∑b=1BRb*.Step 5.
The actual error rate with bootstrap bias correction is(20)eboot=e(F̂;X)-b(F̂).


## 2.1. Multiple-Group Neural Discriminant Model

### 2.1.1. Statistical Inference

The functional representation of the neural network model is considered, as shown in Figure1. The connection weight between the ith unit in the input layer (i=0,…,I) and the jth unit in the hidden layer (j=1,…,H) is αij. Similarly, the weight between the jth unit in the hidden layer (j=0,…,H) and the kth unit in the output layer (k=1,…,K) is βjk. The input to the jth hidden unit is a linear projection of the input vector x=(x1,…,xI), that is,(1)uj=∑i=0Iαijxi,x0≡1,
where α0j is a bias. This is the same idea as incorporating the constant term in the design matrix of a regression by including a column of 1’s [1]. The output of the jth hidden unit is(2)yj=f(uj)=f(∑i=0Iαijxi),
where f(·) is a nonlinear activation function. The most commonly used activation function is the logistic (sigmoid) function:(3)yj=11+exp⁡(-uj).
The input to the kth output unit is(4)vk=∑j=0Hβjkyj,y0≡1,
where β0j is a bias. The activation function of network outputs for the mutually exclusive groups can be achieved using the softmax activation (normalized exponential) function:(5)ok=exp⁡(vk)∑k=1Kexp⁡(vk)=11+exp⁡(-Vk),(6)Vk=vk-ln⁡{∑k′≠kKexp⁡(vk′)},
which can be regarded as a multiclass generalization of logistic function.Figure 1
Single hidden layer neural network model.From (1)–(6), ok can be written in the form(7)ok=exp⁡(vk)∑k′=1Kexp⁡(vk′)=exp⁡(∑j=1Hβjkyj)∑k′=1Kexp⁡(∑j=0Hβjk′yj)=exp⁡{∑j=0Hβjk/(1+exp⁡(-∑i=0Iαijxi))}∑k′=1Kexp⁡{∑j=0Hβjk′/(1+exp⁡(-∑i=0Iαijxi))}.
The output oK to the Kth group can be calculated as oK=1-∑k=1K-1ok. For example, in the case of K=3, it follows that(8)o1=ev1ev1+ev2+ev3,o2=ev2ev1+ev2+ev3.
From o1+o2+o3=1, o3=1-(o1+o2). Thus the number of unit for output layer is 2 (= K-1).By setting the teach value(9)tk〈d〉={1:dthinputvectorx〈d〉=(x1〈d〉,x2〈d〉,…,xI〈d〉),is from thek-group0:o.w.
the log likelihood function for the total sample size D is(10)ln⁡L(θ;x,t)=∑d=1D∑k=1Ktk〈d〉ln⁡ok〈d〉,0≤ok〈d〉≤1,∑k=1Ktk〈d〉=1,
where t〈d〉=(t1〈d〉,t2〈d〉,…,tK〈d〉) and o〈d〉=(o1〈d〉,o2〈d〉,…,oK〈d〉) are the teach and output vectors, respectively, for the dth observation, t=(t〈1〉,t〈2〉,…,t〈D〉), x=(x〈1〉,x〈2〉,…,x〈D〉), α={αij}, and β={βjk}. As usual, the negative log likelihood gives the cross-entropy error function. The unknown parameters θ={α,β} can be estimated by maximizing the log likelihood ((10) with output (7)) by use of batch backpropagation including momentum, in which the training values for unknown parameters are chosen at random. The number of parameters included in the multiple-group neural discriminant model is p=H(I+K-1)+H+K-1.


### 2.1.2. Determination of the Optimum Number of Hidden Units

The criterion based on bootstrapping is demonstrated to be favorable when selecting the optimum number of hidden units. In conventional statistics, various criteria have been developed for assessing the generalization performance.AIC provides us with a decision as to which of several competing network architectures are best for a given problem. However, the usage of AIC may not be justified theoretically when considering a neural network as an approximation to an underlying model [7, 24]. A bootstrap type nonparametric resampling estimator of Kullback-Leibler information by Ishiguro and Sakamoto [25], Konishi and Kitagawa [26], Ishiguro et al. [27], Kullback and Leibler [28], and Shibata [29] and Shao [30] can provide an alternative to AIC computed from a skewed discrete distribution.Let the training samplesX={X1,X2,…,Xd,…,XD}, Xd={x〈d〉,t〈d〉}, and x〈d〉={x1〈d〉,x2〈d〉,…,xI〈d〉} for d=1,2,…,D be independently distributed in an unknown distribution F. Let F̂ be the empirical distribution function that places a mass equal to 1/D at each point X1,X2,…,Xd,…,XD. We propose the bootstrap sampling algorithm given as follows.Step 1.
GenerateB samples X*, each of size D, drawn with replacement from the training sample X={X1,X2,…,Xd,…,XD}. Denote the bth sample as Xb*,b=1,2,…,B.Step 2.
For each bootstrap sampleXb*,b=1,2,…,B, fit a model to obtain the estimator θ̂(Xb*).Step 3.
The bootstrap estimator of biasC* is given as
(11)C*≅1B∑b=1B[ln⁡L{Xb*;θ̂(Xb*)}-ln⁡L{X;θ̂(Xb*)}],
where C* is the average of differences between log likelihood on the bootstrap sample lnL{Xb*;θ̂(Xb*)} and that on the training sample lnL{X;θ̂(Xb*)}, given θ̂(Xb*).
Thus, Extended Information Criterion (EIC) proposed by Ishiguro et al. [27] is defined as
(12)EIC=-2ln⁡L(X;θ̂(X))+2C*.EIC approach selects the number of hidden units with the minimum value of (12) as Shibata [29] and Shao [30] point out that this method is asymptotically equivalent to leaving-one-out CV and AIC.
Note that the bootstrap algorithm requires refitting of the model (retraining the network)B times [31]. The number of replications B is in the range 20≤B≤200, and so B=200 bootstrap replications are used. The competing networks share the same architecture with the only exception being the number of hidden units.


### 2.1.3. Bootstrapping the Deviance

No standard procedure by which to assess the overall goodness-of-fit of the multiple-group neural discriminant model has been proposed. By introducing the maximum likelihood principle, the deviance allows us to test the overall goodness-of-fit of the model:(13)Dev=2[ln⁡Lf-ln⁡L(X;θ̂)]=2[∑d=1D∑k=1K{tk〈d〉ln⁡(tk〈d〉ôk〈d〉)}],
where lnL(X;θ̂) denotes the maximized log likelihood under a current neural discriminant model. Since the log likelihood for the full model lnLf=2∑d=1D∑k=1K{tk〈d〉lntk〈d〉} is zero by using the definition 0ln0=0, we have(14)Dev=-2∑d=1D∑k=1K{tk〈d〉ln⁡ôk〈d〉}.
Note that the deviance is two times log likelihood Equation (10). The greater the deviance, the poorer the fit of the model. However, the deviance given in (14) is not even approximately distributed as χ2 for the case in which binary (Bernoulli) responses are available [32–35]. We therefore provide the bootstrap estimator of the percentile (i.e., the critical point) for the deviance given in (14) according to the following algorithm.Step 1.
GenerateB (= 200) bootstrap samples X* drawn with the replacement from the training sample X with the optimum number of hidden units which was determined by the way in Section 2.1.2.Step 2.
For the bootstrap sampleXb*,b=1,2,…,B, the deviance given in (14) is computed as
(15)Dev(b)=-2ln⁡L(Xb*;θ̂(Xb*)).
This process is independently repeated B times, and the computed values are arranged in ascending order.Step 3.
The value of thejth order statistic Dev* of the B replications can be taken as an estimator of the quantile of order j/(B+1).Step 4.
The estimator of the100(1-α)-th percentile (i.e., the 100α% critical point) of Dev* is used to test the goodness-of-fit of the model using a specified significance level α=1-j/(B+1). If the value of the deviance given in (14) is greater than the estimate of the percentile, then the model fits poorly.


### 2.1.4. Excess Error Estimation

Let error ratee(F;X) be the probability of incorrectly predicting the outcome of a new observation drawn from an unknown distribution F, given a prediction rule on a training sample X. This error rate is defined as the actual error rate, which is of interest in performance assessment of prediction rules. Let F̂ be the empirical distribution function that places a mass equal to 1/D at each point X1,X2,…,Xd,…,XD. We apply a prediction rule η to this training sample X and form the realized prediction rule ηF̂(x〈0〉) for a new observation X0={x〈0〉,t〈0〉}. Let Q(t〈0〉,ηF̂(x〈0〉)) indicate the discrepancy between an observed value t〈0〉 and its predicted value ηF̂(x〈0〉). Let error rate e(F̂;X), referred to as the apparent error rate, be the probability of incorrectly predicting the outcome for the sample drawn from the empirical distribution of the training sample, F̂. Because the training sample is used for both forming and assessing the prediction rule, this proportion (i.e., apparent error rate) underestimates the actual error rate. The difference e(F̂;X)-e(F;X) is the excess error. The expected excess error (i.e., bias) of a given prediction rule [22, 23, 36, 37] is(16)b(F)=E[e(F̂;X)-e(F;X)].When the prediction rule by multiple-group neural discriminant model is allowed to be complicated, overfitting becomes a real danger, and excess error estimation becomes important. Thus we will consider the bootstrapping to estimate the expected excess error when fitting a multiple-group neural discriminant model to the data. The algorithm can be summarized as follows.Step 1.
Generate bootstrap samplesX* from F̂ as described in Section 2.1.2. Let F̂* be the empirical distribution of X1*,X2*,…,Xd*,…,XD*.Step 2.
For each bootstrap sampleX*, fit a model to obtain the estimator θ̂(X*) and construct the realized prediction rule ηF̂* based on X*.Step 3.
The bootstrap estimator of the expected excess error in (16) is given by
(17)R*=1D∑d=1DQ(t〈d〉*,ηF̂*(x〈d〉*))-1D∑d=1DQ(t〈d〉,ηF̂*(x〈d〉)),
where
(18)Q(t〈d〉*,ηF̂*(x〈d〉*))={1:incorrect discriminant.0:o.w.Step 4.
Repeat Step1–Step 3 for bootstrap samples Xb*,b=1,2,…,B(= 200) to get Rb*. The bootstrap estimator of the expected excess error can be obtained as
(19)b(F̂)≅1B∑b=1BRb*.Step 5.
The actual error rate with bootstrap bias correction is(20)eboot=e(F̂;X)-b(F̂).


## 2.1.1. Statistical Inference

The functional representation of the neural network model is considered, as shown in Figure1. The connection weight between the ith unit in the input layer (i=0,…,I) and the jth unit in the hidden layer (j=1,…,H) is αij. Similarly, the weight between the jth unit in the hidden layer (j=0,…,H) and the kth unit in the output layer (k=1,…,K) is βjk. The input to the jth hidden unit is a linear projection of the input vector x=(x1,…,xI), that is,(1)uj=∑i=0Iαijxi,x0≡1,
where α0j is a bias. This is the same idea as incorporating the constant term in the design matrix of a regression by including a column of 1’s [1]. The output of the jth hidden unit is(2)yj=f(uj)=f(∑i=0Iαijxi),
where f(·) is a nonlinear activation function. The most commonly used activation function is the logistic (sigmoid) function:(3)yj=11+exp⁡(-uj).
The input to the kth output unit is(4)vk=∑j=0Hβjkyj,y0≡1,
where β0j is a bias. The activation function of network outputs for the mutually exclusive groups can be achieved using the softmax activation (normalized exponential) function:(5)ok=exp⁡(vk)∑k=1Kexp⁡(vk)=11+exp⁡(-Vk),(6)Vk=vk-ln⁡{∑k′≠kKexp⁡(vk′)},
which can be regarded as a multiclass generalization of logistic function.Figure 1
Single hidden layer neural network model.From (1)–(6), ok can be written in the form(7)ok=exp⁡(vk)∑k′=1Kexp⁡(vk′)=exp⁡(∑j=1Hβjkyj)∑k′=1Kexp⁡(∑j=0Hβjk′yj)=exp⁡{∑j=0Hβjk/(1+exp⁡(-∑i=0Iαijxi))}∑k′=1Kexp⁡{∑j=0Hβjk′/(1+exp⁡(-∑i=0Iαijxi))}.
The output oK to the Kth group can be calculated as oK=1-∑k=1K-1ok. For example, in the case of K=3, it follows that(8)o1=ev1ev1+ev2+ev3,o2=ev2ev1+ev2+ev3.
From o1+o2+o3=1, o3=1-(o1+o2). Thus the number of unit for output layer is 2 (= K-1).By setting the teach value(9)tk〈d〉={1:dthinputvectorx〈d〉=(x1〈d〉,x2〈d〉,…,xI〈d〉),is from thek-group0:o.w.
the log likelihood function for the total sample size D is(10)ln⁡L(θ;x,t)=∑d=1D∑k=1Ktk〈d〉ln⁡ok〈d〉,0≤ok〈d〉≤1,∑k=1Ktk〈d〉=1,
where t〈d〉=(t1〈d〉,t2〈d〉,…,tK〈d〉) and o〈d〉=(o1〈d〉,o2〈d〉,…,oK〈d〉) are the teach and output vectors, respectively, for the dth observation, t=(t〈1〉,t〈2〉,…,t〈D〉), x=(x〈1〉,x〈2〉,…,x〈D〉), α={αij}, and β={βjk}. As usual, the negative log likelihood gives the cross-entropy error function. The unknown parameters θ={α,β} can be estimated by maximizing the log likelihood ((10) with output (7)) by use of batch backpropagation including momentum, in which the training values for unknown parameters are chosen at random. The number of parameters included in the multiple-group neural discriminant model is p=H(I+K-1)+H+K-1.


## 2.1.2. Determination of the Optimum Number of Hidden Units

The criterion based on bootstrapping is demonstrated to be favorable when selecting the optimum number of hidden units. In conventional statistics, various criteria have been developed for assessing the generalization performance.AIC provides us with a decision as to which of several competing network architectures are best for a given problem. However, the usage of AIC may not be justified theoretically when considering a neural network as an approximation to an underlying model [7, 24]. A bootstrap type nonparametric resampling estimator of Kullback-Leibler information by Ishiguro and Sakamoto [25], Konishi and Kitagawa [26], Ishiguro et al. [27], Kullback and Leibler [28], and Shibata [29] and Shao [30] can provide an alternative to AIC computed from a skewed discrete distribution.Let the training samplesX={X1,X2,…,Xd,…,XD}, Xd={x〈d〉,t〈d〉}, and x〈d〉={x1〈d〉,x2〈d〉,…,xI〈d〉} for d=1,2,…,D be independently distributed in an unknown distribution F. Let F̂ be the empirical distribution function that places a mass equal to 1/D at each point X1,X2,…,Xd,…,XD. We propose the bootstrap sampling algorithm given as follows.Step 1.
GenerateB samples X*, each of size D, drawn with replacement from the training sample X={X1,X2,…,Xd,…,XD}. Denote the bth sample as Xb*,b=1,2,…,B.Step 2.
For each bootstrap sampleXb*,b=1,2,…,B, fit a model to obtain the estimator θ̂(Xb*).Step 3.
The bootstrap estimator of biasC* is given as
(11)C*≅1B∑b=1B[ln⁡L{Xb*;θ̂(Xb*)}-ln⁡L{X;θ̂(Xb*)}],
where C* is the average of differences between log likelihood on the bootstrap sample lnL{Xb*;θ̂(Xb*)} and that on the training sample lnL{X;θ̂(Xb*)}, given θ̂(Xb*).
Thus, Extended Information Criterion (EIC) proposed by Ishiguro et al. [27] is defined as
(12)EIC=-2ln⁡L(X;θ̂(X))+2C*.EIC approach selects the number of hidden units with the minimum value of (12) as Shibata [29] and Shao [30] point out that this method is asymptotically equivalent to leaving-one-out CV and AIC.
Note that the bootstrap algorithm requires refitting of the model (retraining the network)B times [31]. The number of replications B is in the range 20≤B≤200, and so B=200 bootstrap replications are used. The competing networks share the same architecture with the only exception being the number of hidden units.


## 2.1.3. Bootstrapping the Deviance

No standard procedure by which to assess the overall goodness-of-fit of the multiple-group neural discriminant model has been proposed. By introducing the maximum likelihood principle, the deviance allows us to test the overall goodness-of-fit of the model:(13)Dev=2[ln⁡Lf-ln⁡L(X;θ̂)]=2[∑d=1D∑k=1K{tk〈d〉ln⁡(tk〈d〉ôk〈d〉)}],
where lnL(X;θ̂) denotes the maximized log likelihood under a current neural discriminant model. Since the log likelihood for the full model lnLf=2∑d=1D∑k=1K{tk〈d〉lntk〈d〉} is zero by using the definition 0ln0=0, we have(14)Dev=-2∑d=1D∑k=1K{tk〈d〉ln⁡ôk〈d〉}.
Note that the deviance is two times log likelihood Equation (10). The greater the deviance, the poorer the fit of the model. However, the deviance given in (14) is not even approximately distributed as χ2 for the case in which binary (Bernoulli) responses are available [32–35]. We therefore provide the bootstrap estimator of the percentile (i.e., the critical point) for the deviance given in (14) according to the following algorithm.Step 1.
GenerateB (= 200) bootstrap samples X* drawn with the replacement from the training sample X with the optimum number of hidden units which was determined by the way in Section 2.1.2.Step 2.
For the bootstrap sampleXb*,b=1,2,…,B, the deviance given in (14) is computed as
(15)Dev(b)=-2ln⁡L(Xb*;θ̂(Xb*)).
This process is independently repeated B times, and the computed values are arranged in ascending order.Step 3.
The value of thejth order statistic Dev* of the B replications can be taken as an estimator of the quantile of order j/(B+1).Step 4.
The estimator of the100(1-α)-th percentile (i.e., the 100α% critical point) of Dev* is used to test the goodness-of-fit of the model using a specified significance level α=1-j/(B+1). If the value of the deviance given in (14) is greater than the estimate of the percentile, then the model fits poorly.


## 2.1.4. Excess Error Estimation

Let error ratee(F;X) be the probability of incorrectly predicting the outcome of a new observation drawn from an unknown distribution F, given a prediction rule on a training sample X. This error rate is defined as the actual error rate, which is of interest in performance assessment of prediction rules. Let F̂ be the empirical distribution function that places a mass equal to 1/D at each point X1,X2,…,Xd,…,XD. We apply a prediction rule η to this training sample X and form the realized prediction rule ηF̂(x〈0〉) for a new observation X0={x〈0〉,t〈0〉}. Let Q(t〈0〉,ηF̂(x〈0〉)) indicate the discrepancy between an observed value t〈0〉 and its predicted value ηF̂(x〈0〉). Let error rate e(F̂;X), referred to as the apparent error rate, be the probability of incorrectly predicting the outcome for the sample drawn from the empirical distribution of the training sample, F̂. Because the training sample is used for both forming and assessing the prediction rule, this proportion (i.e., apparent error rate) underestimates the actual error rate. The difference e(F̂;X)-e(F;X) is the excess error. The expected excess error (i.e., bias) of a given prediction rule [22, 23, 36, 37] is(16)b(F)=E[e(F̂;X)-e(F;X)].When the prediction rule by multiple-group neural discriminant model is allowed to be complicated, overfitting becomes a real danger, and excess error estimation becomes important. Thus we will consider the bootstrapping to estimate the expected excess error when fitting a multiple-group neural discriminant model to the data. The algorithm can be summarized as follows.Step 1.
Generate bootstrap samplesX* from F̂ as described in Section 2.1.2. Let F̂* be the empirical distribution of X1*,X2*,…,Xd*,…,XD*.Step 2.
For each bootstrap sampleX*, fit a model to obtain the estimator θ̂(X*) and construct the realized prediction rule ηF̂* based on X*.Step 3.
The bootstrap estimator of the expected excess error in (16) is given by
(17)R*=1D∑d=1DQ(t〈d〉*,ηF̂*(x〈d〉*))-1D∑d=1DQ(t〈d〉,ηF̂*(x〈d〉)),
where
(18)Q(t〈d〉*,ηF̂*(x〈d〉*))={1:incorrect discriminant.0:o.w.Step 4.
Repeat Step1–Step 3 for bootstrap samples Xb*,b=1,2,…,B(= 200) to get Rb*. The bootstrap estimator of the expected excess error can be obtained as
(19)b(F̂)≅1B∑b=1BRb*.Step 5.
The actual error rate with bootstrap bias correction is(20)eboot=e(F̂;X)-b(F̂).


## 3. Simulation Study

Since the model generally does not encompass unknown functions, but rather only approximations thereof, the model is inherently misspecified. Therefore, we demonstrate results from some Monte Carlo simulations to evaluate the performance. The criterion based on bootstrapping is demonstrated to be favorable when selecting the optimum number of hidden units [38–41]. Vach et al. [41] investigated how regression functions can be approximate specific regression from the class(21)f(x)=β0+∑i=1Iβixi+∑i=1Iγixi2+∑i<i′δii′xixi′,
and pointed out that the comparison using members of this is a little bit unfair. We thus show the superiority of neural network model by using the function of the existence of several local extrema. The influence can be illustrated through a simple simulation using a neural network model with two inputs x1 and x2, because we can visualize the contour plot of unknown population.


### 3.1. Two-Class Classification

The influence can be illustrated through a simple simulation using a neural network model with two inputs, one output and a varying number of hidden units. For two independent continuous covariatesx1 and x2, we simulated the following known (true) model:(22)f(x1,x2)=11+exp⁡[-sin⁡(2πx1)-x1x2-sin⁡(2πx2)].Training and test samples of size 1000 were considered in the present study. Input data(x1,x2) are chosen from data that are uniformly distributed over [0,1]×[0,1], and the binary response y is labeled with 1 if f(x1,x2)>(1/2) and otherwise with 0. Figure 2 shows the distribution of the covariates (x1,x2) and the class membership indicator in the training sample.Contour plots: (a) darker grey scale levels represent lower probabilities ofy=0 and (b) • and ∘ show the class membership indicators y=0 and y=1, respectively, for the covariates (x1,x2).
(a)
Contour plot(b)
Class membership indicatorEIC values with B=200 replications based on bootstrapping pairs for the training sample are shown in Figure 3 after fitting the neural discriminant models having one to five hidden units. For the purpose of comparison, the values of AIC and BIC are also provided. In the case of the simulation study, the known (true) model given in (22) is included in the population. Thus, the differences between EIC and AIC values are slight.Figure 3
EIC, AIC, and BIC values for the simulation using only one training sample (note that the series of EIC and AIC are indistinguishable).Using the simulated training sample, the feed-forward neural networks were fit to the known (true) model given in (22). The tendency of mapping performed by neural networks with hidden units h=1,2,3 to implausibly fit the function given in (22) can also be illustrated.The bootstrap estimate of the95th percentile Dev* (i.e., the 5% critical point) for the training sample with four hidden units is Dev*=203.10. Comparison to the deviance given in (14) (Dev=39.40) suggests that the multiple-group neural discriminant model fits the data fairly well because Dev=39.40 is far from the 5% critical point Dev*=203.10.The actual error rate with the bootstrap bias correction given in (20) for the multiple-group neural discriminant models with four hidden units is calculated as eboot=0.009. Figure 4 illustrates the apparent error rates observed in the training sample- and test sample-based error rates. The apparent error rates for both samples decreased with the increase in the number of hidden units from h=1 to h=4 and then remained constant.Figure 4
Apparent error rates for simulated data after fitting neural networks with one to five hidden units.Figures3 and 4 are based on only one simulated data set. However, the efficacy of the bootstrap procedures would be more convincingly illustrated in a simulation study based on multiple samples. Figure 5 shows the average values of EIC, AIC, and BIC based on multiple samples with 100 replications after fitting the neural discriminant models having one to five hidden units. Figure 6 shows the box-and-whisker plots for EIC in order to evaluate the standard errors and other statistics. Figure 7 illustrates the mean apparent error rates observed in multiple test samples with 100 replicates. Figure 8 also shows the box-and-whisker plots for the mean apparent error rates in multiple test samples with 100 replicates. For the purpose of comparison, the estimates of the actual error rates with bootstrap bias correction for the training sample [42] are also shown in Figure 7. It is concluded that EIC identifies the optimal number of hidden units (i.e., 4) more often than AIC. In addition, the differences between the average values of EIC and AIC are somewhat similar to Figure 3, and the average values of the bootstrap-corrected estimate of the prediction error rate vary around the average apparent error rates for the multiple test samples.Figure 5
Average values of EIC, AIC and BIC for the simulated sample with 100 replications.Figure 6
Box-and-whisker plots for EIC.Figure 7
Error rates of multiple test samples with 100 replicates and actual error rate with bootstrap bias.Figure 8
Box-and-whisker plots for the mean apparent error rates in multiple test samples with 100 replicates.


### 3.2. Multiclass Classification

Input data(x1,x2) are generated from uniformly distributed over x=(x1,x2)∈[-1,1]×[-1,1]. By substituting x=(x1,x2) into(23)Prk=exp⁡{fk(x1,x2)}1+∑k=1K-1exp⁡{fk(x1,x2)},k=1,2,…,K-1x can be grouped into four classes in the case of K=4. As nonlinear function fk(x1,x2),(24)f1(x1,x2)=5sin⁡(2πx1)-3sin⁡(2πx2),f2(x1,x2)=sin⁡(2πx1)+2cos⁡(2πx2),f3(x1,x2)=2-8(x1-0.5)(x1-0.5)-8(x2-0.5)(x2-0.5)
can be used with ∑k=1KPrk=1 [41, 43].By use of definition oftk in (9), the observations can be divided into four class by multinomial random number(25)t~Multinorm(1,Pr){Class1:t=(1,0,0,0)Class2:t=(0,1,0,0)Class3:t=(0,0,1,0)Class4:t=(0,0,0,1)
with t=(t1,t2,t3,t4), Pr=(Pr1,Pr2,Pr3,Pr4).In this paper, training and test samples of size 400 were considered. The apparent error rates for training and test samples of several models are given in Table1. From Table 1, it is found that the apparent error rates of training and test sample for multiple-group neural discriminant model is the smallest.Table 1
Comparison of various discriminant methods for simulated data.
ModelApparent error rateTraining sampleTest sampleMultiple-group neural discriminant model (h = 5)0.2900.308Proportional odds model0.5500.543Linear discriminant model0.5450.525Quadratic discriminant model0.5180.500Tree-based model0.2900.345


## 3.1. Two-Class Classification

The influence can be illustrated through a simple simulation using a neural network model with two inputs, one output and a varying number of hidden units. For two independent continuous covariatesx1 and x2, we simulated the following known (true) model:(22)f(x1,x2)=11+exp⁡[-sin⁡(2πx1)-x1x2-sin⁡(2πx2)].Training and test samples of size 1000 were considered in the present study. Input data(x1,x2) are chosen from data that are uniformly distributed over [0,1]×[0,1], and the binary response y is labeled with 1 if f(x1,x2)>(1/2) and otherwise with 0. Figure 2 shows the distribution of the covariates (x1,x2) and the class membership indicator in the training sample.Contour plots: (a) darker grey scale levels represent lower probabilities ofy=0 and (b) • and ∘ show the class membership indicators y=0 and y=1, respectively, for the covariates (x1,x2).
(a)
Contour plot(b)
Class membership indicatorEIC values with B=200 replications based on bootstrapping pairs for the training sample are shown in Figure 3 after fitting the neural discriminant models having one to five hidden units. For the purpose of comparison, the values of AIC and BIC are also provided. In the case of the simulation study, the known (true) model given in (22) is included in the population. Thus, the differences between EIC and AIC values are slight.Figure 3
EIC, AIC, and BIC values for the simulation using only one training sample (note that the series of EIC and AIC are indistinguishable).Using the simulated training sample, the feed-forward neural networks were fit to the known (true) model given in (22). The tendency of mapping performed by neural networks with hidden units h=1,2,3 to implausibly fit the function given in (22) can also be illustrated.The bootstrap estimate of the95th percentile Dev* (i.e., the 5% critical point) for the training sample with four hidden units is Dev*=203.10. Comparison to the deviance given in (14) (Dev=39.40) suggests that the multiple-group neural discriminant model fits the data fairly well because Dev=39.40 is far from the 5% critical point Dev*=203.10.The actual error rate with the bootstrap bias correction given in (20) for the multiple-group neural discriminant models with four hidden units is calculated as eboot=0.009. Figure 4 illustrates the apparent error rates observed in the training sample- and test sample-based error rates. The apparent error rates for both samples decreased with the increase in the number of hidden units from h=1 to h=4 and then remained constant.Figure 4
Apparent error rates for simulated data after fitting neural networks with one to five hidden units.Figures3 and 4 are based on only one simulated data set. However, the efficacy of the bootstrap procedures would be more convincingly illustrated in a simulation study based on multiple samples. Figure 5 shows the average values of EIC, AIC, and BIC based on multiple samples with 100 replications after fitting the neural discriminant models having one to five hidden units. Figure 6 shows the box-and-whisker plots for EIC in order to evaluate the standard errors and other statistics. Figure 7 illustrates the mean apparent error rates observed in multiple test samples with 100 replicates. Figure 8 also shows the box-and-whisker plots for the mean apparent error rates in multiple test samples with 100 replicates. For the purpose of comparison, the estimates of the actual error rates with bootstrap bias correction for the training sample [42] are also shown in Figure 7. It is concluded that EIC identifies the optimal number of hidden units (i.e., 4) more often than AIC. In addition, the differences between the average values of EIC and AIC are somewhat similar to Figure 3, and the average values of the bootstrap-corrected estimate of the prediction error rate vary around the average apparent error rates for the multiple test samples.Figure 5
Average values of EIC, AIC and BIC for the simulated sample with 100 replications.Figure 6
Box-and-whisker plots for EIC.Figure 7
Error rates of multiple test samples with 100 replicates and actual error rate with bootstrap bias.Figure 8
Box-and-whisker plots for the mean apparent error rates in multiple test samples with 100 replicates.


## 3.2. Multiclass Classification

Input data(x1,x2) are generated from uniformly distributed over x=(x1,x2)∈[-1,1]×[-1,1]. By substituting x=(x1,x2) into(23)Prk=exp⁡{fk(x1,x2)}1+∑k=1K-1exp⁡{fk(x1,x2)},k=1,2,…,K-1x can be grouped into four classes in the case of K=4. As nonlinear function fk(x1,x2),(24)f1(x1,x2)=5sin⁡(2πx1)-3sin⁡(2πx2),f2(x1,x2)=sin⁡(2πx1)+2cos⁡(2πx2),f3(x1,x2)=2-8(x1-0.5)(x1-0.5)-8(x2-0.5)(x2-0.5)
can be used with ∑k=1KPrk=1 [41, 43].By use of definition oftk in (9), the observations can be divided into four class by multinomial random number(25)t~Multinorm(1,Pr){Class1:t=(1,0,0,0)Class2:t=(0,1,0,0)Class3:t=(0,0,1,0)Class4:t=(0,0,0,1)
with t=(t1,t2,t3,t4), Pr=(Pr1,Pr2,Pr3,Pr4).In this paper, training and test samples of size 400 were considered. The apparent error rates for training and test samples of several models are given in Table1. From Table 1, it is found that the apparent error rates of training and test sample for multiple-group neural discriminant model is the smallest.Table 1
Comparison of various discriminant methods for simulated data.
ModelApparent error rateTraining sampleTest sampleMultiple-group neural discriminant model (h = 5)0.2900.308Proportional odds model0.5500.543Linear discriminant model0.5450.525Quadratic discriminant model0.5180.500Tree-based model0.2900.345


## 4. Results and Discussion

Prediction accuracy (error rate) is the most important consideration in the development of prediction model. The assessment of goodness-of-fit is a useful exercise. In particular the goodness-of-fit and error rate from the training data are meaningful because of overfitting issue. The main purpose is to predict the future samples accurately. In other words, in real applications, the test sample population may be different from the training samples. A benchmark data set is thus used to illustrate the advantages of the models and methods developed herein. A multiple-group neural discriminant model having a single hidden layer was applied to a data set of 3,772 training instances and 3,428 testing instances of a thyroid disease database. All of these data sets are available on the World Wide Web athttp://archive.ics.uci.edu/ml/datasets/Thyroid+Disease. The present study considered three groups: hypothyroid, hyperthyroid, and normal. The laboratory profiles on which the differential diagnosis is made consist of 21 attributes (15 attributes are binary, and six attributes are continuous):x1: age, x2: sex, x3: on thyroxine, x4: query thyroxine, x5: on antithyroid, x6: sick, x7: pregnant, x8: thyroid surgery, x9: I131 treatment, x10: query hypothyroid, x11: query hyperthyroid, x12: lithium, x13: goitre, x14: tumour, x15: hypopituitary, x16: psych, x17: TSH, x18: T3, x19: TT4, x20: T4U, x21: FTI.Table2 is a list of the first-five observations for the 21 attributes and the group with respect to the training sample. The training sample is used to determine the neural network model structure. Table 3 is a list of the first-five observations for 21 attributes and the group with respect to the test sample. The goal of discrimination is to assign new observations to one of the mutually exclusive groups. The data in Tables 2 and 3 include six continuous and 15 binary attributes. Fisher’s discriminant model assumed that the inputs are normal distributed. However, it is worth noting that the posterior class probabilities for neural discriminant model can be given by maximizing log likelihood Equation (10) without the normal distributed assumption for inputs.Table 2
List of the first-five observations for 21 attributes and the group with respect to the training sample.
0.730100000100000000.000600.0150.1200.0820.14630.240000000000000000.000250.0300.1430.1330.10830.470000000000000000.001900.0240.1020.1310.07830.641000000000000000.000900.0170.0770.0900.08530.230000000000000000.000250.0260.1390.0900.1533Table 3
List of the first-five observations for 21 attributes and the group with respect to the test sample.
0.290000000000000000.006000.0280.1110.1310.08520.320000000000000000.001000.0190.0840.0780.10730.350000000000000000.000000.0310.2390.1000.23930.210000000000000000.001000.0180.0870.0880.09930.220000100000000000.000000.0220.1340.1350.0993A thyroid disease database has been used as a benchmark test for the neural network model shown in Figure1 with I=21 and K=3. EIC values are shown in Figure 9 after fitting the multiple-group neural discriminant models having one to four hidden units. In this case, the true model is not included in the population. For the purpose of comparison, AIC and BIC values are also provided.Figure 9
EIC, AIC and BIC values for the training sample of a thyroid disease database.Figure9 indicates that the minimum EIC value is obtained for the model having two hidden units, which has an apparent error rate e(F̂;X) of 0.0090. Figure 10 shows a histogram of the bootstrap replications Rb* that are used to estimate the expected excess error. The values of the mean and standard deviation of Rb* are −0.0033 and 0.0022, respectively. The actual error rate with the bootstrap bias correction given in (20) for the multiple-group neural discriminant models with two hidden units is calculated as eboot=0.012.Figure 10
Histogram of bootstrapped replicationsRb*.The histogram of the bootstrappedDev(b) for B=200 is provided in Figure 11. The bootstrap estimate of the 95th percentile Dev* (i.e., the 5% critical point) for the thyroid disease training sample with two hidden units is Dev*=344.41. Comparison to the deviance given in (14) (Dev=204.03) suggests that the multiple-group neural discriminant model fits the data fairly well. For reference, the Q-Q plot of the bootstrapped Dev(b) for B=200 is shown in Figure 12.Figure 11
Histogram of bootstrapped devianceDev(b) for B=200.Figure 12
Q-Q plot of bootstrapped deviance Dev(b) for B=200.Alternatively, if the deviance Equation (14) asymptotically follows the χ2 distribution with D-p=3772 degrees of freedom under the null hypothesis that the model is correct, the probability density function of the χ2 distribution with 3772 degrees of freedom is shown in Figure 13. However, because of large sample size D=3772, the distribution is extremely skewed. By comparing Figure 13 with Figure 11, it is found that the distribution of deviance Equation (14) can not be approximated by χ2 distribution. Furthermore, the mean and deviance of bootstrapped Dev(b) are E[Dev(b)]=205.55 and Var[Dev(b)]=4436.90, respectively, which are not close to those of the χ2 distribution with 3772d.f., that is, E[χ2]=d.f.=3772 and Var[χ2]=2×d.f.=7544. It should be noted that the deviance asymptotically follows χ2 distribution for grouped binary (i.e., binomial) response and a set of predictor variables, as described in Tsujitani and Aoki [44].Figure 13
Probability density function ofχ2 distribution on 3772d.f.The apparent error rates after fitting the multiple-group neural discriminant models having one to four hidden units are shown in Figure14. Figure 14 indicates that (i) the multilayer feedforward neural network can approximate virtually any function up to some desired level of approximation with the number of hidden units increased ad libitum for the training sample, (ii) the actual error rate for the test sample is the smallest when the number of hidden units is two, and (iii) a neural network with a large number of hidden units has a higher error rate for the test sample, because the noise is modeled in addition to the underlying function.Figure 14
Apparent error rates for the thyroid disease database after fitting neural networks with one to hidden units.Although the model fits the training sample as well as possible by increasing the number of hidden units, the model does not generalize very well to the test sample, which is the goal. The apparent error rates for training and test samples of several models are given in Table4: (i) the multigroup logistic discriminant model with linear effect [6, 45] by use of library{VGAM} in free software R [15], (ii) multiple-group logistic discrimination models with linear + quadratic effects, (iii) the tree-based model with mincut = 5, minsize = 10, mindev = 0.01 as tuning parameters [46] by use of library{rpart} in R, (iv) the nearest neighbor smoother using a nonparametric method to derive the classification criterion [6, 47] by use of library (knn) in R, (v) the kernel smoother [47] using normal distribution and a radius r=1.1 to specify a kernel density by use of library{ks} in R, (vi) the support vector machine using the “one-against-one” approach [48, 49] by use of library{e1071} in R, (vii) the proportional odds model [14], and (viii) VGAM based on the proportional odds model with optimum smoothing parameters selected by leaving-one-out cross-validation [20] by use of library{VGAM} in R.Table 4
Comparison of various discriminant methods for a thyroid disease database.
ModelApparent error rateTraining sampleTest sampleMultiple-group neural discriminant model (h = 2)0.0090.017Multiple-group logistic discrimination with linear effects of covariates0.0280.041Multiple-group logistic discrimination with linear + quadratic effects of covariates0.0190.032Tree-based model0.0020.0732-nearest neighbor0.0170.0743-nearest neighbor0.0380.089Kernel smoother0.0540.062SVM0.0450.052Proportional odds model0.0480.055VGAM0.0000.021From Table4, it is found that multiple-group neural discriminant model (h = 2) has the smallest error rate for test sample preserving relatively small error rate for training sample. In order to overcome the stringent assumption of the additive and purely linear effects of the covariates, multiple-group logistic discrimination models with linear and quadratic effects were included. The improvement obtained by the inclusion of the quadratic effect is slight. It should be noted that the apparent error rates for training of VGAM are the smallest, but that for test samples are large. This overfitting leads poor generalization. For example, the estimated smooth function of the covariate “age” for VGAM in Figure 15 shows the overfitting.Figure 15
the estimated smooth function of the covariate “age” for VGAM.Table5 is apartment house data for assessment of land value by the metropolitan area stations, of the metropolitan area stations with four-class classification [50]. By using the four covariates (average price of house built for sale, average house rent, yield, assessment of station value by the number of passengers getting on and off), and assessment of land value by the metropolitan area stations may be grouped into four categories:(i)
the most comfortable,(ii)
very comfortable,(iii)
a little comfortable,(iv)
not comfortable.Table 5
Apartment house data for assessment of land value by the metropolitan area stations.
Name of metropolitan area stationsAverage price of house built for saleAverage house rentYieldAssessment of station value by the number of passengers getting on and offGroupShinjyuku585025.35.288IKita-kashiwa28109.64.180IVMinami-kashiwa28909.94.180IIIAsakusa-bashi431019.35.484IIYokohama374018.45.9100IFigure16 indicates the values of EIC, AIC, and leaving-one-out CV (See the Appendix). The leaving-one-out CV is also included in order to assess the bootstrapping. The minimum EIC and leaving-one-out CV values are obtained for the model having two hidden units. However, the number of hidden unit with the minimum AIC value is three. The actual error rates in the case using EIC and leaving-one-out CV with two hidden units are 0.276 and 0.273, respectively. The bootstrapping is assessed from the point of leaving-one-out CV. The apparent error rates for training samples of several models are given in Table 6. From Table 6, it is found that multiple-group neural discriminant model (h = 2) has the smallest error rate.Table 6
Comparison of various discriminant methods for apartment house data.
ModelApparent error rateMultiple-group neural discriminant model (h = 2)0.262Multigroup logistic discrimination0.268Proportional odds model0.2942-nearest neighbor0.348Figure 16
EIC, AIC and (leaving-one-out) CV values for the training sample of apartment house data.


## 5. Conclusions

We discussed the learning algorithm by maximizing the log likelihood function. Statistical inference based on the likelihood approach for the multiple-group neural discriminant model was discussed, and a method for estimating bias on the expected log likelihood in order to determine the optimum number of hidden units was suggested. The key idea behind bootstrapping is to focus on the optimum tradeoff between the unbiased approximation of the underlying model and the loss in accuracy caused by increasing the number of hidden units. In the context of applying bootstrap methods to a multiple-group neural discriminant model, this paper considered three methods and performed experiments using two data sets to evaluate the methods. The three methods are bootstrap pairs sampling algorithm, goodness-of-fit statistical test, and excess error estimation algorithm.There are two broad limitations to our approach. First, the use of batch backpropagation algorithm including momentum prevents an maximum likelihood estimates from getting trapped in a local minimum, not global minimum. So far, our discussion of neural networks has focused on the maximum likelihood to determine the network parameters (weights and biases). However, a Bayesian neural network approach [51] might provide a more formal framework in which to incorporate a prior parameter distribution. Second, our neural network models assumed the independence of the predictor variables x=(x1,…,xI). More generally, it may be preferable to visualize interactions between predictor variables. The smoothing spline ANOVA models can provide an excellent means for data of mutually exclusive groups and a set of predictor variables [43, 52]. We expect that flexible methods for discriminant model using machine learning theory [47, 53–55] such as penalized smoothing splines and support vector machine [17–19] will be very useful in these real-world contexts.



---

*Source: 820364-2012-03-13.xml*