Discovery of Relevant Weights
by Minimizing Cross-validation Error
Kazumi Saito1 and Ryohei Nakano2
1NTT Communication Science Laboratories
2-4 Hikaridai, Seika, Soraku, Kyoto 619-0237 Japansaito@cslab.kecl.ntt.co.jp
2Nagoya Institute of Technology
Gokiso-cho, Showa-ku, Nagoya 466-8555 Japannakano@ics.nitech.ac.jp

Abstract. In order to discover relevant weights of neural networks, thispaper proposes a novel method to learn a distinct squared penalty factor
for each weight as a minimization problem over the cross-validation error.
Experiments showed that the proposed method works well in discoveringa polynomial-type law even from data containing irrelevant variables anda small amount of noise.

1Introduction

Neural networks can be utilized as a core technique in some KDD (Knowledge
Discovery and Data mining) applications such as scientiﬁc discovery [2, 1]. One
important research subject of neural networks is to improve the generalization
performance. Here the generalization means the performance on new data. It is
widely known that adding some penalty term to a standard training error term
can lead to signiﬁcant improvements in network generalization. As for squaredpenalty, a single penalty factor is often conveniently used. If we can develop
a method that automatically adjusts a distinct penalty factor for each weight,
several advantages can be expected, i.e., the generalization performance will bestill more improved; the readability of discovered laws will be improved; sucha squared penalty term is consistent with any linear scaling of variables; andsuitable penalty factors can be determined without inaccurate estimation.

2Optimal Penalty Factor Calculation

Let (x1 , · · · , xK , y) be a vector of variables describing each example, where xk
is a numeric or nominal explanatory variable and y is a numeric target variable. Here we assume that each nominal explanatory variable is described as adummy variable. As a class of numeric formula y(x; Θ), we consider a generalized
polynomial expressed by
K

JJK




wjky(x; Θ) = w0 +wjxk
= w0 +
wj expwjk ln xk , (1)
j=1

k=1

j=1

k=1

where each parameter wj or wjk is an unknown real number, and J is an unknown integer corresponding to the number of terms. Θ is an M -dimensionalparameter vector constructed by arranging parameters wj , j = 0, · · · , J, andwjk , j = 1, · · · , J, k = 1, · · · , K. Let D = {(xµ , y µ ), µ = 1, · · · , N } be a set oftraining examples, where N is the number of examples. Here we assume thateach training example (xµ , y µ ) is independent and identically distributed. Now,our ultimate goal of the law discovery is deﬁned as a problem of minimizing thegeneralization error, that is, to ﬁnd the optimal estimator Θ ∗ that minimizes
2G(Θ ∗ ) = ED ET (y ν − y(xν ; Θ ∗ (D))) ,ν(2)νwhere T = (x , y ) denotes test data independent of the training data D. The minimizes the error sum of squares
least-squares estimate of Θ ∗ , denoted by Θ,N

E1 (Θ) =1 µ(y − y(xµ ; Θ))2 .
2 µ=1(3)However, this estimation is likely to over-ﬁt to the training data; thus, we cannotusually obtain good results in terms of the generalization performance.
As we have already mentioned, it is widely known that adding some penalty
term to Eq. (3) can lead to signiﬁcant improvements in network generalization.
Here a simple penalized target function using a single factor is given as below.
E2 (Θ) = E1 (Θ) +M
12exp(λ)θm,
2m=1(4)

where exp(λ) is a penalty factor and θm ∈ Θ. Here since the penalty factor must
be non-negative, we adopted exp(λ), instead of a standard parameterization λ.
To improve both the generalization performance and the readability, we consider a distinct penalty factor for each weight. Let λ be an M -dimensional vector(λ1 , · · · , λM )T , and Λ be an M -dimensional diagonal matrix whose diagonal elements are deﬁned by Λmm = exp(λm ) for m = 1, · · · , M , where aT denotes atransposed vector of a. Then, the discovery of laws subject to Eq. (1) can be deﬁned as the following learning problem in neural networks. That is, the problem
is to ﬁnd the Θ that minimizes the following objective function for weights
1(5)E(Θ) = E1 (Θ) + ΘT ΛΘ.
2Now, we introduce an objective function for penalty factors derived from theprocedure of cross-validation, and propose MCV (Minimum Cross-Validation)
regularizer. The procedure of cross-validation divides the data D at random into
S distinct segments (Gs , s = 1, · · · , S), and uses S −1 segments for training, anduses the remaining one for the test. This process is repeated S times by changing
the remaining segment, and the generalization performance is evaluated by using
the following MSE (mean squared error) over all S test results.
S21    ν s) .
y − y(xν ; ΘM SECV =N s=1
ν∈Gs(6) s denotes the optimal weights obtained by minimizing the following obHere Θjective function for weights
Es (Θ s ) =1  µ1(y − y(xµ ; Θs ))2 + Θ Ts ΛΘ s .
22(7)

µ∈Gs

The extreme case of S = N is known as the leave-one-out method, which isoften used for a small size of data. Note that Eq. (6) is regarded as a reasonableapproximation to Eq. (2) for a given data set D. According to the implicit
 s can be regarded as a vector consisting of implicit
function theorem, since Θ
functions of λ, Eq. (6) can be deﬁned as the objective function for penalty
 which minimizes Eq. (6). Then, by using λ,factors. Thus, we can calculate λ which minimizes Eq. (5). Finally, Θ is adopted as the ﬁnal
we can calculate Θweight vector of the discovered law.

3

Evaluation by ExperimentsWe consider an artiﬁcial law (function) described by−0.02+0.02+ 4x−1
y = 2 + 3x+1
1 x2
3 x4(8)where we have 9 numeric explanatory variables. Clearly, variables x5 , · · · , x9 areirrelevant to Eq. (8). Each example is generated as follows: each value of numericvariables x1 , · · · , x9 is randomly generated in the range of (0, 1), and we get thecorresponding value of y by calculating Eq. (8) and adding Gaussian noise with amean of 0 and a standard deviation of 0.1. The number of examples is set to 200(N = 200). Before the analysis, the following scaling was applied to the variables:y = (y − mean(y))/std(y), and ln x
k = ln xk − mean(ln xk ), k = 1, · · · , 9.
In the experiments, the initial values for the weights wjk were independently generated according to a normal distribution with a mean of 0 and astandard deviation of 1; the initial values for the weights wj were set to 0.
The initial values for the penalty factors λ were set to 0, i.e., Λ was set tothe identical matrix. The iteration was terminated when the gradient vector
was suﬃciently small, i.e., maxm {∂/∂θm E(Θ)} < 10−6 for learning over Θ;maxm {∂/∂λm M SECV (λ)} < 10−6 for learning over λ.
MCV regularizer was compared with two conventional methods, no-penalty
method and single-factor method, where the objective functions of these conventional methods are Eq. (3) and Eq. (4), respectively. Figure 1(a) shows thelearning results of these three methods, where the RMSE (root mean squarederror) was used for evaluation; the number of hidden units J was ﬁxed at thecorrect number 2; the cross-validation error was calculated by using the leaveone-out method, i.e., S = N ; and the generalization performance was measured
by using a set of noise-free 10, 000 test examples generated independently to thetraining examples. This ﬁgure shows that the RMSE for the training data wasalmost the same for each method; both the RMSE for the cross-validation andpenalty factor

RMSE (Root Mean Squared Error)
crossvalidation
trainig
error
error

4100.120.1
0.08

0.06
0.04
0.02
0210

testerror

1−210−410

no penalty

single factor

MCV regularizer(a) performance of three methods

Hidden unit 1Hidden unit 2(b) learning result of penalty factors

Fig. 1. Experimental results of artificial data

the RMSE for the test data were clearly decreased by using MCV regularizer;
and the performance of the single factor method was almost comparable to thoseof the no penalty method.
An example of the laws discovered by the no penalty method was as follows:y = 2.0306
x−0.0203
x+0.0035
x−0.0029
x+0.0073
x+0.0056
x+0.0010
x+0.0022x−0.0036+2.9791x+1.0024
12345678
9+3.9993x+0.0008
x+0.0004x−1.0003
x+0.0201x−0.0005
x−0.0011
x−0.0002
x−0.0002
x+0.0011
12
3
4
5
6
789
where the weight values were rounded oﬀ to the fourth decimal place. Note thatthese weight values were transformed so as to correspond to the original scaleof variables. Although a law almost equivalent to the true one was found, it isdiﬃcult to select only the relevant weights from this result. While an exampleof the laws discovered by MCV regularizer was as follows:y = 2.0118
x−0.0190
x−0.0000x−0.0000x+0.0019
x+0.0007
x+0.0000x+0.0000x+0.0000+2.9792x+0.9941
12
3456
7
8
9+0.0001 −0.9999 +0.0197 −0.0000 −0.0006 −0.0000 −0.0000 +0.0003+3.9987x+0.0000
xxxxxxxx
1
2
345
6
78
9
Clearly, the irrelevant weight values were greatly suppressed.
Figure 1(b) shows the learning result of the penalty factors. This ﬁgure indicates that only the penalty factors for the relevant weights became small enough,i.e., we can easily select only the relevant weights. Therefore, it was shown that
the MCV regularizer simultaneously improves the generalization performanceand readability, without care of variable scaling and a candidate determination
for the penalty factors.

References
1. R. Nakano and K. Saito. Discovery of a set of nominally conditioned polynomials.
In Proc. 2nd Int. Conf. on Discovery Science, LNAI 1721, pages 287–298, 1999.
2. K. Saito and R. Nakano. Law discovery using neural networks. In Proc. 15th Int.
Joint Conf. on Artificial Intelligence, pages 1078–1083, 1997.