Law Discovery from Financial Data
using Neural Networks
Kazumi Saito, Naonori Ueda, Shigeru Katagiri
NTT Communication Science Laboratories{saito,ueda,katagiri}@cslab.kecl.ntt.co.jp
Yutaka Fukai, Hiroshi Fujimaru, Masayuki Fujinawa
Tokyo-Mitsubishi Asset Management Ltd.
{y-fukai,fujimaru,m-fujinawa}@tmam.co.jp
Abstract
This paper describes an experimental study for discovering underlying laws of market capitalization
using BS (Balance Sheet) items. For this purpose, we apply law discovery methods based onneural networks: RF5 (Rule Finder) discovers a single numeric law from data containing only
numeric values, RF6 discovers a set of nominally conditioned polynomials from data containing
both nominal and numeric values, and MCV regularizer is used to improve both the generalization
performance and the readability. Our preliminary experimental results show that these methods
are promising for discovering underlying laws from ﬁnancial data.

1

Introduction

The discovery of a numeric law, e.g., Kepler’s third
law T = kr3/2 , from data is the central part of scientiﬁc discovery systems. By using such methods, we
can expect to discover underlying laws in various typesof scientiﬁc and business data including ﬁnancial data.
As a traditional AI (Artiﬁcial Intelligence) approach,the BACON systems (Langley, 1978; Langley et al.,1987) and many variants (Langley and Zytkow, 1989;
Falkenhainer and Michalski, 1990; Nordhausen andLangley, 1993) have employed a recursive combinationof two variables, but these systems have suﬀered fromcombinatorial explosion problems, troublesome preparations of appropriate model functions, and a lack ofrobustness against unknown samples. As an alternative to these conventional methods, we have proposed
a connectionist approach that specially employs a generalized polynomial network structure referred to as a
Rule Finder (RF) (see (Saito and Nakano, 1990)).
Among several implementations of the RF methods,
we use two selections: RF5 (Saito and Nakano, 1997b)
and RF6 (Nakano and Saito, 1999). We are indeed
surrounded by a lot of numeric data, and RF5 is amethod developed to eﬀectively discover numeric laws
from such numeric data. In contrast, RF6 is a method
for coping with another type of data, which consistsof both numeric and nominal values that can be easily
observed in our daily life.
Moreover, data usually contain several irrelevant
variables. Since these variables only work to overﬁt the training data, the generalization performanceis generally degraded. Here the generalization means
the performance with new data. In order to discover
relevant weights of neural networks, we have proposeda new method, called MCV regularizer (Saito andNakano, 2000), to learn a distinct squared penalty factor for each weight as a minimization problem using
the cross-validation error. Since only the irrelevant
weights are severely penalized, and these values are
expected to be very small, the corresponding irrelevant variables will be negligible. In addition, thismakes the interpretation of the discovered laws easy.
This paper describes an experimental study for discovering laws of market capitalization using BS (Balance Sheet) items. Section 2 brieﬂy explains the law
discovery methods used in this study. Section 3 reports our experimental results.

2

Law Discovery Methods

2.1

RF5 algorithm

Let (x1 , · · · , xK , y) be a vector of variables describing each example, where xk is a numeric explanatory
variable and y is a numeric target variable. As a classof numeric formula y(x; Θ), we consider a generalized
polynomial, whose power values are not restricted tointegers, expressed byy(x; Θ) =

w0 +

J

j=1

=

w0 +

J

j=1

wj

K

k=1

wj exp

w

xk jk
K

k=1


wjk ln xk.(1)Here each parameter wj or wjk is an unknown real
number, and J is an unknown integer corresponding
to the number of terms. Θ is an M -dimensional parameter vector constructed by arranging parameters
wj , j = 0, · · · , J, and wjk , j = 1, · · · , J, k = 1, · · · , K.
Note that Eq. (1) can be regarded as a feedforward
computation of a three-layer neural network (Saito
and Nakano, 1997b).
Let D = {(xµ , y µ ), µ = 1, · · · , N } be a set of training data, where N is the number of examples. Here
we assume that each training example (xµ , y µ ) is independent and identically distributed. Now, our ultimate goal in discovering laws is deﬁned as a problemof minimizing the generalization error, that is, ﬁnding
the optimal estimator Θ∗ (D) that minimizesG(Θ∗ )

2= ED ET (y ν − y(xν ; Θ∗ (D))) ,(2)

where T = (xν , y ν ) denotes test data independent of
the training data D.
The least-squares estimate of Θ∗ , denoted by Θ,
minimizes the error sum of squaresN

E1 (Θ) =1 µ2(y − y(xµ ; Θ)) .
2 µ=1

k=1 l=1(r)

where V (r) denotes a vector of parameters vkl . By
setting the adequate values of V (r) , we can see thatthe following formula can closely approximate the ﬁnal
output value from a set of multiple nominally conditioned polynomials, deﬁned by Eq. (5).
F (q, x; Ψ) =RF6 algorithm

Let (q1 , · · · , qK1 , x1 , · · · , xK2 , y) be a vector of variables describing an example, where qk is a nominal explanatory variable. For each qk we introduce a dummy
variable qkl deﬁned by
1
if qk matches the l-th category,
qkl =
(4)
0
otherwise,
where l = 1, · · · , Lk , and Lk is the number of distinct
categories appearing in qk . As a true model governing
data, we consider the following set of multiple rules
if
qkl = 1 then y = y(x; Θ(1) ),





qkl ∈Q(1)
···
···
···
· · · , (5)


(R)

qkl = 1 then y = y(x; Θ ) ifqkl ∈Q(R)where Q(r) denotes a set of dummy variables corresponding to the r-th nominal condition and R is thenumber of rules.

Rg(q; V (r) ) y(x; Θ(r) ).

(7)

r=1

Here Ψ consists of all parameters V (r) , Θ(r) , r =1, · · · , R. With an adequate number J, the following
can completely describe Eq. (7),(8)
y(q, x; Θ) = w0 +
K L

K
J
1
2
k



wj expvjkl qkl +wjk ln xk .
j=1

(3)

In order to eﬃciently and constantly obtain goodlearning results, we employ a second-order learningalgorithm called BPQ (Saito and Nakano, 1997a). By
adopting the quasi-Newton method (Gill et al., 1981;
Luenberger, 1984) as a basic framework, we calculate
the descent direction on the basis of a partial BFGS
update and then eﬃciently calculate a reasonably accurate step-length as the minimal point of a secondorder approximation.

2.2

We introduce a framework to learn nominal conditions from data.

K L
1 
k

(r)
(r)
(6)
vkl qkl ,g(q; V ) = exp

k=1 l=1k=1

Eq. (8) can also be regarded as the feedforward computation of a three-layer neural network.

2.3

MCV regularizer

To improve both the generalization performance andthe readability, we consider a distinct penalty factor for each weight. Hereafter, an M -dimensional
vector λ is deﬁned by λ = (λ1 , · · · , λM )T , andan M -dimensional diagonal matrix Λ is deﬁned bydiag(Λ) = (exp(λ1 ), · · · , exp(λM )), where aT denotes
a transposed vector of a. Here, since the penalty factor must be non-negative, we adopted exp(λm ), instead of a standard parameterization λm . As a result,the discovery of laws subject to Eq. (1) can be deﬁned
as the following learning problem in neural networks.
E(Θ)

1= E1 (Θ) + ΘT ΛΘ.
2(9)In our own experiments (Saito and Nakano, 1997c),
the combination of the squared penalty term and the
BPQ algorithm drastically improves the convergence
performance in comparison to the other combinations,while at the same time bringing about excellent generalization performance.
We introduce an objective function for penalty factors derived from the procedure of cross-validation,
which is frequently used to evaluate the generalization
performance of learned networks (Bishop, 1995). The
procedure of cross-validation divides the data D at
random into S distinct segments (Gs , s = 1, · · · , S);
it uses S − 1 segments for training, and uses the remaining one for the test. This process is repeatedTable 1: target and explanatory variables
names meaningsymarket capitalization
x1
current assets
property and equipment
x2
x3
total assets
x4
current liabilities
long-term debt
x5
x6
total liabilities

Table 2: RF5+MCV results
M SE
M SECV
avg
std
avg
std
J = 1 0.2878 0.0031 0.3090 3.0814
J = 2 0.2350 0.0029 0.2437 2.7761
J = 3 0.2314 0.0026 0.2516 2.6004

S times by changing the remaining segment, and thegeneralization performance is evaluated by using thefollowing MSE (mean squared error) over all S test
results.
S

M SECV

=

1  
N s=1

 s)
y ν − y(xν ; Θ

2

.(10)

ν∈Gs

 s denotes the optimal weights obtained by minHere Θ
imizing the following objective function for weights
Es (Θs ) =1  µ1(y −y(xµ ; Θs ))2 + ΘTs ΛΘs .(11)
2
2
µ∈Gs

The extreme case of S = N is known as the leaveone-out method, which is often used for a small data
size. Note that Eq. (10) is regarded as a reasonableapproximation to Eq. (2) for a given data set D.
According to the implicit function theorem, since s can be regarded as a vector consisting of implicit
Θfunctions of λ, Eq. (10) can be deﬁned as the objectivefunction for penalty factors. Thus, we can calculate that minimizes Eq. (10). Then, with λ,
 we
the λ
 that minimizes Eq. (9). Finally, Θ iscalculate the Θadopted as the ﬁnal weight vector of the discoveredlaw (Saito and Nakano, 2000).

3
3.1

3.3

RF5+MCV result

The number of hidden units was changed from 1 to 3
(J = 1, 2, 3). Table 2 shows the performance of thelearning results. The M SE values were minimized
when J = 3, while the M SECV values were minimized
when J = 2. This indicates the suitable number ofhidden units is considered to be 2. When J = 2, anexample of the discovered laws was as follows:y=

Law Discovery from Financial Data

Experimental settings

Before the analysis, the following scaling was applied to the variables: y = (y − avg(y))/std(y), andln xk = (ln xk − avg(ln xk ))/std(ln xk ), k = 1, · · · , 6.

−25055.171654
x+0.013135
x+0.967862+6.983186x+0.000008
12
3

x+0.000000
x−0.000001
x−0.000001
456−8.205773x+0.000005
x+0.000005
x+0.592110
1
2
3

Experimental data

Our experiments used 953 companies listed on the ﬁrst
section of the Tokyo Stock Exchange, where banks,and insurance, securities, and recently listed companies were excluded. The market capitalization of eachcompany was calculated by multiplying the shares ofoutstanding stocks with the stock price at the end of
October, 1999. As the ﬁrst step in this study, we selected six fundamental items from all of the BS items,
and the values of these items were also calculated at
the end of October, 1999. Table 1 summarizes target
and explanatory variables.

3.2

When a distinct squared penalty factor is adopted foreach weight, this penalty term is known to be consistent with any linear scaling of input and/or output
variables. Here consistency requires that we obtainequivalent networks that diﬀer only by the scaling of
the weights as given (Bishop, 1995).
In the experiments, the initial values for the weights
wjk were independently generated according to a normal distribution with a mean of 0 and a standard deviation of 1; the initial values for the weights wj were set
to 0. The initial values for the penalty factors λ were
set to 0, i.e., Λ was set to the identical matrix. The iteration was terminated when the gradient vector was
suﬃciently small, i.e., maxm {∂/∂θm E(Θ)} < 10−6
for learning over Θ; maxm {∂/∂λm M SECV (λ)} <10−6 for learning over λ. The cross-validation error
was calculated by using the leave-one-out method, i.e.,
S = N.

x+0.000002
x−0.006089
x+0.378892
,
4
5
6

(12)

where the weight values were rounded oﬀ to the seventh decimal. These weight values were transformed
to correspond to the original scale of variables. When
the weight values were rounded oﬀ to the second decimal, the discovered law was as follows:y=0.6 +0.4. (13)−25055.2 + 7.0x1.0
3 − 8.2x3 x6

Note that existing numeric discovery methods cannot ﬁnd such laws, as described in Eq. (13), without
preparing some appropriate prototype functions. This
point is an important advantage in RF5 over existing
methods.

3.4

RF6+MCV result

Since laws of market capitalization can diﬀer according to the type of industry, the 33 classiﬁcations of theTokyo Stock Exchange were used as a nominal variable. In this experiment, to understand the eﬀect of
the nominal variable intuitively, we ﬁxed the numberof hidden units at 1. An example of the discoveredlaws was as follows:y= 12944.3 + 1.9 exp(−1.4q1 − 1.3q2 − 1.0q3−0.4q4 − 0.6q5 − 0.7q6 − 0.5q7 + 0.2q8−1.0q9 − 0.6q11 − 0.9q12 − 0.3q13 − 1.6q20−0.8q21 − 0.7q22 − 0.9q23 − 1.2q24 − 1.1q26−1.0q31 − 0.9q32 )x+0.7
x+1.0
x−0.7
,
(14)
2
3
6

where the above qk denotes the k-th type of industry,
and the weight values are rounded oﬀ to the second
decimal. Here the M SE and M SECV values of the
discovered law were 0.2365 and 0.2668, respectively.
This formula can be easily decomposed into a set ofrules. Since the second term on the right hand sideof Eq. (14) is always positive, a weight for qk indicates the price setting tendency of the k-th type ofindustry for similar BS situations. More speciﬁcally,
Eq. (14) tells us that the pharmaceutical industry (q8 )is likely to have a high setting due to the largest weight(0.2), while the electric power or gas industry (q20 ) is
likely to have a low setting due to the smallest weight(−1.6).

4

Conclusion

This paper has reported an experimental study fordiscovering underlying laws of market capitalization
using BS items. Namely, both a single numeric law
and a set of nominally conditioned polynomials werediscovered by using RF5 and RF6 algorithms. Our
preliminary experimental results showed that thesemethods are promising for discovering laws from ﬁnancial data. Currently, we are investigating possible
interpretations of the laws discovered in experiments.

References
Bishop, C. (1995). Neural networks for pattern recognition. Oxford Press.
Falkenhainer, B. and Michalski, R. (1990). Integrating quantitative and qualitative discovery in the
ABACUS system. In Kodratoﬀ, Y. and Michalski, R., editors, Machine learning: an artificial
intelligence approach, Volume III, pages 153–190.
Morgan Kaufmann, San Mateo, CA.
Gill, P., Murray, W., and Wright, M. (1981). Practicaloptimization. Academic Press, London.
Langley, P. (1978). Bacon.1: a general discovery system. In Proc. 2nd National Conferenee of theCanadian Soeiety for Computational Studies ofIntelligenee, pages 173–180.
Langley, P., Simon, H., Bradshaw, G., and Zytkow, J.
(1987). Scientific discovery: computational explorations of the creative process. MIT Press, Cambridge, MA.
Langley, P. and Zytkow, J. (1989). Data-driven approaches to empirical discovery. Artificial Intelligence, 40:283–312.
Luenberger, D. (1984). Linear and nonlinear programming. Addison-Wesley, Reading, MA.
Nakano, R. and Saito, K. (1999). Discovery of a setof nominally conditioned polynomial. In Proc.
2nd Int. Conference on Discovery Science, LNAI
1721, pages 287–298, Tokyo, Japan.
Nordhausen, B. and Langley, P. (1993). An integrated framework for empirical discovery. Machine Learning, 12(1/2/3):17–48.
Saito, K. and Nakano, R. (1990). Rule extraction fromfacts and neural networks. In Proc. Int. Neural Network Conference, pages 379–382, Paris,
France.
Saito, K. and Nakano, R. (1997a). BFGS update andeﬃcient step-length calculation for three-layer
neural networks. Neural Computation, 9(1):123–
141.
Saito, K. and Nakano, R. (1997b). Law discovery
using neural networks. In Proc. 15th Int. Joint
Conference on Artificial Intelligence, pages 1078–
1083, Nagoya, Japan.
Saito, K. and Nakano, R. (1997c). Second-order learning algorithm with squared penalty term. In Advances in Neural Information Processing Systems
9, pages 627–633, Denver, CO.
Saito, K. and Nakano, R. (2000). Law discovery usingneural networks. In Proc. 4th Pacific-Asia Conference on Knowledge Discovery and Data Mining, Kyoto, Japan (to appear).