Abstract:
A generator in a back propagation type neural network having an input layer, an output layer and an intermediate layer coupled the input and output layers forms initial values for connection parameters. A first generator produces an initial value W10 of a weight coefficient of each connection parameter of the intermediate layer from in-class covariant data SW and inter-class co-variant data SB over data inputted to the input layer. The produced values are set into respective synapses of the intermediate layer as connection parameters.

Description:
This application is a continuation-in-part of application Ser. No. 08/171,980 filed Dec. 23, 1993, now abandoned, which is a continuation of application Ser. No. 08/004,680 filed Jan. 14, 1993, abandoned, which is a continuation of application Ser. No. 07/526,650, filed May 22, 1990, abandoned. 
    
    
     BACKGROUND OF THE INVENTION 
     The present invention relates to a neural network which can be used, for example, for character recognition and, in particular, to a method of assigning initial values for learning the connection parameters of a neural network by utilizing statistical information on input data. 
     For example, as shown in FIG. 1, in a back propagation type neural network, synapse elements of an input layer, an intermediate layer and an output layer are generally connected in a network form. The connection between synapse elements is represented by synapse parameters (or a weight coefficient W and a bias θ). Adjusting these parameters to suitable values is called learning. If the network is used for character recognition, learning is achieved using learning data for character recognition. If it is used for picture image processing, learning is achieved using learning data for picture image processing; if such learning is sufficient to determine parameters appropriately and synapse parameters are set to these determined values, thereafter, this neural network performs character recognition, picture image processing and the like in accordance with the learning. 
     In a neural network, therefore, setting the above-mentioned synapse parameters suitably, i.e., setting efficiently and suitably, is very important. 
     In learning of a neural network, a back propagation method is generally performed. In the learning of a weight coefficient W and a bias θ of the synapse elements of this back propagation type neural network, regarding the weight coefficient W and bias θ, correction is made to the weight coefficient W (n) and bias θ(n) beginning with initial values W(O) and θ(O) in the learning process. However, there are at present no theoretical grounds for preferring any particular set of initial values as the starting point for the learning of the weight coefficient W and bias θ. Under the existing circumstances, a small random number is generally used for each element as the initial value. 
     The problems regarding a weight coefficient that arise in a conventional learning method will be explained in detail. 
     In a back propagation algorithm, a weight coefficient w ij  between two synapse elements i, j is successively corrected by using a correction amount .increment.w ij  . That is, 
     
         w.sub.ij (n+1)=w.sub.ij (n)+.increment.w.sub.ij            ( 1) 
    
     In the above equation, i denotes elements of the input side; j denotes elements of the output side. The correction amount .increment.w ij  is calculated from the following equation. ##EQU1## where η is a positive constant and E is an error of the entire network given by the following equation. ##EQU2## where t j  is a teacher signal, and y j  is the output signal of an output-side synapse element. ∂g/∂w ij  is calculated using outputs of each layer, but the explanation of a concrete calculation method is omitted. 
     The correction of a weight coefficient W ij  (this step is called learning) is performed as described above and it is expected that the weight coefficient will converge on a suitable value through repetition of the learning steps. At this time, to make the value converge on a suitable value quickly, it is important to start with the right initial value w ij  (O). 
     The learning of bias θj for an element j is performed by means of a learning process similarly to w ij  such that the bias θj is regarded as a weight coefficient for input data that may be assigned a &#34;1&#34; at any time as a value. 
     As described above, since a back propagation type neural network is dependent on the initial values mentioned above regarding a weight coefficient and a bias in the learning process, the above-mentioned conventional technology, in which random numbers are used as initial values for a weight coefficient W={w i ,j } and a bias θ═{θ j  }, entails the possibility that (1) the learning time will be inordinately long and (2) a parameter will fall into a minimum value which is not optimum in the midway of learning and this minimum value will be erroneously taken as a suitable value. 
     SUMMARY OF THE INVENTION 
     According to the present invention, in a back propagation type neural network having the three layers --an input layer, an intermediate layer and an output layer, a calculation means is provided for calculating a weight coefficient matrix of an intermediate layer from a in-class covariant matrix and a inter-class covariant matrix of input data of an input layer and initial values of a weight coefficient matrix of an intermediate layer are determined from the input data. 
     According to the present invention, in a back propagation type neural network having the three layers--an input layer, an intermediate layer, and an output layer, a calculation means is provided for calculating a vector representing the bias of an intermediate layer from a total average vector of input data of an input layer and a weight coefficient matrix of an intermediate layer and an initial value of a vector representing the bias of an intermediate layer is determined from the input data. 
     According to the present invention, in a back propagation type neural network having three layers--an input layer, an intermediate layer, and an output layer, a calculation means is provided for calculating the weight coefficient matrix of an output layer from an average vector of each class of input data of an input layer, a weight coefficient matrix of an intermediate layer and a vector representing the bias of an intermediate layer and initial values of a weight coefficient matrix of an output layer are determined from the input data. 
     According to the present invention, in a back propagation type neural network having three layers--an input layer, an intermediate layer, and an output layer, a calculation means is provided for calculating a vector representing the bias of an output layer from a weight coefficient matrix of an output layer and an initial value of the bias of the output layer is determined from the input data. 
     Other objects and advantages besides those discussed above shall be apparent to those skilled in the art from the description of a preferred embodiment of the invention which follows. In the description, reference is made to accompanying drawings, which form a part hereof, and which illustrate an example of the invention. Such example, however, is not exhaustive of the various embodiments of the invention, and therefore reference is made to the claims which follow the description for determining the scope of the invention. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     FIG. 1 is a block diagram showing the configuration of the neural network in one embodiment of the present invention; 
     FIG. 2 is a flowchart showing an initial value determination process; 
     FIG. 3 is a view showing the distribution of input data x; 
     FIG. 4 is a view showing the distribution of vector W 1 .x; 
     FIG. 5 is a view showing the distribution of vector W 1 .x-θ 1  ; 
     FIG. 6 is a view showing the distribution of vector f(W 1 .x-θ 1 ); and 
     FIG. 7 is an view for explaining a method of determining W 20  (K) and bias θ 20  (K) . 
    
    
     DESCRIPTION OF THE PREFERRED EMBODIMENTS 
     One embodiment of the present invention will be explained in detail hereinunder with reference to the drawings. This embodiment is made up of an input layer, intermediate layers, and an output layer, in this embodiment, one layer is formed as an example of intermediate layers. 
     FIG. 1 is a block diagram of a neural network of this embodiment. In this figure, the neural network consists of one input layer 1, one intermediate layer 2, and one output layer 3. 
     Where the parameters of the synapse elements of this network are suitably set, input data X is received by the synapses of the input layer 1 and sent to the synapses of the intermediate layer 2. The intermediate layer 2 multiplies the input data X sent from the input layer 1 by a weight coefficient W 1 , adds a bias θ 1  to operate a transfer function (in this embodiment, a sigmoid function is selected as a transfer function), and outputs the resulting data Y to the output layer 3. The output layer 3 multiplies the input data Y sent from the input layer 2 by a weight coefficient W 2 , add a bias θ 2  to operate a transfer function, and outputs the resulting data Z. 
     The above is the explanation of the flow of data in general data processing, after learning, by the neural network of this embodiment. 
     Shown in FIG. 1 are weight coefficient calculation sections 4 and 6 for calculating weight coefficient matrices W 1  and W 2 , respectively, in a learning process. Also shown are bias calculation sections 5 and 7 for calculating biases θ 1  and θ 2 , respectively. The method of calculating the learning values of W 1 , W 2 , θ 1  and θ 2  by these calculation sections follows a back propagation method. The method is well known and therefore an explanation thereof is omitted. Next, a method of assigning the initial values of these parameters W 10 , W 20 , θ 10  and θ 20 , a feature of this invention, will be explained. 
     The procedure of assigning initial values in this embodiment will be explained with the flowchart in FIG. 2 with reference to FIG. 1. In this embodiment, data input for learning are represented by x, y and z to discriminate the above-mentioned X, Y and Z. From the statistical properties of the learning data x, the calculation sections 4, 5, 6 and 7 assign initial values W 10 , W 20 , θ 10  and θ 20 . Because the learning data x, y, z are represented as vectors, these are also called vectors x, y, z in this specification. 
     First, in step S1, the weight coefficient calculation section 4 calculates the initial value W 10  for a weight coefficient matrix of the intermediate layer from the in-class covariant matrix and the inter-class covariant matrix of the input data x. Next, in step S2, the bias calculation section 5 calculates the bias initial value θ 10  from the total average vector x tot   of input data and the weight coefficient matrix W 10  of the intermediate layer. The   indicates an average value. Next, in step S3, the weight coefficient matrix calculation section 6 calculates the initial value W 20  of the weight coefficient matrix of the output layer from an average vector x.sup.(k)  of each class of input data, the weight coefficient matrix W 10  of the intermediate layer and the bias θ 10  of the intermediate layer. In step S4, the bias calculation section 7 calculates the bias θ 20  of the output layer from the weight coefficient matrix W 20  of the output layer. 
     Each calculation in the above procedure will be explained more in detail below using equations. 
     When the neural network of the present invention is used, for example, in a character recognition apparatus, input data becomes a feature vector representing the feature of a character, extracted from a character image. All input data is represented by an M dimensional vector and it is assumed that the data belongs to any one of K sets of classes. 
     If the number of data contained in each class is set at n and the j-th input data of class k (k=1, 2, . . . , K) is represented by x j .sup.(k) the following statistical quantities can be calculated from the distribution of input data; namely, an input vector average x(k) of each class, an average value x tot   over all the classes, a covariant matrix S.sup.(k) of each class, an in-class covariant matrix S W , and an inter-class covariant matrix S B . ##EQU3## In the above equations, T in () T  represents a transposition. 
     A signal processing in a back propagation type neural network made up of three layers--the input layer 1, the intermediate layer 2, and the output layer 3--can be represented by the following equations if the dimension of the input vector x is set at M, the number of elements of the intermediate layer, at N, and the number of classes to be classified, at K. That is, 
     
         y=f (W.sub.10 x-θ.sub.10)                            (10) 
    
     
         z=f (W.sub.20 y-θ.sub.20)                            (11) 
    
     where 
     x: input data (X εR M ) 
     y: output from intermediate layer (y εR N ) 
     z: output from output layer (z εR K ) 
     W 10  : weight coefficient matrix of intermediate layer (N x M matrix) 
     θ 10  : bias of intermediate layer (θ 10  εR N ) 
     W 20  : weight coefficient matrix of output layer (K x N matrix) 
     θ 20  : bias of output layer (θ 20  εR K ). 
     f indicates that a sigmoid function, ##EQU4## is operated on the components of the vector. 
     As learning is performed according to a back propagation algorithm and the values of the weight coefficient W and the bias θ have come to converge on an ideal value, an output such that ##EQU5## is made from the j-th synapse element of the output layer to the input vector x.sup.(k) belonging to class k. 
     In the present invention, to assign the initial value W 10  of the weight coefficient of the intermediate layer, &#34;The method of linear discriminant mapping&#34; is used. (For example, refer to &#34;Handbook of Multivariate Analysis&#34;, by Yanai and Takagi, Gendai Suugaskusha.) 
     Linear discrimination mapping intends to select, as the evaluation criterion for discrimination, a linear mapping A (y=A. x: x εR M , y εR N ) such that &#34;the inter-class covariation becomes as large as possible and, at the same time, the in-class covariation becomes as small as possible&#34;. 
     When this method of linear discrimination mapping is applied to determine the initial value W 10  of a weight coefficient of the intermediate layer, a problem of finding a mapping A such that an inter-class covariant S B  becomes maximum while keeping an in-class covariant S W  constant, reduces to an eigen value problem of finding an eigen value λ i  and an engine vector a i  that satisfy the relation 
     
         S.sub.w a.sub.i =λ.sub.i S.sub.B a.sub.i            (13). 
    
     When the eigen value λ i  and the eigen vector a i  are found by solving an eigen problem, an eigen vector matrix, in which eigen vectors (a 1 , a 2 , . . . , a N ) corresponding to N pieces of (λ i , λ 2 , . . . , λ N ) that are taken from such an eigen value λ i  in a descending order, becomes a linear discrimination mapping A to be determined. That is, 
     
         A=(a.sub.1, a.sub.2, . . . , a.sub.N)                      (14) 
    
     A transposed matrix of this eigen vector matrix is given as the weight coefficient matrix W 10  of the intermediate layer. That is, 
     
         W.sub.10 =A.sup.T                                          (15) 
    
     The above results will be explained in detail using FIGS. 3 and 4. 
     The distribution of x when the dimension M of the input vector x is set at 2 and the classes are set at 3 of k 1 , k 2  and k 3 , is shown in FIG. 3. As shown in FIG. 3, for example, it is assumed that an input vector of each class has a tendency such that it is distributed in a lateral direction in classes k 1  and k 2 . Thus, if, keeping the inclass covariant S W  constant, a transposed matrix of a mapping A such that the inter-class covariant S B  becomes maximum is defined to be the initial value W 10  of the weight coefficient of the intermediate layer, the input vector of each class shows the distribution shown in FIG. 4 by the mapping of W 10  x. That is, a variance is maximum among input vectors in the different classes by the mapping of W 10 . x and the extension of the variance is small among the input vectors in the same class. In other words, if W 10  (=A T ) is taken as the initial value of a weight coefficient matrix, convergence in a learning process is quickened. 
     Assignment of the initial value θ 10  of a bias is explained next. 
     
         θ.sub.10 =W.sub.10 x.sub.tot                         (16) 
    
     where x tot   is a total average vector obtained by taking the average of vector x over all the classes. FIG. 4 shows W 10  x tot   by mapping W 10 . The assignment of θ 10  as in equation (16) causes the distribution of the vector obtained by the conversion of W 10  x-θ 10  to the input vector x to become as shown in FIG. 5, and the center of the distribution moves to the origin. 
     Further, since the distribution of the output y=f(W 10  x-θ 10 ) from the intermediate layer is obtained by operating the sigmoid function of equation (12) on the components of the vector, the region of the values of each component is suppressed to a section [-1/2]. 
     Next, a method of assigning the initial value W 20  of the weight coefficient matrix of the output layer and the bias θ 20  will be explained. 
     Among the synapse elements of the output layer, element Zk is assumed to output 1/2 for the input of data belonging to solely class k, and output -1/2 for the other inputs. And if the initial value of the weight coefficient vector for the Zk is denoted by W 20 .sup.(k) and the initial value of the bias is denoted by θ 20 .sup.(k). Then the output zk from such elements becomes as follows: 
     
         z.sub.k =f(W.sub.20.sup.(k)T.y-θ.sub.20.sup.(k))     (17) 
    
     T represents transposition. 
     The set of vectors that satisfy (W 20 .sup.(k)T.sub..y-θ 20 .sup.(k))=0 represents a hyperplane that includes f(W 10  x.sup.(k) -θ 10 ), which is an image of an average vector x.sup.(k)  of class k on the positive area side and includes the images of the other classes on the negative area side. To be specific, in FIG. 7, the set of f(W 10  x.sup.(k) -θ 10 ) for the input vector x of class k 1  is included in a positive area and the set of f(W 10  x.sup.(k) -θ 10 ) for the input vector x of classes k 2  and k 3  (≠k 1 ) is included in a negative area. 
     Thus, the following are defined: 
     
         W.sub.20.sup.(k) =f(W.sub.10.x.sup.(k) -θ.sub.10)    (18) 
    
     
         θ.sub.20.sup.(k) =α∥W.sub.20.sup.(k) ∥(19) 
    
     αis a positive constant less than 1 and, for example, may be set to 0.5. 
     In such a case, as shown in FIG. 7, the set of vectors that satisfy z k  =0 passes a midpoint between an image y.sup.(k)  of x.sup.(k)  and the origin and represents a K-1 dimensional hyperplane perpendicular to y.sup.(k). 
     Thus, those in which initial values W 20 .sup.(k) and θ 20 .sup.(k) of a parameter for class k are arrayed become a matrix W 20  representing the initial values of the weight coefficient of the output layer and a matrix θ 20  representing the bias initial value of the output layer, respectively. That is, 
     
         W.sub.20 =(W.sub.20.sup.(1), W.sub.20.sup.(2), . . . , W.sub.20.sup.(k)).sup.T 
    
     
         θ.sub.20 =(θ.sub.20.sup.(1), θ.sub.20.sup.(2), . . . , θ.sub.20.sup.(k)) 
    
     From the above procedure, W 10 , θ 10 , W 20 , and θ 20  are determined. 
     The initial values obtained from the above-mentioned method are good estimate values obtained by the theory of linear mapping. Ordinary input data is nonlinear. However, input data handled by the neural network of the above-mentioned embodiment is thought to have a small non-linearity and therefore they are values more proper than those obtained by a conventional method of adopting a random number as an initial value. 
     Various modifications are possible within the scope not departing from the spirit of the present invention. 
     For example, a suitable normalization may be performed on the weight coefficient matrices W10 and W20 and biases θ 10  and θ 20  used in the above-mentioned embodiment to change the norm. 
     A transfer function f is not limited to the format of equation (12). For example, any function may be used if it is differentiable, monotone increasing and a suppression type. 
     Some of W 10 , θ 10 , W 20 , and θ 20  may be assigned with a random number as in the prior art rather than all values of these being determined according to the above-mentioned embodiment. 
     Accordingly, in the neural network of the present invention, estimate values obtained on the basis of statistical information (average value, covariant matrix) of input data are adopted in place of a conventional method of assigning a random number as an initial value of a weight coefficient of each element and/or an initial value of a bias. As a result, learning time is considerably shortened over the conventional art which uses a random number as an initial value, and the advantage can be obtained that the possibility of falling to a minimum value which is not optimum is lessened. 
     The present invention is not limited to the above embodiments and various changes and modifications can be made within the spirit and scope of the present invention. Therefore, to apprise the public of the scope of the present invention the following claims are made.