Neuronal data processing system and adaptation method therefor

Abstract: An adaptation method for a data processing system comprising a plurality of neurons each of which outputs an output according to a comparison between a sum of multiplied inputs by weights and a threshold, characterized in the following steps: The threshold of the neuron which has generated significant output at certain point of time is compulsorily increased to a maximal value; The weight of the neuron is adapted for a constant value of the inputs; and the threshold is decreased to a value at the point.

Description: The present invention relates to a method for avoiding falling into local minima during learning of a neural network according to the preamble part of claim 1. Such a method is known from EP-A-0 340 742. On learning process of data processing system of this type, when an adequate input and output correlation at certain level occurs and the correlation is strengthened, data processing system cannot escape from the condition thereafter, as a result, further adaptation becomes impossible. This phenomenon is expressed by "fall into local minimum". Bolzman machines (David H. Ackley, Geoffrey E. Hinton, and Terrence J. Sejnowski: A learning algorithm for Bolzman machines: Cognitive Science 9, 1989, 147-169), for example, focuses on energy formula of neural network; This energy is jumped temporarily to predetermined high level so as to escape from a local minimum of above energy formula and reach global minimum. However, optimized correlation of input and output and the condition of global minimum of said energy do not always coincide with each other. In general, since the depth of local minimum is unknown, certainty of escaping from partial local minimum cannot be guaranteed when energy jump level is predetermined. Furthermore, there is no parallelism between the function of organisms and temporal energy jump. From AIP Conference Proceedings 151, Neural Networks for Computing, Snowbird, UT, 1986, pages 404-407, American Institute of Physics, New York, US; P. Sibani: "Firing response of a neural model with threshold modulation and neural dynamics", adaptively tuning a network by use of a quenching threshold (QT) is known. By increasing or decreasing the QT, the criterion of which activities represent functional signals - and hence should be processed and stored in a short term memory - and of which activities represent functional noise - and hence should be suppressed - can be flexibly modified through time. An increase in QT can cause all but the largest activities to be quenched. Then the network behaves like a choice machine. A sudden decrease in the QT can cause all recently presented input signals to be stored. If a novel or unexpected event suddenly decreases the QT, then all recently presented data can be stored in CAM until the cause of the unexpected event can be determined and learned. This property is considered important in actively regulating the focus of attention of a neural network sensory processor. It is the object of the present invention to provide an improved method for avoiding falling into local minimum during a learning process. This object is solved by the subject matter of claims 1 and 2. Preferred embodiments of the present invention are the subject matter of the dependent claims thereto. BRIEF DESCRIPTION OF THE DRAWING Fig.1 shows a graph describing of the change of standard identification ratio when appropriate learning is performed. Fig.2 shows a graph describing of the change of standard identification ratio when inappropriate learning is performed. Fig.3 shows a graph describing of the change of standard identification ratio when dropping to partial local minimal occurred on learning process. Fig.4 shows a schematic diagram of embodiment of ignition pattern for neural network when it drops to partial local minimal. Fig.5 shows a graph of the 1st embodiment of threshold change according to the present invention method. Fig.6 shows a graph of the 2nd embodiment of threshold change according to the present invention method. Fig.7 shows a graph of the 3nd embodiment of threshold change according to the present invention method. Fig.8 shows a graph of the 4th embodiment of threshold change according to the present invention method. Fig.9 shows a graph of the 5th embodiment of threshold change according to the present invention method. Fig.10 shows a graph of the 6th embodiment of threshold change according to the present invention method. Fig.11 shows a graph of the 7th embodiment of threshold change according to the present invention method. Fig.12 shows a graph of the 8th embodiment of threshold change according to the present invention method. Fig.13 shows a graph of the 9th embodiment of threshold change according to the present invention method. Fig.14 shows a graph of the 10th embodiment of threshold change according to the present invention method. Fig.15 shows a graph of the 11th embodiment of threshold change according to the present invention method. Fig.16 shows a graph of the 12th embodiment of threshold change according to the present invention method. Fig.17 shows a diagram of the weight change by the 13th embodiment of the present invention; Fig.18 shows a diagram of weight change by the 14th embodiment. Fig.19 shows a diagram of weight change by the 15th embodiment. Fig.20 shows a diagram of weight change by the 16th embodiment. Fig.21 shows a diagram of weight change by the 17th embodiment. Fig.22 shows a diagram of weight change by the 18th embodiment. Fig.23 shows a diagram of weight change by the 19th embodiment. Fig.24 shows a diagram of weight change by the 20th embodiment. Fig.25 shows a diagram of weight change by the 21th embodiment. Fig.26 shows a diagram of weight change by the 22th embodiment. Fig.27 shows a diagram of weight change by the 23th embodiment. Fig.28 shows a diagram of weight change by the 24th embodiment. Fig.29 is a diagram to show the structure of data processing system. Fig.30 shows a diagram of an embodiment of the initial threshold distribution. Fig.31 shows a diagram of the 2nd embodiment of the initial threshold distribution. Fig.32 shows a diagram of the 3rd embodiment in addition to Fig.19 of the initial threshold distribution. Fig.33 shows a diagram of the 4th embodiment in addition to Fig.19 of the initial threshold distribution. Fig.34 shows a diagram of the 5th embodiment in addition to Fig.19 of the initial threshold distribution. Fig.35 shows a diagram of the 6th embodiment in addition to Fig.19 of the initial threshold distribution. Fig.36 shows a diagram of the 7th embodiment in addition to Fig.19 of the initial threshold distribution. Fig.37 shows a diagram of the 8th embodiment in addition to Fig.19 of the initial threshold distribution. Preferred Embodiment of the Present Invention Hereafter, preferred embodiments of the present invention are described with referring to the attached drawings. Here, standard identification ratio TP is defined as an index for indicating the effect of learning as follows: where, n: number of input phenomena; i,j: input phenomenon number; Pi,j: identification ratio (number of bits identified) of neural network output of ith and jth input phenomena. Above standard identification ratio TP is an index of recognition ability for input phenomenon of neural network. The lower the TP is, the higher the recognition ability is. In general, TP=100% for neural network that has not been trained for any leanings. TP monotonously decreases as learning advances.(Fig.1) However, the learning effect may be deteriorated conversely, when learning method is inappropriate, causing raise of TP that has once been dropped, as shown in Fig.2. As mentioned above, TP is saturated at a high value when falling into local minimum occurs, as shown in Fig.3. This falling into local minimum is nothing to do with said energy formula(1), and can be considered as the result of learning which strengthen a characteristics of a phenomenon corresponding to a weight distribution with steep deviations. It causes confusion among a plurality of phenomena. This is typically described as shown in Fig.4, in which ignition distribution of neurons is inclined due to strong influence of a input but I4, and it becomes difficult for other neurons to ignite because this condition is strengthened. Here, ignited neurons are shown by black circle in this figure. In order to improve such condition, it is necessary to strengthen the participation of neurons with respect to other input bits; I1 to I3 and I5. Here, inventors noticed on absolute and relative refractory period of organism neural network. It is said that these refractory period contribute to restraint and control of stimulation transmissions, according to organism neural network. (Written by Shunichi Amari: A mathematical principal of neural network - information processing manner of neural network- ; Sangyo Tosho) Contrary to the above, inventors considered these refractory period are factors for changing temporarily the balance of influence for each characteristics of input data, and give change to threshold of neuron that ignites for certain input so as to obtain equivalent effect to absolute and relative refractory period. Accordingly, threshold  of neuron which has associatively ignited at a point of time "to" on learning process is raised to the maximal value  max(for example, to the infinite) for a predetermined period "ta" after the ignition. This corresponds to absolute refractory period. During the absolute refractory period, learning is continued and weight of neuron is changed. Here, neuron whose weight is to be changed is the neuron which does not ignite for the input before "to". After the period "ta", threshold is gradually decreased to value  o at "to". This period of gradual decrease corresponds to relative refractory period and expressed by "tr". As for absolute refractory period, weight is strengthen for only neuron which did not ignite at "to" and leaning for modification of deviated neuron distribution is performed. On the other hand, in the relative refractory period, balanced amendment learning for total neural network is performed, gradually, with igniting neurons that fired at "to". As the result, escape from local minimum is realized, and learning is performed to lead standard identification ratio to optimized value (global minimum). As shown in Fig.6 of embodiment 2, it is effective for escaping from a local minimum to give repetitive changes of the threshold during the refractory period. However, convergence of learning may obstructed, conversely, when too many repetitive changes occur, or when repetitive changes are given toward the end of learning. Therefore, repetitive changes shall be given for appropriate number of times at rather initial period of learning. Furthermore, optimization for period "ta" of absolute refractory period, period "ta" of relative refractory period, and relationship therebetween shall be optimized. Fig.7 shows the threshold change according to the 3rd embodiment. As for this embodiment, relative refractory period is prepared, whereas absolute refractory period does not exist. That is, threshold  is first changed to the maximal value  max for a period of "to", then gradually decreases to the original value  o during the period "ta". As for this embodiment, it is possible to complete learning fast due to absence of absolute refractory period, when there is no steep incline in neurons ignition distribution. Fig.8 shows the threshold change according to the 4th embodiment. As for this embodiment, a plurality of refractory periods are repeatedly given, each of which is similar to that in Fig.7. Fig.9 shows the threshold change according to the 5th embodiment. As for this embodiment, only relative refractory period is prepared in which threshold gradually increases. That is, threshold gradually increases during the period "ts" from a point "to". Once threshold reaches to the maximal value  max, it immediately returns to the original value  o . In view of above embodiment, balanced learning for improvement of total neural network can be expected. Fig.10 shows the threshold change according to the 6th embodiment. As for this embodiment, a plurality of cycles of relative refractory periods are given, each of which is similar to that in Fig.9. Fig.11 shows the threshold change according to the 7th embodiment. As for this embodiment, both relative refractory period are prepared, in which threshold gradually increases and decreases. That is, threshold gradually increases during the period "ts" from the point "to". Once threshold reaches to the maximal value  max, it immediately returns to the original value  o . The same effect as that of the above embodiments can be expected. Fig.12 shows the threshold change according to the 8th embodiment. As for this embodiment, the relative refractory period shown in Fig.11 is repeatedly occurs. Fig.13 shows the threshold change according to the 9th embodiment. As for this embodiment, only absolute refractory period is prepared, where as relative refractory period is not. That is, threshold has the maximal value  max during the period ta from the point to, then returns to the original value  o . In this embodiment, balanced amendment learning for total neural network is performed in a short period of time. Fig.14 shows the threshold change according to the 10th embodiment. As for this embodiment, absolute refractory period described in Fig.13 occurs repeatedly. Fig.15 shows the threshold change according to the 11th embodiment. As for this embodiment, relative refractory period, absolute refractory period in which threshold gradually increases, and relative refractory period in which threshold gradually decreases are prepared. That is, threshold gradually increases to the maximal value  max during the period of "ts" from the point "to", and the maximal value  max is maintained during the period "to", then gradually decreases to original value  o during the period "tr". In view of above embodiment, balanced amendment learning for total neural network is possible. Fig.16 shows the threshold change according to the 12th embodiment. As for this embodiment, refractory period as it is described in Fig.15 is repeatedly performed. As mentioned above, the adaptation method of data processing according to the present invention has outstanding effect of escaping from partial local minimal for certain as following steps; a threshold of neuron generating significant output at certain point is compulsorily changed to increase once to maximal value, then decreases to the value at said point; a predetermined weight is adjusted during the above compulsory change of threshold. In the embodiments above, an effect equivalent to refractory period is obtained by changing threshold. A similar effect can be occurred by that weights W1 to Wn of a neuron which fire at a point of time "to" are decreased to the minimum Wmin, for example "0". Fig.17 shows the 13th embodiment of this type. As shown in Fig.6, it is more certain to get out of a local minimum by giving a cycle of weight change. When the number of repetitions of weight change is too many, the convergence of learning is impeded. The convergence of learning is impeded, also when the cycle of weight change is given on the end period of learning. Namely, the cycle of weight change is to be given appropriate number in the initial term of learning. In addition to the above, adaptation is to be considered as to absolute refractory period of "ta", relative refractory period of "tr" and the balance of both periods. Fig.19 shows the weight change in the fifteenth embodiment. In this embodiment, absolute refractory period is not settled by only relative refractory period is settled. After weight wi takes the minimum value Wmin on "to", it immediately increases gradually to the beginning value wo during the period "tr". By the third embodiment, learning can be completed earlier because of absence of absolute refractory period, when there is no steep inclination of the distribution of neurons ignition distribution. Fig.20 shows the weight change in the sixteenth embodiment. In this embodiment, the cycle of relative refractory periods only as in Fig.19 is given plural times. Fig.21 shows the weight change of the seventeenth embodiment. In this embodiment, only relative refractory period is settled in which weight gradually decreases. The weight gradually decreases to reach the minimum value Wmin from "to" during the period "ts", then immediately increases to the beginning value wo. Balanced corrective learning is expected for whole of neural network by this embodiment. Fig.22 shows the weight change of the eighteenth embodiment. In this embodiment, the cycle of only relative refractory period is given in which weight gradually decreases as shown in Fig.9 plural times. Fig.23 shows the weight change of the nineteenth embodiment. In this embodiment, two patterns of the relative refractory periods are settled in which weight gradually decreases and in which weight gradually increases. The weight gradually decreases to reach the minimum value Wmin from "to" during the period "ts", then immediately increases to the original value wo increasing gradually during the period "tr". This embodiment is also expected the same effectiveness of each embodiment above. Fig.24 shows the weight change of the twentieth embodiment. In this embodiment, two patterns of the relative refractory periods are repeated in which weight gradually decreases and in which weight gradually increases as shown in Fig.13. Fig.25 shows the weight change of the twenty-first embodiment. In this embodiment, relative refractory period is not settled but only absolute refractory period is settled. The weight value is the minimum Wmin from "to" during the period "ta", and returns to the original value wo. Balanced corrective learning for whole neural network is possible in short time by the present embodiment, too. Fig.26 shows the weight change of the twenty-second embodiment. In this embodiment, absolute refractory period shown in Fig.13 is repeated. Fig.27 shows the weight change of the twenty-third embodiment. In this embodiment, three patterns of periods are settled: relative refractory period in which weight gradually decreases, absolute refractory period, and relative refractory period in which weight gradually increases. Weight value reaches the minimum Wmin decreasing gradually during the period "ts"; it keeps the minimum Wmin during the period "to"; and it returns the original value wo increasing gradually during the period "tr". Balanced corrective learning for whole neural network is possible by the present embodiment, too. Fig.28 shows the weight change of the twenty-fourth embodiment. In this embodiment, the cycle of refractory period shown in Fig.27 is repeated. As mentioned above, an adaptation method of data processing system according to the present invention has superior effectiveness to get out of the local minimum certainly because of the steps of: a) Giving certain inputs to the data processing system, a weight of neuron generated a useful output on a point is increased to the value on said point after once decreased to minimum value; b) The weight is arranged during the minimum value. It is described below the adaptive method to be executed in the beginning of learning so as to the adaptive method to be more effective one. In Fig.29, a data processing system comprises plural neural layer NL in which plural neurons N are settled parallelly. The neural layer has the structure that the output of a neural layer becomes the input of the neural layer in the next stage. The topology of a neural network can be defined in such a structure. The coordinate of each neuron can be specified as shown in Fig.29. Here, settle the condition below. X-axis is settled in the direction from the input to the output and Y-axis to be settled along the longitudinal direction of each neural layer. The initial threshold distribution of neurons is settled as in Fig.30. In the distribution, neurons in the same layer have the same value, and the value increases as the X coordinate increases. The inclination of threshold's increase is smooth. During learning process, a weight of a synapse transmitting significant output is increased when valid or correct data is input to a neural network. As to threshold distribution in Fig.30, only the neurons in the input side ignite first and gradually the ignition is spread out to the neurons in later steps. On this process, introducing the process of minimizing a weight, it is effective to increase the neurons to be ignited corresponding to a certain input in each neural layer. On the point that neural layer in the last step is ignited, many neurons contribute data processing and ignition pattern does not incline. Therefore, it is possible to prevent falling into local minimum. Fig.31 shows another embodiment of the initial threshold distribution. The threshold of neurons in the center of data processing system is large. As departing from the center of the data processing system, the threshold becomes smaller. Ignition pattern is generated around the base of the mountain of threshold at the beginning of learning. When threshold is changed to be maximum, the distribution of ignited neuron is spread gradually toward the center or the top of threshold mountain because of the existence of the neurons with rather lower threshold adjacent to igniting neuron. Through the processing, many neurons contribute data processing at the end of learning and ignition pattern does not incline. Therefor, it is possible to prevent a cave into local minimum. Fig.32 shows another embodiment of the initial threshold distribution in addition to Fig.31. The neuron's threshold at the middle of Y-axis is settled to be the minimum. It becomes larger toward the maximum and minimum of Y-axis with smooth inclination. The ignition pattern passing through or beside the middle of Y-axis is generated first in such a data processing system. When threshold is changed to be maximum, the distribution of igniting neurons is spread gradually toward both of the maximum and minimum of Y-axis because of the existence of the neurons with rather lower threshold adjacent to neurons igniting. Through the processing, many neurons finally contribute data processing and ignition pattern becomes flat. Therefore, it is possible to prevent the falling into local minimum. Fig.33 shows the 4th embodiment. In this embodiment, unlike the 1st embodiment in Fig.30, thresholds for neurons in the same Y coordinate are equal for each neural layer, whereas threshold of neuron is heightened as Y coordinate increases with respect to the same neural layer. Here, the inclination on threshold's increase is smooth. As to threshold distribution in Fig.33, only neurons larger in Y coordinate ignite first and gradually the ignition is spread out to the neurons smaller in Y coordinate. When maximization of threshold is performed, the distribution of igniting neurons is spread gradually because of the existence of the neurons with rather lower threshold adjacent to neurons igniting. Through the processing, many neurons finally contribute data processing and ignition pattern becomes flat. Therefor, it is possible to prevent the falling into local minimum. Fig.34 shows the 5th embodiment, thresholds for neurons of all neural layer in the same Y coordinate are equal, whereas threshold in the specific Y coordinate M (center, for example) is the maximum and decreases gradually as departing from the center. As to threshold distribution in Fig.34, only neurons at the maximum and minimum value side with respect to Y coordinate ignite first and gradually the ignition is spread out to the center. When maximization of threshold is performed, the distribution of igniting neurons is spread gradually because of the existence of the neurons with rather lower threshold adjacent to neurons igniting. Through the processing, many neurons finally contribute data processing and ignition pattern becomes flat. Therefore, it is possible to prevent the falling into local minimum. Fig.35 shows the 6the embodiment. In this embodiment, the smaller the X and Y coordinate becomes, the smaller the threshold of neurons becomes, and the bigger the X and Y coordinate becomes, the smoother the threshold of neurons decreases. As to this threshold distribution, only neurons at small X and Y coordinate ignite first and gradually the ignition is spread out to the neurons at bigger end of X and Y coordinate. When maximization of threshold is performed, the distribution of igniting neurons is spread gradually because of the existence of the neurons with rather lower threshold adjacent to neurons igniting. Through the processing, many neurons finally contribute data processing and ignition pattern becomes flat. Therefore, it is possible to prevent the falling into local minimum. Fig.36 shows the 7th embodiment of the initial threshold distribution. Threshold of center neuron is settled to the minimum. It increases as it becomes more distant from above center neuron. The ignition pattern is generated from center neuron at the beginning of learning. When maximization of threshold is performed, the distribution of igniting neurons is spread gradually because of the existence of the neurons with rather lower threshold adjacent to neurons igniting. Through the processing, many neurons finally contribute data processing and ignition pattern becomes flat. Therefore, it is possible to prevent the falling into local minimum. Fig.37 shows the 8th embodiment of the initial threshold distribution. Threshold, for example, becomes bigger as it approaches to the center in Y coordinate, whereas it, for example, becomes smaller as it approaches to the center in X coordinate. That is, the shape of threshold distribution is like a saddle. The similar effect of the above each embodiment can be obtained by the present embodiment, too. As to a data processing system according to the present invention, igniting neuron is spread during the learning, for modifying the inclination so as to give the smooth inclination to initial threshold distribution. As the result, it is possible to prevent inclined ignition pattern.

Claims: Method for avoiding falling in local minima during learning of a neural network comprising neurons each of which igniting in accordance to a comparison between a sum of weighted inputs and a threshold value, wherein said weighting of inputs is changed in accordance with learning, characterized by the steps of: determining a neuron which has generated a significant output at a certain time t o , compulsorily increasing the threshold value for said determined neuron from a predetermined value ( o ) to a maximum value ( max ) and subsequently decreasing the threshold value again to the predetermined value ( o ), and continuing, during the time of compulsorily increasing the threshold value above the predetermined value, with learning of the neural network. Method for avoiding falling in local minima during learning of a neural network comprising neurons each of which igniting in accordance to a comparison between a sum of weighted inputs and a threshold value, wherein said weighting of inputs is changed in accordance with learning, characterized by the steps of: determining a neuron which has generated a significant output at a certain time t o , compulsorily decreasing the weight of said determined neuron from a particular value present during said time t o to a minimum value (W min ) and subsequently increasing the weight again to said particular value (W i ), and continuing, during the time of compulsorily decreasing the weight below the particular value, with learning of the neural network. Method according to claim 1 or 2, wherein said steps are performed a plurality of times repeatedly. Method according to claim 1, wherein said threshold is changed so as to obtain an effect equivalent to an absolute refractory period. Method according to claim 1, wherein said threshold is changed so as to obtain an effect equivalent to a relative refractory period. Method according to claim 1, wherein said threshold is changed so as to obtain an effect equivalent to both an absolute and relative refractory period. Method according to claim 5 or 6, wherein said relative refractory period consists of an increasing period and decreasing period of threshold. Method according to claim 2, wherein said weight is changed so as to obtain an effect equivalent to an absolute refractory period. Method according to claim 2, wherein said weight is changed so as to obtain an effect equivalent to a relative refractory period. Method according to claim 2, wherein said weight is changed so as to obtain an effect equivalent to both an absolute and relative refractory period. Method according to claim 9 or 10, wherein said relative refractory period consists of an increasing period and decreasing period of weight. Method according to claim 1 or 2, wherein a smooth inclination is given to said threshold distribution at the initial condition. Method according to claim 12, wherein said threshold increases from an input side to an output side in said data processing system. Method according to claim 12, wherein said threshold decreases from a center portion to a peripheral portion of said data processing system. Method according to claim 12, wherein said threshold increases from a middle portion to both sides in the lateral direction of said data processing system. Method according to claim 12, wherein said threshold increases from a center portion to peripheral portion of said data processing system. Method according to claim 12, wherein said threshold increases laterally from one side to the other of said data processing system. Method according to claim 12, wherein said threshold decreases from a middle portion to both sides in the lateral direction of said data processing system. Method according to claim 12, wherein said threshold increases from one end of an input side to the opposite end of an output side. Method according to claim 12, wherein said threshold has saddle-like distribution with center line passing along lateral direction through a center portion of said data processing system.

Inventors: KUMAGAI RYOHEI, MATSUMOTO KOJI, TAKATORI SUNAO, YAMAMOTO MAKOTO, KUMAGAI, RYOHEI, MATSUMOTO, KOJI, TAKATORI, SUNAO, YAMAMOTO, MAKOTO
Applicants: SHARP KK, YOZAN INC, SHARP KABUSHIKI KAISHA, YOZAN INC.
Application year: 1990
Main classification: G06F15
IPC codes: G06N3