A neural network computer (20) includes a weighting network (21) coupled to a plurality of phase-locked loop circuits (251-25N). The weighting network (21) has a plurality of weighting circuits (C11, . . . , CNN) having output terminals connected to a plurality of adder circuits (311-31N). A single weighting element (Ckj) typically has a plurality of output terminals coupled to a corresponding adder circuit (31k). Each adder circuit (31k) is coupled to a corresponding bandpass filter circuit (31k) which is in turn coupled to a corresponding phase-locked loop circuit (25k). The weighting elements (C1,1, . . . , CN,N) are programmed with connection strengths, wherein the connection strengths have phase-encoded weights. The phase relationships are used to recognize an incoming pattern.

FIELD OF THE INVENTION

The present invention relates, in general, to computational devices and, more particularly, to a recurrent neural network computer based on phase modulation of Phase-Locked Loop (PLL) nonlinear oscillators.

BACKGROUND OF THE INVENTION

Neural network computers, are biologically inspired, that is, they are composed of elements that perform in a manner analogous to the most elementary functions of the biological neuron. In one methodology, a neural network computer is composed of a number (n) of processing elements that may be switches or nonlinear amplifiers. These elements are then organized in a way that may be related to the anatomy of the brain. The configuration of connections, and thus communication routes, between these elements represents the manner in which the neural network computer will function, analogous to that of a program performed by digital computers. Despite this superficial resemblance, such artificial neural networks exhibit a surprising number of the brain's characteristics. For example, they learn from experience, generalize from previous examples to new ones, and abstract essential characteristics from inputs containing irrelevant data. Unlike a von Neumann computer, such a neural network computer does not execute a list of commands (a program). Rather, it performs pattern recognition and associative recall via self-organization of connections between elements.

Artificial neural networks can modify their behavior in response to their environment. Shown a set of inputs (perhaps with desired outputs), they self-adjust to produce consistent responses. A network is trained so that application of a set of inputs produces the desired (or at least consistent) set of outputs. Each such input (or output) set is referred to as a vector. Training can be accomplished by sequentially applying input vectors, while adjusting network weights according to a predetermined procedure, or by setting weights a priori. During training, the network weights gradually converge to values such that each input vector produces the desired output vector.

Because of their ability to simulate the apparently oscillatory nature of brain neurons, oscillatory neural network computers are among the more promising types of neural network computers. Simply stated, an oscillatory neural network computer includes oscillators. Oscillators are mechanical, chemical or electronic devices that are described by an oscillatory signal (periodic, quasi-periodic, almost periodic function, etc.) Usually the output is a scalar function of the form V(ωt+φ) where V is a fixed wave form (sinusoid, saw-tooth, or square wave), ω is the frequency of oscillation, and φ is the phase deviation (lag or lead).

Recurrent neural networks have feedback paths from their outputs back to their inputs. The response of such networks is dynamic in that after applying a new input, the output is calculated and fed back to modify the input. The output is then recalculated, and the process is repeated again and again. Ideally, successive iterations produce smaller and smaller output changes until eventually the outputs become steady oscillations or reach a steady state. Although these techniques have provided a means for recognizing signals, to date they have not been able to do so using associative memory.

Accordingly, a need exists for a neural network computer with fully recurrent capabilities and a method that incorporates the periodic nature of neurons in the pattern recognition process.

SUMMARY OF THE INVENTION

In accordance with the present invention, an oscillatory neural network computer is disclosed that exhibits pattern recognition using the phase relationships between a learned pattern and an incoming pattern, i.e., the pattern to be recognized.

In one aspect of the present invention, the oscillatory neural network computer comprises a weighting circuit having phase-based connection strengths. A plurality of phase-locked loop circuits are operably coupled to the weighting circuit.

In another aspect of the present invention, a method for programming an oscillatory neural network computer is provided wherein programming comprises encoding connection coefficients of the oscillatory neural network computer in accordance with phase relationships of a pattern to be learned.

In yet another aspect of the present invention, a method for recognizing an incoming pattern using an oscillatory neural network computer is provided wherein the method comprises using the phase deviation between a learned pattern and the incoming pattern to create an output signal indicative of the learned pattern.

DETAILED DESCRIPTION

The oscillatory neural network computer of the present invention learns or memorizes information in terms of periodic waveforms having an amplitude, a frequency, and a phase. This information is encoded as connection strengths, Sk,j, using a learning rule such as, for example, the Hebbian learning rule. The connection strengths, in combination with phase-locked loop circuitry, are used to recognize information from signals transmitted to the oscillatory neural network computer.

Copending U.S. patent application Serial No. PCT/US99/26698, entitled “OSCILLATORY NEUROCOMPUTER WITH DYNAMIC CONNECTIVITY” and filed Nov. 12, 1999 by Frank Hoppensteadt and Eugene Izhikevich is hereby incorporated herein by reference in its entirety.

FIG. 1schematically illustrates an oscillatory neural network computer20in accordance with an embodiment of the present invention. Oscillatory neural network computer20comprises a weighting network21coupled in a feedback configuration to a plurality of phase-locked loop circuits251, . . . ,25N-1,25N. Oscillatory neural network computer20has output terminals OUT1, OUT2, . . . , OUTN-1, OUTNfor transmitting output signals V(θ1), V(θ2), . . . , V(θN-1), V(θN), respectively, where the output signals V(θ1), V(θ2), . . . , V(θN-1), V(θN) have equal frequencies and constant, but not necessarily zero, phase relationships. More particularly, weighting network21includes a plurality C1,1, C1,2, . . . , CN,Nof weighting circuits, a plurality311, . . . ,31Nof adder circuits, and a plurality351,35Nof bandpass filter circuits. Weighting circuits C1,1, C1,2, . . . , CN,Nare configured as an N×N matrix. By way of example, weighting network21is a symmetric matrix of weighting circuits where each weighting element is coupled for transmitting an output signal having a connection strength, Sk,j, associated therewith. The connection strength is also referred to as the connection weight, the connection coefficient, the interunit connection strength, or simply the strength or weight. It should be understood that reference number25is used to collectively identify the plurality of phase-locked loop circuits, reference number31is used to collectively identify the plurality of adder circuits, and reference number35is used to collectively identify the plurality of bandpass filter circuits. The subscript notation (1, 2, . . . , N) has been appended to reference numbers25,31, and35to identify individual phase-locked loop circuits, adder circuits, and bandpass filter circuits, respectively. Further, the weighting circuits may also be referred to as connectors or weighting elements.

FIG. 2schematically illustrates an embodiment of a weighting circuit (C1,1, C1,2, . . . , CN,N) in accordance with the present invention. In this embodiment, weighting circuits C1,1, C1,2, . . . , CN,Ncomprise a linear amplifier23having an input terminal connected to a respective output terminal OUT1, OUT2, . . . , OUTN-1, OUTN. An output terminal of linear amplifier23is connected to an input terminal of phase shift circuit24. An output terminal of phase shift circuit24is connected to a corresponding adder circuit31. The output signal appearing on the output terminal of phase shift circuit24is:
Sk,j*V(θ+ψk,j)  (1)

whereSk,jis the connection strength (gain) of weighting circuit Ck,jprovided by its linear amplifier23; andψk,jis the phase shift introduced by phase shift circuit24of the weighting circuit.

Referring toFIGS. 1 and 2, PLL neural network20is a dynamic system that is described mathematically as

θkis the phase of the VCO embedded in the kthPLL circuit;

θjis the phase of the VCO embedded in the jthPLL circuit;

Ω is the natural frequency of the VCO in MegaHertz (MHz);

Sk,jare the connection strengths; and

V(θ) is a 2π periodic waveform function.

PLL neural network computer20has an arbitrary waveform function, V, that satisfies “odd-even” conditions and if connection strengths Sk,jare equal to connection strengths Sj,kfor all k and j, then the network converges to an oscillatory phase-locked pattern, i.e., the neurons or phase-locked loop circuits oscillate with equal frequencies and constant, but not necessarily zero, phase relationships. Thus, the phase relationships between the oscillators can be used to determine the connection strengths of a neural network computer.

An example of using phase relationships to train neural network computer20is described with reference toFIGS. 1,2, and3.FIG. 3illustrates three patterns to be memorized that correspond to the images or symbols “0”, “1”, and “2,” and which are identified by reference numbers41,42, and43, respectively. The phase relationships of a set of key vectors (ξm) or patterns of the images are initially memorized using a well known learning rule such as, for example, the Hebbian learning rule. (Other suitable learning rules include the back propagation learning rule, the template learning rule, the least squares learning rule, the correlation learning rule, the perceptron learning rule as well as other supervised and nonsupervised learning rules). In accordance with the Hebbian learning rule, the images are described as a set of key vectors:
ξm=(ξm1, ξm2, . . . , ξmN), ξmk=±1, m=0, . . . , r, and k=1, . . . , N  (3)
wherem identifies the pattern to be memorized;r+1 is the number of patterns to be recognized; andN is the number of phase-locked loop circuits, i.e., the number of neurons.

Still referring toFIG. 3, patterns “0”, “1”, and “2” are partitioned into sixty (N=60) pixels or subunits, P1, P2, . . . , P60, which are described by key vectors ξ0, ξ1, ξ2, respectively, which key vectors are to be memorized or recognized by oscillatory neural network computer20. For the image or symbol “1,” ξ11is the vector component of the first pixel, P1; ξ12is the vector component of the second pixel, P2; ξ13is the vector component of the third pixel, P3; ξ14is the vector component of the fourth pixel, P4, etc. It should be noted that in this example, vector component ξ11describes a white pattern, vector component ξ12describes a white pattern, vector component ξ13describes a black pattern, vector component ξ14describes a black pattern, etc. Key vectors are also determined for the images “0” and “2.” When ξmk=ξmj, the kthand the jthoscillators are in phase, i.e., φk=φj, and when ξmk=−ξmj, the kthand the jthoscillators are anti-phase, i.e., θk=θj+π. It should be noted that the number of pixels into which the images are partitioned is not a limitation of the present invention.

The key vectors are used in conjunction with the learning rule to determine the connection coefficients of oscillatory neural network computer20. In the example of using the Hebbian learning rule to memorize the images, the connection coefficients, Sk,jare given by:

An advantage of using the Hebbian learning rule to determine the connection coefficients is that it produces symmetric connections Sk,jso that the network always converges to an oscillatory phase-locked pattern, i.e., the neurons oscillate with equal frequencies and constant, but not necessarily zero, phase relations. It should be understood that some information about each memorized image is included in each connection coefficient.

After the initial strengths are memorized, neural network computer20is ready for operation, which operation is described with reference toFIGS. 1-8.

FIG. 4illustrates a pattern44to be recognized which is a degraded or distorted version of the image “1” illustrated inFIG. 3and previously memorized. In operation, the pattern to be recognized, i.e., pattern44, is parsed into a desired number of pixels. Each pixel will correspond to a PLL circuit. Each PLL circuit is the equivalent of a neuron, in our use of the term. In the example shown inFIGS. 4-8pattern44is parsed into sixty pixels, P1-P60. Thus, oscillatory neural network computer20will contain sixty PLL circuits,251-2560. At time t=0, oscillatory neural network computer20is initialized such that the initial signals on PLL circuits251-2560represent degraded pattern44. By way of example, PLL circuits251-2560are initialized by applying an external signal at input terminals IN1, . . . , INNof the form:
Ik(t)=Akcos(Ωt+φ0)  (4)
for k=1, . . . , 60, where:

Ω is the same as the center frequency of the PLL;

Akare large numbers that are positive if the input for the kthchannel is to be initialized at +1 and are negative if the kthchannel is to be initialized at a −1.

After an initialization interval, the external inputs are turned off and the network proceeds to perform its recognition duties.

Another suitable method for initializing neural network computer20is to start the PLL circuits of PLL25such that they have different phases that represent pattern44. Yet another suitable method is to start the PLL circuits of PLL25such that they have the same phase and then shift the phase in accordance with pattern44. Yet another suitable method is to set the initial voltages of the loop filters associated with each PLL circuit of PLL25. It should be understood that the method for initializing oscillatory neural network computer20is not a limitation of the present invention.

Still referring toFIG. 4, at time t=0, pattern44is such that the image or value for the pixel P1associated with key vector ξ11is white, the image or value for the pixel P2associated with key vector ξ12is black, the image or value of the pixel P3associated with key vector ξ13is black, the image or value of the pixel P4associated with key vector ξ14is white, the image or value of the pixel P5associated with key vector ξ15is white, the image or value of the pixel P6associated with key vector ξ16is white, the image or value of the pixel P7associated with key vector ξ17is white, the image or value of the pixel P8associated with key vector ξ18is black, . . . , the image or value of the pixel P60associated with key vector ξ160is black. It should be understood that the superscripted number 1 indicates this is the phase pattern being recognized, as opposed to a “0” or a “2.”

FIG. 5is a plot50of the frequency51and the phase52portions of output signals, V(θ1) and V(θ2), of PLL circuits251and252, respectively, (FIG. 1). At time t=0, the amplitude of output signal V(θ2) is +1 and the phase φ2is approximately 120 degrees (2π/3) which corresponds to the white image associated with key vector ξ11; whereas at time t=0 the amplitude of output signal V(θ2) is −1 and the phase φ2is approximately 0 degrees which corresponds to the black image associated with key vector ξ12.

FIG. 6is a plot55of the frequency56and the phase57portions of output signals, V(θ1) and V(θ3), of PLL circuits251and253, respectively, (FIG. 1). At time t=0, the amplitude of output signal V(θ1) is +1 and the phase φ1is approximately 120 degrees (2π/3) which corresponds to the white image associated with key vector ξ11; whereas at time t=0 the amplitude of output signal V(θ3) is −1 and the phase φ3is approximately 0 degrees which corresponds to the black image associated with key vector ξ13.

Although not shown, it should be understood that there are corresponding output signals V(θ4), . . . , V(θ60) that occur for each of the respective PLL circuits254, . . . ,2560.

Plots50and55further illustrate pattern recognition in accordance with an embodiment of the present invention. Because the patterns of the individual pixels that have been learned are either black or white, output signals V(θ1), . . . , V(θ60) lock in phase or in anti-phase to each other depending on the pattern being recognized. For example, in the pattern for a “1” (FIG. 3) pixels P1, P2, P5, and P6are white and pixels P3and P4are black. Thus, when this pattern is recognized from pattern44, PLL circuits251,252,255, and256should lock in phase to each other and PLL circuits253and254should lock in phase to each other. Further, the PLL circuits for pixels that are of opposite color should lock in anti-phase to each other, i.e., pixels that are white should lock in anti-phase or out of phase to pixels that are black. This is illustrated for PLL circuits251and252inFIG. 5and for PLL circuits251and253inFIG. 6. Referring toFIG. 5, output signals V(θ1) and V(θ2) have a substantially constant in-phase relationship to each other by time t=8.0, i.e., the difference in their phases (φ1-φ2) is less than 30 degrees. It should be understood that an acceptable error value for the phase difference of a signal locked in-phase is a design choice. It should be further understood that the units for time are not included for the sake of clarity. Thus, the units for the time may be seconds, milliseconds, microseconds, etc.

Briefly referring toFIG. 6, output signals V(θ1) and V(θ3) have a substantially constant anti-phase relationship to each other by time t=8.0, i.e., the difference in their phases (φ1-φ3) is greater than approximately 150 degrees. It should be understood that an acceptable error value for the phase difference of a signal locked in anti-phase is a design choice.

FIG. 7illustrates the visual outputs for pixels P1, . . . , P60from time t=8.0 to time t=9.2. At times t=8.0, 8.6, and 9.2, pixels P1and P2are white, whereas at times t=8.3, 8.9, and 9.5 pixels P1and P2are black. It is expected that at times t=8.0, 8.6, and 9.2, the output signals for pixels P1and P2would have substantially the same amplitude and polarity and be in phase and at times t=8.3, t=8.9, and t=9.5, the output signals for pixels P1and P2would have substantially the same amplitude and polarity and be in phase. Comparing these times with the output signals shown inFIG. 5, at times t=8.0, t=8.6, and t=9.2 output signals V(θ1) and V(θ2) have an amplitude of +1 and are substantially in phase. Likewise, at times t=8.3, t=8.9, and t=9.5 output signals V(θ1) and V(θ2) have an amplitude of −1 and are substantially in phase.

Further, it is expected that at times t=8.0, 8.3, 8.6, 8.9, 9.2, and 9.5, the output signals for pixels P1and P3would be in anti-phase and have amplitudes of substantially the same magnitude but opposite polarity. Comparing these times with the output signals shown inFIG. 6, at times t=8.0, 8.6, and 9.2 output signal V(θ1) has an amplitude of +1 and V(θ3) has an amplitude of −1, and at times t-8.3, t=8.9, and t=9.5 output signal V(θ1) has an amplitude of −1 and V(θ3) has an amplitude of +1.

The output signals of neural network computer20can be monitored by multiplying each output signal with a reference output signal. In the example of recognizing pattern44fromFIG. 4, the output signals of each of pixels P1, . . . , P60are multiplied with that of pixel P1. The result of this multiplication is a new sixty pixel image where the image of new pixel P1newis the product of output V(θ1) with itself; the image of new pixel P2newis the product of output signals V(θ1) and V(θ2); the image of new pixel P3newis the product of output signals V(θ1) and V(θ3); the image of new pixel P4newis the product of output signals V(θ1) and V(θ4), etc. This result is illustrated inFIG. 8where the products are shown at time intervals t=0, t=1, t=2, . . . ,t=10.

By now it should be appreciated that a method for recognizing patterns and an oscillatory neural network computer for implementing the method have been provided. An important aspect of this invention is the discovery by the inventors that the output signals for a PLL neural network computer oscillate with equal frequencies and constant, but not necessarily zero, phase relationships. Thus, the phase relationships of the neural network computer are used to determine the connection strengths of the neural network computer. This provides an increased immunity to noise. Another advantage of the present invention is that the type of learning rule used to train the neural network computer is not a limitation of the present invention.