Patent Publication Number: US-4926064-A

Title: Sleep refreshed memory for neural network

Description:
This is a continuous of U.S. Patent Application Serial No. 222,887, filed July 22, 1988, now abandoned. 
    
    
     BACKGROUND OF THE INVENTION 
     1. Field of the Invention 
     The present invention relates to apparatus implementing a neural network, including a dynamic sleep refresh capability for reinforcing memory traces which might otherwise wash out over time. 
     2. Description of Related Art 
     A neural network is a highly parallel form of information processing apparatus, having as its philosophical basis the attempted modeling of the mammalian brain. A typical neural network comprises a large number of nodes, also called neurons or neuronal circuits, each intended to operate in some way analogous to a neuron in a mammalian brain. The neurons are connected to each other by synaptic connective circuits, also referred to as synapses or simply as connections, each intended to operate in some way analogous to a synapse in a mammalian brain. 
     Like the mammalian brain, the organization of a neural network is such that the each neuron is cross-coupled via synapses to a large number of other neurons in the network. The coupling in many cases is bi-directional and in some cases is uni-directional. Because of the philosophical basis for much of the work in the neural network area, it often happens that advances in the field help to improve the scientific understanding of the mammalian brain, and advances in brain research help to advance the state of the art of neural networks. 
     When a neural network operates, environmental input signals are provided to inputs of a subset of the neurons in the network. Following some algorithm, the signals propagate through the network and finally activate one or more output neurons representing a processed version of the input signals. For example, where the inputs are connected to the various pixels in a CCD imaging array, the outputs may be such as to specify which of the 26 letters of the alphabet is imaged on the CCD array. 
     Typically, the transfer functions by which information propagates through the network are modifiable according to a learning algorithm of some kind. In this way the network can learn to recognize different input patterns, depending on the environment to which it is exposed. Usually, the learning algorithm is either of a supervised form or of an unsupervised form. In a network which learns by supervised learning, the output response is measured against some predetermined correct output response either by a human or by an automatic supervisor. The network is then caused to modify its own transfer functions, often in dependence upon the difference between the actual and the predetermined output response, iteratively until the predetermined output response is achieved. The object of such a system is to have it find its own way to a predetermined relation between input signals and output responses. 
     In unsupervised learning networks, the network constructs its own distinctive output response for any given input signal. Such a network should itself be able to determine, for example, that different varieties of the letter &#34;B&#34; belong in a different category from the various forms of the letter &#34;A&#34;. Such a network would modify its own transfer functions in order to provide a different output response for each category of input signals. 
     An example of a supervised learning algorithm is the so-called back-propagation algorithm, illustrated in U.S. Pat. No. 3,950,733 to Cooper. An example of an unsupervised learning algorithm is that discussed in Fukushima, &#34;Neocognitron: A Hierarchical Neural Network Capable of Visual Pattern Recognition,&#34; Neural Networks, Vol. 1, pp. 119-130 (1988). 
     Another unsupervised learning algorithm is used in the ART-type networks proposed by Carpenter and Grossberg. See, for example, Carpenter and Grossberg, &#34;A Massively Parallel Architecture For a Self-Organizing Neural Pattern Recognition Machine,&#34; Computer Vision, Graphics, and Image Processing, Vol. 37, pp. 54-115 (1987) (&#34;Grossberg I&#34;) and Carpenter and Grossberg, &#34;The ART of Adaptive Pattern Recognition by a Self-Organizing Neural Network,&#34; IEEE Computer (March 1988) at 77-88 (&#34;Grossberg II&#34;). In its simplest form, the ART network is composed of two layers F1 and F2 of neurons. The F1 layer is composed of a large number of clusters, each of which, in turn, is composed of a number of nodes with mutually inhibitory interconnections. Each of the input neurons or nodes takes in synaptic signals from three types of sources: one from environmental input stimuli, another from nodes in F2 (top-down connections), and a third from a global control mechanism. The F2 layer is also composed of clusters of nodes. Each F2 cluster has mutually inhibitory nodes only one of which is active at any one time. F2 activity is controlled by input from two main types of sources: one from Fl nodes (bottom-up connections) and the other from a resetting mechanism. Both top-down and bottom-up connections are Long-Term Memory (LTM) connections, meaning that the time-constants or decay constants of synaptic strengths are much longer compared to that of Short-Term Memory (STM) activity in F1 and F2. It is noteworthy that the output signals generated by the neurons are binary, indicating either active on inactive Analog activity levels generated internally to the neuron form only one of several parameters to the formula by which the binary outputs are generated. 
     The transfer functions according to which information propagates through the network usually have a predefined form with coefficients or other constants which are modifiable in accordance with the learning algorithm. For example, in the ART model, each neuron v k  has a respective &#34;STM activity&#34; x k  which obeys a membrane equation of the form: 
     
         εdx.sub.k /dt =-x.sub.k (1-Ax.sub.k)J.sub.k.sup.+ -(B+Cx.sub.k)J.sub.k.sup.-, 
    
     where ε, A, B and C are constants, where J k   +  is the 15 total excitory input to v k , and where J k   -   is the total inhibitory input to v k . Each of the values J k  derives from an equation of the form ##EQU1## where v l  is a neuron the output of which influences neuron v k , f(x l ) is a fixed form binary output function of the internal activity level x in neuron v l , and z lk  is a coefficient which varies in accordance with the learning algorithm. 
     Coefficients such as z, which may be thought of as connectivity weights, can be stored as charge levels on a capacitor in each of many circuits implementing respective synaptic connections. Such implementations suffer from the problem that capacitors by their nature are imperfect, and lose charge over time. Current implementations therefore often use expensive CCDs or floating gate MOSFETs to retain memory as long as possible. In at least one instance, workers have even gone to the extent of cooling the capacitors to -100° C. in order to stem leakage and prevent memory loss. Mackie, et al., &#34;Implementations of Neural Network Models in Silicon,&#34; NATO ASI Series, Vol. F41, &#34;Neural 
     Computers,&#34; pp. 467-476, at 472 (1987). In other instances, memory is retained digitally. These technologies are expensive and do not lend themselves to optimal integration densities. No attempt has been made to stem charge loss through the use of circuitry. 
     In addition to gradual loss of memory due to charge leakage on the storage capacitors, such memory loss is sometimes intentionally designed into a system. For example, Carpenter and Grossberg recognized that even long-term mammalian synapses have a characteristic decay constant related to how fast the chemical transmitters/receptors are lost in the absence of preand post-synaptic activity. Their ART model therefore includes a so-called associative decay rule, which implies that some connectivity weights decay towards zero during learning See Grossberg I at 77. 
     While the associative decay rule models one aspect of the functioning of the mammalian brain, it is not perfect. As an illustration, suppose an ART system has learned six categories representing the letters A-F. Suppose further that for a long time thereafter, the system is presented with environmental inputs representing only the categories A-D. In the ART model, the memory traces by which the system recognizes the categories E and F will eventually wash out. However, we know that human beings can remember images and events that took place many years earlier, even without repetition of the stimuli. 
     It is therefore an object of the present invention to provide neural network apparatus in which memory traces are dynamically refreshed in order to counteract the loss of memory due to any cause. 
     SUMMARY OF THE INVENTION 
     According to the invention, a neural network system is provided which operates in at least two modes: one in which the system is &#34;awake&#34;, and one in which it is &#34;asleep&#34;. While the system is in its waking mode, it learns and solves problems according to any of the various prior art algorithms. When it is in its sleep mode, the system is isolated from its environmental inputs and the connectivity weights which have been learned until that point in time are adjusted according to some sleep refresh algorithm. The sleep refresh algorithm may be of a relatively unsophisticated type, in which the various connectivity weights are merely increased in fixed, or substantially fixed, proportion to their prior values. The algorithm by which this refresh activity takes place may also include an aspect which limits the increases in connectivity weights so as to prevent them from exceeding the electrical limitations of the system. 
     A more sophisticated sleep refresh. algorithm may be used in conjunction with another aspect of the invention, however. This aspect is based partially on the recognition that memory retention in the mammalian cortex is founded not solely on synaptic time constants, but also on the stochastic dynamism of the entire brain. Accordingly, a neural network system includes a large number of neurons, each responsive to a weighted sum of the outputs of at least a subset of the other neurons in the network. The terms in the sum are weighted according to connectivity weights, which are modifiable according to a learning algorithm described in more detail below. Not unlike prior art networks, the various neurons are interconnected in a topology which includes many feedback loops. For simplicity of illustration, the description of the invention describes a network which has only two levels of neurons: a first level, each neuron in which is responsive both to a respective environmental input signal and to each neuron in the second level. The second level neurons form a competitive cluster, in which the output of each of the second level neurons is positively responsive to each of the first level neurons and also negatively responsive to every other second level neuron in the cluster. 
     In prior art systems having this sort of topology, the output of each neuron was either &#34;active&#34; or &#34;inactive&#34;. The neuron response equations would operate with the various feedback loops such that in response to a given input pattern, the network would, in effect, activate different ones of the second level neurons until the positive feedback through the loops is such as to reinforce the choice of one of the second level neurons. At this time the system is said to be in resonance, and the active second level neuron is said to represent the category to which the input pattern belongs. A network constructed according to this aspect of the invention, however, has analog activity levels as the neuron outputs. Further, the network is constructed so that the first level neurons have a non-zero activity level output even in the absence of environmental stimulus. In a network according to this aspect of the invention, not only will resonances be achieved when and input pattern is recognized, as in the prior art networks, but secondary resonances, characterized by low intensity firings of the neurons not due to environmental stimulus, will also take place due to the non-zero activity level outputs of the first level neurons. 
     Advantageously, the activity levels of the first level neuron outputs never fall below respective quiescent values, which values are low enough that the secondary resonances have little influence on the categories chosen by the system in normal operation. When the network is isolated from its inputs, however, the secondary resonances dominate since they are the only resonances present in the system. The recursive nature of the network then causes various ones of the second level neurons to fire (i.e., rise to a temporary high activity level) in a pseudo-random sequence dictated by the topology of the network, the past history of the activity levels of the various neurons, and the connectivity weights quantifying the extent to which the outputs of the neurons influence the activity levels of the other neurons. These secondary resonances continue until inputs are once again applied to the system, at which time the primary resonances resume their dominance. Alternatively or additionally, whatever mechanism ensures the non-zero activity level outputs of first level neurons may be disengaged or suppressed when the network is awake and activated when the network is asleep. 
     In accordance with the invention, whatever learning algorithm is used while the network is awake remains active while the network is asleep. Learning algorithms usually are adapted to reinforce the connections which generated a resonance, and to weaken the connections which generate a faulty choice of categories. Consequently, when the network is in its sleep mode, the connections which cause secondary resonances are strengthened. Moreover, since the pseudo-random sequence by which the various second level neurons fire depends on the connectivity weights of the connections abutting the second level neurons, it will be seen that those second level neurons which have strong connections to first level neurons, or which are activated by a large number of first level neurons, will fire more often than those second level neurons which have a weaker connection with first level neurons, or which are activated by a smaller number of first level neurons. Since the learning algorithm strengthens connections when they are used, the learning algorithm will in effect strengthen all the connections roughly in proportion to their relative weights as they existed at the onset of sleep. 
     A problem may arise if the connections to one of the second level neurons become too strong. These neurons fire more often than other neurons, thereby further increasing the strength of all connections abutting them. The condition is regenerative because connections abutting such favored second level neurons will strengthen more quickly than other connections. This in turn causes such favored states to be even more favored because of increased feedback between the first and second level neurons that represent those states. The strength of these connections may increase indefinitely at the expense of the inactive connections. Instead of contrast-enhancing the connectivity field, therefore, the network may lose some memory that is never exercised. 
     Advantageously, therefore, a network according to the invention includes a mechanism to detect the presence of favored states during sleep, and to check their regenerative advances to prevent such states from overwhelming the system. Preferably, the deep sleep mechanism observes the activity levels of one or more of the second level neurons and, if a favored neuron is detected, reverses the sense by which the connections to that neuron learn. That is, instead of strengthening the connections which are exercised, the learning algorithm will weaken those connections which are exercised. 
     In a preferred embodiment, the deep sleep mechanism generates an analog deep sleep signal coupled to each of the synaptic connections which abut the second level neuron(s) being watched by the deep sleep mechanism, and causes learning to slow more and more as such second level neurons become more favored. The learning sense passes through zero as the neurons become even more favored, and becomes increasingly negative as favoritism further increases. Such a mechanism may be connected to watch either a single second level neuron, in which case its deep sleep signal is connected to affect only the connections abutting that second level neuron, or it may be coupled to observe the outputs of all the second level neurons in the cluster, in which case the deep sleep signal is connected to all the connections abutting any of such second level neurons. Since the learning algorithm affects only those connections which are exercised, only those connections which form part of the favored states Will be affected by the deep sleep signal. in all cases, only the connections which couple the outputs of first level neurons to inputs of second level neurons (&#34;feed-forward&#34; connections) are affected by the deep sleep signal; connections in the opposite direction are allowed to strengthen to their strongest values. Further, as will be seen, the deep sleep mechanism enhances compliance with the Weber Law Rule. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     The invention will be described with respect to particular embodiments thereof, and reference will be made to the drawings in which: 
     FIG. 1 is a block diagram of a neural network according to the invention. 
     FIG. 2 is a functional block diagram of a neuron according to the invention. 
     FIG. 3 is a schematic diagram of a neuron according to the invention. 
     FIG. 4 is a schematic diagram of a feed-forward synaptic connection circuit. 
     FIG. 5 is a schematic diagram of a feed-back synaptic connection circuit. 
     FIG. 6 is a schematic diagram of an inhibitory connection circuit. 
     FIG. 7 shows waveforms of various nodes in the neuronal circuit of FIG. 3; 
     FIG. 8 shows a connectivity field developed by a network according to the invention in response to certain input patterns; 
     FIG. 9 is a schematic diagram of a deep sleep node pulse extender; 
     FIG. 10 is a schematic diagram of a special connection for use with a deep sleep neuron; and 
     FIG. 11 is a schematic diagram of a deep sleep neuronal circuit. 
    
    
     DETAILED DESCRIPTION 
     In FIG. 1 there is shown a block diagram of a neural network according to the invention. The apparatus shown in FIG. 1 may either be an entire neural network, or it may represent only a portion of a much larger neural network. The network includes two levels 10 and 12 of nodes or neurons, with a plurality of connection circuits coupling them together. The neurons in the first level 10 are organized into clusters 14-1, 14-2 and 14-3, each containing a single neuron 16-1, 16-2, and 16-3, respectively. The clusters 14-1, 14-2 and 14-3 could contain more than one neuron each, but for simplicity of illustration, only single neuron first level clusters are shown. 
     The second level neurons 12 are organized into a single cluster of mutually competitive neurons 18-4, 18-5, 18-6 and 18-7. The second level could include other clusters of neurons as well, but for simplicity only the one cluster is shown. 
     Each of the neurons 16 in the first level neurons 10 has an input connected through a respective isolation switch 20-1, 20-2 and 20-3 to respective environmental inputs 22-1, 22-2 and 22-3. As will be seen, the inputs 22 may either connect to the outputs of other neurons on a different level, or may connect through input circuitry of some kind to the external environment. Each of the neurons 16 also receives input signals from all of the second level neurons 18 in a manner described below, so that a large number of signals are received by each of the neurons 16. Semantically, it is equivalent to think of these signals as being provided to separate inputs of the neuron, and then combined, or as being combined and provided to a single input of the neuron. In the circuit embodiment described below, for example, all the input signals are connected together to a common circuit node. Each neuron in the first level 10 also includes a plurality of outputs which, similarly to the inputs, may be considered a single output. 
     The neurons 18 in the second level 12 are preferably the same as those in the first level 10. Like the neurons in the first level 10, the neurons in the second level 12 each have a plurality of inputs (which may be considered a single input) and a plurality of outputs (which may be considered a single output). The environmental output of each of the neurons 18 is coupled through a respective isolation switch 24-4, 24-5, 24-6 and 24-7 to a respective output circuit 26-4, 26-5, 26-6 and 26-7. 
     The output of each of the first level neurons 16 is coupled through feed-forward circuits 30 to inputs of each of the second level neurons 18. These circuits are referred to as feed-forward connection circuits since the destination neuron to which they connect, also referred to as post-synaptic neurons, are neurons from which environmental output signals are taken. Thus, the output of neuronal circuits 16-1 is coupled via feed-forward connection circuits 30-14, 30-15, 30-16 and 30-17 to the respective second level neurons 18-4, 18-5, 18-6 and 18-7. These are the only feed-forward connection circuits shown in FIG. 1, for simplicity of illustration, but it will be understood that each of the first level neurons 14-2 and 14-3 too, are coupled via respective feed-forward nodes to each of the four second level neurons 18. A total of 12 separate feed-forward connection circuits are included. 
     Similarly, each of the second level neurons 18 is coupled via feed-back connection circuits 32 to inputs of each of the first level neurons 16. Shown in FIG. 1 are feed-back connection circuits 32-41, 32-51, 32-61 and 32-71, coupling outputs of respective second level neurons 18-4, 18-5, 18-6 and 18-7 to the input of first level neuron 16-1. The output of each of the second level neurons 18 is also coupled via feed-back circuits (not shown in FIG. 1) to the input of each of the first level neurons 14-2 and 14-3. 
     The designators by which the various elements are referenced in FIG. 1 are chosen for convenience of this description. In particular, the first level neurons 16, and each of the elements in FIG. 1 which correspond one-to-one with the respective first level neurons 16, are given the suffixes 1, 2 and 3. Similarly, the second level neurons 18, and all of the elements in FIG. 1 which correspond one-to-one with the respective neurons 18, are given the suffixes 4, 5, 6 and 7. Thus seven neurons are shown; neurons 1, 2 and 3 are in the first level and neurons 4, 5, 6 and 7 are in the second level. 
     The connection circuits 30 and 32 are given suffixes which represent the source and destination of the information which they transmit. For example, feed-forward connection circuit 30-17 couples the output of neuron I to the input of neuron 7, and feed-back connection circuit 32-51 couples the output of neuron 5 to the input of neuron 1. In general, there are N neurons, M of which are in the first level and N-M of which are in the second level. Feed-forward connection circuits 30-ij couple the output of each i&#39;th neuron to the input of each j&#39;th neuron, and feed-back connection circuits 32-ji couple the output of each j&#39;th neuron to the input of each i&#39;th neuron, i=1, . . . ,M and j=M+1, . . . ,N. 
     The isolation switches 20-1, 20-2 and 20-3 are all controlled by a common buffer 36, the input of which is connected to receive a network mode signal. Similarly, isolation switches 24-4, 24-5, 24-6 and 24-7 are all controlled by a common buffer 38, also connected to receive the network mode signal. In both cases the connection is such that the isolation switches are open when the mode signal is inactive (indicating that the system is &#34;asleep&#34;), and closed when the mode signal is active (indicating that the system is &#34;awake&#34;). The network mode signal is generated by circuit 40, which may do so periodically, or only in response to a manual input. 
     In order to implement the deep sleep mechanism, the network of FIG. 1 further includes a deep sleep neuron 50. As will be seen, the deep sleep neuron 50 is similar, but not identical, to the neurons 16 and 18. Signal inputs to the deep sleep neuron 50 are coupled via respective &#34;special&#34; connection circuits 52-4, 52-5, 52-6 and 52-7 to the outputs of the respective second level neurons 18, and the signal output of deep sleep neuron 50 is coupled via a pulse extender 52 to respective deep sleep control inputs to each of the feed-forward nodes 30. The deep sleep neuro 50 also has an inhibitory input coupled to receive the network mode signal from mode signal source neuron 40. Also included in the network of FIG. 1, but not shown, are inhibitory connection circuits coupling the output of each of the second level neurons 18 to each of the other second level neurons 18. 
     As previously mentioned, unlike prior art networks, the signals transmitted between neurons represent analog activity levels rather than binary indications of whether a neuron is firing. According to an aspect of the invention, the analog activity level is represented as a binary pulse signal having an analog frequency. Preferably, the higher the frequency of the pulses, the higher the activity level represented. Additionally, the frequency of the output signal of any of the first level neurons never falls below a quiescent positive frequency. These features will become apparent following a description of the circuitry involved. 
     Neuron Circuit 
     FIG. 2 shows a functional diagram of a node or neuronal circuit 16 or 18 according to the invention. It comprises an input block 100, Which generates a time integration of the sum of the activity levels of the input signals to the neuron. The output of the input block is summed with other signals, in a summer 102 described below, and forms an input to a voltage-controlled oscillator (VCO) 104. For first level neurons, a constant term C is also connected to the neuron circuit, either as an additional input to the input block 100, or as an adding input of summer 102. The VCO 104 generates a signal having a frequency responsive to the voltage on its input. That signal is integrated by another integrator 106, also called a regulator, the output of which forms a subtracting input to summer 102. While this Figure shows how the neuron operates functionally, it will be understood that actual circuit implementations may not be neatly divisible into the functional blocks shown. 
     Three aspects of FIG. 2 should be noted. First, the output frequency of the neuron will increase rapidly when the sum of the frequencies of the input signals increases rapidly. Similarly, a sudden drop in the sum of the input frequencies will yield a relatively sudden drop in the neuronal output frequency. There will be some smoothing and delay introduced by the integration function of the input block 100, which will depend upon the time period over which the input signals are integrated. In general, the shorter the time period over which they are integrated, the more quickly and faithfully the output frequency of the neuron will follow the sum of the input frequencies. 
     Second, a sustained high activity level on the inputs of the neuron will not yield a sustained high activity level on the output, since the output of regulator 106 will gradually increase. Since the output voltage of regulator 106 subtracts from the output voltage of input block 100 by summer 102, a higher output voltage of regulator 106 will reduce the control voltage input to VCO 104 and thereby gradually return the output frequency of the neuron to a lower value. If the activity levels of the inputs to the neuron subsequently fall, the neuronal output frequency will fall initially, and then due to regulator 106, gradually return to a higher value. 
     Third, the function of regulator 106 does not completely counteract steady-state frequency changes on the neuronal output; it merely tempers them gradually over time. Thus, the inclusion of a constant term as an input to input block 100 or to summer 102 in first level neurons ensures that the steady state output frequency of the neuron will never fall below some quiescent value. 
     FIG. 3 is a circuit diagram of a suitable neuronal circuit implementing the functions of the neuron of FIG. 2. It consists of a summing capacitor 130 having one terminal tied to ground and the other terminal tied to an input 128 of the circuit. The input 128 is also coupled through a resistor 132 to the inverting input node 136 of a comparator 144. The inverting input node 136 is also connected through a resistor 134 to Vcc, and through a capacitor 138 to ground. The non-inverting input 146 of the comparator 144 is coupled through a series of three diodes 148 to the output node 150 of the comparator 144. The inverting input 136 of the comparator 144 is also connected through a resistor 140 and a parallel diode 142 to the output node 150 of the comparator 144. 
     The non-inverting input 146 of the comparator 144 is also connected through a resistor 158 to the output of another comparator 162. The inverting input of the comparator 162 is coupled through a resistor 152 and a series diode 156 to the output 150 of the comparator 144. It is also coupled through a capacitor 168 and parallel resistor 160 to the output 164 of the comparator 162. The non-inverting input of comparator 162 is connected to the tap of a potentiometer 159, the two ends of which are connected, respectively, to Vcc and ground. The output node 150 of the comparator 144 is also connected to the inverting input of an output comparator 170, the non-inverting input of which is connected to Vcc/2. The output 172 of comparator 170 forms the output of the neuronal circuit 16 or 18. 
     Input 128 of the neuronal circuit 16 or 18 is both a temporal and spatial summing point, meaning that it integrates not only the variations in time of each input, but also the current contributed by all the inputs. Connections from other neurons to this neuron all converge at the point 128, where current from such connections are summed and integrated over time by capacitor 130. The time period over which these signals are integrated is a rather complex issue, but it will be seen that because of the current draw through resistor 132 into capacitor 138, the integration is heavily weighted in favor of input signals occurring since the most recent pulse of the neuronal output signal. 
     Resistor 132 transfers electrical charge from the integrating capacitor 130 to regenerative capacitor 138 which, together with comparator 1 44, resistors 140 and 158, and diodes 142 and 148, form a simple oscillator. It can be seen that when this oscillator is in its quiescent state, it nevertheless outputs narrow pulses at a slow rate (for example, one pulse in 5 milliseconds). This is because even if there is no input current charging the integrating capacitor 130, current is still available through leakage resistor 134 to slowly charge the capacitor 138. In a second level neuronal circuit, leakage resistor 134 would be omitted (open circuited) to avoid a non-zero quiescent output frequency. It will be seen further that the non-zero quiescent output frequency of first level neurons can be generated by another mechanism alternatively or additionally to leakage resistor 134, such as by providing a source of low frequency current pulses to the input circuit nodes 128 of each of the fist level neurons 16. 
     The operation of the oscillator can be understood by assuming first that the output 150 of the comparator 144 is high. In this state, diodes 142 and 148 are all reverse biased, the voltage at node 146 equals that at node 164, and capacitor 138 begins to charge through resistor 134 (and also through resistor 132 if the voltage on the summing capacitor 130 is currently higher than the voltage at node 136). After some time, the charge on capacitor 138 reaches a point where the voltage at point 136 exceeds that at point 146. This causes comparator 144 to reverse its output voltage, and pull down toward ground the voltage at its output point 150. This action forward biases the diodes 142 and 148, pulling down the voltage at point 146 to a low threshold voltage V thL  determined by the low output voltage of comparator 144 and the three diodes 148 Diode 142 at this point is also forward biased, beginning a discharge of the capacitor 138 through the resistor 140. Gradually, the voltage on the capacitor 138 will decrease until it falls below the low threshold voltage V thL  on the node 146, at which time the comparator 144 will again reverse the voltage on its output 150 and raise it to a high level. Thus the amount of time for which the voltage at the output node 150 of the comparator 144 remains low, depends on the time constant of capacitor 138 With resistor 140. That time period is also lengthened to the extent of the current charging the capacitor 138 via resistors 132 and 134, but desirably the resistor values are chosen so as to minimize this influence. 
     When the voltage at the output node 150 of comparator 144 goes high, the capacitor 138 is once again gradually recharged by the currents through resistors 132 and 134, thereby repeating the cycle. It can be seen that the voltage output of comparator 144 is normally high except for a sequence of brief, low-going pulses. The duration of each of the pulses is substantially fixed and preferably kept short by choosing a small value for resistor 140. The duration of the high levels of the pulses is typically relatively much longer, and depends on the temporal and spatial sum of the input voltages stored on the summing capacitor 130: the higher the average voltage of the inputs, the shorter the duration of the high levels of the pulses on the output 150 of the comparator 144. Even with no input signal at all, however, the duration of the high levels is never permitted to exceed some maximum length, due to the charging current arriving through leakage resistor 134. 
     The comparator 170 operates merely to reverse the sense of the pulses on node 150, thereby generating at the output node 172 of the neuron a voltage signal consisting of a series of short high-going pulses separated by a time period dependent upon the average input activity level and the current through leakage resistor 134. Thus prior to consideration of the function of the regulator circuit in the neuron 16, the output of the neuron 16 will have a pulse frequency (activity level) responsive to a recent time average of the sum of the activity levels of the inputs to the neuron 16. For example, a sudden increase in the voltage at input node 128 will cause the neuron to &#34;fire&#34;, a condition characterized by a temporary increase in the rate of the output pulses. 
     Maintaining the increased voltage level at point 128, however, does not maintain the increased rate of neuronal output pulses because the output pulse rate is also controlled by a regulating circuit composed of operational amplifier 162, capacitor 168, resistors 152 and 160 and diode 156. The voltage at point 164 is the high threshold voltage V thH  of the oscillator, representing the voltage to which the capacitor 138 must charge in order for the voltage at point 136 to exceed that at point 146. It tends to increase as the output frequency increases because of the charging action of diode 156 and resistor 160 on capacitor 168. This effect eventually decreases the output frequency. Thus, the neuron is sensitive not only to the instantaneous voltage level at its input summing junction 128, but also to changes in that voltage level. 
     The potentiometer 159 defines a resting potential for the regulator circuit of the neuron. In an integrated implementation, this potentiometer could of course be replaced by fixed resistors. In the absence of pulses on the output node 150 of the comparator 144, resistor 160 discharges the capacitor 168 to thereby cause the voltage at point 164 to decay toward the resting potential. Resistor 152 and diode 156 pull down the point 166 whenever the point 150 pulses down. This charges capacitor 168 and raises the high threshold voltage at point 146. 
     Feed-Forward Synaptic Connection 
     In FIG. 4 there is shown a schematic diagram of a circuit suitable for implementing one of the feed-forward synaptic connections 30-ij. It is assumed that the synaptic connection shown in FIG. 4 couples the output signal of first level neuron 16-i to the input of a second level neuron 18-j. 
     At the heart of the feed-forward synaptic connection 30-ij is a capacitor 124, which stores a charge indicative of the connectivity weight w ij  by which the information will be transferred. One terminal of the capacitor 124 is connected to ground, and the other terminal is connected through a unity gain buffer 226 to a first terminal 228 of a switch 278. The opposite terminal of the switch 278 is a Voltage control node 282, and is also connected to one terminal of a switch 280. The other terminal of the switch 280 is connected to the tap of a potentiometer 274, one terminal of which is connected to ground and the other terminal of which is connected through a resistor 276 to Vcc. 
     The signal output of the first level neuron 16-i connects to a signal input terminal 206 of the feed-forward synaptic connection circuit 30-ij . The signal input terminal 206 is connected through a comparator 208 to a circuit node 210. Comparator 208 serves as an impedance transducer and presents a very high input impedance to the pre-synaptic neuron 16-i. Such a high impedance input is important because there may be thousands of connection circuits connected to the same neuronal output. The impedance transducer 208 has its non-inverting input connected to the signal input terminal 206 and its inverting input connected to a Vcc/2 voltage. 
     The circuit node 210 is connected to the control terminal of the switch 278, and also through a voltage inverter 275 to the control terminal of switch 280. It can be seen that when a high level voltage is present on the signal input terminal 206, switch 278 will close and the voltage control node 282 will carry a voltage equal (except for an offset introduced by the follower 226) to the voltage on capacitor 124. The switch 280 will be open. When the signal input terminal 206 carries a low voltage, switch 278 will be open and switch 280 will be closed. Voltage control node 282 will carry a voltage, preferably a low voltage as explained below, dependent upon the resistor divider formed by resistor 276 and potentiometer 274. Any current through resistor 286 does not influence this low voltage since the value of resistor 286 is much greater than the value of resistors 274 and 276. Thus, the voltage on voltage control node 282 will oscillate between the low reference voltage and the voltage on capacitor 124, at a rate equal to the pulse rate of the signal output of first level neuron 16-i. The purpose of the adjustable voltage divider formed by resistors 274 and 276 is to provide a low voltage offset because the offset of follower 226 does not permit node 228 to go all the way to zero when the voltage on capacitor 124 is zero. The voltage divider therefore allows the transconductance converter, described below, to provide a real zero current when capacitor 124 has zero voltage. 
     The voltage control node 282 forms an input to a transconductance converter formed by voltage follower 200 and operational amplifier 294 as active components. The voltage control node 282 is connected through a resistor 284 to the non-inverting input 290 of op-amp 294, the output 298 of which is coupled through resistor 292 to the inverting input 288 of the op-amp 294. The inverting input of op-amp 294 is also connected through a resistor 286 to the tap of potentiometer 274. The output 298 of the op-amp 294 is further coupled through a resistor 202 to the non-inverting input of the voltage follower 200, the output of which is connected directly to its own inverting input. The output of follower 200 is further connected through a resistor 296 to the non-inverting input 290 of the op-amp 294. The non-inverting input of follower 200 also forms the signal output terminal 204 of the feed-forward synaptic connection circuit 30-ij, which is coupled to the signal input of the second level neuron 18-j. 
     The transconductance amplifier, as so constructed, injects current into the summing junction 128 (see FIG. 3) of the second level neuron 18-j to which it connects. The amount of current which is injected is controlled by the voltage across resistor 202, independently of voltage variations at the current output terminal 204. Follower 200 senses the voltage at point 204 and presents the same voltage to a voltage divider formed by resistors 296 and 284, the tap of which is connected to the noninverting input of op-amp 294. The voltage at point 298 is fed back negatively to op-amp 294 through another voltage divider formed by resistors 292 and 286. Op-amp 294 maintains the voltage at point 298 such that the voltages at points 288 and 290 are approximately equal. It can be seen, therefore, that the current through resistor 202 is controlled by the voltage at voltage control node 282. 
     Thus, in conjunction with the storage capacitor 124, the switch 278 and 280, and associated circuitry previously described, the transconductance amplifier generates at the synaptic output terminal 204 a series of current pulses. These current pulses have a frequency and duration equal to that of the voltage pulses entering the feed-forward synaptic connection circuit 30-ij at signal input terminal 206. The current level at the top of each pulse is dictated by the voltage on storage capacitor 124, and the current level between pulses is very small or zero. It can be said that the output node 204 of the feed-forward synaptic connection circuit 30-ij carries an activity level approximately equal to w ij  a i , where a 1  is the activity level output of the first level neuron 16-i to which the input 206 of feed-forward synaptic connection circuit 30-ij is connected. 
     The network of the present embodiment learns categories by strengthening those connectivity weights w ij  in feed-forward synaptic connection circuits 30-ij, to the extent of simultaneous high activity levels on the outputs of both the first level neuron 16-i and the second level neuron 18-j. If both neurons fire simultaneously, then the connectivity weight of the feed-forward synaptic connection circuit which couples them should be strengthened. The learning algorithm also holds that the connectivity weight w ij  should be weakened to an extent governed by the concurrence of a high activity level on the second level neuron 18-j and a low activity level on the first level neuron 16-i. If the activity level output of the second level neuron 18-j has a low activity level, then all the connectivity weights wij, i-1, . . . ,M, should remain substantially unchanged regardless of the activity level of the outputs of first level neurons 16-i. The connectivity weights may weaken nevertheless, due to leakage in memory capacitors 124. 
     Accordingly, a learning circuit is included in the feed-forward synaptic connection of FIG. 4 for charging (or discharging) storage capacitor 124 only during the high levels of the pulses of the output of the second level neuron 18-j. If the voltage output of the first level neuron 16-i (as extended by an extender described below) is high during a second level pulse, capacitor 124 will gain charge. If it is low, capacitor 124 will lose charge. 
     The output of impedance transducer 208, in addition to being connected to the control terminal of switch 278, is also connected through a diode 211 in series with a diode 212 to the non-inverting input 213 of a comparator 217. The non-inverting input 213 of the comparator 217 is also coupled to ground by a resistor 216 in parallel with a capacitor 218. A circuit input node 236 to the feed-forward synaptic connection circuit of FIG. 4 is coupled to the output of a deep-sleep neuron (DSN) extender, described below. The circuit node 236 is coupled through another impedance transducer 238 and series resistor 239 to the inverting input 214 of the comparator 217. The inverting input 214 of the comparator 217 is also connected through a resistor 240 to Vcc. The output 219 of comparator 217 is coupled through a switch 220 in series with a resistor 221 to the non-grounded terminal of the storage capacitor 124. The control terminal for switch 220 is connected to the output 242 of yet another impedance transducer 241, the input of Which is connected to a circuit input node 237. The circuit input node 237 receives the output signal from the second level neuron 18-j, which is the very same second level neuron which has an input connected to the output 204 of the feed-forward synaptic connection circuit 30-ij. 
     The pulse extender portion of the feed-forward synaptic connection circuit operates as follows. If the input 236 is low, resistors 239 and 240 form a voltage divider which provides a threshold voltage at the inverting input 214 of the comparator 217. When a pulse arrives at the signal input terminal 206, diodes 211 and 212 charge the capacitor 218 and thereby rapidly raise the voltage at the non-inverting input 213 of comparator 217, until it exceeds the threshold voltage at the inverting input 214. The output voltage 219 of the comparator 217 then goes high and remains high until resistor 216 discharges capacitor 218 sufficiently to drop the voltage of the non-inverting input 213 to a level below the threshold level on inverting input 214. The output pulse width is therefore determined by resistors 239, 240 and 216 and capacitor 218. 
     In accordance with a feature of the invention described in more detail below, it will further be seen that a high voltage level on circuit input node 236 will disable the pulse extender and force a low voltage on the output 219 of the comparator 217. This can be seen by noting that the output of impedance transducer 238 will be high, thereby bringing the threshold voltage at inverting input 214 to a high level. The voltage at the non-inverting input 213 should never exceed this high threshold voltage, due to the double diode voltage drop introduced by diodes 211 and 212, and the output voltage of the comparator 217 will therefore remain low. In accordance with an aspect of the invention described below, therefore, high-going pulses on the DSN extender output will override the storage capacitor charging action that would otherwise take place when high levels occur on the outputs of both the first and second level neurons 16-i and 18-j simultaneously, and instead cause the storage capacitor 124 to discharge through resistor 221 at a rate dictated by the frequency of pulses output from the second level neuron 18-j. 
     Feed-Back Synaptic Connection Circuit 
     FIG. 5 shows a schematic diagram of a feed-back synaptic connection 32-ji. It is very similar to the feed-forward synaptic connection of FIG. 4, with certain exceptions. First, instead of being controlled by the first level neuron 16-i the switches 278 and 280, which determine the frequency of the output pulses of the synaptic connection, are controlled by the output signal from second level neuron 18-j. This is accomplished by connecting the control input of switch 278, and the inverting input of voltage inverter 275, to the output of impedance transducer 241 instead of to the output of impedance transducer 208. The output of impedance transducer 241 remains connected to the control input of learning switch 220 for reasons which will be apparent below. 
     Second, instead of being connected to the signal input of second level neuron 18-j, the output 204 of the feed-back synaptic connection 32-ji is connected to the signal input of the post-synaptic first level neuron 16-i. 
     The learning algorithm for a feed-back synaptic connection 32-ji is the same as the learning algorithm for the feed-forward synaptic connection coupling the same two neurons. That is, the connectivity weight w ji  should strengthen to the extent of concurrency of high activity levels in both the first level neuron 16-i and the second level neuron 16-j, and should weaken to the extent of concurrence of a high activity level in the output of the second level neuron 18-jand low activity level in the output of the first level neuron 16-i. Thus the input 206 of the pulse extender circuit in the feed-back synaptic connection circuit 32-ji remains coupled to the output of the first level neuron 16-i, as in the feed-forward synaptic connection circuit, and the circuit input node 237 in the feed-back synaptic connection circuit 32-ji, which controls the learning switch 220, remains coupled to the output of the second level neuron 18-j. 
     In one further difference between the feed-back synaptic connection circuit and the feed-forward synaptic connection circuit, as explained below, sleep refresh of the connectivity weights w ji  in the feed-back synaptic connection circuits does not need to be checked by the deep sleep neuron. The feed-back synaptic connection circuit shown in FIG. 5 therefore omits the input 236, Which in the feed-forward synaptic connection circuit of FIG. 4 is connected to the DSN extender output, and instead couples the resistor 239 to ground. The resistor divider formed by resistors 239 and 240 therefore provide a fixed threshold voltage to the inverting input of comparator 217, thereby ensuring that the pulse extender is never disabled. 
     It should be noted that the switch 220 serves both as a refresh and a learning switch. Consequently, as far as any connection circuit is concerned, it does not matter whether the network of which it is a part is sleeping or learning; it functions the same way whether in sleep or learning (awake) mode. 
     Inhibitory Connection Circuit 
     FIG. 6 illustrates an inhibitory connection circuit. As previously mentioned, all the neurons in a second level cluster are mutually competitive. Mutual competition is accomplished by providing an inhibitory connection between the output of each of the second level neurons in the cluster and the input of every other one of the second level neurons in the cluster. The inhibitory connection circuit has a fixed connectivity weight which is negative, thereby making the activity level of the output of each of the second level neurons in the cluster negatively responsive to the activity level of each of the other second level neurons in the cluster. Thus, an inhibitory connection circuit such as that shown in FIG. 6 connects the output of each k&#39;th second level neuron to the input of each j&#39;th second level neuron, j=M+1, . . . N, k=M+1, . . . ,N, k≠j. 
     Inhibitory connection circuit 31-jk couples the output of second level neuron 18-j to an input of second level neuron 18-k. Thus the output of second level neuron 18-j connects to a circuit input node 344 of the inhibitory connection circuit 31-jk, and the output node 382 of the inhibitory connection circuit 31-jk connects to an input of second level neuron 18-k. Resistors 358, 360, 362 and 368 are all approximately equal, and the values of resistors 370 and 374 are approximately equal. The value of resistor 358 is proportional to the value of resistor 370 by a constant of proportionality K. The constant K determines the synaptic connectivity of the circuit because, as will be shown, it determines how much current is drawn from the post-synaptic neuron. 
     Inverter 350 inverts pulses coming in at input 344, while inverter 346 inverts them back to their original polarity. Voltage follower 378 and op-amp 372 maintain the voltage drop across resistor 380 at zero when input 344 is low as follows. Between pulses, the voltage at point 348 is high while that at point 352 is low, so that both diodes 354 and 356 are forward biased. Because resistors 358, 362, 360, and 368 are all of equal value, and because op-amp 372 maintains the voltage at point 376 such that the voltages at points 364 and 366 are equal, the currents through resistors 370 and 374 must be equal. And, since R370 =R374, the voltage at point 376 (referred to herein as V376) must equal V382. Thus, between pulses, the inhibitory connection circuit of FIG. 6 supplies zero current to the post-synaptic neuron 18-k. 
     When a pulse occurs, point 348 is low while point 352 is high. Both diodes 354 and 356 are therefore reverse biased. Using Kirchoff&#39;s Law at point 364, we have 
     
         (VCC-V364) / R362=(V364-V376) / R370. 
    
     Similarly at point 366, 
     
         (V366-0) / R368=(V382-V366) / R374. 
    
     Because op-amp 372 keeps V376 such that V364 is equal to V366, the above equations can be combined and rearranged to yield 
     
         (1/K)* VCC=V382-V376. 
    
     When a pulse occurs, therefore, a current proportional to (1/K)Vcc is drawn from the postsynaptic neuron 18-k. 
     Network Operation 
     In order to understand the general operation of the network of FIG. 1, initially a single one 16-i of the first level neurons 16 and a single one 18-j of the second level neurons 18 to which it is coupled will be examined in isolation as if they formed a complete, albeit simple, network. The environment of this simple network consists of the rest of the much larger network. 
     As previously mentioned, neurons are electronic entities that output pulses all the time, even when they are in a quiescent state. A neuron is said to fire when the frequency or rate of the pulses it generates increases rapidly to a rate which is much faster than the quiescent rate. Recalling the operation of a neuron such as that in FIG. 3, it will be appreciated that a sufficiently quick increase in the input activity levels will cause the neuron to fire. For first level neurons 16, these input activity levels come either from environmental inputs or from the second level neurons 18 to which its inputs are coupled. Recalling the operation of feed-back connection circuit 32-ji, it will be appreciated that the activity level received by the first level neuron 16-i from a second level neuron 18-j equals w ji  f j , where w ji  is the connectivity weight of the connection from second level neuron 18-j to first level neuron 16-i, and where fj is the output frequency of second level neuron 18-j. Thus a high activity level on second level neuron 18- j can cause first level neuron 16-i to fire, but only if the connectivity weight w ji  is sufficiently strong. 
     For second level neurons 18, the input activity levels come from the first level neurons 16 to which its inputs are coupled. Like the first level neurons 16, it will be appreciated that a second level neuron 18-j can be caused to fire by a first level neuron 16-i, but only if the connectivity weight w ij  is sufficiently strong. 
     Consider first a two-neuron network in which the connectivity weights w ij  and w ji  are weak. In this situation there is no inherent reason for both neurons to fire simultaneously. At any given time, therefore, it is much more likely that neither or only one of the neurons is firing, than it is that both are firing. The timing of the pulses generated by first level neuron 16-i therefore will rarely correlate with the timing of the pulses generated by second level neuron 18-j, and the synaptic connectivity weights w ij  and w ji  will remain low. They will actually be reduced if the second level neuron 18-j is firing when the first level neuron 16-i is quiescent. 
     Now consider a two-neuron network in which the connectivity weights w ij  and w ji  are strong. In this situation a high activity level on either of the neurons strongly and positively influences that of the other neuron, significantly increasing the probability that they will fire in synchrony. (Note that the high activity levels of neither neuron will be sustained for long due to the dampening effect of regulator circuit 106 (FIG. 2) in each of the neurons.) Since the peaks of the voltage output waveforms of the two neurons will now occur simultaneously more often, the connections between them will strengthen. However, since there can be no regularity in the environment, there will nevertheless be intervals of time greater than the time constant of connection decay in which the two neurons fire less synchronously than is normally the case. In these intervals, according to the learning algorithm chosen and described above, and also according to the operation of the network implementation described above, the connectivity weights w ij  and w ji  will weaken. 
     In order to counter undesirable loss of memory due to this reason or to simple charge leakage, the inventive network includes a sleep mode into which the network is placed occasionally. The sleep mode is defined herein to refer to the physical isolation of a neural network from its environment such that all input to the network is shut off. Outputs may also be disconnected, since their activation during sleep has little meaning. As will be seen, the network will randomly traverse its state space on its own. 
     In the case of the two-neuron network, only four states exist. The first state is visited when both the first level neuron 16-i and the second level neuron 18-j are quiescent (&#34;off&#34;); the second, when the second level neuron 18-j is quiescent and the first level neuron 16-i is firing (&#34;on&#34;); the third, when the second level neuron 18-j is firing and the first level neuron 16-i is quiescent; and the fourth, when both neurons are firing simultaneously. 
     As previously discussed with respect to FIGS. 2 and 3, each neuron by itself is already an oscillator. As has been noted, for first level neurons, this primary oscillation occurs even when the neuron is in a quiescent state. The two neurons in the network and the two connections between them form still a third feedback oscillator that causes the frequency f of each neuron to oscillate. The spontaneous modulation of frequency, referred to herein as secondary oscillation, occurs as follows. Assume a start state in which both neurons are Off. The quiescent pulses from the first level neuron is temporally summed at the summing input of the second level neuron, thereby increasing the charge on summing capacitor 130 (FIG. 3) in the second level neuron. The magnitude of the increase brought about by each incoming pulse is proportional to the charge on the capacitor 124 (FIG. 4 or 5) of the feed-forward connection circuit. This increase helps to speed up the charging of capacitor 138 (FIG. 3) in the second level neuron through resistor 132 so that capacitor 138 becomes charged more and more quickly on each of the primary oscillation cycles of the second level neuron. Since the rate of charge build-up on capacitor 130 is proportional to the charge on connectivity weight capacitor 124 in the feed-forward connection circuit, the rate of frequency build-up in the second level neuron is also proportional to the charge on connectivity weight capacitor 124. The increase in frequency of the primary oscillation in the second level neuron is fed back via the feed-back connection circuit to the first level neuron, thereby increasing the frequency of the primary oscillation in the first level neuron. This in turn causes further speed-up in the charging of capacitor 138 in the second level neuron. An avalanche effect results until both neurons reach high frequencies. Both neurons then are said to be on. 
     Each neuron will stay on until the charge on its respective capacitor 168, in the regulator circuit, builds up to that point at which the frequency of the primary oscillation drops back toward its respective quiescent value. The secondary oscillatory cycle then repeats itself. The rate at which avalanche will occur (i.e. the frequency of the secondary oscillation) depends on the initial charge on capacitor 124 in each connection circuit. 
     In FIG. 7 there are shown several graphs of neuron activity when the summed input voltage on capacitor 130 increases and then decreases. Curve 400 shows the voltage V130 on capacitor 130. Neuron response, measured as the rate f of pulses produced on circuit output node 172, is shown in curve 402. The regulator voltage V164 is shown as curve 404. Note that although f increases with V130, the decay of f can be more rapid than that of V130 because the rise in the regulator voltage V164 of the neuron has an inhibiting effect (decreases neuron pulse frequency f). Thus a neuron has two main internal states: an inhibitory state in which capacitor 168 is charged, V164 is high, and response to the summed input voltage V130 decreases; and an accommodation state in which V164 is low and the neuron is sensitive to changes in the summed input voltage V130. The exact length of the inhibitory state is not important, as explained in more detail below, as long as it is long enough to discourage firing of the same neurons twice in a row. 
     It can be seen that the secondary oscillation is the main mechanism by which capacitor 124 retains its charge. If the connection strength between the two neurons in the two-neuron network is high at the onset of sleep, the first state (off-off) and the fourth state (on-on) are favored by the network over the second state (on-off) and the third state (off-on). The outputs of the two neurons increase and then decrease in frequency, firing simultaneously more often than not, thereby preserving and strengthening the previous synaptic strengths which caused such harmonious firing originally. With every resonant firing, the charge on memory capacitor 124 is refreshed through switch 220 (FIG. 4 or 5). 
     If the connection strength between the two neurons in the network is low at the onset of sleep, then the second and third states will be favored. The second state (on-off) has little if any effect on the connectivity weights, and the third state (off-on) will actually weaken the connectivity strengths. In general, the rate of simultaneous firing depends on the initial connectivity weights w ij  and w ji , and the time constants by which these connectivity weights change. Thus, since the secondary oscillations by which the network wanders through its state space occur spontaneously when the network is isolated from its environment (asleep), the contrast between the various connectivity values throughout the connection field is thereby increased during sleep. Any washing out of responses which have been learned but not recently visited during wakefulness, are counteracted. 
     Multi-Neuron Network 
     Referring to the network of FIG. 1, a network will be considered which includes three first level neurons 16 and four second level neurons 18. Suppose that this network is taught to distinguish four input pattern combinations as follows: 
     
         ______________________________________                                    
Input        First Level Neuron                                           
Pattern      16-1      16-2       16-3                                    
______________________________________                                    
No. 1:       OFF       OFF        ON                                      
No. 2:       OFF       ON         ON                                      
No. 3:       ON        OFF        ON                                      
No. 4:       ON        ON         ON                                      
______________________________________                                    
 
    
     Because all the second level neurons 18 in the network belong to the same cluster in which all neurons are mutually competitive, it can be expected that each one of the four second level neurons 18 will come to recognize a different one of the four input patterns. 
     FIG. 8 illustrates connections that may develop as a result of learning the four patterns. In this Figure only the strong connections are shown, and the feedforward connection and the feed-back connection are both shown together as a single line. It can be seen that Input Pattern No. 3 (ON-OFF-ON), for example, will cause resonance between second level neuron 18-5 and both first level neurons 16-1 and 16-3, indicating that second level neuron 18-5 recognizes the third input pattern. That is, when the third input pattern is presented to the first level neurons 16, second level neuron 18-5 is the neuron that will fire. Similarly, it can be seen that second level neuron 18-7 recognizes the first input pattern (OFF-OFF-ON), second level neuron 18-6 recognizes the second input pattern (OFF-ON-ON), and second level neuron 18-4 recognizes the fourth input pattern (ON-ON-ON). Depending on the sequence in which the input patterns were presented to the network, and depending on the pre-learning initial connectivity values, a different correlation between input patterns and second level neurons could have been learned instead. 
     Note that the connections between second level neuron 18-6 and each of the first level neurons 16-2 and 16-3 will be weaker than that between second level neuron 18-7 and first level neuron 16-3. Otherwise, if input pattern No. (off-off-on) is applied to the input neurons 16, then second level neuron 18-6 will receive a signal as strong as that received by second level neuron 18-7. Depending on past history and other factors, second level neuron 18-6 may win out over second level neuron 18-7. This is undesirable since second level neuron 18-7, judging by the connections shown in FIG. 8, is the proper neuron to react to input pattern No. 1. On the other hand, if the connections abutting second level neuron 18-6 are weaker than the connection abutting second level neuron 18-7, second level neuron 18-7, properly, will be the neuron which will react to input pattern No. 1. 
     If the network of FIG. 8 is now put to sleep, secondary oscillations will occur and stable states will be visited by the resonant firing of neurons. If the internal inhibitory states of the neurons are made long enough, visiting of the same state twice in a row will be discouraged and the network will visit different ones of the states in some pseudo-random sequence. As each state is visited, the resonant firings of the first level neurons representing a learned input pattern, with the secondlevel neuron which recognizes that input pattern, will tend to further strengthen the connections between such first and second level neurons. In effect, therefore, the normal activity of the inventive network during sleep automatically refreshes the connectivity weights stored in the capacitors in feed-forward connection circuits 30 and in feed-back connection circuits 32. Applicant has observed that the connectivity weights tend to increase in substantially constant proportion to each other, thereby properly maintaining and improving the contrast between the various connectivity values rather than smoothing them all toward an equal level. 
     Deep Sleep 
     When the system is awake, neurons fire primarily because of environmental stimulus to the system. When the system is asleep, however&#39; neurons fire primarily because of secondary resonance or oscillation. Although secondary oscillations can occur even while awake, they are in the background, and firings due to them are much less intense than firings due to environmental input, as explained below. Firings due to environmental stimulus are always of a higher intensity than those due to secondary oscillation, so to distinguish between the two types of firings, the first is referred to as High Intensity Firings (HIF), and the second, Low Intensity Firings (LIF). The difference in intensity arises because an environmental input, when active, is normally sustained at a high frequency, and at such sustained frequency the regulator circuits 106 in both first and second level neurons are too slow to quickly dampen the neuronal response. On the other hand, firings due to secondary oscillations can be dampened as soon as the avalanche starts, unless the feed-forward connections have grown so large as to overcome the dampening effect of the regulator circuit. It is therefore readily appreciated that HIF activity is prevalent among neurons when the network is awake, and when the network sleeps, HIF is rare and LIF dominates. 
     It is preferable that all the learned states in a network be visited with equal probability while the network sleeps, in order to ensure accurate refreshing of the entire connectivity memory terrain. It is possible, however, for connections to increase their affectively to such levels that too much positive feedback between first and second level neurons results and HIF activity occurs even in the absence of environmental input. This phenomenon is referred to herein as deep sleep. 
     When awake, a network according to the invention exercises some connections with HIF activity more often than others. Some neurons may fire most of the time, while others may not fire at all. As a consequence, only connections that connect neurons that do fire get exercised, thereby increasing their connectivities slightly. Also, if the system learns a new category in the course of its wakefulness, new connections may be established. Connections between neurons that do not fire at all decay slowly; and connections exercised by HIF activity grow faster than those exercised by LIF activity. 
     Therefore, at the onset of sleep, network states that are visited by HIF activity while awake are favored over those that are not visited at all. During sleep, second level neurons that represent favored states fire more often (with LIF) than do unfavored ones, thereby increasing the affectivity of all connections to or from that favored neuron. The affectivity of these connections may increase indefinitely at the expense of the inactive connections. Instead of contrast-enhancing the connectivity field, the system may lose some memory that is never exercised. 
     In an aspect of the invention, therefore, apparatus is included in the network for automatically detecting and correcting a deep sleep condition. Referring again to FIG. 1, the network further includes a deep sleep neuron (DSN) 50 having inputs coupled via special connections 52-4, 52-5, 52-6 and 52-7 to the outputs of respective second level neurons 18-4, 18-5, 18-6, 18-7. The deep sleep neuron 50 further includes an inhibitory input coupled to receive a signal from the same mode signal source neuron 40 that operates the isolation switches 22 and 24. The output of DSN 50 is coupled through a DSN pulse extender 52 to the OSN extender signal input 236 (FIG. 4) of each of the feedforward synaptic connection circuits 30. 
     The DSN 50 is similar in design to that of each of the neurons 16 and 18 shown in FIG. 3, except that the regulator circuit is omitted. A DSN 50 is shown in FIG. 11, and as can be seen, the resistor 158 now couples the threshold circuit node 146 to the tap of a potentiometer 502 connected between Vcc and ground, instead of to the output 164 (FIG. 3) of a regulator circuit. The leakage resistor 134 (FIG. 3) is also omitted from the DSN 50. The activity level output of network mode source neuron 40 provides a strong inhibitory (i.e., negative) signal to the circuit input node 128 of the DSN 50 when the network is awake, and provides little or no signal when the network is asleep. In effect, therefore, network mode source neuron 40 inhibits the operation of DSN 50 when the network is awake. 
     The DSN pulse extender 52 can be implemented as shown in FIG. 9. It is essentially the same as the pulse extender on the learning input of the feed-back synaptic connection circuit shown in FIG. 5. The operation of the pulse extender of FIG. 9 will not be described here in detail, since it has already been described with respect to the synaptic connection circuits. It is noted, however, that instead of being connected to a fixed voltage divider as in the circuit of FIG. 5, the inverting input of the DSN pulse extender 52 is connected to the tap of a potentiometer. This permits experimental adjustment of the length of each pulse in the DSN signal. In an integrated implementation, of course, this potentiometer could be replaced by a fixed pair of resistors. 
     It is noted further that the function of the DSN pulse extender 52, instead of being performed once at the output of DSN 50, could instead be performed at the deep sleep signal input 236 of each of the feed-forward synaptic connection circuits 30-ij. This, however, would involve an unnecessary duplication of circuitry. The pulse extender at the signal input terminal 206 of each of the feed-forward and feed-back synaptic connection circuits is duplicated in each of these connection circuits only because, at least in the case of the feed-forward synaptic connection circuits 30-ij, the raw pulse information from the first level neuron is needed as well to operate the switches 278 and 280 (FIG. 4). Similarly, in the feed-back synaptic connection circuit 32-ji, the raw pulse information from the second level neuron 18-j is needed to operate the same switches and also the learning switch 220 (FIG. 5). However, it will be appreciated that the circuitry could be modified to provide the pulse extender on the output of each of the neurons, and provide the raw pulse information as an additional output of each of the neurons. 
     FIG. 10 shows a schematic diagram of one of the special connection circuits 52-j, coupling the output of one of the second level neurons 18-j to the input of the deep sleep neuron 50. Much of it is identical to the circuitry of either of the synaptic connection circuits 30 or 32, and an explanation of those aspects of the circuit will not be repeated here. In particular, the operation and structural connection between the circuit elements which generate the current pulses on the special connection output terminal 204 at a frequency dictated by the frequency of voltage pulses arriving from the output of second level neuron 18-j, and with a magnitude governed by the voltage on storage capacitor 424, is the same as that of feed-back synaptic connection circuit 32-ji (FIG. 5). The learning mechanism of the special connection 52-j is, however, different than that of feed-back synaptic connection circuit 32-ji. Instead of being connected through a switch 220 to the output of a pulse extender (FIG. 5), the terminal of resistor 221 opposite that connected to the storage capacitor 424 is connected to the cathode of a diode 420, the anode of which is connected to the output of impedance converter 241. The diode 420 is also shunted with a resistor 425. 
     The operation of the deep sleep apparatus will be described with reference to FIGS. 10, 3, 9 and 4 (listed in order of information flow). Initially, at the onset of sleep, it is assumed that the charge level on each of the storage capacitors 424 in the special connections 52 are low. Thus the output terminal 204 of each of special connections 52 carries a series of current pulses at a rate equal to the rate at which voltage pulses are arriving from second level neuron 18-j, but with a very small magnitude because of the small charge in connectivity capacitors 424. These current pulses are all summed at the input terminal 128 of the DSN 50, but because they are so small in magnitude, do not influence the DSN 50 to fire. The output terminal 172 of the DSN 50 therefore carries voltage pulses at the quiescent rate, which, since the DSN 50 has no leakage resistor on its input, is substantially zero. Any pulses output by the DSN 50 are extended by DSN pulse extender 52 and provided to the input terminal 236 of all of the feed-forward synaptic connection circuits 30. 
     Each of the feed-forward synaptic connection circuits 30-ij continues to operate almost completely as previously described, strengthening its connectivity weight by adding charge on capacitor 124 to the extent of occurrence of high activity levels on both the first level neuron 16-i and the second level neuron 18-j and weakening the connectivity weight by reducing the charge stored on connectivity capacitor 124 to the extent of concurrence of a high activity level on the second level neuron 8- j  and a low activity on the first level neuron 6-i. At whatever the present frequency of the DSN 50, a pulse will arrive on input terminal 236 to the feed-forward synaptic connection circuit 30-ij, and during the high voltage portion of the pulse, as previously explained, a low, voltage will be forced at the output 219 of the comparator 217. Should such a pulse arrive at the same time that a pulse arrives on terminal 237 from the second level neuron 18-j, switch 220 will close briefly and connectivity capacitor 124 will be slightly discharged through resistor 221. Since the pulses arriving from second level neuron 18- j are not extended, however, as long as the deep sleep signal remains at a low frequency the discharge of capacitor 124 will be negligible. 
     If the second level neuron 18-j to which the special connection 52-j is connected never fires, the special connection 52-j will never influence the DSN 50 to fire. The quiescent pulses coming in on terminal 237 of special connection 52-j charge the capacitor 424 in that circuit slightly, but the lengthy low voltage levels of the signal arriving from second level neuron 18-j permit the capacitor to discharge again through resistor 425. Should the second level neuron 18-j fire, the connectivity of special connection 15- j  will increase somewhat. It will then decrease again, however, if the second level neuron 18-j then quiets down. It can be seen that if the second level neuron 18-j fires with too high a frequency (HIF), or fires too frequently, the connectivity capacitor 124 in the special connection 52-j will charge up. This will send extended higher level current pulses to the input summing terminal of the DSN 50 and will soon cause the DSN 50 to fire. When it does, pulses will arrive at the terminal 236 of each of the feed-forward synaptic connection circuits 30 at a high rate, thereby causing the discharge of connectivity capacitor 124 in the feed-forward synaptic connection circuits in correspondence with the rate at which pulses are arriving from the post-synaptic second level neuron 18 to which it is connected. In effect, therefore, instead of strengthening the connectivity weight w ij , activation of the deep sleep neuron 50 causes a weakening of the connectivity weight w.sub. ij in correspondence with the activity level of the second level neuron 18-j. The sense of learning has temporarily been reversed. 
     As the connectivity weight w ij  weakens. the frequency at which, and the intensity with which, the second level neuron 18-j fires, will drop. As it does, the connectivity weight in the special connection 52-j will also drop, and the DSN 50 will return to its quiescent output frequency. This returns the feedforward synaptic connection circuits 30 to their normal sense of learning. 
     In the above description a single DEN 50 controls all the second level neurons 18 in the cluster. If a single one of the second level neurons becomes favored, the sense of learning will be reversed in all of the feed-forward synaptic connections feeding all of the second level neurons in the cluster. This is not a problem, however, since not all of these connections are actually discharged. Charging or discharging of a connectivity capacitor occurs only when the corresponding second level neuron fires, and in deep sleep it is natural that only the favored second level neurons continue to fire. Only the favored connections, therefore, are actually weakened. It will be appreciated that instead of providing one DSN for an entire cluster of second level neurons, each of the second level neurons 18-j may have associated with it its own DSN that controls only the feed-forward synaptic connections associated with the neuron 18-j. Note also that feed-back synaptic connections are allowed to strengthen to their maximum levels even during deep sleep. Note further that a simpler deep sleep mechanism may be used instead of that described above. For example, a single all-or-nothing connection may be provided from the output of each second level node 18-j to a &#34;reverse learning sense input of all of the feed-forward synaptic connection circuits abutting neuron 18-j. 
     Deep Sleep Naturally Encourages Weber Law Compliance 
     The Weber Law rule can be described as follows. Consider two distinct system states, S1 and S2, each represented by a second level neuron. These two states are such that the set of first level neurons that triggers one state (S1) is a subset of the set that triggers the other state (S2). In other words, all first level neurons in the set that triggers S1 are also in the set that triggers S2, but some first level neurons in the set that triggers S2 are not in the set that triggers S1. The Weber Law Rule states that the feed-forward connections for S1 should be stronger than those for S2. 
     The deep sleep mechanism regulates the feed-forward connections such that this rule is followed. In general, the deep sleep mechanism regulates the feed-forward connections such that connections for more complicated system states (i.e., those states involving activation of a larger number of first level neurons) are weaker than those for less complicated system states. 
     To show this, assume that all feed-forward connections are initially equal in strength. This means that second level neurons that represent more complicated patterns will receive more current at their summing points than those second level neurons that represent less complicated states. It follows that the more complicated states will be visited more often than less complicated ones, and that eventually the process of connectivity strength reduction through deep sleep will reduce the probability of the more complicated states by reducing the strength of the feed-forward connections which feed the second level neurons representing the more complicated states. Thus the deep sleep mechanism of the inventive network naturally encourages compliance with the Weber Law Rule without requiring additional mechanisms. 
     The invention has been described with respect to particular embodiments thereof, and it will be understood that numerous modifications are possible without departing from the scope of the invention. For example, though only two levels of neurons are included in the above-described embodiment, it will be understood that these two levels could actually be only a small part of a multi-level network. As another example, though the two levels described are recursive with each other, it will be understood that a network according to the invention can be designed in which three or more neurons are part of each&#39; loop. For example, a first level neuron might be coupled to second level neurons via feed-forward connections, a second level neuron might be coupled to third level neurons via feed-forward connections, and a third level neuron might be coupled to first level neurons via feed-back connections. Other levels may also be included for pre- and post-processing purposes. All these and other variations are intended to be within the scope of the claims.