NEURAL NET USING DIFFERENTIAL CAPACITIVELY COUPLED INPUT AND OUTPUT LINES

Abstract: Neural nets using capacitive structures are adapted for construction in complementary metal-oxide-semiconductor integrated-circuit technology. In each neural net layer synapse input signals are applied to the inverting and non-inverting input terminals of each of a plurality of differential-input non-linear amplifiers by a respective pair of capacitors, which non-linear amplifiers generate respective axon responses. Alternatively, fully differential input amplifiers in each neural net layer apply their push-pull responses to synapse input signals via a pair of input lines. Each push-pull response is applied via a pair of capacitors of complementary capacitance values and an output line to the input of each of a plurality of non-linear output amplifiers in that neural net layer, which generate respective axon responses for that neural net layer. In certain of these neural nets, arrangements are described that make the capacitive structures bilaterally responsive, so that back-propagation calculations can be performed to alter the relative values of capacitors in each pair thereof, which is done during training of certain of the neural nets described.

Description: NEURAL NET USING CAPACITIVE STRUCTURES CONNECTING INPUT OR OUTPUT LINES AND DIFFERENTIALLY SENSE OR DRIVEN OUTPUT OR INPUT LINE PAIRS The invention relates to computer structures that emulate portions of a brain in operation, and more particularly, to such computer structures as can be realized using complementary metal-oxide-semiconductor (CMOS) technology. RΑr πRoττwn OF THE INVENTION Computers of the von Neumann type architecture have limited computational speed owing to the communication limitations of the single processor. These limitations may be overcome if a plurality of processors are utilized in the calculation and are operated at least partly in parallel. This alternative architecture, however, generally leads to difficulties associated with programming complexity. Therefore, it is often not a good solution. Recently, an entirely different alternative that does not require programming has shown promise. The networking ability of the neurons in the brain has served as a model for the formation of a highly interconnected set of processors, called a "neural network" or "neural net" that can provide computational and reasoning functions without the need of formal programming. The neural nets can learn the correct procedure by experience rather than being preprogrammed for performing the correct procedure. The reader is referred to R.P. Lippmai 's article "An Introduction to Computing With Neural Nets" appearing on pages 4-21 of the April 1987 TF.F.F. ASSP MΆP;Ά7,TNE (0740- 7467/87/0400-0004/$10.00 " 1987 IEEE), incorporated herein by reference, for background about the state of the art in regard to neural nets. - 2 - Neural nets are composed of a plurality of neuron models, processors each exhibiting "axon" output signal response to a plurality of "synapse" input signals. In a type of neural net called a "perceptron", each of these processors calculates the weighted sum of its "synapse" input signals, which are respectively weighted by respective weighting values that may be positive- or negative-valued, and responds non- linearly to the weighted sum to generate the "axon" output response. This relationship may be described in mathematical symbols as follows. Here, i indexes the input signals of the perceptron, of which there are an integral number M, and j indexes its output signals, of which there are an integral number N. Wi. j is the weighting of the i th input signal as makes up the j th output signal at such low input signal levels that the function is approximately linear. At higher absolute values of its argument, the function no longer exhibits linearity but rather exhibits a reduced response to ∑ Wj.. j X±. i-l A more complex artificial neural network arranges a plurality of perceptrons in hierarchic layers, the output signals of each earlier layer providing input signals for the next succeeding layer. Those layers preceding the output layer providing the ultimate output signal(s) are called "hidden" layers. The processing just described normally involves sampled-data analog signals, and prior-art neural nets have employed operational amplifiers with resistive interconnecting elements for the weighting and summing procedures . The resistive elements implement weighted summation being done in accordance with Ohm's Law. The speed of such a processor is limited by capacitances in various portions of the processor, and computations have been slow if the power consumption of a reasonably large neural net is to be held within reasonable bounds. That is, speed is increased by reducing resistance values to reduce RC time constants in the processors, but the reduced resistance values increase the V-2/R power consumption (R, C and V being resistance, capacitance and voltage, respectively.) Using capacitors to perform weighted summation in accordance with Coulomb's Law provide neural nets of given size operating at given speed that consume less power than those the processors which use resistors to implement weighted summation in accordance with Ohm's Law. Y.P. Tsividis and D. Anastassion in a letter "Switched-Capacitor Neural Networks" appearing in ELECT-RONICS . LETTERS, 27th August 1987, Vol. 23, No. 18, pages 958,959 (IEE) describe one method of implementing weighted summatiσif in accordance with Coulomb's Law. Their method, " a switched capacitor method, is useful in analog sampled-data neural net systems. However, a method of implementing weighted summation in accordance with Coulomb's Law that does not rely on capacitances being switched is highly desirable, it is here pointed out. This avoids the complexity of the capacitor switching elements and associated control lines. Furthermore, operation of the neural net with continuous analog signals over sustained periods of time, as well as with sampled data analog signals, is thus made possible. A problem that is encountered when one attempts to use capacitors to perform weighted summation in a neural net layer is associated with the stray capacitance between input" and output lines, which tends to be of appreciable size in neural net layers constructed using a metal-oxide- semiconductor (MOS) integrated circuit technology. The input and output lines are usually laid out as overlapping column and row busses using plural-layer metallization. The column busses are situated in one layer of metallization and the row busses are situated in another layer of metallization separated from the other layer by an intervening insulating oxide layer. This oxide layer is thin, so there is appreciable capacitance at each crossing of one bus over another. The fact of the row and column busses being in different planes tends to increase stray capacitances between them. The stray capacitance problem is also noted where both row and column busses are situated in the same metallization layer with one set of busses being periodically interrupted in their self- connections to allow passage of the other set of busses and being provided with cross-over connections to complete their self-connections. The problem of stray capacitance is compounded by the fact that the capacitive elements used to provide weights in a capacitive voltage summation network have stray capacitances to the substrate of the monolithic integrated circuit in which they are incorporated; a perfect two-terminal capacitance is not actually available in the monolithic integrated circuit. Where capacitive elements having programmable capacitances are used, capacitance is usually not programmable to zero value, either. The problems of stray capacitance are solved in some embodiments of the invention the invention by using output line pairs and sensing the charge conditions on the output lines of each pair differentially so that the effects of stray capacitances tend to cancel each other out. These output line pairs facilitate both excitory and inhibitory weights — that is, both positive- and negative-polarity Wj. fj — in effect to be achieved without having to resort to capacitor switching to achieve negative capacitance. The foregoing problems with stray capacitance are solved in other embodiments of the invention by using balanced input line pairs, so that the effects of stray capacitance from each output line tend to cancel out each other. Driving paired input lines with balanced input signals also allows both excitory and inhibitory weights — that is, both positive- and negative-polarity Wi f j — in effect to be achieved without having to resort to capacitor switching in order to achieve negative capacitance. The Wi, j £ terms are obtained by summing weighted xi and - xi balanced input signals. Neural nets employing capacitors in accordance with the invention lend themselves to being used in performing parts of the computations needed to implement a back- propagation training algorithm. The back-propagation training algorithm is an iterative gradient algorithm designed to minimize the mean square error between the actual output of a multi-layer feed-forward neural net and the desired output. It requires continuous, differentiable non-linearities. A recursive algorithm starting at the output nodes and working back to the first hidden layer is used iteratively to adjust weights in accordance with the following formula. W i j (t+l)=W i j(t)-ηδ j Xi (2} In this equation Wi fj (t) is the weight from hidden node i (or, in the case of the first hidden layer, from an input node) to node j at time t; xi is either the output of node i (or, in the case of the first hidden layer, i an input signal) ; η is a gain term introduced to maintain stability in the feedback procedure used to minimize the mean square errors between the actual output (s) of the perceptron and its desired output(s); and δ j is a derivative of error. The general definition of δ j is the change in error energy from output node j of a neural - 6 - net layer with a change in the weighted summation of the input signals used to supply that output node j . Lippman presumes that a particular sigmoid logistic non-linearity is used. Presuming the non-linearity of processor response is to be defined not as restrictively as Lippmann does, then δ j can be more particularly defined as in equation (2) , following, if node j is an output node, or as in equation (3), following, if node j is an internal hidden node. In equation (3) d j and y are the desired and actual values of output response from the output layer and y j ' is differential response of yj to the non-linearity in the output layer - i.e., the slope of the transfer function of that non- linearity. In equation (4) k is over all nodes in the neural net layer succeeding the hidden node j under consideration and W j .jς is the weight between node j and each such node k. The term y j ' is defined in the same way as in equation (3) . The general definition of the y j ' term appearing in equations (3) and (4) , rather than that general term being replaced by the specific value of yj' associated with a sigmoid logistic non-linearity, is the primary difference between the training algorithm as described here and as described by Lippmann. Also, Lippmann defines δ j in opposite polarity from equations (1), (3) and (4) above. During training of the neural net, prescribed patterns of input signals are sequentially repetitively applied, for which patterns of input signals there are corresponding prescribed patterns of output signals known. The pattern of output signals generated by the neural net, responsive to each prescribed pattern of input signals, is compared to the prescribed pattern of output signals to develop error signals, which are used to adjust the weights per equation (2) as the pattern of input signals is repeated several times, or until the error signals are detected as being negibly valued. Then training is done with the next set of patterns in the sequence. During extensive training the sequence of patterns may be recycled. SUMMARY OF THF. INVENTION The problem of stray capacitance and the problem of providing for both negative and positive weights without using switched capacitors are solved in the invention by using two sets of weighting capacitors, rather than one set of weighting capacitors, in each neuron model. In one embodiment of the invention each processor includes a plurality, M in number, of input lines for receiving respective ones of M input voltage signals. Each processor has first and second output lines. Respective capacitive elements connect each input signal line to each of the first and second output lines. Means are provided for maintaining substantially equal capacitances from the output lines to their respective surroundings. A differential-input non-linear amplifier has inverting and non-invert _ ".g input ports to which the first and second output lines connect and has an output port for sut.:lying neuron-like response to the M input signal voltages. Sensing the charge conditions on the output lines of each pair differentially tends to make the effects of stray capacitances cancel each other out. These output line pairs facilitate both excitory and inhibitory weights — that is, both positive- and negative-polarity Wj., j — in effect to be achieved without having to resort to capacitor switching to achieve negative capacitance. - 8 - In another embodiment of the invention each processor includes an output line on which the weighted summation of input signal voltages appears and a non-linear amplifier for providing neuron-like response to that weighted summation. A plurality, 2M in number, of input lines, crossing by that output line and exhibiting a stray capacitance thereto, are each connected to that output line by a respective capacitive element. Each input line and the capacitive element connecting it to the output line are identified by a respective one of consecutive ordinal,numbers first through 2M th , M being a positive integer. Means are provided for applying M input signal voltages in balanced form each to a respective pair of input lines identified by ordinal numbers M apart . The differences in the capacitances of the capacitive elements connecting to the output line from each pair of input lines determines the weighting of the input signal applied in balanced form on that pair of input lines, as appears in the weighted summation of input signal on the outline line. Accordingly, the effects of stray capacitance on the weighted summation of input signals is reduced. BRIEF DESCRIPTION OF THE DRAWING FIGURE 1 is a schematic diagram of a neural net layer which embodies the invention, using capacitors to perform weighted summations of synapse signals to be subsequently linearly combined and non-linearly amplified to generate axon response signals. FIGURE 2 is a schematic diagram of a prior art fully differential amplifier and a bias network therefor, as implemented with complementary metal-oxide-semiconductor CMOS field effect transistors, which is useful in the construction of neural nets in accordance with the invention. - 9 - FIGURE 3 is a schematic diagram of a non-linear voltage amplifier that is useful in the construction of neural nets in accordance with the invention. FIGURES 4A and 4B together form a FISURE 4 that is a schematic diagram of a modification of the FIGURE 1 neural fiet that can be made manifold times to provide in accordance with a further aspect of the invention, for the programmable weighting of the capacitances used in performing weighted * summation of synapse signals. FIGURE 5 is a schematic diagram illustrating one "way of pulsing the non-linear output driver amplifiers in a FIGURE 1 neural net layer modified manifoldly per FIGURE . FIGURE 6 is a schematic diagram of a prior-art analog multiplier modified to provide balanced output signals, which is useful in the FIGURE 1 neural net modifications shown in FIGURE 4 and in FIGURE 9. FIGURE 7 is a schematic diagram of training apparatus used with the FIGURE 1 neural net layer manifoldly modified per FIGURE 4 (or with the FIGURE 13 neural net layer , modified manifold times per FIGURE 14) . FIGURE 8 is a schematic diagram of a system having a plurality of neural net layers each constructed in accordance with FIGURE 1 modified manifold times per FIGURE 4 (or in accordance with FIGURE 13 modified manifold times per FIGURE 14) . FIGURES 9A and 9B together form a FIGURE 9 that is a schematic diagram of an alternative modification of the FIGURE 1 neural net that can be made manifold times to provide during training for the programmable weighting of the capacitances used in performing weighted summation of synapse signals, in accordance with another aspect of the invention. FIGURE 10 is a schematic diagram of the arrangement of stages in each counter of the FIGURE 1 neural net modified per FIGURE 9. FIGURE 11 is a schematic diagram of the logic elements included in each counter stage. FIGURES 12A, 12B and 12C together form a FIGURE 12 that is a schematic diagram of further modifications to the neural net, which use pairs of input lines driven by balanced input signals for connection to the pairs of differentially sensed output lines by weighting capacitors connected in bridge configurations . FIGURE 13 is a schematic diagram of another neural net layer which embodies the invention, using capacitors to perform weighted summations of synapse signals to be subsequently non-linearly amplified to generate axon response signals. FIGURES 14A and 14B together form a FIGURE 14 that is a schematic diagram of a modification of the FIGURE 13 neural net that can be made manifold times to provide, in accordance with a further aspect of the invention, for the programmable weighting of the capacitances used in performing weighted summation of synapse signals. FIGURE 15 is a schematic diagram illustrating apparatus which can be used instead of the FIGURE 5 apparatus for pulsing the non-linear output driver amplifiers in a FIGURE 13 neural net layer modified manifoldly per FIGURE 14. FIGURES 16A and 16B together form a FIGURE 16 that is a schematic diagram of an alternative modification of the FIGURE 1 neural net that can be made manifold times to provide during training for the programmable weighting of the capacitances used in performing weighted summation of synapse signals, in accordance with another aspect of the invention. DETAILED DESCRIPTION FIGURE 1 shows a neural net comprising a plurality, N in number, of non-linear amplifiers ODi, OD2, • • •OD(N-I), ODN. Each of a plurality, M in number, of input voltage signals xi, X2, .• •X(M-i), X supplied as "synapse" signals is weighted to provide respective input voltages for the non¬ linear voltage amplifiers 0Dχ, OD2, ...OD(N-I), ODN, which generate respective "axon" responses yi, y2, . • .y(N-1) . YN- M is a positive plural integer indicating the number of input synapse signals to the FIGURE 1 (or FIGURE 13) net, and N is a positive plural integer indicating the number of output axon signals the net can generate. To reduce the written material required to describe operation of the FIGURE 1 (or FIGURE 13) neural net, operations using replicated elements will be described in general terms;-using a subscript i ranging over all values one through M for describing operations and apparatuses as they relate to the ' (column) input signals xi, X2, • • -x(M-l) . χ M> * nd using a subscript j ranging over all values one through N for describing operations and apparatus as they relate to the (row) output signals yi, y2, •••y( N -i), γ _ That is, i and j are the column and row numbers used to describe ^ particular portions of the neural net. ' - , In FIGURE 1 input voltage signal xi is applied to the input port of an input driver amplifier IDi that is a voltage amplifier which in turn applies its voltage response to an input line ILi. Respective output lines OLj and 0L(J+N) " connect to the non-inverting input port of output driver amplifier OD j and to its inverting input port. Output driver " amplifier OD j generates at its output port a non-linear voltage response to the cumulative difference in charge on that respective pair of output lines OLj and OL(J+N) . The non-linear output driver amplifier OD j is sh.ow» in FIGURE 1 as simply being a differential-input non-linear voltage amplifier with the quiescent direct potential applied to its (+) and (-) input signal terminals via output lines OL j and OL(J+N) being adjusted by clamping to a desired bias voltage at selected times using a respective direct-current restorer circuit DCR j . The total capacitance of output line OL j to its surroundings and the total capacitance of output line OL(J+ N ) to its surroundings are caused to be substantially the same, as will be more particularly described below. A respective capacitor Ci, j connects each of the input lines ILi to each of the output lines OL j , and a respective capacitor Ci, (j + N) connects each of the input lines ILi to each of the output lines OL(J+ N ) . Since at its output terminal the output driver amplifier ODj responds without inversion to xi input signal voltage applied to its non- inverting (+) input terminal via capacitor Ci f j and responds with inversion to xi input signal voltage applied to its inverting (-) input terminal via capacitor Ci f (J+N) respectively, the electrically equivalent circuit is Xi signal voltage being applied to a single output line OL j by a capacitor having a capacitance that equals the capacitance of Ci ^j minus the capacitance of Ci f (J+ N ) • This technique of single-ended output signal drive to paired output lines that are differentially sensed avoids the need for switched- capacitance techniques in order to obtain inhibitory (or negative) weights as well as excitory (or positive) weights. Thus this technique facilitates operating the neural net with analog signals that are continuous over sustained periods of time, if so desired. FIGURE 1 shows each of the input lines ILi as being provided with a respective load capacitor CLi to cause that capacitive loading upon the output port of the input driver amplifier IDi to be substantially the same as that upon each output port of the other input driver amplifiers. This is desirable for avoiding unwanted differential delay in responses to the input signals xi. Substantially equal capacitive loading can be achieved by making the capacitance of each of the input line loading capacitors CLi - CL M very - 13 - large compared to the total capacitance of the capacitors Ci, j connecting thereto. Preferably, however, this result is achieved by making the capacitance of each of the input line loading capacitors complement the combined value of the other capacitances connecting thereto. This procedure reduces the amount of line loading capacitance required. Where the voltages appearing on the output lines OL j and 0L ( J+N> are sensed directly by the non-linear output driver amplifiers ODi, ...OD N , as shown in FIGURE 1, this procedure makes the voltage division ratio for each input voltage xi , ... x m independent of the voltage division ratios for the other input voltages. FIGURE 1 also shows each of the output lines OL j being loaded with a respective load capacitor CL(M+J) and each of the output lines OL(N+J) being loaded with a respective load capacitor CL( + N +J) • This is done so that the total capacitance on each output line remains substantially the same as on each of the other output lines. This can be done by choosing CL(M+J) to be much larger than other capacitances to output line OL j , and by choosing CL(M+N+J) to be much larger than other capacitances to output line OLN+J ) . Alternatively, this can be done by choosing CL(M+J) and CL( +N+J) to complement the combined value of the other capacitances connecting the same output line. The input voltage to output driver amplifier OD j will (to good approximation) have the following value, VJ, in accordance with Coulomb's Law. -1 Vj = (C j ) (Ci, j ~ Ci, ( j +M) ) Xi i=l (5) The generation of voltage VJ can be viewed as the superposition of a plurality of capacitive divisions between, on the one hand, the effective capacitance (C(i. j ) - Ci,( j+N )) each input voltage has to output line OL j and, on the other hand, the total capacitance Cj of the output line to its surroundings. That is, C j is the total capacitance on output line OLj or the total capacitance on output line OL(N-t-j), which capacitances should be ecpial to each other and fixed in value. Where the difference in charge appearing on the output lines OL j and OL(J+N) is sensed by fully differential charge-sensing amplifiers preceding the non-linear voltage amplifiers in the output driver amplifiers, as will be described later on in this specification in connection with FIGURE 4, the output signals from the charge-sensing amplifiers will be balanced with reference to a reference V BIAS potential. FIGURE 2 shows a fully differential amplifier constructed of MOS field-effect transistors Qi - Q 13 , as may serve for any one of the fully differential amplifiers DA j for j = 1, 2, ...N. Also shown is a bias network constructed of MOS field effect transistors Q 14 - Q 19 for generating direct bias potentials for application to that fully differential amplifier and to others of its kind. This circuitry is described in more detail on pages 255-257 of the book Analog MOS Integrated Circuits for Signal Prnmsslng by R. Gregorian and G.C. Temes, copyright 1986, published by John Wiley & Sons, Inc., of New York, Chichester, Brisbane, Toronto and Singapore. The fully differential amplifier includes a long- tailed-pair connection of n-channel MOSFETs Qi and Q 2 providing common-mode rejection for the input voltages IN and I applied to the (+) and (-) input terminals at their respective gate electrodes. N-channel MOSFET Q 13 is connected as a constant-current sink for tail current from the interconnection between the source electrodes of Qi and QL. Q l and Q 2 are in folded cascode connections with p-channel MOSFETs 0. 7 and Qβ respectively. There is also common mode rejection for output voltages OUT and OUT appearing at the (+) and (-) output terminals connecting from the drain electrodes of Q 7 and Qβ respectively, which is why the differential amplifier comprising Qi - Q 13 is described as being "fully" differential. This common mode rejection is provided by common-mode degenerative feedback connections from the (-) and (+) output terminals to the gate electrodes of p-channel MOSFETs Q 3 and Q 4 , the paralleled source-to-drai,n paths of which supply current to the joined source electrodes of p-channel MOSFETs Q 5 and Qs operated as a current splitter. Q 5 drain current biases the folded cascode connection of Qi and Q 7 , and QQ drain current biases the folded cascode connection of Q 2 and Qβ. N-channel MOSFETs Q 9 and Qn are in a cascode connection biased to provide a high-impedance constant-current sink as drain load to Q 7 , and n-channel MOSFETs Q 10 and Q 12 are in a cascode connection biased to provide a high-impedance constant-current sink as drain load The (+) and (-) output terminals can be biased to the same (+2.5v) potential as applied to the gate electrode of MOSFET Q 14 by causing MOSFETs Q 1 -Q 18 to have the following width-to-length ratios presuming Qi, Q2, Q7 and Qβ to have equal amplitude quiescent channel currents. (W/D 11 (W/L) 12 (W/L) 13 : (W/L) 18 2:2:1:1 (6) (W/L) 3 (W/L) (W/L) 14 : : 1 : 1 : 1 (7) ( /L) 5 (W/L) 6 (W/L) 15 : : 1 : 1 : 1 (8) (W/L) 7 (W/L) 8 (W/L) 16 : : 2 : 2 : 1 (9) (W/L) 9 (W/L) 10 (W/L) 17 : : 2 : 2 : 1 (10) - 16 - The width-to-length ratio of MOSFET Q 1 9 is chosen to provide responsive to the drain current demand of Q 14 a voltage drop across Q 1 9 channel that affords sufficient operating voltage range for signals at terminals out and OUT . FIGURE 3 shows non-linear voltage amplifier circuitry that can be used after linear voltage amplifier circuitry to implement each non-linear output driver amplifier OD j in the FIGURE 1 neural net layer. The FIGURE 3 non-linear voltage amplifier is a cascade connection of two source-follower transistors, one (Q 2 0 A ) being an n-channel MOSFET and the other (Q 2 0 B ) being a p-channel MOSFET. Q 2 0 A is provided a constant-current generator source load by an n- channel MOSFET Q 21 , which is the slave or output transistor of a current mirror amplifier including as its master or input transistor an n-channel MOSFET Q22 self-biased by drain-to- gate feedback. Q20 B is provided a constant-current generator source load by a p-channel MOSFET Q23, which is the slave or output transistor of a current mirror amplifier including as its master or input transistor a p-channel MOSFET Q 24 self- biased by drain-to-gate feedback. Q 22 a d Q2 4 are connected as diodes by their respective drain-to-gate feedback connections, and these diodes are connected in series with another diode-connected n-channel MOSFET Q 25 and with another diode-connected p-channel MOSFET Q26 between Vgs and VDD potentials to implement a bias network. In this bias network a quiescent input current flows from the input port of the current mirror amplifier comprising Q 2 3, Q 24 into the input port of the current mirror amplifier comprising Q 21 , Q2 2 -- Q 21 and Q23 drain current flows are similar-valued by current mirror amplifier action. All the n-channel MOSFETs Q20 r Q21, Q2 2 and Q25 have similar channel widths and lengths and exhibit similar operating characteristics. All the p-channel MOSFETs Q20 B. Q23, Q 24 and Q 2 6 have similar channel widths and lengths and exhibit similar operating characteristics, which are complementary to those of the n-channel MOSFETs. The bias network MOSFETs Q22. Q24. Q2 5 and Q26 may be shared by a plurality of the FIGURE 3 non-linear voltage amplifier circuits to conserve hardware and operating power. Non-linearity of response in the FIGURE 3 voltage amplifier comes about because (1) source-follower action of Q20 A fo r positive-going excursions of its gate electrode potential becomes limited as its source potential approaches its drain potential VH I and (2) source-follower action of Q 20 for negative-going excursions of its gate electrode potential becomes limited as its source potential approaches its drain potential V O- At the source electrode of source-follower Q20 B there is a sigmoidal response to a linear ramp potential applied to the gate electrode of source-follower Q20 A - The voltages V O and VHI can be programmed to control the limiting properties of the FIGURE 3 non-linear amplifier, and the voltages VLO and VHI ma y be selected to provide for symmetry of response or for asymmetry of response. FIGURE 3 shows representative values for VHI and VLQ that provide a substantially symmetrical response about +2.5 volts. Output driver amplifier OD j can use non-linear voltage amplifier circuitry different from that shown in FIGURE 3. For example, source followers Q20 A and Q20 B can be reversed in order of their cascade connection. Either this* alternative circuitry or the FIGURE 3 circuitry can be preceded by a charge-sensing amplifier, rather than a linear voltage amplifier, to realize the type of output driver amplifier used in FIGURE 4, FIGURE 9, FIGURE 14 and FIGURE 16 neural nets. In the FIGURE 1 and FIGURE 13 neural nets the output driver amplifiers can be realized without using *;he FIGURE 3 circuitry or the previously described alternative circuitry. For example, each output driver amplifier can comprise a long-tailed pair connection of transistors having a current mirror amplifier load for converting their output signal voltage to single-ended form. The long-tailed pair connection of transistors is a differential amplifier connection where their source electrodes have a differential- mode connection to each other and to a constant-current generator. Consider now, referring to FIGURE 1, how neuron model behavior is exhibited by input driver amplifier IDi , capacitors Ci. and Ci, (j+N) and non-linear output driver amplifier OD j for particular respective values of i and j . If the capacitance of capacitor Ci.j is larger than the capacitance of capacitor Ci. (j+N) for these particular values of i and j, then the output voltage y j for that j will exhibit "excitory" response to the input voltage i. If the capacitances of Ci, j and Ci f (j+ N ) are equal for these i and j values, then the output voltage y j for that j should exhibit no response to the input voltage y j . If the capacitance of capacitor Ci, j is smaller than the capacitance of capacitor Ci (j + ) for those i and j values, then the output voltage y j for that j will exhibit "inhibitory" response to the input voltage xi. In some neural nets constructed in accordance with the invention the capacitors Ci, j and Ci, (J+ ) for all i and j may be fixed-value capacitors, so there is never any alteration in the weighting of input voltages i where i=l, ...M. However, such neural nets lack the capacity to adapt to changing criteria for neural responses — which adaptation is necessary, for example, in a neural network that is to be connected for self-learning. It is desirable in certain applications, then, to provide for altering the capacitances of each pair of capacitors CL,J and Ci, ( J+ ) associated with a respective pair of values of i and j. This alteration is to be carried out in a complementary way, so the sum of the capacitances of Ci,j and of CKJ+ ) remains equal to C k - For example, this can be implemented along the lines of the inventor's previous teachings in regard to "digital" capacitors, having capacitances controlled in proportion to binary-numbers used as control signals, as particularly disclosed in connection with FIGURE 11 of his U.S. patent No. 3,890,635 issued 17 June 1975, entitled "VARIABLE CAPACITANCE SEMICONDUCTOR DEVICES" and assigned to General Electric Company. Each pair of capacitors Ci f j and Ci, j+ ) is then two similar such capacitors and their capacitances are controlled by respective control signals, one of which is the one's complement of the other. Alternatively, the pair of capacitors Ci, j and Ci, ( j+N ) ma y be formed from selecting each of a set of component capacitors with capacitances related in accordance with powers of two to be a component of one or the other of the pair of capacitors Ci, j and Ci( j + ), the selecting being done by field effect transistors (FETs) operated as transmission gates. Yet another way of realizing the pair of capacitors Ci, j and C f (J+ N ) is to control the inverted surface potentials of a pair of similar size metal-oxide- semiconductσr (MOS) capacitors with respective analog signals developed by digital-to-analog conversion. FIGURE 4, comprising component FIGURES 4A and 4B, shows a representative modification that can be made to the FIGURE 1 neural net near each set of intersections of output lines OLj and 0L(J+N) with an input line ILi from which they receive with differential weighting a synapse input signal xi. Such modifications together make the neural net capable of being trained. Each capacitor pair Ci,j and Ci, (J+ N ) of the FIGURE 1 neural net is to be provided by a pair of digital capacitors DCi,j and ECi, (J+N) . (For example, eafch of these capacitors DCi, j and DCi, (J+ N ) may be as shown in FIGURE li of U.S. patent No. 3,890,635). The capacitances of DCi, j and - 20 - DCi, ( j + N ) are controlled in complementary ways by a digital word and its one's complement, as drawn from a respective word-storage element WSEi,j in an array of such elements located interstitially among the rows of digital capacitors and connected to form a memory. This memory may, for example, be a random access memory (RAM) with each word- storage element WSEi,j being selectively addressable by row and column address lines controlled by address decoders . Or, by way of further example, this memory can be a plurality of static shift registers, one for each column j. Each static shift register will then have a respective stage WSEi_ for storing the word that controls the capacitances of each pair of digital capacitors DCi, j and DCi r (J+ ) . The word stored in word storage element WSEi, j may also control the capacitances of a further pair of digital capacitors DC(i+M),j and DC(i+M) , (J+N) / respectively. The capacitors DC(i+M),j and DC(i+ ), (j+ N ) connect between "ac ground" and output lines OL j and OL(J+ ), respectively, and form parts of the loading capacitors CL( M + j ). The capacitances of DC(i+ 2M , j ) and DCi, j are similar to each other and changes in their respective values track each other. The capacitances of DC(i+M), (j+N) and DCi, (j+N) are similar to each other and changes in their respective values track each other. The four digital capacitors DCi,j, DCi, (J+N), DC(i+ M ), j and DC(ik+M), (j+N) are connected in a bridge configuration having input terminals connecting from the input line ILi and from a-c ground respectively and having output terminals connecting to output lines OLj and 0L(J+NJ respectively. This bridge configuration facilitates making computations associated with back-propagation programming by helping make the capacitance network bilateral insofar as voltage gain is concerned. Alternatively, where the computations for back- propagation programming are done by computers that do not involve the neural net in the computation procedures, the neural net need not include the digital capacitors DC(i + ), j and DC(i+M), (j+N) • When the FIGURE 4 neural net layer is being operated normally, following programming, the φp signal applied to a mode control line MCL is a logic ZERO. This ZERO on mode control line MCL conditions each output line multiplexer OLMj of an N-numbered plurality thereof to select the output line OL j to the inverting input terminal of a respective associated fully differential amplifier DA j . This ZERO on mode control line MCL also conditions each output line multiplexer 0LM(J+ N > to select the output line OL(J+ ) to the non-inverting input terminal of the respective associated fully differential amplifier DA j . Differential amplifier DA j , which may be of the form shown in FIGURE 2, is included in a respective charge-sensing amplifier QS j that performs a charge-sensing operation for output line OL . In furtherance of this charge-sensing operation, a transmission gate TG j responds to the absence of a reset pulse Q R to connect an integrating capacitor CI j between the (+) output and (-) input terminals of amplifier DAJ; and a transmission gate TG(J+5 ) responds to the absence of the reset pulse ΦR to connect an integrating capacitor CI(J+N> between the (-) output and (+) input terminals of amplifier DAj. With integrating capacitors CIj and CI(J+N) SO connected, amplifier DA j functions as a differential charge amplifier. When φ p signal on mode control line MCL is a ZERO, the input signal i induces a total differential change in charge on the capacitors DCi,j and DCi, (j + N) proportional to the difference in their respective capacitances. The resulting displacement current flows needed to keep the input terminals of differential amplifier DAj substantially equal in potential requires that there be corresponding displacement current flow from the integrating capacitor CIj and CI ( J+ N) - 22 - differentially charging those charging capacitors to place thereacross a differential voltage VJ defined as follows. _1 M Vj = (CIj + CI (j+N) ) ∑ (Cj. j - C (i+M) j) Xi J i=l (11) The half V j signal from the non-inverting (+) output terminal of amplifier DA j is supplied to a non-linear voltage amplifier circuit NL j which can be the non-linear voltage amplifier circuit of FIGURE 3 or an alternative circuit as previously described. The non-linear voltage amplifier circuit NL j responds to generate the axon output response y j . It is presumed that this non-linear voltage amplifier NL j supplies y j at a relatively low source impedance as compared to the input impedance offered by the circuit y j is to be suppled to - e.g. on input line in a succeeding neural net layer. If this is so there is no need in a succeeding neural net layer to interpose an input driver amplifier IDi as shown in FIGURE 1. This facilitates interconnections between successive neural net layers being bilateral. An output line multiplexer OLM j responds to the φp signal appearing on the mode control line MCL being ZERO to apply y j to an input line of a succeeding neural net layer if the elements shown in FIGURE 4 are in a hidden layer. If the elements shown in FIGURE 4 are in the output neural net layer, output line multiplexer OLM j responds to the φp signal on the mode control line being ZERO to apply yj to an output terminal for the neural net. From time to time, the normal operation of the neural net is interrupted; and, to implement dc-restoration a reset pulse ΦR is supplied to the charge sensing amplifier QSj. Responsive to ΦR ,the logic complement of the reset pulse ΦR, going low when Φ goes high, transmission gates TG j and TG(J+SN) are no longer rendered conductive to connect the - - integrating capacitors CIj and CI(J+N) from the output terminals of differential amplifier DAj . Instead, transmission gates TG(J+N) and TG( +4 ) respond to Φ going high to connect to VBIAS the plates of capacitor CI j and CI(J+N) normally connected from those output terminals, BI S being the 2.5 volt intermediate potential between the Vss = 0 volt and V D D •= 5 volt operating voltages of differential amplifier DAj. Other transmission gates TG(j+2 N ) and TG(J+3 ) respond to Φ going high to apply direct-coupled degenerative feedback from the output terminal of differential amplifier DA j to its input terminals, to bring the voltage at the output terminals to that supplied to its inverting input terminal from output lines OLj and OL( +N) . During the dc- restoration all xi are "zero-valued". So, the " bharges on integrating capacitor CIj and CI(J+N) are adjusted to compensate for any differential direct voltage error occurring in the circuitry up to the output terminals of differential amplifier DA j . Dc-restoration is done concurrently for all differential amplifiers DA j (i.e., for values of j ranging from one to N) . During training, the φp signal applied to mode control line MCL is a logic ONE, which causes the output line multiplexer OLMj to disconnect the output lines OL j and OL(J+N) from the (+) and (-) input terminals of differential amplifier DAj and to connect the output lines OL j and O ( +N ) " to receive +δ j and -δ j error terms. These +δ j and -δ j error terms are generated as the balanced product output signal of a analog multiplier AM j , responsive to a signal j and to a signa__ y' j which is the change in output voltage y j of non- linear amplifier NLj for unit change in the voltage on output line OL j . The term Δ j for the output neural net layer is an error signal that is the difference between y j actual value and its desired value d j . The term Δ j for a hidden neural - 24 - net layer is also an error signal, which is of a nature that will be explained in detail further on in this specification. Differentiator DFj generates the signal y' j , which is a derivative indicative of the slope of y j change in voltage on output line OLj, superposed on VBIAS- TO determine the y' j derivative, a pulse doublet comprising a small positive-going pulse immediately followed by a similar- amplitude negative-going pulse is introduced at the inverting input terminal of differential amplifier DA j (or equivalently, the opposite-polarity doublet pulse is introduced at the non-inverting input terminal of differential amplifier DAj) to first lower y j slightly below normal value and then raise it slightly above normal value. This transition of y j from slightly below normal value to slightly above normal value is applied via a differentiating capacitor CD j to differentiator DF j . Differentiator DFj includes a charge sensing amplifier including a differential amplifier DA ( J+N ) and an integrating capacitor CI(J+N> . During the time that yj is slightly below normal value, a reset pulse φs is applied to transmission gates TG( j + 4N ) and G(J+SN) to render them conductive. This is done to drain charge from integrating capacitor CI(J+N) . except for that charge needed to compensate for DA ( J+N ) input offset voltage error. The reset pulse φs ends, rendering transmission gates GB(J+4N) and TG(J+SN ) no longer conductive, and the complementary signal φs goes high to render a transmission gate TG(J+3 ) conductive for connecting integrating capacitor CI(J+N) between the output and inverting-input terminals of differential amplifier DA(j+ ) . With the charge-sensing amplifier comprising elements DA(J+N) and CI(J+N) reset, the small downward pulsing of yj from normal value is discontinued and the small upward pulsing of j from normal value occurs. The transition between the two abnormal conditions of y j is applied to the charge-sensing amplifier by electrostatic induction via differentiating capacitor CD j . Differential amplifier DA ( J+N> output voltage changes by an amount y'j from the VBIAS value it assumed during reset. The use of the transition between the two pulses of the doublet, rather than the edge of a singlet pulse, to determine the derivative y' j makes the derivative-taking process treat more similarly those excitory * and inhibiting responses of the same amplitude. The doublet pulse introduces no direct potential offset error into the neural net layer. Responsive to a pulse φi, the value y'j + V B IAS from differentiator DFj is sampled and held by row sample and hold circuit RSH j for application to analog multiplier AM j as an input signal. This sample and hold procedure allows yj to return to its normal value, which is useful in the output layer to facilitate providing y j for calculating (y j - dj) . The sample and hold circuit RSH j may simply comprise an L- section with a series-arm transmission-gate sample switch and a shunt-leg hold capacitor, for example. Analog multiplier AM j is of a type accepting differential input signals, as will be described in greater detail further on in connection with FIGURE 6. The difference between y'j + VBIAS and VBI S voltages is used as a differential input signal to analog multiplier AMj, which exhibits common-mode rejection for the VBIAS term. During training, the φp signal applied to mode control line MCL is a logic ONE, as previously noted. When the FIGURE 4 elements are in the output layer, the ONE on mode control line MCL conditions an output multiplexer OM j to discontinue the application of y j signal from non-linear amplifier NLj to an output terminal. Instead, the output multiplexer OMj connects the output terminal to a charge- sensing amplifier QS j . Charge sensing amplifier QS j includes - 26 - a differential amplifier DA(J+2 ) and an integrating capacitor CI(j+2 N ) and is periodically reset responsive to a reset pulse φu- Reset pulse φu can occur simultaneously with reset pulse φs, for example. Output signal Δ j from charge-sensing amplifier QS j is not used in the output layer, however. Analog multiplier AMj does not use Δj + V B I S and VBI S as a differential input signal in the output layer, (y j - d j ) being used instead. When the FIGURE 4 elements are in a hidden neural net layer, φp signal on the mode control line MCL being a ONE conditions output multiplexer OMj to discontinue the application of y j signal from non-linear amplifier NL j to the input line IL j of the next neural net layer. Instead, output multiplexer OM j connects the input line IL j to a charge- sensing amplifier QS j . Charge-sensing amplifier QSj senses change in the charge on input line IL j during training to develop a Δ j error signal superposed on B χ A g direct potential. The difference between Δj+ V B ι A s and VBI S voltages is used as a differential input signal to analog multiplier AM j , which multiplier exhibits common-mode rejection for the V B IAS term. Charge-sensing amplifier QS j employs a differential-input amplifier DA(J+2N) and an integrating capacitor CI(J+2N) - Transmission gates TG J+9 ), TG(J+ION) and G(j+n ) cooperate to provide occasional resetting of charge conditions on the integrating capacitor CIJ+2N responsive to the reset pulse φ-y. FIGURE 5 shows how each output line OL j for j = 1, ...N may be pulsed during calculation of y' j terms. Each output line OLj is connected by a respective capacitor COj to the output terminal of a pulse generator PG, which generates the doublet pulse. FIGURE 5 shows the doublet pulse applied to the end of each output line OL j remote from the - terminal of the associated differential amplifier DA j in the charge- sensing amplifier QS j sensing the charge on that line. It is also possible to apply the doublet pulses more directly to those - terminals by connecting to these terminals respective ones of the plates of capacitors CO j that are remote from the plates connecting to pulse generator PG. Each output line OLj has a respective capacitor CO j connected between it and a point of reference potential, and each output line OL( + N ) has a respective capacitor CO(J+ N > connected between it and a point of reference potential, which capacitors are not shown in the drawing. The respective capacitances of the capacitors COj and CO(J+N) are all of the same value, so that the back-propagation algorithm is not affected by the presence of these capacitors. Arrangements for adding the doublet pulse to VJ before its application to the non-linear amplifier NL j can be used, rather than using the FIGURE 5 arrangement. FIGURE 6 shows a four-quadrant analog multiplier supplying product output signal in balanced form at its output terminals POUT and POUT. It is a modification of a single-ended-output analog multiplier described by K. Bultt and H. Wallinga in their paper "A CMOS Four-quadrant Analog Multiplier" appearing on pages 430-435 of the IEEE JOURNAL OF SOLID STATE CIRCUITS, Vol. SC-21, No. 3, June 1986, incorporated herein by reference. The FIGURE 6 analog multiplier accepts a first push-pull input signal between input terminals INI and INI, and it accepts a second push- pull input signal between terminals IN2 and IN2. As described by Bultt and Wallinga there are four component analog multipliers: a first comprising n-channel MOSFETs Q 2 7-Q 29 , a second comprising n-channel MOSFETs Q 3 0- Q 32 , a third comprising n-channel MOSFETs Q 33 -Q 35 and a fourth comprises n-channel MOSFETs Q36-Q.38- The component analog multipliers are arranged in cross-coupled pairs to suppress quadratic and offset terms. Constant-current generator IG1 provides for floating of the potentials at input terminals INI and INI, and constant-current generator IG2 provides for floating of the potentials at input terminals IN2 and IN2. The push-pull outputs of the four-quadrant analog multiplier are supplied to diode-connected p-channel MOSFETs Q 39 and Q 40 , which are the master or input transistors of respective current mirror amplifiers., Q3 9 and Q 40 have respective p- channel MOS slave or output transistors Q 41 and Q 42 associated with them in their respective current mirror amplifiers, and as in the Bultt and Wallinga analog multiplier the push-pull.. variations in the drain currents of Q 41 and Q 42 are converted to single-ended form at output terminal POUT using a current mirror amplifier connection of n-channel MOSFETs Q 43 and Q 44 . In FIGURE 6 Q 39 and Q 40 additionally have respective further p-channel MOS slave or output transistors Q 45 and Q 46 associated with them in their respective current mirror amplifiers, which are dual-output rather than single-output in nature. The push-pull variations in the drain currents of Q 45 and Q 46 are converted to single-ended form at terminal POUT using a current mirror amplifier connection of n-channel MOSFETs Q 47 and Q 48 . Since the current mirror amplifier connections of Q 43 and Q 4 7 are driven push-pull, the output signals at output terminals POUT and POUT exhibit variations in opposite senses of swing. FIGURE 7 shows apparatuses for completing the back- propagation computations, as may be used with the FIGURE 1 neural net manifoldly modified per FIGURE 4. The weights at each word storage element WSEi, j in the interstitial memory array IMA are to be adjusted as the i column addresses and j row addresses are scanned row by row, one column at a time. An address scanning generator ASG generates this scan of i and j addresses shown applied to interstitial memory array IMA, assuming it to be a random access memory. The row address j is applied to a row multiplexer RM that selects δ j - - to one input of a multiplier MULT, and the column address i is applied to a column multiplexer CM that selects i to another input of the multiplier MULT. Multiplier MULT is of a type providing a digital output responsive to the product of its analog input signals. Multiplier MULT may be a multiplying analog-to-digital converter, or it may comprise an analog multiplier followed by an analog-to-digital converter, or it may comprise an analog-to-digital converter for each of its input signals and a digital multiplier for multiplying together the converted signals. Multiplier MULT generates the product i δj as reduced by a scaling factor η, which is the increment or decrement to the weight stored in the currently addressed word storage element WSEij in the memory array IMA. The former value of weight stored in word storage element WSEi j is read from memory array IMA to a temporary storage element, or latch, TS. This former weight value is supplied as minuend to a digital subtractor SUB, which receives as subtrahend η xi δ j from multiplier MULT. The resulting difference is the updated weight value which is written into word storage element WSEi,j in memory array IMA to replace the former weight value. FIGURE 8 shows how trained neural net layers Lo, Li and L2 are connected together in a system that can be trained. Lo is the output neural net layer that generates y' output signals, is similar to that described in connection with FIGURES 4 and 5, and is provided with a badk-propagatAon processor BPPo with elements similar to those shown in FIGURE 7 for updating the weights stored in the interstitial memory array of Lo. Li is the first hidden neural net layer which generates yi output signals supplied to the output neural net layer as its i input signals. These yi output signals are generated by layer Li as its non-linear response to the weighted sum of its X input signals. This first hidden - 30 - neural net layer Li is provided with a back-propagation processor BPPi similar to BPPo. L 2 is the second hidden neural net layer, which generates y output signals supplied to the first hidden neural net layer as its X input signals. These y h output signals are generated by layer L2 as its non¬ linear response to a weighted summation of its x g input signals. This second hidden layer is provided with a back- propagation processor similar to BPPo and to BPPi. FIGURE 8 presumes that the respective interstitial memory array IMA of each neural net layer Lo, Li, L 2 has a combined read/write bus instead of separate read input and write output busses as shown in FIGURE 7. FIGURE 8 shows the Δ j , Δi and Δ signals being fed back over paths separate from the feed forward paths for y j , yi and y h signals, which separate paths are shown to simplify conceptualization of the neural net by the reader. In actuality, as shown in FIGURES 4 and 9, a single path may be used to transmit y j in the forward direction and Δ j in the reverse direction, etc. Back-propagation processor BPPo modifies the weights read from word storage elements in neural net layer Lo interstitial memory array by η xi δ j amounts and writes them back to the word storage elements in a sequence of read- modify-write cycles during the training procedure. Back- propagation processor BPPi modifies the weights read from word storage elements in neural net layer Li interstitial memory array by η X δi amounts and writes them back to the word storage elements in a sequence of read-modify-write cycles, during the training procedure. Back-propagation processor BPP2 modifies the weights read and storage elements in neural net layer L2 interstitial memory array by η X g δ h amounts and writes them back to the word storage element in a sequence of read-modify-write cycles during the training procedure. FIGURE 9, comprising component FIGURES 9A and 9B shows an alternative modification that can be manifoldly made to the FIGURE 1 neural net layer to give it training capability. This alternative modification seeks to avoid the need for a high-resolution multiplier MULT and complex addressing during back-propagation calculations in order that training can be implemented. A respective up/down counter UDCi, j is used instead of each word storage element WSEi. j . Correction of the word stored in counter UDCi f j is done a count at a time; and the counter preferably has at least one higher resolution stage in addition to those used to control the capacitances of digital capacitors DCi, j , DC(i+ ), , DCi, (J+ ) and DC(i+ M ) , (j+ N ) • Each up/down counter UDCi.j has a respective counter control circuit CONi. j associated therewith. Each counter control circuit CONi fj may, as shown in FIGURE 9a,and described in detail further on in this specification simply consist of an exclusive-OR gate X0Ri,j. A row sign detector RSDj detects whether the polarity of δj is positive or negative, indicative of whether a row of weights should in general be decremented or incremented, and broadcasts its detection result via a row sign line RSLj to all counter control circuits (CONi, j for i=l, ...,M) in the row j associated with that row sign detector RSDj. Before making a back-propagation calculation, a respective column sign detector CSDi detects whether the polarity of xi is positive or negative for each, columnar pc_ tion along the row which is to be updated, to provide an indication of whether it is likely the associated weight should be decremented or incremented. This indication is stored temporarily in a (column) sample and hold circuit CSHi. Each column sample and hold circuit CSHi is connected to broadcast its estimate via a column sign line CSLi to all ' counter control circuits (CONi f j for j=l, ...N) in the column i associated with that sample and hold circuit CSHi. Responsive to these indications from sign detectors CSDi and RSDj, each respective counter control circuit C0Ni,j decides in which direction up/down counter UDCi,j will count to adjust the weight control signals Di f j and Oi,j stored therein The counter control circuitry CONi. j should respond to the sign of +δ j being positive, indicating the response VJ to be too positive, to decrease the capacitance to output line OLj that is associated with the signal xi or -xi that is positive and to increase the capacitance to output line OL j that is associated with the signal -xi or xi that is negative, for each value of i. The counter control circuitry CONi .j should respond to the sign of +δ j being negative, indicating the response v to be too negative, to increase the capacitance to output line OL j that is associated with the signal -xi or x that is negative and to decrease the capacitance to output line OL j that is associated with the signal i or -xi that is positive. Accordingly, counter control circuitry CONi. j may simply consist of a respective exclusive-OR gate XORi, j as shown in FIGURE 9a, if the following presumptions are valid. Each of the digital capacitors D j^j and C(i+ M ) , (J+N) is presumed to increase or decrease its capacitance as Di, j is increased or decreased respectively. Each of the digital capacitors DC(i+ M ), j and DCi, ( j+ N) is presumed to increase or decrease its capacitance as Di, j is increased or decreased respectively. A ZERO applied as up/down signal to up/down counter UDCi, j is presumed to cause counting down for Di.j and counting up for Di.j. A ONE applied as up/down signal to up/down counter UDCi,j is presumed to cause counting up for Di,j and counting down for i, j . Column sign detector CSDi output indication is presumed to be a ZERO when xi is not negative and to be a ONE when xi is negative. Row sign detector RSD j output indication is presumed to be a ZERO when δ j is not negative and to be a ONE when δj is negative. Since the condition where xi or δj is zero-valued is treated as if the zero-valued number were positive, forcing a false correction which is in fact not necessary, and thus usually creating the need for a counter- correction in the next cycle of back-propagation training, there is dither in the correction loops. However, the extra stage or stages of resolution in each up/down counter UDCi, j prevent high-resolution dither in the feedback correction loop affecting the capacitances of DCi,j , DC(i+ M ), j , DCi f ( +N) and DC(i+M) , (j+N) • Analog multiplier AM j develops balanced product signals, +δ j and -δj. that can be supplied to a voltage comparator that serves as the row sign detector RSD j . Alternatively, since the derivative y' j always has the same sign (normally a positive one) , one can use a voltage comparator to compare the voltages supplied to input terminals of analog multiplier Aamj the other than those which receive y' j + V BIAS and VBIAS for providing the row sign detector RSDj its input signal. FIGURE 10 shows the construction of counter UDCi, j being one that has a plurality of binary counter stages BCSi, BCS2, BCS3 that provide increasingly more significant bits of the weight control signal Di. and of its one's complement Di,j. FIGURE 11 shows the logic within each binary counter stage which is implemented with MOS circuitry that is conventional in the art. FIGURES 10 and 11 make it clear that the opposite directions of counting for Di, j and Di,j can be controlled responsive to a ZERO or ONE up/down control signal in either of two ways, depending on whether Di.j is taken from Q outputs of the flip-flops and Di,j is taken from their Q outputs, as shown, or whether Di. j is taken from the Q outputs of the flip-flops and Di, j is taken from their Q outputs. If the latter choice had been made instead, each counter control circuit CON . j would have to consist of a respective exclusive-NOR circuit, or alternatively the CSDi and RSDj sign detectors would have to be of opposite logic types, rather than of same logic type. FIGURE 12 comprising FIGURES 12A, 12B and 12C shows further modification that can be made to the FIGURE 9 modification for the FIGURE 1 neural net. This modification, as shown in FIGURE 12A provides for a pair of input lines ILi and IL(i+ M ) for driving each bridge configuration of digital capacitors DCi. j , DCi. {j + N ), DC ( i+ M) ,j and DC(i+ M) . (j + N) push- pull rather than single-ended. Push-pull, rather than single-ended drive is provided to the differential charge sensing amplifier DQS j , doubling its output response voltage. Push-pull drive also permits differential charge sensing amplifier DSQ j to be realized with differential-input amplifiers that do not provide for common-mode suppression of their output signals, if one so desires. FIGURE 12A differs from FIGURE 9A in that sign detector CSDi and CSHi do not appear, being relocated to appear in FIGURE 12C as shall be considered further later on. FIGURE 12B differs from FIGURE 9B in that the single-ended charge-sensing amplifier QSi does not appear, being inappropriate for sensing differences in charge appearing on a pair of input lines. Instead, Δj +BBIAS is developed in the following neural net layer and is fed back to analog multiplier AMj via the output multiplexer OM j when the φp signal on mode control line MCL is a ONE. FIGURE 12C shows circuitry that may be used in each neural net layer to provide balanced input signal drive to a pair of input lines ILi and IL(i+ M ) during normal operation - - and to differentially sense the charge on those input lines during back-propagation calculations. A single fully differential amplifier IDi, which is by way of example of the type shown in FIGURE 2 , is multiplexed to implement both functions in duplex circuitry DPXi shown in FIGURE 12C. Alternatively the functions could be implemented with separate apparatus. During normal operation the δp signal appearing on mode control line MCL is a ZERO, conditioning an input multiplexer IMi to apply i signal to the non-inverting (+) input terminal of differential amplifier IDi and conditioning input line multiplexers ILMi and ILM(i+ M ) to connect the non-inverting (+) and inverting (-) output terminals of differential amplifier IDi to input lines ILi and IL i+ ) respectively. A signal θp is a ONE during normal operation and appears in the φu + Θ P control signal applied to a transmission gate between the non-inverting (+) output terminal of differential amplifier IDi and its inverting (-) input terminal, rendering that transmission gate conductive to provide direct-coupled feedback between those terminals. This d-c feedback conditions differential amplifier IDi to provide xi and -xi responses at its (+) and (-) output terminals to the xi signal applied to its (-) input terminal. Other transmission gates within the duplex circuitry DPXi are conditioned to be non-conductive during normal operation. During back-propagation calculations, the φp signal appearing on mode control line MCL is a ONE, conditioning input multiplexer IMi to apply Δi signal from the non- inverting (+) output terminal of differential amplifier IDi to the preceding neural net layer, if any, and conditioning input line multiplexers ILMi and ILM_(i+M) to connect the input lines ILi and I (i + ) to respective ones of the non- inverting (+) and inverting (-) input terminals of differential amplifier IDi. Integrating capacitors ICi and lC(i +M ) connect from the (+) and (-) output terminals of differential amplifier IDi to its (-) and (+) input terminals when transmission gates in duplex circuitry DPXi that are controlled by Φu • Φp signal receive a ZERO during back- propagation calculations. The charge conditions on integrating capacitors ICj. and IC(i+ M ) are reset when φ occasionally pulses to ONE during back-propagation calculations. This happens in response to transmission gates in duplexer circuitry DPXi receptive of φu and φu + Φp control signals being rendered conductive responsive to φu being momentarily a ONE, while transmission gates in duplexer circuitry DPXi receptive of Φu control signal being rendered non-conductive. Column sign detector CSDi and column sample and hold circuit CSHi appear in FIGURE 12C. Column sign detector receives output signal from differential amplifier IDi directly as its input signal and can simply be a voltage comparator for the i and -xi output signals from the differential amplifier IDi. FIGURE 13 shows a neural net comprising a plurality, N in number, of non-linear amplifiers ODi, 0D2, ...OD(N-I), ODN supplied respective input voltages from apparatus for matrix multiplying as described by the inventor in U.S. patent No. 4,156,284 issued 22 May 1979, entitled "SIGNAL PROCESSING APPARATUS" and assigned to General Electric Company. The apparatus for matrix multiplying weights each of a plurality, M in number, of input voltage signals xi, X2, • ■•X( M - l )r XM supplied as "synapse" signals to provide the respective input voltages for the non-linear voltage amplifiers ODi, OD2, ...ODN-1, ODN, which generate respective "axon" responses yi, Input voltage signal i is applied to the input port of an input driver amplifier IDi that in FIGURE 13 in - - turn applies its non-inverted voltage response from its (+) output port to an input line ILi an d applies its inverted voltage response from its (-) output port to an input line lL(i+ M ) • A respective output line OLj connects to the input port of output driver amplifier ODj ,which generates at its output port a non-linear voltage response to the cumulative charge on that respective output line OLj . The non-linear output driver amplifier OD j is shown in FIGURE 13 as being just a non-linear voltage amplifier with the quiescent direct potential applied to its input_ signal terminal via output line OLj being adjusted by clamping to a desired bias voltage at selected times using a respective direct-current restorer circuit DCR j . The restorer circuit DCRj is shown separated from non-linear output driver OD j in FIGURE 13 to improve the layout of the drawing, but customarily is more closely associated with the output driver OD j . in accordance with customary practice in dc restoration. A respective capacitor Ci,j connects each of the input lines ILi to each of the output lines OL j , and a respective capacitor C(i+ ), j connects to each of the output lines OL j the one of the input lines IL(i+ ) paired with that ILi. Since the paired ILi and I (i+M) input lines are driven with xi and - i signal voltages respectively, the electrically equivalent circuit is xi signal voltage be+ng applied to output line OL j by a capacitor having a capacitance that equals the capacitance of Ci,j minus the capacitance of C(i+ M ),j- This balanced input signal drive to paired input lines technique avoids the need for switched-capacitance techniques in order to obtain inhibitory as well as excitory weights, and thus facilitates operating the neural net with analog signals that are continuous over sustained periods of time, if so desired. FIGURE 13 shows each of the input lines ILi or lL(i+M) as being provided with a respective load capacitor tlLj. - 38 - or CL(i+ M ) to cause that capacitive loading upon each of the output terminals of the input driver amplifier IDi to be substantially the same as that upon each output port of the other input driver amplifiers. This is desirable for avoiding unwanted differential delay in responses to the input signals xi. Substantially equal capacitive loading can be achieved by making the capacitance of each of the input line loading capacitors CLi - CL2M very large compared to the total capacitance of the capacitors Ci,j or C(i +M ), j connecting thereto. Preferably, however, this result is achieved by making the capacitance of each of the input line loading capacitors complement 'the combined value of the other capacitances connecting thereto. This procedure reduces the amount of line loading capacitance required. Where the voltage appearing on the output lines is sensed directly by the non-linear output driver amplifiers 0Dχ, ...ODN, as shown in FIGURE 13, this preferable procedure makes the voltage division ratio for each input voltage xi , ... X M independent of the voltage division ratios for the other input voltages. Where the charge appearing on the output lines is sensed by charge-sensing amplifiers preceding the non-linear output driver amplifiers, as will be described later on in this specification in connection with FIGURE 14, this latter consideration is not as important. FIGURE 13 also shows each of the output lines OL j being loaded with a respective load capacitor CL( 2M+ j) to cause the total capacitance on that line to remain substantially the same as on each of the other output lines. Again, this can be done either by choosing CL( 2 +j ) to be much larger than other capacitances to output line OL j , or by choosing CL(2 M +J) to complement the combined value of the other capacitances connecting thereto. The input voltage to output driver amplifier OD j will (to good approximation) have the following value, VJ, in accordance with Coulomb's Law. - - _1 M Vj = (Cj ) ∑ (Ci j - C ( i+M) j ) xi i-1 ( 12 ) Here C j is the total capacitance on output line OL j . The generation of voltage VJ can be viewed as the superposition of a plurality of capacitive divisions between, on the one hand, the effective capacitance (C(i fj ) - C(i+ M ), j ) each input voltage has to output line OLj a nd, on the other hand, the total capacitance Cj of the output line to its surroundings. Consider now how neuron model behavior is exhibited by input driver amplifier IDi , capacitors Ci fj and C ( i +M) , j , and non-linear output driver amplifier OD j for particular respective values of i and j. The voltage responses input driver amplifier IDi applies to input lines ILi and IL(i+M) are the same in amplitude but are of opposing polarity as referred to a common-mode voltage that is designed to be nominally the same as a bias voltage V BI AS midway between the 0-volt Vgs and +5-volt VQ D supply voltages. If the capacitance of capacitor Ci,j is larger than the capacitance of capacitor C(i + ), j for these particular values of i and j, then the output voltage y j for that j will exhibit "excitory" response to the input voltage xi. If the capacitances of Ci, j and C(i +M ), j are equal for these i and j values, then the output voltage yj for that j should exhibit no response to the input voltage y j . If the capacitance of capacitor Ci, j is smaller than the capacitance of capacitor C(i+ M ),j for those i and j values, then the output voltage y j for that j will exhibit "inhibitory" response to the input voltage i. In some neural nets constructed in accordance with the invention the capacitors Ci,j and C(i +M ), for all i and j may be fixed-value capacitors, so there is never any alteration in the weighting of input voltages xi where i=l, ...M. However, such neural nets lack the capacity to adapt to changing criteria for neural responses — which adaptation is necessary, for example, in a neural network that is to be connected for self-learning. It is desirable in certain applications, then, to provide for altering the capacitances of each pair of capacitors Ci,j and C(i+ M ), j associated with a respective pair of values of i and j . This alteration is to be carried out in a complementary way, so the sum of the capacitances of Ci,j and of C(i+ ), j remains equal to C f c. FIGURE 14, comprising component FIGURES 14A and 14B, shows a representative modification that can be made to the FIGURE 13 neural net near each set of intersections of an output line OLj with input lines ILi and IL(i+ M ) driven by opposite senses of a synapse input signal xi. Such modifications together make the neural net capable of being trained. Each capacitor pair Ci,j and C(i+ M ), j of the FIGURE 13 neural net is to be provided by a pair of digital capacitors DCi, j an d DC(i+ M ),j. (For example, each of these capacitors DCi,j an d DC(i+ M ),j may be as shown in FIGURE 11 of U.S. patent No. 3,890,635) . The capacitances of DCi, j and DC(i+ ) , j are controlled in complementary ways by a digital word and its one's complement, as drawn from a respective word-storage element WSEi, j in an array of such elements located interstitially among the rows of digital capacitors and connected to form a memory. The word stored in word storage element WSEi, j may also control the capacitances of a further pair of digital capacitors DCi, (j+N) and DC(i+M) , (J+N) . respectively. The capacitors DCi, (j+N) and DC(i+M), (j+N) connect between "ac ground" and input lines ILi and IL(i+ M ), respectively, and form parts of the loading capacitors CLi and CL(i+ M ), respectively. The capacitances of DC(i+ M , (j+ N ) and DCi, j are similar to each other and changes in their respective values track each other. The four digital capacitors DC^ j , DC(i + M),j, DCi, (J+ ) and DC(i + M), (j+N) are connected in a bridge configuration having input terminals to which the input lines ILi and IL(i+ M ) respectively connect and having output terminals connecting to output line OL j and to ac ground respectively. The capacitances of DCi, (j+N) and DC(i+ M ), j are similar to each other and changes in their respective values track each other. This bridge configuration facilitates making computations associated with back-propagation programming by helping make the capacitance network bilateral insofar as voltage gain is concerned. Alternatively, where the computations for back-propagation programming are done by computers that do not involve the neural net in the computation procedures, the neural net need not include the digital capacitors DCi,J+N and DC(i + M), (j + N) • These digital capacitors DCi, (j + N) and DC(I+M), (j+N) are not needed either where very large loading capacitors are placed on the output lines OL j , but this alternative undesirably reduces sensitivity of the output driver amplifier OD j . When the FIGURE 14 neural net layer is being operated normally, following programming, the φp signal applied to a mode control line MCL is a logic ZERO. This ZERO conditions a respective input line multiplexer ILMi to connect the non-inverting output port at each input driver amplifier IDi to input line ILi. The φp signal on mode control line being a ZERO also conditions a respective input line multiplexer ILM(i + M) to connect the inverting output port of each input driver amplifier IDi to input line IL(i +M ). A ZERO on mode control line MCL also conditions each output line multiplexer OLM j of an n-numbered plurality thereof to select the output line OL j to the inverting input terminal of a respective associated differential-input amplifier DA j , included in a respective charge-sensing amplifier QS j that performs a charge-sensing operation for output line OL j . In furtherance of this charge-sensing - 42 - operation, a transmission gate TG j responds to the absence of a reset pulse Q R to connect an integrating capacitor CI j between the output and inverting-input terminals of differential-input amplifier DAj. Amplifier DA j may be an operational amplifier of the conventional voltage amplifier type or may be an operational transconductance amplifier. With integrating capacitor CIj so connected, amplifier DA j functions as a charge amplifier. When φp signal on mode control line MCL is a ZERO, the input signal xi induces a total change in charge on the capacitors DCj., j and DC ( i +M) , j proportional to the difference in their respective capacitances. The resulting displacement current flow from the inverting input terminal of differential-input amplifier DA j requires that there be a corresponding displacement current flow from the integrating capacitor CI j charging that capacitor to place thereon a voltage VJ defined as follows. Vj = ( CIj ) < C .J " C <i+M).j) i i-i (13) The voltage V j is supplied to a non-linear voltage amplifier circuit NL j , which can be the non-linear voltage amplifier circuit of FIGURE 3 or an alternative circuit as previously described. The non-linear voltage amplifier circuit responds to generate the axon output response j . From time to time, the normal operation of the neural net is interrupted, and to implement dc-restoration a reset pulse Φ R is supplied to each charge sensing amplifier QSj. Responsive to ΦR the logic complement of the reset pulse Φ R , going low when Φ goes high, transmission gate TG j is no longer rendered conductive to connect the integrating capacitor CIj from the output terminal of differential amplifier DAj. Instead, a transmission gate TG(J+ N) responds to Φ going high to connect to V BI AS the plate of capacitor C j normally connected from that output terminal, VBIAS being the 2.5 volt intermediate potential between the Vgs = 0 volt and VDD = 5 volt operating voltages of differential amplifier DA . Another transmission gate G(J+2N) responds to Φ going high to apply direct-coupled feedback from the output terminal of differential amplifier DA j to its inverting input terminal, to bring the voltage at the output terminal to that supplied to its inverting input terminal from output line OL j . During the dc-restoration all xi are "zero-valued". So, the charge on integrating capacitor CIj is adjusted to compensate for any direct voltage error occurring in the circuitry up to the output terminal of differential amplifier DA j . DC- restoration is done concurrently for all differential amplifiers DA j (i.e., for values of j ranging from one to N) . During training, the φp signal applied to mode control line MCL is a logic ONE, which causes the output line multiplexer OLM j to disconnect the output line OL j from the inverting input terminal of differential amplifier DA j and to connect the output line OLj to receive a δ j error term. This δ j error term is generated as the product output signal of a ' analog multiplier AM j , responsive to a signal Δ j and to a signal y' j which is the change in output voltage yj of non¬ linear amplifier NL j for unit change in the voltage on output line OLj . The term Δj is for the output neural net layer tϊ e difference between yj actual value and its desired value d j . The term Δ j is for a hidden neural net layer the Δ j output of the succeeding neural net layer during the back-propagation procedure. As was the case in the FIGURE 4 neural net layer ifi the FIGURE 14 neural net layer differentiator DF j generates the signal y' j , which is a derivative indicative of the slope of y j change in voltage on output line OL j , superposed on VBIAS- TO determine the y'j derivative, a pulse do blet comprising a small positive-going pulse immediately followed - 44 - by a similar-amplitude negative-going pulse is introduced at the inverting input terminal of differential amplifier DA j (or equivalently, the opposite-polarity doublet pulse is introduced at the non-inverting input terminal of differential amplifier DAj) to first lower y j slightly below normal value and then raise it slightly above normal value. This transition of y j from slightly below normal value to slightly above normal value is applied via a differentiating capacitor CD j to differentiator DF j . During training, the φp signal applied to the mode control line MCL is a ONE, as previously noted, and this causes the input line multiplexers ILMi and ILM(i+ M ) to disconnect the input lines ILi and IL(i+ M ) from the input driver amplifier IDi output terminals and connect them instead to the non-inverting and inverting input terminals of a differential charge-sensing amplifier DQSi. The voltage δj induces a differential change in charge between input lines IL j and IL ( i+ M ) proportional to δ j (Cj._ j - C(i+ M ), j ), which differential change in charge is sensed using the differential charge sensing amplifier DQSi. Differential charge-sensing amplifier DQSi includes a fully differential amplifier (as shown in FIGURE 2, for example) provided with integrating capacitors ICi and IC(i+ M ) in respective degenerative feedback connections from each of its output terminals to each of its input terminals. Resetting of differential charge-sensing amplifier DQSi is similar to the resetting of a single-ended amplifier such as QS j , except for involving two integrating capacitors ICi and TC(i+ m ), rather than just the one integrating capacitor CI j . Resetting of differential charge-sensing amplifier DQSi is done responsive to a pulse φu, which occurs during the time when mode control line MCL has a ONE thereon conditioning input line multiplexers ILMi and IM(i+ m ) to connect input lines ILi and IL(i+ m ) to the differential charge-sensing amplifier DQSi. Resetting is normally done shortly after a ZERO to ONE transition appears in the φp signal applied to mode control line MCL and may also be done at other times. This procedure corrects for capacitive unbalances on the input lines ILi and IL(i+ M ) during back-propagation computations that follow the resetting procedure. In these computations voltages +Δi + VBI S an d ~Δi V BIAS ar e developed at the (+) and (-) output terminals of the fully differential amplifier included in differential charge-sensing amplifier DQSi. Tne voltage +Δi + V B j A s is used by the preceding neural net layer during the back-propagation training procedure, if such a preceding neural net layer exists. The use of single- ended +Δi and +Δ j drive is shown in FIGURE 14, presuming the neural net layers to be integrated within separate monolithic integrated circuits, and presuming the limitation on number of pin-outs is restrictive. Where a plurality of neural net layers are integrated within the same monolithic integrated circuitry, or where maximum pin-out count is not a restrictive design factor, balanced Δ signals may be applied from one neural net layer to the preceding one. So, too, if the non-linear voltage amplifier NL j is of a correct type (for example, a long-tailed pair connection of transistors) yj + V BIAS and -yj + V BIA S balanced output signals may be supplied to the next neural net layer. Indeed, the y' j signals applied to the analog multiplier AF- may be generated in balanced form, replacing differentiator DFj and sample-and- hold circuit SH j with balanced circuitry. FIGURE 5 shows how each output line OL j for j — 1, ...N may be pulsed during calculation of y' j terms. Each output line OL j is connected by a respective capacitor CO j to the output terminal of a pulse generator PG, which generates the doublet pulse. FIGURE 5 shows the doublet pulse applied to the end of each output line OLj remote from the - terminal of the associated differential amplifier DA j in the charge- sensing amplifier QS j sensing the charge on that line. It is also possible to apply the doublet pulses more directly to those (-) terminals by connecting to these terminals respective ones of the plates of capacitors COj that are remote from the plates connecting to pulse generator PG. FIGURE 15 shows how the non-inverting (+) input terminal of each differential amplifier DAj may be pulsed during calculation of y' j terms, rather than pulsing the OL j output lines. VBIAS* rather than being applied directly to the non-inverting input terminals of each differential amplifier DA j , is applied from the output terminal of a biasing operational amplifier BOA. Operational amplifier BOA has its non-inverting terminal connected to receive VBIAS . and has degenerative feedback applied via a resistor Rl from its output terminal to its inverting input terminal. This degenerative feed holds the quiescent level of operational amplifier BOA output terminal at VBIAS- A pulse generator PG applies a doublet pulse via the series connection of a resistor R2 having resistance and a dc-blocking capacitor CB to the inverting input terminal of operational amplifier BOA. Presuming the resistances of resistors Rl and R2 to be Ri and R2, respectively, a doublet pulse -R 1 /R2 times the amplitude of the doublet pulse supplied from the pulse generator PG is applied from the output terminal of operational amplifier DBA to the non-inverting input terminals of differential amplifiers DA j . This doublet pulse, having no direct voltage component, does not alter the VBIAS direct component maintained at the output terminal of operational amplifier BOA by the degenerative feedback connection back to the inverting input connection of BOA. Arrangements for adding the doublet pulse to VJ before its application to the non-linear amplifier NL j can be used, rather than using the FIGURE 5 or FIGURE 15 arrangement. The FIGURE 7 apparatuses for completing the back- propagation computations can also be used with the FIGURE 13 neural net layers manifoldly modified per FIGURE 14. In FIGURE 8 the trained neural net layers Lo, Li and L2 may alternatively be constructed using FIGURE 13 neural net layers manifoldly modified per FIGURE 14. FIGURE 16, comprising component FIGURES 16A and 16B shows an alternative modification that can be manifoldly made to the FIGURE 13 neural net layer to give it training capability. This alternative modification seeks to avoid the need for a high-resolution multiplier MULT and complex addressing during back-propagation calculations in order that training can be implemented. A respective up/down counter UDCi, j is used instead of each word storage element WSEi, j . Correction of the word stored in counter UDCi,j is done a count at a time; and the counter preferably has at least one higher resolution stage in addition to those used to control the capacitances of digital capacitors DCi, j , DC ( i+ M) , j , DCi, (J+N) and DC(i+M), (J+N) • Each up/down counter UDCi,j has a respective counter control circuit CONi f j associated therewith. Each counter control circuit CONi, j may, as shown in FIGURE 16A,and described in detail further on in this specification simply consist of an exclusive-OR gate XORi,j. A row sign detector RSDj detects whether the polarity of δ j is positive or negative, indicative of whether a row of weights should in general be decremented or incremented, and broadcasts its detection result via a row sign line RSL j to all counter control circuits (CONi, j for i=l, ...,M) in the row j associated with that row sign detector RSD j . Before making a back-propagation calculation, a respective column sign detector CSDi detects whether the polarity of xi is positive or negative for each columnar position along the row which is to be updated, to provide an indication of whether it is likely the associated weight - 48 - should be decremented or incremented. This indication is stored temporarily in a (column) sample and hold circuit CSHi. Each column sample and hold circuit CSHi is connected to broadcast its estimate via a column sign line CSLi to all counter control circuits (C0Ni,j for j=l, ...N) in the column i associated with that sample and hold circuit CSHi. Responsive to these indications from sign detectors CSDi and RSD j , each respective counter control circuit CONi, j decides in which direction up/down counter UDCi,j will count to adjust the weight control signals Di, j and Di, j stored therein The counter .control circuitry CO i fj should respond to the sign of +δ j being positive, indicating the response VJ to be too positive, to decrease the capacitance to output line OL j that is associated with the signal xi or -X that is positive and to increase the capacitance to output line OL j that is associated with the signal -xi or i that is negative, for each value of . i. The counter control circuitry CONi, should respond to the sign of +δ j being negative, indicating the response v to be too negative, to increase the capacitance to output line OL j that is associated with the signal -xi or xi that is negative and to decrease the capacitance to output line OL j that is associated with the signal xi or -xi that is positive. Accordingly, counter control circuitry CONi, j may simply consist of a respective exclusive-OR gate X0Ri, j as shown in FIGURE 16A, if the following presumptions are valid. Each of the digital capacitors DCi,j and DC(i+ M ), (j+ N ) is presumed to increase or decrease its capacitance as Di, j is increased or decreased respectively. Each of the digital capacitors DC(i+ M ),j and DCi, ( j + N ) is presumed to increase or decrease its capacitance as Di, j is increased or decreased respectively. A ZERO applied as up/down signal to up/down counter UDCi, j is presumed to cause - 4 - counting down for Di, j and counting up for OL,J. A ONE applied as up/down signal to up/down counter UDCi, j is presumed to cause counting up for Di, j and counting down for Di, j . Column sign detector CSDi output indication is presumed to be a ZERO when xi is not negative and to be a ONE when x is negative. Row sign detector RSD j output indication is presumed to be a ZERO when δ j is not negative and to be a ONϊf when δ j is negative. Since the condition where xi or δ j is zero-valued is treated as if the zero-valued number were positive, forcing a false correction which is in fact not- necessary, and thus usually creating the need for a counter correction in the next cycle of back-propagation training, there is dither in the correction loops. However, the extra stage or stages of resolution in each up/down counter UDCi, j prevent high-resolution dither in the feedback correction loop affecting the capacitances of DCi,j , DC(i +M ),j, DCi, ( j+ N) and DC(i +M ), (j+N) • Analog multiplier AM j can <be readily modified to develop its product output in balanced form. The Bult and Wallinga analog multiplier initially develops its product in balanced form and follows this with a push-pull to single- ended converter connection using current mirror multipliers. An additional balanced to single-ended converter can convert the push-pull product to opposite-sense output signal. With an analog multiplier generating balanced product signals, tlie +δ j and -δ j terms can be supplied to a voltage comparator that serves as the row sign detector RSDj. Alternatively, since the derivative y' j always has the same sign (normally a positive one) , one can use a voltage comparator to compare the voltages supplied to the asterisked and double-asterisked input terminals of the analog multiplier AMj, for providing the row sign detector RSD j . A further modification in accordance with FIGURE 12C can be made to the FIGURE 13 neural net layer modified - 50 - per FIGURE 16. Input driver amplifier IDi is time-division- multiplexed in duplex circuitry DPXi for performing the function of the differential charge-sensing amplifier DQS in FIGURE 16A. Only one set of interconnecting lines is used between neural net layers, both for conducting i + V BIA S signals to a neural net layer during forward-propagation when φp is ZERO and for conducting Δi + V BIAS signals from that neural net layer during back-propagation when φp is ONE. An input multiplexer IMIi controlled from the φp, which is signal on the mode control line MCL, performs the time-division- multiplexing for the neural net layer including input driver amplifier IDi• A* 1 output multiplexer OMi for the preceding neural net layer controlled from the φp signal in its mode controlling MCL' selects yi + VBIAS to the interconnecting line between the layers during forward propagation when φp is ZERO, and selects Δi + V B I AS from that interconnecting line during back-propagation when φp is ONE. Similar further modification can be made to the FIGURE 13 neural net layer modified per FIGURE 14. In such case column sign detector CSDi column sample and hold circuit CSHi, and column sign line CSLi would, of course, not be used. The time-division-multiplexed use of the input driver amplifiers IDi saves integrated-circuit die area. The conservation of interconnecting lines between successive neural net layers is of substantial importance, since pin-out limitations on interconnected integrated circuits having different neural net layers therein are less restrictive of neural net layer size. A neural net layer with twice as many inputs and outputs can be constructed within an integrated circuit having given pin-out limitations. The multiplexers employed in various portions of the circuits described above are customarily constructed of single-pole switch elements, each of which single-pole switch elements is conventionally a so-called "transmission gate" connection of one or more field effect transistors in CMOS design. A suitable transmission gate is provided by the paralleled channels of a p-channel FET and an n-channel FET having oppositely swinging control voltages applied to their respective gate electrodes to control the selective conduction of those paralleled channels.

Claims: - 52 - What is claimed is: 1. A processor for providing neuron-like response to a plurality M in number of input voltage signals identified by respective ones of the consecutive ordinal numbers first through M th ; said processor having a first plurality M in number of weighting capacitance elements identified by respective ones of the consecutive ordinal numbers first through M th each charged in response to an input signal voltage identified by the corresponding ordinal numbers . having a charge sensing circuit connected for sensing the combined charge on said first through M th weighting capacitance elements to generate a weighted summation response; and having a non-linear amplier connected to provide said neuron-like response responsive to the weighted summation response; said processor BEING CHARACTERIZED BY a second plurality M in number of weighting capacitance elements identified by respective ones of the consecutive ordinal numbers (M+l) th through 2M th each charged in response to an input signal voltage identified by the ordinal number lower by M or to the negative of that input signal voltage; said charge sensing circuit connected for sensing the combined charge on said (M+l) th through 2M th weighting capacitance elements as well as the combined charge on said first through M th weighting capacitance elements to generate the weighted summation response, the sensing of the combined charge on said (M+l) th through 2M th weighting capacitance elements and of the combined charge on said first through 0 th weighting capacitance elements being done differentially in the case where the (M+l) th through 2M th weighting capacitance elements are charged by the negatives of the input signal voltages and being done supplementarily where the (M+l) th through 2M th weighting capacitance elements are charged by the input signal voltages rather than their negatives. 2. A processor as set forth in claim 1 CHARACTERIZED BY being included as one of a plurality N in number of such processors respectively identified by respective consecutive ordinal numbers first through N th which receive the same set of input signals, thereby forming a neural net layer including said first through N th processors. 3. A processor as set forth in claim 2 CHARACTERIZED BY being included in a neural net layer that receives the same set of input signals from another neural net layer. 4. A processor as set forth in claim 2 CHARACTERIZED BY being included in a neural net layer that supplies its M neuron-like responses as a set of input signals to another neural net layer. 5. A processor as set forth in any of claims 1, 2, 3 or 4 CHARACTERIZED BY each of said weighting capacitance elements being a respective digital capacitor of the same type as each of the others, each of which digital capacitors has a capacitance that is programmed by a respective binary number programming signal. 6. A processor as set forth in claim 5 CHARACTERIZED BY a respective binary counter for each pair of digital capacitors providing ones of said weighting capacitance elements having respective ordinal numbers that differ by M, which respective binary counter supplies its corresponding digital capacitors their binary number programming signals; and circuitry for selectively incrementing or decrementing the count in each respective binary counter. 7. A processor as set forth in claim 5 CHARACTERIZED BY a respective memory array for binary numbers having a respective word storage element for each pair of digital capacitors providing ones of said weighting capacitance elements having respective ordinal numbers that differ by M, which respective word storage element supplies its corresponding digital capacitors their binary number programming signals. 8. A processor as set forth in claim 3 or 4 CHARACTERIZED BY each of said weighting capacitance elements being a respective digital capacitor of the same type as each of the others, each of which digital capacitors has a capacitance that is programmed by a respective binary number programming signal; a respective binary counter for each pair of digital capacitors providing ones of said weighting capacitance elements having respective ordinal numbers that differ by M, which respective binary counter supplies its corresponding digital capacitors their binary number programming signals; and circuitry for selectively incrementing or decrementing the count in each respective binary counter in accordance with back propagation calculations made using said first and second pluralities of weighting capacitance elements for carrying out analog computation. 9. A processor as set forth in claim 3 or 4 CHARACTERIZED BY each of said weighting capacitance elements being a respective digital capacitor of the same type as each of the others, each of which digital capacitors has a capacitance that is programmed by a respective binary number programming signal; a respective memory array for binary numbers having a respective word storage element for each pair of digital capacitors providing ones of said weighting capacitance elements having respective ordinal numbers that differ by M, which respective word storage element supplies its corresponding digital capacitors their binary number programming signals; and circuitry for selectively incrementing or decrementing the binary number stored in each respective word storage element in accordance with back propagation calculations made using said first and second pluralities of weighting capacitance elements for carrying out analog computation.

Inventors: ENGELER WILLIAM ERNEST, ENGELER, WILLIAM, ERNEST
Applicants: GEN ELECTRIC, GENERAL ELECTRIC COMPANY
Application year: 1990
Main classification: G06F15
IPC codes: G06F15, G06G7, G06N1, G06N3