Patent Description:
Numerous embodiments are disclosed for a hybrid output architecture for an analog neural memory in a deep learning artificial neural network.

<CIT> describes a read-out unit for a storage unit array and a storage and calculation integrated chip comprising the read-out unit. The read-out unit comprises an ADC, a decision subunit and amultiplexer switch. The ADC and the input end of the decision subunit both receive input voltage, and the output end of the decision subunit is connected with the control end of the ADC and the address gating end of the multiplexer switch; the first input end of the multiplexer switch is connected with the output end of the ADC, the second input end of the multiplexer switch receives a first level, the third input end of the multiplexer switch receives a second level, and the first level is smaller than the second level.

<CIT> describes a design of a digital-analog hybrid reading circuit applied to eFlash-based storage and calculation integrated circuit, which is used for meeting the requirements of different multiply-add operation combinations during reasoning of a DNN model and achieving higher energy consumption efficiency and higher calculation accuracy. The circuit adopted by the design comprises a TIA(Trans-Impedance Amplifier) with negative feedback, a VGA (Variable Gain Amplifier) and an ADC (Analog-to-Digital Converter). The weighted current is converted into digital output to complete the vector-matrix multiplication and addition operation. The circuit formed by the combination of the TIA and the VGA is called as a VGTIA (Variable Gain Trans-Impedance Amplifier). The design belongs to a part of a storage and calculation integrated vector-matrix multiply-add analog calculation core based on embedded NOR Flash.

Document "Raqibul Hasan et al: On-chip training of memristor crossbar based multi-layer neural networks" describes on-chip training circuits for multi-layer neural networks implemented using a single crossbar per layer and two memristors per synapse.

Document "Yao Peng et al: Fully hardware-implemented memristor convolutional neural network" describes a five-layer memristor-based CNN to perform MNIST image recognition. In addition to parallel convolutions using different kernels with shared inputs, replication of multiple identical kernels in memristor arrays are demonstrated for processing different inputs in parallel.

<CIT> describes a neural network computing circuit, chip and system, and belongs to the technical field of neural networks. The neural network calculation circuit comprises a first calculationunit, a second calculation unit and a processing circuit. The first calculation unit is used for obtaining a first current according to the input end voltage of the first calculation unit, the outputend voltage of the first calculation unit and a set first weight value. And the second calculation unit is used for obtaining a second current according to the input end voltage of the second calculation unit, the output end voltage of the second calculation unit and a set second weight value. The processing circuit is connected with the output end of the first calculation unit and the output endof the second calculation unit, and is used for obtaining a target current difference according to the first current and the second current, and obtaining an output voltage used for indicating a target calculation result according to the target current difference, wherein the target calculation result is used for indicating the calculation result of the neuron on the input data based on the weight value of the input data.

<FIG> illustrates an artificial neural network, where the circles represent the inputs or layers of neurons. The connections (called synapses) are represented by arrows and have numeric weights that can be tuned based on experience. This makes neural networks adaptive to inputs and capable of learning. Typically, neural networks include a layer of multiple inputs. There are typically one or more intermediate layers of neurons, and an output layer of neurons that provide the output of the neural network. The neurons at each level individually or collectively make a decision based on the received data from the synapses.

One of the major challenges in the development of artificial neural networks for high-performance information processing is a lack of adequate hardware technology. Indeed, practical neural networks rely on a very large number of synapses, enabling high connectivity between neurons, i.e., a very high computational parallelism. In principle, such complexity can be achieved with digital supercomputers or specialized graphics processing unit clusters. However, in addition to high cost, these approaches also suffer from mediocre energy efficiency as compared to biological networks, which consume much less energy primarily because they perform low-precision analog computation. CMOS analog circuits have been used for artificial neural networks, but most CMOS-implemented synapses have been too bulky given the high number of neurons and synapses.

Applicant previously disclosed an artificial (analog) neural network that utilizes one or more non-volatile memory arrays as the synapses in <CIT>. The non-volatile memory arrays operate as an analog neural memory. The neural network device includes a first plurality of synapses configured to receive a first plurality of inputs and to generate therefrom a first plurality of outputs, and a first plurality of neurons configured to receive the first plurality of outputs. The first plurality of synapses includes a plurality of memory cells, wherein each of the memory cells includes spaced apart source and drain regions formed in a semiconductor substrate with a channel region extending there between, a floating gate disposed over and insulated from a first portion of the channel region and a non-floating gate disposed over and insulated from a second portion of the channel region. Each of the plurality of memory cells is configured to store a weight value corresponding to a number of electrons on the floating gate. The plurality of memory cells is configured to multiply the first plurality of inputs by the stored weight values to generate the first plurality of outputs.

Non-volatile memories are well known. For example, <CIT> ("the '<NUM> patent"), discloses an array of split gate non-volatile memory cells, which are a type of flash memory cells. Such a memory cell <NUM> is shown in <FIG>. Each memory cell <NUM> includes source region <NUM> and drain region <NUM> formed in semiconductor substrate <NUM>, with channel region <NUM> there between. Floating gate <NUM> is formed over and insulated from (and controls the conductivity of) a first portion of the channel region <NUM>, and over a portion of the source region <NUM>. Word line terminal <NUM> (which is typically coupled to a word line) has a first portion that is disposed over and insulated from (and controls the conductivity of) a second portion of the channel region <NUM>, and a second portion that extends up and over the floating gate <NUM>. The floating gate <NUM> and word line terminal <NUM> are insulated from the substrate <NUM> by a gate oxide. Bitline <NUM> is coupled to drain region <NUM>.

Memory cell <NUM> is erased (where electrons are removed from the floating gate) by placing a high positive voltage on the word line terminal <NUM>, which causes electrons on the floating gate <NUM> to tunnel through the intermediate insulation from the floating gate <NUM> to the word line terminal <NUM> via Fowler-Nordheim (FN) tunneling.

Memory cell <NUM> is programmed by source side injection (SSI) with hot electrons (where electrons are placed on the floating gate) by placing a positive voltage on the word line terminal <NUM>, and a positive voltage on the source region <NUM>. Electron current will flow from the drain region <NUM> towards the source region <NUM>. The electrons will accelerate and become heated when they reach the gap between the word line terminal <NUM> and the floating gate <NUM>. Some of the heated electrons will be injected through the gate oxide onto the floating gate <NUM> due to the attractive electrostatic force from the floating gate <NUM>.

Table No. <NUM> depicts typical voltage and current ranges that can be applied to the terminals of memory cell <NUM> for performing read, erase, and program operations:.

Other split gate memory cell configurations, which are other types of flash memory cells, are known. For example, <FIG> depicts a four-gate memory cell <NUM> comprising source region <NUM>, drain region <NUM>, floating gate <NUM> over a first portion of channel region <NUM>, a select gate <NUM> (typically coupled to a word line, WL) over a second portion of the channel region <NUM>, a control gate <NUM> over the floating gate <NUM>, and an erase gate <NUM> over the source region <NUM>.

<FIG> depicts a three-gate memory cell <NUM>, which is another type of flash memory cell. Memory cell <NUM> is identical to the memory cell <NUM> of <FIG> except that memory cell <NUM> does not have a separate control gate. The erase operation (whereby erasing occurs through use of the erase gate) and read operation are similar to that of the <FIG> except there is no control gate bias applied. The programming operation also is done without the control gate bias, and as a result, a higher voltage must be applied on the source line during a program operation to compensate for a lack of control gate bias.

<FIG> depicts stacked gate memory cell <NUM>, which is another type of flash memory cell. Memory cell <NUM> is similar to memory cell <NUM> of <FIG>, except that floating gate <NUM> extends over the entire channel region <NUM>, and control gate <NUM> (which here will be coupled to a word line) extends over floating gate <NUM>, separated by an insulating layer (not shown). The erase is done by FN tunneling of electrons from FG to substrate, programming is by channel hot electron (CHE) injection at region between the channel <NUM> and the drain region <NUM>, by the electrons flowing from the source region <NUM> towards to drain region <NUM> and read operation which is similar to that for memory cell <NUM> with a higher control gate voltage.

The methods and means described herein may apply to other non-volatile memory technologies such as FINFET split gate flash or stack gate flash memory, NAND flash, SONOS (silicon-oxide-nitride-oxide-silicon, charge trap in nitride), MONOS (metal-oxide-nitride-oxide-silicon, metal charge trap in nitride), ReRAM (resistive ram), PCM (phase change memory), MRAM (magnetic ram), FeRAM (ferroelectric ram), CT (charge trap) memory, CN (carbon-tube) memory, OTP (bi-level or multi-level one time programmable), and CeRAM (correlated electron ram), without limitation.

S0 is the input layer, which for this example is a 32x32 pixel RGB image with <NUM> bit precision (i.e. three 32x32 pixel arrays, one for each color R, G and B, each pixel being <NUM> bit precision). The synapses CB1 going from input layer S0 to layer C1 apply different sets of weights in some instances and shared weights in other instances, and scan the input image with 3x3 pixel overlapping filters (kernel), shifting the filter by <NUM> pixel (or more than <NUM> pixel as dictated by the model). Specifically, values for <NUM> pixels in a 3x3 portion of the image (i.e., referred to as a filter or kernel) are provided to the synapses CB1, where these <NUM> input values are multiplied by the appropriate weights and, after summing the outputs of that multiplication, a single output value is determined and provided by a first synapse of CB1 for generating a pixel of one of the feature maps of layer C1. The 3x3 filter is then shifted one pixel to the right within input layer S0 (i.e., adding the column of three pixels on the right, and dropping the column of three pixels on the left), whereby the <NUM> pixel values in this newly positioned filter are provided to the synapses CB1, where they are multiplied by the same weights and a second single output value is determined by the associated synapse. This process is continued until the 3x3 filter scans across the entire 32x32 pixel image of input layer S0, for all three colors and for all bits (precision values). The process is then repeated using different sets of weights to generate a different feature map of layer C1, until all the features maps of layer C1 have been calculated.

The purpose of the pooling function P1 is to average out the nearby location (or a max function can also be used), to reduce the dependence of the edge location for example and to reduce the data size before going to the next stage. The synapses CB2 going from layer S1 to layer C2 scan maps in layer S1 with 4x4 filters, with a filter shift of <NUM> pixel.

<FIG> is a block diagram of an array that can be used for that purpose. Vector-by-matrix multiplication (VMM) array <NUM> includes non-volatile memory cells and is utilized as the synapses (such as CB1, CB2, CB3, and CB4 in <FIG>) between one layer and the next layer. Specifically, VMM array <NUM> includes an array of non-volatile memory cells <NUM>, erase gate and word line gate decoder <NUM>, control gate decoder <NUM>, bit line decoder <NUM> and source line decoder <NUM>, which decode the respective inputs for the non-volatile memory cell array <NUM>. Input to VMM array <NUM> can be from the erase gate and wordline gate decoder <NUM> or from the control gate decoder <NUM>. Source line decoder <NUM> in this example also decodes the output of the non-volatile memory cell array <NUM>. Alternatively, bit line decoder <NUM> can decode the output of the non-volatile memory cell array <NUM>.

Non-volatile memory cell array <NUM> serves two purposes. Second, the non-volatile memory cell array <NUM> effectively multiplies the inputs by the weights stored in the non-volatile memory cell array <NUM> and adds them up per output line (source line or bit line) to produce the output, which will be the input to the next layer or input to the final layer. By performing the multiplication and addition function, the non-volatile memory cell array <NUM> negates the need for separate multiplication and addition logic circuits and is also power efficient due to its in-situ memory computation.

The output of non-volatile memory cell array <NUM> is supplied to a differential summer (such as a summing op-amp or a summing current mirror) <NUM>, which sums up the outputs of the non-volatile memory cell array <NUM> to create a single value for that convolution. The differential summer <NUM> is arranged to perform summation of positive weight and negative weight.

The summed-up output values of differential summer <NUM> are then supplied to an activation function block <NUM>, which rectifies the output. The activation function block <NUM> may provide sigmoid, tanh, or ReLU functions. The rectified output values of activation function block <NUM> become an element of a feature map as the next layer (e.g. C1 in <FIG>), and are then applied to the next synapse to produce the next feature map layer or final layer. Therefore, in this example, non-volatile memory cell array <NUM> constitutes a plurality of synapses (which receive their inputs from the prior layer of neurons or from an input layer such as an image database), and summing op-amp <NUM> and activation function block <NUM> constitute a plurality of neurons.

The input to VMM array <NUM> in <FIG> (WLx, EGx, CGx, and optionally BLx and SLx) can be analog level, binary level, or digital bits (in which case a DAC is provided to convert digital bits to appropriate input analog level) and the output can be analog level, binary level, or digital bits (in which case an output ADC is provided to convert output analog level into digital bits).

<FIG> is a block diagram depicting the usage of numerous layers of VMM arrays <NUM>, here labeled as VMM arrays 32a, 32b, 32c, 32d, and 32e. As shown in <FIG>, the input, denoted Inputx, is converted from digital to analog by a digital-to-analog converter <NUM> and provided to input VMM array 32a. The input D/A conversion for the first layer could be done by using a function or a LUT (look up table) that maps the inputs Inputx to appropriate analog levels for the matrix multiplier of input VMM array 32a. The input conversion could also be done by an analog to analog (A/A) converter to convert an external analog input to a mapped analog input to the input VMM array 32a.

The output generated by input VMM array 32a is provided as an input to the next VMM array (hidden level <NUM>) 32b, which in turn generates an output that is provided as an input to the next VMM array (hidden level <NUM>) 32c, and so on. The various layers of VMM array <NUM> function as different layers of synapses and neurons of a convolutional neural network (CNN). Each VMM array 32a, 32b, 32c, 32d, and 32e can be a stand-alone, physical non-volatile memory array, or multiple VMM arrays could utilize different portions of the same physical non-volatile memory array, or multiple VMM arrays could utilize overlapping portions of the same physical non-volatile memory array. The example shown in <FIG> contains five layers (32a,32b,32c,32d,32e): one input layer (32a), two hidden layers (32b,32c), and two fully connected layers (32d,32e). One of ordinary skill in the art will appreciate that this is merely exemplary and that a system instead could comprise more than two hidden layers and more than two fully connected layers.

<FIG> depicts neuron VMM array <NUM>, which is particularly suited for memory cells <NUM> as shown in <FIG> and is utilized as the synapses and parts of neurons between an input layer and the next layer.

As described herein for neural networks, the non-volatile memory cells of VMM array <NUM>, i.e. the memory cells <NUM> of VMM array <NUM>, are preferably configured to operate in a sub-threshold region.

The non-volatile reference memory cells and the non-volatile memory cells described herein are biased in weak inversion (sub threshold region): <MAT> where w = e (- Vth)/nVt where Ids is the drain to source current; Vg is gate voltage on the memory cell; Vth is threshold voltage of the memory cell; Vt is thermal voltage = k*T/q with k being the Boltzmann constant, T the temperature in Kelvin, and q the electronic charge; n is a slope factor = <NUM> + (Cdep/Cox) with Cdep = capacitance of the depletion layer, and Cox capacitance of the gate oxide layer; Io is the memory cell current at gate voltage equal to threshold voltage, Io is proportional to (Wt/L)*u*Cox* (n-<NUM>) * Vt<NUM> where u is carrier mobility and Wt and L are width and length, respectively, of the memory cell.

For an I-to-V log converter using a memory cell (such as a reference memory cell or a peripheral memory cell) or a transistor to convert input current into an input voltage: <MAT> where, wp is w of a reference or peripheral memory cell.

For a memory array used as a vector matrix multiplier VMM array with the current input, the output current is: <MAT> namely <MAT> <MAT> Here, wa = w of each memory cell in the memory array. Vthp is effective threshold voltage of the peripheral memory cell and Vtha is effective threshold voltage of the main (data) memory cell. Note that the threshold voltage of a transistor is a function of substrate body bias voltage and the substrate body bias voltage, denoted Vsb, can be modulated to compensate for various conditions, on such temperature. The threshold voltage Vth can be expressed as: <MAT>.

Where Vth0 is threshold voltage with zero substrate bias, ϕF is a surface potential, and gamma is a body effect parameter.

Alternatively, the flash memory cells of VMM arrays described herein can be configured to operate in the linear region: <MAT> <MAT> meaning weight W in the linear region is proportional to (Vgs-Vth).

For an I-to-V linear converter, a memory cell (such as a reference memory cell or a peripheral memory cell) or a transistor operating in the linear region can be used to linearly convert an input/output current into an input/output voltage.

Alternatively, the memory cells of VMM arrays described herein can be configured to operate in the saturation region: <MAT> Wα (Vgs-Vth)<NUM>, meaning weight W is proportional to (Vgs-Vth)<NUM>.

Alternatively, the memory cells of VMM arrays described herein can be used in all regions or a combination thereof (sub threshold, linear, or saturation) for each layer or multi layers of a neural network.

Other embodiments for VMM array <NUM> of <FIG> are described in <CIT>. As described in that application. a sourceline or a bitline can be used as the neuron output (current summation output).

<FIG> depicts neuron VMM array <NUM>, which is particularly suited for memory cells <NUM> as shown in <FIG> and is utilized as the synapses between an input layer and the next layer.

Table No. <NUM> depicts operating voltages and currents for VMM array <NUM>.

<FIG> depicts neuron VMM array <NUM>, which is particularly suited for memory cells <NUM> as shown in <FIG> and is utilized as the synapses and parts of neurons between an input layer and the next layer.

<FIG> depicts neuron VMM array <NUM>, which is particularly suited for memory cells <NUM> as shown in <FIG> and is utilized as the synapses and parts of neurons between an input layer and the next layer.

VMM array <NUM> implements uni-directional tuning for non-volatile memory cells in memory array <NUM>. That is, each non-volatile memory cell is erased and then partially programmed until the desired charge on the floating gate is reached. If too much charge is placed on the floating gate (such that the wrong value is stored in the cell), the cell is erased and the sequence of partial programming operations starts over. As shown, two rows sharing the same erase gate (such as EG0 or EG1) are erased together (which is known as a page erase), and thereafter, each cell is partially programmed until the desired charge on the floating gate is reached.

Table No. <NUM> depicts operating voltages and currents for VMM array <NUM>.

<FIG> depicts neuron VMM array <NUM>, which is particularly suited for memory cells <NUM> as shown in <FIG> and is utilized as the synapses and parts of neurons between an input layer and the next layer. In VMM array <NUM>, the inputs INPUT<NUM>. , INPUTN are received on bit lines BL<NUM>,. BLN, respectively, and the outputs OUTPUT<NUM>, OUTPUT<NUM>, OUTPUT<NUM>, and OUTPUT<NUM> are generated on source lines SL<NUM>, SL<NUM>, SL<NUM>, and SL<NUM>, respectively.

<FIG> depicts neuron VMM array <NUM>, which is particularly suited for memory cells <NUM> as shown in <FIG> and is utilized as the synapses and parts of neurons between an input layer and the next layer. In this example, the inputs INPUT<NUM>, INPUT<NUM>, INPUT<NUM>, and INPUT<NUM> are received on source lines SL<NUM>, SL<NUM>, SL<NUM>, and SL<NUM>, respectively, and the outputs OUTPUT<NUM>,. OUTPUTN are generated on bit lines BL<NUM>,.

In this example, the inputs INPUT<NUM>,. , INPUTM are received on word lines WL<NUM>,. , WLM, respectively, and the outputs OUTPUT<NUM>,. OUTPUTN are generated on bit lines BL<NUM>,.

In this example, the inputs INPUT<NUM>,. , INPUTn are received on vertical control gate lines CG<NUM>,. , CGN, respectively, and the outputs OUTPUT<NUM> and OUTPUT<NUM> are generated on source lines SL<NUM> and SL<NUM>.

In this example, the inputs INPUT<NUM>,. , INPUTN are received on the gates of bit line control gates <NUM>-<NUM>, <NUM>-<NUM>,. , <NUM>-(N-<NUM>), and <NUM>-N, respectively, which are coupled to bit lines BL<NUM>,. , BLN, respectively. Exemplary outputs OUTPUT<NUM> and OUTPUT<NUM> are generated on source lines SL<NUM> and SL<NUM>.

<FIG> depicts neuron VMM array <NUM>, which is particularly suited for memory cells <NUM> as shown in <FIG>, memory cells <NUM> as shown in <FIG>, and memory cells <NUM> as shown in <FIG>, and is utilized as the synapses and parts of neurons between an input layer and the next layer. In this example, the inputs INPUT<NUM>,. , INPUTM are received on word lines WL<NUM>,. , WLM, and the outputs OUTPUT<NUM>,. , OUTPUTN are generated on bit lines BL<NUM>,. , BLN, respectively.

<FIG> depicts neuron VMM array <NUM>, which is particularly suited for memory cells <NUM> as shown in <FIG>, memory cells <NUM> as shown in <FIG>, and memory cells <NUM> as shown in <FIG>, and is utilized as the synapses and parts of neurons between an input layer and the next layer. In this example, the inputs INPUT<NUM>,. , INPUTM are received on control gate lines CG<NUM>,. Outputs OUTPUT<NUM>,. , OUTPUTN are generated on vertical source lines SL<NUM>,. , SLN, respectively, where each source line SLi is coupled to the source lines of all memory cells in column i.

<FIG> depicts neuron VMM array <NUM>, which is particularly suited for memory cells <NUM> as shown in <FIG>, memory cells <NUM> as shown in <FIG>, and memory cells <NUM> as shown in <FIG>, and is utilized as the synapses and parts of neurons between an input layer and the next layer. In this example, the inputs INPUT<NUM>,. , INPUTM are received on control gate lines CG<NUM>,. Outputs OUTPUT<NUM>,. , OUTPUTN are generated on vertical bit lines BL<NUM>,. , BLN, respectively, where each bit line BLi is coupled to the bit lines of all memory cells in column i.

The prior art includes a concept known as long short-term memory (LSTM). LSTM units often are used in neural networks. LSTM allows a neural network to remember information over predetermined arbitrary time intervals and to use that information in subsequent operations. A conventional LSTM unit comprises a cell, an input gate, an output gate, and a forget gate. The three gates regulate the flow of information into and out of the cell and the time interval that the information is remembered in the LSTM. VMMs are particularly useful in LSTM units.

<FIG> depicts an exemplary LSTM <NUM>. LSTM <NUM> in this example comprises cells <NUM>, <NUM>, <NUM>, and <NUM>. Cell <NUM> receives input vector x<NUM> and generates output vector h<NUM> and cell state vector c<NUM>. Cell <NUM> receives input vector x<NUM>, the output vector (hidden state) h<NUM> from cell <NUM>, and cell state c<NUM> from cell <NUM> and generates output vector hi and cell state vector c<NUM>. Cell <NUM> receives input vector x<NUM>, the output vector (hidden state) h<NUM> from cell <NUM>, and cell state c<NUM> from cell <NUM> and generates output vector h<NUM> and cell state vector c<NUM>. Cell <NUM> receives input vector x<NUM>, the output vector (hidden state) h<NUM> from cell <NUM>, and cell state c<NUM> from cell <NUM> and generates output vector h<NUM>. Additional cells can be used, and an LSTM with four cells is merely an example.

<FIG> depicts an exemplary implementation of an LSTM cell <NUM>, which can be used for cells <NUM>, <NUM>, <NUM>, and <NUM> in <FIG>. LSTM cell <NUM> receives input vector x(t), cell state vector c(t-<NUM>) from a preceding cell, and output vector h(t-<NUM>) from a preceding cell, and generates cell state vector c(t) and output vector h(t).

LSTM cell <NUM> comprises sigmoid function devices <NUM>, <NUM>, and <NUM>, each of which applies a number between <NUM> and <NUM> to control how much of each component in the input vector is allowed through to the output vector. LSTM cell <NUM> also comprises tanh devices <NUM> and <NUM> to apply a hyperbolic tangent function to an input vector, multiplier devices <NUM>, <NUM>, and <NUM> to multiply two vectors together, and addition device <NUM> to add two vectors together. Output vector h(t) can be provided to the next LSTM cell in the system, or it can be accessed for other purposes.

<FIG> depicts an LSTM cell <NUM>, which is an example of an implementation of LSTM cell <NUM>. For the reader's convenience, the same numbering from LSTM cell <NUM> is used in LSTM cell <NUM>. Sigmoid function devices <NUM>, <NUM>, and <NUM> and tanh device <NUM> each comprise multiple VMM arrays <NUM> and activation function blocks <NUM>. Thus, it can be seen that VMM arrays are particular useful in LSTM cells used in certain neural network systems. The multiplier devices <NUM>, <NUM>, and <NUM> and the addition device <NUM> are implemented in a digital manner or in an analog manner. The activation function blocks <NUM> can be implemented in a digital manner or in an analog manner.

An alternative to LSTM cell <NUM> (and another example of an implementation of LSTM cell <NUM>) is shown in <FIG>. In <FIG>, sigmoid function devices <NUM>, <NUM>, and <NUM> and tanh device <NUM> share the same physical hardware (VMM arrays <NUM> and activation function block <NUM>) in a time-multiplexed fashion. LSTM cell <NUM> also comprises multiplier device <NUM> to multiply two vectors together, addition device <NUM> to add two vectors together, tanh device <NUM> (which comprises activation function block <NUM>), register <NUM> to store the value i(t) when i(t) is output from sigmoid function block <NUM>, register <NUM> to store the value f(t) * c(t-<NUM>) when that value is output from multiplier device <NUM> through multiplexor <NUM>, register <NUM> to store the value i(t) * u(t) when that value is output from multiplier device <NUM> through multiplexor <NUM>, and register <NUM> to store the value o(t) * c~(t) when that value is output from multiplier device <NUM> through multiplexor <NUM>, and multiplexor <NUM>.

Whereas LSTM cell <NUM> contains multiple sets of VMM arrays <NUM> and respective activation function blocks <NUM>, LSTM cell <NUM> contains only one set of VMM arrays <NUM> and activation function block <NUM>, which are used to represent multiple layers in the embodiment of LSTM cell <NUM>. LSTM cell <NUM> will require less space than LSTM <NUM>, as LSTM cell <NUM> will require <NUM>/<NUM> as much space for VMMs and activation function blocks compared to LSTM cell <NUM>.

It can be further appreciated that LSTM units will typically comprise multiple VMM arrays, each of which requires functionality provided by certain circuit blocks outside of the VMM arrays, such as a summer and activation function block and high voltage generation blocks. Providing separate circuit blocks for each VMM array would require a significant amount of space within the semiconductor device and would be somewhat inefficient. The embodiments described below therefore reduce the circuitry required outside of the VMM arrays themselves.

An analog VMM implementation can be utilized for a GRU (gated recurrent unit) system. GRUs are a gating mechanism in recurrent neural networks. GRUs are similar to LSTMs, except that GRU cells generally contain fewer components than an LSTM cell.

<FIG> depicts an exemplary GRU <NUM>. GRU <NUM> in this example comprises cells <NUM>, <NUM>, <NUM>, and <NUM>. Cell <NUM> receives input vector x<NUM> and generates output vector h<NUM>. Cell <NUM> receives input vector x<NUM>, the output vector h<NUM> from cell <NUM> and generates output vector h<NUM>. Cell <NUM> receives input vector x<NUM> and the output vector (hidden state) h<NUM> from cell <NUM> and generates output vector h<NUM>. Cell <NUM> receives input vector x<NUM> and the output vector (hidden state) h<NUM> from cell <NUM> and generates output vector h<NUM>. Additional cells can be used, and an GRU with four cells is merely an example.

<FIG> depicts an exemplary implementation of a GRU cell <NUM>, which can be used for cells <NUM>, <NUM>, <NUM>, and <NUM> of <FIG>. GRU cell <NUM> receives input vector x(t) and output vector h(t-<NUM>) from a preceding GRU cell and generates output vector h(t). GRU cell <NUM> comprises sigmoid function devices <NUM> and <NUM>, each of which applies a number between <NUM> and <NUM> to components from output vector h(t-<NUM>) and input vector x(t). GRU cell <NUM> also comprises a tanh device <NUM> to apply a hyperbolic tangent function to an input vector, a plurality of multiplier devices <NUM>, <NUM>, and <NUM> to multiply two vectors together, an addition device <NUM> to add two vectors together, and a complementary device <NUM> to subtract an input from <NUM> to generate an output.

<FIG> depicts a GRU cell <NUM>, which is an example of an implementation of GRU cell <NUM>. For the reader's convenience, the same numbering from GRU cell <NUM> is used in GRU cell <NUM>. As can be seen in <FIG>, sigmoid function devices <NUM> and <NUM>, and tanh device <NUM> each comprise multiple VMM arrays <NUM> and activation function blocks <NUM>. Thus, it can be seen that VMM arrays are of particular use in GRU cells used in certain neural network systems. The multiplier devices <NUM>, <NUM>, <NUM>, the addition device <NUM>, and the complementary device <NUM> are implemented in a digital manner or in an analog manner. The activation function blocks <NUM> can be implemented in a digital manner or in an analog manner.

An alternative to GRU cell <NUM> (and another example of an implementation of GRU cell <NUM>) is shown in <FIG>. In <FIG>, GRU cell <NUM> utilizes VMM arrays <NUM> and activation function block <NUM>, which when configured as a sigmoid function applies a number between <NUM> and <NUM> to control how much of each component in the input vector is allowed through to the output vector. In <FIG>, sigmoid function devices <NUM> and <NUM> and tanh device <NUM> share the same physical hardware (VMM arrays <NUM> and activation function block <NUM>) in a time-multiplexed fashion. GRU cell <NUM> also comprises multiplier device <NUM> to multiply two vectors together, addition device <NUM> to add two vectors together, complementary device <NUM> to subtract an input from <NUM> to generate an output, multiplexor <NUM>, register <NUM> to hold the value h(t-<NUM>) * r(t) when that value is output from multiplier device <NUM> through multiplexor <NUM>, register <NUM> to hold the value h(t-<NUM>) *z(t) when that value is output from multiplier device <NUM> through multiplexor <NUM>, and register <NUM> to hold the value h^(t) * (<NUM>-z(t)) when that value is output from multiplier device <NUM> through multiplexor <NUM>.

Whereas GRU cell <NUM> contains multiple sets of VMM arrays <NUM> and activation function blocks <NUM>, GRU cell <NUM> contains only one set of VMM arrays <NUM> and activation function block <NUM>, which are used to represent multiple layers in the embodiment of GRU cell <NUM>. GRU cell <NUM> will require less space than GRU cell <NUM>, as GRU cell <NUM> will require <NUM>/<NUM> as much space for VMMs and activation function blocks compared to GRU cell <NUM>.

It can be further appreciated that GRU systems will typically comprise multiple VMM arrays, each of which requires functionality provided by certain circuit blocks outside of the VMM arrays, such as a summer and activation function block and high voltage generation blocks. Providing separate circuit blocks for each VMM array would require a significant amount of space within the semiconductor device and would be somewhat inefficient. The embodiments described below therefore reduce the circuitry required outside of the VMM arrays themselves.

The input to the VMM arrays can be an analog level, a binary level, a pulse, a time modulated pulse, or digital bits (in this case a DAC is needed to convert digital bits to appropriate input analog level) and the output can be an analog level, a binary level, a timing pulse, pulses, or digital bits (in this case an output ADC is needed to convert output analog level into digital bits).

In general, for each memory cell in a VMM array, each weight W can be implemented by a single memory cell or by a differential cell or by two blend memory cells (average of <NUM> cells). In the differential cell case, two memory cells are needed to implement a weight W as a differential weight (W = W+ - W-). In the two blend memory cells, two memory cells are needed to implement a weight W as an average of two cells.

<FIG> depicts VMM system <NUM>. In some embodiments, the weights, W, stored in a VMM array are stored as differential pairs, W+ (positive weight) and W- (negative weight), where W = (W+) - (W-). In VMM system <NUM>, half of the bit lines are designated as W+ lines, that is, bit lines connecting to memory cells that will store positive weights W+, and the other half of the bit lines are designated as W- lines, that is, bit lines connecting to memory cells implementing negative weights W-. The W- lines are interspersed among the W+ lines in an alternating fashion. The subtraction operation is performed by a summation circuit that receives current from a W+ line and a W- line, such as summation circuits <NUM> and <NUM>. The output of a W+ line and the output of a W- line are combined together to give effectively W = W+ - W- for each pair of (W+, W-) cells for all pairs of (W+, W-) lines. While the above has been described in relation to W- lines interspersed among the W+ lines in an alternating fashion, in other embodiments W+ lines and W- lines can be arbitrarily located anywhere in the array.

<FIG> depicts another embodiment. In VMM system <NUM>, positive weights W+ are implemented in first array <NUM> and negative weights W- are implemented in a second array <NUM>, second array <NUM> separate from the first array, and the resulting weights are appropriately combined together by summation circuits <NUM>.

<FIG> depicts VMM system <NUM>. the weights, W, stored in a VMM array are stored as differential pairs, W+ (positive weight) and W- (negative weight), where W = (W+) - (W-). VMM system <NUM> comprises array <NUM> and array <NUM>. Half of the bit lines in each of array <NUM> and <NUM> are designated as W+ lines, that is, bit lines connecting to memory cells that will store positive weights W+, and the other half of the bit lines in each of array <NUM> and <NUM> are designated as W- lines, that is, bit lines connecting to memory cells implementing negative weights W-. The W- lines are interspersed among the W+ lines in an alternating fashion. The subtraction operation is performed by a summation circuit that receives current from a W+ line and a W- line, such as summation circuits <NUM>, <NUM>, <NUM>, and <NUM>. The output of a W+ line and the output of a W- line from each array <NUM>, <NUM> are respectively combined together to give effectively W = W+ - W- for each pair of (W+, W-) cells for all pairs of (W+, W-) lines. In addition, the W values from each array <NUM> and <NUM> can be further combined through summation circuits <NUM> and <NUM>, such that each W value is the result of a W value from array <NUM> minus a W value from array <NUM>, meaning that the end result from summation circuits <NUM> and <NUM> is a differential value of two differential values.

Each non-volatile memory cells used in the analog neural memory system is to be erased and programmed to hold a very specific and precise amount of charge, i.e., the number of electrons, in the floating gate. For example, each floating gate must hold one of N different values, where N is the number of different weights that can be indicated by each cell. Examples of N include <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>.

Similarly, a read operation must be able to accurately discern between N different levels.

There is a need in VMM systems for improved output blocks that can quickly and accurately receive outputs from an array and discern the values represented by those outputs.

The present invention is set out in independent claim <NUM> and independent claim <NUM>. Preferred aspects are defined in dependent claims <NUM>-<NUM>, <NUM>-<NUM>.

<FIG> depicts a block diagram of VMM system <NUM>. VMM system <NUM> comprises VMM array <NUM>, row decoder <NUM>, high voltage decoder <NUM>, column decoder <NUM>, bit line drivers <NUM>, input circuit <NUM>, output circuit <NUM>, control logic <NUM>, and bias generator <NUM>. VMM system <NUM> further comprises high voltage generation block <NUM>, which comprises charge pump <NUM>, charge pump regulator <NUM>, and high voltage analog precision level generator <NUM>. VMM system <NUM> further comprises (program/erase, or weight tuning) algorithm controller <NUM>, analog circuitry <NUM>, control engine <NUM> (that may include special functions such as arithmetic functions, activation functions, embedded microcontroller logic, without limitation), and test control logic <NUM>. The systems and methods described below can be implemented in VMM system <NUM>.

Input circuit <NUM> may include circuits such as a DAC (digital to analog converter), DPC (digital to pulses converter, digital to time modulated pulse converter), AAC (analog to analog converter, such as a current to voltage converter, logarithmic converter ), PAC (pulse to analog level converter), or any other type of converters. The input circuit <NUM> may implement normalization, linear or non-linear up/down scaling functions, or arithmetic functions. The input circuit <NUM> may implement a temperature compensation function for input levels. The input circuit <NUM> may implement an activation function such as ReLU or sigmoid. The output circuit <NUM> may include circuits such as a ADC (analog to digital converter, to convert neuron analog output to digital bits), AAC (analog to analog converter, such as a current to voltage converter, logarithmic converter), APC (analog to pulse(s) converter, analog to time modulated pulse converter), or any other type of converters.

Output circuit <NUM> may implement an activation function such as rectified linear activation function (ReLU) or sigmoid. The output circuit <NUM> may implement statistic normalization, regularization, up/down scaling/gain functions, statistical rounding, or arithmetic functions (e.g., add, subtract, divide, multiply, shift, log) for neuron outputs. Output circuit <NUM> may implement a temperature compensation function for neuron outputs or array outputs (such as bitline output) so as to keep power consumption of the array approximately constant or to improve precision of the array (neuron) outputs such as by keeping the IV slope approximately the same.

<FIG> depicts output block <NUM> falling under the scope of protection of the present invention. Output block comprises current-to-voltage converters (ITV) <NUM>-<NUM> through <NUM>-i, where i is the number of bit line W+ and W- pairs received by output block <NUM>; multiplexor <NUM>; sample and hold circuits <NUM>-<NUM> through <NUM>-i, channel multiplexor <NUM>, and analog-to-digital converter (ADC) <NUM>. Output block <NUM> receives differential weight outputs W+ and W- from bit line pairs in the array, and ultimately generates a digital output, DOUTx, representing the output of one of the bit line pairs (e.g., W+ and W- lines) from the ADC <NUM>.

Current-to-voltage converters <NUM>-<NUM> through <NUM>-i each receive analog bit line current signals BLw+ and BLw- (which are bit line outputs generated in response to inputs and stored W+ and W-weights, respectively) and convert them into differential voltages ITVO+ and ITVO-.

ITVO+ and ITVO- are then received by multiplexor <NUM>, which time-multiplexes the outputs from current-to-voltage converters <NUM>-<NUM> through <NUM>-I to the S/H circuits <NUM>-<NUM> to <NUM>, where k is smaller than i.

S/H circuits <NUM>-<NUM> to <NUM>-k each samples its received differential voltages and holds them as a differential output.

Channel multiplexor <NUM> then receives a control signal to select one of the bit line W+ and W- channels, i.e., one of the bit line pairs, and outputs the differential voltages held by the respective sample and hold circuit <NUM> to ADC <NUM>, which converts the analog differential voltages that are output by the respective sample and hold circuit <NUM> into a set of digital bits, DOUTx. As shown, the S/H <NUM> arc shared across the multiple ITV circuits <NUM>, and the ADC <NUM> operates on multiple ITV circuits in a time-multiplexed manner. Each S/H <NUM> can be just a capacitor or a capacitor followed by a buffer (e.g., operational amplifier).

ADC <NUM> can be of a hybrid ADC architecture, meaning it has more than one ADC architecture to perform conversion. For example, if DOUTx is an <NUM>-bit output, ADC <NUM> can comprise an ADC sub-architecture to generate bits B7-B4 and another ADC sub-architecture to generate bits B3-B0 from the differential inputs ITVSH+ and ITVSH-. That is, ADC circuit <NUM> can include multiple ADC sub0architectures.

Optionally, an ADC sub-architecture can be shared among all channels while another ADC sub-architecture is not shared among all channels.

In another embodiment, channel mux <NUM> and ADC <NUM> can be removed, and the output instead can be analog differential voltages from a S/H <NUM>, which can be buffered by an operational amplifier. For example, the use of an analog voltage can be implemented in an all-analog neural network (i.e. one where a digital output or digital input is not needed for the neural memory array).

<FIG> depicts output block <NUM>. Output block comprises current-to-voltage converters (ITV) <NUM>-<NUM> through <NUM>-i, where i is the number of bit line W+ and W- pairs received by output block <NUM>; multiplexor <NUM>; differential to single ended converter Diff-to-S Converter <NUM>, sample and hold circuits <NUM>-<NUM> through <NUM>-k (where k is the same as or different than i), channel multiplexor <NUM>, and analog-to-digital converter (ADC) <NUM>. Diff-to-S converter <NUM> is used to convert the differential outputs from the ITV <NUM> signal provided by mux <NUM> into a singled-ended output. The singled-ended output is then input to the S/H <NUM>. mux <NUM>, and ADC <NUM>.

<FIG> depicts output block <NUM> falling under the scope of protection of the present invention. Output block comprises summation circuits <NUM>-<NUM> through <NUM>-i (such as a current mirror circuit), where i is the number of bit line BLw+ and BLw- pairs received by output block <NUM>; current-to-voltage converter circuits (ITV) <NUM>-<NUM> through <NUM>-i, multiplexor <NUM>; sample and hold circuits <NUM>-<NUM> through <NUM>-k (where k is the same as or different than i), channel multiplexor <NUM>, and ADC <NUM>. Output block <NUM> receives differential weight outputs BLw+ and BLw- from bit line pairs in the array, and ultimately generates a digital output from ADC <NUM>, DOUTx, representing the output of one of the bit line pairs at a time.

Current summation circuits <NUM>-<NUM> through <NUM>-i each receive current from a pair of bit lines and subtract the BLw- value from the BLw- value and output the result as a summation current.

Current-to-voltage converters <NUM>-<NUM> through <NUM>-i receive the output summation current and convert the respective summation current into differential voltages ITVO+ and ITVO-, which are then received by multiplexor <NUM> and selectively provided to sample-and-hold circuits <NUM>-<NUM> through <NUM>-k.

Each sample and hold circuit <NUM> receives differential voltages ITVOMX+ and ITVOMX-, samples the received differential voltages, and hold them as a differential voltage output, OSH+ and PSH-.

Channel multiplexor <NUM> receives a control signal to select one of the bit line pairs, i.e., channels, BLw+ and BLw- and outputs the voltage held by the respective sample and hold circuit <NUM> to ADC <NUM>, which converts the voltage into a set of digital bits as DOUTx.

<FIG> depicts current-to-voltage converter <NUM>. Current-to-voltage converter <NUM> comprises operational amplifiers <NUM> and <NUM> and variable resistors <NUM>, <NUM>, and <NUM>, configured as shown. Current-to-voltage converter <NUM> receives differential output currents BLw+ from a W+ bit line and BLw- from a W- bit line, shown as variable current sources, and generates a single-ended output, Vout. The output voltage Vout is = (BLw+ - BLw-) * R, with resistors <NUM>, <NUM> and <NUM> each having value equal to R. The variable resistors in <FIG> can be used for scaling the output.

<FIG> depicts current-to-voltage converter <NUM>. Current-to-voltage converter <NUM> comprises operational amplifiers <NUM>, <NUM>, and <NUM> and variable resistors <NUM>, <NUM>, <NUM>, and <NUM>, configured as shown. Current-to-voltage converter <NUM> receives an output current BLw+ from a W+ bit line, shown as a variable current source, and generates output Vout+ for that line and receives an output current Blw- from a W- bit line, shown as a variable current source. and generates output Vout- for that line. Thus, unlike in output block <NUM>, output block <NUM> generates differential voltages, rather than a single-ended output, respectively representing differential values BLw+ and BLw-. The output voltage Vout+ = Iw+*R and Vout-= -Rw-*R, with resistors <NUM>, <NUM>, <NUM> and <NUM> each having value equal to R. The variable resistors in <FIG> can be used for scaling the outputs.

Optionally, differential output voltages Vout+ and Vout- can be input to ADC <NUM>, which converts them into a set of digital output bits, Doutx.

<FIG> depicts current-to-voltage converter <NUM>. Current-to-voltage converter <NUM> comprises operational amplifiers <NUM> and <NUM>; variable capacitors <NUM>, <NUM>, and <NUM>; and controlled switches <NUM> and <NUM>, configured as shown. Current-to-voltage converter <NUM> receives differential output currents BLw+ from a W+ bit line, shown as a variable current source, and BLw- from a W- bit line, shown as a variable current source, and generates single-ended output Vout. The output voltage Vout is = (Iw+ - Iw-) * t_integration / C , with capacitors <NUM>, <NUM> and <NUM> each having a capacitance value equal to C. A control circuit (not shown) controls the opening and closing of switches <NUM>, <NUM> to provide the integration time t_integration.

<FIG> depicts current-to-voltage converter <NUM>. Current-to-voltage converter <NUM> comprises operational amplifiers <NUM>, <NUM>, and <NUM>; variable capacitors <NUM>, <NUM>, <NUM>, and <NUM>; and switches <NUM>, <NUM>, and <NUM>. Current-to-voltage converter <NUM> receives an output current BLw+ from a W+ bit line, shown as a variable current source, and generates output Vout+ for that line and receives an output current BLw- from a W- bit line, shown as a variable current source, and generates output Vout- for that line. Thus, unlike in output block <NUM>, output block <NUM> generates two voltages representing respective differential values BLw+ and BLw-. The output voltage Vout+ = BLw+ * t_integration/C and Vout- = BLw- * t_integration/C, with capacitors <NUM>, <NUM>, <NUM> and <NUM> each having a capacitance value equal to C. A control circuit (not shown) controls the opening and closing of switches <NUM>, <NUM> and <NUM> to provide the integration time t_integration.

<FIG> depicts current-to-voltage converter <NUM>. Current-to-voltage converter <NUM> comprises operational amplifier <NUM>; variable integrating resistors <NUM> and <NUM>; controlled switches <NUM>, <NUM>, <NUM>, and <NUM>; and sample and hold capacitors <NUM> and <NUM>, configured as shown. Current-to-voltage converter <NUM> receives differential current BLw+ from a W+ bit line and BLw- from a W- bit line and outputs voltages Vout+ and Vout-, respectively. The output voltage Vout+ = (BLw+) * R and Vout- = (BLw-) * R, , with resistors <NUM> and <NUM> each having value equal to R. Capacitors <NUM> and <NUM> each serves as holding S/H capacitor to hold the output voltage once the resistors <NUM> and <NUM> and the input current are shut off. A control circuit (not shown) controls the opening and closing of switches <NUM>, <NUM>, <NUM> and <NUM> to provide an integration time.

<FIG> depicts a differential voltages to singed ended voltage converter (Diff-to-S) <NUM>. The Diff-to-S converter <NUM> comprises operational amplifier <NUM>; and variable integrating resistors <NUM> and <NUM>. The output voltage Vout - (Vin1 - Vin2) * (R_3852/R_3953). This is, for example, used as block <NUM> in <FIG>.

<FIG> depicts output block <NUM>, which is a hybrid output conversion block. Output block <NUM> comprises multiple sub-architectures such as SAR and serial ADC sub-architectures as shown. Output block <NUM> receives differential signals Iw+ and Iw-. Successive approximation register analog-to-digital converter SAR <NUM> converts differential signals Iw+ and Iw- into higher order digital bits, and serial block ADC <NUM> then converts the signal that remans after the higher bit conversion into the lower order bits and outputs all the output digital bits together. In one example, SAR ADC <NUM> converts a portion of the received differential voltages into MSB bits B7-B4 and serial ADC <NUM> converts a portion of the received differential voltages into the LSB bit B3-B0 for <NUM>-bit ADC conversion.

<FIG> depicts output block <NUM>. Output block <NUM> comprises multiple sub-architectures, such as algorithmic ADC and serial ADC sub-architectures as shown. Output block <NUM> receives differential signals Iw+ and Iw-. Algorithmic analog-to-digital converter <NUM> converts differential signals Iw+ and Iw- into high order digital bits, and serial ADC block <NUM> then converts the signal that remains after the higher bit conversion into the lower order bits and outputs all the output digital bits together In one example, Algorithmic ADC <NUM> converts a portion of received differential voltages into MSB bits B7-B4 and the serial ADC <NUM> converts a portion of the received differential voltages into the LSB bit B3-B0 for <NUM>-bit ADC conversion.

<FIG> depicts output block <NUM>. Output block <NUM> receives differential signals Iw+ and Iw-. Output block <NUM> comprises hybrid analog-to-digital converter, which converts differential signals Iw+ and Iw- into digital bits by combining different conversion schemes (such as those shown in <FIG>) into one block.

<FIG> depicts configurable serial analog-to-digital converter <NUM>. It includes integrator <NUM> which integrates the neuron output current INEU, shown as a variable current source, into the integrating capacitor <NUM> (Cint). Integrator <NUM> comprises a differential amplifier <NUM>, controlled switches <NUM> and <NUM>, and a control circuit (not shown) which controls the opening and closing of switches <NUM> and <NUM> to provide an integration time.

In one embodiment, VRAMP <NUM> is provided to the inverting input of comparator <NUM>. The digital output (count value) <NUM> is produced by ramping VRAMP <NUM> until the output of comparator <NUM>, shown as EC <NUM>, switches polarity, with counter <NUM> counting clock pulses from the beginning of the ramp of VRAMP <NUM> and stopping when the output of comparator <NUM> switches polarity, responsive to AND gate <NUM> preventing the passage of clock <NUM> as pulse series <NUM> from reaching counter <NUM>.

In another embodiment, VREF <NUM> is provided to the inverting input of comparator <NUM>. VC <NUM> is ramped down by ramp current <NUM> (IREF) until VOUT <NUM> reaches VREF <NUM>, at which point the output of comparator <NUM>, EC <NUM>, switches polarity which disables the count of counter <NUM>. Thus, counter <NUM> is enabled with the closing of switch S2, (which is after the opening of S2, and disabled when output of comparator <NUM>, EC <NUM>, switches polarity). S3 is used to initialize (equalize) at the beginning of the operation. The (n-bit) ADC <NUM> is configurable to have a lower precision (fewer than n bits) or a higher precision (more than n bits), depending on the target application. The configurability of precision is done by configuring the capacitance of capacitor <NUM>, the current <NUM> (IREF), the ramping rate of VRAMP <NUM>, or the clocking frequency of clock <NUM>, without limitation.

In another embodiment, the ADC circuit of a VMM array is configured to have a precision lower than n bits and the ADC circuits of another VMM array is configured to have high a precision greater than bits.

In another embodiment, one instance of serial ADC circuit <NUM> of one neuron (array output) circuit is configured to combine with another instance of serial ADC circuit <NUM> for an adjacent neuron circuit to produce an ADC circuit with higher than n-bit precision, such as by combining the integrating capacitor <NUM> of the two instances of serial ADC circuits <NUM>.

<FIG> depicts a configurable SAR (successive approximation register) analog-to-digital converter <NUM> used for neuron output circuit (array output circuit). This circuit is a successive approximation converter based on charge redistribution using binary capacitors. It includes a binary CDAC (capacitor Digital to Analog Converter) <NUM>, comparator <NUM>, and SAR logic and registers <NUM>. As shown GndV <NUM> is a low voltage reference level, for example ground level. SAR logic and register <NUM> provides digital outputs <NUM>. Other non-binary capacitor structures can be implemented with weighted reference voltages or correction with the outputs.

<FIG> depicts a pipelined SAR ADC circuit <NUM> that can be used to combine with the next SAR ADC to increase the number of bits in a pipelined fashion. SAR ADC circuit <NUM> comprises binary CDAC <NUM>, comparator <NUM> (operates as a op amp or comparator), op-amp/comparator <NUM>, SAR logic and registers <NUM>. As shown GndV <NUM> is a low voltage reference level, for example ground level. SAR logic and register <NUM> provides digital outputs <NUM>. Vin is in the input voltage, VREF is a reference voltage, and GndV is a low voltage, such as a ground voltage. Vresidue is generated by capacitor <NUM> and is provided as an input to the next stage of an SAR ADC conversion sequence.

<FIG> depicts hybrid SAR +serial ADC circuit <NUM> that can be used to increase the number of bits in a hybrid fashion. SAR ADC circuit <NUM> comprises binary CDAC <NUM>, comparator <NUM>, and SAR logic and registers <NUM>. As shown, GndV is a low voltage reference level, for example ground level during the SAR ADC operation. SAR logic and registers <NUM> provides digital outputs. Vin is the input voltage. The VREFRAMP is used as a reference ramping voltage during the serial ADC operation with appropriate control circuit and signal muxing (not shown).

Other hybrid ADC architectures that can be used include SAR ADC plus sigma delta ADC, Flash ADC plus serial ADC, Pipelined ADC plus serial ADC, Serial ADC plus SAR ADC, and other architectures.

<FIG> depicts hybrid differential SAR + serial ADC circuit <NUM> that can be used to increase the number of bits in a hybrid fashion.

<FIG> depicts Algorithmic ADC output block <NUM>. Output block <NUM> comprises sample-and-hold circuit <NUM>, <NUM>-bitanalog-to-digital converter <NUM>, <NUM>-bit digital-to-analog converter <NUM>, summer <NUM>, operational amplifier <NUM>, and controlled switches <NUM> and <NUM>, configured as shown. Operational amplifier <NUM> is shown configured to provide a gain of <NUM>.

<FIG> depicts tracking voltage reference generator <NUM> that is used to generate a reference voltage that can be used by output circuits described herein and components of such output circuits, such as in <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, and <FIG>.

Tracking voltage reference generator <NUM> comprises bias current <NUM> and variable resistor <NUM> and generates an output VREFx <NUM> = i * R, where i is the current from bias current <NUM> and R is the resistance of variable resistor <NUM>.

Claim 1:
An output circuit (<NUM>) configured to generate a digital output (DOUTx) from one or more arrays of non-volatile memory cells, comprising:
a plurality of current-to-voltage converters (<NUM>), each of the plurality of current-to-voltage converters configured to receive currents from a respective bitline pair (BLw+, BLw-), each bitline pair providing differential weight , W+, W-, outputs from a respective bitline coupled to one or more non-volatile memory cells of the one or more arrays storing W+ values and from a respective bitline coupled to one or more non-volatile memory cells of the one or more arrays storing W- values, and to convert said values into differential voltage outputs (ITVO+, ITVO-);
a multiplexor (<NUM>) configured to receive respective differential voltage outputs from the plurality of current-to-voltage converters and to time-multiplex respective differential voltage outputs;
a plurality of sample-and-hold circuits (<NUM>) configured to be shared across the plurality of current-to-voltage converters through the multiplexer, the number (k) of sample-and-hold circuits thus being smaller than the number (i) of current-to-voltage converters, and configured to generate respective held differentail voltage outputs (OSH+, OSH-);
a channel multiplexor (<NUM>) configured to receive the respective held differential voltage outputs (OSH+, OSH-) from the plurality of sample-and-hold circuits and to provide as an output a selected pair (ITVSH+, ITVSH-) of the
held differential voltage outputs in response to a control signal received by the channel multiplexor; and
an analog-to-digital converter (<NUM>) configured to convert the selected pair of differential voltage outputs (ITVSH+, ITVSH-) received from the channel multiplexer into the digital output representing the output of one of the bit line pairs at a time.