Patent Description:
<FIG> illustrates an artificial neural network, where the circles represent the inputs or layers of neurons. The connections (called synapses) are represented by arrows, and have numeric weights that can be tuned based on experience. This makes neural networks adaptive to inputs and capable of learning. Typically, neural networks include a layer of multiple inputs. There are typically one or more intermediate layers of neurons, and an output layer of neurons that provide the output of the neural network. The neurons at each level individually or collectively make a decision based on the received data from the synapses.

One of the major challenges in the development of artificial neural networks for high-performance information processing is a lack of adequate hardware technology. Indeed, practical neural networks rely on a very large number of synapses, enabling high connectivity between neurons, i.e. a very high computational parallelism. In principle, such complexity can be achieved with digital supercomputers or specialized graphics processing unit clusters. However, in addition to high cost, these approaches also suffer from mediocre energy efficiency as compared to biological networks, which consume much less energy primarily because they perform low-precision analog computation. CMOS analog circuits have been used for artificial neural networks, but most CMOS-implemented synapses have been too bulky given the high number of neurons and synapses.

Applicant previously disclosed an artificial (analog) neural network that utilizes one or more non-volatile memory arrays as the synapses in <CIT>. The non-volatile memory arrays operate as an analog neuromorphic memory. The neural network device includes a first plurality of synapses configured to receive a first plurality of inputs and to generate therefrom a first plurality of outputs, and a first plurality of neurons configured to receive the first plurality of outputs. The first plurality of synapses includes a plurality of memory cells, wherein each of the memory cells includes spaced apart source and drain regions formed in a semiconductor substrate with a channel region extending there between, a floating gate disposed over and insulated from a first portion of the channel region and a non-floating gate disposed over and insulated from a second portion of the channel region. Each of the plurality of memory cells is configured to store a weight value corresponding to a number of electrons on the floating gate. The plurality of memory cells is configured to multiply the first plurality of inputs by the stored weight values to generate the first plurality of outputs.

Each non-volatile memory cells used in the analog neuromorphic memory system must be erased and programmed to hold a very specific and precise amount of charge, i.e., the number of electrons, in the floating gate. For example, each floating gate must hold one of N different values, where N is the number of different weights that can be indicated by each cell. Examples of N include <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>.

One challenge of implementing analog neuro memory systems is that various layers containing arrays of different sizes are required. Arrays of different sizes have different needs for supporting circuitry outside of the array. Providing customized hardware for each system can become costly and time-consuming.

What is needed is a configurable architecture for an analog neuro memory system that can provide various layers of vector-by-matrix multiplication arrays of various sizes, along with supporting circuitry of the right size, such that the same hardware can be used in analog neural memory systems with different requirements. Document "<NPL>, refers to neuromorphic networks using highly optimized, nanoscale, non-volatile floating gate memory cells which are used in embedded NOR flash memories.

<CIT> discloses an artificial neural network device that utilizes one or more non-volatile memory arrays as the synapses. The synapses are configured to receive inputs and to generate therefrom outputs. Neurons are configured to receive the outputs. The synapses include a plurality of memory cells, wherein each of the memory cells includes spaced apart source and drain regions formed in a semiconductor substrate with a channel region extending there between, a floating gate disposed over and insulated from a first portion of the channel region and a non-floating gate disposed over and insulated from a second portion of the channel region. Each of the plurality of memory cells is configured to store a weight value corresponding to a number of electrons on the floating gate. The plurality of memory cells are configured to multiply the inputs by the stored weight values to generate the outputs.

Document "<NPL>), refers to long short-term memories (LSTM) which can be implemented with a memristor crossbar, having a small circuit footprint to store a large number of parameters and in-memory computing capability that circumvents the 'von Neumann bottleneck'.

Numerous embodiments are disclosed for a configurable hardware system for use in an analog neural memory system for a deep learning neural network. The components within the configurable hardware system that are configurable can include vector-by-matrix multiplication arrays, summer circuits, activation circuits, inputs, reference devices, neurons, and testing circuits. These devices can be configured to provide various layers or vector-by-matrix multiplication arrays of various sizes, such that the same hardware can be used in analog neural memory systems with different requirements.

Digital non-volatile memories are well known. For example, <CIT> ("the '<NUM> patent"), which is incorporated herein by reference, discloses an array of split gate non-volatile memory cells, which are a type of flash memory cells, and is incorporated herein by reference for all purposes. Such a memory cell <NUM> is shown in <FIG>. Each memory cell <NUM> includes source region <NUM> and drain region <NUM> formed in a semiconductor substrate <NUM>, with a channel region <NUM> there between. A floating gate <NUM> is formed over and insulated from (and controls the conductivity of) a first portion of the channel region <NUM>, and over a portion of the source region <NUM>. A word line terminal <NUM> (which is typically coupled to a word line) has a first portion that is disposed over and insulated from (and controls the conductivity of) a second portion of the channel region <NUM>, and a second portion that extends up and over the floating gate <NUM>. The floating gate <NUM> and word line terminal <NUM> are insulated from the substrate <NUM> by a gate oxide. Bitline <NUM> is coupled to drain region <NUM>.

Memory cell <NUM> is programmed (where electrons are placed on the floating gate) by placing a positive voltage on the word line terminal <NUM>, and a positive voltage on the source region <NUM>. Electron current will flow from the source region <NUM> towards the drain region <NUM>. The electrons will accelerate and become heated when they reach the gap between the word line terminal <NUM> and the floating gate <NUM>. Some of the heated electrons will be injected through the gate oxide onto the floating gate <NUM> due to the attractive electrostatic force from the floating gate <NUM>.

Other split gate memory cell configurations, which are other types of flash memory cells, are known. For example, <FIG> depicts a four-gate memory cell <NUM> comprising source region <NUM>, drain region <NUM>, floating gate <NUM> over a first portion of channel region <NUM>, a select gate <NUM> (typically coupled to a word line, WL) over a second portion of the channel region <NUM>, a control gate <NUM> over the floating gate <NUM>, and an erase gate <NUM> over the source region <NUM>. This configuration is described in <CIT>, which is incorporated herein by reference for all purposes).

<FIG> depicts a three-gate memory cell <NUM>, which is another type of flash memory cell. Memory cell <NUM> is identical to the memory cell <NUM> of <FIG> except that memory cell <NUM> does not have a separate control gate. The erase operation (whereby erasing occurs through use of the erase gate) and read operation are similar to that of the <FIG> except there is no control gate bias applied. The programming operation also is done without the control gate bias, and as a result, a higher voltage must be applied on the source line during a program operation to compensate for a lack of control gate bias.

<FIG> depicts stacked gate memory cell <NUM>, which is another type of flash memory cell. Memory cell <NUM> is similar to memory cell <NUM> of <FIG>, except that floating gate <NUM> extends over the entire channel region <NUM>, and control gate <NUM> (which here will be coupled to a word line) extends over floating gate <NUM>, separated by an insulating layer (not shown). The erase, programming, and read operations operate in a similar manner to that described previously for memory cell <NUM>.

S0 is the input layer, which for this example is a 32x32 pixel RGB image with <NUM> bit precision (i.e. three 32x32 pixel arrays, one for each color R, G and B, each pixel being <NUM> bit precision). The synapses CB1 going from input layer S0 to layer C1 apply different sets of weights in some instances and shared weights in other instances, and scan the input image with 3x3 pixel overlapping filters (kernel), shifting the filter by <NUM> pixel (or more than <NUM> pixel as dictated by the model). Specifically, values for <NUM> pixels in a 3x3 portion of the image (i.e., referred to as a filter or kernel) are provided to the synapses CB1, where these <NUM> input values are multiplied by the appropriate weights and, after summing the outputs of that multiplication, a single output value is determined and provided by a first synapse of CB1 for generating a pixel of one of the layers of feature map C1. The 3x3 filter is then shifted one pixel to the right within input layer S0 (i.e., adding the column of three pixels on the right, and dropping the column of three pixels on the left), whereby the <NUM> pixel values in this newly positioned filter are provided to the synapses CB1, where they are multiplied by the same weights and a second single output value is determined by the associated synapse. This process is continued until the 3x3 filter scans across the entire 32x32 pixel image of input layer S0, for all three colors and for all bits (precision values). The process is then repeated using different sets of weights to generate a different feature map of C1, until all the features maps of layer C1 have been calculated.

<FIG> is a block diagram of an array that can be used for that purpose. Vector-by-matrix multiplication (VMM) array <NUM> includes non-volatile memory cells and is utilized as the synapses (such as CB1, CB2, CB3, and CB4 in <FIG>) between one layer and the next layer. Specifically, VMM array <NUM> includes an array of non-volatile memory cells <NUM>, erase gate and word line gate decoder <NUM>, control gate decoder <NUM>, bit line decoder <NUM> and source line decoder <NUM>, which decode the respective inputs for the non-volatile memory cell array <NUM>. Input to VMM array <NUM> can be from the erase gate and wordline gate decoder <NUM> or from the control gate decoder <NUM>. Source line decoder <NUM> in this example also decodes the output of the non-volatile memory cell array <NUM>. Alternatively, bit line decoder <NUM> can decode the output of the non-volatile memory cell array <NUM>.

Non-volatile memory cell array <NUM> serves two purposes. Second, the non-volatile memory cell array <NUM> effectively multiplies the inputs by the weights stored in the non-volatile memory cell array <NUM> and adds them up per output line (source line or bit line) to produce the output, which will be the input to the next layer or input to the final layer. By performing the multiplication and addition function, the non-volatile memory cell array <NUM> negates the need for separate multiplication and addition logic circuits and is also power efficient due to its in-situ memory computation.

The output of non-volatile memory cell array <NUM> is supplied to a differential summer (such as a summing op-amp or a summing current mirror) <NUM>, which sums up the outputs of the non-volatile memory cell array <NUM> to create a single value for that convolution. The differential summer <NUM> is arranged to perform summation of positive weight and negative weight.

The summed up output values of differential summer <NUM> are then supplied to an activation function circuit <NUM>, which rectifies the output. The activation function circuit <NUM> may provide sigmoid, tanh, or ReLU functions. The rectified output values of activation function circuit <NUM> become an element of a feature map as the next layer (e.g. C1 in <FIG>), and are then applied to the next synapse to produce the next feature map layer or final layer. Therefore, in this example, non-volatile memory cell array <NUM> constitutes a plurality of synapses (which receive their inputs from the prior layer of neurons or from an input layer such as an image database), and summing op-amp <NUM> and activation function circuit <NUM> constitute a plurality of neurons.

The input to VMM array <NUM> in <FIG> (WLx, EGx, CGx, and optionally BLx and SLx) can be analog level, binary level, or digital bits (in which case a DAC is provided to convert digital bits to appropriate input analog level) and the output can be analog level, binary level, or digital bits (in which case an output ADC is provided to convert output analog level into digital bits).

<FIG> is a block diagram depicting the usage of numerous layers of VMM arrays <NUM>, here labeled as VMM arrays 32a, 32b, 32c, 32d, and 32e. As shown in <FIG>, the input, denoted Inputx, is converted from digital to analog by a digital-to-analog converter <NUM>, and provided to input VMM array 32a. The input D/A conversion for the first layer could be done by using a function or a LUT (look up table) that maps the inputs Inputx to appropriate analog levels for the matrix multiplier of input VMM array 32a. The input conversion could also be done by an analog to analog (A/A) converter to convert an external analog input to a mapped analog input to the input VMM array 32a.

The output generated by input VMM array 32a is provided as an input to the next VMM array (hidden level <NUM>) 32b, which in turn generates an output that is provided as an input to the next VMM array (hidden level <NUM>) 32c, and so on. The various layers of VMM array <NUM> function as different layers of synapses and neurons of a convolutional neural network (CNN). Each VMM array 32a, 32b, 32c, 32d, and 32e can be a stand-alone, physical non-volatile memory array, or multiple VMM arrays could utilize different portions of the same physical non-volatile memory array, or multiple VMM arrays could utilize overlapping portions of the same physical non-volatile memory array. The example shown in <FIG> contains five layers (32a,32b,32c,32d,32e): one input layer (32a), two hidden layers (32b,32c), and two fully connected layers (32d,32e). One of ordinary skill in the art will appreciate that this is merely exemplary and that a system instead could comprise more than two hidden layers and more than two fully connected layers.

The non-volatile reference memory cells and the non-volatile memory cells described herein are biased in weak inversion: <MAT> where <MAT>.

For an I-to-V log converter using a memory cell (such as a reference memory cell or a peripheral memory cell) or a transistor to convert input current into an input voltage: <MAT> Here, wp is w of a reference or peripheral memory cell.

For a memory array used as a vector matrix multiplier VMM array, the output current is: <MAT> , namely <MAT> <MAT> Here, wa = w of each memory cell in the memory array.

Alternatively, the flash memory cells of VMM arrays described herein can be configured to operate in the linear region: <MAT> <MAT>.

A wordline or control gate or bitline or sourceline can be used as the input for the memory cell operated in the linear region for the input voltage.

For an I-to-V linear converter, a memory cell (such as a reference memory cell or a peripheral memory cell) or a transistor operating in the linear region can be used to linearly convert an input/output current into an input/output voltage.

Other embodiments for VMM array <NUM> of <FIG> are described in <CIT>, which is incorporated by reference herein. As described in that application. a sourceline or a bitline can be used as the neuron output (current summation output).

In effect, the first and second non-volatile reference memory cells are diode-connected through multiplexors <NUM> with current inputs flowing into them.

In effect, the first and second non-volatile reference memory cells are diode-connected through multiplexors <NUM> with current inputs flowing into them through BLR0, BLR1, BLR2, and BLR3.

VMM array <NUM> implements uni-directional tuning for non-volatile memory cells in memory array <NUM>. That is, each non-volatile memory cell is erased and then partially programmed until the desired charge on the floating gate is reached. This can be performed, for example, using the novel precision programming techniques described below. If too much charge is placed on the floating gate (such that the wrong value is stored in the cell), the cell must be erased and the sequence of partial programming operations must start over. As shown, two rows sharing the same erase gate (such as EG0 or EG1) need to be erased together (which is known as a page erase), and thereafter, each cell is partially programmed until the desired charge on the floating gate is reached.

The prior art includes a concept known as long short-term memory (LSTM). LSTM units often are used in neural networks. LSTM allows a neural network to remember information over predetermined arbitrary time intervals and to use that information in subsequent operations. A conventional LSTM unit comprises a cell, an input gate, an output gate, and a forget gate. The three gates regulate the flow of information into and out of the cell and the time interval that the information is remembered in the LSTM. VMMs are particularly useful in LSTM units.

<FIG> depicts an exemplary LSTM <NUM>. LSTM <NUM> in this example comprises cells <NUM>, <NUM>, <NUM>, and <NUM>. Cell <NUM> receives input vector x<NUM> and generates output vector h<NUM> and cell state vector c<NUM>. Cell <NUM> receives input vector x<NUM>, the output vector (hidden state) h<NUM> from cell <NUM>, and cell state c<NUM> from cell <NUM> and generates output vector h<NUM> and cell state vector c<NUM>. Cell <NUM> receives input vector x<NUM>, the output vector (hidden state) h<NUM> from cell <NUM>, and cell state c<NUM> from cell <NUM> and generates output vector h<NUM> and cell state vector c<NUM>. Cell <NUM> receives input vector x<NUM>, the output vector (hidden state) h<NUM> from cell <NUM>, and cell state c<NUM> from cell <NUM> and generates output vector h<NUM>. Additional cells can be used, and an LSTM with four cells is merely an example.

<FIG> depicts an exemplary implementation of an LSTM cell <NUM>, which can be used for cells <NUM>, <NUM>, <NUM>, and <NUM> in <FIG>. LSTM cell <NUM> receives input vector x(t), cell state vector c(t-<NUM>) from a preceding cell, and output vector h(t-<NUM>) from a preceding cell, and generates cell state vector c(t) and output vector h(t).

LSTM cell <NUM> comprises sigmoid function devices <NUM>, <NUM>, and <NUM>, each of which applies a number between <NUM> and <NUM> to control how much of each component in the input vector is allowed through to the output vector. LSTM cell <NUM> also comprises tanh devices <NUM> and <NUM> to apply a hyperbolic tangent function to an input vector, multiplier devices <NUM>, <NUM>, and <NUM> to multiply two vectors together, and addition device <NUM> to add two vectors together. Output vector h(t) can be provided to the next LSTM cell in the system, or it can be accessed for other purposes.

<FIG> depicts an LSTM cell <NUM>, which is an example of an implementation of LSTM cell <NUM>. For the reader's convenience, the same numbering from LSTM cell <NUM> is used in LSTM cell <NUM>. Sigmoid function devices <NUM>, <NUM>, and <NUM> and tanh device <NUM> each comprise multiple VMM arrays <NUM> and activation circuit blocks <NUM>. Thus, it can be seen that VMM arrays are particular useful in LSTM cells used in certain neural network systems.

An alternative to LSTM cell <NUM> (and another example of an implementation of LSTM cell <NUM>) is shown in <FIG>. In <FIG>, sigmoid function devices <NUM>, <NUM>, and <NUM> and tanh device <NUM> share the same physical hardware (VMM arrays <NUM> and activation function block <NUM>) in a time-multiplexed fashion. LSTM cell <NUM> also comprises multiplier device <NUM> to multiply two vectors together, addition device <NUM> to add two vectors together, tanh device <NUM> (which comprises activation circuit block <NUM>), register <NUM> to store the value i(t) when i(t) is output from sigmoid function block <NUM>, register <NUM> to store the value f(t) * c(t-<NUM>) when that value is output from multiplier device <NUM> through multiplexor <NUM>, register <NUM> to store the value i(t) * u(t) when that value is output from multiplier device <NUM> through multiplexor <NUM>, and register <NUM> to store the value o(t) * c~(t) when that value is output from multiplier device <NUM> through multiplexor <NUM>, and multiplexor <NUM>.

Whereas LSTM cell <NUM> contains multiple sets of VMM arrays <NUM> and respective activation function blocks <NUM>, LSTM cell <NUM> contains only one set of VMM arrays <NUM> and activation function block <NUM>, which are used to represent multiple layers in the embodiment of LSTM cell <NUM>. LSTM cell <NUM> will require less space than LSTM <NUM>, as LSTM cell <NUM> will require <NUM>/<NUM> as much space for VMMs and activation function blocks compared to LSTM cell <NUM>.

It can be further appreciated that LSTM units will typically comprise multiple VMM arrays, each of which requires functionality provided by certain circuit blocks outside of the VMM arrays, such as a summer and activation circuit block and high voltage generation blocks. Providing separate circuit blocks for each VMM array would require a significant amount of space within the semiconductor device and would be somewhat inefficient. The embodiments described below therefore attempt to minimize the circuitry required outside of the VMM arrays themselves.

An analog VMM implementation can be utilized for a GRU (gated recurrent unit) system. GRUs are a gating mechanism in recurrent neural networks. GRUs are similar to LSTMs, except that GRU cells generally contain fewer components than an LSTM cell.

<FIG> depicts an exemplary GRU <NUM>. GRU <NUM> in this example comprises cells <NUM>, <NUM>, <NUM>, and <NUM>. Cell <NUM> receives input vector x<NUM> and generates output vector h<NUM>. Cell <NUM> receives input vector x<NUM>, the output vector (hidden state) h<NUM> from cell <NUM> and generates output vector h<NUM>. Cell <NUM> receives input vector x<NUM> and the output vector (hidden state) h<NUM> from cell <NUM> and generates output vector h<NUM>. Cell <NUM> receives input vector x<NUM> and the output vector (hidden state) h<NUM> from cell <NUM> and generates output vector h<NUM>. Additional cells can be used, and an GRU with four cells is merely an example.

<FIG> depicts an exemplary implementation of a GRU cell <NUM>, which can be used for cells <NUM>, <NUM>, <NUM>, and <NUM> of <FIG>. GRU cell <NUM> receives input vector x(t) and output vector h(t-<NUM>) from a preceding GRU cell and generates output vector h(t). GRU cell <NUM> comprises sigmoid function devices <NUM> and <NUM>, each of which applies a number between <NUM> and <NUM> to components from output vector h(t-<NUM>) and input vector x(t). GRU cell <NUM> also comprises a tanh device <NUM> to apply a hyperbolic tangent function to an input vector, a plurality of multiplier devices <NUM>, <NUM>, and <NUM> to multiply two vectors together, an addition device <NUM> to add two vectors together, and a complementary device <NUM> to subtract an input from <NUM> to generate an output.

<FIG> depicts a GRU cell <NUM>, which is an example of an implementation of GRU cell <NUM>. For the reader's convenience, the same numbering from GRU cell <NUM> is used in GRU cell <NUM>. As can be seen in <FIG>, sigmoid function devices <NUM> and <NUM>, and tanh device <NUM> each comprise multiple VMM arrays <NUM> and activation function blocks <NUM>. Thus, it can be seen that VMM arrays are of particular use in GRU cells used in certain neural network systems.

An alternative to GRU cell <NUM> (and another example of an implementation of GRU cell <NUM>) is shown in <FIG>. In <FIG>, GRU cell <NUM> utilizes VMM arrays <NUM> and activation function block <NUM>, which when configured as a sigmoid function applies a number between <NUM> and <NUM> to control how much of each component in the input vector is allowed through to the output vector. In <FIG>, sigmoid function devices <NUM> and <NUM> and tanh device <NUM> share the same physical hardware (VMM arrays <NUM> and activation function block <NUM>) in a time-multiplexed fashion. GRU cell <NUM> also comprises multiplier device <NUM> to multiply two vectors together, addition device <NUM> to add two vectors together, complementary device <NUM> to subtract an input from <NUM> to generate an output, multiplexor <NUM>, register <NUM> to hold the value h(t-<NUM>) * r(t) when that value is output from multiplier device <NUM> through multiplexor <NUM>, register <NUM> to hold the value h(t-<NUM>) *z(t) when that value is output from multiplier device <NUM> through multiplexor <NUM>, and register <NUM> to hold the value h^(t) * (<NUM>-z(t)) when that value is output from multiplier device <NUM> through multiplexor <NUM>.

Whereas GRU cell <NUM> contains multiple sets of VMM arrays <NUM> and activation function blocks <NUM>, GRU cell <NUM> contains only one set of VMM arrays <NUM> and activation function block <NUM>, which are used to represent multiple layers in the embodiment of GRU cell <NUM>. GRU cell <NUM> will require less space than GRU cell <NUM>, as GRU cell <NUM> will require <NUM>/<NUM> as much space for VMMs and activation function blocks compared to GRU cell <NUM>.

It can be further appreciated that GRU systems will typically comprise multiple VMM arrays, each of which requires functionality provided by certain circuit blocks outside of the VMM arrays, such as a summer and activation circuit block and high voltage generation blocks. Providing separate circuit blocks for each VMM array would require a significant amount of space within the semiconductor device and would be somewhat inefficient. The embodiments described below therefore attempt to minimize the circuitry required outside of the VMM arrays themselves.

The input to the VMM arrays can be an analog level, a binary level, or digital bits (in this case a DAC is needed to convert digital bits to appropriate input analog level) and the output can be an analog level, a binary level, or digital bits (in this case an output ADC is needed to convert output analog level into digital bits).

For each memory cell in a VMM array, each weight w can be implemented by a single memory cell or by a differential cell or by two blend memory cells (average of <NUM> cells). In the differential cell case, two memory cells are needed to implement a weight w as a differential weight (w = w+ - w-). In the two blend memory cells, two memory cells are needed to implement a weight w as an average of two cells.

<FIG> depicts configurable flash analog neuromorphic memory system <NUM>. Configurable flash analog neuro memory system <NUM> comprises macro blocks 2201a, 2201b, 2201c, 2201d, 2201e, and 2201f; neuron output (such as summer circuit and a sample and hold S/H circuit) blocks 2202a, 2202b. 2202c, 2202d, 2202e, and 2202f; activation circuit blocks 2203a, 2203b, 2203c, 2203d, 2203e, and 2203f; horizontal multiplexors 2204a, 2204b, 2204c, and 2204d; vertical multiplexors 2205a, 2205b, and 2205c; and cross multiplexors 2206a and 2206b. Each of macro blocks 2201a, 2201b, 2201c, 2201d, 2201e, and 2201f is a VMM sub-system containing a VMM array.

In one embodiment, neuron output blocks 2202a, 2202b. 2202c, 2202d, 2202e, and 2202f each includes a buffer (e.g., op amp) low impedance output type circuit that can drive a long, configurable interconnect. In one embodiment, activation circuit blocks 2203a, 2203b, 2203c, 2203d, 2203e, and 2203f provide the summing, high impedance current outputs. Alternatively, neuron output blocks 2202a, 2202b. 2202c, 2202d, 2202e, and 2202f can include the activation circuits, in which case additional low impedance buffers will be needed to drive the outputs.

It is to be understood by one of ordinary skill in the art that activation circuit blocks 2203a, 2203b, 2203c, 2203d, 2203e, and 2203f are just one example of a type of input block, and that configurable flash analog neuro memory system <NUM> instead can be designed with other input blocks in place of activation circuit blocks 2203a, 2203b, 2203c, 2203d, 2203e, and 2203f, such that those blocks become input blocks 2203a, 2203b, 2203c, 2203d, 2203e, and 2203f.

In one embodiment, neuron output blocks 2202a, 2202b. 2202c, 2202d, 2202e, and 2202f each comprises analog-to-digital conversion block <NUM> that output digital bits instead of analog signals. Those digital bits are then routed to the desired location using configurable interconnects of <FIG>. In this embodiment, activation circuit blocks 2203a, 2203b, 2203c, 2203d, 2203e, and 2203f each comprises digital-to-analog conversion block <NUM> that receives digital bits from the interconnects of <FIG> and converts the digital bits into analog signals.

In instances where configurable system <NUM> is used to implement an LSTM or GRU, output blocks 2202a, 2202b. 2202c, 2202d, 2202e, and 2202f and/or input blocks 2203a, 2203b, 2203c, 2203d, 2203e, and 2203f may include multiplier block, addition block, subtraction (output = <NUM> - input) block as needed for LSTM/GRU architecture, and optionally may include analog sample-and-hold circuits (such as circuits <NUM> or <NUM> in <FIG>) or digital sample-and-hold circuits (e.g., a register or SRAM) as needed.

Configurability includes the width of neurons (number of outputs convolution layer, such as bitlines), the width of inputs (number of inputs per convolution layer, such as number of rows) by combining multiple macros and/or configuring each individual macros to have only parts of neuron output and/or input circuit active.

Within a VMM array, time multiplexing is used to enable multiple timed passes to maximize usage of the array. For example first N rows or N columns of an array can be enabled (sampled) at time t0 and its result is held in a t0 sample and hold S/H circuit, the next N rows or N columns can be enabled at time t1 and its result is held in a t1 sample and hold S/H circuit, and so on. And at final time tf, all previous S/H results is combined appropriately to give final output.

As can be appreciated, one requirement of an analog neuro memory system is the ability to collect outputs from one layer and provide them as inputs to another layer. This results in a complicated routing scheme where the outputs from one VMM array might need to be routed as inputs to another VMM array that is not necessarily immediately adjacent to it. In <FIG>, this routing function is provided by horizontal multiplexors 2204a, 2204b, 2204c, and 2204d; vertical multiplexors 2205a, 2205b, and 2205c; and cross multiplexors 2206a and 2206b. Using these multiplexors, the outputs from any of the macro blocks 2201a, 2201b, 2201c, 2201d, 2201e, and 2201f can be routed as inputs to any of the other macro blocks in 2201a, 2201b, 2201c, 2201d, 2201e, and 2201f. This functionality is critical to creating a configurable system.

Configurable flash analog neuro memory system <NUM> also comprises controller or control logic <NUM>. Controller or control logic <NUM> optionally is a microcontroller running software code to perform the configurations described herein (controller), or hardware logic for performing the configurations described herein (control logic), including activation of horizontal multiplexors 2204a, 2204b, 2204c, and 2204d; vertical multiplexors 2205a, 2205b, and 2205c; and cross multiplexors 2206a and 2206b to perform the needed routing functions at each cycle.

<FIG> depicts configurable flash analog neuro memory system <NUM>. Configurable flash analog neuro memory system <NUM> comprises macro blocks 2301a, 2301b, 2301c, 2301d, 2301e, and 2301f; neuron output blocks (such as summer circuit and a sample and hold S/H circuit) 2302a, 2302b. and 2302c; activation circuit blocks 2303a, 2303b, 2303c, 2303d, 2303e, and 2303f; horizontal multiplexors 2304a, 2304b, 2304c, and 2304d; vertical multiplexors 2305a, 2305b, 2305c, 2305d, 2305e, and 2305f; and cross multiplexors 2306a and 2306b. Each of macro blocks 2301a, 2301b, 2301c, 2301d, 2301e, and 2301f is a VMM sub-system containing a VMM array. Neuron output blocks 2302a, 2302b, and 2302c are configured to be shared across macros.

As can be seen, the systems of <FIG> and <FIG> are similar except that the system of <FIG> has shared configurable neuron output blocks (i.e., neuron output blocks 2302a, 2302b, and 2302c). In <FIG>, the routing function is provided by horizontal multiplexors 2304a, 2304b, 2304c, and 2304d, vertical multiplexors 2305a, 2305b, 2305c, 2305d, 2305d, and 2305f and cross multiplexors 2306a and 2306b. Using these multiplexors, the outputs from any of the macro blocks 2301a, 2301b, 2301c, 2301d, 2301e, and 2301f can be routed as inputs to some (but not all) of the other macro blocks in 2301a, 2301b, 2301c, 2301d, 2301e, and 2301f. This allows some configurability with a lesser space requirement than the system of <FIG> due to the lack of vertical multiplexors.

Neuron output blocks 2302a, 2302b, and 2302c may include current summer circuit blocks and/or activation circuit blocks. Neuron output block 2302a, for example, can be configured to connect to an output of the macro block 2301a or to an output of the macro block 2301d. Or the neuron output block 2302a, for example, can be configured to connect to part of an output of the macro block 2301a and part of an output of the macro block 2301d.

It is to be understood by one of ordinary skill in the art that activation circuit blocks 2303a, 2303b, 2303c, 2303d, 2303e, and 2303f are just one example of a type of input block, and that configurable flash analog neuro memory system <NUM> instead can be designed with other input blocks in place of activation circuit blocks 2303a, 2303b, 2303c, 2303d, 2303e, and 2303f, such that those blocks become input blocks 2303a, 2303b, 2303c, 2303d, 2303e, and 2303f.

In one embodiment, neuron output blocks 2302a, 2302b, and 2302c each comprises analog-to-digital conversion block <NUM> that output digital bits instead of analog signals. Those digital bits are then routed to the desired location using configurable interconnects of <FIG>. In this embodiment, activation circuit blocks 2303a, 2303b, 2303c, 2303d, 2303e, and 2303f each comprises digital-to-analog conversion block <NUM> that receives digital bits from the interconnects of <FIG> and converts the digital bits into analog signals.

In instances where configurable system <NUM> is used to implement an LSTM or GRU, output blocks 2302a, 2302b. 2302c, 2302d, 2302e, and 2302f and/or input blocks 2303a, 2303b, 2303c, 2303d, 2303e, and 2303f may include multiplier block, addition block, subtraction (output = <NUM> - input) block as needed for LSTM/GRU architecture, and optionally may include analog sample-and-hold circuits (such as circuits <NUM> or <NUM> in <FIG>) or digital sample-and-hold circuits (e.g., a register or SRAM) as needed.

Configurable flash analog neuro memory system <NUM> also comprises controller or control logic <NUM>. As in <FIG>, controller or control logic <NUM> optionally is a microcontroller running software code to perform the configurations described herein (controller), or hardware logic for performing the configurations described herein (control logic), including activation of horizontal multiplexors 2304a, 2304b, 2304c, and 2304d; vertical multiplexors 2305a, 2305b, 2305c, 2305d, 2305e, and 2305f; and cross multiplexors 2306a and 2306b to perform the needed routing functions at each cycle.

<FIG> depicts VMM system <NUM>. VMM system <NUM> comprises macro block <NUM> (which can be used to implement macro blocks 2201a, 2201b, 2201c, 2201d, 2201e, 2201f, 2301a, 2301b, 2301c, 2301d, 2301e, and 2301f in <FIG> and <FIG>) and activation function block <NUM> and summer block <NUM>.

VMM system <NUM> comprises VMM array <NUM>, low voltage row decoder <NUM>, high voltage row decoder <NUM>, and low voltage reference column decoder <NUM>. Low voltage row decoder <NUM> provides a bias voltage for read and program operations and provides a decoding signal for high voltage row decoder <NUM>. High voltage row decoder <NUM> provides a high voltage bias signal for program and erase operations.

VMM system <NUM> further comprises redundancy arrays <NUM> and <NUM>. Redundancy arrays <NUM> and <NUM> provides array redundancy for replacing a defective portion in array <NUM>. VMM system <NUM> further comprises NVR (non-volatile register, aka info sector) sector <NUM>, which are array sectors used to store user info, device ID, password, security key, trimbits, configuration bits, manufacturing info, etc. VMM system <NUM> further comprises reference sector <NUM> for providing reference cells to be used in a sense operation; predecoder <NUM> for decoding addresses for decoders <NUM>, <NUM>, and/or <NUM>; bit line multiplexor <NUM>; macro control logic <NUM>; and macro analog circuit block <NUM>, each of which performs functions at the VMM array level (as opposed to the system level comprising all VMM arrays).

<FIG> depicts examples of array configurability, which can be used in the embodiments of <FIG>. Configurable array <NUM> comprises an array of M rows by N columns. Configurable array <NUM> can be a flash memory cell array containing cells of the types shown in <FIG>. In the embodiments of <FIG>, each VMM array can be configured into one or more sub-arrays of different sizes that are smaller than configurable array <NUM>. For instance, configurable array can be divided into sub-array <NUM> of A rows by B columns, sub-array <NUM> of C rows by D columns, and sub-array <NUM> of E rows by F columns. This configuration can be implemented by controller or control logic <NUM>. Once each of the desired sub-arrays is created, controller or control logic <NUM> can configure the horizontal, vertical, and cross multiplexors of <FIG> and <FIG> to perform the appropriate routing from each sub-array to the appropriate location at the appropriate time. Only one sub-array in each configurable array will be accessed during any given cycle at time t (through array time multiplexing). Only one of the sub-arrays in configurable array <NUM> will be accessed during a single cycle. However, the sub-arrays can be accessed during different time cycles, which allows the same physical array to provide multiple sub-arrays for use in a time-multiplexed fashion.

Examples of embodiments of the circuit blocks shown in <FIG> will now be described.

<FIG> depicts neuron output summer block <NUM> (which can be used as neuron output summer blocks 2202a, 2202b, 2202c, 2202d, 2202e, and 2201f in <FIG>; neuron output summer blocks <NUM>, 2302b, 2302c, 2302d, 2302e, and 2302f in <FIG>; and neuron output summer block <NUM> in <FIG>. It can be seen that neuron output summer block <NUM> comprises a plurality of smaller summer blocks 2601a, 2601b,. 2601i, each of which can operate on a portion of a corresponding VMM array (such as a single column in the array). Controller or control logic <NUM> can activate the appropriate summer blocks 2601a, 2601b,. 2601i during each cycle as needed. The summer circuit can be implemented as an op amp based summer circuit or a current mirror circuit. The summer circuit may include an ADC circuit to convert analog into output digital bits.

<FIG> depicts adaptable neuron circuit <NUM> that comprises on an op amp that provides low impedance output, for summing multiple current signals and converting the summed current signal into a voltage signal, and which is an embodiment of each summer block within summer block 2601a,. , 2601i in <FIG>. Adaptable neuron circuit <NUM> receives current from a VMM, such as VMM array <NUM> (labeled I_NEU), which here is represented as current source <NUM>, which is provided to the inverting input of operational amplifier <NUM>. The non-inverting input of operational amplifier <NUM> is coupled to a voltage source (labeled VREF). The output (labeled VO) of operational amplifier <NUM> is coupled to NMOS R_NEU transistor <NUM>, which acts as a variable resistor of effective resistance R_NEU in response to the signal VCONTROL, which is applied to the gate of NMOS transistor <NUM>. The output voltage, Vo, is equal to I_NEU * R_NEU - VREF. The maximum value of I_NEU depends on the number of synapses and weight value contained in the VMM. R_NEU is a variable resistance and can be adapted to the VMM size it is coupled to. For instance, R_NEU, can be altered by changing IBIAS and/or VDREF and /or VREF in <FIG>. Further, the power of the summing operational amplifier <NUM> is adjusted in relation the value of the R_NEU transistor <NUM> to minimize power consumption. As the value of R_NEU transistor <NUM> increases, the bias (i.e., power) of the operational amplifier <NUM> is reduced via current bias IBIAS_OPA <NUM> and vice versa. Since the op amp based summer circuit can provide low impedance output, it is suitable to be configured to drive a long interconnect and heavier loading.

<FIG> depicts activation function circuit <NUM>. Activation function circuit <NUM> can be used for activation circuit blocks 2203a, 2203b, 2203c, 2203d, 2203e, and 2203f in <FIG> and activation circuit blocks 2303a, 2303b, 2303c, 2303d, 2303e, and 2303f in <FIG>, and activation block <NUM> in <FIG>.

Activation function circuit <NUM> converts an input voltage pair (Vin+ and Vin-) into a current (Iout_neu) using a tanh function, and which can be used with the VMM arrays described above. Activation function circuit <NUM> comprises PMOS transistors <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> and NMOS transistors <NUM>, <NUM>, <NUM>, and <NUM>, configured as shown. The transistors <NUM>, <NUM>, and <NUM> serve as cascoding transistors. The input NMOS pair <NUM> and <NUM> operates in sub-threshold region to realize the tanh function. The current I_neu_max is the maximum neuron current that can be received from the attached VMM (not shown).

<FIG> depicts operational amplifier <NUM> that can be used as operational amplifier 2701in <FIG>. Operational amplifier <NUM> comprises PMOS transistors <NUM>, <NUM>, and <NUM>, NMOS transistors <NUM>, <NUM>, <NUM>, and <NUM>, and NMOS transistor <NUM> that acts as a variable bias, in the configuration shown. The input terminals to operational amplifier <NUM> are labeled Vinn (applied to the gate of NMOS transistor <NUM>) and Vin- (applied to the gate of NMOS transistor <NUM>), and the output is VO.

<FIG> depicts high voltage generation block <NUM>, control logic block <NUM>, analog circuit block <NUM>, and test block <NUM>.

High voltage generation block <NUM> comprises charge pump <NUM>, charge pump regulator <NUM>, and high voltage operational amplifier <NUM>. The voltage of the output of charge pump regulator <NUM> can be controlled based on the signals sent to the gates of the NMOS transistors in charge pump regulator <NUM>. Control logic block <NUM> receives control logic inputs and generates control logic outputs. Analog circuit block <NUM> comprises current bias generator <NUM> for receiving a reference voltage, Vref, and generating a current that can be used to apply a bias signal, iBias, as used elsewhere. Analog circuit block <NUM> also comprises voltage generator <NUM> for receiving a set of trim bits, TRBIT_WL, and generating a voltage to apply to word lines during various operations. Test block <NUM> receives signals on a test pad, MONHV_PAD, and outputs various signals for a designer to monitor during testing.

<FIG> depicts program and sensing block <NUM>, which can be used during program and verify operations. Program and sensing block <NUM> comprises a plurality of individual program and sense circuit blocks 3101a, 3101b,. Controller or control logic <NUM> can activate the appropriate program and sense circuit blocks 3101a, 3101b,. 3101j during each cycle as needed.

<FIG> depicts reference system <NUM>, which can be used in place of reference sector <NUM> in <FIG>. Reference system <NUM> comprises reference array <NUM>, low voltage row decoder <NUM>, high voltage row decoder <NUM>, and low voltage reference column decoder <NUM>. Low voltage row decoder <NUM> provides a bias voltage for read and program operations and provides a decoding signal for high voltage row decoder <NUM>. High voltage row decoder <NUM> provides a high voltage bias signal for program and erase operations.

<FIG> depicts VMM high voltage decode circuits, comprising word line decoder circuit <NUM>, source line decoder circuit <NUM>, and high voltage level shifter <NUM>, which are appropriate for use with memory cells of the type shown in <FIG>.

Word line decoder circuit <NUM> comprises PMOS select transistor <NUM> (controlled by signal HVO_B) and NMOS de-select transistor <NUM> (controlled by signal HVO_B) configured as shown.

Source line decoder circuit <NUM> comprises NMOS monitor transistors <NUM> (controlled by signal HVO) , driving transistor <NUM> (controlled by signal HVO), and de-select transistor <NUM> (controlled by signal HVO_B), configured as shown.

High voltage level shifter <NUM> received enable signal EN and outputs high voltage signal HV and its complement HVO_B.

<FIG> depicts VMM high voltage decode circuits, comprising erase gate decoder circuit <NUM>, control gate decoder circuit <NUM>, source line decoder circuit <NUM>, and high voltage level shifter <NUM>, which are appropriate for use with memory cells of the type shown in <FIG>.

Erase gate decoder circuit <NUM> and control gate decoder circuit <NUM> use the same design as word line decoder circuit <NUM> in <FIG>.

Source line decoder circuit <NUM> uses the same design as source line decoder circuit <NUM> in <FIG>.

High voltage level shifter <NUM> uses the same design as high voltage level shifter <NUM> in <FIG>.

<FIG> depicts adaptable neuron circuit <NUM> that converts an output neuron current into a voltage. Adaptable neuron circuit <NUM> uses only one PMOS transistor <NUM> and essentially is configured to mirror itself (i.e., a sample and hold mirror) using switches <NUM>, <NUM>, and <NUM>. Initially, switch <NUM> and switch <NUM> are closed and switch <NUM> is open, at which time PMOS transistor <NUM> is coupled to I_NEURON, which is a current source that represents the current from a VMM. Then, switch <NUM> and <NUM> are opened and switch <NUM> is closed, which causes PMOS transistor <NUM> to send current I_NEURON from its drain to variable resistor <NUM>. Thus, adaptable neuron <NUM> converts a current signal (I_NEURON) into a voltage signal (VO). Basically, transistor <NUM> samples the current I_NEURON and holds it by storing a sampled gate-source voltage on its gate. An op amp circuit can be used to buffer the output voltage VO to drive the configurable interconnect.

<FIG> depicts current sample and hold S/H circuit <NUM> and voltage sample and hold S/H circuit <NUM>. Current S/H circuit <NUM> includes sampling switches <NUM> and <NUM>, S/H capacitor <NUM>, input transistor <NUM> and output transistor <NUM>. Input transistor <NUM> is used to convert input current <NUM> into an S/H voltage on the S/H capacitor <NUM> and is coupled to gate of the output transistor <NUM>. Voltage S/H circuit <NUM> includes sampling switch <NUM>, S/H capacitor <NUM>, and op amp <NUM>. Op amp <NUM> is used to buffer the S/H voltage on the capacitor <NUM>. S/H circuits <NUM> and <NUM> can be used with the output summer circuits and/or activation circuits described herein. In an alternative embodiment, digital sample and hold circuits can be used instead of analog sample and hold circuits <NUM> and <NUM>.

<FIG> shows an array architecture that is suitable for memory cells operating in linear region. System <NUM> comprises input block <NUM>, output block <NUM>, and array <NUM> of memory cells. Input block <NUM> is coupled to the drains (source lines) of the memory cells in array <NUM>, and output block <NUM> is coupled to the bit lines of the memory cells in array <NUM>. Alternatively, input block <NUM> is coupled to the wordlines of the memory cells in array <NUM>, and output block <NUM> is coupled to the bit lines of the memory cells in array <NUM>.

In instances where system <NUM> is used to implement an LSTM or GRU, output block <NUM> and/or input block <NUM> may include multiplier block, addition block, subtraction (output = <NUM> - input) block as needed for LSTM/GRU architecture, and optionally may include analog sample-and-hold circuits (such as circuits <NUM> or <NUM> in <FIG>) or digital sample-and-hold circuits (e.g., a register or SRAM) as needed.

Claim 1:
A configurable vector-by-matrix multiplication system (<NUM>, <NUM>), comprising:
an array (<NUM>) of memory cells arranged into rows and columns;
an activation block (2203a, 2203b, 2203c, 2203d, 2203e, 2203f, 2303a, 2303b, 2303c, 2303d, 2303e, 2303f) coupled to the array for generating a vector of input currents in response to a vector of input voltages and providing the vector of input currents to a plurality of memory cells in the array during a vector matrix multiplier operation;
an output block (2202a, 2202b, 2202c, 2202d, 2202e, 2202f, 2302a, 2302b, 2302c) coupled to the array for generating a vector of output voltages in response to current received from a plurality of memory cells in the array during a vector matrix multiplier operation;
a controller or control logic (<NUM>) configured to generate during a first operation cycle a first sub-array in the array of memory cells and to couple during the first operation cycle the output block and the activation block to the first sub-array, and generate during a second operation cycle a second sub-array in the array of memory cells and to couple during the second operation cycle the output block and the activation block to the second sub-array; and
routing circuitry (2204a, 2204b, 2204c, 2204d, 2205a, 2205b, 2205c, 2206a, 2206b, 2304a, 2304b, 2304c, 2304d, 2305a, 2305b, 2305c, 2305d, 2305d, 2305f, 2306a, 2306b) configured to route, in response to current received by the output block from the first sub-array, a vector of output voltages from the output block to the activation block for performing a vector matrix multiplier operation by the second sub-array.