Patent Description:
Neural networks are machine learning models that employ one or more layers of models to generate an output, e.g., a classification, for a received input. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer of the network.

<NPL>, describes a 3D-Dot product engine for accelerating neural networks.

<NPL>, describes on-chip training circuits for memristor based deep neural networks utilizing unsupervised and supervised learning methods.

According to an aspect of the present disclosure, there is provided a circuit for performing neural network computations for a neural network comprising a plurality of neural network layers, as defined by claim <NUM>.

In some implementations, the circuit includes an activation unit communicatively coupled to the matrix computation unit and configured to, for one or more of the neural network layers, apply an activation function to accumulated values generated by the matrix computation unit to generate a plurality of activated values for the neural network layer.

In some implementations, the circuit includes a digital to analog converter (DAC) device, the DAC device being connected to a nanowire of the first set of nanowires of a first crossbar array. In some implementations, the circuit includes an analog to digital converter (ADC) device, the ADC device being connected to the second set of nanowires of a second crossbar array. The matrix computation unit is coupled to the activation unit by the ADC devices. In some implementations, the ADC device is configured to recognize an encoded signal from a nanowire of the second set of nanowires, the encoded signal representing a sign of an accumulated value of the matrix computation unit.

In some implementations, the activation unit is formed in the common substrate with the plurality of crossbar arrays.

In some implementations, the activation unit normalizes each activated value to generate a plurality of normalized values. In some implementations, the activation unit pools one or more activated values to generate a plurality of pooled values.

In some implementations, the crosspoint devices include a memristor device, and the electrical property that is tuned to the value is a conductance of the memristor device. In some implementations, the crosspoint devices include a memcapacitor device, and the electrical property that is tuned to the value is a capacitance of the memcapacitor device.

In some implementations, a first crossbar array of the plurality of crossbar arrays is connected, in the stacked configuration, to a second crossbar array of the plurality of crossbar arrays by metal vias, and the metal vias are configured to transmit activated values from the first crossbar array to the second crossbar array. In some implementations, the metal vias have a length between <NUM>-<NUM>.

In some implementations, a crossbar array of the plurality of crossbar arrays includes dimensions of approximately <NUM><NUM>. In some examples, one or more of the crossbar arrays may include dimensions of at least <NUM><NUM>, optionally at least <NUM><NUM>, and/or less than <NUM><NUM>, optionally less than <NUM><NUM>.

In some implementations, the circuit includes a processing unit configured to receive instructions and generate a plurality of control signals from the instructions, and the plurality of control signals control dataflow through the circuit. In some implementations, the circuit includes a multiplexer communicatively coupled to the processing unit and the matrix computation unit, and the multiplexer is configured to send the plurality of activation inputs to the matrix computation unit.

In some implementations, the circuit includes a memory unit configured to send a plurality of weight inputs to the matrix computation unit, and the direct memory access engine is configured to send the plurality of weight inputs to the memory unit.

In some implementations, the plurality of weight inputs is applied to the plurality of crossbar arrays to preload the set of crosspoint devices with the value of the tunable electrical property.

In some implementations, the circuit includes a shift-add circuit configured to sum the plurality of activated values; a sum-in register configured to store the summed plurality of activated values; and summation circuitry communicatively coupled to the matrix computation unit and the sum-in register, and the summation circuitry is configured to output a sum of a product and the summed plurality of activated values.

In some implementations, each crossbar array of the plurality of crossbar arrays is connected to a shift-add circuit, each shift-add circuit configured to store a respective sum in a respective accumulator unit, where the respective sum is an accumulated value. In some implementations, an output value of a crossbar array represents a sign of the accumulated value.

In some implementations, each crossbar array includes approximately one million crosspoint devices. In some examples, each crossbar array may include at least <NUM>,<NUM>, optionally at least <NUM>,<NUM>, crosspoint devices and/or less than <NUM>,<NUM>,<NUM>, optionally less than <NUM>,<NUM>,<NUM>, crosspoint devices.

In some implementations, the matrix computation unit comprises approximately <NUM> crossbar arrays. In some examples, the matrix computation unit may comprise at least <NUM>, optionally at least <NUM>, crossbar arrays, and/or less than <NUM>, optionally less than <NUM>, crossbar arrays.

In some implementations, each crosspoint device is configured to store an <NUM>-bit weight value represented by the electrical property. In some implementations, the matrix computation unit is configured to perform functions for a recurrent neural network, and the crosspoint devices are pre-loaded once for performing the functions of the recurrent neural network.

The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages. The stacked crossbar array can include all the weights of a neural network model on a single chip die. Limitations of memory bandwidth which are a limitation of conventional neural network accelerators can be bypassed or mitigated. Compared with CMOS implementations, which require several circuit elements per weight, the much higher storage density of crossbars (one <NUM>-bit weight per crosspoint device) enables larger production models for the stacked crossbar array, such as at least an order of magnitude larger.

The stacked crossbar array consumes approximately an order of magnitude less energy than a corresponding CMOS configuration. The stacked crossbar array reduces timesharing requirements needed for a two-dimensional configuration, reducing latencies of using analog neural network layers that are two dimensional. In some implementations, all parameters of the neural network are stored in the stacked crossbar array, eliminating the need to retrieve the parameters from circuitry that is off-chip. The stacked crossbar arrays have shorter bitlines between arrays than two dimensional systolic array configurations, reducing or eliminating the need for drive buffers between layers. The stacked crossbar arrays can include millions or billions of operators compared to thousands of operators for CMOS configurations of a similar size. The smaller footprint and lower power requirements enable specialized neural network chips for mobile devices and other devices in which chip size and power consumption are limited.

Other potential features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

This specification describes computer hardware and software systems that can be used to accelerate machine learning workloads such that a processing unit performing the workload can achieve high performance and low energy consumption while executing computations. The hardware and software systems described below include an integration of analog and digital logic. Machine learning is performed by a computing system that includes a multilayer circuit, including analog operators, that communicates with one or more digital processing units.

In particular, this specification describes accelerating the processing of inferences by Deep Neural Networks (DNNs) that include multiple layers that feed into each other. A computation process performed within a neural network layer can include a multiplication between an input tensor and a weight tensor. Each multiplication operation occurs at an operator (e.g., synapse). A neural network layer can include many operators, and each layer can produce many products, such as for a matrix multiplication. A neural network layer can sum the products of each operator to produce an accumulated value. Thus, an input-weight multiplication may be written as the sum-of-product of each weight element multiplied with a row of the input or matrix. An activation function is applied to the accumulated value, such as by an activation unit, to produce an activated value. The activated value can represent an input to a subsequent layer of the neural network.

Computing systems can perform neural network operations using hardware that is configured to perform the multiplication operations and activation functions described above. A portion of a neural network layer can be represented by a matrix of hardware operators, each configured to multiply an input tensor and a weight tensor. The hardware operators can be arranged into layers, where each layer represents a neural network layer. The hardware layers can be arranged into a multilayer circuit.

The multilayer circuit includes interconnected crossbar arrays that are each configured to operate as a neural network layer. The crossbar arrays each include a number of electronic operators (e.g., crosspoint devices) that together define the output of the layer as a function of one or more inputs. The crossbar arrays are stacked vertically, increasing the density of operators of the network and increasing the number of operators that can be placed in given chip frame.

The stacked configuration of the crossbar arrays allows for individual neural network layers to be larger and include more operators than two-dimensional crossbar layer configurations, such as a systolic array configuration. The stacked configuration of the crossbar arrays that includes larger layer sizes eliminates at least a portion of timesharing operations, required by smaller layers, for computing an accumulated value output of the same number of input parameters. Eliminating timesharing operations reduces a time and energy overhead for computing activation values, as timesharing operations require extra digital/analog conversions for inputting data to a crossbar array and retrieving the result from the crossbar array. Rather, the three-dimensional stacking of the crossbar arrays enables implementation of fully connected neural networks, without requiring sequential analog to digital conversions.

The crossbar arrays in the stack each include more operators for a given cross-sectional size than a CMOS implementation of equal size. For example, the crosspoint devices of a crossbar array can be configured to perform both storage functionality for the parameter/weight values of the layer and perform the multiplication operations of the layer. As such, transistor requirements for crossbar array layers are reduced by an order of magnitude compared to CMOS implementations. The crossbar arrays can include a number of operators on the order of millions, while comparable CMOS implementations can include thousands. The crossbar arrays enable recurrent neural network functionality using layers of a size less than <NUM><NUM>- <NUM><NUM>. The stacked crossbar array configuration enables the neural network to scale to millions or billions of operators for a single chip die. The smaller footprint of the neural network layers of the multilayer circuit described below enables specialized hardware acceleration in smaller chip frames, such as those required for mobile devices.

Additionally, once the crosspoint devices of a layer of the stacked crossbar arrays have been preloaded with the parameter/weight values, the parameter/weight values do not need to be fetched again for recursive computations, saving time and energy. This increases the performance speed by a factor of <NUM> or more and reduces energy consumption by an order of magnitude in comparison to CMOS implementations and two-dimensional systolic array implementations.

<FIG> shows a block diagram of an example computing system <NUM> that includes a crossbar array stack <NUM> for performing computations for a neural network. As shown, computing system <NUM> includes a processing unit <NUM>, a storage medium <NUM>, multiply accumulate (MAC) system <NUM> hardware that includes the crossbar array stack <NUM>, and an activation unit <NUM>. In some implementations, the system <NUM> includes additional crossbar array stacks that are each a part of additional MAC systems.

The processing unit <NUM> is configured to process instructions for execution within the computing system <NUM>, including instructions or program code stored in the storage medium <NUM> or other instructions/code stored in another storage device. The processing unit <NUM> may include one or more processors. Storage medium <NUM> can include one or more memory banks or units, including first bank <NUM> for storing activation inputs and second bank <NUM> for storing weights. In some implementations, storage medium <NUM> is a volatile memory unit or units. In some other implementations, storage medium <NUM> is a non-volatile memory unit or units such as, for example, read-only memory (ROM) and/or electrically erasable programmable read-only memory (EEPROM). The storage medium <NUM> may also be another form of computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations.

Instruction set <NUM>, when executed by the processing unit <NUM>, causes the processing unit <NUM> to perform one or more tasks that include, for example, storing activation inputs in memory address locations of first bank <NUM> and storing weights in memory address locations of second bank <NUM>. Moreover, instructions <NUM> can also cause processing unit <NUM> to access the stored data from first bank <NUM> and second bank <NUM> and provide the accessed data to MAC system <NUM>. As described in more detail below, the MAC system <NUM> can perform multiply operations that can include multiplying an activation with a weight to produce either a partial sum(s) or to produce an output activation(s) that is provided to activation unit <NUM>.

The host interface <NUM> can send the instructions to the processing unit <NUM>, which converts the instructions into low level control signals that control the circuit to perform the neural network computations. In some implementations, the control signals regulate dataflow in the circuit, e.g., how the sets of weight inputs and the sets of activation inputs flow through the circuit. The processing unit <NUM> can send the control signals to the first and second banks, a MAC system <NUM>, and an activation unit <NUM>. In some implementations, the processing unit generates clock signals. The processing unit <NUM> can use timing of the clock signals to, at appropriate times, send the control signals to each component of the circuit system <NUM>. In some other implementations, an external processor controls the clock signal. Clock rates can be any conventional computing clock rate, but typically ranges between <NUM> to <NUM>.

As discussed in more detail below with reference to <FIG>, activation inputs can be loaded from the first bank <NUM> in respective memory address locations that correspond to input bitline positions in a layer of the stacked crossbar arrays <NUM>. For example, each input bitline of a crossbar array can be connected to a multiplexer (not shown) that selects the appropriate bitline to receive the activation input based on the addresses of the stored activation values. When the activation value is needed by the MAC system <NUM>, the processing unit <NUM> controls the multiplexer to load each bitline with the appropriate activation value via a digital to analog conversion (DAC) interface <NUM>. Likewise, weights can be loaded to second bank <NUM> in respective memory address locations that correspond to positions of crosspoint devices in the stacked crossbar array <NUM>. Each crosspoint device connected to a bitline is pre-loaded with a weight value received from a weight value from the second bank <NUM> through the DAC interface <NUM>. The crosspoint devices store the weights in the form of a tunable electric property of the crosspoint devices, as described in further detail in relation to <FIG>, below. In some implementations, instructions, activation inputs, and weights are provided to system <NUM> from an external or higher level control device associated with a neural network hardware computing system.

The MAC system <NUM> receives the weight values from the second bank <NUM> and the activation inputs from the first bank <NUM> as instructed by the processing unit <NUM>. The processing unit is configured to preload each crosspoint device of the crossbar array stack <NUM> with a weight value before the MAC system <NUM> receives the activation inputs. The processing unit <NUM> controls which bitlines of the crossbar arrays receive which activation and weight signals using one or more multiplexer devices (not shown). The multiplexer devices are connected to each bitline via the DAC interface <NUM>. Once the weights are preloaded into the crossbar arrays, the processing unit <NUM> sends each bitline the corresponding activation signal from the first bank <NUM> via the DAC interface. The preloaded weights can be used for multiple different sets of inputs, such as to compute many different inferences, without requiring a second preloading process.

The crossbar arrays, which each perform operations of a neural network layer, are interconnected to form the stack, as described in greater detail with respect to <FIG>. Each layer is accessible, via a buffer, for loading weights and for receiving activation values. Each layer is configured to receive the output activation value from the preceding layer of the stack, such that activation values are communicated between the layers. In some implementations, each layer is fully connected, such that all crosspoint devices are used as operators for a given multiplication operation. In some implementations, a layer can be partially connected. For example, one or more layers can be partially connected to implement specialized operations of a layer (such as to function as a convolutional layer). In some implementations, the specialized operations are reprogrammable by the processing unit <NUM>. In some implementations, activation signals are provided to a single crossbar array, and the signal propagates through the stack and out a single analog to digital (ADC) interface <NUM>. In some implementations, activation signals are provided to more than one crossbar array. In some implementations, accumulated values are read from a single crossbar layer of the crossbar array stack <NUM> via the ADC interface <NUM>. In some implementations, accumulated values can be read from several or each crossbar array of the stack, such as through multiple ADC devices of a ADC interface <NUM> and from sample and hold registers connected to each bitline of the crossbar arrays.

The accumulated signals generated by the crossbar arrays of the stack <NUM> are sent to the activation unit <NUM>. In some implementations, the activation unit applies an activation function to the accumulated signals. The function can be specified by the processing unit <NUM>. The activated signals can be output <NUM> by the system <NUM>, stored in the storage medium <NUM>, or sent back into the neural network.

<FIG> shows an unclaimed hardware configuration of a crossbar array stack <NUM> (e.g., crossbar array stack <NUM> of <FIG>). The crossbar array stack <NUM> includes a first crossbar array <NUM> and a second crossbar array <NUM>. The first and second crossbar arrays <NUM>, <NUM> each represent a neural network layer. The first and second crossbar arrays <NUM>, <NUM> are connected by a transition layer <NUM>. While two crossbar layers <NUM>, <NUM> are shown, in a claimed hardware configuration the crossbar array stack <NUM> can include tens of layers, hundreds of layers, etc. In some implementations, additional drive buffers can be required to add additional stacking layers. For example, stacking additional crossbar layer dies using a micro bump reflow approach can require an adjustment to a buffer drive strength between crossbar layer dies. However, in this context, driving a Through-Silicon-Via (TSV) requires less power than driving long chip wires. TSVs can be shorter than nanowires required for full size crossbar layers since a crossbar die can be thinned for a stacking application, and so buffer drive strength can be maintained by using smaller layer dimensions. In some implementations, additional drive buffers are not required to add additional stacking layers. Some die-die bonding techniques (e.g., homogeneous (oxide-to-oxide) direct bonding) that include finer pitch than micro bump reflow (and thus smaller electrical parasitics). Such techniques also offer lower cost, lower warpage, and lower delimitation. Such bond techniques can bypass a requirement for a stronger buffer for stacking applications.

Crossbar array <NUM> includes a first set of nanowires, such as nanowire <NUM>, and a second set of nanowires, such as nanowire <NUM>. Each nanowire <NUM>, <NUM> can serve as a bitline for matrix multiplication operations. The first set of nanowires and the second set of nanowires are connected by a plurality of crosspoint devices, such as crosspoint device <NUM>. Each nanowire <NUM> of the first set of nanowires is connected to each nanowire <NUM> of the second set of nanowires by a crosspoint device <NUM>. Crossbar array <NUM> is similarly configured as crossbar array <NUM>, except that the activation inputs to crossbar array are the output activation values of crossbar <NUM>.

To compute an activation value, activation inputs are each sent to a nanowire <NUM> of the first set of nanowires. The activation input value is converted to a voltage level by a digital to analog conversion for each nanowire <NUM> of the set. Each crosspoint device has a tunable electrical property, such as resistance, capacitance, etc., that determines the current amount that is contributed from the first nanowire <NUM> to the second nanowire <NUM>. Each nanowire <NUM> of the second set receives some current from each of the nanowires <NUM> of the first set. The sum of all the currents represents the value dot product operation of M activation values by N weights, wherein M is the number of nanowires of the first set and where N is the number of crosspoint devices connected to each nanowire of the second set. In some implementations, the crossbar arrays <NUM>, <NUM> are squareMxMarrays, where the number M of nanowires in the first set of nanowires is the same as the number M of nanowires of the second set. In some implementations, the crossbar arrays have differing numbers of nanowires to create an M × N array.

The crossbar arrays <NUM>, <NUM> can compute matrix multiplication up to the size of the crossbar arrays <NUM>, <NUM>. For example, for a crossbar array <NUM> of M × N size, where M represents of the number of nanowires <NUM> of the first set and N represents the number of nanowires <NUM> in the second set, the crossbar array <NUM> is capable of handling M activation inputs for M × N operators, resulting in N activation outputs. The crossbar array <NUM> can perform the matrix multiplication operations in parallel. For larger input vectors, timesharing of the array can be performed, wherein the vector is divided into pieces, matrix multiplication is performed, and the results can be combined by an accumulator or pooled by a pooling layer.

The crossbar arrays <NUM>, <NUM> are connected by metal vias, such as via <NUM>. Via <NUM> can be a solid piece of metal that conductively connects the second set of nanowires of crossbar array <NUM> to the first set of nanowires of crossbar array <NUM>. For example, each activation output of the first crossbar array <NUM> can be the activation input of the second crossbar array <NUM>. The metal vias are long enough to allow enough substrate to be between crossbar arrays <NUM>, <NUM> such that capacitive effects are reduced or eliminated between neural network layers, without increasing inductance effects too greatly. In some implementations, the metal via <NUM> is approximately <NUM>-<NUM>. However, the metal via <NUM> can be approximately <NUM>-<NUM>. The vias are approximately <NUM>-<NUM> wide. In some implementations, the vias are <<NUM> wide, such as if homogeneous bonding techniques are used to connect the stacking layer dies. Other lengths and widths can be used depending on the dimensions of the crossbar arrays, the size of the nanowires, and the current levels of the crossbar arrays.

Although the vias <NUM> are depicted as connecting ends of the first and second nanowires <NUM>, <NUM>, the vias can be placed anywhere along the bitline, such as in the center of the array stack. In some implementations, the vias <NUM> need not be aligned as depicted, but can be offset to reduce capacitive effects between vias. In some implementations, the vias <NUM> connect the output & input of the crossbars (i.e., the neurons/activations). A connection density based on the number of crosspoints in a network layer, typically square root of the number of crosspoints.

In some implementations, the metal vias <NUM> are each in sequence with a device <NUM>. Device <NUM> represents one or more analog or digital logic devices, signal inputs, or signal outputs disposed between crossbar layers. For example, device <NUM> can include a drive buffer that refreshes the signal after a number of crossbar layers and that prevents electrical characteristics of crossbar layer <NUM> from affecting analog values of the crossbar array <NUM>. For example, device <NUM> can include a buffered output line such that the activation outputs of the crossbar layer <NUM> can be read from the crossbar array stack. For example, device <NUM> can include a buffered input line that is used to preload the crosspoint devices of crossbar array <NUM>.

The crossbar arrays <NUM>, <NUM> can compute a number of matrix multiplication operations without the requirement of timesharing a crossbar array in the stack. The crossbar array stack <NUM> enables recurrent neural network operations to be performed without requiring intermediate analog/digital conversions of data between layers. Timesharing requires the crossbar array to be reinitialized, such as with preloaded weights for the crosspoint devices, before additional matrix multiplication operations can be performed. The crossbar array stack <NUM> can send activation outputs to a second layer without requiring weights to be preloaded after the matrix multiplication has occurred in the first layer. Additionally, the crossbar array <NUM> can send activation outputs as analog signals directly to the second crossbar array <NUM>, avoiding a time and energy overhead of converting to a digital signal and back to an analog signal.

The crossbar array stack <NUM> consumes less power compared to a systolic array of crossbar cells. A two dimensional configuration of crossbar arrays requires relatively long wires (e.g., <NUM>-<NUM> for a 1000x1000 crossbar layer) between cells to transmit the activation output from a first cell to a second cell. The crossbar array stack <NUM> places the crossbar arrays <NUM>, <NUM> in a relatively compact volume envelope. The crossbar arrays can be connected by the metal vias <NUM>, which are relatively short in comparison (e.g., <NUM>-<NUM>). The crossbar array stack <NUM> can operate at lower voltages because the crossbar array stack can use lower voltage drive buffers than a systolic array configuration and still have signal fidelity between crossbar arrays. In addition to consuming less power, the crossbar array stack <NUM> produces less waste heat than a two dimensional configuration. As such, by avoiding timesharing operations and using lower power drive buffers, energy consumption can be reduced by a factor of <NUM> over two dimensional and digital configurations, and the speed of computations can be increased by a factor of <NUM> in comparison to two dimensional and digital configurations.

The crosspoint devices, such as crosspoint device <NUM>, are tuned by a processing device (e.g., processing device <NUM>) of the system. The crosspoint device <NUM> can be tuned by a tuning voltage applied during a tuning phase of the operation of the crossbar array stack <NUM>. For example, the processing device sends a signal, such as a voltage, to device <NUM>. The signal tunes an electrical property of the device <NUM> to control a conductance of the device, as explained in further detail in relation to <FIG>. The device <NUM> stores a weight value that is represented by the conductance of the device <NUM>. During calculation of an activation value, the conductance of each crosspoint device <NUM> determines how much of the signal from an activation input is transmitted to the activation output, such as how much current flows from the first nanowire <NUM> to the second nanowire <NUM>. The crosspoint devices <NUM> thus have dual functionality, as the crosspoint devices store weights of each neural network layer and also form the synapses of the neural network layer by functioning as operators. In some implementations, the crosspoint devices <NUM> are loaded using the crossbar lines without any dedicated loading input line. The operator values are stored in the crosspoint device <NUM> and are reused for many inferences (up to hundreds, thousands, millions, etc.). Since the operator values remain loaded for many inferences of the neural network, loading time overhead for loading the crosspoint devices <NUM> becomes a smaller percentage of operation time as the number of inferences increases.

The dimensions of the crossbar arrays <NUM>, <NUM> can vary based on the size of the chip frame and the transistor technology that is included, such as for the crosspoint device <NUM>. To avoid including relatively long nanowires in the crossbar arrays <NUM>, <NUM>, which require higher drive voltages and more power consumption, the dimensions of the crossbar arrays <NUM>, <NUM> can be limited to between <NUM><NUM> and <NUM><NUM>. Each crossbar array <NUM>, <NUM> can include approximately a million operators, represented by crosspoint devices <NUM>. In some implementations, fewer crosspoint devices <NUM> can be used in a crosspoint array <NUM>, <NUM> to increase robustness of operation by reducing capacitances of crossbars and enable lower operating voltages, but at a cost of the overhead of the connection space between crossbar arrays <NUM>, <NUM>. For example, a crossbar array <NUM>, <NUM> can include only 10x10 crosspoint devices <NUM>. In some implementations, a greater number of crosspoint devices <NUM> can be used per crosspoint layer <NUM>, <NUM> to reduce overhead of crossbar connections and increase the number of operators in the network, but this can require higher drive voltages and more waste heat. For example, crossbar arrays <NUM>, <NUM> can be 2000x2000 crosspoint devices <NUM>. Other configurations of the crossbar layers <NUM>, <NUM> are possible. For example, <FIG> shows an unclaimed alternative layout of a crossbar array stack <NUM> to the crossbar array stack <NUM> of <FIG>.

In some implementations, the crossbar array stack can be formed from a single die, and can be formed in the same die as complementary analog or digital circuitry, such as drive buffers, ADC and DAC interfaces, etc. Turning to <FIG>, a crossbar array stack <NUM> is shown with additional circuitry represented by blocks <NUM> and <NUM>. The blocks <NUM>, <NUM> can include specialized circuitry to handle signals from the crossbar array stack. For example, the blocks <NUM>, <NUM> can include interfaces for the crossbar array stack, multiplexing devices, reprogrammable circuit layers, buffers, registers, and so forth. In some implementations, the specialized circuitry can be configured to optimize a specific neural network task, such as speech recognition, image recognition, etc..

In some implementations, blocks <NUM> and <NUM> can include analog to digital conversion (ADC) and digital to analog conversion (DAC) devices. The ADC and DAC devices are used to interface the analog crossbar stack with the digital devices of the computing system. Various types of ADC devices and DAC devices can be used, as described in greater detail below in relation to <FIG>.

In some implementations, blocks <NUM> and <NUM> can include sample and hold devices. The sample and hold devices acquire the accumulated signals from the crossbar arrays and hold the signal until the ADC device is ready to convert the signal. A sample and hold device can be attached to each bitline of a crossbar array.

Blocks <NUM>, <NUM> include an activation unit that applies an activation function to the accumulated values. The activation unit receives the accumulated values, such as from the ACD device, and applies an activation function to generate activation values. Such a configuration keeps operations of the neural network on a single chip die, resulting in the advantages described above. In some implementations, activation hardware can be in a separate device.

In some implementations, blocks <NUM>, <NUM> include a shift-add circuit configured to sum the plurality of activated values. The shift-add circuit can be connected to a sum-in register configured to store the summed plurality of activated values. Summation circuitry can be communicatively coupled to the matrix computation unit and the sum-in register, where the summation circuitry is configured to output a sum of a product and the summed plurality of activated values. Other configurations of the crossbar layers <NUM>, <NUM> are possible. For example, <FIG> shows an unclaimed alternative layout of a crossbar array stack <NUM> to the crossbar array stack <NUM> of <FIG>.

<FIG> show examples of crosspoint devices <NUM>, <NUM> for the crossbar arrays, such as crossbar arrays <NUM>, <NUM>. The crosspoint devices have dual functionality. The crosspoint devices store the weight values from the second bank (e.g., bank <NUM> of <FIG>). The weights can be preloaded into the crossbar arrays, such as before matrix multiplication operations are to take place. When the bitlines of the crossbar arrays (e.g., nanowires <NUM>) receive activation inputs, the crosspoint devices act as operators for the matrix multiplication, and convert the activation input from the input bitline to an activation output on an output bitline, such as on nanowire <NUM>.

The crosspoint devices store weights using an electrical property of the crosspoint devices. The preloading process of the crossbar array tunes the electrical properties of the crosspoint devices, such as by using a tuning voltage or current. During the preloading process, each bitline of the crossbar array to be sent a tuning signal, such as from a tuning source. In some implementations, the preloading processes for each crossbar array of the stack can be performed in parallel. In some implementations, the tuning of each crossbar array is performed in sequence. Each time the crossbar array is to perform a matrix multiplication, the crosspoint devices are tuned based on the desired weight values. In recurrent neural networks, the crosspoint devices need not be tuned between operations, but instead, for a second (and subsequent) operation, the crosspoint devices exhibit hysteresis from the prior operation.

When the matrix multiplication operations are performed, the crosspoint devices act as operators on the bitline signals received from the nanowires <NUM> of the first set of nanowires. The exact mechanism by which the operator performs the operation depends on the type of device being used for the crosspoint device. For example, <FIG> shows a crossbar array <NUM> including a memristor array for the crosspoint devices, such as memristor <NUM>. Memristors can include a tunable resistance based on a current that is applied to the memristor device. For example, for a lower weight to apply to the crosspoint, the memristor <NUM> is turned to a higher resistance value. Less current is contributed from the first nanowire <NUM> connected to the memristor <NUM> to the second nanowire <NUM> connected to the memristor. The second nanowire <NUM> receives some or no current from each memristor crosspoint device connected to the second nanowire. The activation value output of the second nanowire <NUM> represents the dot product of each activation input and the weights of each memristor connected to the second nanowire. The activation output can be passed to a subsequent layer of the crossbar stack (repeating the process), sent to an output register, or both.

<FIG> shows a memcapacitor array for the crosspoint devices, such as memcapacitor <NUM>. The memcapacitors function in a similar manner to the memristors. The memcapacitor <NUM> is charged to a voltage that represents a weight for the crossbar array <NUM>. The memcapacitor <NUM> can store the voltage value until matrix multiplication operations are to be performed. When an activation input is sent to a bitline connected to the memristor <NUM> (e.g., nanowire <NUM>), the memcapacitor induces a voltage on the output bitline (e.g., nanowire <NUM>) that is linearly proportional to the weight voltage value and input voltage value.

Memristor <NUM> and memcapacitor <NUM> of the crossbar array stacks <NUM>, <NUM> can form denser storage than digital storage devices. For example, memristors can store analog weight values that have converted from an eight-bit digital signal using two transistors. The low number of transistors required for the crosspoint devices enables scalability to millions or billions of operators in a single crossbar array stack.

<FIG> shows a three-dimensional representation of a crossbar array stack <NUM>. Crossbar array <NUM> is stacked over a number of crossbar layers, terminating in crossbar layer <NUM>. Crossbar layer <NUM> is connected to a DAC interface <NUM> by each bitline of the crossbar array <NUM>. The DAC device <NUM> converts the activation input, the preload weight value, etc. to an analog representation of the signal. In some implementations, a single-bit drive buffer can be used to drive the analog input signal. In some implementations, the DAC device <NUM> converts four-bit signal, <NUM>-bit signals, and <NUM>-signals. In some implementations, each bit of a multi-bit signal is handled by a different bitline, and the results of the operation are merged after conversion back to a digital signal, such as using the ADC interface <NUM>. For example, if an <NUM>-bit signal is being operated upon, a bit can be sent to each bitline of the first crossbar array <NUM>. The synaptic weights of the crossbar array <NUM> can be replicated to be identical for each bitline. If the bit size of the signal exceeds the number of bitlines, the signal can be divided to a more significant portion and a less significant portion, processed over multiple cycles, and merged after matrix multiplications have been performed on each bit of the signal.

Likewise, to represent high-precision weight values, such as <NUM>-bit weight values, weights can be represented by multiple crosspoint devices on the same bitline. For example, if the crosspoint devices are <NUM>-bit memristors, the <NUM>-bit weight value can be represented in four crosspoint devices of the row. The results of each operation are later merged.

The output activation values are converted back to digital signals through the ADC device <NUM>. The ADC device <NUM> can retrieve values from buffers at the end of each output bitline, such as sample and hold buffers, and convert to the digital representation of each result. The resolution of the ACD device <NUM> can be reduced by encoding the input signal, such as by using a unit column bitline. The unit column bitline can represent a value that is recognized by the ADC during conversions back to a digital signal. For example, the unit column bitline can be used to represent signed operations. Any known encoding schemes for efficient ADC and DAC can be used - no special ACD/DAC scheme is required.

<FIG> represents an example method <NUM> for performing neural network operations using the stacked crossbar arrays described above. The computing system receives (<NUM>) the weight values and an instruction set, such as from another system. The computing system preloads (<NUM>) the crosspoint devices with the weight values. Once all the crosspoint devices have been tuned by the preloading signals, the system sends (<NUM>) activation values to the crossbar array stack. The crossbar array stack sums (<NUM>) the activation values to generate activation outputs. The activation outputs can be combined to form accumulated values (<NUM>). The accumulated values can be sent back to the crossbar array as activation values, stored in the computing system storage, or otherwise be analyzed by the system.

Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory program carrier for execution by, or to control the operation of, data processing apparatus. The program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

A computer program (which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array), an ASIC (application specific integrated circuit), a GPGPU (General purpose graphics processing unit), or some other type of processor.

Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub combination.

Claim 1:
A circuit for performing neural network computations for a neural network comprising a plurality of neural network layers, the circuit comprising:
a matrix computation unit comprising a plurality of crossbar arrays (<NUM>, <NUM>) formed in a common substrate configured in a vertical stack of crossbar arrays, wherein each crossbar array of the plurality of crossbar arrays (<NUM>, <NUM>) in the vertical stack corresponds to operations of a different neural network layer of the plurality of neural network layers, and the plurality of neural network layers of the neural network are allocated to one vertical stack of crossbar arrays, wherein the plurality of crossbar arrays has at least three crossbar arrays, each crossbar array comprising:
a set of crosspoint devices (<NUM>), wherein a respective electrical property of each of the crosspoint devices is adjustable to represent a weight value that is stored for each respective crosspoint device;
a first set of nanowires (<NUM>), each nanowire of the first set of nanowires being configured to receive an activation input; and
a second set of nanowires (<NUM>), each nanowire of the second set of nanowires being connected to each nanowire of the first set of nanowires by a respective crosspoint device of the set of crosspoint devices, wherein each nanowire of the second set of nanowires is configured to output a value that is a function of signals received from each nanowire of the first set of nanowires and the respective electrical properties of the respective crosspoint devices; and
a processing unit (<NUM>) configured to adjust the respective electrical properties of each of the crosspoint devices by pre-loading each of the crosspoint devices with a tuning signal that corresponds to a weight value, wherein a value of the tuning signal for each crosspoint device is a function of the weight value represented by each respective crosspoint device, and wherein the computation unit is configured to perform a plurality of computations using the pre-loaded weight values without reloading the weight values to the crosspoint devices.