Conductance Mapping Technique for Neural Networks

Various implementations described herein are directed to a device having neural network circuitry with an array of synapse cells arranged in columns and rows. The device may have input circuitry that provides voltage to the synapse cells by way of row input lines for the rows in the array. The device may have output circuitry that receives current from the synapse cells by way of column output lines for the columns in the array. Also, conductance for the synapse cells in the array may be determined based on the voltage provided by the input circuitry and the current received by the output circuitry.

BACKGROUND

In some conventional memory architecture designs, various challenges arise in machine learning with respect to scalability, such as in reference to scale-up computations for training and inference while remaining energy efficient. In recent times, some neural networks have been proposed to address scalability challenges, wherein a broad goal of neuromorphic architecture research has led to creation of neural networks for designing electronic components in a manner that takes inspiration from (or at least tries to mimic) the architecture of the human brain. This may be achieved with the desire that one would obtain considerable energy efficient advantages over some conventional neural network designs similar to the often-touted computational efficiency of the human brain. However, substantial challenges remain such as finding effective ways to train neural networks and implementing various techniques for mapping neural networks to the physical substrate, which may be resource limited and thus substantially difficult to implement.

DETAILED DESCRIPTION

Various implementations described herein are directed to configurable neural networking schemes and techniques for energy efficient applications. For instance, the various schemes and techniques described herein may provide for energy efficient online training of spiking neural networks (SNN) using non-volatile memories (NVM), such as, e.g., resistive random access memory (RRAM), magnetic RAM (MRAM), spin-transfer-torque magnetic RAM (STT-MRAM), and correlated-electron RAM (CeRAM). Therefore, various aspects of the present disclosure may provide for performing online training using a spiking neural network (SNN) that is designed with RRAM, MRAM, STT-MRAM and/or CeRAM NVM synapse cells, in a manner as described herein.

Some benefits of neuromorphic computing may refer to the event-driven nature of computational paradigms in that there is a large amount of sparsity in neural network circuitry. In some instances, neuromorphic computing may refer to the instantiation of a computing paradigm that may enable computations on highly sparse representations, which may drive the possibility of making dense deep neural networks sparser to thereby improve energy efficiency. Thus, neural networks may be designed with sparsity from the onset, and with event-driven networks, computation may only occur where and when it is necessary and, in this manner, these computations may lead to energy efficiency benefits and thus scaling neural networks may become easier. In addition to these considerations, some neural network circuitry pursue energy efficiency advantages by performing some calculations in the analog domain, and these analog based calculations may use non-volatile memories (NVM) along with resistive intersecting crossbars.

In a spiking neural network (SNN), information is exchanged between neurons via short messages or voltage spikes with actual information content of each transmission encoded in the time of arrival or dispatch of the spike and/or the rate at which spikes are transmitted. In some approximations of a biological model, charge accumulates as spikes arrive at a neuron (e.g., when inputs of connected neurons fire). Also, this accumulation of charge may lead to a corresponding increase in voltage, which may cause a neuron to fire when the potential difference exceeds a particular voltage threshold. In some models, the accumulated charge may leak away or decay such that the neuron slowly returns to its inactive state, if and when the neuron subsequently fails to fire.

In some neural network applications, training neuromorphic hardware involves considerable effort. In reference to Spike-Timing-Dependent Plasticity (STDP), strength of the connection between neurons may be modulated based on relative timing of input and output spikes. This idea models a biological process that refers to an instantiation of a more general concept of Hebbian learning. The STDP learning rules stipulate that if an input spike arrives shortly before the output spike is generated, then the weight of the corresponding synapse is increased (potentiation). Conversely, if the input spike arrives after the output spike is generated, then weight of a corresponding synapse is decreased (depression). The degree to which each weight is adjusted may be variable, and some formulations use transfer functions for potentiation and depression.

Various implementations of neural networking schemes and techniques will be described in detail herein with reference toFIGS.1-9and10A-10B.

FIGS.1-2illustrate various schematic diagrams of neural network circuitry in accordance with various implementations described herein. In particular,FIG.1shows a schematic diagram100of neural network circuitry104, andFIG.2shows a schematic diagram200of neural network circuitry204.

In various implementations, the neural network circuitry104may refer to a system or device having various integrated circuit (IC) components that are arranged and coupled together as an assemblage or combination of parts that provide for a physical circuit design and related structures. In some instances, a method of designing, providing and building the neural network circuitry104as an integrated system or device that may be implemented with various IC circuit components is described herein so as to thereby implement various neural networking schemes and techniques associated therewith. The neural network circuitry104may be integrated with various neural network computing circuitry and related components on a single chip, and the neural network circuitry104may be implemented in various embedded systems for automotive, electronic, mobile and Internet-of-things (IoT) applications, including remote sensor nodes.

As shown inFIG.1, the neural network circuitry104refers to neural network architecture having an array of synapse cells (C) that are arranged in columns (column_1, column_2, . . . , column_n) and rows row_1, row_2, . . . , row_m). In some implementations, the synapse cells (C) are accessible with a column control voltage via the columns and with a row control voltage via the rows. The memory circuitry104may receive the column control voltage and the row control voltage as input, and also, the memory circuitry104may provide a corresponding output. The memory circuitry104may be adapted to record neural network training profile data to determine an extent to which various connections between neurons are adjusted. The synapse cells (C) may include non-volatile memory (NVM) cells108, and each NVM synapse cell108may include a transistor (T) and resistor (R) that are coupled in series. The synapse cells (C) may refer to programmable synapse cells for use in non-volatile memory (NVM) applications including at least one of resistive random access memory (RRAM) applications, magnetic RAM (MRAM) applications, spin-transfer-torque magnetic RAM (STT-MRAM) applications, and correlated-electron RAM (CeRAM) applications.

In various implementations, the synapse cells (C) may be disposed at neuronal junctions, e.g., where corresponding columns and rows intersect. The synapse cells (C) may be positioned in the array at crossbar intersection points of the columns and the rows, which may refer to neuronal junctions. The array may include passgates that are used to interconnect sub-blocks of synapse cells (C) within the array. For instance, NVM synapse cells108may have a transistor (T) that functions as a passgate for access to the resistive state of the N.VM synapse cells108as stored by the resistor (R) in each cell.

In various implementations, non-volatile memories (NVM) such as, e.g., FeFET or Resistive-Switching (RS) technologies, including, e.g., RRAMs, ST-RAMs or CeRAM, are used for their applicability to store data in non-conventional processor architectures in various applications. Matrix-vector multiplications for base operations in dense-algebra applications (e.g., machine-learning (ML), hyper-dimensional computing or compressive sensing) may be deployed in RS crossbars that achieve impressive improvements in energy consumption and speed. However, RS technologies may be limited in the number of states they are programed in, which may limit precision. However,FIG.1represents an NVM crossbar having N×M RS devices interconnected as shown in the neural network circuitry104. Also, depending on NVM technology, each NVM synapse cell may include a single NVM element or device that is accessible by a selector (e.g., either diode-based or transistor) that may refer to a 2-terminal cell, or a 3-4 terminal cell.

As shown inFIG.2, the neural network circuitry204refers to neural network architecture having an array of synapse cells (C) arranged in columns and rows. In some implementations, the neural network circuitry204includes input circuitry214that provides voltage (V1, V2,3, V4) to the synapse cells (C) by way of row input lines (r1, r2, r3, r4) for the rows in the array. Also, the neural network circuitry204includes output circuitry218that receives current (I1, I2, I3, I4) from the synapse cells (C) by way of column output lines (c1, c2, c3, c4) for the columns in the array. Also, in various instances, conductance (G) for the synapse cells (C) in the array is determined based on the voltage (V1, V2, V3, V4) provided by the input circuitry214and the current (I1, I2, I3, I4) received by the output circuitry218. Also, each synapse cell (C) has a corresponding conductance (G) that may be mapped based on parasitics associated with each synapse cell (C).

In some implementations, the input circuitry214may refer to digital-to-analog converter (DAC) circuitry, wherein each row (r1, r2, r3, r4) has its own corresponding DAC coupled thereto. The DAC circuitry may be configured to receive digital voltage signals (V1, V2, V3, V4) as input, convert the digital voltage signals to analog voltage signals, and then provide the analog voltage signals to the synapse cells (C) by way of the row input lines (r1, r2, r3, r4). In addition, the output circuitry218may refer to analog-to-digital (ADC) converter (DAC) circuitry, wherein each column (c1, c2, c3, c4) has its own corresponding ADC coupled thereto. Also, the ADC circuitry may be configured to receive analog current signals from the synapse cells by way of the column output lines (c1, c2, c3, c4), convert the analog current signals to digital current signals, and then provide the digital current signals as output. Also, in some instances, current (I) may be calculated based on voltage (V) multiplied by conductance (G), wherein I=V×G. Thus, based on this equation, conductance (G) for each synapse cell may be calculated as G=I/V, with voltage (V) as the inputs, current (I) as the outputs, and G as the mapped matrix.

In some implementations, the conductance (G) for the synapse cells (C) in the array is mapped based on positional orientation of each synapse cell (C) in the array. In some instances, the conductance of the synapse cells may include parasitic conductance based on the positional orientation of the synapse cells such that effective conductance is selectively tunned by adjusting a programmable weight of each synapse cell. For instance, as shown inFIG.2, the first row (r1) of synapse cells (C) has corresponding conductances, such as, e.g., G11, G12, G13, G14, wherein added row parasitics224are applied to synapse cells (C) based on location across the row (r1) such that conductance G14has the most added row parasitics224. In some instances, added column parasitics228may be applied to synapse cells (C) based on location across the columns (c1, c2, c3, c4) such that conductance G14has the most added column parasitics228.

The second row (r2) of synapse cells (C) also has corresponding conductances, such as, e.g., G21, G22, G23, G24, wherein added row parasitics224are applied to synapse cells (C) based on location across the row (r2) such that conductance increases with added row parasitics224. In some instances, added column parasitics228may be applied to synapse cells (C) based on location across the columns (c1, c2, c3, c4) such that conductance increases with added column parasitics228.

The third row (r3) of synapse cells (C) also has corresponding conductances, such as, e.g., G31, G32, G33, G34, wherein added row parasitics224are applied to synapse cells (C) based on location across the row (r3) such that conductance increases with added row parasitics224. In some instances, added column parasitics228may be applied to synapse cells (C) based on location across the columns (c1, c2, c3, c4) such that conductance increases with added column parasitics228.

The fourth row (r4) of synapse cells (C) has corresponding conductances, such as, e.g., G41, G42, G43, G44, wherein added row parasitics224are applied to synapse cells (C) based on location across the row (r4) such that conductance G41has the least added row parasitics224. In some instances, added column parasitics228may also be applied to synapse cells (C) based on location across the columns (c1, c2, c3, c4) such that conductance G41has the least added column parasitics228.

Therefore, as shown inFIG.2, the conductance (G) for the synapse cells (C) in the array may be mapped based on positional orientation of each synapse cell (C) in the array. Also, in some implementations, the conductance (G) for the synapse cells (C) in the array may be mapped at programming time based on positional orientation of each synapse cell (C) in the array.

In various implementations, each synapse cell (C) may have a programmable resistance value, and the conductance (G) for each synapse cell (C) may be calculated based on the programmable resistance value for each synapse cell (C). The conductance (G) for each synapse cell (C) may be mapped according to the programmable resistance value for each synapse cell (C). Also, the conductance (G) of the synapse cells (C) may include parasitic conductance based on the positional orientation of the synapse cells (C) in the array. Also, the parasitic conductance may be cumulative along row lengths of the row input lines (r1, r2, r3, r4) for the rows and along column lengths of the column output lines (c1, c2, c3, c4) for the columns between the input DAC circuitry214and the output (ADC) circuitry218.

In various implementations, the parasitic conductance for the synapse cells (C) may be selectively modified by adjusting a resistance value associated with the synapse cells (C). Also, accumulation of the parasitic conductance may be selectively modified by adjusting resistance values associated with the synapse cells (C) along the row lengths of the row input lines (r1, r2, r3, r4) for the rows and the column lengths of the column output lines (c1, c2, c3, c4) for the columns between the input DAC circuitry214and the output ADC circuitry218. Also, the conductance (G) of the synapse cells (C) may include parasitic conductance based on one or more characteristics associated with the synapse cells (C) including one or more of positional orientation, conductance drift, temperature and input amplitudes of the synapse cells.

In various implementations, in reference toFIGS.1-2, positional orientation of the synapse cells (C) may be defined by points of intersection for the rows (r1, r2, r3, r4) and columns (c1, c2, c3, c4) in the array. Also, the conductance (G) for the synapse cells (C) may be mapped in a manner so as to correspond to the points of intersection for the rows (r1, r2, r3, r4) and columns (c1, c2, c3, c4) in the array. Also, in various instances, the points of intersection may refer to resistive-switching (RS) crossbars, and positional orientation of the synapse cells (C) may refer to the location of the synapse cells (C) in the array at the RS crossbars.

FIGS.3-4illustrate various diagrams of neural network circuitry with synapse cells in accordance with implementations described herein. In particular,FIG.3shows a diagram300of a neural network304with 2-terminal NVM synapse cells, andFIG.4shows a diagram400of a neural network404with 3-terminal NVM synapse cells.

As shown inFIG.3, the neural network304may include a pseudo-crossbar array308with 2-terminal NVM synapse cells, including, e.g., RRAM synapse cells having a transistor (T) and a resistor (R) coupled together to form NVM synapse cells. The neural network304may have a wordline/bitline (WL/BL) switch matrix314coupled to the RRAM synapse cells (S1, S2, S3, S4) via wordlines (WL) and bitlines (BL). The neural network304may have a source line (SL) switch matrix318coupled to the RRAM synapse cells (S1, S2, S3, S4) via source lines (SL). Also, in some instances, the wordlines (WL) may be coupled to corresponding gates of transistors (T) in the synapse cells (S1, S2, S3, S4), and the transistors (T) and resistors (R) in the synapse cells (S1, S2, S3, S4) are coupled in series between corresponding source lines (SL) and bitlines (BL).

Also, in some implementations, the neural network304may include a column multiplexer (column mux)328coupled to the pseudo-crossbar array308via the source lines (SL), and the neural network304may include a mux decoder324coupled to the column mux328. Moreover, the neural network304may include a number (N) of ADC converters (332A, . . . ,332N), adders (334A, . . . ,334N) and shift registers (336A, . . . ,336N) that are coupled in the column mux328in column groups.

As shown inFIG.4, the neural network404may include a pseudo-crossbar array408with 3-terminal NVM synapse cells, e.g., ferroelectric field-effect transistor (Fe FET) synapse cells having a transistor (T1) and a FeFET transistor (T2) coupled together to form NVM synapse cells in the array408. Also, the neural network404may include a wordline/resistive switching line (WL/RS) switch matrix414coupled to FeFET synapse cells (S1, S2, S3, S4) via wordlines (WL) and resistive switching lines (RS). The neural network404may include a bitline (BL) switch matrix418coupled to FeFET synapse cells (S1, S2, S3, S4) via bitlines (BL). In some instances, the wordlines (WL) may be coupled to corresponding gates of transistors (T1) in the synapse cells (S1, S2, S3, S4), and the transistors (T1) in the synapse cells (S1, S2, S3, S4) may be coupled in series between corresponding bitlines (BL) and gates of FeFETs (T2). Also, in some instances, FeFETs (T2) may be coupled between resistive switching lines (RS) and source lines (SL) for each synapse cell (S1, S2, S3, S4) in the array408.

Also, in some implementations, the neural network404may include a column multiplexer (column mux)428coupled to the pseudo-crossbar array408via the source lines (SL), and the neural network404may include a mux decoder424coupled to the column mux428. Moreover, the neural network404may include a number (N) of ADC converters (432A, . . . ,432N), adders (434A, . . . ,434N) and shift registers (436A, . . . ,436N) that are coupled in the column mux428in column groups.

As shown inFIG.5, the neural network circuitry504may have crossbar sub-divisions or sub-block layers (L1/SB1, L2/SB2, L3/SB3) and conductance layer (G). Also, the neural network circuitry504may have DAC circuitry514for input of voltages (V1, V2, V3), and the neural network circuitry504may have ADC circuitry518for output of current (I1, I2, I3). In some instances, the neural network circuitry604may include an array with synapse cells arranged in at intersections points of columns and rows.

As shown inFIG.6, the neural network circuitry604may have crossbar sub-divisions or sub-block layers (L1/SB1, L2/SB2, L3/SB3) and conductance layer (G). Also, the neural network circuitry604may have DAC circuitry614for input of voltages (V1, V2, V3), and the neural network circuitry604may have ADC circuitry618for output of current (I1, I2, I3). In some instances, the neural network circuitry604may include an array with flying bitlines (FBL) that are used to couple the sub-blocks (L1/SB1, L2/SB2, L3/SB3) to the output ADC circuitry618so as to receive current (I1, I2, I3) from the synapse cells in the array by way of the column output lines for the columns in the array. Also, the neural network circuitry604may include passgate switches (PGS) that are used to interconnect the sub-blocks (L1/SB1, L2/SB2, L3/SB3) of the synapse cells within the array.

As shown inFIG.7, the neural network circuitry704may have crossbar sub-divisions or sub-block layers (L1/SB1, L2/SB2, L3/SB3) and conductance layer (G). Also, the neural network circuitry704may have DAC circuitry714for input of voltages (V1, V2, V3), and the neural network circuitry704may have ADC circuitry718for output of current (I1, I2, I3). In some instances, the neural network circuitry704may include an array with flying bitlines (FBL) that are used to couple the sub-blocks (L1/SB1, L2/SB2, L3/SB3) to the output ADC circuitry718so as to receive current (I1, I2, I3) from the synapse cells in the array by way of the column output lines for the columns in the array. Also, the neural network circuitry704may include passgate switches (PGS) that are used to interconnect the sub-blocks (L1/SB1, L2/SB2, L3/SB3) of the synapse cells within the array. Also, the neural network circuitry704may include an activated crossbar area724that is activated with activated passgate switches (APGS) via the flying bitlines (FBL).

As shown inFIG.8, the neural network circuitry804may have crossbar sub-divisions or sub-block layers (L1/SB1, L2/SB2, L3/SB3) and conductance layer (G). Also, the neural network circuitry804may have DAC circuitry814for input of voltages (V1, V2, V3), and the neural network circuitry804may have ADC circuitry818for output of current (I1, I2, I3). In some instances, the neural network circuitry804may include an array with flying bitlines (FBL) that are used to couple the sub-blocks (L1/SB1, L2/SB2, L3/SB3) to the output ADC circuitry818so as to receive current (I1, I2, I3) from the synapse cells in the array by way of the column output lines for the columns in the array. Also, the neural network circuitry804may include passgate switches (PGS) that are used to interconnect the sub-blocks (L1/SB1, L2/SB2, L3/SB3) of the synapse cells within the array. Also, the neural network circuitry804may include an activated crossbar area824that is activated with activated passgate switches (APGS). In this instance, the flying bitlines (FBL) may not be needed to couple the activated crossbar area824to the ADC818.

Various implementations described herein refer to neural network circuitry and post/during training weight techniques that apply various conductance mapping methods to parasitic alleviation on analog MAC (multiplication-accumulation) accelerators. Various schemes and techniques described herein may be configured to deploy high-precision matrix-vector multiplication in switching-resistor based crossbars whose devices may only be low-precision programmed and avoid higher precision ADCs and intermediate large buffers. Moreover, these techniques allow systems based on dense-algebra operations (e.g., machine learning) to be ported to more power efficient analog crossbars that may be limited by device precision and system-noise levels, and that may also need higher precision ADCs with consequent overhead in area and power consumption.

FIG.9illustrates a diagram of a method900for mapping conductance of a neural network in accordance with various implementations described herein. In various implementations, the method900refers to a target conductance estimation technique for neural networks in accordance with various implementations described herein.

It should be understood that even though method900may indicate a particular order of operation execution, in some cases, various portions of the operations may be executed in a different order, and on different systems. In other cases, other operations and/or steps may be added to and/or omitted from method900. Also, method900may be implemented in hardware and/or software. If implemented in hardware, method900may be implemented with components and/or circuitry, as described herein in reference toFIGS.1-8. Also, if implemented in software, method900may be implemented as a program and/or software instruction process configured for estimating target conductance in various schemes and techniques described herein. Also, if implemented in software, instructions related to implementing method900may be recorded in memory and/or a database. For instance, various types of computing devices having at least one processor and memory may be configured to perform method900.

In various implementations, method900may provide for a method of designing, building, fabricating and/or manufacturing neural network architecture as an integrated system, device and/or circuitry that involves use of various circuit components described herein so as to implement various neural networking schemes and techniques associated therewith. In some implementations, neural network architecture may be integrated with computing circuitry and related components on a single chip, and also, the neural network architecture may be implemented in various embedded chip-level systems for various electronic, mobile and Internet-of-things (IoT) applications.

At block910, method900may be configured to pre-train the neural network (NN). For each layer in a multi-layered neural network structure, method900may unroll weights following a desired (or predetermined) approach, such as, e.g., by following a weight-to-conductance technique. In some instances, method900may provide neural network circuitry with multiple layers such that each layer has synapse cells arranged in an array. Also, for each layer in the neural network circuitry, method900may unroll weights of each layer by following a pre-determined approach.

At block914, method900may be configured to provide for a layer-to-crossbar process. In some instances, method900may iterate through the layers in a multi-layered neural network structure so as to find (or determine and/or identify) a maximum throughput and/or a maximum utilization. In various instances, method900may iterate through the layers so as to find (or determine and/or identify) at least one of an upper boundary for throughput and an upper boundary for utilization.

At block916, method900may estimate target conductances so as to minimize deviation from the target conductance to real conductance. In various instances, method900may estimate target conductances and then find (or determine and/or identify) a lower boundary for deviation from the target conductance and the real conductance.

At decision block918, method900may be configured to evaluate one or more or all related scenarios. If yes, at block924, method900may program conductances for each synapse cell in each layer so as to minimize errors in the neural network. Otherwise, if no, at block920, method900may be configured to compute final conductance for each layer and/or each synapse cell in each layer based on synapse cell location, technology parasitics and/or state. In some instances, for each layer in the neural network, and for each synapse cell in each layer in the neural network, method900may be configured to compute a final conductance (or final target conductance) based one or more of synapse cell location, technology parasitics, and state. Also, in some instances, method900may program the target conductances so as to reduce error related to conductance.

FIGS.10A-10Billustrate process diagrams of methods for providing training operations for a neural network in accordance with implementations described herein. In particular,FIG.10Ashows a method1000A for providing a training operation for neural networks using forward/backward stages1004, including, e.g., a backward stage1004A and a forward stage1004B, and also,FIG.10Bshows a method1000B for providing the forward stage1004B of an i-th layer1004B in the neural network.

It should be understood that even though methods1000A,1000B may indicate a particular order of operation execution, in some cases, various portions of the operations may be executed in a different order, and on different systems. In other cases, operations and/or steps may be added to and/or omitted from methods1000A,1000B. Also, methods1000A,1000B may be implemented in hardware or software. If implemented in hardware, methods1000A,1000B may be implemented with various components and circuitry, as described herein in reference toFIGS.1-9. Also, if implemented in software, methods1000A,1000B may be implemented as a program and/or software instruction process that is configured for training neural networks in various schemes and techniques described herein. Further, if implemented in software, instructions related to implementing methods1000A,1000B may be recorded in memory and/or a database. For instance, some types of computing devices having at least one processor and memory may be configured to perform methods1000A,1000B shown inFIGS.10A-10B.

In various implementations, methods1000A,1000B may provide for a method of designing, building, fabricating and/or manufacturing neural network architecture as an integrated system, device and/or circuitry that involves use of various circuit components described herein so as to implement various neural networking schemes and techniques associated therewith. In various implementations, some neural network architecture may be integrated with computing circuitry and related components on a single chip, and also, neural network architecture may be implemented in various embedded chip-level systems for various electronic, mobile and Internet-of-things (IoT) applications.

As shown inFIG.10A, method1000A provides a training operation for neural networks using forward/backward stages1004, including, e.g., the backward stage1004A and the forward stage1004B. In some instances, the backward stage1004A may include one or more backward stage operations1014,1018of an i-th layer in a neural network, and further, the forward stage1004B may include one or more forward stage operations1024of an i-th layer in a neural network.

In some instances, method1000A may provide the neural network with multiple layers such that each layer has synapse cells arranged in an array. Also, method1000A may perform the backward stage1004A on the neural network, and method1000A may perform the forward stage1004B on the neural network.

At block1014, method1000A may perform the backward stage as per defined by the layer algorithm. In some instances, the backward stage may be performed without using the quantized weights, and the backward stage may be performed without using the conductance minimization error.

At block1018, if a neural architecture search (NAS) is used so as to alter layer characteristics (filter size, pruning, etc.) of the multiple layers, then method1000A may include information from a CiM structure (i.e., Compute-in-Memory including information related to area, number of operations, power consumed, throughput, et.) into the neural network, and information related to parasitics for each synapse cell in each layer may be used during layer-to-crossbar mapping so as to reduce the effect of the parasitics on the neural network. In some instances, Compute-in-Memory (CiM) may refer to process-in-Memory (PiM). Also, in some instances, if NAS is used to alter layer characteristics of the multiple layers, then weight training may be used to reduce the effect of parasitics on the neural network by including information from an equivalent CiM structure into the neural network. Therefore, in some instances, a weight training method may lead to a reduction of an effect of parasitics on the neural network by including information from an equivalent CiM structure into the neural network.

In some implementations, information related to parasitics may be used during layer-to-crossbar mapping to minimize parasitics. Also, if the parasitics degrade accuracy over a threshold, then each layer may be split so that the dynamic range is non-degraded with a corresponding reduction of throughput.

In some implementations, the forward stage1004B may provide an output (A) to block1018. For instance, the forward stage1004B may estimate target conductances so as to reduce the effect of errors by finding a lower boundary for deviation from a target conductance and a real conductance. Thus, in some instances, the output (A) may refer to the forward stage1004B providing the estimated target conductances to the backward stage1004A via the output (A) so that the backward stage1004A is performed based on the target conductances, as provided by the forward stage1004B.

As shown inFIG.10B, method1000B may provide the forward stage1004B of an i-th layer1004B in the neural network. In some implementations, the forward stage1000B is configured to perform the forward stage operations1024, and also, the forward stage1004B is configured to provide the output (A) from block1064.

At block1050, method1000B may be configured to provide layer-to-crossbar processing for the neural network. In some instances, method1000B may iterate through the layers in the multi-layered neural network structure so as to find (or determine and/or identify) maximum throughput and/or maximum utilization. In various instances, method1000B may iterate through the layers so as to find (or determine and/or identify) at least one of an upper boundary for throughput and an upper boundary for utilization.

At block1054, method1000B estimates target conductances so as to minimize deviation from the target conductance to real conductance. In various instances, method1000B may estimate target conductances and so as to find (or determine and/or identify) a lower boundary for deviation from target conductance and real conductance.

At decision block1058, method1000B may be configured to evaluate one or more related scenarios. If yes, at block1064, method1000B estimate conductances for each synapse cell in each layer so as to minimize errors in the neural network. Also, from block1064, the method1000B may use the forward stage1004B to provide output (A) to block1018inFIG.10A. For instance, the forward stage1004B may estimate the target conductances so as to reduce the effect of errors by finding a lower boundary for deviation from a target conductance and a real conductance. Thus, in some instances, the output (A) may refer to the forward stage1004B providing the estimated target conductances to backward stage1004A via the output (A) so that the backward stage1004A is performed based on the target conductances, as provided by the forward stage1004B.

Otherwise, if no, at block1060, method1000B may be configured to compute final conductance for each layer and/or each synapse cell in each layer based on synapse cell location, technology parasitics and/or state. Also, for each layer in the neural network, and for each synapse cell in each layer in the neural network, method1000B is configured to compute final conductance (or final target conductance) based one or more of synapse cell location, technology parasitics, and state. Also, method1000B may program target conductances so as to reduce error related to conductance.

In some implementations, at block1064, method1000B may compute one or more layers with optimized conductances and then provide this information to block1068, wherein at block1068, method1000B performs layer MACS based on real domain input activations or CiM domain activations. Next, at block1070, method1000B may compute bias/activations and CiM output activations to real domain activations. Also, from block1070, method1000B may provide the forward stage output based on bias/activations and CiM output activations to real domain activations.

It should be intended that the subject matter of the claims not be limited to the implementations and illustrations provided herein, but include modified forms of those implementations including portions of implementations and combinations of elements of different implementations in accordance with the claims. It should be appreciated that in the development of any such implementation, as in any engineering or design project, numerous implementation-specific decisions should be made to achieve developers' specific goals, such as compliance with system-related and business related constraints, which may vary from one implementation to another. Moreover, it should be appreciated that such a development effort may be complex and time consuming, but would nevertheless be a routine undertaking of design, fabrication, and manufacture for those of ordinary skill having benefit of this disclosure.

Described herein are various implementations of a device with neural network circuitry. The neural network circuitry may include an array of synapse cells arranged in columns and rows. The device may include input circuitry that provides voltage to the synapse cells by way of row input lines for the rows in the array. The device may include output circuitry that receives current from the synapse cells by way of column output lines for the columns in the array. Also, conductance for the synapse cells in the array may be determined based on the voltage provided by the input circuitry and the current received by the output circuitry.

Described herein are various implementations of a method. The method may provide a neural network with multiple layers such that each layer has synapse cells arranged in an array. For each layer in the neural network, the method may unroll weights of the layers following a pre-determined approach. The method may iterate through the layers to find at least one of an upper boundary for throughput and an upper boundary for utilization. The method may estimate target conductances and finding a lower boundary for deviation from a target conductance and a real conductance.

Described herein are various implementations of a method. The method may provide a neural network with multiple layers such that each layer has synapse cells arranged in an array. The method may perform a backward stage on the neural network, and the method may also perform a forward stage on the neural network. If a neural architecture search (NAS) is used to alter layer characteristics of the multiple layers, then the method may use weight training to provide a reduction of an effect of parasitics on the neural network by including information from a compute-in-memory (CiM) structure into the neural network.

Reference has been made in detail to various implementations, examples of which are illustrated in the accompanying drawings and figures. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the disclosure provided herein. However, the disclosure provided herein may be practiced without these specific details. In some other instances, well-known methods, procedures, components, circuits and networks have not been described in detail so as not to unnecessarily obscure details of the embodiments.

While the foregoing is directed to implementations of various techniques described herein, other and further implementations may be devised in accordance with the disclosure herein, which may be determined by the claims that follow.