Counter based multiply-and-accumulate circuit for neural network

Disclosed herein includes a system, a method, and a device for improving computation efficiency of a neural network. In one aspect, adder circuitry is configured to add input data from processing of the neural network and a first number of bits of accumulated data for the neural network to generate summation data. In one aspect, according to a carry value of the adding from the adder circuitry, a multiplexer is configured to select between i) a second number of bits of the accumulated data and ii) incremented data comprising the second number of bits of the accumulated data incremented by a predetermined value. The summation data appended with the selected one of the second number of bits of the accumulated data or the incremented data may form appended data.

FIELD OF DISCLOSURE

The present disclosure is generally related to a processing mechanism for a neural network, including but not limited to a multiply and accumulate circuit for the neural network using a counter or an incrementor.

BACKGROUND

Recent development in neural networks allows various artificial intelligence (AI) tasks such as computer vision, speech recognition, machine translation, social network filtering, playing board and video games, and medical diagnosis, etc., to be automatically performed. In one implementation, a neural network includes a network of nodes that model neurons in a human brain. Nodes may perform computations such as multiply-accumulate operations on their inputs, and provide, transmit or forward computation results to their subsequent nodes. The subsequent nodes then may perform computations such as multiply-accumulate operations on the computation results from their preceding nodes as their inputs, and provide, transmit, or forward additional computation results to their subsequent nodes. According to the computations through a large network of nodes (e.g., including more than thousands or more than millions), AI tasks may be performed. However, performing multiply-accumulate operations for a large number of nodes may be computationally expensive and may delay the AI tasks.

SUMMARY

Various embodiments disclosed herein are related to a multiply and accumulate circuit for a neural network. In some embodiments, the multiply and accumulate circuit includes adder circuitry configured to generate summation data by adding input data from processing of the neural network and a first number of bits of accumulated data for the neural network. In some embodiments, the multiply and accumulate circuit includes a multiplexer configured to select, according to a carry value of the adding by the adder circuitry, between i) a second number of bits of the accumulated data and ii) incremented data comprising the second number of bits of the accumulated data incremented by a predetermined value. In some embodiments, the summation data appended with the selected one of the second number of bits of the accumulated data or the incremented data form appended data.

In some embodiments, the multiplexer is configured to select the second number of bits of the accumulated data, in response to the carry value of the adding being zero. In some embodiments, the multiplexer is configured to select the incremented data, in response to the carry value of the adding being one. In some embodiments, the adder circuitry is configured to provide the first number of bits of the appended data to an input of the adder circuitry for a subsequent clock cycle. In some embodiments, the multiplexer is configured to provide the second number of bits of the appended data to an input of the multiplexer for a subsequent clock cycle. In some embodiments, the multiply and accumulate circuit includes incrementing circuitry configured to increment the second number of bits of the appended data by one to generate the incremented data and provide the incremented data to an additional input of the multiplexer. In some embodiments, the multiplexer includes a control input coupled to a carry output of the adder circuitry and configured to receive the carry value of the adding from the carry output of the adder circuitry.

In some embodiments, the multiply and accumulate circuit includes multiplier circuitry including an output coupled to an input of the adder circuitry. In some embodiments, the multiplier circuitry is configured to multiply first data and second data to generate the input data. In some embodiments, the first data is weight of the neural network and the second data is an activation value of the neural network. In some embodiments, the first data has a third number of bits, and the second data has the third number of bits, where the first number is twice of the third number.

Various embodiments disclosed herein are related to a method for a neural network. In some embodiments, the method includes adding, by adder circuitry, input data from processing of the neural network and a first number of bits of accumulated data for the neural network to generate summation data. In some embodiments, the method includes selecting, by a multiplexer, according to a carry value of the adding from the adder circuitry, between i) a second number of bits of the accumulated data and ii) incremented data comprising the second number of bits of the accumulated data incremented by a predetermined value. In some embodiments, the summation data appended with the selected one of the second number of bits of the accumulated data or the incremented data form appended data.

In some embodiments, the selecting includes selecting, by the multiplexer, the second number of bits of the accumulated data, in response to the carry value of the adding being zero. In some embodiments, the selecting includes selecting, by the multiplexer, the incremented data, in response to the carry value of the adding being one. In some embodiments, the method includes providing, by the adder circuitry, the first number of bits of the appended data from an output of the adder circuitry to an input of the adder circuitry for a subsequent clock cycle. In some embodiments, the method includes providing, by the multiplexer, the second number of bits of the appended data from an output of the multiplexer to an input of the multiplexer for a subsequent clock cycle. In some embodiments, the method includes incrementing, by incrementing circuitry, the second number of bits of the accumulated data from the output of the multiplexer by one to generate the incremented data. In some embodiments, the method includes providing, by the incrementing circuitry, the incremented data to an additional input of the multiplexer. In some embodiments, the method includes receiving, at a control input of the multiplexer, the carry value of the adding from a carry output of the adder circuitry.

In some embodiments, the method includes multiplying, by multiplier circuitry, first data and second data to generate the input data. In some embodiments, the first data is weight of the neural network and the second data is an activation value of the neural network. In some embodiments, the first data has a third number of bits, and the second data has the third number of bits, where the first number is twice of the third number.

DETAILED DESCRIPTION

For purposes of reading the description of the various embodiments of the present invention below, the following descriptions of the sections of the specification and their respective contents may be helpful:Section A describes an environment, system, configuration and/or other aspects useful for practicing or implementing an embodiment of the present systems, methods and devices; andSection B describes embodiments of devices, systems and methods for multiply-accumulate operations for neural networks based on a counter or an incrementor.
A. Environment for Artificial Intelligence Related Processing

Prior to discussing the specifics of embodiments of systems, devices and/or methods in Section B, it may be helpful to discuss the environments, systems, configurations and/or other aspects useful for practicing or implementing certain embodiments of the systems, devices and/or methods. Referring now toFIG. 1A, an embodiment of a system for performing artificial intelligence (AI) related processing is depicted. In brief overview, the system includes one or more AI accelerators108that can perform AI related processing using input data110. Although referenced as an AI accelerator108, it is sometimes referred as a neural network accelerator (NNA), neural network chip or hardware, AI processor, AI chip, etc. The AI accelerator(s)108can perform AI related processing to output or provide output data112, according to the input data110and/or parameters128(e.g., weight and/or bias information). An AI accelerator108can include and/or implement one or more neural networks114(e.g., artificial neural networks), one or more processor(s)24and/or one or more storage devices126.

Each of the above-mentioned elements or components is implemented in hardware, or a combination of hardware and software. For instance, each of these elements or components can include any application, program, library, script, task, service, process or any type and form of executable instructions executing on hardware such as circuitry that can include digital and/or analog elements (e.g., one or more transistors, logic gates, registers, memory devices, resistive elements, conductive elements, capacitive elements).

The input data110can include any type or form of data for configuring, tuning, training and/or activating a neural network114of the AI accelerator(s)108, and/or for processing by the processor(s)124. The neural network114is sometimes referred to as an artificial neural network (ANN). Configuring, tuning and/or training a neural network can refer to or include a process of machine learning in which training data sets (e.g., as the input data110) such as historical data are provided to the neural network for processing. Tuning or configuring can refer to or include training or processing of the neural network114to allow the neural network to improve accuracy. Tuning or configuring the neural network114can include, for example, designing, forming, building, synthesizing and/or establishing the neural network using architectures that have proven to be successful for the type of problem or objective desired for the neural network114. In some cases, the one or more neural networks114may initiate at a same or similar baseline model, but during the tuning, training or learning process, the results of the neural networks114can be sufficiently different such that each neural network114can be tuned to process a specific type of input and generate a specific type of output with a higher level of accuracy and reliability as compared to a different neural network that is either at the baseline model or tuned or trained for a different objective or purpose. Tuning the neural network114can include setting different parameters128for each neural network114, fine-tuning the parameters128differently for each neural network114, or assigning different weights (e.g., hyperparameters, or learning rates), tensor flows, etc. Thus, setting appropriate parameters128for the neural network(s)114based on a tuning or training process and the objective of the neural network(s) and/or the system, can improve performance of the overall system.

A neural network114of the AI accelerator108can include any type of neural network including, for example, a convolution neural network (CNN), deep convolution network, a feed forward neural network (e.g., multilayer perceptron (MLP)), a deep feed forward neural network, a radial basis function neural network, a Kohonen self-organizing neural network, a recurrent neural network, a modular neural network, a long/short term memory neural network, etc. The neural network(s)114can be deployed or used to perform data (e.g., image, audio, video) processing, object or feature recognition, recommender functions, data or image classification, data (e.g., image) analysis, etc., such as natural language processing.

As an example, and in one or more embodiments, the neural network114can be configured as or include a convolution neural network. The convolution neural network can include one or more convolution cells (or pooling layers) and kernels, that can each serve a different purpose. The convolution neural network can include, incorporate and/or use a convolution kernel (sometimes simply referred as “kernel”). The convolution kernel can process input data, and the pooling layers can simplify the data, using, for example, non-linear functions such as a max, thereby reducing unnecessary features. The neural network114including the convolution neural network can facilitate image, audio or any data recognition or other processing. For example, the input data110(e.g., from a sensor) can be passed to convolution layers of the convolution neural network that form a funnel, compressing detected features in the input data110. The first layer of the convolution neural network can detect first characteristics, the second layer can detect second characteristics, and so on.

The convolution neural network can be a type of deep, feed-forward artificial neural network configured to analyze visual imagery, audio information, and/or any other type or form of input data110. The convolution neural network can include multilayer perceptrons designed to use minimal preprocessing. The convolution neural network can include or be referred to as shift invariant or space invariant artificial neural networks, based on their shared-weights architecture and translation invariance characteristics. Since convolution neural networks can use relatively less pre-processing compared to other data classification/processing algorithms, the convolution neural network can automatically learn the filters that may be hand-engineered for other data classification/processing algorithms, thereby improving the efficiency associated with configuring, establishing or setting up the neural network114, thereby providing a technical advantage relative to other data classification/processing techniques.

The neural network114can include an input layer116and an output layer122, of neurons or nodes. The neural network114can also have one or more hidden layers118,119that can include convolution layers, pooling layers, fully connected layers, and/or normalization layers, of neurons or nodes. In a neural network114, each neuron can receive input from some number of locations in the previous layer. In a fully connected layer, each neuron can receive input from every element of the previous layer.

Each neuron in a neural network114can compute an output value by applying some function to the input values coming from the receptive field in the previous layer. The function that is applied to the input values is specified by a vector of weights and a bias (typically real numbers). Learning (e.g., during a training phase) in a neural network114can progress by making incremental adjustments to the biases and/or weights. The vector of weights and the bias can be called a filter and can represent some feature of the input (e.g., a particular shape). A distinguishing feature of convolutional neural networks is that many neurons can share the same filter. This reduces memory footprint because a single bias and a single vector of weights can be used across all receptive fields sharing that filter, rather than each receptive field having its own bias and vector of weights.

For example, in a convolution layer, the system can apply a convolution operation to the input layer116, passing the result to the next layer. The convolution emulates the response of an individual neuron to input stimuli. Each convolutional neuron can process data only for its receptive field. Using the convolution operation can reduce the number of neurons used in the neural network114as compared to a fully connected feedforward neural network. Thus, the convolution operation can reduce the number of free parameters, allowing the network to be deeper with fewer parameters. For example, regardless of an input data (e.g., image data) size, tiling regions of size 5×5, each with the same shared weights, may use only 25 learnable parameters. In this way, the first neural network114with a convolution neural network can resolve the vanishing or exploding gradients problem in training traditional multi-layer neural networks with many layers by using backpropagation.

The neural network114(e.g., configured with a convolution neural network) can include one or more pooling layers. The one or more pooling layers can include local pooling layers or global pooling layers. The pooling layers can combine the outputs of neuron clusters at one layer into a single neuron in the next layer. For example, max pooling can use the maximum value from each of a cluster of neurons at the prior layer. Another example is average pooling, which can use the average value from each of a cluster of neurons at the prior layer.

The neural network114(e.g., configured with a convolution neural network) can include fully connected layers. Fully connected layers can connect every neuron in one layer to every neuron in another layer. The neural network114can be configured with shared weights in convolutional layers, which can refer to the same filter being used for each receptive field in the layer, thereby reducing a memory footprint and improving performance of the first neural network114.

The hidden layers118,119can include filters that are tuned or configured to detect information based on the input data (e.g., sensor data, from a virtual reality system for instance). As the system steps through each layer in the neural network114(e.g., convolution neural network), the system can translate the input from a first layer and output the transformed input to a second layer, and so on. The neural network114can include one or more hidden layers118,119based on the type of object or information being detected, processed and/or computed, and the type of input data110.

In some embodiments, the convolutional layer is the core building block of a neural network114(e.g., configured as a CNN). The layer's parameters128can include a set of learnable filters (or kernels), which have a small receptive field, but extend through the full depth of the input volume. During the forward pass, each filter is convolved across the width and height of the input volume, computing the dot product between the entries of the filter and the input and producing a 2-dimensional activation map of that filter. As a result, the neural network114can learn filters that activate when it detects some specific type of feature at some spatial position in the input. Stacking the activation maps for all filters along the depth dimension forms the full output volume of the convolution layer. Every entry in the output volume can thus also be interpreted as an output of a neuron that looks at a small region in the input and shares parameters with neurons in the same activation map. In a convolutional layer, neurons can receive input from a restricted subarea of the previous layer. Typically, the subarea is of a square shape (e.g., size 5 by 5). The input area of a neuron is called its receptive field. So, in a fully connected layer, the receptive field is the entire previous layer. In a convolutional layer, the receptive area can be smaller than the entire previous layer.

The first neural network114can be trained to detect, classify, segment and/or translate input data110(e.g., by detecting or determining the probabilities of objects, events, words and/or other features, based on the input data110). For example, the first input layer116of neural network114can receive the input data110, process the input data110to transform the data to a first intermediate output, and forward the first intermediate output to a first hidden layer118. The first hidden layer118can receive the first intermediate output, process the first intermediate output to transform the first intermediate output to a second intermediate output, and forward the second intermediate output to a second hidden layer119. The second hidden layer119can receive the second intermediate output, process the second intermediate output to transform the second intermediate output to a third intermediate output, and forward the third intermediate output to an output layer122for example. The output layer122can receive the third intermediate output, process the third intermediate output to transform the third intermediate output to output data112, and forward the output data112(e.g., possibly to a post-processing engine, for rendering to a user, for storage, and so on). The output data112can include object detection data, enhanced/translated/augmented data, a recommendation, a classification, and/or segmented data, as examples.

Referring again toFIG. 1A, the AI accelerator108can include one or more storage devices126. A storage device126can be designed or implemented to store, hold or maintain any type or form of data associated with the AI accelerator(s)108. For example, the data can include the input data110that is received by the AI accelerator(s)108, and/or the output data112(e.g., before being output to a next device or processing stage). The data can include intermediate data used for, or from any of the processing stages of a neural network(s)114and/or the processor(s)124. The data can include one or more operands for input to and processing at a neuron of the neural network(s)114, which can be read or accessed from the storage device126. For example, the data can include input data, weight information and/or bias information, activation function information, and/or parameters128for one or more neurons (or nodes) and/or layers of the neural network(s)114, which can be stored in and read or accessed from the storage device126. The data can include output data from a neuron of the neural network(s)114, which can be written to and stored at the storage device126. For example, the data can include activation data, refined or updated data (e.g., weight information and/or bias information from a training phase for example, activation function information, and/or other parameters128) for one or more neurons (or nodes) and/or layers of the neural network(s)114, which can be transferred or written to, and stored in the storage device126.

In some embodiments, the AI accelerator108can include one or more processors124. The one or more processors124can include any logic, circuitry and/or processing component (e.g., a microprocessor) for pre-processing input data for any one or more of the neural network(s)114or AI accelerator(s)108, and/or for post-processing output data for any one or more of the neural network(s)114or AI accelerator(s)108. The one or more processors124can provide logic, circuitry, processing component and/or functionality for configuring, controlling and/or managing one or more operations of the neural network(s)114or AI accelerator(s)108. For instance, a processor124may receive data or signals associated with a neural network114to control or reduce power consumption (e.g., via clock-gating controls on circuitry implementing operations of the neural network114). As another example, a processor124may partition and/or re-arrange data for separate processing (e.g., at various components of an AI accelerator108, in parallel for example), sequential processing (e.g., on the same component of an AI accelerator108, at different times or stages), or for storage in different memory slices of a storage device, or in different storage devices. In some embodiments, the processor(s)124can configure a neural network114to operate for a particular context, provide a certain type of processing, and/or to address a specific type of input data, e.g., by identifying, selecting and/or loading specific weight, activation function and/or parameter information to neurons and/or layers of the neural network114.

In some embodiments, the AI accelerator108is designed and/or implemented to handle or process deep learning and/or AI workloads. For example, the AI accelerator108can provide hardware acceleration for artificial intelligence applications, including artificial neural networks, machine vision and machine learning. The AI accelerator108can be configured for operation to handle robotics related, internet of things (IoT) related, and other data-intensive or sensor-driven tasks. The AI accelerator108may include a multi-core or multiple processing element (PE) design, and can be incorporated into various types and forms of devices such as artificial reality (e.g., virtual, augmented or mixed reality) systems, smartphones, tablets, and computers. Certain embodiments of the AI accelerator108can include or be implemented using at least one digital signal processor (DSP), co-processor, microprocessor, computer system, heterogeneous computing configuration of processors, graphics processing unit (GPU), field-programmable gate array (FPGA), and/or application-specific integrated circuit (ASIC). The AI accelerator108can be a transistor based, semiconductor based and/or a quantum computing based device.

Referring now toFIG. 1B, an example embodiment of a device for performing AI related processing is depicted. In brief overview, the device can include or correspond to an AI accelerator108, e.g., with one or more features described above in connection withFIG. 1A. The AI accelerator108can include one or more storage devices126(e.g., memory such as a static random-access memory (SRAM) device), one or more buffers, a plurality or array of processing element (PE) circuits, other logic or circuitry (e.g., adder circuitry), and/or other structures or constructs (e.g., interconnects, data buses, clock circuitry, power network(s)). Each of the above-mentioned elements or components is implemented in hardware, or at least a combination of hardware and software. The hardware can for instance include circuit elements (e.g., one or more transistors, logic gates, registers, memory devices, resistive elements, conductive elements, capacitive elements, and/or wire or electrically conductive connectors).

In a neural network114(e.g., artificial neural network) implemented in the AI accelerator108, neurons can take various forms and can be referred to as processing elements (PEs) or PE circuits. The neuron can be implemented as a corresponding PE circuit, and the processing/activation that can occur at the neuron can be performed at the PE circuit. The PEs are connected into a particular network pattern or array, with different patterns serving different functional purposes. The PE in an artificial neural network operate electrically (e.g., in the embodiment of a semiconductor implementation), and may be either analog, digital, or a hybrid. To parallel the effect of a biological synapse, the connections between PEs can be assigned multiplicative weights, which can be calibrated or “trained” to produce the proper system output.

A PE can be defined in terms of the following equations (e.g., which represent a McCulloch-Pitts model of a neuron):
ζ=Σiwixi(1)
y=σ(ζ)  (2)
Where ζ is the weighted sum of the inputs (e.g., the inner product of the input vector and the tap-weight vector), and σ(ζ) is a function of the weighted sum. Where the weight and input elements form vectors w and x, the ζ weighted sum becomes a simple dot product:
ζ=w·x(3)

This may be referred to as either the activation function (e.g., in the case of a threshold comparison) or a transfer function. In some embodiments, one or more PEs can be referred to as a dot product engine. The input (e.g., input data110) to the neural network114, x, can come from an input space and the output (e.g., output data112) are part of the output space. For some neural networks, the output space Y may be as simple as {0, 1}, or it may be a complex multi-dimensional (e.g., multiple channel) space (e.g., for a convolutional neural network). Neural networks tend to have one input per degree of freedom in the input space, and one output per degree of freedom in the output space.

In some embodiments, the PEs can be arranged and/or implemented as a systolic array. A systolic array can be a network (e.g., a homogeneous network) of coupled data processing units (DPUs) such as PEs, called cells or nodes. Each node or PE can independently compute a partial result as a function of the data received from its upstream neighbors, can store the result within itself and can pass the result downstream for instance. The systolic array can be hardwired or software configured for a specific application. The nodes or PEs can be fixed and identical, and interconnect of the systolic array can be programmable. Systolic arrays can rely on synchronous data transfers.

Referring again toFIG. 1B, the input x to a PE120can be part of an input stream132that is read or accessed from a storage device126(e.g., SRAM). An input stream132can be directed to one row (horizontal bank or group) of PEs, and can be shared across one or more of the PEs, or partitioned into data portions (overlapping or non-overlapping data portions) as inputs for respective PEs. Weights134(or weight information) in a weight stream (e.g., read from the storage device126) can be directed or provided to a column (vertical bank or group) of PEs. Each of the PEs in the column may share the same weight134or receive a corresponding weight134. The input and/or weight for each target PE can be directly routed (e.g., from the storage device126) to the target PE (e.g., without passing through other PE(s)), or can be routed through one or more PEs (e.g., along a row or column of PEs) to the target PE. The output of each PE can be routed directly out of the PE array (e.g., without passing through other PE(s)), or can be routed through one or more PEs (e.g., along a column of PEs) to exit the PE array. The outputs of each column of PEs can be summed or added at an adder circuitry of the respective column, and provided to a buffer130for the respective column of PEs. The buffer(s)130can provide, transfer, route, write and/or store the received outputs to the storage device126. In some embodiments, the outputs (e.g., activation data from one layer of the neural network) that are stored by the storage device126can be retrieved or read from the storage device126, and be used as inputs to the array of PEs120for processing (of a subsequent layer of the neural network) at a later time. In certain embodiments, the outputs that are stored by the storage device126can be retrieved or read from the storage device126as output data112for the AI accelerator108.

Referring now toFIG. 1C, one example embodiment of a device for performing AI related processing is depicted. In brief overview, the device can include or correspond to an AI accelerator108, e.g., with one or more features described above in connection withFIGS. 1A and 1B. The AI accelerator108can include one or more PEs120, other logic or circuitry (e.g., adder circuitry), and/or other structures or constructs (e.g., interconnects, data buses, clock circuitry, power network(s)). Each of the above-mentioned elements or components is implemented in hardware, or at least a combination of hardware and software. The hardware can for instance include circuit elements (e.g., one or more transistors, logic gates, registers, memory devices, resistive elements, conductive elements, capacitive elements, and/or wire or electrically conductive connectors).

In some embodiments, a PE120can include one or more multiply-accumulate (MAC) units or circuits140. One or more PEs can sometimes be referred to (singly or collectively) as a MAC engine. A MAC unit is configured to perform multiply-accumulate operation(s). The MAC unit can include a multiplier circuit, an adder circuit and/or an accumulator circuit. The multiply-accumulate operation computes the product of two numbers and adds that product to an accumulator. The MAC operation can be represented as follows, in connection with an accumulator operand a, and inputs b and c:
a←a+(b×c)  (4)
In some embodiments, a MAC unit140may include a multiplier implemented in combinational logic followed by an adder (e.g., that includes combinational logic) and an accumulator register (e.g., that includes sequential and/or combinational logic) that stores the result. The output of the accumulator register can be fed back to one input of the adder, so that on each clock cycle, the output of the multiplier can be added to the accumulator register.

As discussed above, a MAC unit140can perform both multiply and addition functions. The MAC unit140can operate in two stages. The MAC unit140can first compute the product of given numbers (inputs) in a first stage, and forward the result for the second stage operation (e.g., addition and/or accumulate). An n-bit MAC unit140can include an n-bit multiplier, 2n-bit adder, and 2n-bit accumulator. An array or plurality of MAC units140(e.g., in PEs) can be arranged in a systolic array, for parallel integration, convolution, correlation, matrix multiplication, data sorting, and/or data analysis tasks.

Various systems and/or devices described herein can be implemented in a computing system.FIG. 1Dshows a block diagram of a representative computing system150. In some embodiments, the system ofFIG. 1Acan form at least part of the processing unit(s)156(or processors156) of the computing system150. Computing system150can be implemented, for example, as a device (e.g., consumer device) such as a smartphone, other mobile phone, tablet computer, wearable computing device (e.g., smart watch, eyeglasses, head mounted display), desktop computer, laptop computer, or implemented with distributed computing devices. The computing system150can be implemented to provide VR, AR, MR experience. In some embodiments, the computing system150can include conventional, specialized or custom computer components such as processors156, storage device158, network interface151, user input device152, and user output device154.

Network interface151can provide a connection to a local/wide area network (e.g., the Internet) to which network interface of a (local/remote) server or back-end system is also connected. Network interface151can include a wired interface (e.g., Ethernet) and/or a wireless interface implementing various RF data communication standards such as Wi-Fi, Bluetooth, or cellular data network standards (e.g., 3G, 4G, 5G, LTE, etc.).

User input device152can include any device (or devices) via which a user can provide signals to computing system150; computing system150can interpret the signals as indicative of particular user requests or information. User input device152can include any or all of a keyboard, touch pad, touch screen, mouse or other pointing device, scroll wheel, click wheel, dial, button, switch, keypad, microphone, sensors (e.g., a motion sensor, an eye tracking sensor, etc.), and so on.

User output device154can include any device via which computing system150can provide information to a user. For example, user output device154can include a display to display images generated by or delivered to computing system150. The display can incorporate various image generation technologies, e.g., a liquid crystal display (LCD), light-emitting diode (LED) including organic light-emitting diodes (OLED), projection system, cathode ray tube (CRT), or the like, together with supporting electronics (e.g., digital-to-analog or analog-to-digital converters, signal processors, or the like). A device such as a touchscreen that function as both input and output device can be used. User output devices154can be provided in addition to or instead of a display. Examples include indicator lights, speakers, tactile “display” devices, printers, and so on.

B. Methods and Devices for Multiply-Accumulate Operations for Neural Networks

Disclosed herein include embodiments of a system, a method, and a device for multiply-accumulate operations for neural networks based on a counter or an incrementor. In some aspects, this disclosure is directed to improving efficiency in accumulation operation of a multiply-accumulate (MAC) unit in a neural network, implemented in an AI accelerator for instance. A MAC unit may perform multiplications of weights with corresponding activations and accumulation of different multiplication results. Accumulation of different multiplication results may involve a large number of bits, and may be computationally exhaustive and power inefficient. In one aspect, an output of an accumulation adder can be divided into two portions, a least significant bit (LSB) portion and a most significant bit (MSB) portion. The LSB portion can be added with a multiplication result. Meanwhile, a multiplexer can receive the MSB portion and a summation of the MSB portion incremented by one (e.g., 1 bit, or some other value) for example. Depending on the carry output of the sum of the LSB portion and the multiplication result (which can be provided as an overflow indication or overflow bit of an accumulation adder that performs the summation), the multiplexer can output either the MSB portion or the summation of the MSB portion incremented by one. Then, the output of the multiplexer can be combined with the sum of the LSB portion and the multiplication result as the output of the accumulation adder. In some embodiments, multiple possible increment values for summation with the MSB portion can be implemented to provide additional inputs to the multiplexer for selection according to the carry output (which can have three or more possible values for instance).

Advantageously, such an accumulation adder can improve operation speed and reduce latency. For example, accumulation by a conventional approach involves propagating a carry through a large number of bits. Propagating a carry through a large number of bits may delay the process of accumulation. By performing accumulation only on the LSB portion for instance, and selecting either i) the MSB portion or ii) the MSB portion incremented by one, according to the carry output from the accumulation of LSB portion, a carry can be propagated through a fewer number of bits than in a conventional approach. Hence, a critical path for performing accumulation operation can be reduced by propagating a carry through a fewer number of bits.

In one aspect, the input data is a multiplication result from multiplying weight with a corresponding activation value or an activation function. Appended data may be applied as accumulated data for a next clock cycle. Hence, in one embodiment, a multiplication result from multiplying weight with a corresponding activation value or an activation function can be added for a clock cycle, and additional multiplication results for multiplying another weight with a corresponding activation value or an activation function can be added for a subsequent clock cycle.

Referring toFIG. 2, illustrated is a block diagram of a multiply-accumulate (MAC) unit140ofFIG. 1C, according to an example implementation of the present disclosure. In some embodiments, the MAC unit140includes multiplier circuitries210A,210B,210C . . .210N, adder circuitry215, an accumulator220, and a register230. These components operate together to receive weights205A,205B . . .205N and activation values208A,208B . . .208N to generate appended data225(also referred to as “updated accumulated data225”), for example, as described above with respect to Eq. (4). In other embodiments, the MAC unit140includes more, different, or fewer components than shown inFIG. 2. For example and in some embodiments, the adder circuitry215may be omitted, and the outputs of the multiplier circuitries210may be provided to the accumulator220as input data.

In one configuration, each multiplier circuitry210X receives a weight205X and an activation value208X and multiplies the weight205X and the activation value208X to generate a multiplication result212X, which is provided to the adder circuitry215. The adder circuitry215may add the multiplication results from the multiplier circuitries210A-210N to generate a summation data218, and provide the summation data218to the accumulator220as an input data. In one configuration, the accumulator220receives the summation data218from the adder circuitry215and accumulated data235from the register230to generate appended data (or updated accumulated data225) and provide the appended data (or the updated accumulated data225) to the register230.

In one approach, the multiply-accumulation operation is performed for each clock cycle. For example, the multiplier circuitries210A-210N may receive a set of weights205A-205N and a corresponding set of activation values208A-208N during a clock cycle, and multiply the set of weights205A-205N with the corresponding set of activation values208A-208N to generate a set of multiplication results212A-212N during the clock cycle. The adder circuitry215may add the set of multiplication results from the multiplier circuitries210A-210N to generate summation data218, and provide the summation data218to the accumulator220during the clock cycle. The accumulator220may add the summation data218with the accumulated data235to generate the updated accumulated data225, and provide the updated accumulated data225to the register230. The register230may output the updated accumulated data225received during the clock cycle, as the accumulated data235during the subsequent clock cycle. The multiplier circuitries210A-210N, the adder circuitry215, the accumulator220, and the register230may repeat the process for an additional set of weights and a corresponding set of activation values208during the subsequent clock cycle. Assuming for instance that inputs of a neuron or a node is coupled to outputs of 50 neurons or nodes, the MAC unit140may employ 10 multiplier circuitries210(e.g., N is 10) that can perform computations in parallel for a clock cycle, such that the MAC unit140can perform multiply-accumulation operations of 50 weights and 50 activation values in 5 clock cycles.

In one implementation, accumulation of a large number of bits by the accumulator220may be computationally inefficient. Assuming for an example that the weight205has 8 bits and the activation value has 8 bits, then the multiplication result212has 16 bits. For N number of additions, the summation data218has a K number of bits corresponding to ceil(16+log 2N), where ceil( ) is a ceiling function. Moreover, accumulating the summation data218renders the accumulated data235to have a J number of bits (e.g., 28 bits) larger than the K number of bits. In one implementation, the accumulator220includes J number of adders to add the summation data218having a fixed K number of bits to the K number of bits of an LSB portion of the accumulated data, and propagate a carry through the remaining bits of the accumulated data. However, performing addition through the J number of adders may be slow and consume a large amount of power.

Advantageously, the disclosed accumulator220improves computational speed and efficiency by segregating the accumulated data into two portions, where the first portion is added with the input data, and either i) the second portion or ii) the second portion added by a predetermined value (e.g., ‘1’) is selected according to a carry value from the addition. By obviating addition or accumulation for a larger number of bits, a critical path for the accumulation can be reduced to improve operation speed or improve power efficiency, as described below with respect toFIGS. 3A and 3Bbelow.

Referring toFIG. 3A, illustrated is a block diagram of the accumulator220A ofFIG. 2, according to an example implementation of the present disclosure. In some embodiments, the accumulator220A includes a K bit adder circuitry310, a J-K bit incrementor350, and a J-K bit multiplexer320. J-K can refer to a difference between the value of J and the value of K. These components can operate together to receive the summation data218or the multiplication result212as input data, and accumulated data235, and can generate updated accumulated data225equivalent to a sum of the summation data218and the accumulated data235. In some example embodiments, J is 28 and K is between 16 and 20. In other embodiments, the accumulator220A includes more, fewer, or different components than shown inFIG. 3A.

The K bit adder circuitry310can be or include a component that receives a first portion of the accumulated data235A and input data218, and can add the first portion of the accumulated data235A with the input data218to generate a first portion of updated accumulated data225A. For example, the K bit adder circuitry310receives K bits of a LSB portion of the accumulated data235A from the register230and input data218having K bits from the adder circuitry215to generate K bits of a LSB portion of the updated accumulated data225A. The K bit adder circuitry310may provide the K bits of the LSB portion of the updated accumulated data225A to the register230.

The J-K bit multiplexer320can be or include a component, switch or circuitry that selects between inputs: i) a second portion of the accumulated data235B and ii) incremented data355, according to a carry value of the addition from the K bit adder circuitry310, to provide as an output. In one aspect, the incrementor350(or a counter) receives the second portion of the accumulated data235B, and increments the second portion of the accumulated data235B by a predetermined value (e.g., ‘1’ or other value) to generate the incremented data355. For example, the J-K bit multiplexer320can receive J-K bits of MSB portion of the accumulated data235B and incremented data355having J-K number of bits, and can select between i) the J-K bits of MSB portion of the accumulated data235B and ii) the incremented data355having J-K number of bits, according to the carry value. In one configuration, a one-bit carry output of the K bit adder circuitry310can be coupled to a control input of the J-K bit multiplexer320. In response to the carry value being ‘1’, the J-K bit multiplexer320may select the incremented data355. Similarly, in response to the carry value being ‘0’, the J-K bit multiplexer320may select the second portion of the accumulated data235B.

By segregating computations into two portions (e.g., LSB portion and MSB portion) performed in parallel, an addition or an accumulation can be performed by the K bit adder circuitry310for a reduced number of bits (e.g., for the LSB portion) to improve operation speed and achieve computational efficiency. Specifically, the critical path for the K bit adder circuitry310can be reduced, compared to a full J bit adder circuitry. In one aspect, J-K may be less than K, such that the output of the multiplexer320can be selected immediately after the carry value at the carry output of the K bit adder circuitry310is generated. Accordingly, the accumulator220A can achieve a low critical path for the accumulation to improve operation speed and power efficiency.

Referring toFIG. 3B, illustrated is a block diagram of the accumulator220B ofFIG. 2, according to an example implementation of the present disclosure. The accumulator220B ofFIG. 3Bis similar to the accumulator220A ofFIG. 3A, except that the adder circuitry310may add the first portion of the accumulated data235A with a plurality of multiplication results212(or a plurality of summation data218) to generate a carry value having multiple bits, and the multiplexer320may select and output one from more than two input data, according to the carry value having multiple bits.

In one example implementation, the adder circuitry310may add the accumulated data235A with three or four multiplication results212from different multipliers210, simultaneously or sequentially. By adding the accumulated data235A with three or four multiplication results212, the adder circuitry310may generate the carry value having two bits. The multiplexer320may receive the second portion of the accumulated data235B and incremented data355A,355B,355C from different incrementors350A,350B,350C (or counters). The incrementor350A may receive and increment the second portion of the accumulated data235B by ‘1’ to generate the incremented data355A, the incrementor350B may receive and increment the second portion of the accumulated data235B by ‘2’ to generate the incremented data355B, and the incrementor350C may receive and increment the second portion of the accumulated data235C by ‘3’ to generate the incremented data355C. Alternatively, the incrementors350A,350B,350C are connected in series or in cascade, such that the incrementor350A may receive and increment the second portion of the accumulated data235B by ‘1’ (or other value) to generate the incremented data355A, the incrementor350B may receive and increment the incremented data355A by ‘1’ (or other value) to generate the incremented data355B, and the incrementor350C may receive and increment the incremented data355B by ‘1’(or other value) to generate the incremented data355C.

In one aspect, the multiplexer320may select one from its multiple input data, according to the carry value at its control input. For example, if the carry value is ‘00’, the multiplexer320may select and output the second portion of the accumulated data235B. For another example, if the carry value is ‘01’, the multiplexer320may select and output the incremented data355A. For another example, if the carry value is ‘10’, the multiplexer320may select and output the incremented data355B. For another example, if the carry value is ‘11’, the multiplexer320may select and output the incremented data355C. By performing accumulation of a plurality of multiplication results212and selecting a MSB portion according to a carry value having multiple bits, an operation speed of the accumulator220B can be further improved. Although inFIG. 3B, the multiplexer320receives a carry value having two bits, then selects and outputs one of four inputs235B,355A,355B,355C according to the carry value, in other embodiments, the multiplexer320may receive a carry value having any number of bits, then select and output one of any number of inputs according to the carry value.

Referring toFIG. 4, illustrated is a flow chart illustrating a process400of performing multiply-accumulate operation for a neural network, according to an example implementation of the present disclosure. In some embodiments, the process400is performed by the accumulator220ofFIG. 3AorFIG. 3B. In some embodiments, the process400is performed by one or more other entities. In some embodiments, the process400includes more, fewer, or different steps than shown inFIG. 4.

In one approach, the accumulator220adds410input data and a first number of bits of accumulated data. The input data may be generated, obtained, computed, or received from processing of a neural network. For example, the input data may be a result from multiplying a weight and a corresponding activation value. The input data may have K bits, where the accumulated data may have J bits higher than the K bits. The accumulator220may add K bits of a LSB portion of the accumulated data with the input data having K bits. In one aspect, J-K (e.g., difference between the value of J and the value of K) is equal to or less than K.

In one approach, the accumulator220can check or confirm415whether a carry value from the addition in the step410is a predetermined threshold (e.g., ‘1’) to control selection. The accumulator220may select between i) a second number of bits of the accumulated data and ii) incremented data, according to a carry value from the addition in the step410, according to the carry value being “1” or “0”. In one approach, the accumulator220receives the second portion of the accumulated data, and increments the second portion of the accumulated data by a predetermined value (e.g., ‘1’) to generate the incremented data. For example, the accumulator220receives J-K bits of MSB portion of the accumulated data and incremented data having J-K number of bits and selects between i) the J-K bits of MSB portion of the accumulated data235B and ii) the incremented data355having J-K bits, according to the carry value. For example, in response to the carry value from the addition in the step410being ‘0’, the accumulator220may select420the second portion of the accumulated data235B. For example, in response to the carry value from the addition in the step410being ‘1’, the accumulator220may select430the incremented data355.

In one approach, the accumulator220can output440, generate and/or provide the result of the addition in the step410with the selected data in the step420as updated accumulated data or appended data. For example, the result of the addition has K number of bits that corresponds to K bits of the LSB portion of the appended data, and the selected data has J-K number of bits that corresponds to J-K bits of the MSB portion of the appended data. The accumulator220may send, output, convey, direct and/or provide the appended data to a register (e.g., register230ofFIG. 2, which can be an array or a bank of registers that can be activated by a clock signal). In one aspect, the appended data for a clock cycle may be received by the accumulator220as an accumulated data in a subsequent clock cycle through the register. For example, a K number of bits of the appended data from an output of an adder circuitry (e.g., the adder circuitry310ofFIG. 3A) may be provided to an input of the adder circuitry for a subsequent clock cycle through the register. Similarly, a J-K number of bits of the appended data from an output of the multiplexer (e.g., the multiplexer320ofFIG. 3A) may be sent, fed, or provided to an input of the multiplexer for a subsequent clock cycle through the register.

In some embodiments, the input data is produced, derived, generated and/or provided by multiplying first data and second data. The first data may be or include a weight of (e.g., a node of a layer of) the neural network, and the second data may be an activation value of (e.g., the node of the layer of) the neural network. In one aspect, the first data has M number of bits and the second data has M number of bits, where the input data has K number of bits that is at least twice of the M number of bits.

Systems and methods described herein may be embodied in other specific forms without departing from the characteristics thereof. References to “approximately,” “about” “substantially” or other terms of degree include variations of +/−10% from the given measurement, unit, or range unless explicitly indicated otherwise. Coupled elements can be electrically, mechanically, or physically coupled with one another directly or with intervening elements. Scope of the systems and methods described herein is thus indicated by the appended claims, rather than the foregoing description, and changes that come within the meaning and range of equivalency of the claims are embraced therein.