Patent Description:
Deep neural networks (DNNs) may be accelerated by Neural Processing Units (NPUs). The operand sparsity associated with General Matrix Multiply (GEMM) operations in DNNs may be used to accelerate operations performed by NPUs. Fine-grained structured sparsity, especially N:M sparsity (N nonzero elements out of M weight values), may be helpful to maintain accuracy and save hardware overhead compared to random sparsity. Existing technology related to structured sparsity, however, only supports weight sparsity.

<NPL>, relates to Sparse Optimization for Deep Learning Architectures.

<NPL>, relates to a NPU architecture for a mobile system on chip.

In the following section, the aspects of the subject matter disclosed herein will be described with reference to the figures, in which:.

In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the disclosure. It will be understood, however, by those skilled in the art that the disclosed aspects may be practiced without these specific details. In other instances, well-known methods, procedures, components and circuits have not been described in detail to not obscure the subject matter disclosed herein.

Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment disclosed herein. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "according to one embodiment" (or other phrases having similar import) in various places throughout this specification may not necessarily all be referring to the same embodiment. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner in one or more embodiments. In this regard, as used herein, the word "exemplary" means "serving as an example, instance, or illustration. " Any embodiment described herein as "exemplary" is not to be construed as necessarily preferred or advantageous over other embodiments. Additionally, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. Also, depending on the context of discussion herein, a singular term may include the corresponding plural forms and a plural term may include the corresponding singular form. Similarly, a hyphenated term (e.g., "two-dimensional," "pre-determined," "pixel-specific," etc.) may be occasionally interchangeably used with a corresponding non-hyphenated version (e.g., "two dimensional," "predetermined," "pixel specific," etc.), and a capitalized entry (e.g., "Counter Clock," "Row Select," "PIXOUT," etc.) may be interchangeably used with a corresponding non-capitalized version (e.g., "counter clock," "row select," "pixout," etc.). Such occasional interchangeable uses shall not be considered inconsistent with each other.

Also, depending on the context of discussion herein, a singular term may include the corresponding plural forms and a plural term may include the corresponding singular form. It is further noted that various figures (including component diagrams) shown and discussed herein are for illustrative purpose only, and are not drawn to scale. Further, if considered appropriate, reference numerals have been repeated among the figures to indicate corresponding and/or analogous elements.

The terminology used herein is for the purpose of describing some example embodiments only and is not intended to be limiting of the claimed subject matter. The terms "first," "second," etc., as used herein, are used as labels for nouns that they precede, and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) unless explicitly defined as such. Furthermore, the same reference numerals may be used across two or more figures to refer to parts, components, blocks, circuits, units, or modules having the same or similar functionality. Such usage is, however, for simplicity of illustration and ease of discussion only; it does not imply that the construction or architectural details of such components or units are the same across all embodiments or such commonly-referenced parts/modules are the only way to implement some of the example embodiments disclosed herein.

It will be understood that when an element or layer is referred to as being on, "connected to" or "coupled to" another element or layer, it can be directly on, connected or coupled to the other element or layer or intervening elements or layers may be present. In contrast, when an element is referred to as being "directly on," "directly connected to" or "directly coupled to" another element or layer, there are no intervening elements or layers present.

The terms "first," "second," etc., as used herein, are used as labels for nouns that they precede, and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) unless explicitly defined as such. Furthermore, the same reference numerals may be used across two or more figures to refer to parts, components, blocks, circuits, units, or modules having the same or similar functionality. Such usage is, however, for simplicity of illustration and ease of discussion only; it does not imply that the construction or architectural details of such components or units are the same across all embodiments or such commonly-referenced parts/modules are the only way to implement some of the example embodiments disclosed herein.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this subject matter belongs.

As used herein, the term "module" refers to any combination of software, firmware and/or hardware configured to provide the functionality described herein in connection with a module. For example, software may be embodied as a software package, code and/or instruction set or instructions, and the term "hardware," as used in any implementation described herein, may include, for example, singly or in any combination, an assembly, hardwired circuitry, programmable circuitry, state machine circuitry, and/or firmware that stores instructions executed by programmable circuitry. The modules may, collectively or individually, be embodied as circuitry that forms part of a larger system, for example, but not limited to, an integrated circuit (IC), system on-a-chip (SoC), an assembly, and so forth.

The subject matter disclosed herein provides an NPU architecture, generally referred to as a hybrid-sparsity architecture, that supports activation sparsity in a NPU that is also configured for N:M fine-grain structured weight sparsity. The hybrid-sparsity NPU may operate in several sparse modes, such as, structured weight sparsity, random weight sparsity, random activation sparsity, dual sparsity with random activation sparsity and structured weight sparsity, and dual sparsity with random activation sparsity and random weight sparsity.

Two different types of sparse NPU core architectures are disclosed herein. A first type of sparse NPU core architecture supports structured weight sparsity, random weight sparsity, random activation sparsity, dual sparsity with random activation sparsity and structured weight sparsity, and dual sparsity with random activation sparsity and random weight sparsity. A second type of sparse NPU core architecture supports dual sparsity and uses an ANDing technique. A significant benefit provided by the subject matter disclosed herein is that a hybrid sparsity NPU may be efficiently implemented using sparse logic.

While an existing system only supports N:M=<NUM>:<NUM> structured weight sparsity, the hybrid-sparsity NPU disclosed herein supports dual sparsity (i.e., activation sparsity and weight sparsity) in which activation sparsity may be random sparsity and weight sparsity may be structured weight sparsity (N:M = <NUM>:<NUM>, <NUM>:<NUM>, <NUM>:<NUM>, <NUM>:<NUM>) and random weight sparsity.

The "Hybrid Sparsity-V1" architecture may run DNN tasks with various modes, such as: (<NUM>) a N:M=<NUM>:<NUM> structured weight-sparse mode that provides a 4x speed up and ~<NUM> power/area efficiency improvement (as compared to a dense baseline architecture); (<NUM>) a dual sparsity mode with a N:M=<NUM>:<NUM> structured sparsity with ~<NUM>. 5x power/area efficiency improvement; (<NUM>) a dual sparsity mode with random weight sparsity with ~<NUM> power efficiency improvement; (<NUM>) a random weight-sparsity mode with ~3x power efficiency improvement; and (<NUM>) a random activation-sparsity mode with ~<NUM> power efficiency improvement.

The Hybrid Sparsity-V1 architecture may be configured to use weight-preprocessing techniques that may be more efficient for DNN inference-type operations. A second embodiment, referred to herein as a Hybrid-V2 architecture, a NPU architecture uses AND-gates that may be more efficient for DNN training-type operations.

<FIG> depicts an example of a NPU architecture <NUM> and an example of a dual-sparsity NPU architecture <NUM>' according to the subject matter disclosed herein. The NPU architecture <NUM> is depicted at the top of <FIG> and the example of the dual-sparsity NPU architecture <NUM>' is depicted at the bottom of <FIG> The example NPU architecture <NUM> is configured for a <NUM>:<NUM> fine-grain structured weight sparsity and the dual-sparsity NPU architecture <NUM>' is configured for a <NUM>:<NUM> fine-grain structured weight sparsity that includes a <NUM>-cycle activation lookahead. The dual-sparsity NPU architecture <NUM>' is referred to herein as a Hybrid Sparsity-V1.

The NPU architecture <NUM> may include a multiply and accumulate (MAC) unit having an array of four multipliers (each indicated by a block containing an X). The accumulator portion of the MAC unit includes an adder tree (indicated by a block containing a +) and an accumulator ACC. Additionally, the NPU architecture <NUM> may include a weight buffer WBUF array that contains <NUM> weight register WREG for each multiplier of the MAC unit, and an activation buffer ABUF that contains a depth of <NUM> activation registers AREG for each multiplier of the MAC unit. An activation multiplexer AMUX may include an activation multiplexer (indicated by a trapezoidal shape) for each multiplier of the MAC unit. Although not explicitly shown, each activation multiplexer has a fan in of <NUM>. That is, the fan in of each activation multiplexer is connected (not shown) to <NUM> separate AREGs. In operation, a weight value in a WREG is input to a multiplier as a first input. Weight metadata is used to control the multiplexers of the AMUX to select an appropriate AREG in the ABUF corresponding to each weight value. The activation value in a selected AREG is input to a multiplier as a second input corresponding to first input to the multiplier. The NPU architecture <NUM> provides a speed up of 2x over a NPU architecture configured only for weight sparsity. Additional details of the NPU architecture <NUM> are provided in (attorney docket <NUM>-<NUM>), which is incorporated by reference herein.

By adding a <NUM>-cycle random activation lookahead, the NPU architecture <NUM> may be reconfigured to be the Hybrid Sparsity-V1 architecture <NUM>', which is configured for dual sparsity that includes a <NUM>:<NUM> fine-grain structured weight sparsity that includes a <NUM>-cycle random activation lookahead. The MAC unit of the Hybrid Sparsity-V1 architecture <NUM>' is not changed from the NPU architecture <NUM>. The WBUF is reconfigured to include a WREG depth of <NUM>. A weight multiplexer array WMUX is added to select an appropriate weight value stored in the WBUF. The ABUF is also increased in size to provide the <NUM>-cycle random activation lookahead, so that the reconfigured ABUF stores <NUM> cycles for each multiplier of the MAC unit. The AMUX is reconfigured so that each multiplexer has a fan in of <NUM>. That is, each multiplexer of the AMUX is a <NUM>-to-<NUM> multiplexer. A control unit is also added that in operation receives an activation zero-bit mask (A-zero-bit mask) and weight metadata in order to control (ctrl) the multiplexers of the AMUX to select appropriate AREGs. The example of the dual-sparsity NPU architecture <NUM>' provides a speed up of ~3x over a NPU architecture configured for only weight sparsity.

The Hybrid Sparsity-V1 architecture <NUM>' may also be used for random weight sparsity operations. <FIG> depicts the example of a NPU architecture <NUM>, and an example of a dual-sparsity NPU architecture <NUM>' according to the subject matter disclosed herein. The NPU architecture <NUM> is depicted at the top of <FIG> and the example of the dual-sparsity NPU architecture <NUM>' is depicted at the bottom of <FIG>. The example NPU architecture <NUM> is configured for a random weight sparsity of (Tw=<NUM>,Cw=<NUM>,Ta=<NUM>). Configuration details for the NPU architecture <NUM> are described above in connection with <FIG>.

The <NUM>-cycle random activation lookahead reconfiguration described above of the NPU architecture <NUM> to form the Hybrid Sparsity-V1 architecture <NUM>' may also be used for random weight sparsity operations that includes a <NUM>-cycle random activation lookahead. By adding the <NUM>-cycle random activation lookahead, the random sparsity mode of the Hybrid Sparsity-V1 architecture provides a random sparsity mode of (Tw=<NUM>,Cw=<NUM>,Ta=<NUM>). Although configured to operate with a different sparsity mode, the dual-sparsity NPU architecture <NUM>' depicted in <FIG> is still referred to as the Hybrid Sparsity-V1. The Hybrid Sparsity-V1 architecture <NUM>' is the same as described above in connection with <FIG>.

For random weight sparsity, the effective activation lookahead the NPU architecture <NUM>' is <NUM> cycles based on the <NUM> AREG depth of the ABUF with a maximum speed up of 6x (typically 2x) over a NPU architecture configured for only weight sparsity. Regarding weight preprocessing of random weight sparsity, if the weight mask is updated infrequently, software-based preprocessing may be used. If the weight mask is updated frequently, then hardware-based preprocessing by adding a weight-preprocessing unit may be a better approach.

Using similar design logic, the Hybrid Sparsity-V1 architecture <NUM>' may be used to support different sparsity modes. That is, the Hybrid Sparsity-V1 architecture <NUM>' also supports <NUM>:<NUM> and <NUM>:<NUM> sparsity modes when configured for weight-only structured sparsity based on a <NUM>:<NUM> structured-sparsity mode architecture that is reconfigured for the other modes, such as <NUM>:<NUM>, <NUM>:<NUM> and <NUM>:<NUM> structured sparsity modes. Table <NUM> sets forth some aspects of Hybrid Sparsity-V1 architecture <NUM>'. In Table <NUM>, an "S" means "structured" for structured sparsity, and an "R" means "random" for random sparsity. The top row of Table <NUM> shows that the Hybrid Sparsity-V1 architecture may be used to support <NUM>:<NUM> structured weight sparsity and random activation sparsity with <NUM>-cycle ahead. The next row down shows that the Hybrid Sparsity-V1 architecture may be used to support <NUM>:<NUM> structured <NUM>:<NUM> weight sparsity alone. If <NUM>:<NUM> structured weight sparsity is desired, then all of the sparse logic may be used on the weight side of the architecture, which provides a 4x speed up. The third row down shows that the Hybrid Sparsity-V1 architecture supports random weight sparsity with <NUM>-cycle activation lookahead and <NUM>-channel weight lookaside, which provides a maximum of a 4x speed up. The fourth row down shows that the Hybrid Sparsity-V1 architecture may be configured for only for activation sparsity by adding <NUM>-channel activation lookaside.

The Hybrid Sparsity-V1 architecture supports four different sparsity modes for a hardware cost per multiplier of a <NUM>-AREG depth in the ABUF, a <NUM>-to-<NUM> AMUX per multiplier, a <NUM>-WREG depth in the WBUF, and a <NUM>-to-<NUM> WMUX. Additionally, the additional hardware includes a controller (arbiter) to arbitrate the activations and a control unit to control both weight and activation movement for dual sparsity.

<FIG> depicts an example of a NPU architecture <NUM> and an example of a Hybrid Sparsity-V2 architecture <NUM>' according to the subject matter disclosed herein. The NPU architecture <NUM> is depicted at the top of <FIG> and the example embodiment of the NPU architecture <NUM>' is depicted at the bottom of <FIG>. The example NPU architecture <NUM> is configured for both a <NUM>:<NUM> fine-grain structured and a random weight sparsity (Tw=<NUM>,Cw=<NUM>). The dual-sparsity NPU architecture <NUM>' configured for dual sparsity and uses an ANDing technique. The dual-sparsity NPU architecture <NUM>' is referred to herein as a Hybrid Sparsity-V2.

The NPU architecture <NUM> may include a multiply and accumulate (MAC) unit having an array of four multipliers (indicated by a block containing an X). The accumulator portion of the MAC unit includes an adder tree (indicated by a block containing a +) and an accumulator ACC. Additionally, the NPU architecture <NUM> may include a weight buffer WBUF that contains <NUM> weight register WREG for each multiplier of the MAC unit, and an activation buffer ABUF that contains a depth of <NUM> activation registers AREG for each multiplier of the MAC unit. An activation multiplexer AMUX may include an activation multiplexer (indicated by a trapezoidal shape) for each multiplier of the MAC unit. Although not explicitly shown, each activation multiplexer has a fan in of <NUM>. That is, the fan in of each activation multiplexer is connected (not shown) to <NUM> separate AREGs. In operation, a weight value in a WREG is input to a multiplier as a first input. Weight metadata is used to control the multiplexers of the AMUX to select an appropriate AREG in the ABUF corresponding to each weight value. The activation value in a selected AREG is input to a multiplier as a second input corresponding to first input to the multiplier. Weight metadata is used to control the multiplexers of the AMUX to select an appropriate AREG in the ABUF. The NPU architecture <NUM> provides a speed up of 4x for a <NUM>:<NUM> fine-grain structured weight sparsity, or a speed up of <NUM>. 7x for a random weight sparsity. Additional details of the NPU architecture <NUM> are provided in (attorney docket <NUM>-<NUM>), which is incorporated by reference herein.

By using an ANDing technique and increasing the size of the WBUF, the example NPU architecture <NUM> may be reconfigured to form the Hybrid Sparsity-V2 architecture <NUM>,' which is configured for dual sparsity and provides a speed up of 3x. The MAC unit of the Hybrid Sparsity-V2 architecture <NUM>' is not changed from the NPU architecture <NUM>. The WBUF is reconfigured to include a WREG depth of <NUM>. The ABUF and the AMUX are not changed from the NPU architecture <NUM>. A control unit (CU) that includes an ANDing (&) functionality is added that may receive an activation zero-bit mask (A-zero-bit mask), a weight zero-bit-mask (W-zero-bit mask) and weight metadata, and ANDs the two bit masks and uses the weight metadata to control (ctrl) the multiplexers of the AMUX to select appropriate AREGs in operation. The example embodiment of the NPU architecture <NUM>' provides a speed up of ~3x.

Table <NUM> sets forth some configuration aspects of Hybrid Sparsity-V2 architecture <NUM>'. In Table <NUM>, an "S" means "structured" for structured sparsity, and an "R" means "random" for random sparsity. Additionally, an "L" associated with the Hybrid Sparsity-V1 indicates a relatively large hardware overhead configuration, and an "S" associated with the Hybrid Sparsity-V1 indicates a relatively small hardware overhead configuration. The different sparsity configurations are indicated below the Hardware heading. The register depths are shown below the ABUF and the WBUF headings, and the multiplexer fan ins are shown below the AMUX and WMUX headings for the different sparsity configurations. The number of adder trees for each sparsity configuration are shown below the ADT heading. The CTRL heading indicates that type of control that is used for a sparsity configuration in which "CU" stands for Control Unit, "Arb" stands for an Arbitrator unit, "Preproc" stands for preprocessing, and "On-the-Fly" stands for on-the-fly processing. Speed up for the different sparsity configurations is shown below the Speed Up heading. The "-/(<NUM>)" and the "-/(<NUM>) appearing towards the bottom of Table <NUM> respectively indicate the WBUF and WMUX resources associated with "preprocessing" and "on-the-fly" control modes. A "-" indicates that no WBUF or WMUX resources are needed with the weight tensors are compressed and metadata is generated in compile time. If "on-the-fly" mode is used in which weight tensors are not compressed in compile time and metadata is not generated prior to the runtime, the WBUF uses a fan-in of <NUM> and WMUX has a depth of <NUM> registers.

<FIG> depicts an electronic device <NUM> that may include at least one NPU that supports dual-sparsity modes according to the subject matter disclosed herein. Electronic device <NUM> and the various system components of electronic device <NUM> may be formed from one or modules. The electronic device <NUM> may include a controller (or CPU) <NUM>, an input/output device <NUM> such as, but not limited to, a keypad, a keyboard, a display, a touchscreen display, a 2D image sensor, a 3D image sensor, a memory <NUM>, an interface <NUM>, a GPU <NUM>, an imaging-processing unit <NUM>, a neural processing unit <NUM>, a TOF processing unit <NUM> that are coupled to each other through a bus <NUM>. In one embodiment, the 2D image sensor and/or the 3D image sensor may be part of the imaging processing unit <NUM>. In another embodiment, the 3D image sensor may be part of the TOF processing unit <NUM>. The controller <NUM> may include, for example, at least one microprocessor, at least one digital signal processor, at least one microcontroller, or the like. The memory <NUM> may be configured to store a command code to be used by the controller <NUM> and/or to store a user data. The neural processing unit <NUM> may include at least one NPU that supports dual-sparsity modes according to the subject matter disclosed herein.

The interface <NUM> may be configured to include a wireless interface that is configured to transmit data to or receive data from, for example, a wireless communication network using a RF signal. The wireless interface <NUM> may include, for example, an antenna. The electronic system <NUM> also may be used in a communication interface protocol of a communication system, such as, but not limited to, Code Division Multiple Access (CDMA), Global System for Mobile Communications (GSM), North American Digital Communications (NADC), Extended Time Division Multiple Access (E-TDMA), Wideband CDMA (WCDMA), CDMA2000, Wi-Fi, Municipal Wi-Fi (Muni Wi-Fi), Bluetooth, Digital Enhanced Cordless Telecommunications (DECT), Wireless Universal Serial Bus (Wireless USB), Fast low-latency access with seamless handoff Orthogonal Frequency Division Multiplexing (Flash-OFDM), IEEE <NUM>, General Packet Radio Service (GPRS), iBurst, Wireless Broadband (WiBro), WiMAX, WiMAX-Advanced, Universal Mobile Telecommunication Service - Time Division Duplex (UMTS-TDD), High Speed Packet Access (HSPA), Evolution Data Optimized (EVDO), Long Term Evolution - Advanced (LTE-Advanced), Multichannel Multipoint Distribution Service (MMDS), Fifth-Generation Wireless (<NUM>), Sixth-Generation Wireless (<NUM>), and so forth.

Embodiments of the subject matter and the operations described in this specification may be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer-program instructions, encoded on computer-storage medium for execution by, or to control the operation of data-processing apparatus. Alternatively or additionally, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer-storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial-access memory array or device, or a combination thereof. Moreover, while a computer-storage medium is not a propagated signal, a computer-storage medium may be a source or destination of computer-program instructions encoded in an artificially-generated propagated signal. The computer-storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices). Additionally, the operations described in this specification may be implemented as operations performed by a data-processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

While this specification may contain many specific implementation details, the implementation details should not be construed as limitations.

Moreover, it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Thus, particular embodiments of the subject matter have been described herein. Other embodiments are within the scope of the following claims. In some cases, the actions set forth may be performed in a different order and still achieve desirable results. Additionally, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claim 1:
A neural processing unit (<NUM>, <NUM>', <NUM>, <NUM>', <NUM>), comprising:
a weight buffer (WBUF) configured to store weight values in an arrangement selected from a group comprising a structured weight sparsity arrangement and a random weight sparsity arrangement;
a weight multiplexer configured to output one of the weight values stored in the weight buffer (WBUF) as a first operand value based on the selected weight sparsity arrangement;
an activation buffer (ABUF) configured to store activation values;
an activation multiplexer (AMUX) coupled to the activation buffer (ABUF), the activation multiplexer (AMUX) configured to output one of the activation values stored in the activation buffer (ABUF) as a second operand value, the second operand value and the first operand value forming an operand value pair; and
a multiplier unit (MAC) configured to output a product value for the operand value pair, wherein:
the weight buffer (WBUF) comprises an array of weight registers (WREG), each weight register (WREG) being configured to store a weight value that is in the arrangement selected from the group comprising the structured weight sparsity arrangement and the random weight sparsity arrangement,
the weight multiplexer is configured to select a weight register (WREG) based the weight sparsity arrangement of the weight values stored in the weight buffer (WBUF) and output the weight value stored in the selected weight register (WREG) as the first operand value,
the activation buffer (ABUF) comprises an array of activation registers (AREG), each activation register (AREG) being configured to store an activation value, and
the activation multiplexer (AMUX) is configured to select and output an activation value stored in the activation buffer (ABUF) as the second operand value, the second operand value corresponding to the first operand value and forming the operand value pair.