CHIPLET BASED ARTIFICIAL INTELLIGENCE ACCELERATORS AND CONFIGURATION METHODS

A processing unit can include a plurality of chiplets coupled in a cascade topology by a plurality of interfaces. A set of the plurality of cascade coupled chiplets can be configured to execute a plurality of layers or blocks of layers of an artificial intelligence model. The set of cascade coupled chiplets can also be configured with parameter data of corresponding ones of the plurality of layers or blocks of layers of the artificial intelligence model.

BACKGROUND OF THE INVENTION

A growing use of computing devices involve intelligent applications utilizing artificial intelligence, machine learning, deep learning and the like applications. Intelligent applications can include image recognition, language processing, autonomous vehicle controls, medical diagnostics, search engines, and the like. Artificial intelligence as used herein refers to techniques that enable devices to mimic human intelligence, using logic, if-then rules, decision trees, and the like. Machine learning includes a subset of artificial intelligence that includes abstruse statistical techniques that enable machines to improve at tasks with experience. Deep learning includes a subset of machine learning that includes algorithms that permit software to train itself to perform tasks by exposing multilayered artificial neural networks, recurrent neural networks (RNN), convolution neural networks (CNN) or the like to vast amounts of data. For ease of explanation, artificial intelligence, as used herein, also includes machine learning, deep learning and the like. In addition, neural network, as used herein, also includes artificial neural networks (ANN), recurrent neural networks (RNN), convolution neural networks (CNN), deep neural networks (DNN), graph neural networks (GNN) and the like.

Large-scale artificial intelligence models can include many layers or modules, as illustrated inFIGS. 1A and 1B. The layers or modules and associate data can be mapped to the resources of a processing unit such as, but not limited to, an artificial intelligence accelerator. However, the artificial intelligence accelerator is limited by the physical size of its monolithic integrated circuit (IC) chip. As the size of artificial intelligence models increase, it is difficult or impossible to fit all the parameters of the entire artificial intelligence model on a processing chip. Therefore, a common approach is to bring the parameters for a layer or a block of layers onto the chip, perform the associated computations, and write the result for the layer of block of layers back to memory, and then fetch parameters for another layer or block of layers onto the chip. However, bring parameters for each layer or block of layers on chip, writing back results of the given layer or block of layers, and repeating consumes a substantially amount of data communication bandwidth resulting in substantially processing latency. The substantial amount of data communication can also consume a substantial amount of power. Accordingly, there is a continuing need for improved systems and methods for executing artificial intelligence models.

SUMMARY OF THE INVENTION

The present technology may best be understood by referring to the following description and accompanying drawings that are used to illustrate embodiments of the present technology directed toward artificial intelligence accelerators.

In one embodiment, a processing unit can include a plurality of chiplets including interfaces to communicatively cascade the plurality of chiplets or a subset thereof together. The plurality of chiplets or subsets thereof can be configured to execute layers or blocks of layers of one or more artificial intelligence models. The plurality of chiplets or subsets thereof can also be configured with parameter data of the one or more artificial intelligence models.

In another embodiment, a method of configuring a processing unit can include mapping a plurality of layers of an artificial intelligence model to a set of cascaded chiplets. The cascaded chiplets can be configured to execute the plurality of layers of the artificial intelligence model based on the mapping. The cascaded chiplets can also be configured with parameter data of corresponding ones of the plurality of layers of the artificial intelligence model.

DETAILED DESCRIPTION OF THE INVENTION

Some embodiments of the present technology which follow are presented in terms of routines, modules, logic blocks, and other symbolic representations of operations on data within one or more electronic devices. The descriptions and representations are the means used by those skilled in the art to most effectively convey the substance of their work to others skilled in the art. A routine, module, logic block and/or the like, is herein, and generally, conceived to be a self-consistent sequence of processes or instructions leading to a desired result. The processes are those including physical manipulations of physical quantities. Usually, though not necessarily, these physical manipulations take the form of electric or magnetic signals capable of being stored, transferred, compared and otherwise manipulated in an electronic device. For reasons of convenience, and with reference to common usage, these signals are referred to as data, bits, values, elements, symbols, characters, terms, numbers, strings, and/or the like with reference to embodiments of the present technology.

It should be borne in mind, however, that these terms are to be interpreted as referencing physical manipulations and quantities and are merely convenient labels and are to be interpreted further in view of terms commonly used in the art. Unless specifically stated otherwise as apparent from the following discussion, it is understood that through discussions of the present technology, discussions utilizing the terms such as “receiving,” and/or the like, refer to the actions and processes of an electronic device such as an electronic computing device that manipulates and transforms data. The data is represented as physical (e.g., electronic) quantities within the electronic device's logic circuits, registers, memories and/or the like, and is transformed into other data similarly represented as physical quantities within the electronic device.

In this application, the use of the disjunctive is intended to include the conjunctive. The use of definite or indefinite articles is not intended to indicate cardinality. In particular, a reference to “the” object or “a” object is intended to denote also one of a possible plurality of such objects. The use of the terms “comprises,” “comprising,” “includes,” “including” and the like specify the presence of stated elements, but do not preclude the presence or addition of one or more other elements and or groups thereof. It is also to be understood that although the terms first, second, etc. may be used herein to describe various elements, such elements should not be limited by these terms. These terms are used herein to distinguish one element from another. For example, a first element could be termed a second element, and similarly a second element could be termed a first element, without departing from the scope of embodiments. It is also to be understood that when an element is referred to as being “coupled” to another element, it may be directly or indirectly connected to the other element, or an intervening element may be present. In contrast, when an element is referred to as being “directly connected” to another element, there are not intervening elements present. It is also to be understood that the term “and or” includes any and all combinations of one or more of the associated elements. It is also to be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting.

Referring now toFIG. 2, deployment of an artificial intelligence model on a processing unit, in accordance with aspects of the present technology, is illustrated. The processing unit200can be an artificial intelligence accelerator including a plurality of chiplets205,210. The plurality of chiplets can be packaged as a multi-chiplet module or the like. As used herein, a chiplet is a monolithic integrated circuit die that has been designed to work with other similar chiplets to form a larger more complex chip. The plurality of chiplets205,210can be homogeneous or heterogeneous. The plurality of chiplets205,210can be configurable to execute one or more layers and or one or more blocks of layers215,220of one or more artificial intelligence models. In one implementation, the one or more artificial intelligence models can include one or more neural network models. The plurality of chiplets205,210can also be configurable with parameter data, including but not limited to configuration data, activation data, weights or the like, of the one or more artificial intelligence models.

The plurality of chiplets205,210can include interfaces225-240to communicatively cascade the plurality of chiplets205,210together. Each of the chiplets205,210, can include an ingress interface225,230, and an egress interface235,240. The interfaces225-240can be configured to transfer one or more feature map data streams between adjacent cascaded ones of the plurality of chiplets205,210during training and inference modes. The interfaces225-240can also be configured to transfer commands between adjacent cascaded ones of the plurality of chiplets205,210during configuration and or during training and inference modes. The interfaces225-240can include one or more clock lines configured to transmit one or more clock signals to synchronize the plurality of chiplets205,210. The interfaces225-240can also include one or more control buses configured to transmit one or more control signals. The interfaces225-240can also include one or more data buses configured to transmit one or more data streams.

Referring now toFIG. 3, a method of configuring a processing unit for executing an artificial intelligence model, in accordance with aspects of the present technology, is shown. The method can include mapping a plurality of layers or blocks of layers of an artificial intelligence model to a set of cascaded chiplets, at310. For example, a first set of one or more layers and or blocks of layers of the artificial intelligence model can be mapped to the compute resources of a first chiplet. A second set of one or more layers and or blocks of layers of the artificial intelligence model can be mapped to the compute resources of a second chiplet.

At320, a plurality of cascaded chiplets can be configured to execute the plurality of layers of the artificial intelligence model based on the mapping. For example, compute resources of respective chiplets can be configured to perform compute functions of respective layers of the artificial intelligence model. On-chiplet memory, interfaces and dataflow between the compute resources, the interfaces and the on-chiplet memory can also be configured based on the mapping. In one implementation, the on-chiplet memory can include on-chiplet volatile memory buffers, such as static random-access memory (SRAM), for storing feature map data. The on-chiplet memory can also include on-chiplet non-volatile memory, such as Flash, resistive random-access memory (RRAM), phase change random access memory (PRAM) or the like, for storing activation data. Interfaces for cascade coupling the chiplets together can also be configured as respective egress and ingress interfaces. The egress and ingress interfaces can be symmetrical. The interfaces can be configured to transfer data streams, such as but not limited to feature maps, between adjacent cascaded ones of the plurality of chiplets. The interfaces can also be configured to transfer commands, clock signals and the like between adjacent cascaded ones of the plurality of chiplets. At least one interface of at least one chiplet can also be configured as an external input/output interface, such as a universal serial bus (USB), peripheral component interface express (PCIe) bus or the like, of the processing unit.

At330, the plurality of cascaded chiplets can be configured with parameter data of corresponding ones of the plurality of layers of the artificial intelligence model. For example, respective portions of activation data for the artificial intelligence model can be loaded into on-chiplet memory of respective chiplets.

Optionally, a plurality of instance of one or more layers of the artificial intelligence model can be mapped to the set of the cascaded chiplets. Additional instances of the one or more layers of the artificial intelligence model can be mapped to the set of the cascaded chiplets to improve performance. The cascaded chiplets can then be configured to execute the plurality of instances of the one or more layers, and the cascaded chiplets can also be configured with parameter data of corresponding instances of the one or more layers of the artificial intelligence model. Similarly, multiple copies of the artificial intelligence model can be mapped to different sets of the cascaded chiplets so that the processing unit can run multiple copies of the artificial intelligence model at the same time.

Optionally, a plurality of layers of a second artificial intelligence model can also be mapped to a second set of the cascaded chiplets, at340. The first and second set of chiplets can be separate, overlapping or mutually exclusive sets within the plurality of cascaded chiplets. The cascaded chiplets can also be configured to execute the plurality of layers of the second artificial intelligence mode based on the corresponding mapping, at350. The cascaded chiplets can also be configured with parameter data of corresponding ones of the plurality of layers of the second artificial intelligence model, at360. Configuring the cascaded plurality of chiplets to perform a second artificial intelligence model can be employed to improve utilization of the processing unit, where the resources of the cascaded plurality of chiplets are not fully consumed by the first artificial intelligence mode. In addition, the mapping and configuration can be further extended to configure the processing unit for executing even more artificial intelligence models if computing resources are available.

Referring now toFIG. 4, an exemplary deployment of an artificial intelligence model on a processing unit, in accordance with aspects of the present technology. The processing unit can include a plurality of artificial intelligence accelerator chiplets410-430coupled together in a package440. The plurality of chiplets410-430can be coupled together in a cascade topology by a plurality of interfaces450,460. The cascade interface450,460provide an efficient interface that supports data streaming from adjacent chiplets to adjacent chiplets.

A set of the plurality of cascade coupled chiplets can be configured to execute a plurality of layers and or blocks of layers of a given artificial intelligence model. For example, a first chiplet410can be configured to execute a first block of an artificial intelligence model, a second chiplet420can be configured to execute a second block of the artificial intelligence model, and a third chiplet430can be configured to execute a third block of the artificial intelligence model. The cascade chiplets410-430can also be configured with parameter data of corresponding ones of the plurality of blocks of the artificial intelligence model. The parameters can stay on the respective chiplet410-430and do not need to be repeatedly fetched from and or written to off-chiplet memory.

The plurality of interfaces450,460can include one or more data buses configured to transmit one or more data streams. For example, the plurality of interface450,460can be configured to transfer a feature map data stream between adjacent ones of the plurality of cascade coupled chiplets410-430. The plurality of interfaces450,460can also include one or more clock lines configured to transmit one or more clock signals to synchronize the plurality of chiplets410-430. The plurality of interfaces450,460can also include one or more control buses configured to transmit control information. For example, the plurality of interfaces450,460can transmit one or more control signals to identify data stream numbers. The plurality of interfaces450,460can also transmit one or more control signals to identify commands. Control signals can indicate bandwidth requirements, supported commands and data stream numbers, and the like.

AlthoughFIG. 4illustrates a package including three artificial intelligence accelerator chiplets410-430and that the given artificial intelligence model is mapped to all three chiplets,410-430, a package can include any number of chiplets. Furthermore, a given artificial intelligence model can be implemented on a subset of the chiplets. The other chiplets can be utilized to execute additional copies of the same artificial intelligence model, or one or more other artificial intelligence models.

Referring now toFIG. 5, package elements of an artificial intelligence processing unit, in accordance with aspects of the present technology, is shown. The package can include a package substrate510and one or more chiplets520. A first side (bottom) of the package substrate510can include a first set of contacts530configured for external coupling of the processing unit. For example, a ball grid array of contacts of a first pitch can be configured for coupling the package to a printed circuit board (PCB) or the like. A second side of the package substrate510can include a plurality of second sets of contacts540configured for coupling chiplets520to respective ones of the second sets of contacts540. Similarly, the chiplets520can include a set of contacts550configured for coupling the chiplet520to a respective one of the second set of contacts540of the package substrate510. For example, a ball grid array of contacts of a second pitch can be configured for coupling the chiplets520to corresponding contacts on the second side of the package substrate, wherein the first pitch is less than the second patch. The package substrate510can also include a plurality of interconnects (now shown) configured to communicatively cascade the interfaces of the plurality of chiplets together. The plurality of interconnects of the package substrate510can also be configured to communicatively couple at least one interface of at least one of the chiplets520to the first set of contacts530of the package substrate510.

Referring now toFIGS. 6A, 6B and 6C, a package of an artificial intelligence processing unit, in accordance with aspects of the present technology, is shown. The package of the artificial intelligence processing unit can be a multi-chiplet module or the like package. The package can include a package substrate510and one or more chiplets520laterally disposed on the package substrate510. The package can optionally include an encapsulation605. The package can be adapted to manufacture a family of products including different numbers of cascaded chiplets.

In a first example, a package610can be manufactured with a single chiplet520, as illustrated inFIG. 6A. A package610including a single chiplet520can be manufactured for applications in which an entire artificial intelligence model can fit on a single chiplet520. The chiplet520can be communicatively coupled to the package substrate510by the respective set of contacts on the chiplet520and a respective one of the second set of contacts of the package substrate510. The chiplet520can include an external interface615, such as but not limited to a universal serial bus interface, and a chiplet interface620. Because the package610includes a single chiplet520, the chiplet interface for cascade coupling chiplets can be turned off. The external interface of the single chiplet520, however is configured to be on to provide for communicating out of the package610to a host device for example.

In a second example, a package625can be manufactured with two chiplets520, as illustrated inFIG. 6B. The package625including two chiplets630,635can be manufactured for applications in which an entire artificial intelligence model cannot fit on a single chiplet520. The chiplets630,635can be communicatively coupled to the package substrate510by respective sets of contacts on the chiplets630,635and respective ones of the second sets of contacts of the package substrate510. Interconnects655in the package substrate510can couple at least one interface of at least one of the chiplets630,635to the external set of contacts of the package substrate510. Interconnects660in the package substrate510can also cascade coupled the chiplets630,635together. Because the package625includes two chiplets630,635, the interfaces for cascade coupling chiplets are configured to be on. The external interface of at least one of the chiplet630,635is also configured to be on to provide for communicating out of the package625to a host device for example.

In a third example, a package665can be manufactured with four chiplets670-685, as illustrated inFIG. 6C. A package665including four chiplets670-685can be manufactured for applications in which an entire artificial intelligence model cannot fit on a lesser number of chiplets520. Although the exemplary packages610,625and665illustrates embodiments including one, two and four chiplets, aspects of the present technology can be extended to packages including any number of chiplets by a package substrate510providing for cascade coupling of the desired number of chiplets. The package substrate510can provide a family of products with different numbers of chiplets in a common package size (e.g., PCB footprint). The packages can also have the same external interface. The packages with different numbers of chiplets can be utilized to target different market segments. Alternatively, the package substrate510can be fully populated with chiplets, and fuses or the like can be used to disable chiplets to configure the different product offerings.

Referring now toFIG. 7, a package of an artificial intelligence processing unit, in accordance with aspects of the present technology, is shown. The package700can include a package substrate710and one or more chiplets720-740stacked on the package substrate710. The stack of chiplets720-740can be coupled to the package substrate710utilizing wire bonding750. The package700can optionally include an encapsulation (not shown). The stacking of the chiplet720-740reduces the package size (e.g., PCB footprint).

Referring now toFIG. 8, a system-in-package (SiP) including a plurality of artificial intelligence accelerators chiplets, in accordance with aspects of the present technology, is shown. The SiP800can include a plurality of artificial intelligence accelerator chiplet dice810-860disposed on a package substrate870. The SiP800can also include one or more additional dice. For example, the SiP800can include a host controller chip880configured to control configuration and or operation of the plurality of AI accelerator chiplets810-860. In another example, the SiP800can include a host communication controller chip (not shown) configured to mange the external interface of the plurality of artificial intelligence accelerator chiplets810-860and or the internal chiplet interfaces of the plurality of artificial intelligence accelerator chiplets810-860. In another example, the SiP800can include a memory chip890configured to store data such as configuration libraries or the like.

Aspects of the present technology advantageously enable a large artificial intelligence model to be partitioned into layers and or blocks of layers and mapped to compute resources of a plurality of cascaded chiplets. Artificial intelligence model parameters can advantageously stay on the respective chiplets. Only the output of a block is streamed from one chiplet to another. Accordingly, the cascaded chiplets advantageously reduce data movement. As compared to a single chip artificial intelligence accelerator, chiplets are smaller, cheaper to manufacture, and can be better utilized. A scalable package of artificial intelligence chiplet accelerators can advantageously scale for various size artificial intelligence models. The artificial intelligence chiplet based accelerators can also run multiple artificial intelligence models at the same time, can run multiple copies of the same artificial intelligence model, and or multiple instances of layers or blocks of layers to speed up execution of a given artificial intelligence model. The cascaded interface of the chiplets advantageously provide an efficient interface that supports data streaming. The cascaded interfaces can transfer single or multiple feature map data streams between chiplets, and can also transfer commands such as chip configuration from a host device.