Patent ID: 12242564

DETAILED DESCRIPTION

Advantages and features of the present invention and methods to achieve the same will become clear with reference to example embodiments described in detail along with the accompanying drawings. However, the present invention is not limited to example embodiments disclosed blow and may be implemented in various forms. Here, the example embodiments are provided to make the disclosure of the present invention complete and to fully inform one of ordinary skill in the art to which the present invention pertains of the scope of the present invention and the present invention is defined by the scope of the claims. Like reference numerals used herein refer to like elements throughout.

When it is described that one component is “connected to” or “coupled to” another component, it may be understood that the one component is directly connected to or coupled to the other component or that still other component is interposed between the two components. In contrast, it should be noted that when it is described that one component is “directly connected to” or “directly coupled to” to another component, still other component may not be present therebetween. As used herein, the expression “and/or” includes any one and any combination of the associated listed items.

The terms used herein are to explain the example embodiments and not to be limiting of the present invention. Herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated components, steps, operations, and/or elements, but do not preclude the presence or addition of one or more other components, steps, operations, and/or elements.

Although terms of “first,” “second,” and the like are used to explain various components, the components are not limited to such terms. These terms are used only to distinguish one component from another component. For example, a first component may be referred to as a second component, or similarly, the second component may be referred to as the first component within the scope of the present invention.

Unless otherwise defined herein, all terms used herein (including technical or scientific terms) have the same meanings as those generally understood by one of ordinary skill in the art. Also, terms defined in dictionaries generally used should be construed to have meanings matching contextual meanings in the related art and are not to be construed as an ideal or excessively formal meaning unless otherwise defined herein.

In a training process for a large language model (LLM), an input amount and a computational amount are large, so a multi-device-based operation may be efficiently processed using a graphics processing unit (GPU) capable of well processing a throughput calculation. In contrast, in an inference process, an input amount is small and a lot of memory access is required, so performing a multi-device-based operation using the GPU degrades the efficiency.

For example, for training and inference, fundamentally different types of communication operation are required and operational characteristics are different. Accordingly, optimal solutions are also different. Training is a large batch operation with a large input amount and inference is a small batch operation with a small input amount. While multi-device-based training requires four types of communication operation, 1) Reduce Scatter, 2) All-Gather, 3) All Reduce, and 4) All-to-All, inference requires only one type of communication operation, All-Gather. Therefore, a solution for training may be applied to inference, but performance is degraded compared to a solution for inference.

Also, to operate inference of the large language model requires based on a multi-device, parallelism is required. Types of parallelism are largely divided into data parallelism and model parallelism. The data parallelism refers to a method of mapping the same model to all devices and then separating input and transmitting the same to each device, without separating a model. The model parallelism refers to a method of separating a model and mapping the same to each device. The model parallelism is divided again into inter-layer parallelism and intra-layer parallelism. The inter-layer parallelism is also called pipeline parallelism and is a method of separating a model based on a layer unit without separating a layer of the model. This may reduce a size of the model mapped to each device and may improve computational throughput, but may not reduce latency for a single request. On the contrary, the intra-layer parallelism refers to a method of separating each layer. This is a method that may improve throughput and also reduce latency, but separates each layer, which requires communication between devices after calculation in each device.

Example embodiments provide an inference performance method and system as a network technique capable of effectively performing a multi-device-based operation.

FIG.1illustrates an example of a hardware structure of a multi-device that implements an inference performance system according to an example embodiment. A device100according to an example embodiment shown inFIG.1may include a register file110, a matrix unit120, a load/store unit130, a memory140, and a peer-to-peer (P2P) network150. Here, each multi-device that implements the inference performance system may have a hardware structure identical or similar to the device100ofFIG.1.

A process of (1) matrix multiplication may be an example of a process in which the device100reads data of the register file110and performs matrix multiplication on the data through the matrix unit120.

A process of (2) store may be an example of a process in which the device100stores a sub-result in the memory140through the load/store unit130to transmit sub-results to a network without waiting the entire operation result of matrix multiplication.

A process of (3) TX (transmission) may be a process in which the device100reads the sub-result stored in the memory140from the memory140and transmits the same to the other device through the P2P network150.

A process of (4) RX (reception) may be an example of a process in which the device100all-gathers data (sub-results) transmitted from each multi-device and stores the same in the memory140. Even in this case, the device100may immediately store the sub-results in the memory140without waiting for all the data of the multi-device.

A process of (5) load may be an example in which the device100reads the sub-results stored in the memory140in the process of (4) from the memory140and writes back the same to the register file110. Here, the sub-results to be written back to the register file110may also include the sub-result stored in the memory140in the process of (2), that is, the sub-result of matrix multiplication performed by the device100.

FIG.2illustrates an example of a timeline of an operation process of matrix multiplication using a multi-device according to an example embodiment. As shown in the example embodiment ofFIG.2, the device100may substantially reduce communication overhead by processing the processes of (1) to (5) in a continuously pipelined manner, and may reduce latency excluding tail latency for the last operation result. This tail latency is very small and, as a result, the inference performance system according to the example embodiment may provide very high scalability.

FIG.3illustrates an example of a sub-result in intra-layer parallelism according to an example embodiment. The example embodiment ofFIG.3shows an example of an output vector330generated as a result of matrix multiplication between a shared input vector310and a partitioned weight matrix320. Here, each multi-device (e.g., each of device1to device4) included in the inference performance system according to the example embodiment may perform matrix multiplication between each partition of the partitioned weight matrix320and the shared input vector310for intra-layer parallelism. Here, a first sub-result350may be generated as a result of device1performing matrix multiplication between a first column340of the partitioned weight matrix320and the shared input vector310. Here, device1may transmit the earlier generated first sub-result350to each multi-device without waiting for a result of the entire partitioned weight matrix320or a result of the corresponding entire partition. That is, each device of the inference performance system according to the example embodiment may simultaneously perform matrix multiplication and All-Gather while transmitting data according to the sub-result to another device in real time, without waiting for the matrix multiplication to be completed.

To this end, instruction fusion may be required. For example, a matrix multiplication instruction and an All-Gather instruction may be fused. Also, architecture capable of supporting the instruction fusion is required. That is, a structure capable of simultaneously performing transmission for matrix multiplication and All-Gather is required. Also, the device needs to be capable of simultaneously processing reception. That is, each multi-device included in the inference performance system may have a structure that allows partitioned matrix multiplication, data transmission, and data reception to be simultaneously performed in real time.

FIG.4illustrates an example of processing data synchronization through overlapping when processing an operation of matrix multiplication using a multi-device according to an example embodiment. A method of effectively performing partitioned matrix multiplication and subsequent necessary data synchronization is most important when performing a multi-device-based inference operation. In general, a processor may start All-Gather after matrix multiplication is completed. That is, since target data of matrix multiplication that is a first instruction is source data of All-Gather that is a second instruction, All-Gather may be executed after the matrix multiplication is completed due to a dependency check logic. The example embodiment ofFIG.4shows latency in the case of performing data synchronization (e.g., data synchronization through All-Gather) after an operation of matrix multiplication is completed in a timeline and latency in the case of processing data synchronization by overlapping matrix multiplication. The inference performance system according to the example embodiment may simultaneously perform matrix multiplication and All-Gather while transmitting data of the sub-result from each multi-device to another device, without waiting for the matrix multiplication to be completed, and may significantly reduce latency through this overlapping. For example, it can be easily understood that the greater latency decrease effect may be achieved according to an increase in the size of the partitioned weight matrix320ofFIG.3.

FIG.5is a diagram illustrating an example of an internal configuration of an inference performance system according to an example embodiment, andFIG.6is a flowchart illustrating an example of an inference performance method according to an example embodiment.

An inference performance system500according to an example embodiment shown inFIG.5may include a plurality of devices (device1510and other devices520) mapped to partitions that separate a large language model into columns of a matrix of each layer according to an intra-layer parallelism method. In this case, the inference performance system500may process inference using the large language model through the plurality of devices510and520that are mapped to the partitions of the large language model.

Here, referring toFIG.5, the device1510may include a matrix processing unit511, a sub-result storage512, a memory513, a transmitter514, a receiver515, and a synchronizer516. Here, the matrix processing unit511, the sub-result storage512, the transmitter514, the receiver515, and the synchronizer516may be functional representations for an operation of a physical processor includable in the device1510. Each of the other devices520may include components identical or similar to those of the device1510.

The inference performance method ofFIG.6may include operation610, and operation610may include operation611to operation614.

In operation610, the matrix processing unit511may perform matrix multiplication on data. For example, the matrix processing unit511may continuously calculate matrix multiplication between an input vector and each column of a weight matrix and may continuously generate a first sub-result for each matrix multiplication between the input vector and each column. In this case, operation611to operation614may be performed while the matrix multiplication is being performed in operation610.

In operation611, the sub-result storage512may store, in the memory513, the first sub-result that is calculated in real time. For example, assuming that the matrix processing unit511calculates four sub-results from a (1-1)-th sub-result to a (1-4)-th sub-result, the (1-1)-th sub-result may be stored in the memory513during a process of calculating the (1-2)-th sub-result and the (1-2)-th sub-result may be stored in the memory513during a process of calculating the (1-3)-th sub-result. Even in the following operation612to operation614, each sub-result may be also processed during a process of calculating a subsequent sub-result.

In operation612, the transmitter514may read the first sub-result stored in the memory513and may transmit the same to each of the other devices520included in the inference performance system500. Here, each of the other devices520may calculate the second sub-result may transmit the same to each of other devices excluding the corresponding device itself while performing matrix multiplication.

In operation613, the receiver515may receive the second sub-result calculated by each of the other devices520and may store the same in the memory513. In this case, all of the first sub-result and the second sub-result may be stored in the memory513. Depending on example embodiments, the transmitter514and the receiver515may also process a process of storing the first sub-result again in the memory513in association with the second sub-result.

In operation614, the synchronizer516may synchronize the data using the first sub-result and the second sub-result. For example, the synchronizer516may load the first sub-result and the second sub-result to the register file and may synchronize the data.

As described above, according to example embodiments, it is possible to provide an inference performance method and system as a network technique capable of effectively performing a multi-device-based operation. Also, it is possible to reduce communication overhead and latency and to provide very high scalability for an inference performance system by simultaneously performing matrix multiplication and All-Gather.

Although the example embodiments are described above with reference to the accompanying drawings, it will be understood by one of ordinary skill in the art that the present invention can be implemented in other specific forms without changing technical spirit or essential features of the invention. Therefore, the example embodiments should be understood in all respects as illustrative and not construed as limiting.