Hardware environment and method of performing matrix multiplication in artificial intelligence applications

A plurality of hardware accelerators are interconnected and include a special processing unit and accelerator memory. At least one host computer is coupled to each of the plurality of hardware accelerators and includes a general processing unit and host memory. The plurality of hardware accelerators exchange data in a ring communication pattern in computing a linear layer of a neural network.

BACKGROUND

The present application relates generally to computers and computer applications, and more particularly to hardware configuration and environment for matrix multiplication in artificial intelligence applications.

The computation associated with a linear layer in a neural network is a multiplication between two matrices, also called General Matrix to Matrix Multiplication (GEMM) in Basic Linear Algebra Subprograms (BLAS) terminology. While existing computer systems, for example, heterogeneous systems, can implement GEMM, traditional BLAS implementation on a heterogeneous system are not optimized for the GEMM shapes and sizes used in inferencing with neural networks.

BRIEF SUMMARY

A system, in one aspect, may include a plurality of hardware accelerators interconnected via an accelerator interconnect, each of the plurality of hardware accelerators comprising a special processing unit and accelerator memory. At least one host computer may be coupled to each of the plurality of hardware accelerators via an accelerator link. The at least one host computer may include a general processing unit and host memory. The plurality of hardware accelerators may exchange data in a ring communication pattern in computing a linear layer of a neural network.

A computer-implemented method of performing a matrix to matrix operation involving a first matrix, a second matrix and a third matrix, in one aspect, may include splitting the first matrix by row into P partitions and the third matrix by row into P partitions, P representing a number of hardware accelerators involved in the matrix to matrix operation, each partition of the first matrix and each partition of the third matrix stored on a different hardware accelerator. The method may also include splitting the second matrix by row into P partitions, each partition of the second matrix stored by column on the different hardware accelerator. The method may further include each of the P hardware accelerators in parallel multiplying one block of the second matrix stored locally by corresponding columns of the partition of the first matrix stored locally and accumulating a result into a local partition of the third matrix. The method may also include, each of the P hardware accelerators in parallel reading a block of the second matrix stored on its neighbor accelerator in a ring communication pattern and multiplying the block of the second matrix read by the corresponding columns of the partition of the first matrix stored locally and accumulating a result into the local partition of the third matrix. The method may further include, each of the P hardware accelerators repeating the reading of the block of the second matrix stored on its neighbor accelerator in a ring communication pattern and multiplying the block of the second matrix read by the corresponding columns of the partition of the first matrix stored locally and accumulating a result into the local partition of the third matrix, until all partitions of the second matrix have taken part in the multiplying.

A computer-implemented method of performing a matrix to matrix operation involving a first matrix, a second matrix and a third matrix, in another aspect, may include splitting the first matrix by row into P partitions and the third matrix by row into P partitions, P representing a number of hardware accelerators involved in the matrix to matrix operation, all partitions of the first matrix stored on a host computer and each partition of the third matrix stored on a different hardware accelerator. The method may further include splitting the second matrix by row into P partitions, each partition of the second matrix stored by column on the different hardware accelerator. The method may also include, each of the P hardware accelerators in parallel fetching a block of matrix A from the host computer corresponding to a locally stored block of matrix B. The method may further include, each of the P accelerators in parallel multiplying one block of the second matrix stored locally by corresponding columns of the partition of the first matrix fetched from the host computer and accumulating a result into a local partition of the third matrix. The method may also include, each of the P accelerators in parallel reading a block of the second matrix stored on its neighbor accelerator in a ring communication pattern and at the same time fetching a next block of the first matrix from the host computer, and repeating the multiplying, reading and fetching until all partitions of the second matrix have taken part in the multiplying.

A computer-implemented method of performing a matrix to matrix operation involving a first matrix, a second matrix and a third matrix, in another aspect, may include splitting the first matrix by row into P partitions and the third matrix by row into P partitions, P representing a number of hardware accelerators involved in the matrix to matrix operation, all partitions of the first matrix and the third matrix stored on a host computer. The method may also include splitting the second matrix by row into P partitions, all partitions of the second matrix stored by column on the host computer. The method may further include, each of the P hardware accelerators in parallel fetching a block of the third matrix wherein all of the P hardware accelerators work on a separate partition of the third matrix. The method may also include, each of the P hardware accelerators in parallel fetching a block of the first matrix and a block of the second matrix from the host computer, wherein the block of the first matrix and the block of the second matrix are multiplied to produce a contribution to a local partition of the third matrix. The method may also include, each of the P accelerators in parallel multiplying the block of the second matrix by corresponding columns of the partition of the first matrix fetched from the host computer and accumulating a result into the local partition of the third matrix. The method may further include, each of the P accelerators in parallel reading a block of the second matrix stored on its neighbor accelerator in a ring communication pattern and at the same time fetching a next block of the first matrix from the host computer, and repeating the multiplying, reading and fetching until all partitions of the second matrix have taken part in the multiplying.

DETAILED DESCRIPTION

A method, system and techniques of performing General Matrix to Matrix Multiplication (GEMM) on a heterogeneous system are disclosed.FIG. 1is a diagram illustrating an architectural overview of a heterogeneous system in one embodiment. A heterogeneous system may include multiple host nodes102a,102ninterconnected by a host interconnect104, and by multiple accelerators106a,106b,106c,106nconnected by and accelerator interconnect108. Accelerators106a,106b,106c,106nprovide compute functionality and are equipped with their own memory. Each accelerator106a,106b,106c,106nis attached to a host node102a,102nby means of an accelerator link. Examples of the accelerators may include, but are not limited to, a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and/or others. A host interconnect104may include interconnect architecture such as, but not limited to, NVLink (from NVIDIA, Santa Clara, Calif.), for instance, for connecting a host computer (e.g.,102a,102n) to an accelerator (e.g.,106a,106b,106c,106n), and may provide for low latency, high speed, direct memory access connectivity between devices, for example, of different instruction set architectures. A host interconnect104may include other standards or connections. In an embodiment the accelerators may be connected, for example, by interconnects such as Cache Coherent Interconnect for Accelerators (CCIX).

In one aspect, a method operates on matrices that can occupy more memory than available on a single accelerator (e.g., one of106a,106b,106c,106n), and that can take advantage of the multiple kinds of links available in the system. Given an m×k matrix A, a k×n matrix B, and an m×n matrix C, GEMM can be formalized as:
C←αAB+βC

The method in one embodiment operates on matrices with k»m>n. The method, system and techniques may handle cases in which: all the matrices reside in accelerator memory (e.g., memory associated with one or more of106a,106b,106c,106n); matrices B and C reside in accelerator memory (e.g., memory associated with one or more of106a,106b,106c,106n); and matrices B and C reside in host memory (e.g., memory associated with one or more of102a,102n), but their size does not exceed the accelerator memory minus temporary work buffers.

In some embodiments, linear layers of a neural network are computed on a system comprising multiple accelerators, for example, as shown inFIG. 1. In some embodiments, the computational data is exchanged among accelerators in a ring fashion, for example, with input being provided in the memory of the accelerators106a,106b,106c,106n.

In another aspect, the input may be provided in part in the memory of the accelerators106a,106b,106c,106n, in part in the memory of the hosts102a,102n. The data may be transferred onto and between the accelerators106a,106b,106c,106n. There may be overlapping transfers and computation between accelerators and between host-accelerator pairs.

In one aspect, input may be provided initially entirely in the memory of the hosts, and data then may be transferred onto and between the accelerators. There may be overlapping transfers and computation between accelerators and between host-accelerator pairs.

Briefly, an accelerator (e.g.,106a,106b,106c,106n) refers to a hardware device, for example, designed to improve the overall performance of the computer. An example of an accelerator is a graphics accelerator with its own processor such as a graphics processing unit (GPU) and memory such as random access memory (RAM). Hardware accelerators106a,106b,106c,106nperform a given specific function more efficiently than a process running on a general-purpose computer, which for example, runs its processes on a central processing unit (CPU). A host computer (e.g.,102a,102n) may be a general-purpose computer, for example, which includes a processor such as a CPU and also may include memory such as RAM, and/or others.

FIG. 2A-2Cillustrate an embodiment of a system configuration in performing GEMM in one embodiment. In this embodiment, the system includes P accelerators202,204,206and matrices A, B, and C reside in accelerator memory. For example, in this embodiment, all the matrices reside in accelerator memory. In one embodiment, a host processor or computer208and accelerators202,204,206are connected by a uniform mesh. Each accelerator202,204,206hosts n (in this example case, 2) sub-blocks of matrix B as well as one partition of matrix A.

FIG. 2Aillustrates an initial state of the system with three accelerators as an example. In one embodiment, matrices A and C are split by row into P partitions, stored in column-major order. A partition of matrix A is defined as a submatrix of size m/P by k, a partition of matrix B defined as a submatrix of size k/P by n while matrix C′s partition is defined as a submatrix of size m/P by n. Each partition of matrix A and matrix C is stored on a different accelerator. Matrix B is split by row into P partitions, stored in column-major order on the P accelerators' random access memory. Briefly, in a column-major order, the consecutive elements of a column reside next to each other in memory, e.g., contiguous in memory.

Partitions of matrices A, B and C are split into 2-dimensional submatrices to perform GEMM on smaller matrices. In some embodiments, when a partition P is split column-wise into x submatrices, a submatrix of matrix A has the size (m*k)/(P*x), that is, height (number of rows) is m/P, and width (number of columns) is k/x). Subsequently, matrix B is split among rows into x matrices as well. A submatrix of matrix B has the size (k*n)/(P*x), that is, height (number of rows) is k/(P*x), and width (number of columns) is n (in this example, width of matrix B is not split). InFIG. 2A, submatrices are, for simplicity, linearly labeled.

In one embodiment, each accelerator202,204,206, in parallel execution, multiplies one block of matrix B stored locally, by the corresponding columns of the partition of matrix A stored locally. The result is accumulated into the local partition of matrix C. If the multiplication is complete (e.g., if all blocks of matrix B have taken part in the multiplication) the method exits and the result is left in matrix C. As the input matrix C is distributed across multiple accelerators, the output matrix C is distributed in the same way at the end of the multiplication. Depending on the computation that follows, matrix C may be kept in the accelerators, or copied to the host memory and reassembled in a single matrix.

FIG. 2Billustrates a buffer flow with three accelerators as an example. Each accelerator202,204,206, in parallel, reads the block of B stored on its neighbor in a ring communication pattern. In one embodiment, the ring communication pattern employs a consistently same direction, for example, a clock-wise or a counter-clockwise ring. The pattern may be pre-defined, and both directions may have the same performance metrics. In the given example, accelerator 0 needs to compute C0. C0 is computed by performing A0*B0+A1*B1+A2*B2+A3*B3+A4*B4+A5*B5 (where ‘*’ represents a matrix multiplication). As B0 and B1 is given (residing on accelerator 0's memory), accelerator 0 requires A0 and A1 to perform first matrix multiplication. Subsequently, B2 and B3 are required that can be fetched from accelerator 2. At the same time A2 and A3 can be streamed from the host, if those partitions are not already in accelerator 0. Since accelerator 2 streamed B4 and B5 from accelerator 1 in a previous iteration step, accelerator 0 can fetch both B4 and B5 in the last two steps from accelerator 2. Hence, the order of performing matrix multiplication and transfers is derived by GEMM itself. In some embodiments, before computation and transfer start, the host208sets up the accelerators202,204,206and the host is responsible for orchestrating the cooperation between accelerators or informs the accelerator of the proper buffer flow. Each of the accelerators202,204,206then repeats the multiplication processing of the block of B that is read with corresponding columns of the partition of matrix A stored locally.

FIG. 2Billustrates the clock-wise movement of the sub-blocks of matrix B. These movements form a ring pattern among the accelerators, for instance, shown by buffer movements at210,212,214. Since each accelerator needs to see the entire matrix B, and n sub-blocks of matrix B are already within its own memory, (P−1)/P of matrix B is streamed into and out of the accelerator. For instance, if there are 3 partitions (P), ⅔ of matrix B is streamed into and out of an accelerator. In some embodiments, to allow computation and data exchange to proceed concurrently, a free work-buffer of the size of a submatrix of matrix B is allocated per accelerator. After a buffer has been used in a computation, it can be subsequently used freely to stream in a new submatrix of matrix B. As an example, B0 is used for computation in the first step by accelerator 0. In the first step, accelerator 0 also copies B2 to the empty work-buffer. In the subsequent step, B0 is no longer required by accelerator 0 so that accelerator 0 can transfer B3 from accelerator 2 into the buffer of B0.

FIG. 2Cillustrates a communication timeline of three accelerators as an example. In one embodiment, accelerators202,204,206execute in lock step. During each step one sub-block of matrix B is pushed to the direct neighbor, hence at each step a sub-block of matrix B is received. At the same time one multiplication is performed on an accelerator. After all of the sub-blocks of matrix B have been seen by each accelerator, communication can stop. Multiplications continue until all elements of matrix A are consumed, for example, as shown at time steps216.

FIGS. 3A-3Cillustrate an embodiment of a system configuration in performing GEMM in one embodiment. In this embodiment, matrix A resides in host memory, and matrices B and C in accelerator memory. P accelerators in the system participate in a GEMM operation in one embodiment.FIG. 3Ashows an initial state of the system configuration, in which the system includes three accelerators302,304,306and a host computer308participating in a GEMM operation in one embodiment. A host computer308and accelerators302,304,306are connected by a uniform mesh. Each accelerator302,304,306hosts n (in this case, 2) sub-blocks of matrix B. The entire matrix A resides on the host308.

Matrices A and C are split by row into P partitions, stored by column. All partitions of matrix A are stored in host memory of the host computer308, and each partition of matrix C is stored on a different accelerator, e.g.,302,304,306. Matrix B is split by row into P partitions, stored by column on the P accelerators302,304,306.

FIG. 3Bshows a buffer flow in a heterogeneous system with three accelerators in one embodiment. In this embodiment, each accelerator302,304,306, in parallel, fetches a block of matrix A from a memory of the host computer308, corresponding to the block of matrix B contained in it. The embodiment shown inFIG. 3Bmay leverage the same blocking mechanism described with reference toFIGS. 2A-2Cto rotate matrix B. In addition, submatrices of matrix A are transferred from the host to the individual accelerators depending on which submatrix of matrix A is required at a specific point in time within the accelerator. This is derived from a tiling of the entire GEMM and the rotation. For example, accelerator 0302computes C0 by performing A0*B0+A1*B1+A2*B2+A3*B3+A4*B4+A5*B5 (where ‘*’ represents a matrix multiplication). As B0 and B1 are given, and B2, B3, B4 and B5 are streamed in the mentioned sequence, submatrices A0, A1, A2, A3, A4, A5 need to be streamed into accelerator 0302. Each accelerator302,304,306, in parallel, multiplies one block of matrix B stored locally by the corresponding columns of the partition of matrix A fetched from the host memory. The result is accumulated into the local partition of matrix C. If the multiplication is complete (e.g., if all blocks of matrix B have been used) the computation stops and the result is left in C.

Each accelerator302,304,306, in parallel, reads a block of matrix B stored on its neighbor in a ring communication pattern. At the same time, each accelerator302,304,306fetches the next block of matrix A from the host memory of the host308. In one embodiment the same direction is used consistently in the transfers of matrix B blocks, for example, a clock-wise or a counter-clockwise direction can be employed. The processing repeats fetching and multiplying until all blocks are processed. For example, accelerator 0302, for example, needs to compute A0*B0+A1*B1+A2*B2+A3*B3+A4*B4+A5*B5 to obtain C0, whereas accelerator 1304, for example, computes A6*B0+A7*B1+A8*B2+A9*B3+A10*B4+A11*B5 to obtain C1, while accelerator 2306computes A12*B0+A13*B1+A14*B2+A15*B3+A16*B4+A17*B5 to obtain C2 (where ‘*’ represents a matrix multiplication). As a few submatrices of matrix B are given, each accelerator computes the matrix multiplications that can be performed with the given start B submatrices, for example, in case of accelerator 0302, A0*B0 and A1*B1, while the result is always accumulated with C0. As B2 is streamed into accelerator 0302, the next computation is B2*A2, and so on. The same mechanism applies to the other accelerators.

As an example,FIG. 3Billustrates a clock-wise movement of sub-blocks of matrix B. Movements form a ring pattern among the accelerators302,304,306, for instance, as shown by buffer movements310,312,314. Since each accelerator302,304,306needs to see the entire matrix B, and n sub-blocks of matrix B are already within its own memory, (P−1)/P of matrix B is streamed into and out of an accelerator302,304,306. As sub-blocks of matrix A are not in the accelerators memory, sub-blocks of A are streamed into the accelerators at the same time matrix B is rotated. In one embodiment, the sub-blocks of matrix A and sub-blocks of matrix B use different connections, and therefore, no interference exists. For example, a connection316is used between accelerator 0302and accelerator 1304, a connection318is used between accelerator 1304and accelerator 2306, and a connection318is used between accelerator 2306and accelerator 0302; a connection322is used between accelerator 0302and the host308, a connection324is used between accelerator 1304and the host308, and a connection326is used between accelerator 2306and the host308.

FIG. 3Cillustrates a communication timeline of three accelerators as an example. Accelerators302,304,306execute in lock step. In one embodiment, as no sub-block of matrix A is given, the first or initial step demands the host308to push one sub-block of matrix A into each accelerator memory. In some embodiments, the sequence of computation steps is pre-programmed and depends on the dimensions of the given matrices. For each further step, one further sub-block of matrix A is pushed from the host308into each accelerator302,304,306. This processing continues until the entire partition of matrix A is streamed into each accelerator302,304,306. During each step, one sub-block of matrix B is pushed to a direct neighbor, hence at each step a sub-block of matrix B is received. In some embodiments, the direct neighbor is pre-configured by the host. At the same time, one multiplication is performed. After all of the sub-blocks of matrix B have been seen by each accelerator302,304,306, communication for sub-blocks of matrix B can stop. Multiplications continue until all elements of matrix A are consumed.

FIGS. 4A-4Cillustrate an embodiment of a system configuration in performing GEMM in one embodiment. In this embodiment, P accelerators and a host computer in a heterogeneous system participate in GEMM operation, and matrices A, B, and C are stored or reside in host memory.FIG. 4Ashows an initial state of the system configuration, in which the system includes three accelerators402,404,406and a host computer408participating in a GEMM operation in one embodiment. Matrices A and C are split by row into P partitions, stored in column-major order. All partitions of matrices A and C are stored in host memory of the host408. Matrix B is split by row into P partitions, stored in column-major order on the host408as well. In one embodiment, the host408and accelerators402,404,406are connected by a uniform mesh. Entire matrix A and entire matrix B reside on the host408.

FIG. 4Bshows a buffer flow in a heterogeneous system with three accelerators in one embodiment. Each accelerator402,404,406, in parallel, fetches a block of matrix C, so that all accelerators402,404,406work on a separate partition of matrix C. Each accelerator402,404,406, in parallel, fetches one block of matrix A and one block of matrix B, such that they can be multiplied to produce a contribution to the local partition of matrix C.

Each accelerator402,404,406, in parallel, multiplies one block of matrix B by the corresponding columns of the partition of matrix A fetched from the host408. The result is accumulated into the local partition of matrix C. This processing continues until each accelerator out of P given accelerators loaded 1/P of matrices A, B and C.

As just 1/P of matrix B was read from the host, each accelerator402,404,406, in parallel, reads the block of matrix B stored on its neighbor in a ring communication pattern. At the same time, each accelerator402,404,406fetches the next block of matrix A. In one embodiment, the same direction is used consistently in the transfers of B, for example, a clock-wise or a counter-clockwise ring. If the multiplication is complete, the processing terminates, with the result in matrix C, otherwise the processing restarts with fetching of block of matrix A and a block of matrix B. As none of the matrices reside in the accelerator memories, matrix A, matrix B and matrix C are streamed from the host408to the accelerators402,404,406until 1/P of matrices A, B and C are streamed. As matrix B can leverage the ring pattern among the accelerators402,404,406, just one partition of matrix B is streamed into each accelerator, while the other P−1)/P of B are streamed into and out of the other accelerators. As sub-blocks of matrix A are not in the accelerators memory, sub-blocks of matrix A are streamed into the accelerators402,404,406at the same time matrix B is rotated and moved from the host memory of the host408. After a partition of matrix B is streamed into each accelerator402,404,406, only matrix A needs to be further streamed from host memory to the accelerators memory.

FIG. 4Cillustrates a communication timeline of three accelerators as an example. Accelerators402,404,406execute in lock step. As no sub-block of matrix A and matrix B is given, initially, the host408pushes one sub-block of matrix A and one sub-block of matrix B into each accelerator memory, for example, as shown at time step410. For each further step one further sub-block of matrix A and one further sub-block of matrix B is pushed from the host408into each accelerator, as shown at time step412. During each step (after the initial step), a cached sub-block of matrix B already in an accelerator is pushed to the direct neighbor. At the same time one multiplication is performed. In this example, sub-blocks of matrix B are being streamed into the accelerators' memory until 1/P per accelerator have been read from host memory.FIG. 4Cat412depicts how the re-use of blocks of matrix B is being performed. The subsequent steps following412do not require any blocks of matrix B from the host anymore as all of the blocks of matrix B are stored distributed across the accelerators' memory. Hence, subsequent steps can leverage the rotation of matrix B previously introduced to acquire all blocks of matrix B. After all of the sub-blocks of matrix B have been seen by each accelerator, communication for sub-blocks of matrix B can stop. Multiplications continue until all elements of matrix A are consumed.

FIG. 5is a diagram illustrating a machine learning model running on a heterogeneous hardware environment in one embodiment. As an example, particularly, the figure shows two accelerators running a convolutional neural network (CNN) flow. The CNNs, for example, perform classification of a given input image. Both accelerators execute separately all stages, up to the fully connected layer of a convolutional neural network. For example, accelerator 0502receives image data504and performs convolution and pooling (and/or other layer computations), and for instance, as shown, accelerator 0's memory holds feature maps506,508,510,512resulting from convolution and pooling and a fully connected layer matrix514(e.g., referred to as matrix B) Likewise, in one embodiment accelerator 1516receives image data518and performs convolution and pooling (and/or other layer computations), and for instance, as shown, accelerator 1's memory holds feature maps520,522,524,526resulting from convolution and pooling and matrix B528as a result of a fully connected layer.

At the fully connected layer530, matrix B is distributed by row-block among the accelerators. As the layout of matrix B does not suit the column-based embodiment, (P−1)/P of matrix B is exchanged with a neighbor accelerators. For instance, in this example, where P=2, ½ of matrix B is exchanged with a neighbor accelerator (accelerator 0 or accelerator 1), for example, as shown at532. Subsequently, according to an embodiment of a method in the present disclosure, the GEMM operation534is performed to achieve a column-distributed C (output). Thus, for example, each accelerator uses one or more other accelerators and corresponding memory space in performing computations of its neural network.

FIG. 6illustrates a neural network layer, for example, the fully connected layer of a convolutional neural network and GEMM performed on the neural network layer in one embodiment.FIG. 6, for example, highlights in detail a data shuffle532and GEMM534steps shown inFIG. 5, for instance, last step of the convolutional neural network. Taking a GEMM into consideration, the blocks602generated up to the fully connected layer represent the B matrix, while the output represents matrix C604(also shown at536inFIG. 5). The weights involved to perform a reduction of matrix B to matrix C, can be held by a weight matrix606that takes the role of matrix A in the embodiment. Any one of the GEMM operation techniques described above (e.g., with reference withFIGS. 2A-C,FIGS. 3A-C, orFIGS. 4A-C), for example, may be employed to perform this stage this of the CNN.

In some embodiments, the GEMM operation may include a pipelined operation. For instance, the P partitions of matrix B may be split each into n sub-blocks, and the method may operate on two sub-blocks at a time in a pipelined fashion. After an initial set up stage of multiplying the first sub-block of matrix B by the corresponding columns of matrix A, the stages of multiplying a sub-block of matrix B by the corresponding columns of matrix A and also reading a next sub-block stored in its neighboring accelerator may operate in parallel (e.g., executed concurrently), respectively on the i-th and the (i+1)-th sub-block of B. Such parallel processing overlaps communication and computation.

In some embodiments, by using additional buffers the copies (reading blocks from another computing element (e.g., host and/or accelerator)) can begin immediately, avoiding the initial setup stage. In these example embodiments, once the copies are completed the last block multiplications do not overlap.

In some embodiments, double ring pattern may be employed. With multiple sub-blocks per partition, it is possible to use two rings operating in parallel in opposite directions for two subsets of sub-blocks. Such bi-directional connections between the accelerators may utilize the full bandwidth available in both directions, for example, providing for efficient use of available bandwidth in a network of computer. This optimization also improves the performance of GEMM, for instance, in cases in which matrices A, B and C reside in accelerator memory.

In some embodiment matrix C can be copied from a hardware accelerator to a host memory device, for instance, depending on computation that follows. If, for example, matrix C is required for a new operation on the accelerators, it may be kept in the partitioned format on the accelerator memories. If, for example, the output of matrix C is required for further computations on the host, matrix C can be copied to the host memory and reassembled into a single matrix.

In some embodiments, e.g., responsive to determining that appropriate scatter, gather and/or strided access direct memory access (DMA) functionalities are available on the accelerator links and/or accelerator interconnect, the matrices need not be stored in blocks, but blocks can be created dynamically by copying from a contiguous source layout to a blocked representation in accelerator memory.

FIG. 7is a diagram illustrating a computer-implemented method in one embodiment of performing a matrix to matrix operation involving a first matrix, a second matrix and a third matrix. The method, in one embodiment, is performed on a system or apparatus which includes hardware accelerators interconnected, for example, via an accelerator interconnect. Each of the plurality of hardware accelerators in one embodiment includes a special processing unit and accelerator memory. The system or apparatus, in one embodiment, also include at least one host computer coupled to each of the plurality of hardware accelerators, for example, via an accelerator link. The host computer, in one embodiment, includes a general processing unit and host memory.

At702, the method includes splitting the first matrix by row into P partitions and the third matrix by row into P partitions. P represents the number of hardware accelerators involved in the matrix to matrix operation. Each partition of the first matrix and each partition of the third matrix stored on a different hardware accelerator.

At704, the method also includes splitting the second matrix by row into P partitions. Each partition of the second matrix is stored by column on the different hardware accelerator.

At706, each of the P hardware accelerators in parallel multiplies one block of the second matrix stored locally by corresponding columns of the partition of the first matrix stored locally and accumulates a result into a local partition of the third matrix.

At708, if multiplication is complete, the method exits. For example, if all blocks of the second matrix have taken part in the matrix multiplication, the method exists at710and the result is left in the third matrix.

Otherwise, at712, each of the P hardware accelerators in parallel reads a block of the second matrix stored on its neighbor accelerator in a ring communication pattern and multiplies the block of the second matrix, which is read, by the corresponding columns of the partition of the first matrix stored locally and accumulates a result into the local partition of the third matrix. Multiplying and reading repeats until all partitions of the second matrix have taken part in multiplication.

FIG. 8is a diagram illustrating a computer-implemented method in another embodiment of performing a matrix to matrix operation involving a first matrix, a second matrix and a third matrix. The method, in one embodiment, is performed on a system or apparatus which includes hardware accelerators interconnected, for example, via an accelerator interconnect. Each of the plurality of hardware accelerators in one embodiment includes a special processing unit and accelerator memory. The system or apparatus, in one embodiment, also include at least one host computer coupled to each of the plurality of hardware accelerators, for example, via an accelerator link. The host computer, in one embodiment, includes a general processing unit and host memory.

At802, the method includes splitting the first matrix by row into P partitions and the third matrix by row into P partitions, P representing a number of hardware accelerators involved in the matrix to matrix operation, all partitions of the first matrix stored on a host computer and each partition of the third matrix stored on a different hardware accelerator.

At804, the second matrix is split by row into P partitions, each partition of the second matrix stored by column on the different hardware accelerator.

At806, each of the P hardware accelerators in parallel fetches a block of the first matrix from the host computer corresponding to a locally stored block of the second matrix.

At808, each of the P accelerators in parallel multiplies one block of the second matrix stored locally by corresponding columns of the partition of the first matrix fetched from the host computer and accumulates a result into a local partition of the third matrix.

At810, if multiplication is complete, the method exits. For example, if all blocks of the second matrix have been used, the method exists at812and the result is left in the third matrix.

Otherwise, at814, each of the P accelerators in parallel reads a block of the second matrix stored on its neighbor accelerator in a ring communication pattern and at the same time fetches the next block of the first matrix from the host computer. In one embodiment, a clock-wise direction may be employed in the ring communication pattern. In another embodiment, a counter-clock-wise direction may be employed in the ring communication pattern. In one embodiment, the same consistent pattern is employed throughout the iterations (e.g., repetitions of808to814). The method repeats from the processing at808, for example, multiplying, reading and fetching until all partitions of the second matrix have taken part in the multiplying.

FIG. 9is a diagram illustrating a computer-implemented method in another embodiment of performing a matrix to matrix operation involving a first matrix, a second matrix and a third matrix. The method, in one embodiment, is performed on a system or apparatus which includes hardware accelerators interconnected, for example, via an accelerator interconnect. Each of the plurality of hardware accelerators in one embodiment includes a special processing unit and accelerator memory. The system or apparatus, in one embodiment, also include at least one host computer coupled to each of the plurality of hardware accelerators, for example, via an accelerator link. The host computer, in one embodiment, includes a general processing unit and host memory.

At902, the method includes splitting the first matrix by row into P partitions and the third matrix by row into P partitions. P represents a number of hardware accelerators involved in the matrix to matrix operation. All partitions of the first matrix and the third matrix stored on a host computer, for example, local memory associated with the host computer.

At904, the method includes splitting the second matrix by row into P partitions. All partitions of the second matrix are stored by column on the host computer.

At906, each of the P hardware accelerators in parallel fetches a block of the third matrix, for example, so that all of the P hardware accelerators work on a separate partition of the third matrix.

At908, each of the P hardware accelerators in parallel fetches a block of the first matrix and a block of the second matrix from the host computer. The fetched block of the first matrix and fetched the block of the second matrix can be multiplied to produce a contribution to a local partition of the third matrix.

At910, each of the P accelerators in parallel multiplies the block of the second matrix by corresponding columns of the partition of the first matrix fetched from the host computer and accumulates a result into the local partition of the third matrix.

At912, if loading of the second matrix is complete, that is, considering there are P accelerators participating in the method, if each of the P accelerators loaded 1/P of blocks of the second matrix into its accelerator memory, the processing continues to914, otherwise the processing continues to908.

At914, each of the P accelerators in parallel reads a block of the second matrix stored on its neighbor accelerator in a ring communication pattern and at the same time fetches the next block of the first matrix from the host computer.

At916, if multiplication is complete, the method exits. For example, if all blocks of the second matrix have been used, the method exits at918and the result is stored in the third matrix. Otherwise, the processing continues to908.

In one embodiment, a clock-wise direction may be employed in the ring communication pattern. In another embodiment, a counter-clock-wise direction may be employed in the ring communication pattern. In one embodiment, the same consistent pattern is employed throughout the iterations (e.g., repetitions of908to916). The method repeats from the processing at808, for example, multiplying, reading and fetching until all partitions of the second matrix have taken part in the multiplying.