MULTI-PROCESSING UNIT INTERCONNECTED ACCELERATOR SYSTEMS AND CONFIGURATION TECHNIQUES

A compute system providing hierarchical scaling can include one or more sets of parallel processing units. The parallel processing units in a set can be organized into subsets of parallel processing units. Each parallel processing unit can be configurably couplable to two nearest neighbor parallel processing units in a same subset by two communication links, and each parallel processing unit can be configurably couplable to farthest neighbor parallel processing unit in the same subset by one communication link. Furthermore, each parallel processing unit can be configurably couplable to a corresponding parallel processing unit in the other subset by two communication links. The compute system can be configured by configuring the communication links of a set of parallel processing units into one or more compute clusters including a corresponding number of communication rings based on a specified compute parameter. Input data for computing on a given compute cluster divided and loaded onto respective parallel processing units of the given compute cluster. A function can be computed on the loaded input data by the given compute cluster using a parallel communication ring algorithm of the function.

BACKGROUND OF THE INVENTION

A current methodology for parallel/distributed training of deep neural networks includes applying synchronized large minibatch stochastic gradient descent (SDG) processing on many distributed computing nodes to explore data parallel based acceleration.

Referring toFIG. 1, an exemplary minibatch SDG process, including pseudo code, for running on a CPU host is illustrated. The process is subject to the synchronization parts bottlenecking the whole process of parallel acceleration. To reduce bottlenecking, building up the bandwidth of an accelerator-side network and/or reducing the frequency of host accelerator communication is needed, as illustrated inFIG. 2.

There are a number of algorithms for the synchronization of minibatch SDG processing. Some common inter-computing-note communication mode functions are the Reduce and All_Reduce functions. Referring now toFIG. 3, the Reduce function is illustrated. In the Reduce function, a set of values of each of a plurality nodes310-340are passed to a given one310of the plurality of nodes310-340, which adds the respective values together. The sum of the set of values is stored by the given node310. For example, a first node310receives the values of 5, 2, 7 and 4 from the plurality of nodes310-340, the first node adds the received values of 5, 2, 7 and 4 together, and the first node310stores the resulting sum of 18. The first node310also adds the values of 1, 3, 8 and 2 together and stores the resulting sum of 14. Referring now toFIG. 4, the All-Reduction function is illustrated. In the All_Reduce function, a set of values of each of a plurality of nodes410-440are passed to a given one410of the plurality of nodes410-440, which adds the respective values together. The set of sum values is broadcast by the given node410to the plurality of nodes410-440, and the plurality of nodes410-440store the set of sum values. For example, a first node410adds the values of 5, 2, 7 and 4 received from the plurality of nodes410-440together. The first node410also adds the values of 1, 3, 8 and 2 together. The first node410broadcast the set of sum values of 18 and 14 to the plurality of nodes410-440, which each store the set of sum values. As illustrated, the Reduce function and All_Reduce function are applied to a bunch of variables simultaneously.

Although a straightforward topology implementation of the Reduce and All_Reduce functions is a tree-based implementation, ring-based implementation can achieve a higher bandwidth utilization rate and efficiency. Referring now toFIG. 5, a conventional ring-based All_Reduce implementation on a distributed computing system is illustrated. In the All_Reduce function, each of N nodes of a distributed computing system communicate with two of its peer nodes2*(N−1) times. During the communications, a node sends and receives set of values. In the first N−1 iterations, received values are added to the values in the respective nodes' buffers. In the second N−1 iterations, received values replace the values held in the respective nodes' buffers. For example,FIG. 5. illustrates three nodes (N=3)510each buffering a respective set of input values. In a first iteration520, the first node passes a first set of input values to a second node. The second node adds the set of input values received from the first node to corresponding input values held by the second node. The first node also receives a third set of input values from a third node. The first node adds the set of input values received from the third node to corresponding values held by the first node. The second and third nodes also pass and add corresponding sets of input values in the first iteration520. In a second iteration530, the first node passes a third set of input values to the second node, which the second node adds to corresponding values held by the second node. The first node also receives a second set of values from the third node, which the first node adds to corresponding values held by the first node. The second and third nodes again pass and add corresponding sets of values in the second iteration530. In a third iteration540, the first node passes a second set of sum values to the second node, which the second node stores. The first node also receives a first set of sum values from the third node, which the first node stores. The second and third nodes also pass and store corresponding sets of the sum values. In a fourth iteration550, the first node passes a first set of sum values to the second node, which the second node stores. The first node also received a third set of the sum values from the third node, which the first node stores. The second and third nodes also pass and store corresponding sets of the sum values. After the fourth iteration, each node has the set of sum values. If the buffer is large enough, the ring-based All_Reduce function illustrated inFIG. 5can optimally utilize the available network of a distributed computing system.

However, there is a need for an improved chip-to-chip high-speed serial/deserialization (SerDes) interconnection so that such a distributed system for computing the All_Reduce function can be implemented within a cluster of chips instead of on distributed computers connected via slower ethernet, infiniband or the like communication links.

SUMMARY OF THE INVENTION

The present technology may best be understood by referring to the following description and accompanying drawings that are used to illustrate embodiments of the present technology directed toward multi-processing unit interconnected accelerator systems and configuration techniques thereof.

In one embodiment, a compute system can include one or more sets of parallel processing units. The parallel processing units in a set can be organized into subsets of parallel processing units. Each parallel processing unit can be configurably couplable to two nearest neighbor parallel processing units in a same subset by two communication links, and each parallel processing unit can be configurably couplable to a farthest neighbor parallel processing unit in the same subset by one communication link. Furthermore, each parallel processing unit can be configurably couplable to a corresponding parallel processing unit in the other subset by two communication links.

In another embodiment, a compute method can include configuring communication links of a set of parallel processing units into one or more compute clusters including a corresponding number of communication rings based on a specified compute parameter. A function can be computed on input data by the one or more compute clusters using a parallel communication ring algorithm. The function can be, but is not limited to, a Reduce function or a All_Reduce function.

DETAILED DESCRIPTION OF THE INVENTION

Some embodiments of the present technology which follow are presented in terms of routines, modules, logic blocks, and other symbolic representations of operations on data within one or more electronic devices. The descriptions and representations are the means used by those skilled in the art to most effectively convey the substance of their work to others skilled in the art. A routine, module, logic block and/or the like, is herein, and generally, conceived to be a self-consistent sequence of processes or instructions leading to a desired result. The processes are those including physical manipulations of physical quantities. Usually, though not necessarily, these physical manipulations take the form of electric or magnetic signals capable of being stored, transferred, compared and otherwise manipulated in an electronic device. For reasons of convenience, and with reference to common usage, these signals are referred to as data, bits, values, elements, symbols, characters, terms, numbers, strings, and/or the like with reference to embodiments of the present technology.

It should be borne in mind, however, that these terms are to be interpreted as referencing physical manipulations and quantities and are merely convenient labels and are to be interpreted further in view of terms commonly used in the art. Unless specifically stated otherwise as apparent from the following discussion, it is understood that through discussions of the present technology, discussions utilizing the terms such as “receiving,” and/or the like, refer to the actions and processes of an electronic device such as an electronic computing device that manipulates and transforms data. The data is represented as physical (e.g., electronic) quantities within the electronic device's logic circuits, registers, memories and/or the like, and is transformed into other data similarly represented as physical quantities within the electronic device.

In this application, the use of the disjunctive is intended to include the conjunctive. The use of definite or indefinite articles is not intended to indicate cardinality. In particular, a reference to “the” object or “a” object is intended to denote also one of a possible plurality of such objects. The use of the terms “comprises,” “comprising,” “includes,” “including” and the like specify the presence of stated elements, but do not preclude the presence or addition of one or more other elements and or groups thereof. It is also to be understood that although the terms first, second, etc. may be used herein to describe various elements, such elements should not be limited by these terms. These terms are used herein to distinguish one element from another. For example, a first element could be termed a second element, and similarly a second element could be termed a first element, without departing from the scope of embodiments. It is also to be understood that when an element is referred to as being “coupled” to another element, it may be directly or indirectly connected to the other element, or an intervening element may be present. In contrast, when an element is referred to as being “directly connected” to another element, there are not intervening elements present. It is also to be understood that the term “and or” includes any and all combinations of one or more of the associated elements. It is also to be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting.

Referring now toFIG. 6, a plurality of parallel processing units (PPUs) providing for hierarchical scaling, in accordance with aspects of the present technology, is shown. The plurality of PPUs can include one or more sets of eight PPUs Each PPU can include seven communication ports. The eight PPUs in a set can be organized in a first subset of four PPUs and a second subset of four PPUs. Each PPU can be configurably couplable to two nearest neighbor PPUs in a same subset by two communication links. Each PPU can also be configurably couplable to a farthest neighbor PPU in the same subset by one communication link. Each PPU can also be configurably couplable to a corresponding PPU in the other subset by two communication links. In one implementation, the PPUs can be coupled by configurable bi-directional communication links. The configurably couplable communications links can be configured as up to three communication rings710-730coupling eight PPUs together, as illustrated inFIG. 7. For example, a first bi-directional ring illustrated by the dashed lines710can communicatively link the first PPU305to the fourth PPU320, the fourth PPU320to the fifth PPU330, the fifth PPU330to the third PPU315, the third PPU315to the eighth PPU325, the eighth PPU325to the fifth PPU340, the fifth PPU340to the second PPU310, the second PPU310to the sixth PPU335, and the sixth PPU335back to the first PU305. There is also some communication links740in addition to the three communication rings710-730, as represented by the solid lines. It is appreciated that the communication rings710-730are just an exemplary set of three communication rings that can be configured from the communication links between the sets of nearest neighbors of PPUs in each subset, one bi-directional communication link between each set of farthest neighbors of PPUs in each subset, and two bi-directional communication links between corresponding PPUs of the two subsets of PPUs

The hierarchical scaling of the PPUs will be further explained with reference toFIG. 8. The communication links of the set of eight PPUs can be configured into one or more compute clusters including a corresponding number of communication rings based on a specified compute parameter, at810. In one implementation, the compute parameter can be a number of PPUs for a given compute cluster, such as eight, four or two PPUs for the given compute cluster. In another implementation, the compute parameter can be an amount of compute processing bandwidth. The compute processing bandwidth can be mapped to a given number of PPUs. In one implementation, the eight PPUs can be configured as one cluster of eight PPUs communicatively coupled by three bi-directional communication rings, as illustrated inFIG. 7. In other cases, an application may not need a cluster of eight PPUs to compute Reduce, All_Reduce or other similar functions. In yet other cases, such as cloud compute services, a customer may want to choose whether to pay for the compute processing bandwidth of eight, four or two PPUs.

Accordingly, in another implementation, the eight PPUs can be configured as two compute clusters905,910of four PPUs305-320,325-340each, as illustrated inFIG. 9A. The communication links can be configured by enabling a given subset of the communication links and disabling the other communications links such that the PPUs in each compute cluster905,910are communicatively coupled by two bi-directional communication rings915-920,925-930. For example, a first915and second920bi-directional ring can couple the first PPU305to the fourth PPU320, the fourth PPU320to the third PPU315, the third PPU315to the second PPU310, and the second PPU310to the first PPU305. Similarly, a third925and fourth930bi-directional ring can couple the fifth PPU340to the sixth PPU335, the sixth PPU325to the seventh PPU330, the seventh PPU330to the eighth PPU325, and the eighth PPU325to the fifth PPU340. The other communication links935can be disabled or utilized for other purposes. With two bi-direction communication rings, each compute cluster of four PPUs can be configured to compute different Reduce, All_Reduce or the like functions. The exemplary configuration illustrated inFIG. 9Ais just one possible configuration of the eight PPUs into two compute clusters of four PPUs. Other possible configurations of the eight PPUs into two compute cluster of four PPUs are illustrated inFIGS. 9B and 9C.

In yet other implementations, the eight PPUs can be configured as four compute clusters of1005,1010,1015,1020of two PPUs305-310,315-320,325-330,335-340, as illustrated inFIG. 10. The PPUs in each compute cluster1005,1010,1015,1020can be communicatively coupled by a respective bi-directional communication ring. For example, the first PPU305can be coupled to the second PPU310by first and second bi-directional communication links. The other communication links can be disabled or utilized for other purposes. Each compute cluster1050,1010,1015,1020of two PPUs can be configured to compute different Reduce, All_Reduce or the like functions. Again, the exemplary configuration illustrated inFIG. 10is just one possible configuration of the eight PPUs into four compute clusters of two PPUs.

In yet other implementations, the eight PPUs can be configured as a combination of one compute cluster1105of four PPUs305-320, and two compute clusters1110,1115of two PPUs325-330,335-340, as illustrated inFIG. 11. Again, each compute cluster can be configured to compute different Reduce, All_Reduce or the like functions. In addition, the exemplary configuration illustrated inFIG. 11is just one possible configuration of the eight PPUs into one compute cluster of four PPUs, and two compute cluster of two PPUs.

Referring again toFIG. 8, input data can be divided for computing on a given compute cluster and loaded onto respective PPUs of the given compute cluster, at820. For a compute cluster of eight PPUs coupled by three bi-directional communication rings, input data for a Reduce, All_Reduce or similar function can be divided into six groups, three groups for propagation in a first direction on the three parallel rings of bi-directional communication links and three groups for propagation in a second direction on the three parallel rings of the bi-directional communication links. For a compute cluster of four PPUs coupled by two bi-directional communication rings, the input data for the Reduce, All_Reduce or similar function can be divided into four groups, two groups for propagation in a first direction on the two parallel rings of bi-directional communication links and two groups for propagation in a second direction on the two parallel rings of the bi-directional communication links. For a compute cluster of two PPUs coupled by two bi-directional communication links, the input data for the Reduce, All_Reduce or similar function can be divided into two groups, one group for propagation in a first direction on the two bi-directional communication links and one group for propagation in a second direction on the two bi-directional communication links.

At830, the Reduce, All_Reduce or similar function can be computed on the input data by the given compute cluster using a parallel ring Reduce, All_Reduce or similar parallel ring algorithm. In a parallel ring algorithm, each of the plurality of PPUs (e.g., N nodes) communicates with its two nearest neighbor PPUs 2*(N−1) times, exchanging a respective group on a respective ring in a respective direction. In the first N−1 iterations, a given PPU sends respective values on respective rings to its nearest neighbors. In the first N−1 iterations, the given PPU also receives respective values on respective rings from its nearest neighbors, and adds the received values to respective values in the given PPU's buffer. In the second N−1 iterations, the given PPU sends respective values on respective rings to its nearest neighbors. In the second N−1 iterations, the given PPU also receives respective values on respective rings from its nearest neighbors, and replaces the respective values in the given PPU's buffer with the respective received values.

Referring now toFIG. 12, an exemplary computing system including a plurality of parallel processing units (PPUs), in accordance with aspects of the present technology, is shown. The exemplary computer system1200can include a plurality of parallel processing units (PPUs)1210,1220coupled together by one or more high-bandwidth inter-chip networks1230. The plurality of PPUs1210,1220can be, but are not limited to, a plurality of neural processing accelerators. The PPUs1210-1220can also be coupled to a plurality of host processing units1240,1250by one or more communication busses1260,1270. The one or more communications busses1260,1270can be, but are not limited to, one or more peripheral component interface express (PCIe) busses. The one or more host processing units1240,1250can be coupled to one or more host side networks1280by one or more network interface cards (NICs)1290,1295.

Referring now toFIG. 13, an exemplary parallel processing unit (PPU), in accordance with aspects of the present technology, is shown. The PPU1300can include a plurality of compute cores1305,1310, a plurality of inter-chip links (ICL)1315,1320, one or more high-bandwidth memory interfaces (HBM I/F)1325,1330, one or more communication processors1335, one or more direct memory access (DMA) controllers1340,1345, one or more command processors (CP)1350, one or more networks-on-chips (NoCs)1355, shared memory1360, and one or more high-bandwidth memory (HBM)1365,1370.

The PPU1300can also include one or more joint test action group (JTAG) engines1375, one or more inter-integrated circuit (I2C) interfaces and or serial peripheral interfaces (SPI)1380, one or more peripheral component interface express (PCIe) interfaces1385, one or more codecs (CoDec)1390, and the like. In one implementation, the plurality of compute cores1305,1310, the plurality of inter-chip links (ICL)1315,1320, one or more high-bandwidth memory interfaces (HBM I/F)1325,1330, one or more communication processors1335, one or more direct memory access (DMA) controllers1340,1345, one or more command processors (CP)1350, one or more networks-on-chips (NoCs)1355, shared memory1360, one or more high-bandwidth memory (HBM)1365,1370, one or more joint test action group (JTAG) engines1375, one or more inter-integrated circuit (12C) interfaces and or serial peripheral interfaces (SPI)1380, one or more peripheral component interface express (PCIe) interfaces1385, one or more codecs (CoDec)1390, and the like can be fabricated in one monolithic integrated circuits (ICs)

The ICLs1315,1320can be configured for chip-to-chip communication between a plurality of PPUs. In one implementation, the PPU1300can include seven ICLs1315,1320. The communication processor1335and direct memory access engines1340,1345can be configured to coordinate data sent and received through the ICLs1315,1320. The network-on-chip (NoC)1355can be configured to coordinate data movement between the compute cores1305,1310and the shared memory1360. The communication processor1335, direct memory access engines1340,1345, network on chip1355and high-bandwidth memory interfaces (HBM I/F)1325,1330can be configured to coordinate movement of data between the high-bandwidth memory1365,1370, the shared memory1360and the ICLs1315,1320. The command processor1350can be configured to serve as an interface between the PPU1300and one or more host processing units. The plurality of the PPUs1300can advantageously employed to compute a Reduce, All_Reduce or other similar functions as described above with reference toFIGS. 7, 8, 9A-9C, 10 and 11.

In accordance with aspects of the present technology, hierarchical enables a plurality of PPUs to be configured as one or more compute clusters coupled by a corresponding number of parallel communication rings. Hierarchical scaling the plurality of PPUs can be advantageous when an application requires a smaller portion of the computational resources of the plurality of PPUs than can be serviced by a compute cluster of a subset of the plurality of PPUs. Likewise, hierarchical scaling can be advantageously employed in a cloud computing platform to readily enable clients to purchase the computing bandwidth of a cluster of the PPUs instead of the entire plurality of PPUs.