Patent ID: 11886934
Assignee: GRAPHCORE LIMITED
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 12:
13. A method implemented in a data processing system having a plurality of processing nodes, a first processing node of the plurality of processing nodes comprising a memory configured to store an array of data items, wherein the first processing node belongs to a first set of processing nodes and to a second set of processing nodes, wherein the first processing node is the only processing node that is common to the first set of processing nodes and the second set of processing nodes, wherein the first processing nodes comprises:
a processing unit; and
a gateway device connected to the processing unit, the gateway device configured to interface the first processing node with a host storage, and wherein the method comprises, at the first processing node:
taking part in a reduce-scatter collective using the array of data items to obtain a first reduced subset of the data items, wherein the taking part in the reduce-scatter collective is performed by the processing unit of the first processing node, and wherein the reduce-scatter collective is further performed between a second processing node and a third processing node of the second set of processing nodes;
exchanging the first reduced subset of the data items by participating in an all-reduce collective to obtain a second reduced subset of the data items, wherein the exchanging the first reduced subset is performed by the gateway device of the first processing node, and wherein the all-reduce collective is performed among the first set of processing nodes; and
taking part in an all-gather collective using the second reduced subset of the data items to obtain a reduced array of data items, wherein the taking part in the all-gather collective is performed by the processing unit of the first processing node, and wherein the all-gather collective is performed with the second processing node and the third processing node of the second set of processing nodes,

wherein each of the plurality of first processing nodes executes compute instructions during a compute phase and, following an internal barrier synchronisation, enters an external barrier synchronisation including at least one exchange phase, wherein each of the plurality of first processing nodes:
takes part in the reduce-scatter collective during the at least one exchange phase;
performs the exchange of the respective reduced subset of the array of data items by participating in the all-reduce collective during the at least one exchange phase; and
takes part in the one or more all-gather collectives during the at least one exchange phase.