Patent ID: 11886934
Assignee: GRAPHCORE LIMITED
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 22:
23. A non-transitory computer readable storage medium storing a computer program comprising computer readable instructions, which when executed by a first processing node within a processing system having a plurality of processing nodes, causes the first processing node to:
take part in a reduce-scatter collective using an array of data items to obtain a first reduced subset of the data items, wherein the taking part in the reduce-scatter collective is performed by a processing unit of the first processing node, and wherein the first processing node is associated with a first set of processing nodes and with a second set of processing nodes, wherein the first processing node is the only processing node that is common to the first set of processing nodes and the second set of processing nodes, and wherein the reduce-scatter collective is further performed between a second processing node and a third processing node of the second set of processing nodes;
exchange the first reduced subset of the data items by participating in an all-reduce collective to obtain a second reduced subset of the data items, and wherein the all-reduce collective is performed among the first set of processing nodes; and
take part in an all-gather collective using the second reduced subset of the data items to obtain a reduced array of data items, wherein the taking part in the all-gather collective is performed by the processing unit of the first processing node, and wherein the all-gather collective is performed with the second processing node and the third processing node of the second set of processing nodes,
wherein the computer readable instructions, when executed by the at least one processor of the first processing node within a processing system having a plurality of processing nodes, further cause the first processing node to:
execute compute instructions during a compute phase and, following an internal barrier synchronisation, enter an external barrier synchronisation including at least one exchange phase,
take part in the reduce-scatter collective during the at least one exchange phase;
perform the exchange of the respective reduced subset of the array of data items by participating in the all-reduce collective during the at least one exchange phase; and
take part in the one or more all-gather collectives during the at least one exchange phase.