Patent ID: 11886982
Assignee: GRAPHCORE LIMITED
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 17:
18. A non-transitory computer readable medium storing a computer program comprising sets of computer readable instructions, wherein when each of the sets of computer readable instructions are executed causes a data processing system to perform operations to implement a process for performing an iteration of a neural network over a plurality of time periods, the operations comprising:
during a first time period of the plurality of time periods, load weights associated with a second layer of the neural network and required during a subsequent time period of the plurality of time periods for performing second calculations to determine activations of the second layer of the neural from a data storage, wherein loading the load weights associated with the second layer is performed by a first processing node of the data processing system;
during the first time period, perform first calculations to determine activations of a first layer of the neural network using weights associated with the first layer loaded from the data storage during an earlier one of the plurality of time periods, wherein the first calculations are performed by a second processing node of the data processing system;
receive the weights associated with the second layer of the neural network at the second processing node from the first processing node;
during the subsequent time period, perform the second calculations using the weights associated with the second layer of the neural network,
wherein the plurality of processing nodes are configured to, as part of the process, use the activations of the second layer of the neural network to determine activations of one or more further layers of the neural network, including determining output values of the neural network, and
wherein the plurality of processing nodes are configured to use the process to perform training of the neural network by:
determining the output values of the neural network by performing the process for performing the iteration of the neural network;
comparing the output values of the neural network to labels of the neural network to calculate loss; and
performing a backward propagation through the neural network of the loss to produce a revised set of weights of the neural network; and

receive, by the first processing node, the activations of the first layer from the second processing node during an exchange phase of the first processing node and the second processing node, wherein each of the plurality of time periods corresponds to a compute phase of the second processing node; and
execute, by the second processing node, a compiled code sequence comprising a synchronisation instruction indicating a barrier between the compute phase and the exchange phase.