Patent ID: 11886982
Assignee: GRAPHCORE LIMITED
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 11:
12. A method implemented in a data processing system having a data storage and a plurality of processing nodes configured to perform operations to implement a process for performing an iteration of a neural network over a plurality of time periods, wherein the method comprises:
a first processing node of the plurality of processing nodes, during a first time period of the plurality of time periods, loading weights associated with a second layer of the neural network and required during a subsequent time period of the plurality of time periods for performing second calculations to determine activations of the second layer of the neural network from the data storage;
a second processing node of the plurality of processing nodes, during the first time period, performing first calculations to determine activations of a first layer of the neural network using weights associated with the first layer loaded from the data storage during an earlier one of the plurality of time periods;
the first processing node, receiving the activations of the first layer from the second processing node;
during the subsequent time period, performing the second calculations using the activations of the first layer and using the weights associated with the second layer loaded during the first time period;
wherein the plurality of processing nodes are configured to, as part of the process, use the activations of the second layer of the neural network to determine activations of one or more further layers of the neural network, including determining output values of the neural network,
wherein the plurality of processing nodes are configured to use the process to perform training of the neural network by:
determining the output values of the neural network by performing the process for performing the iteration of the neural network;
comparing the output values of the neural network to labels of the neural network to calculate loss; and
performing a backward propagation through the neural network of the loss to produce a revised set of weights of the neural network;

receiving, by the first processing node, the activations of the first layer from the second processing node during an exchange phase of the first processing node and the second processing node, wherein each of the plurality of time periods corresponds to a compute phase of the second processing node; and
executing, by the second processing node, a compiled code sequence comprising a synchronisation instruction indicating a barrier between the compute phase and the exchange phase.