Patent ID: 11948352
Assignee: AMAZON TECHNOLOGIES, INC.
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A distributed neural network training system comprising:
a plurality of processing nodes, each of the processing nodes comprising a computing device having:
a communication interface; and
neural network computation circuitry,

wherein each of the processing nodes is configured to:
perform a first iteration of a training process using the neural network computation circuitry with a first set of weights to generate a first output data set from an input data set associated with the processing node;
derive a set of local weight gradients for the processing node based on a comparison of the first output data set of the first iteration and a reference output data set;
update the first set of weights from the first iteration using the set of local weight gradients of the processing node to derive a set of speculative weights for the processing node;
perform a second iteration of the training process using the neural network computation circuitry with the set of speculative weights to generate a second output data set while waiting for a second set of weights to be ready;
obtain the second set of weights intended for the second iteration of the training process, wherein the second set of weights is computed from a set of averaged weight gradients calculated over a plurality of sets of local weight gradients, and wherein each set of local weight gradients is derived by a corresponding processing node of the plurality of processing nodes; and
compare the second set of weights to the set of speculative weights of the processing node to determine a difference between the second set of weights and the set of speculative weights, and based on determining the difference:
continue the training process using the second output data set that was generated from the set of speculative weights when the difference between the second set of weights and the set of speculative weights is at or below a threshold difference; or
repeat the second iteration of the training process using the second set of weights instead of the set of speculative weights when the difference between the second set of weights and the set of speculative weights is above the threshold difference.