Patent ID: 11948352
Assignee: AMAZON TECHNOLOGIES, INC.
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 3:
4. A method comprising:
performing, by a processing node of a plurality of processing nodes in a neural network training system, a first iteration of a training process using a first set of weights to generate a first output data set from an input dataset, wherein each processing node includes a computing device having a communication interface and neural network computation circuitry;
deriving, by the processing node, a set of local weight gradients based on a comparison of the first output data set of the first iteration and a reference output data set;
updating, by the processing node, the first set of weights from the first iteration using the set of local weight gradients to derive a set of speculative weights;
performing, by the processing node, a second iteration of the training process using the set of speculative weights to generate a second output data set while waiting to obtain a second set of weights derived with the plurality of processing nodes for the second iteration of the training process;
obtaining the second set of weights for the second iteration of the training process, wherein the second set of weights is computed from a set of averaged weight gradients calculated over a plurality of sets of local weight gradients, and wherein each set of local weight gradients is derived by a corresponding processing node of the plurality of processing nodes; and
determining a difference between the second set of weights and the set of speculative weights, and based on determining the difference:
continuing the training process using the second output data set generated from the set of speculative weights when the difference between the second set of weights and the set of speculative weights is at or below a threshold difference; or
repeating the second iteration of the training process using the second set of weights instead of the set of speculative weights when the difference between the second set of weights and the set of speculative weights is above the threshold difference.