Patent ID: 11948352
Assignee: AMAZON TECHNOLOGIES, INC.
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 19:
20. A non-transitory computer readable medium having stored therein instructions that, when executed by a computing device in a neural network training system, cause the computing device to implement operations including:
performing a first iteration of a training process using a first set of weights to generate a first output data set from an input dataset;
deriving a set of local weight gradients based on a comparison of the first output data set of the first iteration and an expected output data set;
updating the first set of weights from the first iteration using the set of local weight gradients to derive a set of speculative weights;
performing a second iteration of the training process using the set of speculative weights to generate a second output data set while waiting to obtain a second set of weights for the second iteration of the training process;
obtaining the second set of weights for the second iteration of the training process, wherein the second set of weights is computed from a set of averaged weight gradients calculated over a plurality of sets of local weight gradients, and wherein each set of local weight gradients is derived by a plurality of processing nodes that each include a corresponding computing device having a communication interface and neural network computation circuitry; and
determining a difference between the second set of weights and the set of speculative weights, and based on determining the difference:
continuing the training process using the second output data set generated from the set of speculative weights when the difference between the second set of weights and the set of speculative weights is at or below a threshold difference; or
repeating the second iteration of the training process using the second set of weights instead of the set of speculative weights when the difference between the second set of weights and the set of speculative weights is above the threshold difference.