Patent ID: 11928598
Assignee: ALIBABA GROUP HOLDING LIMITED
Field: Computer technology (Electrical engineering)
Classification: CPC G  H | IPC G

Claim 0:
1. A method implemented by a neural network processing system, the neural network processing system comprising a plurality of heterogeneous computation units (HCUs), the method comprising:
computing, by the plurality of HCUs, a plurality of first gradients from a plurality of first samples;
aggregating the plurality of first gradients to generate an aggregated gradient;
computing, by the plurality of HCUs, a plurality of second gradients from a plurality of second samples;
aggregating, at each of the plurality of HCUs, the aggregated gradient of the plurality of first gradients with the each HCU's corresponding second gradient among the second plurality of second gradients to generate a local gradient update for the each HCU; and
updating, at each of the plurality of HCUs, a local copy of a neural network with the corresponding local gradient update for the each HCU.