Patent ID: 11928571
Assignee: VISA INTERNATIONAL SERVICE ASSOCIATION
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 9:
10. A system for training a distributed machine learning model, comprising:
a plurality of computing devices, each computing device of the plurality of computing devices comprising a respective subset of a plurality of computational nodes of a distributed machine learning model, each computational node comprising at least one parameter, each computing device comprising at least one processor and at least one non-transitory computer-readable medium including one or more instructions that, when executed by the at least one processor, cause the at least one processor to:
forward propagate each sample of a plurality of samples of training data through the distributed machine learning model to generate an output for each sample of the plurality of samples;
determine a loss for each sample of the plurality of samples based on the output;
backward propagate the loss for each sample of the plurality of samples to each computing device of the plurality of computing devices;
asynchronously update the at least one parameter of each computational node based on the loss for each sample as the loss for each sample is backward propagated while at least one of the plurality of samples is forward propagating through the distributed machine learning model;
store, at each computing device of the plurality of computing devices, the at least one parameter of each computational node of the respective subset of the plurality of computational nodes as updated;
communicate, from each computing device of the plurality of computing devices to all other computing devices of the plurality of computing devices, data associated with the at least one parameter of each computational node of the respective subset of the plurality of computational nodes as updated, each of the other computing devices of the plurality of computing devices storing the at least one parameter of each computational node as updated;
determine the loss for a first sample of the plurality of samples satisfies a threshold associated with at least one computing device of the plurality of computing devices becoming unavailable;
in response to determining the loss for the first sample satisfies the threshold, determine a variance in the at least one parameter for each computational node; and
generate at least one new computational node on at least one computing device of the plurality of computing devices based on the variance of at least one computational node of the respective subset of the plurality of computational nodes.