Patent ID: 11928571
Assignee: VISA INTERNATIONAL SERVICE ASSOCIATION
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 16:
17. A computer program product for training a distributed machine learning model, the computer program product comprising at least one non-transitory computer-readable medium including one or more instructions that, when executed by at least one processor, cause the at least one processor to:
initialize a respective subset of a plurality of computational nodes of a distributed machine learning model, each computational node comprising at least one parameter;
receive training data associated with a plurality of samples;
forward propagate each sample of the plurality of samples through the respective subset of the plurality of computational nodes of the distributed machine learning model to generate an intermediate output for each sample of the plurality of samples;
determine a gradient value associated with a loss associated with the respective subset of the plurality of computational nodes for each sample of the plurality of samples;
backward propagate the gradient value for each sample of the plurality of samples through the respective subset of the plurality of computational nodes of the distributed machine learning model;
asynchronously update the at least one parameter of each computational node based on the gradient value associated with the loss for each sample as the gradient value associated with the loss for each sample is backward propagated while at least one of the plurality of samples is forward propagating through the distributed machine learning model;
store the at least one parameter of each computational node of the respective subset of the plurality of computational nodes as updated;
communicate, to at least one computing device, data associated with the at least one parameter of each computational node of the respective subset of the plurality of computational nodes as updated to cause the at least one computing device to store the at least one parameter of each computational node as updated;
determine the loss for a first sample of the plurality of samples satisfies a threshold associated with at least one computing device of the plurality of computing devices becoming unavailable;
in response to determining the loss for the first sample satisfies the threshold, determine a variance in the at least one parameter for each computational node; and
generate at least one new computational node on at least one computing device of the plurality of computing devices based on the variance of at least one computational node of the respective subset of the plurality of computational nodes.