Patent ID: 11928571
Assignee: VISA INTERNATIONAL SERVICE ASSOCIATION
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A computer-implemented method for training a distributed machine learning model, comprising:
initializing a distributed machine learning model on a plurality of computing devices, the distributed machine learning model comprising a plurality of computational nodes, each computing device of the plurality of computing devices comprising a respective subset of the plurality of computational nodes, each computational node comprising at least one parameter;
receiving training data associated with a plurality of samples at a first computing device of the plurality of computing devices;
forward propagating each sample of the plurality of samples through the distributed machine learning model to generate an output for each sample of the plurality of samples;
determining a loss for each sample of the plurality of samples based on the output;
backward propagating the loss for each sample of the plurality of samples to each computing device of the plurality of computing devices;
asynchronously updating the at least one parameter of each computational node based on the loss for each sample as the loss for each sample is backward propagated while at least one of the plurality of samples is forward propagating through the distributed machine learning model;
storing, at each computing device of the plurality of computing devices, the at least one parameter of each computational node of the respective subset of the plurality of computational nodes as updated;
communicating, from each computing device of the plurality of computing devices to all other computing devices of the plurality of computing devices, data associated with the at least one parameter of each computational node of the respective subset of the plurality of computational nodes as updated, each of the other computing devices of the plurality of computing devices storing the at least one parameter of each computational node as updated;
determining the loss for a first sample of the plurality of samples satisfies a threshold associated with at least one computing device of the plurality of computing devices becoming unavailable;
in response to determining the loss for the first sample satisfies the threshold, determining a variance in the at least one parameter for each computational node; and
generating at least one new computational node on at least one computing device of the plurality of computing devices based on the variance of at least one computational node of the respective subset of the plurality of computational nodes.