Patent Document ID: 20180240039
Application ID: 15707104
Patent Flag: 0

Claim One:
1. A method for parallel processing training data with low latency, the method comprising: training a respective replica of a machine learning model on each node of a plurality of nodes organized in a torus topology comprising rows and columns of nodes, wherein each node is trained on a respective batch of training data in parallel, whereby after the training each node holds a respective gradient vector resulting from the training; combining the respective gradient vectors in the nodes to generate a final gradient vector by performing operations comprising: performing, by code executing on the nodes, a first circle algorithm, the first circle algorithm being an improved rotated pincer algorithm, on the rows of the torus in parallel without doing a broadcast to generate in each row combined data for each respective disjoint field of the gradient vector, whereby when the row processing is complete, each column contains all the combined data for a respective same disjoint field of the gradient vector data; and then performing, by code executing on the nodes, a second circle algorithm in each column of the columns of the torus, including broadcasting a respective final column result in each column to all nodes in the column; and then replicating the final column results across each row of the torus, whereby each node of the torus holds the final gradient vector.