A method for parallel processing training data, the method comprising: training a respective replica of a machine learning model on each node of multiple nodes (125a-n) organized in an n -dimensional network topology, wherein n is an integer greater than or equal to 1, wherein a plurality of the multiple nodes are trained on a respective batch of training data in parallel, wherein one or more nodes in the plurality of nodes have been classified as degraded based on a hardware failure, a communication failure and/or a software issue, and wherein each non-degraded node stores a respective individual gradient vector resulting from training the respective replica for the node on the respective batch of training data; and combining the respective individual gradient vectors in the nodes to generate a final gradient vector by performing operations comprising, for a dimension of n dimensions in the network topology: designating each group of nodes along the dimension as either a forwarding group or a critical group based on whether the group of nodes includes any degraded nodes, for each non-degraded node in a forwarding group of nodes along the dimension, forwarding a respective individual gradient vector for the node along the dimension until forwarding the respective individual gradient vector to a respective receiving node in a respective critical group of nodes along the dimension, updating, for each receiving node, a respective individual gradient vector with an intermediate gradient vector, wherein the intermediate gradient vector is computed from the respective individual gradient vector and one or more received individual gradient vectors, performing a reduction on each critical group of nodes along the dimension to generate a respective partial final gradient vector for the critical group, updating, for each critical group of nodes, an individual gradient vector for a representative node with the respective partial final gradient vector; and determining the final gradient vector based on the partial final gradient vectors for each critical group of nodes by performing one or more further reductions of the partial final gradient vectors. training a respective replica of a machine learning model on each node of multiple nodes (125a-n) organized in an n -dimensional network topology, wherein n is an integer greater than or equal to 1, wherein a plurality of the multiple nodes are trained on a respective batch of training data in parallel, wherein one or more nodes in the plurality of nodes have been classified as degraded based on a hardware failure, a communication failure and/or a software issue, and wherein each non-degraded node stores a respective individual gradient vector resulting from training the respective replica for the node on the respective batch of training data; and combining the respective individual gradient vectors in the nodes to generate a final gradient vector by performing operations comprising, for a dimension of n dimensions in the network topology: designating each group of nodes along the dimension as either a forwarding group or a critical group based on whether the group of nodes includes any degraded nodes, for each non-degraded node in a forwarding group of nodes along the dimension, forwarding a respective individual gradient vector for the node along the dimension until forwarding the respective individual gradient vector to a respective receiving node in a respective critical group of nodes along the dimension, updating, for each receiving node, a respective individual gradient vector with an intermediate gradient vector, wherein the intermediate gradient vector is computed from the respective individual gradient vector and one or more received individual gradient vectors, performing a reduction on each critical group of nodes along the dimension to generate a respective partial final gradient vector for the critical group, updating, for each critical group of nodes, an individual gradient vector for a representative node with the respective partial final gradient vector; and determining the final gradient vector based on the partial final gradient vectors for each critical group of nodes by performing one or more further reductions of the partial final gradient vectors. designating each group of nodes along the dimension as either a forwarding group or a critical group based on whether the group of nodes includes any degraded nodes, for each non-degraded node in a forwarding group of nodes along the dimension, forwarding a respective individual gradient vector for the node along the dimension until forwarding the respective individual gradient vector to a respective receiving node in a respective critical group of nodes along the dimension, updating, for each receiving node, a respective individual gradient vector with an intermediate gradient vector, wherein the intermediate gradient vector is computed from the respective individual gradient vector and one or more received individual gradient vectors, performing a reduction on each critical group of nodes along the dimension to generate a respective partial final gradient vector for the critical group, updating, for each critical group of nodes, an individual gradient vector for a representative node with the respective partial final gradient vector; and determining the final gradient vector based on the partial final gradient vectors for each critical group of nodes by performing one or more further reductions of the partial final gradient vectors.