A method, comprising: performing a matrix operation associated with a plurality of input matrices, wherein performing the matrix operation comprises: obtaining the plurality of input matrices from memory ; partitioning the plurality of input matrices into a plurality of input partitions (Fig. 5), wherein the plurality of input matrices is partitioned based on a number of a plurality of matrix processing chips; distributing the plurality of input partitions among the plurality of matrix processing chips , wherein each input partition is distributed to a particular matrix processing chip of the plurality of matrix processing chips, wherein the plurality of matrix processing chips is configured in a cyclic arrangement such that each matrix processing chip is communicatively coupled to a first neighbor matrix processing chip and a second neighbor matrix processing chip; performing a plurality of partial matrix operations using the plurality of matrix processing chips, wherein the plurality of partial matrix operations is performed in a plurality of stages and the method further comprises: simultaneously transmitting and receiving partial matrix data for a subsequent stage of the plurality of partial matrix operations to and from each of the first neighbor matrix processing chip and the second neighbor matrix processing chip while performing the plurality of partial matrix operations; and determining a result of the matrix operation based on the plurality of partial matrix operations. obtaining the plurality of input matrices from memory ; partitioning the plurality of input matrices into a plurality of input partitions (Fig. 5), wherein the plurality of input matrices is partitioned based on a number of a plurality of matrix processing chips; distributing the plurality of input partitions among the plurality of matrix processing chips , wherein each input partition is distributed to a particular matrix processing chip of the plurality of matrix processing chips, wherein the plurality of matrix processing chips is configured in a cyclic arrangement such that each matrix processing chip is communicatively coupled to a first neighbor matrix processing chip and a second neighbor matrix processing chip; performing a plurality of partial matrix operations using the plurality of matrix processing chips, wherein the plurality of partial matrix operations is performed in a plurality of stages and the method further comprises: simultaneously transmitting and receiving partial matrix data for a subsequent stage of the plurality of partial matrix operations to and from each of the first neighbor matrix processing chip and the second neighbor matrix processing chip while performing the plurality of partial matrix operations; and determining a result of the matrix operation based on the plurality of partial matrix operations. simultaneously transmitting and receiving partial matrix data for a subsequent stage of the plurality of partial matrix operations to and from each of the first neighbor matrix processing chip and the second neighbor matrix processing chip while performing the plurality of partial matrix operations; and determining a result of the matrix operation based on the plurality of partial matrix operations. <EOS>