A method comprising: receiving a request to perform computations for a neural network on a hardware circuit having a matrix computation unit, wherein the matrix computation unit is a systolic array which includes multiple cells arranged in rows and columns, the request specifying a transpose operation to be performed on a first neural network matrix associated with the neural network; and generating instructions that when executed by the hardware circuit cause the hardware circuit to transpose the first neural network matrix by performing first operations, wherein the first operations comprise: determining that the first neural network matrix is not a i x i matrix,where i is equal to the number of columns or rows of the cells in the systolic array; in response, updating the first neural network matrix to generate a i x i matrix by zero-padding the first neural network matrix prior to performing all iterations of the second operations; and repeatedly performing the following second operations: for a current subdivision of the first neural network matrix that divides the first neural network matrix into one or more current submatrices: updating the first neural network matrix by swapping an upper right quadrant and a lower left quadrant of each current submatrix in the current subdivision using the matrix computation unit, and subdividing each current submatrix in the current subdivision into a respective plurality of new submatrices to update the current subdivision, each of the respective plurality of new submatrices being a respective quadrant of the current submatrix; and after performing all iterations of the second operations, converting the first neural network matrix to its condition before the update by removing the zeros padded during the update. receiving a request to perform computations for a neural network on a hardware circuit having a matrix computation unit, wherein the matrix computation unit is a systolic array which includes multiple cells arranged in rows and columns, the request specifying a transpose operation to be performed on a first neural network matrix associated with the neural network; and generating instructions that when executed by the hardware circuit cause the hardware circuit to transpose the first neural network matrix by performing first operations, wherein the first operations comprise: determining that the first neural network matrix is not a i x i matrix,where i is equal to the number of columns or rows of the cells in the systolic array; in response, updating the first neural network matrix to generate a i x i matrix by zero-padding the first neural network matrix prior to performing all iterations of the second operations; and repeatedly performing the following second operations: for a current subdivision of the first neural network matrix that divides the first neural network matrix into one or more current submatrices: updating the first neural network matrix by swapping an upper right quadrant and a lower left quadrant of each current submatrix in the current subdivision using the matrix computation unit, and subdividing each current submatrix in the current subdivision into a respective plurality of new submatrices to update the current subdivision, each of the respective plurality of new submatrices being a respective quadrant of the current submatrix; and determining that the first neural network matrix is not a i x i matrix,where i is equal to the number of columns or rows of the cells in the systolic array; in response, updating the first neural network matrix to generate a i x i matrix by zero-padding the first neural network matrix prior to performing all iterations of the second operations; and repeatedly performing the following second operations: for a current subdivision of the first neural network matrix that divides the first neural network matrix into one or more current submatrices: updating the first neural network matrix by swapping an upper right quadrant and a lower left quadrant of each current submatrix in the current subdivision using the matrix computation unit, and subdividing each current submatrix in the current subdivision into a respective plurality of new submatrices to update the current subdivision, each of the respective plurality of new submatrices being a respective quadrant of the current submatrix; and updating the first neural network matrix by swapping an upper right quadrant and a lower left quadrant of each current submatrix in the current subdivision using the matrix computation unit, and subdividing each current submatrix in the current subdivision into a respective plurality of new submatrices to update the current subdivision, each of the respective plurality of new submatrices being a respective quadrant of the current submatrix; and after performing all iterations of the second operations, converting the first neural network matrix to its condition before the update by removing the zeros padded during the update.