A method executed by a system comprising a parallel processing device with a plurality of independent processing units, the method comprising: receiving weight matrices of a layer of a plurality of layers of a signal generation neural network, wherein each layer of one or more layers in the neural network has a residual connection to a subsequent layer, and wherein each layer of the plurality of layers has a skip connection, wherein for each layer, a respective first weight matrix (202, 204) comprises values for a first matrix operation of the layer, and second and third weight matrices (242; 244) comprise values for second and third matrix operations of the layer, and wherein the second and third matrix operations depend on a result of the first matrix operation; interleaving rows of a first matrix for the layer by assigning groups of rows of the first matrix to respective thread blocks of a plurality of thread blocks, each thread block being a computation unit for execution by an independent processing unit of the plurality of independent processing units of the parallel processing device; assigning groups of rows of the second weight matrix to a first subset of the plurality of thread blocks (206, 207) and assigning groups of rows of the third weight matrix to a second subset of the plurality of thread blocks (208, 209); receiving, by the layer, an input vector; performing, by the plurality of thread blocks, the first matrix operation in parallel; performing, by the first subset of the plurality of thread blocks, the second matrix operation in parallel with performing, by the second subset of the plurality of thread blocks, the third matrix operation; and outputting a result computed from the second matrix operation to the residual connection in the network and outputting a result computed from the third matrix operation to the skip connection in the network. receiving weight matrices of a layer of a plurality of layers of a signal generation neural network, wherein each layer of one or more layers in the neural network has a residual connection to a subsequent layer, and wherein each layer of the plurality of layers has a skip connection, wherein for each layer, a respective first weight matrix (202, 204) comprises values for a first matrix operation of the layer, and second and third weight matrices (242; 244) comprise values for second and third matrix operations of the layer, and wherein the second and third matrix operations depend on a result of the first matrix operation; interleaving rows of a first matrix for the layer by assigning groups of rows of the first matrix to respective thread blocks of a plurality of thread blocks, each thread block being a computation unit for execution by an independent processing unit of the plurality of independent processing units of the parallel processing device; assigning groups of rows of the second weight matrix to a first subset of the plurality of thread blocks (206, 207) and assigning groups of rows of the third weight matrix to a second subset of the plurality of thread blocks (208, 209); receiving, by the layer, an input vector; performing, by the plurality of thread blocks, the first matrix operation in parallel; performing, by the first subset of the plurality of thread blocks, the second matrix operation in parallel with performing, by the second subset of the plurality of thread blocks, the third matrix operation; and outputting a result computed from the second matrix operation to the residual connection in the network and outputting a result computed from the third matrix operation to the skip connection in the network.