A system to perform a convolution transpose operation between an input tensor comprising a plurality of input elements and a filter comprising a plurality of filter weights, the system comprising: one or more convolution engines that comprises hardware logic configured to perform a direct convolution between the input tensor and each of a plurality of sub-filters to generate a plurality of sub-output tensors comprising a plurality of output elements, each sub-filter comprising a subset of the filter weights of the filter; an interleave engine that comprises hardware logic configured to interleave the output elements of the plurality of sub-output tensors to generate a final output tensor for the convolution transpose; an input buffer that comprises memory configured to store input elements of the input tensor to be processed by the one or more convolution engines and hardware logic configured to provide the stored input elements to the one or more convolution engines for processing; and a coefficient buffer that comprises memory configured to store filter weights of the plurality of sub-filters to be processed by the one or more convolution engines and hardware logic configured to provide the stored filter weights to the one or more convolution engines for processing; wherein interleaving the output elements of the plurality of sub-output tensors to form the final output tensor for the convolution transpose comprises storing the output elements of each of the plurality of sub-output tensors in a storage unit and reading the output elements of the plurality of sub-output tensors from the storage unit in a predetermined order to form the final output tensor; wherein the system is configured to perform the convolution transpose over a plurality of hardware passes of the system by splitting the plurality of sub-filters over the plurality of hardware passes of the system such that in each hardware pass of the plurality of hardware passes the one or more convolution engines process a different subset of the plurality of sub-filters to generate a different subset of the plurality of sub-output tensors; wherein the plurality of sub-filters are split over the plurality of hardware passes such that the sub-filters that generate sub-output tensors with output elements that are in a same row of the final output tensor are processed in the same hardware pass of the system; wherein a hardware pass of the system comprises storing a plurality of input elements in the input buffer and storing a plurality of filter weights in the coefficient buffer and processing the input elements in the input buffer and the filter weights in the coefficient buffer. one or more convolution engines that comprises hardware logic configured to perform a direct convolution between the input tensor and each of a plurality of sub-filters to generate a plurality of sub-output tensors comprising a plurality of output elements, each sub-filter comprising a subset of the filter weights of the filter; an interleave engine that comprises hardware logic configured to interleave the output elements of the plurality of sub-output tensors to generate a final output tensor for the convolution transpose; an input buffer that comprises memory configured to store input elements of the input tensor to be processed by the one or more convolution engines and hardware logic configured to provide the stored input elements to the one or more convolution engines for processing; and a coefficient buffer that comprises memory configured to store filter weights of the plurality of sub-filters to be processed by the one or more convolution engines and hardware logic configured to provide the stored filter weights to the one or more convolution engines for processing; wherein interleaving the output elements of the plurality of sub-output tensors to form the final output tensor for the convolution transpose comprises storing the output elements of each of the plurality of sub-output tensors in a storage unit and reading the output elements of the plurality of sub-output tensors from the storage unit in a predetermined order to form the final output tensor; wherein the system is configured to perform the convolution transpose over a plurality of hardware passes of the system by splitting the plurality of sub-filters over the plurality of hardware passes of the system such that in each hardware pass of the plurality of hardware passes the one or more convolution engines process a different subset of the plurality of sub-filters to generate a different subset of the plurality of sub-output tensors; wherein the plurality of sub-filters are split over the plurality of hardware passes such that the sub-filters that generate sub-output tensors with output elements that are in a same row of the final output tensor are processed in the same hardware pass of the system; wherein a hardware pass of the system comprises storing a plurality of input elements in the input buffer and storing a plurality of filter weights in the coefficient buffer and processing the input elements in the input buffer and the filter weights in the coefficient buffer.