A method comprising: receiving, by a computing device, one or more parameter matrices of a neural network, wherein each parameter matrix corresponds to a layer of the neural network, wherein the row direction of the parameter matrix represents different inputs of the layer and the column direction of the parameter matrix represents different outputs of the layer, wherein at least a first parameter matrix has parameter values stored in a contiguous sparsity pattern, the contiguous sparsity pattern being a two-dimensional pattern composed of zero values and a plurality of non-square rectangular blocks of non-zero values, wherein each of the blocks has size nx1, where n is greater than one, the number of zero values in the first parameter matrix being greater than the number of non-zero values; storing the first parameter matrix in storage associated with the computing device; receiving, by the computing device, an input vector of activation values and storing the input vector of activation values in the storage associated with the computing device; and performing, by the computing device, an inference pass of the neural network to generate an output vector, including performing operations comprising: reading, from the storage associated with the computing device, a first activation value from the input vector, reading, from the storage associated with the computing device, a block of n contiguous non-zero parameter values, and generating multiple output values of the output vector by a process comprising multiplying the first activation value by the block of n contiguous non-zero parameter values before reading another activation value from the input vector. receiving, by a computing device, one or more parameter matrices of a neural network, wherein each parameter matrix corresponds to a layer of the neural network, wherein the row direction of the parameter matrix represents different inputs of the layer and the column direction of the parameter matrix represents different outputs of the layer, wherein at least a first parameter matrix has parameter values stored in a contiguous sparsity pattern, the contiguous sparsity pattern being a two-dimensional pattern composed of zero values and a plurality of non-square rectangular blocks of non-zero values, wherein each of the blocks has size nx1, where n is greater than one, the number of zero values in the first parameter matrix being greater than the number of non-zero values; storing the first parameter matrix in storage associated with the computing device; receiving, by the computing device, an input vector of activation values and storing the input vector of activation values in the storage associated with the computing device; and performing, by the computing device, an inference pass of the neural network to generate an output vector, including performing operations comprising: reading, from the storage associated with the computing device, a first activation value from the input vector, reading, from the storage associated with the computing device, a block of n contiguous non-zero parameter values, and generating multiple output values of the output vector by a process comprising multiplying the first activation value by the block of n contiguous non-zero parameter values before reading another activation value from the input vector. reading, from the storage associated with the computing device, a first activation value from the input vector, reading, from the storage associated with the computing device, a block of n contiguous non-zero parameter values, and generating multiple output values of the output vector by a process comprising multiplying the first activation value by the block of n contiguous non-zero parameter values before reading another activation value from the input vector.