A neural processing unit (100, 100', 200, 200', 370), comprising: a weight buffer (WBUF) configured to store weight values in an arrangement selected from a group comprising a structured weight sparsity arrangement and a random weight sparsity arrangement; a weight multiplexer configured to output one of the weight values stored in the weight buffer (WBUF) as a first operand value based on the selected weight sparsity arrangement; an activation buffer (ABUF) configured to store activation values; an activation multiplexer (AMUX) coupled to the activation buffer (ABUF), the activation multiplexer (AMUX) configured to output one of the activation values stored in the activation buffer (ABUF) as a second operand value, the second operand value and the first operand value forming an operand value pair; and a multiplier unit configured to output a product value for the operand value pair, wherein: the weight buffer (WBUF) comprises an array of weight registers (WREG), each weight register (WREG) being configured to store a weight value that is in the arrangement selected from the group comprising the structured weight sparsity arrangement and the random weight sparsity arrangement, the weight multiplexer is configured to select a weight register (WREG) based the weight sparsity arrangement of the weight values stored in the weight buffer (WBUF) and output the weight value stored in the selected weight register (WREG) as the first operand value, the activation buffer (ABUF) comprises an array of activation registers (AREG), each activation register (AREG) being configured to store an activation value, and the activation multiplexer (AMUX) is configured to select and output an activation value stored in the activation buffer (ABUF) as the second operand value, the second operand value corresponding to the first operand value and forming the operand value pair. a weight buffer (WBUF) configured to store weight values in an arrangement selected from a group comprising a structured weight sparsity arrangement and a random weight sparsity arrangement; a weight multiplexer configured to output one of the weight values stored in the weight buffer (WBUF) as a first operand value based on the selected weight sparsity arrangement; an activation buffer (ABUF) configured to store activation values; an activation multiplexer (AMUX) coupled to the activation buffer (ABUF), the activation multiplexer (AMUX) configured to output one of the activation values stored in the activation buffer (ABUF) as a second operand value, the second operand value and the first operand value forming an operand value pair; and a multiplier unit configured to output a product value for the operand value pair, wherein: the weight buffer (WBUF) comprises an array of weight registers (WREG), each weight register (WREG) being configured to store a weight value that is in the arrangement selected from the group comprising the structured weight sparsity arrangement and the random weight sparsity arrangement, the weight multiplexer is configured to select a weight register (WREG) based the weight sparsity arrangement of the weight values stored in the weight buffer (WBUF) and output the weight value stored in the selected weight register (WREG) as the first operand value, the activation buffer (ABUF) comprises an array of activation registers (AREG), each activation register (AREG) being configured to store an activation value, and the activation multiplexer (AMUX) is configured to select and output an activation value stored in the activation buffer (ABUF) as the second operand value, the second operand value corresponding to the first operand value and forming the operand value pair. the weight buffer (WBUF) comprises an array of weight registers (WREG), each weight register (WREG) being configured to store a weight value that is in the arrangement selected from the group comprising the structured weight sparsity arrangement and the random weight sparsity arrangement, the weight multiplexer is configured to select a weight register (WREG) based the weight sparsity arrangement of the weight values stored in the weight buffer (WBUF) and output the weight value stored in the selected weight register (WREG) as the first operand value, the activation buffer (ABUF) comprises an array of activation registers (AREG), each activation register (AREG) being configured to store an activation value, and the activation multiplexer (AMUX) is configured to select and output an activation value stored in the activation buffer (ABUF) as the second operand value, the second operand value corresponding to the first operand value and forming the operand value pair.