A method, performed by one or more processors, for learning a rank of an convolutional neural network for image classification, comprising: presenting a digital image to the convolutional neural network for producing a classification output; decomposing a weight tensor of the convolutional neural network into a first weight tensor and a second weight tensor, wherein the decomposing comprises applying singular value decomposition to the weight tensor; applying a set of rank selection parameters to the first weight tensor and the second weight tensor to truncate the rank of the first weight tensor and the second weight tensor; during a backward pass of the convolutional neural network, updating the set of rank selection parameters by averaging updates calculated for each rank selection parameter of the set of rank selection parameters, wherein the same update is applied to all of the rank selection parameters in the set of rank selection parameters; and applying a threshold to each rank selection parameter of the set of rank selection parameters during computation of an inference in a forward pass of the convolutional neural network, wherein the threshold is applied to a rank selection parameter such that applied value of the rank selection parameter is set to 1 if an actual value of the rank selection parameter is greater than or equal to the threshold and the applied value of the rank selection parameter is set to 0 if the actual value of the rank selection parameter is less than the threshold, wherein in the case that the applied value of the rank selection parameter is set to 0, an output to which the applied value of the rank selection parameter is applied is removed without passing it to a subsequent layer. presenting a digital image to the convolutional neural network for producing a classification output; decomposing a weight tensor of the convolutional neural network into a first weight tensor and a second weight tensor, wherein the decomposing comprises applying singular value decomposition to the weight tensor; applying a set of rank selection parameters to the first weight tensor and the second weight tensor to truncate the rank of the first weight tensor and the second weight tensor; during a backward pass of the convolutional neural network, updating the set of rank selection parameters by averaging updates calculated for each rank selection parameter of the set of rank selection parameters, wherein the same update is applied to all of the rank selection parameters in the set of rank selection parameters; and applying a threshold to each rank selection parameter of the set of rank selection parameters during computation of an inference in a forward pass of the convolutional neural network, wherein the threshold is applied to a rank selection parameter such that applied value of the rank selection parameter is set to 1 if an actual value of the rank selection parameter is greater than or equal to the threshold and the applied value of the rank selection parameter is set to 0 if the actual value of the rank selection parameter is less than the threshold, wherein in the case that the applied value of the rank selection parameter is set to 0, an output to which the applied value of the rank selection parameter is applied is removed without passing it to a subsequent layer.