SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 18: The Jaccard index between the selected pruning criteria in time. For each of the methods in the rows, we run the standard DST procedure. Then, every 3200 iterations, we verify what set of weights would be chosen at such a point by the remaining pruning criteria. Note that this relation does not need to be symmetrical.
Figure 19: The number of weights retained by all the pruning criteria relative to the number of all unpruned weights (in blue). The number of weights removed by all the pruning criteria relative to the number of all pruned weights (in pink).
Figure 20: The mean standard deviation computed for the Jaccard index between the sets of weights chosen for removal during the first update of the sparse connectivity.
Figure 21: The standard deviation of the same index computed between the masks obtained by different pruning criteria at the end of training. The rows and columns represent the pruning criteria between which the index is computed.
Figure 22: The performance of the pruning criteria with respect to the density on different datasets. Each row corresponds to a different model-dataset pair. The first column corresponds to the random growth, the second one corresponds to the gradient growth. Please note the logarithmic scale in the x-axis. We may observe that the differences between the methods become more visible predominantly in the low-density regime. DST almost always outperforms the static sparse training (cyan line).
C Magnitude
$\mathcal{C}_{\mathrm{MEST}}$
$\mathcal{C}_{\mathrm{RSensitivity}}$
CSNIP
Figure 23: The validation accuracy versus the training epoch for different pruning criteria and densities on the LeNet-5-Caffe (FashionMNIST dataset).
$\mathcal{C}_{\mathrm{SET}}$
CMagnitude
$\mathcal{C}_{\mathrm{SET}}$
Figure 24: The validation accuracy versus the training epoch for different pruning criteria and densities on the small-MLP (Higgs dataset).
CSNIP
$\mathcal{C}_{\mathrm{RSensitivity}}$
CSNIP
Figure 25: The validation accuracy versus the training epoch for different pruning criteria and densities on the small CNN (CIFAR10 dataset).
$\mathcal{C}_{\mathrm{RSensitivity}}$
$\mathcal{C}_{\mathrm{SET}}$
Figure 26: The validation accuracy versus the training epoch for different pruning criteria and densities on the ResNet-56 (CIFAR10 dataset).
Figure 27: The validation accuracy versus the training epoch for different pruning criteria and densities on the ResNet-56 (CIFAR100 dataset).
Figure 28: The validation accuracy versus the training epoch for different pruning criteria and densities on the large MLP (CIFAR10 dataset).
Figure 29: The validation accuracy versus the training epoch for different pruning criteria and densities on the EfficientNet (TinyImagenet dataset).
Figure 30: The validation accuracy versus the training epoch for different pruning criteria and densities on the VGG (CIFAR100 dataset).