Before revision:  For instance, at 40\% structured sparsity, our sparsified DeiT-Base can achieve 0.42\% accuracy gain, at 33.13\% and 24.70\% running time savings, compared to its dense counterpart . Perhaps most surprisingly, we find that the proposed sparse (co-)training can even improve the ViT accuracy rather than compromising it, making sparsity a tantalizing "free lunch".
After revision:  For instance, at 40\% structured sparsity, our sparsified DeiT-Base can achieve 0.42\% accuracy gain, at 33.13\% and 24.70\% running time validate the effectiveness of our proposals which obtain significantly reduced computational cost and almost unimpaired generalization . Perhaps most surprisingly, we find that the proposed sparse (co-)training can even improve the ViT accuracy rather than compromising it, making sparsity a tantalizing "free lunch".