[5] T. Dettmers and L. Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.04840, 2019. 3, 6, 8, 10
[6] A. Dietrich, F. Gressmann, D. Orr, I. Chelombiev, D. Justus, and C. Luschi. Towards structured dynamic sparse pre-training of bert. arXiv preprint arXiv:2108.06277, 2021. 2, 10
[7] X. Dong and Y. Yang. Network pruning via transformable architecture search. Advances in Neural Information Processing Systems, 32, 2019. 17
[8] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 1
[9] E. Olsen, M. Dukhan, T. Gale, and K. Simonyan. Fast sparse convnets. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14629-14638, 2020. 2, 10
[10] U. Evci, T. Gale, J. Menick, P. S. Castro, and E. Olsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pages 2943–2952. PMLR, 2020. 2, 3, 4, 5, 6, 7, 8, 9, 16
[11] T. Gale, E. Elsen, and S. Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09574, 2019. 6
[12] T. Gale, M. Zaharia, C. Young, and E. Olsen. Sparsegpu kernels for deep learning. arXiv preprint arXiv:2006.10901, 2020. 10
[13] E. García-Martín, C. F. Rodrigues, G. Riley, and H. Grahn. Estimation of energy consumption in machine learning. Journal of Parallel and Distributed Computing, 134:75-88, 2019. 1
[14] S. Gray, A. Radford, and D. P. Kingma. GPU kernels for block-sparse weights. arXiv preprint arXiv:1711.09224, 3:2, 2017. 2
[15] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016. 6
[16] Y. He, P. Liu, Z. Wang, Z. Hu, and Y. Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 17
[17] Y. He, X. Zhang, and J. Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 1389–1397, 2017. 2
[18] J. Hestness, S. Narang, N. Ardalani, G. Diamos, H. Jun, H. Kianinejad, M. Patwary, M. Ali, Y. Yang, and Y. Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017. 1
[19] Z. Hou, M. Qin, F. Sun, X. Ma, K. Yuan, Y. Xu, Y.-K. Chen, R. Jin, Y. Xie, and S.-Y. Kung. Chex: Channel exploration for cnn model compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12287-12298, 2022. 10
[20] I. Hubara, B. Chmiel, M. Island, R. Banner, S. Naor, and D. Soudry. Accelerated sparse neural training: A provable and efficient method to find n: M transposable masks. arXiv preprint arXiv:2102.08124, 2021. 2
[21] E. Iofinova, A. Peste, M. Kurtz, and D. Alistarh. How well do sparse imagenet models transfer? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12266-12276, 2022. 10
[22] A. K. Jaiswal, H. Ma, T. Chen, Y. Ding, and Z. Wang. Training your sparse neural network better with any mask. In International Conference on Machine Learning, pages 9833-9844. PMLR, 2022. 2