[62] A. T. Suresh, Z. Sun, J. Ro, and F. Yu. Correlated quantization for distributed mean estimation and optimization. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 20856-20876. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/suresh22a.html. (Cited on pages 3 and 4)
[63] S. Swaminathan, D. Garg, R. Kannan, and F. Andres. Sparse low rank factorization for deep neural network compression. Neurocomputing, 398:185-196, 2020. ISSN 0925-2312. doi: https://doi.org/10.1016/j.neucom.2020.02.035. URL https://www.sciencedirect.com/science/article/pii/S0925231220302253. (Cited on page 1)
[64] M. Tahaei, E. Charlaix, V. Nia, A. Ghodsi, and M. Rezagholizadeh. KroneckerBERT: Significant compression of pre-trained language models through kronecker decomposition and knowledge distillation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2116-2127, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.nacl-main.154. URL https://aclanthology.org/2022.nacl-main.154. (Cited on page 1)
[65] C. Tai, T. Xiao, Y. Zhang, X. Wang, and W. E. Convolutional neural networks with low-rank regularization, 2016. URL https://arxiv.org/abs/1511.06067. (Cited on page 1)
[66] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. (Cited on page 9)
[67] J. A. Tropp, A. Yurtsever, M. Udell, and V. Cevher. Practical sketching algorithms for low-rank matrix approximation. SIAM Journal on Matrix Analysis and Applications, 38(4):1454-1485, 2017. doi: 10.1137/17M1111590. URL https://doi.org/10.1137/17M1111590. (Cited on page 2)
[68] J. A. Tropp, A. Yurtsever, M. Udell, and V. Cevher. Streaming low-rank matrix approximation with an application to scientific simulation. SIAM Journal on Scientific Computing, 41(4):A2430-A2463, 2019. doi: 10.1137/18M1201068. URL https://doi.org/10.1137/18M1201068. (Cited on page 45)
[69] M. Udell and A. Townsend. Why are big data matrices approximately low rank? SIAM Journal on Mathematics of Data Science, 1(1):144-160, 2019. doi: 10.1137/18M1183480. URL https://doi.org/10.1137/18M1183480. (Cited on page 1)
[70] M. Valipour, M. Rezagholizadeh, I. Kobyzev, and A. Ghodsi. DyLoRa: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation, 2023. URL https://arxiv.org/abs/2210.07558. (Cited on page 1)
[71] S. Vargaftik, R. Ben-Basat, A. Portnoy, G. Mendelson, Y. Ben-Itzhak, and M. Mitzenmacher. DRIVE: One-bit distributed mean estimation. In Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=KXRTmcv3dQ8. (Cited on page 3)
[72] S. Vargaftik, R. B. Basat, A. Portnoy, G. Mendelson, Y. B. Itzhak, and M. Mitzenmacher. Eden: Communication-efficient and robust distributed mean estimation for federated learning. In International Conference on Machine Learning, pages 21984-22014. PMLR, 2022. (Cited on page 3)
[73] R. Vershynin. Introduction to the non-asymptotic analysis of random matrices, 2011. URL https://arxiv.org/abs/1011.3027. (Cited on pages 21 and 22)
[74] R. Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018. doi: 10.1017/9781108231596. (Cited on page 22)
[75] M. J. Wainwright. *Basic tail and concentration bounds*, page 21–57. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019. doi: 10.1017/9781108627771.002. (Cited on page 21)