[22] Y. Wang, S. Tang, F. Zhu, L. Bai, R. Zhao, D. Qi, and W. Ouyang, “Revisiting the transferability of supervised pretraining: an mlp perspective,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 6
[23] R. M. French, “Catastrophic forgetting in connectionist networks,” Trends in cognitive sciences, 1999. 8
[24] V. V. Ramasesh, E. Dyer, and M. Raghu, “Anatomy of catastrophic forgetting: Hidden representations and task semantics,” in International Conference on Learning Representations, 2020. 8, 9, 28, 29
[25] S. I. Mirzadeh, A. Chaudhry, D. Yin, H. Hu, R. Pascanu, D. Gorur, and M. Farajtabar, "Wide neural networks forget less catastrophically," in Proceedings of the 39th International Conference on Machine Learning, 2022. 9
[26] S. I. Mirzadeh, A. Chaudhry, D. Yin, T. Nguyen, R. Pascanu, D. Gorur, and M. Farajtabar, "Architecture matters in continual learning," 2022. 9
[27] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, "How transferable are features in deep neural networks?," Advances in neural information processing systems, 2014. 9
[28] A. H. Williams, E. Kunz, S. Kornblith, and S. Linderman, "Generalized shape metrics on neural representations," Advances in Neural Information Processing Systems, 2021.
[29] Y. Lou, C. E. Mingard, and S. Hayou, “Feature learning and signal propagation in deep neural networks,” in International Conference on Machine Learning, 2022. 9
[30] N. Maheswaranathan, A. Williams, M. Golub, S. Ganguli, and D. Sussillo, "Universality and individuality in neural dynamics across large populations of recurrent networks," Advances in neural information processing systems, vol. 32, 2019. 9
[31] M. Raghu, J. Gilmer, J. Yosinski, and J. Sohl-Dickstein, "Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability," 2017. 9
[32] J. A. Thompson, Y. Bengio, and M. Schonwiesner, "The effect of task and training on intermediate representations in convolutional neural networks revealed with modified rv similarity analysis," 2019.
[33] A. Jacot, F. Gabriel, and C. Hongler, "Neural tangent kernel: Convergence and generalization in neural networks," Advances in neural information processing systems, 2018.
[34] A. Lewkowycz, Y. Bahri, E. Dyer, J. Sohl-Dickstein, and G. Gur-Ari, "The large learning rate phase of deep learning: the catapult mechanism," arXiv preprint arXiv:2003.02218, 2020.
[35] C. Wei, J. D. Lee, Q. Liu, and T. Ma, "Regularization matters: Generalization and optimization of neural nets vs their induced kernel," Advances in Neural Information Processing Systems, vol. 32, 2019. 9
[36] Y. Chen, A. Yuille, and Z. Zhou, "Which layer is learning faster? a systematic exploration of layer-wise convergence rate for deep neural networks," in The Eleventh International Conference on Learning Representations, 2023. 9
[37] B. Neyshabur, H. Sedghi, and C. Zhang, “What is being transferred in transfer learning?” in Advances in Neural Information Processing Systems (H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, eds.), 2020. 9
[38] S. I. Mirzadeh, A. Chaudhry, D. Yin, H. Hu, R. Pascanu, D. Gorur, and M. Farajtabar, "Wide neural networks forget less catastrophically," in International Conference on Machine Learning, 2022. 9
[39] S. Lee, S. S. Mannelli, C. Clopath, S. Goldt, and A. M. Saxe, "Maslow's hammer in catastrophic forgetting: Node re-use vs. node activation," in International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, Proceedings of Machine Learning Research, PMLR, 2022. 9