[22] J. Hron, Y. Bahri, J. Sohl-Dickstein, and R. Novak. Infinite attention: Nngp and ntk for deep attention networks. In Proceedings of the International Conference on Machine Learning (ICML), 2020.
[23] A. Iscen, T. Furon, V. Gripon, M. Rabbat, and H. Jégou. Memory vectors for similarity search in high-dimensional spaces. IEEE transactions on big data, 4(1):65-77, 2017.
[24] A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems (NeurIPS), 2018.
[25] S. Jelassi, M. Sander, and Y. Li. Vision transformers provably learn spatial structure. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[26] Y. Jiang and C. Pehlevan. Associative memory in iterated overparameterized sigmoid autoencoders. In Proceedings of the International Conference on Machine Learning (ICML), 2020.
[27] A. Joulin and T. Mikolov. Inferring algorithmic patterns with stack-augmented recurrent nets. Advances in Neural Information Processing Systems (NIPS), 2015.
[28] S. Karp, E. Winston, Y. Li, and A. Singh. Local signal adaptivity: Provable feature learning in neural networks beyond kernels. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[29] T. Kohonen. Correlation matrix memories. IEEE Transactions on Computers, 1972.
[30] D. Krotov and J. J. Hopfield. Dense associative memory for pattern recognition. Advances in Neural Information Processing Systems (NIPS), 2016.
[31] Y. Li, Y. Li, and A. Risteski. How do transformers learn topic structure: Towards a mechanistic understanding. In Proceedings of the International Conference on Machine Learning (ICML), 2023.
[32] B. Liu, J. T. Ash, S. Goel, A. Krishnamurthy, and C. Zhang. Transformers learn shortcuts to automata. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
[33] Z. Liu, O. Kitouni, N. S. Nolte, E. Michaud, M. Tegmark, and M. Williams. Towards understanding grokking: An effective theory of representation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[34] R. McEliece, E. Posner, E. Rodemich, and S. Venkatesh. The capacity of the hopfield associative memory. IEEE transactions on Information Theory, 33(4):461-482, 1987.
[35] K. Meng, D. Bau, A. Andonian, and Y. Belinkov. Locating and editing factual associations in gpt. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[36] W. Merrill, A. Sabharwal, and N. A. Smith. Saturated transformers are constant-depth threshold circuits. Transactions of the Association for Computational Linguistics, 10:843-856, 2022.
[37] T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. In Proceedings of the International Conference on Learning Representations (ICLR), 2013.
[38] S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022.
[39] N. Nanda, L. Chan, T. Liberum, J. Smith, and J. Steinhardt. Progress measures for grokking via mechanistic interpretability. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.