[26] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. (Cited on 2, 5, 6, 8, 9, 25, 26)
[27] V. P. Dwivedi, C. K. Joshi, T. Laurent, Y. Bengio, and X. Bresson. Benchmarking graph neural networks. arXiv, 2020. (Cited on 8, 9, 23, 25)
[28] V. P. Dwivedi, L. Rampasek, M. Galkin, A. Parviz, G. Wolf, A. T. Liu, and D. Beaini. Long range graph benchmark. In NeurIPS, 2022. (Cited on 10, 23)
[29] W. Falcon. Pytorch lightning. https://github.com/Lightning-AI/lightning, 2019. (Cited on 25, 26)
[30] M. Finzi, M. Welling, and A. G. Wilson. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups. In ICML, 2021. (Cited on 4, 19, 22)
[31] F. Fuchs, D. E. Worrall, V. Fischer, and M. Welling. Se(3)-transformers: 3d roto-translation equivariant attention networks. In NeurIPS, 2020. (Cited on 8, 22, 23)
[32] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl. Neural message passing for quantum chemistry. In ICML, 2017. (Cited on 23)
[33] A. Grover, E. Wang, A. Zweig, and S. Ermon. Stochastic optimization of sorting networks via continuous relaxations. In ICLR, 2019. (Cited on 4, 24)
[34] D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv, 2018. (Cited on 25)
[35] K. Hornik, M. B. Stinchcombe, and H. White. Multilayer feedforward networks are universal approximators. Neural Networks, 1989. (Cited on 2, 3, 22)
[36] W. Hu, B. Liu, J. Gomes, M. Zitnik, P. Liang, V. S. Pande, and J. Leskovec. Strategies for pre-training graph neural networks. In ICLR, 2020. (Cited on 10)
[37] A. Immer, T. F. A. van der Ouderaa, G. Ratsch, V. Fortuin, and M. van der Wilk. Invariance learning in deep neural networks with differentiable laplace approximations. In NeurIPS, 2022. (Cited on 22, 23)
[38] A. Jaegle, S. Borgeaud, J. Alayrac, C. Doersch, C. Ionescu, D. Ding, S. Koppula, D. Zoran, A. Brock, E. Shelhamer, O. J. Henaff, M. M. Botvinick, A. Zisserman, O. Vinyals, and J. Carreira. Perceiver IO: A general architecture for structured inputs & outputs. In ICLR, 2022. (Cited on 22)
[39] A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira. Perceiver: General perception with iterative attention. In ICML, 2021. (Cited on 22)
[40] C. K. Joshi, C. Bodnar, S. V. Mathis, T. Cohen, and P. Lio. On the expressive power of geometric graph neural networks. arXiv, 2023. (Cited on 1, 4, 22)
[41] S. Kaba, A. K. Mondal, Y. Zhang, Y. Bengio, and S. Ravanbakhsh. Equivalence with learned canonicalization functions. arXiv, 2022. (Cited on 2, 4, 5, 6, 7, 8, 15, 17, 18, 19, 21, 22, 26, 27, 29, 30)
[42] P. Kicki, P. Skrzypczynski, and M. Ozay. A new approach to design symmetry invariant neural networks. In IJCNN, 2021. (Cited on 2, 22)
[43] J. Kim, D. Nguyen, S. Min, S. Cho, M. Lee, H. Lee, and S. Hong. Pure transformers are powerful graph learners. In NeurIPS, 2022. (Cited on 22)
[44] J. Kim, S. Oh, S. Cho, and S. Hong. Equivariant hypergraph neural networks. In ECCV, 2022. (Cited on 22)
[45] J. Kim, S. Oh, and S. Hong. Transformers generalize deepsets and can be extended to graphs and hypergraphs. In NeurIPS, 2021. (Cited on 1, 22)
[46] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2015. (Cited on 25)
[47] D. P. Kingma and M. Welling. Auto-encoding variational bayes. In ICLR, 2014. (Cited on 2)
[48] T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017. (Cited on 7, 9, 10)
[49] J. Kohler, L. Klein, and F. Noé. Equivariant flows: Sampling configurations for multi-body systems with symmetric energies. arXiv, 2019. (Cited on 8)
[50] D. Kreuzer, D. Beaini, W. L. Hamilton, V. Létourneau, and P. Tossou. Rethinking graph transformers with spectral attention. NeurIPS, 2021. (Cited on 10, 22)
[51] J. Lee, Y. Lee, J. Kim, A. R. Kosiorek, S. Choi, and Y. W. Teh. Set transformer: A framework for attention-based permutation-invariant neural networks. In ICML, 2019. (Cited on 22)
[52] J. Li, D. Li, S. Savarese, and S. C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv, 2023. (Cited on 22)
[53] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In  $ICLR$ , 2019. (Cited on 25, 26)