Title: Distilling Expressive GNNs into MPNNs
Primary area: learning on graphs and other geometries & topologies
Keywords: Graph Neural Networks, Expressivity, Knowledge Distillation

Abstract: We show that expressive graph neural networks (GNNs) can be distilled into simple message passing graph neural networks (MPNNs). Benchmarking experiments exhibit a significant performance gap between expressive GNNs and simple MPNNs which we close with the knowledge distillation by 50\% to 100\%. Importantly, the distilled networks are 2 to 33 times faster than the expressive GNNs. While the assumption that higher expressivity leads to better predictions has served as a motivation for many recent GNNs, the strong predictive performance of distilled MPNNs on many popular benchmark datasets invalidates this assumption.

TL;DR: We show that expressive graph neural networks can be distilled into simple message passing graph neural networks for significant improvements in predictive performance and speed.