Where Did the Gap Go? Reassessing the Long-Range Graph Benchmark
Authors: Jan Tönshoff
Martin Ritzert
Eran Rosenbluth
Martin Grohe
Venue: Accepted by TMLR ()

Abstract: The recent Long-Range Graph Benchmark (LRGB, Dwivedi et al. 2022) introduced a set of graph learning tasks strongly dependent on long-range interaction between vertices. Empirical evidence suggests that on these tasks Graph Transformers significantly outperform Message Passing GNNs (MPGNNs). In this paper, we carefully reevaluate multiple MPGNN baselines as well as the Graph Transformer GPS (Rampášek et al. 2022) on LRGB. Through a rigorous empirical analysis, we demonstrate that the reported performance gap is overestimated due to suboptimal hyperparameter choices. It is noteworthy that across multiple datasets the performance gap completely vanishes after basic hyperparameter optimization. In addition, we discuss the impact of lacking feature normalization for LRGB's vision datasets and highlight a spurious implementation of LRGB's link prediction metric. The principal aim of our paper is to establish a higher standard of empirical rigor within the graph machine learning community.

Published in Transactions on Machine Learning Research (05/2024)
Where Did the Gap Go? Reassessing the Long-Range Graph Benchmark
Jan Tönshoff
RWTH Aochen University
Martin Ritzert
Georg-August- EelhgusstUnnnrrrsaate Universität Göttingen
Eran Rosenbluth RWTH Aachen University
Martin Grohe
RWTH Auchen University
Reviewed on OpenReview: https: //tps://ol opwwrevee.......eoemeo net/ forum? id- HmOWX666Ky
Abstract
The recent Long-Range Graph Benchmark (LRGB, Dwivedi et al. 2022) introduced set of graph learning tasks strongly dependent on long-range interaction between vertices. Em- pirical evidence suggests that on these tasks Graph Transformers significantly outperform Mcssage Passing GNNs (MPGNNs). In this paper, we carcfully roevaluate multiple MPGNN baselines as well the Graph Transformer GPS (Rampáek et. al. 2022) on LRGB. Through rigorous empirical analysis, demonstrate that the reported performance gap is overes- timated due to suboptimal hyperparameter choices. It is noteworthy that across multiple datasets the performance gap completely vanishes after basic hyperparameter optimiza- tion. In addition, we discuss the impact of lacking feature normalization for LRGB's vision datasets and highlight a spurious implementation of LRGB's link prediction metric. The principal aim of our paper is establish a higher standard of empirical rigor within the graph machine learning community.
1
Introduction
Graph Transformers (GTss) have recently emerged as popular alternative to conventional Message Pass- ing Graph Neural Networks (MPGNNs) which dominated deep learning on graphs for years. central premise underlying GTs is their ability to model long-range interactions between vertices through a global allention mechanism. This could give GTs an advantage on tasks where MPGNNs may be limited through phenomenons like over-smoothing, over-squashing, and under-reaching, thereby justifying the significant runtime overhcad of sclf-attention.
The Long-Range Graph Benchmark (LRGB) has been roduedd by Dwivedi et al. 2022 as collection of five datasets:
Reptides-func and Peptides-stie strruct are graph-level classification and regression tasks, respectively. Their aim to predict various properties of peptides which are modelled as molecular graphs.
IuscalVOC-St and COCO-SP model semantic image segmentation as node-olassification task on superpixel graphs.
PCQM-Cone Contact link prediction task on molecular graphs. The task to predict pairs of vertices which are distant in the graph but in contact in 3D space.

---

Published in Transactions on Machine Learning Research (05/2024)
Dataset
Peptides- Func
Peptides-Struct
VOCCSI
COCO-SP
PCQM-Contact
Domain
Chemistry Graph Cls.
Chemistry Graph Reg.
Vision Node Cls.
Vision Node Cls. 123,286
Chemistry
Task
Link Pred.
# Graphs
15,535
15,535
11,355
529,434
Avg. Nodes
150.94
150.94
479.40
476.88 27.39
30.14
Avg. Diameter
56.99
56.99
27.62
9.86
Table 1: Overview of the five Long-Range Graph Benchmark datasets.
Table provides the main statistics of each dataset. The five LRGB datasets contain graphs that are larger and of higher diameter than those used in prior datasets for similar prediction tasks. A core hypothesis of LRGB is that the chosen learning tasks depend strongly on long-range dependencies between vertices and are therefore suitable for comparing architectures that specifically aim to capture such dependencies, such as GTss.
The experiments provided by Dwivedi et al. (2022) report a strong performance advantage of GTTs over the MPGNN architectures GCN (Kipf & Welling 2017], GINE (Hu et al. 2020b), and GatedGCN (Bessson & Laurent, 2017); in accordance with the expectations. Subsequently, GPS (Rampášek et al.] 2022) reached similar conclusions on LRGB. We note that these two works are strongly related and built on shared code base. Newer rescarch on GTs (see Section 1.1 is commonly based on forks of this code base and often cites the baseline performance reported by Dwivedi et al. (0022) to represent MPGNNs.
Our contribution is thlde-fold First, we show that the three MPGNN baselines GCN, GINE, and GaLedGCN all profit massively from further hyperparameter luning, reducing and even closing the gap to graph transformers on multiple datasets. In fact, GCN yields state-of-the-art results on Peptides-Struct, surpassing several newer graph transformers. On this dataset in particular, most of the performance boost is due to & multi-layer prediction head instead of linear one, again highlighting the importance of hy- perparameters. Second, show that the vision datasets PascalVOC-SP and COCO-SP normalization of the imput features is highly beneficial. We argue that, in the vision domain, feature normalization should be the default setting. Third and last we take a closer look at the MRR metric used to evaluate PCQM-Contact. There, we demonstrate different. filtering strategies have a major impact on the results and must be implemented exactly to specification to facilitate reliable comparisons.
1.1 Related Work
Our primary focus are the commonly used MPGNNs GCN (Kipf & Welling, 2017), GINE (u et al. 2020b and GatedGCN (Bresson & Laurent 2017) well as the graph transformer GPS (Rampšskk el al. 2022 GCN performs simple message passing without edge features. GINE extends Graph Isomorphism Networks (GIN) Xu et al. 2019) by additionally incorporating edge features. GatedGCN also incorporates edge features to compute a channel-wise 'importance weighting' messages. These three architectures are all standard MPGNNs (Gilmer et al., 2017) where the depth of the receptive field of each vertex is equal to the number of message-passing layers. The graph transformer GPS applies message-passing and self- attention (Vaswani et al. 2017) in parallel and combines the results in each layer. With this additional global information exchange facilitated through self-atiention, each vertex generally has unbounded receptive field and can directly aggregate information from the entire graph. This should in principle help graph transformers like GPS capture long-range dependencies on graphs
There are many more MPGNN architectures (Hamilton et al. 2017 Xu et al.
2018
Chen et al. 2020: Kreuzer
Corso al. 2020), as well as graph transformers (Dwivedi Bresson 2020 Ying al. 2021
2t all 2021 Shi et al. 2020 Park et al. 2022 Weis et al. 2021 Rampášek et al.
2022
Shirzad et al. 2023
Kim et al.
2022 Ma et
al. 2023 He et al. 2023) see also the survey by Min et al.
(0022). Many newer
graph transformer architectures have reported results on LRGB datasets, including Exphormer Shirzad
Source code: http:://gthhbb.ccmmoenhtt//////
2

---

Published in Transactions on Machine Learning Research (05/2024)
1-layer head
2-layer head
fully tuned
GPS baseline
METHHDD
PEITIDES-FUL FUNC
PEPTIDES SSPTIDES-STRUCT
STRUCT
TEST AP
TIST MAE
0.375
0.70
GCN
0.5930 + 0.0023
0.3496 + 0.0013
0.350
GINT
0.5498 1. 0.0079
0.3547 0.0045
Mean Absolute Error (MAE)
Average Precision (AP).
GATEDGON
0.0035
0.65
LRANSPOREER
0.6326 1. 0.0126
0.2529 1 0.0016
0.300
SAN
0.6433 + 0.0073
0.25413 0.0012
0.6535 1 0.0041
0,2500 0.0005
0.275
0.60
GCN
0.6860 + 0.0050
- 16%
0.2460 + 0.0007
$ 30%
0.250
GINE
0.9021 + 0.0067
20%
0.2473
0.0017
+30%
0.55
GAMEDGCN
0.6765 0.0047
11%
0.2477
0.0009
26%
0.225
GPS
0.6634 + 0.0091
= 0%
0,2509 + 0.0014
+ 0%
0.200
0.50
CRAWL
0.7074 + 0.0032
0.2500 0.0022
0.175
Onives
0.7150
EXPHORMER
0,6527 + 0.0013
0.2181 = 0.0107
GRIT
0.6088 + 0.0082
0.2460 0.0012
GatedGGN GINE
GCN
GatedGCN GINE
GCN
GRAPH VIT
0.6912 + 0.0075
0.2419 0.0016
Peptides-Struct
Peptides-Func
G-ML
0.6921 + 0.0054
0.2475 + 0.0015
(b) Exchanging the linear prediction head by MLP ac- (a) Previous and updated results on the peptides counts for most of the additional performance of all three MPGNNs, especially Peptides-Struct
datasels. Best results (within sidev) in bold.
Figure 1: On both Peptides datasets, all three MPGNNs surpass GPS. On Peptides-Struct basic GCN model even achieves SOTA results.
at a 2023 GRIT Ma et al. 2023 and Graph ViT GraphMLPMixer (He et al., 2023 Several other
architectures not based on transformers have also been evaluated on LRGB, including CRaWI Tönshoff
et al., 2023). DRew (Gutteridge et al. 2023) and Virtual Nodes (Gilmer al. 2017 Cai al. 2023). In general, long-range interactions are typically handled through GNNs with many layers (c.g. Li et al. 2020
or the addition of global information exchange through graph transformers and virtual nodes (Gilmer el al. 2017). Finally, we do see connection of our work to graph learning benchmarking projects (Dwivedi et al.
2020
Hu et al. 2020a) that also advocate for rigorous testing of graph learning architectures.
2 Concerns
Hyperparameters In this paper, we argue that the results reported by Dwivedi et al. (2022) are not representative of MPGNNs and suffer from suboptimal hyperparameters. We provide new results for the same MPGNN architectures that are obtained after a basic hyperparameter sweep. We tune the main hyperparameters (such as depth, dropout rate, ...) in pre-defined ranges while strictly adhering to the official 500k parameter budget. The exact hyperparameter ranges and all final configurations are provided in Appendix A.T. In particular, the prediction head from to 3 (which turned out to be very relevant), and also considered the dropout and learning rate of the network. In addition to those "standard" byperparameters, we also view the usage of positional or structural encoding (none / LapPE (Dwivedi & Bresson 2020) / RWSE (Dwivedi et al. 2021 as hyperparameter that is tuned for each mothod. As a point of reference, we reevalute GPS in an identical manner, additionally tuning the inner MPGNN and the normalization strategy, and also achieve significantly improved results on three datasets with this Graph Transformer. The results reported for GPS may therefore also be subject to suboptimal configurations.
looked at networks with to 10 layers, varied the number of layers in
Feature Normalization The vision datasets PascalVOC-SP and COCO-SP have multi-dimensional node and edge fcatures with values spanning different orders of magnitude for different fcature channels. Passing this input to a neural netw without channel-wise normalization can cause poorly conditioned activations. While feature normalization is standard practice in deep learning and computer vision in particular, neither Dwivedi et al. (2022) nor any subsequent works using LRGB utilize it, except CRAWi (Tönshoff et al. 2023). We apply channel-wise linear normalization all input features as visualized in Figure 2a] that all models (baselines and GPS) profit from it in ablation in Figure 2b
We show
3

---

Published in Transactions on Machine Learning Research (05/2024)
Unnormalized
Normalized
reported
1.5
+normalization
0.4
+tuning
400
1..0
0.3
0.5
300
0.0
0.2
200
-0.5
0.1
100
-1.0
-155
0.0
GCN
GINE
GatedGCN
GPS
(a) Vertex feature matrices ([V) x 14) on PascalVOC-SP before and after channel-wise normalization.
(b) The effects of feature normalization and hyperpa-
rameter tuning PascalVOC-SP.
Figure 2:
On PascalVOC-SP and COCO-SP feature normalization and further tuning improves performance across all compared methods.
Link Prediction Metrics The evaluation metric on the link-prediction dataset PCQM-Contact Dwivedi at al. 2022) the Mean Reciprocal Rank (MRR) in a filtered setting, as defined by Bordes et al. (2013). For predicted edge scores the MRR measures how given true edge (h,t) ranked compared to all possible candidate edges (h) of the same head. As there might be multiple true tails t for cach head h, the filtered MRR removes those other true tails (false negatives) from the list of candidates before computing the metric. This filtering avoids erroneously low MRR values due to the model preferring other true edges and is common in link-prediction tasks. Even though Dwivedi et al. (2022) explicitly define the metric to be the filtered MRR, the provided code computes the raw MRR, i.c keeping other true tails in the list. We report results on PCQM-Contact in a corrected filtered setting. We additionally provide results with an extended filtering procedure where self-loops of the form (h, h) are also removed from the of candidates since these are semantically meaningless and never positive. This impactful as the scoring function used by Dwivedi al.) 2022 is based symmetric dot- dot-producc and therefore exhibits strong bias towards self-loops.
3 Experiments
Peptides-Func and Peptides-Struct Table Ta provides the results obtained on the test splits of the Peptides-Func and Peptides-Struct For the MPGNN baselines, we observe considerable improvements on both datascts all three MPGNNs outperform GPS after tuning. The average precision increased relatively by around 10% to 20%. GCN achieves score of 68.60%, which is competitive with newer GTss such GRIT or Graph VIT. The improvement on Peptides-Struct is even more significant with relative reduction of the MAE of 30%, fully closing the gap to recently proposed GTss Surprisingly, simple GCN all you need to match the best-known results on Peptides-Strucct Struuct. The results for GPS effectively stayed the same as the original paper (Rampášek et al. 2022 Those values thus seem to be representative for GPS.
Peptides-Func
We observed that the key hyperparameter underlying the improvements of all three MPGNNs is the depth of the prediction head. To show this Figure 1b contains ablation where we exchanged the linear prediction head configured by Dwivedi et al. (2022) with 2-layer perceptron, keeping all other hyperparameters the same. While the benefit on Peptides-Fuur Func is considerable and highly significant, on Peptides-Struct the head depth accounts for almost the complete performance gap between MPGNNs and GTs. GPS' performance with lincar and deeper prediction heads is largely unchangod. For example, our GPS configurations in Table la use 2-layer prediction head. Our results indicate that the prediction targets of both datasets appear depend non-linearly on global graph information. In this case, MPGNNs with linear prediction heads are
4

---

Published in Transactions on Machine Learning Research (05/2024)
PASCALVOC-SP
COCO-SP
GCN
'ESS F1 1 REL IPP
TEST F1 7
REL IMP
GINE
GON
0.1298 + 0.0080"
0.0841 0.001O
0.45
GatedGCN
GINE
0,1265 + 0,0076'
0. 1339
0.001d
GPS
0,2873 + 0.0219"
0.2641
0.0045"
TRANSPOREER
0,2691 + 0,0098
0.2618 0.0051
Test MRR
0.40
SAN
0.3230 1. 0.0039
0,2592 0.0158
0.3746 + 0.0109"
0.3412 0.0014"
0.35
GCN
0:2078 + 0.0031
+-64%
0. 13388
0.00007
+599
Ommens Qugs
GINE
+115%
+59%
GATEDGON
0.3880 + 0.0044
+-35%
0.2922
0.00118 +11%
GPS
CRAWG
0.4588 + 0.0079
0.3314 + 0.0024"
0.25
reported (raw) a)
ours (raw) b))
purs (filter) c)
ours (ext, filter) d)
EXPRORMER
0,3960 1 0.0027
0,3130 10,0008
(a) Tuning results on vision datasets VOCCSI and COCO-SP. "No normalization used.
(b) Results PCQM-Contact. For our own models we provide the MRR scores with varying levels filtering.
unable to model the larget function. Graph Transformers are not sensitive to linear prediction heads, since each layer can process global graph information with a deep feed-forward network. However, we would argue that switching to deep predictive head represents simpler and computationally cheaper solution to the same issue.
VOC-SS and COCO-SP Table 3a provides the results obtained on the test splits of the superpixel datasets PascalVOC-SP and COCO-SP. We observe significant improvements for all evaluated methods. On PascalVOC-SP the Fl score of GatedGCN increases to 38.80% which exceeds the original performance reported for GPS by Rampášek et al. 2022). GPS also improves significantly to 44.40% F1. This is only one percentage point below the results achieved by CRAW1, which currently is the only reported result with normalized features. The previously large performance gap between GPS and CRAWi WL is therefore primarily explained by GPS processing raw input signals. On COCO-SP, we observe similar results. Here GPS sets state-of-the-art art Fl score of 38.84%.
Note that these improvements are achieved entirely through data normalization and hyperparameter tuning, Figure 26 provides an ablation on the individual effect of normalization. We train intermediate models with configurations identical to those used by Dwivedi et al. (2022) and Rampášek et al. (2022), but with feature normalization. For GatodGCN we observe slight performance increase but a large reduction in the variance across random seeds. For the remaining methods, including GPS, normalization of node and edge features already accounts for at least. half of the observed performance gain, emphasizing its importance in practice.
PCQM-Contact Figure 3b plots the MRR scores obtained on the test split with various evaluation settings as described in the link prediction paragraph of Section 2 First, provide the results originally reported for LRGB in the literature (a). Recall that these values are obtained in raw setting with false negatives present. We then provide results obtained after training our own model with new hyperparameters (chosen based on the raw MRR) in b). We still use the raw MRR for evaluation in b) to measure the impact of hyperparameter tuning. Tuning yields an absolute improvement of around 3%. The previonsly reported slight performance edge of GPS not observable this setting after tuning.
In subplot e) we measure the MRR of our models in the filtered setting. Note that these values are based the exact same predictions as in b), but false negatives are removed. The measured MRR increases by roughly 3% when compared to the raw setting. This shift could erroneously be interpreted
significant improvement when comparing to litcrature values obtained in raw setting. In d) we evaluate our models (still using the same predictions) in an extended filtered setting where we additionally remove self-loops from the candidate pool. Compared to the filtered MRR in c) the MRR metric increases by about percentage points, indicating that self-loops strongly affect the results. Note that in d) GPS again slightly outperforms the MPGNN baselines, in contrast b) and c). This means that GPS' predictions seem suffer overproportionally when self-loops are not filtered. Therefore, the specific choice of how negative
5

---

Published in Transactions on Machine Learning Research (05/2024)
samples are filtered on PCQM-Contact can directly affect the ranking of compared methods and must be considered and implemented with care.
4 Conclusion
In our experiments, we observed considerable performance gains for all three MPGNN baselines. First, this indicates that extensive bascline tuning is important for properly assessing onc's own method, especially on relatively recent. datasets. Second, only on the two superpixel datasets do graph transformers exhibit clear performance benefits against MPGNNs, indicating that either there are ways to solve the other tasks without long-range interactions or graph transformers are not inherently better at exploiting such long-range dependencies. Evaluating this further appears to be promising direction for future research. In addition, we would invite discussion on the best-suited link prediction metric on PCQM-Contact..
Going forward, additional benchmark datasets that test for LRI would also be highly useful to further evaluate existing and future architectures. A stronger focus on synthetic tasks may be helpful to overcome the caveats of the LRGB datasets, where it is unclear how strongly the proposed real-world tasks truly depend on LRI. Classical algorithmic tasks such as path-finding or vertex coloring could be used to define target functions that, by construction, require LRI on synthetic high-diameter graphs.
To evaluate the potential virtues of graph transformers it may further be important to evaluate models datasets that are large-scale, rather than long-range. When vision transformers were initially shown to outperform CNNs certain vision tasks, this was only achieved when pre-training on massive datasets with over 100 Million images (Dosovitskiy et al. 2021]. While more data-efficient vision transformers have since been developed (Touvron et al.) 2021), the general consensus that transforms are "data hungry" provails across domains. Many popular benchmark datasets in graph learning are comparatively small and often contain less than 100K graphs. These datasets may therefore simply be to small to observe potential scaling benefits that transformers have demonstrated in other domains. Recent work has begun to investigate the scaling of MPGNNs and graph transformers to new large-scale datasets (Beaini et al. 2024 Sypetkkoski et al. 2024b. This research direction may open up new perspectives on the strengths and weaknesses of graph transformers.
Acknowledgments
This work was supported by the German Research Foundation (DFG) under grants GR 1492/16-1 and KT 2348/1-1. The project also received funding from the European Research Council (ERC) under the European Unions Horizon Europe rescarch and innovation program (Grant Agrecment No. 101041669).
References
Dominique Beaini, Shenyang Huang, Joao Alex Cunha, Zhiyi Li, Gabriela Moisescu-Pareje. Oleksandr Dy- mov, Samuel Maddrell-Mandeer Callum McLean, Frederik Wenkel, Luis Müller, Jama Hussein Mohamud, Ali Parviz, Michael Craig, Michal Koziarski, Jiarui Lu, Zhaocheng Zhu, Cristian Gabellini, Kerstin Klaser, Josef Dean, Cas Wognum, Macicj Sypetkowski, Guillaume Rabusseau, Reihanch Rabbany, Jian Tang, Christopher Morris, Mirco Ravanelli, Guy Wolf, Prudencio Tossou, Hadrien Mary, Therence Bois, An- drew William Fitzgibbon, Blazej Banaszewski, Chad Martin, and Dominic Masters. Towards foundational models for molecular learning on large-scale multi-task datascts. In The Twelfth International Conference on Learning Representations, 2024. URL http::://ppeerreiii..eettto/e
Antoine Bordes, Nicolas Usunier, Alberto Carcia-Durann, Jason Weston, and Oksana Yakhnenko. Translat- ing embeddings for modeling multi-relationa data. In C.J. Burges, Bottou, M. Welling, Z. Ghahra- mani, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 26. Cur- ran Associates, Inc., 2013. URL https: htttp:::/ppooceedigg..eeeee neurips..pppp... cctpspc.cppppp.cepccce htt.erppp.ccaaprrre...ee///pp//p filees/papee//03////iie// Leecr/aar72223333333228822222223
pdf..
Xavier Bresson and Thomas Laurent. Residual gated graph convnets. arXiv preprint arXiw:1711.07553. 2017.
6

---

Published in Transactions on Machine Learning Research (05/2024)
Chen Cai, Truong Son Hy, Rose Yu, and Yusu Wang. On the connection between mpnn and graph trans- former. arxiv preprint urXi::200..1 11..111111 1956, 2023.
Ming Chen, Zhewei Wei, Zengleng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International conference on machine learning, pp. 1725-1735. PMLR, 2020.
Gabricle Corso, Luca Cavalleri, Dominique Beaini, Pictro Liò, and Petar Velickovic. Principal neigh- bourhood aggregation for graph nets. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Had- sell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advonces in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. URL https://prcooiing..noot www.oooddinng...urii...o. cttpsscccpppppa wwtpppppppppppeeeeeeeeceeeeeeeee 2att//222/hhassi
0eea255a117860220011510000000000 Abstract..btt........... htmml.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https://openreeiiee witp://ww.nn//toommddd////.//a YiccbFdNNTTy
Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. arXiv preprinl 2.96:9001.096999 2020.
Vijay Prakash Dwivedi, Chaitanya K Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Bench- marking graph neural networks. arXic preprint arXiw:2003.0098: 2020.
Vijay Prakash Dwivedi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Graph neural networks with learnable structural and positional representations. In International Conference on Learning Represenlations, 2021.
Vijay Prakash Dwivedi, Ladislav Rampášek, Michael Galkin, Ali Parviz, Guy Wolf, Anh Tuan Luu, and Dominique Beaini. Long range graph benchmark. Advances in Neural Information Processing Systems, 2022.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In Proceedings of the Phurth-Fourth International Conference Machine Learning (ICML), pp. 1263 1272, 2017.
Benjamin Gutteridge, Xiaowen Dong, Michael M. Bronstein, and Francesco Di Giovanni. DRew: Dynami- cally rewired message passing with delay. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 23-29.1 Jul 2023. URL
PMLR,
mlr welir.pree wrwsss/esvvv w202//222////// http://22222g/2eeeeeee gutteridgee2a.htt!)
William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. pp. 1024-1032 2017.
Xiaoxin He, Bryan Hooi, Thomas Laurent, Adam Perold, Yann Lecun, and Xavier Bresson. generalization of ViT/MMP-mixxe to graphs. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 12724 12745. PMLR, 23 29 Jul 2023. URL htttps://proceeieee mlr.iigggm.rrr....mmmmm.. press/ Titess/vv02/// 2222/he3a.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08. 2016.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020a. URL https::/prroceedingg neurips. cttppppprccpppr htmll
7

---

Published in Transactions on Machine Learning Research (05/2024)
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovee. Strategies for pre-training graph neural networks. In International Conference on Learning Represenla- tions, 2020b. URL
Jinwoo Kim, Dat Nguyen, Seonwoo Min, Sungjun Cho, Moontae Lee, Honglak Lee, and Seunghoon Hong. Pure transformers are powerful graph learners. Advances in Neural Information Processing Systems, 35: 14582 14595, 2022.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.
Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral attention. Advances in Neural Information Processing Systems, 34: 21618 21629, 2021.
Guohao Li, Chenxin Xiong, Ali Thabet, and Bernard Ghanem. DeeperGCN: All you need to train deeper GCNs. Xiei preprint arv:220006 07739, 2020.
Liheng Ma, Chen Lin, Derek Lim, Adriana Romero-Soriano, K. Dokania, Mark Coates, Philip H.S. Torr, and Ser-Nam Lim. Graph Inductive Biases in Transformers without Message Passing, In Proc. Inl. Conf. Mach. Learn., 2023.
Erxue Min, Runfa Chen, Yatao Bian, Tingyang Xu, Kangfei Zhao, Wenbing Huang, Peilin Zhao, Junzhou Huang, Sophia Ananiadou, and Yu Rong. Transformer for graphs: An overview from architecture per- spective. arXiv preprint arXiv:200..08. 2022.
Wonpyo Park, Woonggi Chang, Donggeon Lee, Juntae Kim, and Seung- won Hwang. Grpe: Relative posi- tional encoding for graph transformer. arXiv preprint arXiv:2201.12787 2022.
Ladislav Rampášek, Michael Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a general, powerful, scalable graph transformer. Advances Neural Information Processing Systems, 35:14501-14515, 2022.
Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjin Wang, and Yu Sun. Masked label prediction: Unified message passing model for semi-supervised classification. arXiv preprint arxiv:2009. Ni0:2009.00500,, 2020.
Hamed Shirzad, Ameya Velingker, Balaji Venkatachalam, Danica J Sutherland, and Ali Kemal Sinop. Ex- phormer: Sparse transformers for graphs. In International Conference on Machine Learning, 2023.
Macicj Sypetkowski, Frederik Wenkel, Farimah Poursafaci, Nia Dickson, Karush Suri, Philip Fradkin, and Dominique Beaini. ON THE SCALABILITY OF GNNS FOR MOLECULAR GRAPHS. In ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2024. URL https:/7
Spenreviev.ne//formrffe ISSN 2835-8856. URL htt.://ooorrmiie//iilllllllll.l.
Jan Tönshoff, Martin Ritzert, Hinrikus Wolf, and Martin Grohe. Walking out of the weisfeiler leman hierarchy: Graph learning beyond message passing. Transactions on Machine Learning Research, 2023.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou, Training data-cfficient image transformers & distillation through attention. In International conference machine learning, pp. 10347-10357. PMLR, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you noed. Advances in neural information processing systems, 30, 2017.
Marissa Weis, Laura Pede, Timo Lüddecke, and Alexander S Ecker. Self-supervised representation learning of neuronal morphologies. Xix preprint arXiu:21 12. 12482, 2021.

---

Published in Transactions on Machine Learning Research (05/2024)
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint or Xiv: 1att1810.000000 000000 2018.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In Proceedings of the Seventh International Conference Learning Representations (TCL.R, 2019.
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie- Yan Liu. Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems, 34:28877-28888, 2021.
Jiaxuan You, Zhitao Ying, and Jure Leskovec. Design space for graph neural networks. Advances in Neural Information Processing Systems, 33:17009- 177009917721,, 2020.

---

Published in Transactions on Machine Learning Research (05/2024)
A Experiment Details
A.1 Hyperparameters
In the following we describe our methodology for tuning hyperparameters on the LRGB datasets. We did not conduct a dense grid search, since this would be infeasible for all methods and datasets. Instead we perform a "linear" hyperparameter search. We start from a empricially chosen default config and tune each hyperparameter individually within a fixed range. Afterwards, we also evaluate the configuration obtained by combining the best choices of every hyperparameter. From all tried configurations we then select the one with the best validation performance single run per configuration and for the larger datasets slightly reduced the number of epochs. For the final evaluations runs we average results across four different random seeds specified by the LRGB dataset.
final setting. For this hyperparameter sweep, we resorted to
Overall, we tried to incorporate the most important hyperparameters which we selected to be dropout, model depth, prediction head depth, learning ratc, and the used positional or structural encoding. For GPS we additionally evaluated the internal MPGNN (but only between GCN and GatedGCN) and whether to use BatchNorm or LayerNorm. Thus, our hyperparamters and ranges were as follows:
Dropout [0, 0.1, 0.2], default 0.1
Depth [6,8,10] default 8. The hidden dimension is chosen stay within a hard limit of 500k parameters
learning rate [0.001, 0.0005, 0.0001], default 0.001
Head depth [1,2,3], default 2
Encoding [none, LapPE, RWSE] default none
Internal MPGNN [GCN, GatedGCN), default GatedGCN (only for GPS)
Normalization [BatchNorm, LayerNorm default BatchNorm (only for GPS)
On the larger datasets PCQM-Contact and COCO we reduce the hyperparameters budget slightly for effi- ciency. There, we did not tune the learning rate (it had been 0.001 in every single other case) and omitted dropout rate of 0. We note that the tuning procedure used here is relatively simple and not exhaustive. The ranges we searched are rather limited, especially in terms of network depth, and could be expanded in the future. TTalls 2 to 0 provide all final model configurations after tuning. [Table 7 provides the final performance on all datasets and Figure 4 shows the results from all tuning runs on Peptides Struct and PascalVOC. From the tuning results we observed comparable slability of MPGNN and GT archilectures with respect their hyperparameters.
We make some additional setup changes based preliminary experiments. All models are trained with AdamW optimizer using cosine annealing learning rate schedule and linear warmup. This differs from Dwivedi et al. (2022), who optimized the MPGNN models with a "Reduce on Plateau" schedule and instead matches the learning rate schedule of GPS Rampášek et al. [2022). We set the weight decay to 0.0 in all five datasets and switch to slightly larger batch sizes to speed up convergence. We also choose GeLU Hendrycks & Gimpel (2016) as our default activation function. Furthermore, we change the prediction head for graph-level tasks such that all hidden layers have the same hidden dimension as the GNN itself. These were previously configured to become more narrow with depth, but could not observe any clear benefit from this design choice. Last, all MPGNN models use proper skip connections which go around the entire GNN layer. The original LRGB results use an implementation of GCN as provided by GraphGym You BL al. 2020h. The skip connections in this implementation do not skip the actual non-linearity at the end of each GCN layer, possibly hindering the flow of gradients. We reimplement GCN with skip connections that go around the non-lincarity. Note that these additional tweaks are not used in our ablation studies Figure 1b and Figure 26 when training the intermediate models where we only change the head depth and normalization, respectively. There, we use identical model configurations to those used in the literature.
10

---

Published in Transactions on Machine Learning Research (05/2024)
0.45
0.425
0.400
0.40
Score (higher better)
MAE Blower is batter)
0.375
0.35
0.350
0.30
0.325
0.25
0.300
zije
GINE
GatedGCN
GCN
GPS
GINE
GatedGCN
GCN
GPS
(a) Peptides Struct
(b)
PascalVOC
Figure 4: Validation results from all tuning runs on Peptides Struct and PascalVOC. Most results cluster tightly around the best-performing variant of the model.
A.2 Feature Normalization
On PascalVOC-SP and COCO-SP we apply channcl-wise normalisation the node and edge features. For each dataset, we compute the channel-wise mean μ E Rd and standard deviation a E I2 on the train split. Here, d is the feature dimension. Each feature vector E RRd is then normalized linearly before beigng passed to the model:
3
0i
11

---

Published in Transactions on Machine Learning Research (05/2024)
Table 2: Hyperparameters on Peptides-Funi Func
GCN
GINE
GatedGCN
GPS
0.001
0.001
0.001
0.001
drapout
0.1
0.1
0.1
0.1
#layers
6
8
10
6
hidden dim.
235
160
95
76
head depth
3
3
PE/SE
RWSE
RWSE
RWSE
LapPE
batch size
200
200
200
200
#epochs
250
250
250
BatchNorm
MPNN
GatedGCN
#Param.
486k
491k
479k
Table 3: Hyperparameters on Peptides-Stree -trruc..
GCN
GINE
GateIGCN
GPS
0.001
0.001
0.001
0.001
dropout
0.1
0.1
#layers
hidden dim.
235
145
head depth
PE/SE
LapPE
LapPE
LapPE 250
LapPE
batch
200
200
200
#epochs
250
250
250
morm
BatchNorm
MPNN
GatedGCN
# PParam.
488k
492k
445k
452k
Table 4: Hyperparameters on Pascal VOC-SP.
GCN
GINE
GaueAGCN
GPS
0.001
0.001
0.001
0.001
dropout
0.0
0.2
0.2
0.1
#layers
10
10
10
hidden dim.
200
145
95
68
head depth
2
2
PE/SE
RWSE
none
none
LapPE
batch size
50
50
#epochs
200
200
BatchNorm
MPNN
GatedGCN
#Param.
490k
450k
473k
501k
Table 5: Hyperparameters on COCO-SP.
GCN
GINE
GatedGCN
GPS
Ir
0.001
0.001
0.001
0.001
dropout
0.1
0.1
0.1
0.1
#layers
hidden dim.
head depth
PE/SE
none
none
none
none
batch size
50
50
50
50
#epochs
200
200
200
200
Norm
MPNN
GatedGCN
# Paramm.
500k
478k
459k
500k
12

---

Published in Transactions on Machine Learning Research (05/2024)
Table 6: Hyperparameters on PCQM-Contact.
GCN
GINE
GatedGCN
GPS
0.001
0.001
0.001
0.001
dropout
0.1
0.0
#layers
hidden dim.
215
105
76
head depth
1
PE/SE
LapPE
LapPE 500
LapPE 300
LapPE
batch size
500
500
#epochs
150
150
150
150
morm
LayerNorm
MPNN
GatedGCN
# Param.
456k
466k
477k
478k
Table 7: Performance of our models on the Long-Range Graph Benchmark.
METROR
PASCALVOC-SP TEST F1 f
COCO-SP TEST F1
PEPTIDES-FUNC TEST AP f
PEPTIDES-STRUCT TEST MAE
PCQM-CONTACT TEST MBR
FILTER
EXT.
GCN
0.2078
0.0031
0.1338 1 0.0007
0.6880
0.0050
0,2460 0.0007
0.3124 1. 0.0007
0.3631
0.0006
0.1526
0.0006
GINE
0.2718
0.0054
0.2125 + 0.0008
0.6621
0067
0.2473
0.0011
0.3508 + 0.0006
0.3725 + 0.00006
0.4617
0.0005
GATEDGON
0.0084
GPS
0.44-40
0.0065
0.3884 + 0000
0.6534
0.00 0.00
0.2509 4: 0.0014
0.3405 4: 0.0006
0.3722
0. 0.0005
0.4703
0.0014
13

---

Published in Transactions on Machine Learning Research (05/2024)
B Additional Experiments
In Table 8 we provide extended results for the ablation study from Figure 1b, More specifically, we evaluate both GPS and GCN models on Peptides-Struct with predictive heads of various depths. As in Figure Ib we study the isolated effect of this hyperparameter on the performance and leave all other hyperparameters identical to those used by Dwivedi et al. (2022)
The main observation is the large performance jump of GCN when configured with head of depth two three instead of the linear head that originally chosen. The results of GPS do not suffer from linear prediction head, but also do not improve further for deeper configurations. This is probably explained by the fact that each GPS layer can already process global graph information with deep foed-forward network. A deeper prediction head may therefore be redundant in GPS models.
Table 8: Detailed performance on Peptidos-Func for GPS and GCN with the depth of the prediction head varying from to 3 layers.
depth
GCN (MAE)
GPS (MAE)
0.3496 + 0.0013
0.2500
0.0005
0.2547 + 0.0019
0.2516 = 0.0012
0.2534 + 0.0013
0.2546 0.0020
14


Reviews:
[{"review_id": "hWoVGHfveK", "reviewer": "Reviewer_vdfV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe rigor of experiments is very important for the machine learning community, and the paper makes contribution in this perspective.\n\nWeakness:\n\nThis paper is mainly adjusting the hyperparameters, the normalization, and metric of the models, but does not provide technical or theoretical contribution to machine learning or representation learning. Therefore, I'm not sure whether TMLR is a suitable venue for it.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper target the phenomenon 'graph transformers significantly outperform mpgnns' claimed in the LRGB work, and revaluate different MPGNNs and graph transformers. The first conclusion is that the performance gap in LRGB is overestimated because of the suboptimal hyperparameter choice. Second, feature normalization is founda to be beneficial. Finally, different filtering strategies have a majot impact on the results.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Question:\n\nAlthough reevaluating the experiments by other researchers is beneficial, how to justify that the experiments in this paper is perfectly rigorous? The search space of the hyperparameters is infinite, and the authors cannot prove that the hyperparameters found in this paper is a global optimum. Therefore, it is still possible that other researchers also reevaluate the experiments in this work, and find hyperparameters leading to completely contradictory conclusions.", "broader_impact_concerns": "No ethical concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3IpPy6yRaN", "reviewer": "Reviewer_Whiu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper addresses an important question regarding the performance of different models on graph learning tasks that heavily rely on long-range interactions.\n2. The writing style of the paper is well-executed, ensuring clarity and coherence.\n\nWeaknesses:\n\n1. While I found the paper enjoyable to read, there are areas that could benefit from further improvement before publication. One such area is the lack of sufficient background introduction, particularly for readers who are not familiar with the LRGB benchmark. Providing more context would enhance the readability for a broader audience.\n2. It would be valuable to include additional experiments to support the findings. While it is expected that tuning hyperparameters can improve GNN performance, it would be interesting to explore if Graph Transformers exhibit more robust performance across different hyper-parameters.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper re-evaluates the performance of Message Passing GNNs and Graph Transformers on the Long-Range Graph Benchmark (LRGB). Through a rigorous empirical analysis, the authors find that the reported superiority of Graph Transformers is overestimated due to suboptimal hyperparameter choices. In fact, after basic hyperparameter optimization, the performance gap between the two model classes completely vanishes across multiple LRGB datasets. The main contribution of the paper is to highlight the importance of thorough empirical evaluation and establishing higher standards of rigor in graph machine learning research.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weaknesses section.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "A1eKPb8E1t", "reviewer": "Reviewer_9SYh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is a clear accept and it is important for broader community to immediately adopt its findings.\n\n**Strengths**\n\nThis paper enjoys being simple and to the point. The message is quite effective and there is no extra fluff in the presentation which helps deliver the main message: carefully tuning and understanding your network through robust visualization is the key enabler of progress in deep learning. To this effect, this paper identifies key areas where the supposed supremacy of Graph Transformer architecture falls short. In particular, the issue with feature normalization in the vision dataset is an important finding. Furthermore, the finding about the MRR metric and the erroneous code piece that is probably now standard in many repositories is something very important to highlight to the broader community. I believe it is extremely important that evaluation metrics are free of these clear issues and as a result, this paper is a great contribution to this effort.\n\n**Weaknesses**\nThere are no gross weaknesses in this work. The presentation can be slightly improved though. In particular, the hyperparameter subsection in section 2 felt like it could benefit from a deeper discussion.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the empirical performance of message passing GNNs in comparison to Graph Transformers on the Long Range Graph benchmark. The principal findings demonstrate that after appropriate tuning, judicious choice of normalization, and filtering the gap between Graph Transformers and MPGNNs diminishes. The findings raise questions regarding the empirical rigor of past graph transformer  architectures.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please give a bit more information about the important hyperparameters tuned to obtain the improved results for MPGNNs in the main text.\n\n- Please also consider including a GAT and GIN architecture as baselines.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tG0h9Cb0Kj", "reviewer": "Reviewer_oXh1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros\n1. The topic raised by the authors is very important for the graph learning community \n2. The experiments and comparisons provided by the authors clearly show the importance of the author's study.\n\nCons. \n1. The related work should be rewritten since it is not informative and does not answer how the covered methods tackle the problem of propagating the information from distant nodes for working on LRGBs.\n2. Since authors argue that the reported performance gap is overestimated due to suboptimal hyperparameter choices in LRGB datasets, it is also essential to show other graph benchmarks, inductive/transductive and heterophilic, showing how the phenomena observed by the LRGBs generalize there.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the performance gaps in Graph transformer architectures compared to MPNNs on Long-Range Graph Benchmarks (LRGBs). \nThe authors observed that the performance gap is mostly due to suboptimal hyperparameter choices. In addition, the paper discusses the impact of lacking feature normalization on LRGB’s vision datasets and highlights the drawbacks of current LRGB’s link prediction metric.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I kindly request that the authors address my concerns raised in the \"Weaknesses\" section. Once the authors respond, I will re-evaluate my assessment accordingly.", "broader_impact_concerns": "No ethical concerns have been found.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors investigate the empirical performance of message passing GNNs in comparison to Graph Transformers on the Long Range Graph benchmark. The principal findings demonstrate that after appropriate tuning, judicious choice of normalization, and filtering the gap between Graph Transformers and MPGNNs diminishes. The findings raise questions regarding the empirical rigor of past graph transformer architectures.

The topic raised by the authors is very important for the graph learning community. Specifically, the authors aim to address an important question regarding the performance of different models on graph learning tasks that heavily rely on long-range interactions. The experiments and comparisons provided by the authors clearly show the importance of the author's study. The writing style of the paper is well-executed, ensuring clarity and coherence. However, this paper is mainly adjusting the hyperparameters, the normalization, and metric of the models, but does not provide technical or theoretical contribution to machine learning or representation learning. Moreover, this paper lacks the sufficient background introduction, particularly for readers who are not familiar with the LRGB benchmark. It would be valuable to include additional experiments to support the findings. While it is expected that tuning hyperparameters can improve GNN performance, it would be interesting to explore if Graph Transformers exhibit more robust performance across different hyper-parameters. Therefore, based on four qualified reviews, this paper can be accepted with minor revision and the authors are encouraged to merge the comments into their update versions.
Decision: Accept (poster)