Title: value: Link Prediction with Untrained Message Passing Layers
Authors: value: Lisi Qarkaxhija
Anatol Eugen Wegner
Ingo Scholtes
Venue: ICLR 2025
Keywords: value: graph neural networks
untrained message passing layers
link prediction
path-based similarity measures

Abstract: value: In this work, we explore the use of untrained message passing layers in graph neural networks for link prediction. The untrained message passing layers we consider are derived from widely used graph neural network architectures by removing trainable parameters and nonlinearities in their respective message passing layers. Experimentally we find that untrained message passing layers can lead to competitive and even superior link prediction performance compared to fully trained message passing layers while being more efficient and naturally interpretable, especially in the presence of high-dimensional features. We also provide a theoretical analysis of untrained message passing layers in the context of link prediction and show that the inner product of features produced by untrained message passing layers relate to common neighbour and path-based topological measures which are widely used for link prediction. As such, untrained message passing layers offer a more efficient and interpretable alternative to trained message passing layers in link prediction tasks.


Review discussion:
1. The paper introduces untrained message-passing architectures, extending existing research from node classification to link prediction. This perspective is relatively novel and addresses the computational limitations of GNNs. The untrained models, by eliminating learnable parameters, are shown to be faster and more resource-efficient, making them suitable for large-scale applications.
2. Theoretical results provide a deeper understanding of how UTMP layers approximate traditional path-based link prediction metrics (e.g., random walks and common neighbors), making the models highly interpretable.
1. Limited baselines are compared. Path-based methods and edge-wise methods should be compared. This doesn't mean the authors should change them into untrained models and do comparison. The author discuss the theoretical relationship with path-based methods, so the emperical comparison is needed to validate the theory.
2. Limited datasets are included. Large datasets like OGB datasets are not included so the application is limited.
3. For non-attributed graphs the results of original GNNs are much better than on the attributed graphs, compared to S-models and UT-models. Can the authors do some ablations to discuss this observation? Maybe it's because of the one-hot encoding? Can the authors show the results of one-hot encoding in attributed graphs?
See weakness.
No ethics review needed.
We thank the reviewer for their insightful comments. Individual concerns are addressed below.
## Reply to weaknesses:
1- In the paper we focus on comparing trained MP layers with their untrained counterparts. This also motivates our choice of using GAEs as an experimental setting as their simplicity allows for an objective comparison between layers.  LP performance of path based measures is widely reported in the literature, however as expected such purely topological measures that do not make use of node features are usually outperformed significantly by methods that make use of features information.   Some path based methods (CN, AA, RA) are already included in Table 9 in the appendix and Table 10 will also be updated to include these measures, in the updated version of the manuscript. We have now also updated the manuscript to include experiments that use NCNC [2] with trained and untrained GCN layers.
2-Due to the large size of the ogb-datasets performing systematic large scale experiments, which in our case would have to include fully trained models, is not feasible with reasonable computational resources.
For instance in [1] the authors report multiple OOM (>50 GB) errors on ogb-datasets for GAEs even for rather restricted ranges of hyperparameters. This is not to say that UTMP layers can’t be used in conjunction with ogb-datasets, it is just that performing the type of systematic experiments we do in the paper are extremely time and resource intensive for these data sets.
3- The results do not change significantly when performed with random features as the use of random features can be seen as a simple change of basis in feature space.
Moreover the use of one-hot encodings is not realistic for the larger attributed data sets as some of these have almost 20k nodes and there is limited value in transforming attributed data sets into unattributed ones.
[1] Li, Juanhui, et al. "Evaluating graph neural networks for link prediction: Current pitfalls and new benchmarking." Advances in Neural Information Processing Systems 36 (2024).
[2] Wang et al. Neural common neighbor with completion for link prediction.arXiv:2302.00890, 2023.
* To the best of my knowledge, this is the first application of untrained message-passing layers to link prediction.
* The empirical results show that untrained layers perform reasonably well.
* Some theoretical observations are included to complement the empirical analysis.
* Lack of novelty: Most of the work builds directly on top of Wu et al. and mirrors many parts of it. The architecture and setup are almost exactly the same, and even some claims, like the benefit of efficiency and interpretability, are taken straight from there. This is not to say that they are not true, but for example in the case of interpretability, there is not much evidence provided beyond the fact that the architecture is simpler.
* The theoretical contribution in the paper feels somewhat limited, with unclear takeaways.
Although the authors aim to support their work with theory, the analysis falls short of substantiating the core claims.
For instance, the authors state that “Our theoretical analysis further provides insights into the effectiveness of widely used node initialization schemes such as one-hot encodings and random features.” However, as this theoretical analysis is restricted to feature vectors that are pairwise orthonormal - often true for one-hot encodings and random features - it doesn’t convincingly explain why these should be more effective than other potential initializations that lack this precondition.
Additionally, this analysis seems somewhat peripheral to the main focus of the paper: comparing untrained and trained MPNNs. To strengthen this aspect, I suggest clearly outlining the theoretical contributions by structuring them into theorems with proofs and more directly linking them to the paper’s central claim.
* The main point of the paper is the fact that untrained message-passing layers perform very well in comparison to their trained counterparts. To evaluate this properly, a good benchmarking framework is necessary that guarantees that the difference in predictive performance is coming from what the authors claim it’s coming from, and especially that the comparison is fair. This is where I have my main problem with this paper. No established benchmarking framework is used, and almost no attention is paid to the fact that link prediction tasks are notoriously hard to evaluate. I would recommend the authors to consult recent works like [1] and [2], which go into more detail on various problems, but let me name some important ones here: Looking at the provided code, it looks like negative sampling of edges is done randomly, which is likely to cause bad performance on these tasks.
This is problematic because it’s not clear if untrained layers really perform that well in comparison or if the training procedure was just not good. From Table 2, I can tell that in several cases, the final trained linear layer (which was also trained with random negative sampling) performs worse than the untrained one. This is really surprising to me and could be due to the negative samples that were not chosen well.
While looking at the code, I also noticed that the test dataset for the untrained variants is not the same as for the trained ones because the random link split in the dataset transform is initialized with `num_val=0.00, num_test=0.1` in contrast to `num_val=0.05, num_test=0.1` for the trained counterpart.
While I don’t expect this one to make a huge difference, it just goes to show that the benchmarking is not done thoroughly enough to warrant the claims made in the paper. Getting link prediction right is actually quite hard, considerably more so than for node and graph-level prediction tasks, and a paper that builds on top of these results that much should put more scrutiny into it. My proposal is this: Use an existing benchmarking framework. This also makes it possible to compare the results to other papers and to run with more recent datasets from ogb, which are completely missing from this analysis. On a side note, I think that these would be quite important as they are bigger and could demonstrate the claimed scalability advantage of untrained layers.
[1] Li, Juanhui, et al. "Evaluating graph neural networks for link prediction: Current pitfalls and new benchmarking." Advances in Neural Information Processing Systems 36 (2024).
[2] Zhu, Jing, et al. "Pitfalls in link prediction with graph neural networks: Understanding the impact of target-link inclusion & better practices." Proceedings of the 17th ACM International Conference on Web Search and Data Mining. 2024.
* While running the code I noticed that the inference for the untrained layers uses considerably more memory than even training the full GNN.
So much so that I ran out of memory on my laptop. I’m wondering, could your code actually be used for much larger graphs?
* The meaning of this sentence was a bit unclear to me: “ Since the simplified architectures consist of UTMP layers followed by a trainable linear layer, the consideration of UT models which do not include the linear layer also covers all possible ablation studies” Could you clarify?
* Some other suggestions for improvements were already described together with the weaknesses.
In light of the weaknesses I mentioned, I tend towards a reject. Especially the empirical analysis has to be improved considerably.
No ethics review needed.
We thank the reviewer for their detailed and insightful comments. Individual points raised by the reviewer are addressed below.
## Replies to weaknesses
1- We clearly attribute the introduction of UTMP to Wu et al. and follow their formulation.
Consequently some similarities in terms of training efficiency improvements with the ones reported by Wu et al. are to be expected.  Our goal is to analyse  UTMP layers and their surprising performance from the perspective of LP.
Although the simplicity of UTMP certainly plays a factor in them being more interpretable, the claim of interpretability stems from the theoretical analysis which shows how features produced by UTMP layers encode paths,  and hence also neighbourhood structures, which are essential for LP tasks. We further relate these to topological similarity measures that are widely studied in the literature and are also used as subcomponents of state of the art LP methods.
2- Our theoretical results show how features resulting from UTMP can capture essential topological characteristics, namely paths between nodes, by leveraging orthogonality. As paths capture essential topological information that is also relevant in tasks other than LP this explains why orthogonal initialisation schemes work well in practice. This does not imply, and we do not claim, that OHE or random initialisations will be more effective than other initialisation schemes that do not warrant orthogonality, under all circumstances.
However, some degree of orthogonality should be expected for any initialisation scheme that uses high dimensional features as orthogonality is a typical property of collections of high dimensional vectors.
The theoretical results provide an explanation as to why UTMP layers perform well in LP tasks (often better than then their trained counterparts) and hence is directly relevant to the central claim of the paper. Once UTMP layers are formulated in matrix form the results follow from simple matrix algebra hence in our opinion there is no need to formalise these in terms of theorems and proofs.
- We follow  a standard benchmarking procedure that is widely used across the LP literature including all recent papers that introduce  sota LP methods e.g [3-5].
- Random sampling of negative edges is standard in the field, in the absence of predetermined data splits. There is no reason for this to lead to a suboptimal training procedure, moreover the results we report for fully trained architectures are in agreement with those reported in the literature.
- The difference in the split settings is due to the fact that UT models do not require any form of training and hence there is no need to create a validation set for UT models.
Otherwise, we use the same split settings for all models with trainable parameters i.e. fully trained models and simplified (S) models.
- [1] introduces a benchmarking procedure for LP, however is not widely adopted and hence using it would have distracted from the main results of the paper. In [2] the authors consider the effects of including target links while training, however this simply does apply in our case. Incidentally, the benchmarking procedure of [1] seems to include target links during training.
Although, we agree that [1] is an interesting benchmarking framework, it does not imply that the standard benchmarking procedure we use in the paper is flawed in any way.
- Due to the large size of the ogb-datasets performing systematic large scale experiments, which in our case would have to include fully trained models, is not feasible with reasonable computational resources.
For instance in [1] the authors report multiple OOM (>50 GB) errors on ogb datasets for GAEs even for rather small ranges of hyperparameters.
- The fact that for some of the attributed datasets the fully untrained architectures (UT) perform best can be explained by the fact that the inclusion of the linear layer significantly reduces the dimensionality of the feature vector which might impede the ability of the features to effectively encode neighbourhood information.
Moreover, the unattributed datasets are in general smaller which inevitably limits the amount of training data putting trained methods at a disadvantage.
## Replies to questions
- This is due to the pre-computation of features for UT models on the CPU to increase training efficiency, however this is an optional step that can omitted for larger dataset.
- The simplified models consist of two components UTMP followed by a a trained linear layer.
Hence removing the linear layer is the only possible ablation which results in the UT models.
- see above.
[3] Yun et al. Neo-gnns: Neighborhood overlap-aware graph neural networks for link prediction. Neurips, 2021.
[4] Wang et al. Neural common neighbor with completion for link prediction.arXiv:2302.00890, 2023.
[5] Chamberlain et al. Graph neural networks for link prediction with subgraph sketching.arXiv:2209.15486, 2022.
Thank you for your rebuttal. Here are my thoughts:
**Interpretability of UTMP Layers**
You state:
> "The claim of interpretability stems from the theoretical analysis which shows how features produced by UTMP layers encode paths, and hence also neighbourhood structures, which are essential for LP tasks."
I was unable to find this information in the paper itself; the text only mentions that UTMPs are “highly interpretable,” without further explanation or evidence. To argue that untrained layers are more interpretable based solely on this reasoning feels like a stretch. At a minimum, this claim requires stronger support. Can you provide concrete examples, perhaps from specific datasets, to show how this enhances interpretability?
**Benchmarking Practices**
The benchmarking practices you reference ([3-5]) are not particularly recent and are already discussed in [1], which outlines the limitations of these approaches.
For example, [1] offers detailed explanations of why negative sampling may not be ideal.
Another example of this issue lies in Table 9, where you compare your results with those in the literature, yet the random split percentages differ.
For instance, NCNC uses a 70%/10%/20% split, while your method uses 85%/5%/10%.
Moreover, the sampling strategies for these splits are not consistent.
Overall, I don't think that the paper is ready for acceptance.
The paper is written well and the presentation is good.
The analysis of the inter layer values to path-based measures is nice, though not unexpected.
Experimental results have been carried out on the usual link prediction benchmarks.
The idea of untrained and linear layers (as acknowledged by the authors) has previously appeared for node classification in Wu et al, ICML 2019.
So, the idea has limited novelty.
The assumption of orthogonality may not always hold especially under conditions of homophily where neighboring nodes have similar features.
Link prediction has been studied in many works and the impact of another paper is limited.
Can you comment on why orthogonality would be expected in homophilic networks?
No ethics review needed.
We thank the reviewer for the review. Individual concerns are addressed below.
## Reply to weaknesses:
- ‘the idea has limited novelty.’: We clearly attribute the formulation of UTMP layers to Wu et al. our goal is to provide a complementary analysis of UTMP layers from the perspective of LP where we find that they can lead to significant gains in terms of performance and training efficiency, as was observed by Wu et al. in the case of node classification.  Our theoretical analysis provides additional insights on the type of information captures by UTMP layers which not only explains why UTMP works well in practical settings but also provides a theoretical interpretation of LP algorithms that use UTMP.
Moreover, the theoretical analysis provides a direct link between MPGNNs and path based measures, both of which are essential components of state-of-the art LP methods.
- Orthonormality is a mathematical assumption that we use to establish the connection between path based measures and features produced by UTMP layers.
Although we do not expect this assumption to hold exactly in real world datasets, orthogonality is a typical/expected property for collections of high dimensional vectors/features. In the homophilic case where connected nodes tend to have similar features, as is the case for the attributed datasets we consider (see Figure 2), the deviation from orthogonality actually improves the link prediction performance as homophily leads to larger inner products/scores for positive node pairs/links.
- ‘Link prediction has been studied in many works and the impact of another paper is limited.’ -  LP is one of the core tasks in graph ML.
Furthermore, the main motivation of the paper is not to propose another LP method but to study UTMP in the context of LP. Our findings indicate that in many instances replacing trained MP layers with their untrained counterparts leads to better performing methods. In order to further support this finding we have now included additional experiments that uses the state-of-the-art LP method NCNC [1] instead of GAEs, where again we study the effects of replacing the trained GCN layers in NCNC with untrained UTGCN  layers and observe that this consistently improves LP performance.
Answer to questions:
- By definition orthogonality is not satisfied in homophilic graphs.
However, in homophilic graphs this deviation improves LP performance as it leads to higher inner products/scores for node pairs that are connected compared to the case of orthogonal features where the inner product would be based purely on the path structure of the graph.
Moreover, for sparse graphs where only a small fraction of all possible links are present, pairwise orthogonality can still hold to a high degree of accuracy even if the graph is  homophilic as disconnected node pairs form the overwhelming majority of all possible node pairs. This can also be confirmed empirically for the datasets we consider (see Figure 2).
[1] Wang et al. arXiv:2302.00890, 2023.
Thanks for the clarifications.
The pointed weaknesses regarding limited novelty and orthonormality remain.
If the performance is even better without orthonormal features (as in the case of homophilic networks), are the authors able to substantiate this with a proof?
Using untrained message passing layers for GNN can be a computationally efficient approach, which is an important topic in the community.
Weakness:
Overall, the presented work gives the impression of an early draft, and I find it challenging to fully assess its contributions in its current form. Below are some clear issues:
1. The presented “theoretical results” are poorly organized, and it is very challenging to judge its correctness given there is no clear distinction between the authors’ contributions and existing results. To be honest, I am not very sure what is the theoretical contributions provided by the authors. The results rely on oversimplified assumptions (e.g., orthonormality line 304); and the authors were linking random things together (e.g., PageRank line 348). It is very challenging for me to decipher what the authors what to convey here.
2. It does not seem like the experimental results support the authors’ claim. In many cases, the untrained variant of the network performs very poorly, especially in Hits@100 dataset. If the authors want to claim the simplified network performs very well, the paper should be written as such.
3. Lack of ablation experiments, intuitive results on synthetic datasets, or example results/visualizations on representative datasets.
4. The presented figure 1 is of low quality. Maybe at least show the standard deviation across different runs?
NA
No ethics review needed.
We thank the reviewer for the concise review. Individual points raised by the reviewer are addressed below:
- ‘it is very challenging to judge its correctness’ : the theoretical results follow directly from the definitions provided in the paper and simple matrix algebra. We would be happy to provide further clarification if the reviewer could be more specific about the parts they believe are unclear.
- ‘there is no clear distinction between the authors’ contributions and existing results.’:It would be helpful if the reviewer could provide specific references to the existing results they are referring to. Needless to say in the case we missed any such results we would be happy to acknowledge and discuss them.
- ‘ To be honest, I am not very sure what is the theoretical contributions provided by the authors’ : as clearly stated multiple times in the paper the main theoretical result of the paper is to establish a direct correspondence between the features produced by UTMP layers and path based measures which are widely used as link prediction heuristics on their own as well as subcomponents in state-of-the-art LP methods. Path based measures have been widely studied in the literature in the context of LP and our results demonstrate how MPGNNs can capture and leverage the path structure of graphs in LP tasks.
- ‘The results rely on oversimplified assumptions (e.g., orthonormality line 304)’ : From a mathematical standpoint we simply select an assumption that allows us to derive meaningful results. Whether the assumption of orthonormality applies to practically relevant settings is discussed at length in the paper. Just to reiterate the main points: orthonormality applies both for OHE and high dimensional features which are widely used in practice. Some degree of orthogonality is a typical/expected and a well known feature of high dimensional features spaces as can be verified experimentally (See Figure 2 in the appendix). As discussed in the paper orthogonality is not necessary for UTMP layers to perform in LP tasks and indeed deviations from orthogonality can in certain cases even increase the LP performance for instance when connected nodes tend to have more similar/non-orthogonal features as is  the case in many real world data sets ( see Figure 2).
- ‘and the authors were linking random things together (e.g., PageRank line 348). It is very challenging for me to decipher what the authors what to convey here.’ :  PageRank and the other measures discussed in the paper are defined in terms of path structures in graphs and are widely used and studied in the context of LP and hence clearly relate to both UTMP layers and LP.
2-As stated in the paper the fully untrained (UT) models should be considered as a baseline for how informative the features produced by UTMP layers are for LP tasks. Note that in the paper we focus on MP layers and not specific LP methods consequently in our experiments we are comparing ‘MP layers’ and not methods. For the sake of simplicity we choose a GAE setting where we consider various MP layers and replace trained layers by their untrained (UT) counterparts resulting in what is termed 'simplified' models and observe a general improvement in LP performance when UT layers are used. We have now updated the manuscript to include experiments based on NCNC [1], where we replace trained GCN layers with untrained UTGCN layers and again observe that replacing trained layers with untrained layers improves performance.
- ‘Lack of ablation experiments’: because the GAE architecture we consider consists of only two components: MP layers followed by a linear layer the only possible ablation is the removal of the linear layer which is results in the UT models.
- ‘intuitive results on synthetic datasets’: to the best of our knowledge there are no widely used synthetic benchmarks for LP and none of the recent papers on LP methods use synthetic data sets. The underlying reason for this might be that synthetic datasets usually consist of either random graphs where the prospects of LP are fundamentally limited or highly regular graphs such as grids where LP is trivial.
- ‘example results/visualizations on representative datasets’: we would be happy to include such additional examples and visualisations if the reviewer could elaborate these further.
- ‘The presented figure 1 is of low quality.’: The figure shows how LP performance changes as the number of UTMP layers is increased. We checked the figure again and there does not seem to be any issues regarding the resolution or readability of the figure.
- ‘Maybe at least show the standard deviation across different runs?’: the typical standard deviations can be found in Table 1 which in many cases would be smaller than the symbols.
Moreover it is unclear how the inclusion of std deviations would improve the figure/add any information that is relevant to what the figure is trying to convey.
[1] Wang et al. arXiv:2302.00890, 2023.

Meta-review:
role: AC
data: metareview: value: This paper introduces an approach for link prediction in graphs with untrained message passing layers. The authors show that untrained MP layers can outperform fully trained models while offering better efficiency.

The reviewers appreciated the idea that untrained MP layers could improve efficiency, there were a number of issues raised about the contributions of the work, whether the approach could be used on large graphs, and the high amount of overlap with previous work from Wu in 2019. Additionally, there was a major concern raised about the benchmarking procedure utilized in this work and how it may lead to inconclusive claims about what leads to the benefits of their approach.
additional_comments_on_reviewer_discussion: value: The authors did provide some justifications for their benchmarking approach and a discussion of the novelty over Wu. However, the reviewers were not convinced and therefore they all agreed that the paper was not ready for publication.
role: PC

Decision: Reject