Revisiting Embeddings for Graph Neural Networks
Authors: Skye Purchase
Yiren Zhao
Robert D. Mullins
Venue: Submitted to ICLR 2023 (2023)
Keywords: Graph Neural Networks
Embeddings
Graph Attention
Large Pretrained Models
Transfer Learning

Abstract: Current graph representation learning techniques use Graph Neural Networks
(GNNs) to extract features from dataset embeddings. In this work, we examine
the quality of these embeddings and assess how changing them can affect the ac-
curacy of GNNs. We explore different embedding extraction techniques for both
images and texts; and find that the choice of embedding biases the performance
of different GNN architectures and thus the choice of embedding influences the
selection of GNNs regardless of the underlying dataset. In addition, we only see
an improvement in accuracy from some GNN models compared to the accuracy of
models trained from scratch or fine-tuned on the underlying data without utilising
the graph connections. As an alternative, we propose Graph-connected Network
(GraNet) layers to better leverage existing unconnected models within a GNN.
Existing language and vision models are thus improved by allowing neighbour-
hood aggregation. This gives a chance for the model to use pre-trained weights, if
possible, and we demonstrate that this approach improves the accuracy compared
to traditional GNNs: on Flickr v2, GraNet beats GAT2 and GraphSAGE by 7.7%
and 1.7% respectively.

Under review as a conference paper at ICLR 2023
REVISITING EMBEDDINGS FOR GRAPH NEURAL NET- WORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Current graph representation learning techniques use Graph Neural Networks (GNNs) extract features from dataset embeddings. this work, we examine the quality of these embeddings and assess how changing them can affect the ac- curacy of GNNs. We explore different embedding extraction techniques for both images and texts; and find that the choice of embedding biases the performance of different GNN architectures and thus the choice of embedding infuences the selection of GNNs regardless of the underlying dataset. In addition, we only see an improvement in accuracy from some GNN models compared to the accuracy the underlying data without utilising
models trained from scratch or fine-tuned the graph connections. As an alternative, we propose Graph-connected Network (GraNet) layers to better leverage existing unconnected models within GNN. Existing language and vision models are thus improved by allowing neighbour- hood aggregation. This gives a chance for the model use pre- pre-trained weights, if possible, and we demonstrate that this approach improves the accuracy compared traditional GNNs: on Flickr v2, GraNet beats GAT2 and GraphSAGE by 7.7% and 1.7% respectively.
INTRODUCTION
Graph Neural Networks (GNNs) have been successful on a wide array of applications ranging from computational biology (Zitnik & Leskovec (017) to social networks (Hamilton al. 2017). The input for GNNs, although sourced from many different domains, is often data that has been prepro- cessed to computationally digestible format. These digestible formats are commonly known as embeddings.
Currently, improvements made GNN architecture tested against these embeddings and the state of the art determined based on those results. However, this does not necessarily correlate with the GNNs accuracy on the underlying dataset and ignores the inffuence that the source and style of these embeddings have on the performance of particular GNN architectures. To test existing GNN architectures, and demonstrate the importance of the embeddings used in training them, provide three new datasets each with set of embeddings generated using different methods.
We further analyse the benefit of using GNNs on fixed embeddings. We compare GNNs to stan- dard models that have been trained or fine-tuned on the target raw data; these models treat each data point as unconnected, ignoring the underlying graph information in data. This simple uncon- nected baseline surprisingly outperforms some strong GNN models. This then prompts the question: Will mixing the two approaches unlock the classification power of existing unconnected models by allowing them to utilize the gruph structure in our data?
Based on the question above, we propose new method of mixing GNNs with unconnected models, allowing them to train simultancously. To achieve this introduce variation of the standard message passing framework. With this new framework subset of the unconnected model's layers each be graph-connected exploiting useful graph structure information during the forward pass. We demonstrate that this new approach improves the accuracy using only a pre-trained or fine-tuned model and outperforms stand-alone GNN fixed embedding.
We call this new approach GraNet (Graph-connected Network), and in summary, this paper has the following contributions:

---

Under review as a conference paper at ICLR 2023
We provide new datasets and a rich set of accompanying embeddings to better test the performance of GNNs.
We empirically demonstrate that only some existing GNNs improve on unconnected model accuracy and those that do vary depending on the embeddings used. We urge that uncon- nected models be used
baseline for assessing GNN performance.
We provide a new method, named GraNet, that combines GNNs and models (fine-tuned or trained from scratch) to efficiently exploit the graph structure in raw data.
We empirically show that GraNet outperforms both unconnected models (the strong base- line) and GNNs on a range of datasets and accompanying embeddings.
2 RELATED WORK
Graph Augmented Networks Chen et al. (2021) introduce Graph suaphauggmeted MultiLayer Percep- trons (GA-MLPs) as simplified alternative to Graph Neural Netwroks (GNNs), These models involve two step process augmenting the node features of the graph based on the topology and then using these node features applying a learnable function at the node level. This allows fixed graph operator and two sets of MultiLayer Perceptrons (MLPs) be used extract features from the graph. This approach is related to similar simplified GNN techniques (Wu et al. 2019 Nt & Maehara 2019). The paper proves that this simplified approach is not as expressive as standard GNNs when looking at the Weisfeiler-Lehman test for distinguishing non-isomorphic graphs. This suggests that GNNs are well suited for inferring information based on graph structure but the paper does not comment which approach is best in practice. We differ in our approach to augmenting networks with graph structure by using existing GNNs and do not attempt to simplify the network. We do provide graph-connected MLP but this looks adding message passing MLPs rather than separte funcitons on the graph data.
Effect of training on GNN performance /Shchur al. (2018) look at the effect of hyperparameters and training in GNNs to show that these have dramatic effect on model ordering. Simply chang- ing the split on a dataset caused large changes in accuracy and which GNN performed best, even though the hyperparameters of the GNNs remained constant. We show similar large difference when considering different embeddings with the same splits across embeddings.
Ablation studies on GNNs Further to these discoveries Nt & Maehara (2019) demonstrate that GNNs only utilise the graph structure to de-noise already highly informative features. They go as far as to demonstrate in certain conditions GNNs and MLPs perform the same. Chen et al. (2019) demonstrate that linearising the graph filter stage of GNNs does not hinder but actually increases the performance. Similarly Wu al. (2019) simplify GNNs by removing non-linearity between layers allow for pre-computing the & message passes. This reduces graph representation learning to simple linear regression. In all of these cases they demonstrate that the major contribution of
GNNs is in their graph structure capabilites. We do not analyse these aspects but look at how this capability can be used in existing unconnected networks.
BACKGROUND
Table 1: An overview of popular datasets
Name
Info Source
Classes
Feature Length
Embedding
Amazon Shchur al.
2018
Text
10,
767,
Bag of Words
Arozucnroducts Zeng cf al.
2019
Text
107
200
4-gram with SVD
Flickr Zeng et 2019
Image
500
Bag of Words
Reddit ZZng 2201 Hamiloo
2017
Text
41
602
Avg. Glo vectors
Cora Kkpp Welling 2017)
Text
1,433
Bag of Words
CiteSeer (kpp & Welling 2017
Text
6
3,703
Bag of Words
PuhMed Kipl & Welling 2017
Text
500
Bag of Words
We compare our new method (GraNet) against some standard Graph Neural Networks to demon- strate the improvements that GraNet makes in classifying datasets.
2

---

Under review as a conference paper at ICLR 2023
Kipf & Welling (2017) introduce GCN (Graph Convolutional Networks) method of applying convolutional layers from CNNs to graph neural networks. focuses on spectral filters applied to the whole graph structure rather than at the node level.
Hamilton et al. (017 introduce the GraphSAGE model which builds on prior work from GCN focusing on individual node representations. This gives rise the iterative message passing process on the node level. Though simpler than newer models we find that this approach, when given the right embedding style, can outperform some recently published GNNs.
Veličković et al. 2018) introduce the idea of graph attention which alters how a node aggregates its neighbours representation. This adds additional attention mechanism to discern which aspects of the node representations in nodes neighbourhood are important at given layer.
Brody al. (2021) provide a more attentive version based on the graph attention system introduced in Veličkovic et al. (2018). We base the graph attention mechanism used in our GraNet models on this improved version of graph attention. We provide both versions of graph attention in our results to compare to our new approach.
3.1 NOTATIONS
Graph Data Let X) denote a graph where V = {U1, 02 ..., Un. is the set of nodes and N = | is the number of nodes in the graph, & is the set of edges between nodes in V such that ei,j E E denotes a directed connection from node v; to node vy. Ei, may itself be a feature vector. We say each node v; has a neighbourhood N; such that by N2 < Cj,i and say that vj a neighbour node to v,. Where is the raw data matrix where X., = I; where æ; is the feature vector for node Us.
Embeddings There normally exists a transformation function, Se. to project the raw data to a more compact feature X. space such that X. = (X)
For instance, we can transform a set of images (X E NXNNCCHWW where C, H and W are the number of channels, width and height of an image) ID features (X, E Nx where F denotes the feature dimension). In this case, we have an embedding function fe BXNXCXHWW + +RNXF for the dimensional reduction.
This paper puts special focus on the design of fe, and reveals later how the design choice of fa can influence the performance of GNN models without making any changes to the underlying data G(V, E, X). An overview of popular datasets, and the embeddings that they use. is presented in Table We see that the popular graph datasets (Zeng al. 2019; Kipf & Welling 2017 Shehur et al. 2018; |ammiloo et al., 2017) focus heavily on Bag of Words (BoW) and word vectors. This implies that current GNNs are being tested on and designed for a very narrow class of embedding styles. A more detailed discussion is available in Appendix A
Graph Neural Networks Current GNNs can be thought of as message passing layers, the I-th layer can be represented
h = 70, (M/-1 vbyEN()) (oo, (hi-1 h// ee.))))
where v is a differentiable aggregation function and Ye, and do, represent differentiable functions with trainable parameters 0,, and 02 respectively. ht, is the node representation of at layer I, with
h fe(;)
We focus on the improved graph attention mechanism (Brody et al. 2021 when looking at potential
GNNs for GraNet. Using this new notation we can formulate it as such
aij = ΣKEN, LeakyeLLUO0) (h,l)))
(2)
where a is a learnable parameter representing the attention of the network. Further discussion of graph attention and why this is useful in GraNet is available in Appendix C
3

---

Under review as a conference paper at ICLR 2023
Pre-trained models are specific neural network architecturwa that have been trained dataset D for a specific task, this could be ImageNet classification (Deng et al. (2099) for vision networks (He et al., 2016) or the entire English Wikipedia for language models (Liu et al., 2019). These networks therefore have pre-trained weights A that can be loaded into the model for further training evaluation.
We denote these pre-trained models as fe that is parameterised by weights 0. We say that a pre- trained model has set of functions { -0. ".. f for M-layer model, where fă, R Rf and fe. : =R= + FR and J. is the feature dimension. A pass through single layer, of network would be So (a)), shorthand for (a;; 0t). If we concatenate these layers form a full pass through the network, we obviously have [a( - f (...Sa_ (So, (æ)))
Fine-tuning is therefore adapting I to a new dataset DD' which is related to G(V, E, X) as illustrated in Section 3.1] In more standard setup, the target dataset fine-tune to has the same underlying data-structure as the pre-training dataset, for instance, they might both be images, but the target dataset is a different type of classification. This may involve adding, removing altering specific layers within the model or simply retraining the model with different labels on D.
If the architecture of pre-trained model is altered then a new weight matrix 0' must be created from @ by adding, removing or reshaping weights.
Freezing layers is the process whereby a selection of weights 0 C do not have gradients and thus do not change during back-propagation
These ideas allow us alter these pre-trained models to use information about the graph connections whilst utilising their pre- rre-trained weight O.
Blending and Fine-tuning models is therefore the process of using existing models, fo, and ge,- with defined set of layers, (sa, "...). and ......... I0 J. and creating a new model, he,.. such that he,, f o "e. We can use pre-trained weights f or fg and/or freeze cither model, and where one of these models a GNN we say this fine-tuning on a graph dataset.
Unconnected models are models that, unlike GNNs, do not use any information about graph con- nections within a dataset. These are trained on datasets that are not graph-connecte where each datapoint is consider isolated. We focus complex unconnected models with multiple layers, such as vision networks, which we call large models. Due to training cost we use pre-trained large models.
METHOD
Our proposed method of converting standard neural network models into graph-comnecte models blends the two networks. This approach can easily be broken down into individual layers. Taking fh as the 1-th layer in standard model, fo. where we may be given pre-trained weights O we can describe this new layer by reformulating Equation (T) as such
h = =0 (ne (n!-1. (bSeNte) (oo. (F%_1 (h/ (6-1,, (h)-1) e ee,)))
(3)
where hi rather than applying an embedding function.
Figure [ is a representation of Equation (3) specifically the case where the pre-trained model is CNN and ((b_)) ) is convolutional layer. The light blue regions perform the standard convolution feature extraction, these extracted feature maps are then adjusted, the light orange region, by Message Passing layer with graph attentions. These two representations then combined.
The light blue regions and resulting channel stacks, A E XIxGxx,,W,, through An e x xWW, represent the forward pass through single CNN layer f/ 1' A represents the forward pass the current node h, and A(j...... represent the forward pass of the neighbours of h,. The light orange region and resulting channel stacks, B, represent the graph-based Message
4

---

Under review as a conference paper at ICLR 2023
Figure 1: Graphical representation of how proposed GraNet layer operates for image networks
A
70,
4
h
(A Ag E.
4
N
N2
B
Passing stage where the new representations are altered (do,).. aggregated (+) and finally combined with the current node representation (70,). following the description in Equation
4.1 GRANET FOR FLICKR_V2
Interconnecting every layer in a large pre-trained model is computationally intensive. Therefore, rather than interconnect every single layer pre-trained network fo we can graph-connect only the final layer. We thus split the model into two portions: the first set of unconnected layers and the final GraNet layer.
We can therefore look at equation Equation (3) and see in this case that we partially carry out the forward pass of fo which we will denote as Je and then carry out forward pass through GraNet layer. This allows to ignore all the steps involved in applying fe and focus the final GraNet layer. Denoting this final GraNet as g, and output classification y, we achieve the following equation
y = (X))
(4)
fe(X) is the same as described in Section and indeed if were to freeze So this would be equivalent to training GraNet with a single layer on the embeddings created by embedding function fe- So instead also allow to train thus fine-tuning the weights O of fe. This indirectly allows fo to learn the graph structure by providing g with better embeddings.
As GraphSAGE performed the best ResNet embeddings already pre-trained but ge, is not fine-tuning the model produces poor results. We therefore initially train 9e, frozen feay= after this short training period unfreeze Sea, allowing both to train fine-tuning the weights 0g and 0,.
this
GNN, g. As fo, is
4.2 GRANET FOR AMAZON
In the case of the Amazon dataset we found that Bag of Words embedding performed the best. As this does not have an associated pre-trained model we design multi-layer perceptron (MLP) to compare against. We then convert all the layers within the MLP to GraNet layers. This method is therefore not fune- but trained from initialised weights.
We also find that GAT (Brody et al. 2021 Veličković et al., 2018) models perform the best on this task. We therefore use graph attention message passing as shown in Equation (2) for our GraNet model. Keeping in line with Equation 3 the new graph attention mechanism becomes
LeakyeLLe (h.))l (h₂)))
Ouj EvEN, Leppaa"LLaayRLLU(111 (h.)l.f,, (h,)))
(5)
However, a single layer of an MLP is fo, (x) = 0,x. This would therefore mean that Equation (6) becomes

---

Under review as a conference paper at ICLR 2023
expla' (ea 0,h,l)) 0,h;]))
aij Σ KEN, LappaalLeakRRe....
(6)
This is very similar to Equation 2 with the only difference being when we apply the weight ma- trix, concatenation and attention mechanism. We find this model behaves the same GAT2 and therefore, given the small size tried different approach as comparison.
Rather than use the vector parameter a we introduce a linear function oo, IR2FF + RF that takes the LeakyReI.U of the concatenation of node, fo, (h:), and its neighbour, So, (h;), as input. The result a feature vector in a new feature space, though in our case we have F³ F. This allows more complex interactions between the node representations to be exploited by attention. This approach is too costly to apply to pre-trained CNN as the size of fa is far too large.
5 EXPERIMENTAL SETUP
For all test results we run train-test runs each with different random seed and take the arithmetic mean and include the standard deviation. The three random seeds are the same for every entry in the table for fair comparison. The architecture used for each benchmark GNN identical across all datasets and embeddings. Training takes 300 epochs of training unless otherwise stated.
Where Graph Neural Network (GNN) is used part of GraNet model (as described Section 4 the same sampler is used for consistency. The specific architectures used are designed using the hyperparameters in Zeng et al. (2019). In the case of GraphSAGE the learning rate was decreased to improve convergence. For specific details of each architecture and learning rate see Appendix L
5.1 DATASETS
An overview of the datasets is provided below with the specific metrics for each dataset shown in Table 6 A more detailed discussion is available in Appendix B It is important to note that though these datasets mirror prior datasets due to the need for raw data we diverge from these datasets. Therefore we do not make any direct comparisons to previous datasets though the results we achieve on our new datasets are on par with results seen in prior papers.
5.1.1 FLICKR V2
The underlying data X is raw images and so Convolutional Neural Networks (CNNs) are used as embedding functions. As a sample of existing pre-trained CNNs we use ResNet18, ResNet50 (He et al. 2016) and VGG16 (Simonyan & Zisserman 2015). In all three cases we use the pre-trained models provided by Torch Vision, using the feature vectors after the final pooling stage before the classification stage.
is important to note that there is no Bag of Words (BoW) embedding for Flickr y2 because there is sensible object that can be considered "word" for raw image data. The available Flickr (Zeng et al. 2019) uses BoW because underlying data is image text descriptions not raw images. This limits how GNN, or general neural network, can classify images the images must first be processed to provide text descriptions.
5.1.2 AMAZONELECTRONICE AND AMAZONINSTRUMENT
The underlying data X is text so text classification transformers such as RoBERTa (Liu et al.) 2019) are ideal, specifically we use pre-trained RoBERTa. We extract three different embeddings from RoBERTa using the pre-trained model provided by fairseq toolkit (Ott et al. 2019). The first is the byte pair tokenisation used by RoBERTa, the second is the feature extraction provided by fairseq which occurs after RoBERTa's transformer heads and before classification, and the final is the fea- ture vector present before the last fully connected layer. Due to restrictions in the token size for RoBERTa remove all nodes that have reviews with greater than 512 tokens.

---

Under review as a conference paper at ICLR 2023
We also provide the standard Bag of Words embedding in the of text classification this is common embedding practice, keeping in line with prior datasets (Kipf & Welling 2017 Zeng et al. use the top 500 words create our Bag of Word embeddings.
2019)
EVALUATION
6.1 RE-EVALUATING EMBEDDINGS
Language Tasks Table demonstrates that the particular embedding function used determines which GNN model performs best on the dataset. For instance, GAT2 has the best performance the data is embedded as BoW (Bag of Words), but performs poorly other embeddings generated from RoBERTa. GraphSAGE. in contrast, performs poorly on BoW but shows a good performance otherwise. Thus which embeddings are used when comparing models has a large effect on which model appears to be the better model. We see the same effect in Table Comparing the two tables see that in the case of the RoBERTa Encoded and RoBERTa embeddings the order of the models changes, this likely comes from the training effects described in Shchhr et al. (2018).
Table also signifies the importance of good embeddings in this case BoW is better than RoBERTa. The complexity of the embeddings does not necessarily improve the efficiency of the classification. This ties into the results of Nt & Maehara (219)) the BoW is more informative of the classification containing the label words in the BoW vector. What is also important to note that when looking at the performance of the GNN against the unconnected model (A 1) we see consistency in the difference across the RoBERTa encodings. This suggests that the "quality", ", de- noted by how well the simple MLP performs, a strong indicator of how well model will perform, rather than the model architecture itself.
We see an overall decrease in accuracy in Table 3 across the models but this attributed to the fact that there are more classes for the dataset. We also see that for RoBERTa embeddings the MLP performs poorly though it does occasionally improve on the simpler models, primarily GCN. With more resources would be better suited to fine-tur RoBERTa (or some other language transformer) to our dataset and use this model as our unconnected model.
Table 2: Test accuracy on Alazonelestrroics with different embeddings compared against a stan- dard unconnected MLP model. The embedding styles are explained Appendix B Table
Embedding styles
Model
Bag of Words
Byte Pair
RoBERTa Encoded
RoBERTa
Unconnected MLP
71.6% (+0.0)
21.6% (+0.0)
55.8% (+0.0)
51.9% (+0.0)
GCN
69.1% (-2.5)
21.7% (+0.1)
22.7% (-33.1)
22.3% (-29.6)
GAT
81.1% (+.5)
22.2% (+0.6)
46.1% (-9.7)
40.3% (-11.6)
GAT2
81.8% (+10.2)
22.2% (+0.6)
41.8% (-14.0)
35.7% (-16.2)
GraphSAGE (Random)
71.3% (-0.3)
26.8% (+5.2)
57.0% (+1.2)
53.7% (+1.8)
GraphSAGE (Neighbour)
76.4% (+4.8)
40.4% (+20.8)
67.8% (+12.0)
66.4% (+12.5)
Table 3: Test accuracy on AmazonInstruments with different embeddings compared against stan- dard unconnected MLP model. Included is the difference A of each model to the unconnected MLP and the standard deviation of each result. The embedding styles are explained Appendix B
Embedding Styles
Model
Bag of Words
Byte Pair
RoBERTa Encoded
RoBERTa
Unconnected MLP
66.1% (+0.0)
21.0% (+0.0)
43.9% (+0.0)
39.8% (+0.0)
GCN
64.0% (-2.1)
20.8% (-0.2)
20.4% (-23.5)
20.4% (-19.4)
GAT
79.3% (+13.2)
21.6% (+0.6)
47.5% (43.6)
46. 1%.1% (+6.3)
GAT2
79.4% (+13.3)
21.2% (+0.2)
49,8% (+5.9)
47.8% (+8.0)
GraphSAGE (Random)
67.5 (+1.4)%
23.9% (+2.9)
45.1% (+1.2)
41.9% (+2.1)
GraphSAGE (Neighbour)
72.6% (+6.5)
43.4% (+22.8)
62.4% (+18.5)
59.9% (+20.1)

---

Under review as a conference paper at ICLR 2023
Vision Tasks Table 4 demonstrates the same pattern, that the embedding function (in this case pre-trained vision model) influences which GNN performs the best. It is interesting to note that we see that none of the GAT models achieve the highest accuracy any of the Flickr v2 embeddings. Instead, similar to the RoBERTa embeddings, we see that GraphSAGE performs the best.
It is important note for VGG16 that the surprisingly poor performance of GNNs is more likely due to the large vector size of more than 25K. With a smaller embedding space better results on par with ResNet may be achieved. Of course, there is also the possibility that the embeddings provided by VGG16 are inferior to ResNet.
Table Test accuracy on Flickr_v2 with different embeddings, compared against the corresponding unconnected vision model. Included is the difference A of each model to the unconnected model and the standard deviation of each result. The details of the embedding styles are available in Appendix
Embedding Styles
Model
ResNet18
ResNet50
VGG16
Unconnected Model
45.2% (+0.0)
46.9% (+0.0)
47.0 (+0.0)
GCN
41.8% (-3.4)
38.3% (-8.6)
35.5% (-11.5)
GAT
38.1% (-7.1)
37.1% (-9.8)
27.3% (-19.7)
GAT2
42.1% (-3.1)
41.0% (-5.9)
34.2% (-12.8)
GraphSAGE (Random)
45.4% (+0.2)
47.0% (+0.1)
35.2% (-11.8)
GraphSAGE (Neighbour)
45.8% (0.6)
44.5% (-2.4)
34.5% (-12.5)
Tables 2 and 3 compared to Table 4 have far larger increases in accuracy from the best performing
GNN compared the unconnected models. This is mainly due to the fact that we were unable to fully fine-tune RoBERTa to our datasets given limited hardware and time. We hypothesise that the improvements seen in Table 2 would be smaller when compared a fine-tuned RoBERTa. Similarly, did not attempt to create GraNet model using VGG16 the results on Flickr-v2 are worse than the ResNet models and therefore for Flickr a GraNet ResNet is idcal.
Tables 2 and 3 specifically with BoW, are the only instances where GAT and GAT2 are the best models. Furthermore, the entries for BoW follow results from past papers, in contrast to the results shown on all other embeddings. With the prevalence of BoW as shown in Table begs the question as to whether we are optimising for BoW extraction rather than graph information extraction.
From these results in both language and vision tasks, make the following key observations:
The function fe used to extract embeddings influences the performance of different GNNs, the embeddings should influence the choice of GNNs regardless of the underlying data. GNN models do not always outperform simple unconnected models, graph structure not enough compete against good classifiers.
The choice of an embedding function contributes more to the final performance compared the choice of GNN model.
6.2 GRANET
Table 5 demonstrates that extending our standard models with graph connections provides sig- nificant improvement. We see that these GraNet models beat the best performing GNN. The table includes two forms of GraNet models, the fine-tuned models and the trained models.
Fine-tuned In Flickr_v2, both GraNet and Unconnected Model use either ResNet18 or ResNer50 weights pre-trained on ImageNet (Deng al. 2009) and then fine-tuned the target dataset. We observe a significant increase (+ 1.5%, + +18%% from the GraNet style of training compared to both unconnected models, and improvement (1.1%, 1.7%) the best performing GNN model.
Intuitively, GraNet layers reframe graph representation learning from training a GNN on a fixed pre- extractedde embedding to training both the GNN and the embedding function (fe) together the underlying data. There are obviously trade-offs between time and versatility as rather than just training GNN the embedding function must be trained well. But the current approach of GNN
8

---

Under review as a conference paper at ICLR 2023
training on images requires human, other annotators, which is hidden time cost. By combining the embedding function and GNN we provide a more general model that does not need external annotators and thus work any raw images.
Trained from Scratch In the Amazon datasets we do not have any pre-trained and train the GraNet model from initialised weights rather than fine-tuning. However, we still observe improvement on both the unconnected model and best performing GNN. The improvement is smaller than in Flickr_v2 v2 (+0.11,, +0.3% compared to 1.1%, 1.7%). But the training time quicker (~ 1hr com- pared to ~ 10hrs for 300 epochs).
In this case do not have a general model we rely BoW embeddings for the GraNet model as well. But we can see that improvements are possible on existing GNN techniques on the embeddings themselves, without having to use large pre-trained models.
Table Comparison of GraNet models against the best performing GNNs for specific embedding. Unconnected Model refers to ResNet18 or ResNet50, in the case of Flickr v2, and an MLP other-
wise. The setup of each GraNet model detailed in Sections 4.1 and 4.2 and the specifies of the
datasets can be found in Appendix B
Flickr. v2
Electronics
Instruments
Model
ResNet18
ResNet50
Bag of Words
Bag of Words
Unconnected Model
45.2% (+0.0)
46.9% (+0.0)
71.6% (+0.0)
66.4% (+0.0)
GAT2
42.1% (-3.1)
41.0% (-5.9)
81.8% (+10.2)
79.4% (+13.0)
Grap (Random)
45.4% (+0.2)
47.0% (+0.1)
71.3% (-0.3)
67.5% (+1.1)
GraphSAGE (Neighbour)
45.8% (+0.4)
44.5% (-2.4)
76.4% (+4.8)
72.6% (+6.2)
GraNet
46.7% (+1.5)
48.7% (+1.8) 81.9% (+10.3)
79.7% (+13.3)
We also make the following key observations:
Graph-connecting a pre-trained network improves performance by fine-tuning feature ex- traction based on the graph structure.
GraNet models outperform their counterparts by facilitating fine-tuning on the graph dataset.
The architecture for each GraNet is a mixture of the best performing GNN for that embedding and the embedding extraction model (Ja) (a Multi-Layer Perceptron in the case of Bag of Words).
7 CONCLUSION
In this paper, we reveal that GNN model designs are overfitting to certain embeddings styles (e.g. BoW and word vectors). To demonstrate this we introduced three new datasets each with a range of embedding styles to be used as a more comprehensive benchmark of GNN performance.
We demonstrated that embedding style influences the performance of GNNs regardless of the un- derlying dataset. Equally, the quality of the embedding, measured by how well unconnected baseline model performs, is greater indicator of GNN accuracy than the GNN architecture chosen. We therefore stress the importance of creating high quality embeddings and choosing the best GNN architecture based on the style of embedding created rather than using (or trying to improve) the same GNN model for every task.
We then introduced a new approach named GraNet. This approach allows for any large pre-trained model to be fine-tuned to a graph-connected dataset by altering the standard message passing func- tion. In this way we exploit graph structure information to enhance the pre-trained model perfor- mance on graph-connected datasets. We have demonstrated that GraNet outperforms both uncon- nected pre-trained models and GNNs on a range of datasets.
There is an increasing trend towards large pre-trained models and graph-connected datasets. Our work demonstrates potential pitfalls in the way GNN architectures are currently evaluated and pro- poses a new technique fully exploit the benefits of pre-trained models within GNN.

---

Under review as a conference paper at ICLR 2023
8 ETHICS STATEMENT
With all computer based research there the issue of carbon footprint. Each train, validation and test run of a model requires electricity which is currently largely sourced from fossil fuels. This training cost carbon footprint is present in this picture, and attempts to keep this low by running short test on new iterations rather than long runs which could waste GPU time and energy.
REPRODUCIBILITY
We discuss how we setup all of our experiments in Section SectioonSincludee including how many runs we completed for each data-point the specific random seeds that we used for the three runs were 42, 9001 and 27032002. The number of runs for each run also listed here and for the models that have a different number of epochs the specifics are mentioned in Appendix F.
All test runs were carried out on single Nvidia 100 80GB GPU, we used pytorch 1.12 with cuda 1 11.3. All other packages where installed based on these requirements.
Appendix E also contains the specific architectures, samplers, learning rates and learning rate sched- ulers used in each of the model test runs. These are also linked back to Zeng et al. (2099) to provide comparable results to the ones achieved that paper for similar datasets Flickr and Amazon.
Appendix B details the specifics of how we formed our datasets including how we downloaded the raw data, steps we took to wash the data, how we formed the graph and how we created our embed- dings. The specifics of how we labelled the data and the labels that we chose are also available.
We have anonymised the dataset including the specific code used to create GraNet models, the config files used and the code to download, wash and build the datasets available here.
REFERENCES
Shaked Brody, Uri Alon, and Eran Yahav. How attentive graph attention networks? CoRR, abs/2105. 14491, 2021. BRLLhttps://arrtt http:///awwww.oooo////////////// 14491
3 5 15
Lei Chen, Zhengdao Chen, and Joan Bruna. On graph neural networks versus graph-augmente
[mlp}s. In International Conference on Learning Representations, 2021. URL https://
openreview net www.eeeetteee fert/forumme0 (ttprrummiii-iiiili1a11 7 2
Ting Chen, Song Bian, and Yizhou Sun. Are powerful graph neural nets necessary? a dissection on
graph classification. 2019. 2
Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yan-Tao Zheng. NUS- WIDE: A real-world web image database from national university of singapore. In Proc. of ACM Conf. on Image and Video Retrieval (CIVR (C)VR'09), Santorini, Greece., July 8-10, 2009. 14
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.
Ieee, 2009,
8
M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL visual object classes challenge: A retrospective. International Journal of Computer Vision, 111(1):98 -13,, January 2015. 14
William I.. Hamilton, Zhitao Ying, and Jure Leskovee. Inductive Representation Learning Large Graphs. In NIPS, pp. 1024-1034, 2017.
13
18
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Weihua Hu, Matthias Fey, Marinka Zitik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:221 01:22118-2211 18-22133, 2020. 12
10

---

Under review as a conference paper at ICLR 2023
Mark J. Huiskes and Michael S. Lew. The MIR Flickr retrieval evaluation. In MIR '08: Proceedings of the 2008 ACM International Conference on Multimedia Information Retrieval, New York, NY, USA, 2008. ACM. [4
Thomas N. Kipf and Max Welling. Semi-Supervised Classification with Graph Convolutional Net- works. In ICLR, 2017.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. 2019.
Julian McAuley and Jure Leskovec. Image labeling network: using social-network metadata for Springer, 2012.
image classification. In European conference on computer vision, pp.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. Image-based rec- ommendations on styles and substitutes. In Proceedings of the 38th International ACM SIGIR
Conference on Research and Development in Information Retrieval, SIGIR '15, pp. 43-52, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 978 4781450336215. doi:
10.1145/266462.2777773 URL
1145/2 66462 .26.222..765 2767755
Hoang Nt and Takanori Machara. Revisiting graph neural networks: All we have is low-pass filters.
arXiv preprint arXiv: /XXi::/900005500 2019. 2
Myle Out, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)) pp. 48 48-53,2015 2019.
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnnemann. Pitfalls of graph neural network evaluation. 2018.
K Simonyan and A Zisserman. Very deep convolutional networks for large-scale image recognition. 2015. 6
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks. In ICLR, 2018.
14
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Sim- plifying graph convolutional networks. In International conference on machine learning, pp. 6861-6871. PMLR, 2019. 2
Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. Graph- saint: Graph sampling based inductive learning method. International Conference on Learning Representations, 2019. 2
10
12
18
Marinka Zitnik and Jure Leskovec. Predicting multicellular function through multi-layer tissue networks. Bioinformatics, 33(14):110--1198 2017. 1
11


Reviews:
[{"review_id": "tuvEN0rJASn", "reviewer": "Reviewer_mwDM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies how the quality of embedding affects the performance of GNNs. The authors selected two types of data, images and texts, and tested different embedding extraction techniques. Then a general framework Graph-connected Network (GraNet) is proposed to combine GNNs and unconnected models to better learn embeddings of input data. Experiments on both images and texts with different embedding methods have been conducted to demonstrate the effectiveness of the proposed framework.", "summary_of_the_review": "This paper studies an interesting and important problem. Overall this paper is well-organized and easy to read. Experiments on different types of datasets have been conducted. However, I have some concerns about the novelty, and experiments, as well as the presentation of this paper. Therefore, I would like to reject this paper.", "strength_and_weaknesses": "Strengths:\n- The problem of investigating the influence of embedding for GNN is very interesting and there are rare studies on this problem.\n- Experiments on different types of datasets with different settings (embeddings) have been conducted to validate the effectiveness of the proposed framework.\n\nWeakness:\n- The novelty of this paper is limited. The proposed framework is a straightforward combination of neural networks and GNNs.\n- Presentation in this paper, especially introducing the method, sometimes is not very clear: in Sections 4.1 ad 4.2, the authors mentioned that \"GraphSAGE performed the best on ResNet embeddings\" and \"we also find that GAT models perform the best on this task\", I wonder where are these conclusions from.\n- It seems that the proposed method is not generalized enough. For different types of datasets introduced in this paper, selecting which GNN architecture is based on prior knowledge (the claims above). A question is how should one select the appropriate GNN architecture if a new dataset is used.\n- For the dataset construction, since the unconnected model can already achieve comparable or even better performance than GNN such as GAT in Flickr_v2 data, does it mean that the structure in this data is unnecessary?\n- In the experiments, there are some issues:\n1) Details of the data split are not provided.\n2) Although for different datasets, different GNNs are used to demonstrate the effectiveness of the proposed framework. It will be interesting to show the results of mixing other GNNs with certain unconnected models. Such comparison can better verify the effectiveness of the proposed combination strategy.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Tg887o2lrBX", "reviewer": "Reviewer_rePb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper explores different embedding extraction techniques in Graph Neural Networks (GNNs), and proposes Graph-connected Network (GraNet) layers, claiming that this approach improves the accuracy compared to traditional GNNs. ", "summary_of_the_review": "The main goal and contribution of the paper is confusing: is this a comprehensive analysis of embeddings for GNNs, the proposal of a new GNN model/approach, or the proposal of \"new datasets and a rich set of accompanying embeddings to better test the performance of GNNs\"?\n\nThere are statements that are not clear: for instance, in the abstract, the authors state that \"As an alternative, we propose Graph-connected Network (GraNet) layers to better leverage existing unconnected models within a GNN. Existing language and vision models are thus improved by allowing neighbourhood aggregation. This gives a chance for the model to use pre-trained weights\". This statement is far from trivial in my opinion and it would need further justification. Why the proposal made by the authors increases the chances of using pre-trained weights?\n\nI hesitate about the technical and methodological novelty of the GraNet approach. For instance, what is the main originality and contribution with respect to Brody et al. (2021)? What are the main intuitions behind GraNet? A better visual description of GraNet and its comparison with prior art will be very welcome. In fact, to be totally honest, I'm not sure I fully understand the merit, motivation and rationale behind their proposal. ", "strength_and_weaknesses": "- Weaknesses: \n-- it's not clear to me what is the main goal or contribution of the paper. For instance, the abstract starts talking about the exploration and analysis of different embedding techniques in GNNs. Later, it's claimed that a new type of layers is proposed, and even, as first contribution (included at the end of the Introduction) the proposal of \"new datasets and a rich set of accompanying embeddings to better test the performance of GNNs\" is mentioned.\n-- writing should be reviewed to make the reading less hard for the reader. \n\n- Strengths: quantitative results are very promising. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1Q04rsKY9P", "reviewer": "Reviewer_r5aW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work aims to examine the quality of different types of input embeddings for GNNs, and design a model architecture by combining both GNNs and non-GNN models (which are referred to as unconnected models in the paper). The authors provide empirical results to demonstrate the importance of having a high-quality input embedding, then propose the GraNet model to combine GNNs and non-GNN models, and finally provide 3 newly-processed datasets to justify the proposed model.", "summary_of_the_review": "My major concern with this work is, the contribution of this work is limited and the writing is hard to follow for me. The authors provide some empirical results to demonstrate that it is important for GNN layers to get a good input feature but does not really make it clear why and when will each type of input feature leads to good or bad performance. The proposed GraNet is not well-motivated and if I understand correctly, the model is just adding a feature mapping function on the input of the GNN, which is not a surprising idea. Therefore, I think this paper is not ready to get published in this venue at this time and needs further improvement.", "strength_and_weaknesses": "strengths:\n1. The problem of how to combine GNNs with the unconnected models to take advantage of both is worth thinking discussing, this topic is meaningful.\n2. The code is provided in the supplementary, which enhances the reproducibility of this work.\n\nweakness:\n1. Novelty of this work is not enough. \nThough the topic of getting the benefit from both GNN and non-GNN models is worth exploring, the idea of combining both is straightforward. And if I understand correctly, the design of GraNet is just adding a projection step to the network input (i.e. applying some function to the input) before conducting the neighborhood aggregation and transformation steps in GNN models, and the projection step is an existing NN model (MLP/CNN), which does not have enough novelty.\n\n\n2. Writing can be improved.\n\n1) The organization of this work is not easy to follow for me. \n- In the related work section, why it is important to include the paragraph \"Effect of training on GNN performance\"? This paragraph seems not very related to this paper.\n\n- In the background section, why do we have Table 1? These datasets are not used in the experiments, and if the authors only want to use this to explain most of the existing datasets rely on the BoW embedding, then I don't think it needs this big table to do so.\n\n- The background section only has one subsection (3.1), then why do we need this separate subsection? And the paragraphs are more than just notations, so I don't feel it is proper to put things under the subsection \"Notations\".\n\n- The method section is slightly confusing for me. First, the model itself is not clearly introduced, it needs some effort to figure out how many components are there in this model, and how is each component designed and how is the model trained. Second, it looks to me that the two subsections are the instantiation of the proposed model, I don't feel it needs these 2 long paragraphs for instantiation.\n\n- For the evaluation section, first, the titles are not informative, especially the \"GraNET\" one (which is actually not talking about GraNET, but is the main result of comparing GraNET against different baseline models). Second, it would be better if the author can provide richer results (like ablation studies) instead of using long paragraphs to give a long but not concise statement for the observations which we can clearly find in the table.\n\n2) The motivation for this work is not clearly delivered. \n- mostly the authors are just trying to make it clear what is this model doing, but do not explain why this design can be a good choice\n\n3) The experimental results are not explained with enough thought and analysis. \n- I think it would be important to explain why we get this kind of results instead of just pointing out what can we observe from experimental results\n\n4) There are a few typos:\n- page 4 first line, \"architecturwa\" -> \"architectures\"\n- page 5 last line, \"Equation (6)\" -> \"Equation (5)\".\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "01I7X7IxXXP", "reviewer": "Reviewer_FPbc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers the impact of (pre-trained) embeddings on the performance of different GNN architectures. The datasets under considered are either images and texts with some form of \"graph structures\", e.g., based on “Co-viewed”, “Co-bought” and “Similar Items” (for the Amazon datasets). The authors fund that only some GNN models yield  an improvement in accuracy compared to the accuracy of\nmodels trained from scratch or fine-tuned on the underlying data without utilizing the graph connections. As an alternative,the authors propose Graph-connected Network (GraNet) layers to better leverage existing unconnected models within a GNN, and show 7.7% and  1.7% improvements on Flickr v2, GraNet over  GAT2 and GraphSAGE.", "summary_of_the_review": "The paper aims to investigate the impact of (pre-trained) embeddings on the performance of different GNN architectures. However, both the problem considered and the method proposed are really specific to datasets where nodes are images or texts that are \"connected\" via some (weak) forms of connections. The findings are hard to interpret, and they do not shed light on the performance of general GNNs.", "strength_and_weaknesses": "Strength: \n\n + Exploring  the impact of (pre-trained) embeddings on the performance of different GNN architectures. \n\n Weaknesses: \n\n  - The paper is rather poorly written and claims more than what is actually accomplished. \n \n  - The proposed GraNet framework is difficult to understand and needs better explanation and  justification.\n\n  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject