FIMP: Foundation Model-Informed Message Passing for Graph Neural Networks

## Abstract

Abstract Foundation models have achieved remarkable success across many domains, relying on pretraining over vast amounts of data. Graph-structured data often lacks the same scale as unstructured data, making the development of graph foundation models challenging. In this work, we propose Foundation-Informed Message Passing (FIMP), a Graph Neural Network (GNN) message-passing framework that leverages pretrained non-textual foundation models in graph-based tasks. We show that the self-attention layers of foundation models can effectively be repurposed on graphs to perform cross-node attention-based message-passing. Our model is evaluated on a real-world image network dataset and two biological applications (single-cell RNA sequencing data and fMRI brain activity recordings) in both finetuned and zero-shot settings. FIMP outperforms strong baselines, demonstrating that it can effectively leverage state-of-the-art foundation models in graph tasks.

## 1 Introduction

Foundation models have emerged as a new paradigm in artificial intelligence, shifting from narrow, task-specific training to large-scale pretraining of more generalized models []. Through pretraining on vast amounts of data, foundation models serve as a base model which can be adapted to a variety of downstream tasks []. Pretraining is typically done in self-supervised fashion through autoregressive language modeling [] or masked language/image modeling [, ]. Standard foundation models have emerged in fields such as Natural Language Processing (NLP) with BERT [], GPT-3 [], and CLIP [], as well as in Computer Vision (CV) []. After pretraining, these models exhibit strong positive transfer to downstream applications, outperforming task-specific models []. More recently, fields such as single-cell RNA sequencing and neuroscience have also seen the emergence of large-scale foundation models such as scGPT [], Geneformer [], and BrainLM [].

In contrast to the success of foundation models in domains such as language and vision, replicating large-scale pretraining for foundation models on graphs has proved more challenging. Often, there is less graph-structured data publicly available relative to the amount of unstructured data, with cost or technological limits hindering our ability to scale graph-structured data to rival unstructured data. In single-cell RNA sequencing (scRNAseq) data, for instance, technological advancements have fueled an exponential rise in the amount of unstructured single-cell transcriptomes available for analysis [], however spatial sequencing methods which preserve spatial organization of the cells within the tissue are not as well-developed. Furthermore, tokenization differences separate traditional Graph Neural Networks (GNNs) from pretrained transformer-based models. Traditional GNNs represent nodes using a single node embedding vector, whereas transformers tokenize input samples into into sequences of feature tokens, which places token representation at the feature level rather than the node level. Notable examples include gene tokenization in single-cell foundation models such as scGPT [] and Geneformer [], as well as image patching used in Vision Transformers (ViTs) [, ]. Bridging the gap between traditional GNNs and pretrained foundation models, and by extension unstructured and structured data, remains an open challenge.

Figure: Figure 1: The proposed framework for FIMP. Pre-existing foundation models, pretrained on vast amounts of unstructured data, are repurposed into message creation modules by adapting their self-attention layers for cross-node attention between node feature sequences.
Refer to caption: extracted/5703935/figures/FIMP_fig1_v7.png

Existing works have begun leveraging pretrained foundation models in graph-based tasks, focusing on the application of LLMs on text-attributed graphs. One-for-All [] used LLMs as an encoding module for text-attributed graphs, and trained a GNN model to do node, edge, and graph-level classification. Talk Like a Graph [], NLGraph [], and GPT4Graph [] evaluated LLM reasoning capabilities on graph reasoning tasks, proposing benchmark datasets for simple graph tasks. These methods, however, focus only on improving performance on text-attributed graphs using LLMs, leaving non-textual foundation models and domains underexplored in graph-based settings.

To address these challenges, we propose Foundation-Informed Message Passing (FIMP), a message-passing framework for GNNs which repurposes the attention layers of pretrained non-textual foundation models for message-passing on graphs. We first unify node entity tokenization between foundation models and GNNs, and develop a cross-node attention-based message creation module which can be learned from scratch or initialized from pretrained foundation models. We evaluate our model on a real-world street-view image dataset [], several spatial transcriptomics datasets, and fMRI brain activity recordings, incorporating state-of-the-art (SOTA) foundation models for images [], scRNAseq [], and brain recordings [] as message creators. FIMP demonstrates improvements over strong baselines, demonstrating that non-textual foundation models can be effectively leveraged in graph-based tasks. Additionally, FIMP demonstrates zero-shot embedding capabilities on image networks by leveraging pretrained ViTs [], matching GNNs trained from scratch. This demonstrates the potential of non-textual foundation models for zero-shot applications on graphs.

Contributions. In summary, the key contributions of our work are:

- 1.
Introducing FIMP, a message passing framework which leverages pretrained non-textual foundation models for graph-based tasks.
- 2.
Evaluating FIMP across image, spatial transcriptomics, and brain activity recording datasets, incorporating SOTA foundation models in each domain as message creators.
- 3.
Demonstrating zero-shot embedding capabilities of FIMP with ViTs on image networks.

## 2 Preliminaries

### 2.1 Graph Neural Networks

Graph Neural Networks are a versatile class of neural network architectures which operate over graph-structured data. The core idea of GNNs is to learn node and/or edge attributes through iterative local aggregation steps, which is commonly implemented through Message-Passing Neural Networks (MPNNs) []. Below we define our notations for describing GNNs.

Let $G=(V,E)$ denote a graph with a set of nodes $V$ and edges $E$. Each node has an input feature vector $\vec{x}_{i}\in\mathbb{R}^{f}$, where $f$ is the number of input features per node. GNNs iteratively pass messages between neighboring nodes, and in the process use both node features and graph structure to learn node representations $\vec{h}_{i}\in\mathbb{R}^{d}$, where $d$ is the hidden dimension of node embeddings. After $K$ message-passing iterations, node representation $\vec{h}_{i}$ will contain information from its $K$-hop neighborhood within the graph. The general update rule for the $k$-th layer of a GNN can be represented as follows:

$$ $\vec{h}_{\mathcal{N}(i)}^{(k)}=\texttt{AGGREGATE}^{(k)}\big{(}\big{\{}\vec{h}_ {j}^{(k-1)},\forall j\in\mathcal{N}(i)\big{\}}\big{)}$ (1) $$

$$ $\vec{h}_{i}^{(k)}=\texttt{COMBINE}^{(k)}\big{(}\vec{h}_{i}^{(k-1)},\vec{h}_{ \mathcal{N}(i)}^{(k)}),$ (2) $$

where $\mathcal{N}(i)$ denotes the neighborhood of node $i$ and $h_{i}^{(k)}$ is the representation of node $i$ in layer $k$. The choice of AGGREGATE and COMBINE vary among different GNN architectures, with the constraint that AGGREGATE should be a permutation-invariant aggregator. A readout function is used to map learned node representations into predictions for feature, node, or graph-level tasks.

### 2.2 Attention-based GNNs

Graph Attention Networks (GATs) [] proposed to learn attention coefficients between neighboring nodes, replacing the AGGREGATE function in equation 1 with a linear combination of neighboring node vector embeddings weighted by normalized attention coefficients:

$$ $e_{ji}=a(\textbf{W}\vec{h}_{i}||\textbf{W}\vec{h}_{j})$ (3) $$

$$ $\alpha_{ji}=\text{softmax}_{j}(e_{ji})$ (4) $$

where $\alpha_{ji}$ represents the final normalized attention coefficient between nodes $i$ and $j$, $a$ is a learned attention mechanism shared across all node pairs, and W represents a shared weight matrix.

We note that FIMP is fundamentally different from GATs and other attention-based GNNs such as graph transformers (covered in Section C). In GATs and graph transformers, nodes are represented by a single embedding vector $\vec{h}_{i}$, and a scalar attention value is learned between neighboring nodes. In FIMP, nodes are tokenized into sequences of feature vectors $H_{i}$, and cross-node attention between the feature sequences of neighboring nodes is used to create messages which are passed along edges. Further details of FIMP’s novel message creation method are covered in Section 3.

### 2.3 Foundation models

Foundation models are generalized Deep Learning models which have been pretrained on large amounts of data, and which can be finetuned for a variety of downstream tasks. In this work, we focus on non-textual foundation models, which define a tokenization procedure for continuous-valued data and typically do pretraining using a masked reconstruction objective. In single-cell RNA sequencing, for instance, scGPT [] tokenizes an input cell as a sequence of gene tokens, and learns a gene embedding table analogous to word embeddings learned in LLMs. Pretraining is done through a masked gene expression prediction objective. In the image domain, ViT-based architectures [, ] encode images as a sequence of patches, and similarly for fMRI brain activity recordings, BrainLM [] tokenizes segments of brain activity signal per brain region into tokens.

## 3 Foundation-Informed Message Passing

We propose a novel message-passing framework, depicted in Figure 1, that uses pretrained non-textual foundation models to generate messages between neighboring nodes in a graph. This leverages the pretrained knowledge of the foundation model to inform message-passing, allowing for pretraining on unstructured data before training on less-abundant graph-structured data.

Figure: Figure 2: Graph structure present in real-world datasets. (A) In spatially resolved RNA transcriptomics, cells are connected to adjacent cells in the 2D tissue section. (B) In the Mapillary street-view image dataset [], images form a geographical proximity graph based on latitude and longitude coordinates. (C) For fMRI recordings, the brain is parcellated into 424 regions, which are connected using a K-nearest neighbors graph based on the 3D spatial coordinates of each brain region.
Refer to caption: extracted/5703935/figures/data_graph_creation_fig_v3.png

### 3.1 Node tokenization

To unify the node tokenization procedure for GNNs with that of pretrained transformers, which tokenize inputs into sequences of feature tokens, we define transformation $\tau$ as a function which takes as input node features $X_{i}\in\mathbb{R}^{f\times c}$, where $f$ is the number of features per node and $c$ is the dimensionality of each feature, and outputs a sequence of $f$ $d$-dimensional feature vectors which will represent node $i$. A general formulation of $\tau$ can be written as:

$$ $H_{i}=\tau(X_{i})=\texttt{CONCAT}(X_{i}\textbf{W},P)\in\mathbb{R}^{f\times d}$ (5) $$

where W is a $c\times d$ learned projection into a $d$-dimensional feature vector, and $P\in\mathbb{R}^{f\times d}$ are positional encodings for each feature.

We note that this general formulation of $\tau$ becomes dataset-specific following tokenization schemes defined by foundation models on different data modalities. For instance, on datasets with input node feature vectors $\vec{x}_{i}\in\mathbb{R}^{f}$, such as a gene expression vector for a cell containing $f$ genes, we can see $X_{i}$ as an expanded feature vector with $c=1$, and W as a projection of a scalar gene expression value into a $d$-dimensional vector embedding. The positional encoding $P$ would then represent a learned gene embedding dictionary $P\in\mathbb{R}^{f\times d}$, analogous to word embeddings in natural language. The concatenation operation in equation 5 would then combine the learned projection of each expression value with its corresponding gene positional encoding, as in scGPT [] and Geneformer [].

For experiments on image datasets, $\tau$ is formulated as a patch encoding procedure following standard ViTs [], where an input image is divided into $f$ patches, each with $c$ pixels, that are embedded via a learned patch projector W. Positional encoding $P$ is done through fixed 2D sinusoidal positional encoding which is concatenated with each patch embedding. For fMRI brain activity recordings, $\tau$ follows a spatiotemporal patching process as in the BrainLM foundation model [], where for each brain region, segments of $c=20$ signal timepoints are embedded via a learned projection W. Spatial positional encoding is done through a learned projection of XYZ coordinates of each brain region, and temporal positional encoding is done using sinusoidal positional encoding.

### 3.2 Message creation

Our objective is to formulate message creation between two nodes such that pretrained foundation models can be leveraged to create the messages while fitting into the rest of the message-passing framework. Our key observation is that transformer-based foundation models operate using self-attention over sequences of feature tokens (depicted in Figure 1), and contain learned attention weights per layer which are trained to highlight important interactions between feature tokens. Message creation between neighboring node feature sequences can be viewed as a problem of highlighting relevant information which source node $j$ must pass to destination node $i$, and thus the pretrained attention weights can be repurposed for message creation between two nodes.

We define a cross-node attention-based message creation module which takes as input node feature sequences $H_{i}$ and $H_{j}$, and outputs a message token sequence $H_{ji}$ which will be passed from node $j$ to node $i$:

$$ $Q=H_{i}\textbf{W}_{Q},\ K=H_{j}\textbf{W}_{K},\ V=H_{j}\textbf{W}_{V},\$ (6) $$

$$ $H_{ji}={\texttt{softmax}\left(\frac{QK^{\top}}{\sqrt{d}}\right)V}$ (7) $$

where WQ, WK, and WV are learned weight matrices which parameterize the attention mechanism. Note that the attention weights can be randomly initialized and learned from scratch, or initialized from pretrained attention weights. Messages $H_{ji}$ can then be aggregated and used to complete the regular message passing aggregation and update steps, with each node represented by a sequence of feature tokens rather than a single embedding vector. The full algorithm is detailed in Algorithm 1.

We note that the cross-attention-based message passing operation in FIMP is fundamentally different from other attention-based GNNs. FIMP is the first method that uses feature-based cross-node attention to construct messages for message passing on graphs. In contrast, attention-based GNNs, particularly GATs and Graph Transformers, do node-level attention and learn scalar attention coefficients between nodes. An overview of attention-based GNNs is provided in the Related Works (section C), along with a summary of key differences with FIMP.

Figure: Algorithm 1 FIMP

### 3.3 Leveraging non-textual foundation models

In its base formulation, cross-attention message passing can be done with a simple cross-attention mechanism which is learned from scratch during training. We denote this base version of our architecture as FIMP-base in our experiments. Pretrained foundation models, however, can be repurposed to do the message creation in order to leverage their pretraining over vast amounts of unstructured data. Given a pretrained foundation model $\mathcal{F}$ with learned attention weights per each transformer layer, we adapt the self-attention mechanism in each layer to do cross attention between node feature sequences from neighboring nodes. This adaptation is done in each layer by using the pretrained WQ, WK, and WV weights to project both the source and destination node feature sequences $H_{j}$ and $H_{i}$, and computing the scaled dot product attention outlined in equation 7. The final hidden representation output of the foundation model is then taken as the message $H_{ji}$.

Figure: Figure 3: Performance summary across different tasks for FIMP + foundation model versus the best traditional GNN baseline. FIMP improves over traditional GNNs across multiple datasets, highlighting the benefits of leveraging foundation models pretrained on unstructured data.
Refer to caption: extracted/5703935/figures/fig_unstructured_pretraining_performance_benefit_barplot_v3.png

## 4 Experiments

In this section, we demonstrate the effectiveness of our proposed framework on a diverse range of tasks in graph-structured settings: (i) gene expression reconstruction and cell type classification on spatial transcriptomics datasets, (ii) image classification on the Mapillary street-view image dataset, and (iii) brain activity reconstruction on fMRI brain recordings from the UK Biobank (UKB) dataset []. The graph structure inherent in each of these datasets is depicted in Figure 2. Our main goal is to show that FIMP allows for the effective integration of pretrained non-textual foundation models into a message-passing framework on graphs.

### 4.1 Datasets

Spatial transcriptomics. We benchmark FIMP on gene expression prediction and cell type classification using three publicly-available spatial transcriptomics datasets. The Slideseq-V2 spatial transcriptomics dataset [] is a mouse hippocampus dataset consisting of $41,786$ cells and $4,000$ genes, with $14$ different cell types. A second spatial dataset of human heart tissue was obtained from the 10X Genomics public spatial data repository, consisting of $4,247$ cells each with $36,601$ measured genes. A third spatial dataset, SeqFISH [], consists of $15,000$ cells and $342$ genes taken from mouse embryo tissue sections. For all spatial transcriptomics datasets, we follow standard preprocessing and normalization procedures for RNA sequencing data, including count normalization and log transformation []. Full dataset details are in Appendix section A.

Mapillary image dataset. The Mapillary planet-scale image dataset [] is a dataset of 750,000 street-view images collected from over 170 countries around the world. Images are 1000-2000 pixels in height and width, originating from a variety of cameras and conditions depicting natural landscapes and buildings. Each image has a recorded latitude and longitude coordinate, forming a geographical proximity graph where each node represents a full image, connected to nearby image nodes if they are within 10 miles of one another. We evaluate FIMP on a task where the aim is to classify the country of origin based on the visual features of each image node and its neighborhood. We train on 100,000 training images, and test on the predefined 10,000 test image set, with country labels determined for each image based on its latitude and longitude coordinates.

fMRI brain activity recordings. The UK Biobank dataset [] comprises of 76,296 task-based and resting-state functional MRI (fMRI) recordings from 41,986 patients aged 40 to 69 years old. All recordings went through standard preprocessing steps for fMRI recordings [, ], and was parcellated into 424 brain regions using the AAL-424 atlas []. We apply robust scaling per brain region by subtracting the median and dividing by the interquartile range computed across subjects. Our training set comprised of $60,000$ recordings, with the rest reserved for validation and test.

Table: Table 1: Gene expression prediction results on the mouse hippocampus and human heart spatial transcriptomics datasets. Performance is reported across 5 runs in terms of MSE and $R^{2}$. FIMP outperforms baseline methods on predicting gene expression on both datasets.

**Table 2: Cell type classification results on the mouse hippocampus and embryo spatial transcriptomics datasets. Performance is reported in terms of accuracy and F1-score. FIMP outperforms baseline models at predicting cell types.**
|  | Mouse Hippocampus | Mouse Embryo |  |  |
| --- | --- | --- | --- | --- |
| Method | Accuracy ($\uparrow$) | F1-score ($\uparrow$) | Accuracy ($\uparrow$) | F1-score ($\uparrow$) |
| GCN | 47.59 $\pm$ 3.788 | 0.445 $\pm$ 0.050 | 74.23 $\pm$ 1.250 | 0.720 $\pm$ 0.008 |
| GraphSAGE | 51.81 $\pm$ 3.229 | 0.495 $\pm$ 0.036 | 80.77 $\pm$ 3.071 | 0.793 $\pm$ 0.031 |
| GAT | 46.21 $\pm$ 3.110 | 0.442 $\pm$ 0.031 | 71.07 $\pm$ 1.452 | 0.690 $\pm$ 0.014 |
| GIN | 52.71 $\pm$ 0.421 | 0.507 $\pm$ 0.008 | 75.51 $\pm$ 1.398 | 0.743 $\pm$ 0.012 |
| GPS | 52.89 $\pm$ 1.176 | 0.510 $\pm$ 0.008 | 81.77 $\pm$ 3.175 | 0.813 $\pm$ 0.038 |
| FIMP-base | 49.04 $\pm$ 1.215 | 0.464 $\pm$ 0.019 | 81.35 $\pm$ 2.285 | 0.807 $\pm$ 0.026 |
| scGPT | 53.50 $\pm$ 0.424 | 0.518 $\pm$ 0.005 | 82.93 $\pm$ 0.419 | 0.820 $\pm$ 0.005 |
| FIMP-scGPT | 57.05 $\pm$ 1.393 | 0.554 $\pm$ 0.004 | 83.33 $\pm$ 2.250 | 0.821 $\pm$ 0.022 |

### 4.2 Experimental setup

All models were implemented in Pytorch Geometric [] and Pytorch [], and trained using the Adam optimizer []. Flash Attention [] is used to improve the computational footprint during message passing. Hyperparameter tuning was done through a grid search over standard values for learning rate, dropout, attention dropout, and weight decay. For all experiments, a 24GB NVIDIA GPU (RTX3090 or A5000) was used for training. Experimental setup details for specific datasets are provided in the Appendix B due to space limitations.

Foundation models. For experiments on single-cell datasets, the scGPT [] whole-human checkpoint is incorporated for message creation in FIMP-scGPT, consisting of a 12-layer transformer with 54 million parameters. scGPT is pretrained using a masked gene expression prediction objective on over 33 million cells from a diverse array of human tissues and organs. The pretrained gene embedding table is also utilized from the pretrained scGPT checkpoint, representing pretrained knowledge about gene identities in transcriptomics datasets. Additionally, we also utilize the gene embeddings obtained by GenePT [], which are GPT-3.5 embeddings of gene function descriptions based on biomedical literature, as another pretrained gene embedding experiment. For image classification, a standard ViT [] with 12 transformer layers and 86 million parameters is used as a message creator. The patch encoder from the ViT is also reused from the ViT embedding module. For experiments on fMRI brain recordings, the BrainLM [] model was used, which consists of a Masked Autoencoder transformer with an 8-layer encoder and 4-layer decoder, totaling 26 million parameters.

Baselines. For both supervised and self-supervised tasks, we compare FIMP against popular message-passing GNN architectures, including GCN [], GraphSAGE [], Graph Attention Networks (GATs) [], and Graph Isomorphism Networks (GINs) []. We also compare FIMP against more recent GNN architectures, namely GraphMAE [], a masked graph autoencoder model, and GPS Graph Transformer [], a SOTA graph transformer framework. For supervised classification tasks, we additionally compare to the pretrained foundation model in each domain, which does not take graph structure as input and instead treats each node as an individual sample.

**Table 3: Image classification results on the Mapillary street-view image dataset. FIMP significantly improves over baseline models in image classification, and creates zero-shot embeddings of the image network on par with trained GNN baseline models.**
| Setting | Method | Accuracy ($\uparrow$) | F1-score ($\uparrow$) |
| --- | --- | --- | --- |
| Finetuned | GCN | 23.9 $\pm$ 1.152 | 0.182 $\pm$ 0.0151 |
| GraphSAGE | 22.2 $\pm$ 1.703 | 0.164 $\pm$ 0.0129 |  |
| GAT | 22.9 $\pm$ 0.596 | 0.189 $\pm$ 0.0042 |  |
| GIN | 26.4 $\pm$ 1.240 | 0.254 $\pm$ 0.0143 |  |
| GraphMAE | 15.8 $\pm$ 0.828 | 0.083 $\pm$ 0.0056 |  |
| GPS | 27.4 $\pm$ 1.046 | 0.268 $\pm$ 0.0157 |  |
| FIMP-base (ours) | 38.6 $\pm$ 1.174 | 0.422 $\pm$ 0.0170 |  |
| ViT | 56.5 $\pm$ 3.187 | 0.597 $\pm$ 0.0065 |  |
| FIMP-ViT (ours) | 63.2 $\pm$ 0.764 | 0.684 $\pm$ 0.0076 |  |
| Zero-shot | Majority class | 17.0 $\pm$ 3.162 | – |
| GraphSAGE | 23.6 $\pm$ 4.037 | 0.129 $\pm$ 0.0309 |  |
| ViT | 34.0 $\pm$ 3.391 | 0.282 $\pm$ 0.0389 |  |
| FIMP-ViT (ours) | 40.6 $\pm$ 6.269 | 0.371 $\pm$ 0.0550 |  |

### 4.3 Results

Spatial transcriptomics. Table 1 contains results for gene expression prediction on the human heart and mouse hippocampus datasets. From these results, we observe that FIMP-base, trained from scratch with a randomly initialized cross-attention layer as a message creator, is able to outperform baseline GNNs at predicting masked gene expression values. We attribute this to improved gene tokenization, with the learned gene embedding table capturing information about different genes from the data. When we leverage pretrained gene embeddings learned on unstructured data, either from GenePT [] or scGPT [] (denoted as FIMP-GenePT and FIMP-scGPT, respectively), we observe further increases in gene expression prediction performance.
Interestingly, we note that using an out-of-domain foundation model such as ViT as the message creator does not improve performance, suggesting that performance improvements are not trivially caused by increased model capacity, and rather depend on the pretraining domain being sufficiently aligned with the graph features.

Table 2 contains results for cell type classification on the mouse hippocampus and embryo spatial transcriptomics datasets. We note that in this supervised classification task, FIMP-scGPT achieves the highest classification performance on both datasets.

**Table 4: Brain activity reconstruction results on the UK Biobank dataset. Performance is reported across 5 runs in terms of Mean Squared Error (MSE) and $R^{2}$. FIMP improves upon baselines by $25.8\%$, with a further improvement of $2.8\%$ by leveraging BrainLM [] for message creation.**
| Method | Masking Strategy | MSE ($\downarrow$) | $R^{2}$ ($\uparrow$) |
| --- | --- | --- | --- |
| GCN | Replace noise | 0.554 $\pm$ 0.00002 | 0.189 $\pm$ 0.00003 |
| Fill in mean | 0.513 $\pm$ 0.00019 | 0.248 $\pm$ 0.00028 |  |
| Linear interpolation | 0.535 $\pm$ 0.00137 | 0.217 $\pm$ 0.00200 |  |
| GraphSAGE | Replace noise | 0.534 $\pm$ 0.00107 | 0.218 $\pm$ 0.00157 |
| Fill in mean | 0.464 $\pm$ 0.00039 | 0.320 $\pm$ 0.00057 |  |
| Linear interpolation | 0.500 $\pm$ 0.00094 | 0.268 $\pm$ 0.00138 |  |
| GAT | Replace noise | 0.548 $\pm$ 0.00004 | 0.197 $\pm$ 0.00007 |
| Fill in mean | 0.505 $\pm$ 0.00005 | 0.260 $\pm$ 0.00007 |  |
| Linear interpolation | 0.527 $\pm$ 0.00052 | 0.229 $\pm$ 0.00076 |  |
| GIN | Replace noise | 0.564 $\pm$ 0.00131 | 0.174 $\pm$ 0.00192 |
| Fill in mean | 0.533 $\pm$ 0.00185 | 0.220 $\pm$ 0.00271 |  |
| Linear interpolation | 0.559 $\pm$ 0.00061 | 0.181 $\pm$ 0.00090 |  |
| GraphMAE | Replace noise | 0.582 $\pm$ 0.00070 | 0.147 $\pm$ 0.00103 |
| Fill in mean | 0.544 $\pm$ 0.00030 | 0.203 $\pm$ 0.00044 |  |
| Linear interpolation | 0.573 $\pm$ 0.00091 | 0.160 $\pm$ 0.00134 |  |
| GPS Graph Transformer | Replace noise | 0.577 $\pm$ 0.00279 | 0.154 $\pm$ 0.00408 |
| Fill in mean | 0.547 $\pm$ 0.01030 | 0.198 $\pm$ 0.01506 |  |
| Linear interpolation | 0.557 $\pm$ 0.01034 | 0.184 $\pm$ 0.01512 |  |
| FIMP-base | Tokenization + PE | 0.288 $\pm$ 0.00713 | 0.578 $\pm$ 0.01043 |
| FIMP-BrainLM | Tokenization + PE | 0.267 $\pm$ 0.00493 | 0.606 $\pm$ 0.00972 |

Image classification. Table 3 summarizes results for image classification on the Mapillary image dataset. We observe that FIMP-base outperforms baseline GNNs by over $10\%$ due to its improved tokenization of image patches, despite being learned from scratch. The best performance is obtained by FIMP-ViT, which utilizes a pretrained ViT [] for cross-node message creation.

Zero-shot node embedding. We furthermore explore a zero-shot setting for embedding image networks, to evaluate the capability of FIMP to leverage the pretrained ViT model without any graph-specific training. We embed subgraphs of the Mapillary dataset with FIMP, and compare it to embeddings generated by a randomly initialized GraphSAGE model [] and the ViT model itself with no graph structure, with 400 image embeddings obtained per model. We evaluate the quality of embeddings by training a linear classifier on $75\%$ of the embeddings and predicting labels for the remaining $25\%$. We observe that FIMP-ViT is able to generate zero-shot embeddings which get over $40\%$ classification accuracy, on par with finetuned baseline GNNs despite having no graph-specific training. This strongly indicates that FIMP is able to effectively leverage pretrained non-textual foundation models, and enables exciting zero-shot application scenarios which were previously not possible with non-textual foundation models operating on unstructured data.

fMRI recording reconstruction. Table 4 summarizes results for fMRI recording reconstruction on the UK Biobank [] dataset. FIMP-base improves upon baseline GNNs by $25\%$ in terms of reconstruction performance on masked brain signals, with a further performance improvement of around $3\%$ from leveraging the pretrained BrainLM [] model for cross-node message creation.

## 5 Conclusions, Limitations, and Future Research

In this work, we introduce Foundation-Informed Message Passing (FIMP), a message-passing framework which repurposes the self-attention layers of pretrained non-textual foundation models for message-passing on graphs. Our approach represents the first broad exploration of utilizing non-textual foundation models pretrained on vast amounts unstructured data in graph-based tasks. FIMP demonstrates improved performance over baselines across multiple tasks in image networks, spatial transcriptomics data, and fMRI brain activity recordings, confirming the performance benefits of leveraging non-textual foundation models in graph-based tasks. Furthermore, FIMP demonstrates zero-shot embedding capabilities on image networks that are on par with trained GNNs. This enables zero-shot applications on graphs, previously impossible with non-textual foundation models.

There are several avenues for improvement upon our method, which we leave for future work. Currently, our evaluation of FIMP is limited to image and biological data. Protein design and social networks are promising areas of future research. Additionally, extending FIMP to a more general pretraining scheme for graph foundation models, as well as supporting multimodal graphs, heterogeneous graphs, and edge features, would all expand the potential applications of FIMP. Finally, improving the computational footprint of FIMP through strategies such as feature selection and efficient attention mechanisms beyond Flash Attention is an important future direction.

## Appendix A Datasets (Extended)

Spatial transcriptomics. We use three publicly-available spatial transcriptomics datasets. The Slideseq-V2 spatial transcriptomics dataset [] is a mouse hippocampus dataset consisting of $41,786$ cells and $4,000$ genes, with $14$ different cell type classes. A second spatial dataset of human heart tissue was obtained from the 10X Genomics public spatial data repository, consisting of $4247$ cells each with $36601$ measured genes. A third spatial dataset, SeqFISH [], consists of $15,000$ cells and $342$ genes taken from mouse embryo tissue sections. For all spatial transcriptomics datasets, we follow standard preprocessing and normalization procedures for RNA sequencing data, including count normalization and log transformation []. For all datasets, we take the intersection of gene features which are present in the scGPT [] pretrained foundation model, and split nodes into training, validation, and test sets with a 70/10/20 split. For graph adjacency information, we utilize the neighbor connectivity matrix present in each spatial transcriptomics dataset, which is derived from the original tissue section coordinates.

Mapillary image dataset. The Mapillary planet-scale image dataset [] is a dataset of 750,000 street-view images collected from over 170 countries around the world. Images are 1000-2000 pixels in height and width, originating from a variety of cameras and conditions depicting natural landscapes and buildings. Each image has a recorded latitude and longitude coordinate, forming a geographical proximity graph where each node represents a full image, connected to nearby image nodes if they are within 10 miles of one another. We evaluate FIMP on a geoguesser task, where the aim is to classify the country of origin based on the visual features of each image node and its neighborhood of nearby images. We train on 100,000 training images, and test on the predefined 10,000 test image set, with country labels determined for each image based on its latitude and longitude coordinates.

fMRI brain activity recordings. The UK Biobank dataset [] comprises of 76,296 task-based and resting-state functional MRI (fMRI) recordings from 41,986 patients aged 40 to 69 years old. Recordings were acquired on
a Siemens 3T scanner at 0.735s temporal resolution. All recordings went through standard preprocessing steps, including motion correction, normalization, temporal filtering, and ICA denoising [, ]. We parcellated the brain into 424 brain regions using the AAL-424 atlas [], yielding 424-dimensional scan sequences sampled at ª1 Hz. Finally, robust scaling was applied by subtracting the median and dividing by the interquartile range computed across subjects for each brain region. Our training set comprised of $60,000$ of the fMRI recordings, with the rest reserved for validation and test sets.

## Appendix B Experimental Setup (Extended)

The following section gives additional details about experimental setup across different datasets.

### B.1 Image classification

For image classification experiments, random 512x512 crops were taken from each image during training, with a 512x512 center crop taken at test time. Per-channel normalization was done on each image using statistics calculated across training images in the Mapillary image dataset. For FIMP and FIMP-ViT experiments, images were divided into 32x32 patches following the standard ViT patch encoding procedure []. For baseline GNNs, pixel values for each image were flattened and encoded using a learned projection.

### B.2 Gene expression prediction

For gene expression prediction experiments on spatial transcriptomics datasets, we limit the number of cells in each dataset to 5% of the original dataset size, leaving $1000$ cells for the mouse hippocampus spatial dataset, and $200$ cells for the human heart spatial dataset. This creates a challenging limited data setting for predicting gene expression values on each spatial dataset. We sample $50$ nonzero expressed genes in each cell for all models and mask out 80% of the gene expression values, taking MSE loss against only masked out genes.

### B.3 fMRI recording reconstruction

In brain activity reconstruction experiments , we sample 320 consecutive timepoints from each fMRI recording, giving a recording of 424 brain regions with 320 timepoints of signal for each region. Each brain region is represented as 1 node in the graph, with node features being the 320 timepoints of signal. We segment the timepoints for each brain region into patches of 20 timepoints, and perform masked reconstruction of brain recording signals. For FIMP and variants of FIMP leveraging foundation models, masked patches are replaced with a mask token, and the signals are predicted back by the model. For baseline GNN models, node features comprise of the 320 timepoints of signal, and we explore three methods for replacing masked out patch values: i) replacing with random noise, ii) filling in with the mean value of the brain region, and iii) linearly interpolating between adjacent non-masked timepoint values. All models mask out $50\%$ of patches per each brain region, with mean squared error (MSE) taken against the original data.

### B.4 Foundation models

For experiments on single-cell datasets, the scGPT [] whole-human checkpoint is incorporated for message creation in FIMP-scGPT, consisting of a 12-layer transformer with 54 million parameters. scGPT is pretrained using a masked gene expression prediction objective on over 33 million cells from a diverse array of human tissues and organs. The pretrained gene embedding table is also utilized from the pretrained scGPT checkpoint, representing pretrained knowledge about gene identities in transcriptomics datasets. Additionally, we also utilize the gene embeddings obtained by GenePT [], which are GPT-3.5 embeddings of gene function descriptions based on biomedical literature, as another pretrained gene embedding experiment. For image classification, a standard ViT [] with 12 transformer layers and 86 million parameters is used as a message creator. The patch encoder from the ViT is also reused from the ViT embedding module. For experiments on fMRI brain recordings, the BrainLM [] model was used, which consists of a Masked Autoencoder transformer with an 8-layer encoder and 4-layer decoder, totaling 26 million parameters.

### B.5 Baselines

For both supervised and self-supervised tasks, we compare FIMP against popular message-passing GNN architectures, including GCN [], GraphSAGE [], Graph Attention Networks (GATs) [], and Graph Isomorphism Networks (GINs) []. We also compare FIMP against more recent GNN architectures, namely GraphMAE [], a masked graph autoencoder model, and GPS Graph Transformer [], a SOTA graph transformer framework. For supervised classification tasks, we additionally compare to the pretrained foundation model with no graph structure input.

## Appendix C Related works

### C.1 Attention-based GNNs and Graph Transformers

GATs [] first introduced the idea of attention-based GNN architectures, learning attention coefficients between neighboring nodes and performing message-passing with a weighted aggregation of neighboring node embeddings. Graph transformers sought to bring the performance and expressivity of the full transformer architecture into the graph domain by modeling graphs as a sequence of node embeddings that represented a fully-connected graph. Graph Transformer Networks (GTNs) [] proposed the first graph transformer architecture, which could learn new graph structures and multi-hop connections. Graph-BERT [] proposed pretraining on subgraphs and finetuning for node classification and graph clustering tasks. Graph Transformer [] proposed utilizing laplacian eigenvectors as positional encodings for node tokens. SAN [] improved upon it by introducing learnable spectral positional encodings, and Graphormer [] further proposed spatial and centrality encodings for nodes to capture structural relation and node importance in graphs. GPS Graph Transformer [] proposed a general framework for building expressive graph transformers composed of positional and structural encodings, graph features, and GNN and attention layers.

In contrast to these works, FIMP views a single node as sequence of feature tokens similar to transformer input sequences, rather than a single node embedding vector as in GATs and graph transformers. Cross-attention is computed between the node feature sequences of two neighboring nodes, resulting in a message which is passed between the two nodes. FIMP is the first method that uses feature-level attention between tokenized nodes to create a message for message-passing over graphs. The node tokenization procedure allows FIMP to leverage pretrained non-textual foundation models as the message creator, something which other attention-based GNNs and graph transformers are unable to do.

### C.2 LLMs on Text-Attributed Graphs

More recent works have explored using Large Language Models (LLMs) in conjunction with LLMs on text-attributed graphs. GPT4Graph [] evaluated LLM reasoning capabilities on graph reasoning tasks, establishing a benchmark of graph-related tasks for language models. Talk Like a Graph [] and NLGraph [] conducted similar studies exploring graph reasoning capabilities of LLMs, and released the GraphQA and NLGraph benchmark datasets, respectively. One-for-all [] used LLMs as an encoding module for text-attributed graphs, and trained a unified GNN model to do node, edge, and graph-level classification using node-of-interest (NOI) subgraphs and prompt nodes. In contrast to these works, we focus on non-textual foundation models and graphs, which have not been explored extensively in graph-based tasks. Our work can be seen as a parallel work to LLM-based works on graphs, aiming to effectively leverage foundation models pretrained on other data domains besides natural language.

## Appendix D Attention Visualizations

Figure: Figure 4: Visualizations of FIMP feature-level attention between different functional groups in the brain. (A) Averaged attention heatmaps between functional regions of the brain for different age populations, with the difference in attention by age group visualized on the right subplot. (B) Similar heatmaps visualized for post-traumatic stress disorder (PTSD) scores, highlighting differences in attention in patients with low vs high PTSD score.
Refer to caption: extracted/5703935/figures/ukb_ampnet_attention_figure_AB.png

### D.1 Functional Region Attention in fMRI Recordings

During message passing on the fMRI recording graphs, FIMP generates cross-attention matrices during message-creation between feature tokens of neighboring brain regions in the K-nearest neighbors graph. We group the 424 brain voxels into 7 functional regions, namely the visual, sensorimotor, ventral salience, dorsal salience, central executive, default mode, and subcortical regions of the brain. Taking 100 unseen test set recordings, we extract attention matrices between all connected nodes, average the attention matrices across timepoints per node, and split patient recordings according to conditions such as Age and post-traumatic stress disorder (PTSD) score. We then average attention values across patient recordings with the same condition, and aggregate the node attention into the 7 functional regions, allowing us to examine differences in functional region attention between patients with different conditions.

In Figure 4A, the attention between functional regions is shown between patients below 65 years of age (left) and those above 65 (middle). The difference in attention between the two groups, as visualized on the rightmost plot, indicates that older patients tend to have higher attention between the dorsal salience regions and visual cortex regions. This follows previous literature that shows changes in dorsal pathways as people age []. Furthermore, Figure 4B shows similar visualizations for patients with high and low PTSD scores, revealing higher attention between sensorimotor areas and central executive, and subcortical areas. This also follows previous literature on the somatosensory basis of PTSD, where arousal and higher-order capacities get affected []. These patterns in attention reveal potential differences in functional region attention picked up by FIMP among patients of varying conditions.

Figure: Figure 5: (A) Averaged attention between 15 genes across edges connecting neighboring astrocyte cells in the mouse hippocampus dataset. (B) UMAP of learned gene embeddings from FIMP, colored by average expression value of each gene across astrocyte cells.
Refer to caption: extracted/5703935/figures/slideseqv2_figure_panel.png

### D.2 Attention Case Study 2: Gene Interactions in Spatial Transcriptomics

In spatial transcriptomics datasets, each node corresponds to a cell which is represented by a set of expressed genes. Message-creation in FIMP provides cross-attention matrices representing interactions between genes of neighboring cells. Gene interactions receiving higher attention between nodes can highlight possible biological connections which can be avenues of potential further exploration in the data. For example, Figure 5A shows an averaged attention heatmap across all self-edges connecting astrocyte cells in a subgraph sampled from the mouse hippocampus dataset []. This astrocyte-astrocyte feature-level attention matrix identifies a key interaction between CD63, a member of the tetraspanin family of cell surface proteins, and CKAP2L, a mitotic spindle protein controlling cellular division. Previous work has identified that CD63 may be either pro- or anti-tumorigenic, depending on tissue context []. CD63 expression is also highly enriched in glioblastoma, a highly lethal malignancy of the astrocytes, and may play a role in progression of these cancers []. This hints that CD63 may play an important role in controlling cellular division through astrocyte-astrocyte cellular communication, which may represent an exciting new target for antitumoral agents.

Figure 5B shows a UMAP embedding of the gene embeddings learned by FIMP-base during masked gene expression prediction training. Each gene is colored by its average expression value across all astrocyte cells in the mouse hippocampus dataset. We see that the learned embeddings form distinct structures during training, and that highly-expressed genes for astrocytes are clustered together in one region in the bottom-right. We hypothesize that this ability to learn gene vectors in embedding space and contextualize them for different cell types allows FIMP to outperform other methods in gene expression prediction tasks.

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: please see the above section.
soundness: 3
strengths: I think it seems like the authors are trying hard to emphasize how FIMP is fundamentally different from existing attention-based GNNs like GATs and graph transformers. The statement with boldface might indicate that they have faced challenges in communicating the novelty of their work or distinguishing it from prior approaches in earlier submissions.
**However, in my opinion, this research is both timely and important because it aligns well with the current era of foundation models dominating various domains like NLP, computer vision, and even biomedicine.**
Existing foundation models excel with unstructured data (like text and images), but applying them directly to graph-structured data is challenging. 
FIMP addresses this gap by allowing foundation models pretrained on unstructured data to be repurposed for graph-based tasks through its innovative tokenization and message-passing approach.
The authors provide comprehensive evaluations across diverse datasets, demonstrating that FIMP outperforms existing GNN models like GCN, GraphSAGE, and GATs. Notably, FIMP with pretrained models (e.g., scGPT and BrainLM) achieves the best results in gene expression prediction and brain activity reconstruction.
The zero-shot embedding experiments on the Mapillary image dataset highlight FIMP's ability to leverage pretrained foundation models without additional training, which is a significant advantage in scenarios where labeled graph data is scarce.
summary: The authors introduce Foundation-Informed Message Passing (FIMP), a novel message-passing framework designed to leverage pretrained non-textual foundation models for graph neural networks (GNNs). 
Unlike existing GNNs, which utilize a single embedding for nodes, FIMP represents nodes as sequences of feature tokens. 
This enables cross-node attention-based message-passing using the self-attention layers from pretrained models. 
The proposed method is evaluated on tasks across image networks, spatial transcriptomics, and fMRI brain activity recordings, showing significant improvements over baseline GNNs. 
Additionally, FIMP demonstrates zero-shot capabilities on image classification tasks using pretrained ViT.
weaknesses: - While I acknowledge the novelty and potential impact of the research direction taken in this work, I must also express some concerns. The proposed module, despite its creative approach (direction), appears somewhat simplistic in its current form. It would benefit greatly from a deeper theoretical analysis to clarify why the architecture is effective and how it fundamentally differs from existing methods. This would not only strengthen the authors’ claims but also provide a clearer understanding of the model’s underlying principles.



- I noticed that while the paper benchmarks against various GNNs, it does not include comparisons with several main baselines (non-foundation models) that are well-established within each specific domain. Is there a particular reason why these comparisons were omitted? Including these baselines would provide a more comprehensive evaluation and help validate the effectiveness of FIMP beyond the GNN-specific context.


- I would like to address the reporting of experimental results. The performance metrics are averaged over five runs, but it is unclear if this number of repetitions is statistically sufficient to ensure robust conclusions, especially given the variability in results. For example, in the Mouse Embryo cell type classification task, the reported standard deviations appear quite significant. Increasing the number of runs or providing a justification for why five is adequate would enhance the reliability of the reported outcomes.


- (minor) Additionally, regarding the reference to Graph Transformer Networks (GTNs) on line 803, I believe there might be a mischaracterization of the original work by Yun et al. (2019). The GTNs discussed in that paper are not graph transformers in the sense typically understood in this context. Instead, they are better described as a graph-adapted version of Spatial Transformer Networks [1]. As stated by Yun et al.:
> Yun et al. (2019): "GTNs can be viewed as a graph analogue of Spatial Transformer Networks which explicitly learn spatial transformations of input images or features."

[1] M. Jaderberg, K. Simonyan, A. Zisserman, et al. Spatial transformer networks. In Advances in neural information processing systems, pages 2017–2025, 2015.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: See weaknesses.
soundness: 2
strengths: 1. The writing and organization are good. The presentation is easy to follow.2. 
2. The proposed methods can be applied to many fields. 
3. Experimental evaluations from different fields demonstrate its effectiveness.
summary: This paper incorporates pre-trained LM for cross-node message creation for a non-textual graph neural network. To this end, it tokenizes each node's content into a sequence and employs self-attention between sequences from pairs of nodes. Evaluations on genomics, brain, and street-view images demonstrate its effectiveness.
weaknesses: 1. The novelty is very limited. The proposed method is the extension of attention in GAT to non-textual node content. This aims at jointly considering source and target nodes for message passing. However, this is not novel.
2. The proposed method lacks clear motivation. It is direct for non-textual node content. Thus, the significance is weak. It is not clear why the cross-node attention can improve the performance. 
3. The evaluations are not convincing. Only the GNN baselines are compared. Besides, it is not clear how the node attribute is constructed for GNNs. How about using the embedding from LM as the node’s attribute?
4. Figure 3 lacks a description in the main body.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: The practical application of traditional GNNs is hampered by a significant computational burden. As shown in Figure 6, the time cost of FIMP is twice that of traditional GNNs. This concerns me about the practical value of FIMP. I would like to see a comparison of F1-Score and consumed time between FIMP + ViT and the ViT model alone on image classification dataset. I also hope to see the training time expenditure of FIMP on larger graph datasets.

In summary, if the authors can address my concerns, I would be open to increase my score.
soundness: 3
strengths: 1. The expression is very fluent. Readers can easily understand the content of the work.

1. The study of graph foundation models and large graph models is very meaningful.

1. The paper demostrates FIMP outperforms traditonal GNN models on image networks, single-cell RNA sequencing, and fMRI brain activity recordings. FIMP can effectively leverage SOTA foundation models in graph tasks.
summary: This paper introduces a message-passing framework that leverages pretrained non-textual foundation models for graph-based tasks. The proposed method, FIMP, aligns node tokenization in Graph Neural Networks with tokenization schemes used by foundation models, allowing for more granular feature-level interactions during message passing. The authors demonstrate FIMP's effectiveness across diverse domains, including image networks, single-cell RNA sequencing, and fMRI brain activity recordings, showing improvements over strong baselines and highlighting the potential of repurposing non-textual foundation models for graph tasks.
weaknesses: 1. The algorithm proposed in this paper appears to be incremental. Although the paper discusses the differences from works like GAT, in essence, it merely transforms node features from vectors($x_v \in \mathbb{R}^d$) to matrices($x_v \in \mathbb{R}^{f \times d}$).

2. The current experiments are limited to image networks, single-cell RNA sequencing, and fMRI brain activity. It is unclear whether this method is effective in other domains, especially traditional graph datasets. I hope to see experiments conducted on larger practical datasets like OGB.

3. As shown in Table 1, the gene expression prediction results on the mouse hippocampus and human heart spatial transcriptomics datasets, FIMP + scGPT performs much better than FIMP-base and FIMP + ViT. This indicates that to unleash the potential of the FIMP algorithm, a suitable and powerful foundation model is needed. However, in many scenarios, there may not be powerful foundation models available. Pretraining a suitable foundation model from scratch is also expensive and challenging. This limits the practical application of FIMP.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Lines 191-193: "By aligning the tokenization in FIMP with the tokenization scheme of pretrained foundation models, we reduce distribution shift in token representation when applying these models to graph-structured data." Does this mean that you are directly adopting the tokenization strategy of the pretrained foundation model? Clarification on how closely you follow the original tokenization would be helpful.
2. In Algorithm 1, which of the weights are learned from scratch, and which are (possibly) borrowed from the pretrained foundation model? A clearer distinction between learned and reused weights would improve understanding.
3. How does FIMP-base specifically leverage the knowledge embedded in a foundation model? From what I understand, given lines 251-253: “In its base formulation, cross-attention message passing can be done with a simple cross-attention mechanism which is learned from scratch during training. We denote this base version of our architecture as FIMP-base in our experiments.”, the knowledge transfer in this case occurs primarily through the tokenization process. 
4. On the FIMP versions that rely on a foundation model’s weights in the cross attention, It seems unclear how effective using the pretrained weights of a foundation model would be, given that cross-node attention differs from what the foundation model encountered during training (e.g., ViTs typically see only single images per training step). Could you elaborate on how the cross-node attention mechanism adapts to this discrepancy? Or did it just appear to work well in practice?

**References**:

[1] Graph Inductive Biases in Transformers without Message Passing. Ma et. al. 2023
soundness: 3
strengths: 1. The concept of leveraging pretrained foundation models for message passing in GNNs is innovative. This approach shows how pretrained knowledge can inform finer-grained interactions in graph structures.
2. The suggested method achieves improvements over the discussed baselines.
summary: The paper introduces FIMP, a framework that adapts pretrained non-textual foundation models for graph-based tasks. FIMP repurposes existing pretrained non-textual foundation models for message-passing on graphs. FIMP is evaluated across diverse domains, such as image classification, spatial transcriptomics, and fMRI brain activity recordings, utilizing state-of-the-art foundation models like ViTs, scGPT, and BrainLM. It demonstrates competitive performance in a zero-shot setting for embedding image networks, achieving notable results without task-specific retraining.
weaknesses: 1. The paper is not easy to follow, with several concepts and sections lacking clarity (see questions below).
2. While the idea of integrating pretrained foundation models with MPNNs is interesting, the proposed approach is overly simplistic (a straightforward approach -- if I understood correctly –- see questions below). Simplicity in methodology is not inherently problematic in my opinion; however, the combination of simplicity with a lack of a clear motivation for addressing the problem raises concerns — see Weakness 4 for further details.
3. The experimental results are not entirely convincing, as the baselines used for comparison (e.g., GCN, GAT) are somewhat outdated. It would strengthen the paper to include comparisons with more recent state-of-the-art graph learning (e.g.,  GRIT [1]) methods that could offer a more rigorous benchmark for evaluating FIMP's performance.
4. In my opinion, the paper lacks sufficient motivation. The tasks presented appear to be crafted primarily to emphasize FIMP’s strengths, rather than addressing meaningful, real-world applications. If I understand correctly, the authors construct a graph from publicly available datasets (e.g., the Mapillary image dataset). For example, line 173 states: “images form a geographical proximity graph.” However, this implies that the method is evaluated exclusively on datasets where the graphs were constructed by the authors, and these graphs are not publicly accessible.
To strengthen the paper, the authors can either (1) provide a compelling rationale for how such a setting could arise naturally in real-world scenarios, or (2) identify and utilize publicly available datasets that align with their setup.

Meta-review: This paper introduces a novel message-passing framework by incorporating existing non-textural pre-trained models for graph-structured tasks. To this end, it employs multi-channel cross-attention to create the message. The performance is verified on multiple fields. Although the proposed model is clear and universal to different fields, its novelty and motivation are of concern. Firstly, the pre-trained models are only used as the initial node feature without any improvements. Secondly, the cross-attention is not novel, although the multi-channel variant is introduced. Thirdly, the graph structure, which is the most important of graph learning, is not properly considered. Therefore, it is not good enough for published.

Decision: Reject