FoGE: Fock Space inspired encoding for graph prompting
Authors: Sotirios Panagiotis Chytas
Rudrasis Chakraborty
Vikas Singh
Venue: NeurIPS 2025 poster ()
Keywords: Graphs
Fock Space
Geometric Algebra
Vector Symbolic Architectures
Primary area: general_machine_learning

Abstract: Recent results show that modern Large Language Models (LLM) are indeed capable of understanding and answering questions about structured data such as graphs. This new paradigm can lead to solutions that require less supervision while, at the same time, providing a model that can generalize and answer questions beyond the training labels. Existing proposals often use some description of the graph to create an ``augmented'' prompt fed to the LLM. For a chosen class of graphs, if a well-tailored graph encoder is deployed to play together with a pre-trained LLM, the model can answer graph-related questions well. Existing solutions to graph-based prompts range from graph serialization to graph transformers. In this work, we show that the use of a parameter-free graph encoder based on Fock space representations, a concept  borrowed from mathematical physics, is remarkably versatile in this problem setting. The simple construction, inherited directly from the theory with a few small adjustments, can provide rich and informative graph encodings, for a wide range of different graphs. We investigate the use of this idea for prefix-tuned prompts leveraging the capabilities of a pre-trained, frozen LLM. The modifications lead to a model that can answer graph-related questions -- from simple graphs to proteins to hypergraphs -- effectively and  with minimal, if any, adjustments to the architecture. Our work significantly simplifies existing solutions and generalizes well to multiple different graph-based structures effortlessly.

FoGE: Fock Space inspired encoding for graph prompting
Sotirios Panagiotis Chytas'
Rudrasis Chakraborty
Vikas Singh'
Unniversity of Wisconsin-Madiset Lawrence Livermore National Lab
chytas@visc.edu, vsinghabiostat. wisc. rudrasischa@@mail.com
Abstract
Recent results show that modern Large Language Models (L..M) are capable understanding and answering questions about structured data such as graphs. Existing proposals often use some description of the graph to create "augmented" prompt fed to the LLM. For a chosen class of graphs, if well-tailored graph encoder is deployed to play together with pre-trained LLM, the model can answer graph-related questions well. Current solutions to graph- based prompts range from graph serialization to graph transformers. In this work, show that the use of a parameter- free graph encoder based on Fock space representations, a concept borrowed from physics, is remarkably versatile in this problem setting. The simple construction, with a few small adjustments, can provide rich and informative graph encodings, for wide range of different graphs. We investigate the use of this idea for prefix-tuned prompts leveraging the capabilities of pre-trained, frozen LLM. The modifications lead model that can answer graph-related questions from simple graphs to proteins to hypergraphs effectively and with minimal, if any, adjustments to the architecture. Our work significantly simplifies existing solutions and generalizes well to multiple different graph-based structures effortlessly.
Introduction
Large Language Models (LLMs) excel at tasks like question answering. sentence completion, translation, and even solving undergraduate--eved math problems [1] (). However, they some- times need additional data unavailable during training. For instance, model trained data up to a specific date may struggle with the ever-changing news cycle 3 4 To prevent responses from data and
Fock Sparen graph encoding
hours
becoming outdated, or to integrate non-public/proprietaa domain-specific terminology, models need extra context. Retrieval Augmented Generation (RAG) describes this process of retrieving and integrating extra information to an LLM during its generation process. While multiple different approaches have been proposed for the retrieval piece, a common solution for integrating additional information is In-Context Learning (ICL) [5 6 7 9. ICL allows additional information to be included with a prompt, guiding the model to generate responses aligned with the extra context. This method is useful it does not require retraining the L.M and can be applied to proprietary models like GPT 100 by adding a text description of the extra information.
long
Augmenting L.M''s ca- pabilities by prompting them with carefully encoded graphs.
Figure 1:
ICL-type ideas are also being studied for utilizing not just additional/new data but also novel input formats/modalities, such as tables and graphs [II 12 13 [4] While specialized models will still perform better at specific tasks, LLMs can serve general-purpor purepae reasoning machines, capable of answering questions about the provided modality beyond the training labels. Several recent results
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

have reported success at "serializing" such structured data-types into a text-form description that can be easily used within ICL. For tables, the serialization is not too complicated [TT] but more care is needed for graphs. While different types of graphs can all be handled by the same pipeline, the efficacy of the model varies [5 13 Further, it has been observed that specific design choices to "textify" the graph can influence performance and additionally, prompting techniques can have more than a small impact on results [15] What will work well in a specific setting depends on both the
question at hand as well as the characteristics of the data [16 17
Prefix-tuning.
A specialized graph encoder translates the underlying graph into embeddings that can be fed directly to an LLM, removing the need for a text description. Although not training-free free, the LLM remains frozen, and only the relatively smaller graph encoder is trained. This approach works well, often surpassing -Csed methods [19 20, 21.. However, using specialized graph encoder can be challenging due to the variety of graph types, and multiple works have proposed modifications of GNNs that suit their needs. For example, GraphToken [16] can encode only simple graphs, while GNP [2 constructs a complex pipeline to handle large graphs and extract subgraphs. GraphLLM [17 combines transformer and GNN (about 100M parameters), requiring detailed text descriptions for each node, Adapting these models to different graph types (e.g., protein-derived graphs or hypergraphs) is difficult; even familiar graph types need adjustments for new tasks.
One option to address the issues above is "prefix-tunii tiiti-ttninn"
18]
Context of this paper. ICL-based approaches for graphs primarily involve converting graphs to text, while teffi-tuning with graphs uses modules to extract richer, task-relevant structures, requiring larger sample sizes and more compute. A key question is whether we can achieve powerful, task- agnostic graph representations that are as easy to obtain as ICI.-based methods. Could lightweight adapter map these rich (but task-independent) representations into the LLM embedding space, making prefix-tuning effective for various tasks? Recent results hint that this may be viable (23.. For instance, single linear layer can transform arbitrary image encoder's s outputs to align with CLIP's text encoder embeddings [24]. If our graph encoding captures the graph's s information and structure well enough, similar adapter could work with a pre-trained LLM to offer good performance. This approach's 's success depends on the quality of the graph representations. We ensure this by invoking a mature concept from mathematical physics, called Fock Spaces [25]. whose practical instantiation yields almost lossless task agssstgg graph embeddings. Our findings show that a linear adapter with these representations yields competitive performance, handling complex graph questions and diverse structures like hypergraphs and proteins. The main contribution of this paper is the Fock-space inspired encoding of diverse graph-based structures, ranging from simple graphs those obtained from proteins. We provide code for grounding LLMs using our graph encodings as prompts and profile the performance of this pipeline relative to baselines, on diverse datasets.
2 Deriving Fock space based Graph Representations
We will first review notations/results which will together provide the conceptual pipeline. While graphs serve as representative examples here, the rationale for structured data such as tables is similar.
Setup/rationale. Consider graph G E) with vertex set V and an edge set E; | denotes set cardinal- ity, We define the incidence matrix (26), I to be of shape [V] x E where lig = 1 if edge j ends at vertex i, -1 if edge j starts vertex and 0 elsewhere. Let [V| n. It is common to represent graphs via graph spectra derived from the Laplacian's 's eigenvalues. This effective for studying global properties of graphs like connectivity/symmerrii [27 12 but less so for capturing localized relationships between individual entities (nodes, edges, faces) within the graph. It turns out that an interesting direction using Clif- ford Algebra, shown to be effective in geometric problems
HVV
w
Figure 2: Single, Bi- and Tri-vectors Clif- ford Algebra with wedge products.
in machine learning (29 30 29 31 32), provides us tools for representing various graph elements
(nodes, edges, faces) in a nice algebraic structure K3 at once. Why? Graphs can be embedded and manipulated a geometric space Kd), and in principle, their spectral properties can also be studied. We briefly summarize the concept to assess its benefits and challenges.

---

2.1 Clifford Algebra and Graph Representations
Clifford Algebra. We start with a vector space W over a field K (e.g,, or C). Vectors in W support operations like addition, scalar multiplication, and subtraction. We also equip W with an inner product () that measures relationships between vectors. On top of our vector space W. we construct the tensor algebra T(W). We can think of T(W) containing all possible ways to multiply vectors together using the tensor product: includes the original vectors from W, all pairs of vectors, all triplets. and so on. It also includes sums and scalar multiples of these products. For creating the Clifford Algebra, the main step is to modify this tensor algebra by imposing a specific constraint. We define ideal I e special subset that acts as a of the form w @ w (w, w)1. Here, w & w is a vector multiplied (via tensor product) by itself, and 1 is the multiplicative identity. This ideal has closure property: multiplying any element from I(W) with any element from T(W) yields another element in I(W. This constraint will impose the algebraic relations (particularly anticommutation) we need for representing graph structures.
by expressions
Definition 2.1. Let W be vector space over field K, equipped with quadratic form q W K. The Clifford algebra of (W, q), denoted C(W,, q). is the quotient algebra
Essentially, we take T() and "divide out" the ideal I(). This filters out redundant information. More specifically, when we quotient by this ideal, we are asking that all expressions in I(W) equal zero, which enforces the constraint w @ w - (w, w)] for all vectors w. From this, we can derive the anticommutation relations we will use for distinct vectors: wu + en 22, v) (for basis
elements, this gives e Ej + ej e 2(es, e,)). More details are available in [35 36).
One choice of Clifford Algebra representation. We have described Clifford algebra as an abstract object, but we need a way to work with it for graph encoding. A representation of a K-algebra is homomorphism that maps algehra elements linear operators. Such a representation allows us to work with Clifford algebra elements as concrete matrices rather than abstract objects. For our practical setup, we will use the formal blueprint this algebra gives, which connects its generators to creation and annihilation operators.
Practical considerations in Clifford Algebra operations. Following Clifford algebra axioms exactly allows us to build higher-order elements (like edges, hyperedges) while preserving graph structure. However, a full implementation faces practical issues: n-dimensional space requires a 2" dimeennnsoo Clifford algebra, and this is impractical. Therefore, we make design choices that balance rigor with computational feasibility.
2.2 From Graphs to Clifford Algebra to Fock Spaces
Laplacian and the Dirac operator. For graph G, consider the Laplacian A II" R where / is the incidence matrix (37] But A represents only one component of the Hodge Laplacian L. which acts the full exterior algebra of the graph. This captures relations between all grades: scalars (grade-O), nodes (represented as grade-1 vectors), edges (as grade-2 bivectors), and so on. Here, the Dirac operator D serves as a "square root" of the Hodge Laplacian [88], sat- isfying 2 L [77. Because the Dirac operator and the graph Laplacian are connected through this identity, we can use the algebraic machinery associated with Dirac operators, i.e. Clifford algebra, for our graph encoding problem.
Algiir (er (LCC) - Mom(CS, C*))
Dinc openner
M
(C/4))
Figure 3: From graph Fock space rep-
resentations.
Connection to Clifford algebra. The Dirac operator arises naturally from Clifford algebra 139 Recall that the Clifford algebra is generated by [V| abstract basis vectors, which denote as (ei...., }. }. These are the grade-l elements of the algebra. The full structure (dimension 2"), ), is then built up from these generators through repeated application of the Clifford product, yielding elements of all grades up to the top grade-n element. One feature of this algebra is the anticommutation relations that these generators satisfy: ege, + e,e; 2(e,, e for basis elements. This algebraic structure guarantees the 2 property holds (40 36).
Spinors and Fock space. We now need to identify the space on which this algebra acts. For the complex Clifford algebra, there exists a special irreducible representation on complex vector space
3

---

S of dimension 2/1V1/21, called the Spinor space [40] 36 41. This Spinor space is where the abstract Clifford algebra elements, the generators ej and their products become concrete operators. However, for graph encoding purposes, we need to handle grades. The Fock space F D k-n provides what we want: graded structure that holds objects of different types concurrently: scalars (grade-O), nodes (as grade- vectors), edges (as grade-2 bivectors), and so on. The key point is that for the Spinor space 8 and the Fock space F. there exists an isomorphism S = between them [40] (6). This identification is a link: the Clifford algebra's 's action is defined on S, and the isomorphism allows us to translate this action onto E, which is the graded "container" of our graph elements.
| ^ (CCIV//2))
Why is this useful? We can now ask: what do the generators e₁ of our Clifford algebra look like when they act on this graded container? When the abstract Clifford generators 62 act the Fock space, they decompose into two operations: a creation operator Ti and an annihilation operator Th- That is, Ek + тk + Tk [40]. The creation operator TË increases the grade of an element (e.g., acting on grade-O to create a grade-1 vector, or acting on a grade-1 vector to form grade-2 bivector), while the annihilation operator тı decreases the grade. Here, the Dirac operator itself is combining all these creation and annihilation operations: D E + Tx [36.. The main takeaway for our graph encoding is that the algebraic structure underlying our graph is one of creation and annihilation. Nodes (grade-1) 1) combine to form edges (grade-2), which in turn can combine to form higher-order structures. Unfortunately, this is intractable due to 211V|/21 dimensions.
2.3 Translating Theory to Practice: Instantiating Graph Representation
The Fock space formulation provides ideas for representing multi-partie systems, viewing particles as nodes in a graph. As noted above, implementing the full structure, in high dimensions, is infeasible. Vector Symbolic Architectures (VSA), as explored in recent works 442 43), offer practical approximation of Fock spaces with compute efficiency. In VSA, the binding operation (circular convolution) approximates the creation/annihilation operators, while the superposition operation (vector addition) resembles the direct sum in Fock spaces. Although the VSA <+ quantum mechanics connection is not new 44.. in this context, provides specific ideas for efficiency.
Representing nodes, sums, and products. In our implementation, we assign a high nigh-dimensional vector to each concept (node, edge, and on). These vectors are analogous to the basis elements above. While idcally, these vectors would be orthogonal, similar to the properties of basis elements in Fock space, we simply approximate this by sampling from a normal distribution A N( 1/d). This leads to nearly orthogonal vectors, with the maximum absolute cosine similarity between any two vectors typically below 0.1 [45].
To emulate operations in Fock space, we use dimnensionaliyy-preeerviii operations instead of tensor products, avoiding exponential growth in dimensionality 44], This ensures all embeddings maintain the same dimensionality. We define sum (+) as element-wise addition and product (x) as circular convolution, analogous to Fock space's 's creation/annihilattoon operators. Circular convolution is done via clement-wise multiplication in the Fourier domain followed by an inverse Fourier transform. As d increases, these operations asymptotically satisfy Fock space's algebraic properties, with complexity (d d). This framework also supports inverse vectors, where a & = 1. Other properties like commutation relations, superposition, and self-commutation are mostly satisfied. Note that experiments are not tied to this specific implementation (improved choices can be dropped in).
Dealing with infinitely many concepts. In some datasets, vertices include text descriptions, making random vector initialization unsuitable. To address this, we use text encoders like CLIP (4). BERT 46). and RoBERTa [47). which map text to vectors that preserve information and place similar sentences in close proximity. This approach allows us to: (1) generate infinitely many vectors, and (2) ensure similar vectors represent similar concepts. When dimensionality allows, we retain default sampling and explicitly note the use of text encoders in our experiments.
Other works using Vector Symbolic Architectures. Vector Symbolic Architecture (VSA), rooted in symbolic AI, leverages digh-dimensional representations alongside logical rules for combining symbols/vectors [88 44 Many studies mechanistically derive ways to construct symbols and im- plement merge operations. The use of Fock space for symbolic manipulation has been explored, with applications in trajectory analysis [44]. Additionally, VSAs have been employed for computa- tional efficiency in self-attention calculations as seen in HRRFormers (50) 42). while a preliminary investigation of the application of VSAs to graphs can be found in [51].

---

Fock Graph Encoder (FoGE)
Based the concepts from 2 use a parameter-free scheme (denoted FoGE) to obtain rich graph embed- dings. Our approach is general and can handle large spectrum of differ- ent graph types, and its extension to novel graph-types is straightforward. Diverse graph types such as hyper- graphs, attributed graphs, as well as proteins (Figure can all be modeled easily providing an alternative or good initialization for more intensive trainable models. This approach translates the concepts of Fock spaces into prac- tical/efficient method for graph representation, where graph features obtained by the encoding are analogous multi-partice states in a Fock space.
Figure 4: Graphs, Hypergraphs, Attributed graphs. Proteins. these types can be efficiently encoded using FoGE.
For graph = (V, E') we have set of vectors [p.],_IV] using to index the nodes. We also use extra vector for the graph's size, practical design choice will explain shortly. Then, with these n + vectors, we obtain a lossless Fock-space based representation g as:
pn)
(pi PP))
(I)
Our formulation follows from Each edge's endpoints fused together using & and then aggregate all edges together using c Finally, the graph's 's size is also added using the special vector s.
+
6
8
P: +
Lossless representation. The above representation is lossless. Assuming we use Equation IT to get a graph's embedding g. Then, simply by evaluating the expression P (p, @ g). we can determine whether the edge (i,j) exists in the edge set of that particular graph. In this way, we can recover, one by one, all edges of the graph and correctly reconstruct it, if desired. It is instructive check the importance of s. By evaluating the expression p. (s @ g), Vi, we can first obtain the size of the graph. This can inform the edge retrieval above because an expression of the form p (p, @ g) could, in practice, produce number close to although there no such edge. By first obtaining the size of the graph, we have "safeguard" against such phantom edges beyond the real vertex-set.
Vertex attributes. Consider graph G Attr), where the set Attr (with |Atrr| [V) consists of attributes, one for each vertex. There is no restriction on the type of attributes: it can denote numerical values or of vertex i E V (using an appropriate text-encoder if needed). Then, we can augment Equation (T to absorb the extra information in the following way:
any other concept. Let a; be the vector associated with the attribute
g = (s 2 p.. + (p.
+
θ @a) sEV
(2)
(UJ)
The graph again, fully reconstructable. We have also encoded each vertex's attribute (which can be recovered by the expression ař (p, & g)). We should think of proteins as a graph with vertex attributes where each vertex is a specific amino acid (possibly with 3-D coordinates).
Hypergraphs (Theory versus Practice). Hypergraphs are generalizations of graphs: each edge is connected to an arbitrary number of vertices, instead of just (Figure 4). In theory, we can easily augment Equation (I) so that can handle hypergraphs as follows:
g =
Pm))
+
P
(3)
(k, JE
In practice, aggregating many multiple vectors together may be unstable. This true for our particular design choices for calculations (c.g., circular convolution),
alternative approach. We
5

---

can start by observing that each edge can be interpreted as unique cluster of vertices, so we simply assign a unique vector e,, i [|E]] to each edge in the hypergraph. This modification allows to encode the hypergraph similar to how graph is encoded as a dictionary, in the following way:
g (s P.)
+
e
+ JER,
P
(4)
3.1 Fock Space-based grounding of LLMs (FoGE-LLM)
Recent works showed that (a) textu-
alizing a graph and pre-appending to question results in better-than- random responses from the LLM (al- though far from perfect), and (b) us- ing a specialized graph encoder such as GNN or graph transformer and training along with a frozen LLM re- sults in big improvement in perfor- mance, resulting essentially in LLMs that can understand, to some extent, graphical structures. One takeaway is that
A: Yes, this graph cycle.
LLM
Layor
enooding
Adapter
baring
Q: the graph has a eycle?
bypass the most tedious stage of designing application-specine graph encoders. Instead, we can use a parameter-free method for a wide range of graph types, as described above. Thus, the only trainable parts of the pipeline are simple linear adapters that convert the raw graph encodings to a format "understandable" by an LLM. Our FoGE-LLM is shown in Figure 5 After getting the graph encodings, we train one/more linear adapters and append the transformed encodings to the question's 's embeddings fed to the LLM.
Parameterffree graph encoding
Lightweight adapter
3 Applicable complicated graphs
Easy to train
Figure 5: LoGE-LLM overview. Using a parameter-fie free graph
encader we get graph embeddings for range of different graphs. Then, use lincar adapters with frozen LLM for prefix tuning.
Summary and Takeaway. We highlight some key advantages. First, our graph encoding is parameter- free and efficient. The complexity of aggregation is Ord d) (d is vectors' dimension) and the number of aggregation operations is linear (in graph size). Second, our encoder is not restricted to specific graph types: it works easily for simple graphs, for proteins and for hypergraphs just via small modifications. In contrast, GraphToken [16] uses a specific GNN whose output size is dependent on the underlying task whereas GraphLLM [17 uses a transformer model together with GNN (also specific the underlying task). These properties simplify our training and eliminates any tunable components. Third, our open-source code offers scalable way to train FoGE-LLM even on consumer GPUs, by using FSDP [52] For reference, GraphToken [16] is trained on TPUs (code unavailable) whereas GraphLLM 17 has large memory/compute footprint (trained on A100 80GB).
4 Experimental results
We examine our Fock-space based encoding in two separate settings: (a) as stand-alone input of simple model, and (b) as an extra prefix in a frozen LLM (FoGE-LLM), for graph prompting. Our initial experiments ($4.1) show that simple models can process FoGE embeddings quite well for traditional tasks. In 4.2 we present that FoGE can be successfully combined with an LLM, leading to two advantages over stand-alone models: (1) language interface for flexible graph queries without pre-defining task types, and (2) easier integration with mature software ecosystems built around LLMs that reduce deployment overhead.
Datasets and Models. We performed experiments on multiple graph reasoning datasets: from simple graph-understanding tasks to hypergraphs and proteins and aim to cover different aspects of graph understanding//easoning Specifically, we consider the 7 following datasets/dataset collections:
(i) GraphQA
[5] (ii) GraphReasoning [17] (iii) HyperGraphQA (iv) PPI [33 (v) OBNB
54 (vi) mol-HIV
(5 (vii) SabDab [56]. More details about the datasets can found in the
appendix. Exploring diverse graph reasoning datasets helps evaluate model's performance and

---

Table 1: Using small neural network with a single layer on the obtained graph representations allows us to perform near-perfectly on tasks such number nodes and number edges, for both synthetic and real data.
GraphQA num edges
HyperGraphQA
Jaffe
num nodes
has cycle
num nodes
num edges
num acids
num links
MSE/Acc 0.67 32K
0.03
9%%
1.12
0.63
2.95
11.9
Model size
8K
16K
32K
4K
32K
16K
generalization across various graph structures and domains, from traditional graph-based QA to hypergraph understanding and biological network analysis. By including datasets like PPI and BioGRID, we seek to check the practical relevance of our result, with potential applications in biology, network analysis, and more. We use the Llama2 (7B) model [57] as the frozen LLM, and we use only extra linear adapters for the graph embeddings obtained using our formulation. We adjust vector dimensionality from 512 to 2048 and use just single adapter for the entire model one adapter per layer in FoGE-LLM.
4.1 Proof of Principle Evaluations for Graph Understanding
Setup and Results. While our key goal is graph-prompting. we first perform multiple preliminary checks of the effec- tiveness of our graph encoding. We conduct three different types of experiments.
Table
2: Results on mol-HIV 55
58
The full details can be found
on https://ogb.staaftf stw.forsaooord.... edu/docs/
leader_graphprop
First, we evaluate whether graph embeddings are infor- mative (i.e., they preserve the graph's 's structure), by using small, 1 1-hidden-layer FFN for basic graph-understanding tasks. The results in T that our representations rich and informative. More specifically, quality of the embeddings by first encoding graphical struc- tures from 3 different classes of graphs (graphs, hypergraphs, and proteins) and then training a small model (one hidden layer) to predict graph attributes like number of nodes and edges. The results indicate that our representations are rich and informative and only few parameters suffice achieve Almost-perfect performance on such tasks.
ROC-AUC
HyperFusion
0.8475 0.0003
PAS Fingerprint
0.8420 + 0.0015
HIG
0.8403 + 0.0021
assess the
DeepAUC FoGE + Fingerppint
0.8352 0.0054
0.8305 0.0068
GMAN + Fingerprint
0.8244 0.0033
RF + Fingerprint
0.8208 0.0037
FoGE
0.7614 + 0.0051
GCN
0.7606 + 0.0097
GIN
0.7558 0.0140
Additionally, for more involved tasks, Table 3: Results on two real protein datasets from OBNB. Our we consider the Open Graph Bench- mark (OGB) (58) (more specifically, the mol-HIV dataset) and we show that our encoding is better than multiple, heavy, specialized, and trainable methods while being training-free and unsupervised! In Table we show the results. Inter- estingly, because our method allows seamless integration of additional graph information (like a molecule's footprint), we find that this can help us achieve even better results and, in some cases, be competitive with sub- missions at the top of the leaderboard. To obtain the final results, we used AutoGluon [6] on unsupervised embeddings, with a time limit of 10 minutes, similarly to the strategy of many of the other baselines.
method is the strongest unsupervised scheme to obtain node embed- dings, especially for DisGeNet. Its performance is comparable to trainable, graph-specifer models. More details on all baselines in [54]. The reported metric is the APOP (Average test Precision Over Prior).
BioGRID
HumanNet
Model
DisGeNet
GOBP
DisGeNet
GOBP
LabelProp
0.931
1.885
3.059
3.806
Adj + LR
0.743
2.528
3.053
3.964
Node2Vec + LR
0.836
2.571
2.433
4.036
LapEigMap + LR FoGE
0.864
2.149
2.301
3.778
1.062
2.433
3.254
3.916
GCN 59
1.012
2.572
3.116
3.812
GAT 60
1.063
2.562
3.065
3.963
Second, we examine whether our graph encodings preserve important biological markers of the data. To test this, we use small dataset of about 900 proteins (SabDab (56)) which are accompanied by affinity data that corresponds to each protein's 's clade. Briefly, clades are protein superfamilies, based on common ancestry (more information can be found in the appendix). In principle, proteins from the same clade are more similar than across clades,
examine whether this is also preserved
7

---

in our obtained embeddings. Although the dataset has only few samples and some of the clades are scarcely populated, we can observe that there a clear separation between the most populated clades in the embeddings space.
Third, examine if the same encoding practice can generate rich node-level encodings, by encoding for each node, the sub- graph that generated by itself and its neighbors. We examine performance on nineteen real protein datasets (OBNB [54]] and PPI (53).. The detailed results on the complete OBN Benchnark can be found on the appendix (4 of them are presented in T while demoontratss FoGE's performance on PPI. We see that our approach is, in all datasets, among the best unsupervised approaches, and is also competitive (if not better) than special- ized supervised approaches that leverage trainable, graph-specific models such as GCN [59] and GAT [60. Specifically, we achieve ofate-of-the-ar performance in PPI. We also achieve the best re- sults (among both unsupervised and supervised) in seven out of the eighteen datasets of OBNB.
Table 4: Micro Fl-score on PPL. Our approach is better than the best unsupervised approaches and het- ter/comparable to the supervised ap- proaches.
Model
F1
Random
39.2
Node2
40.9
Raw features 66
42.2
Unsupervised
GraphSAGE [53
46.5
GraphSAGE-mi max 53
50.2
DGI 63
63.8
GRACE 64
66.2
FoGE
99.2
These results provide encouraging evidence that (a) our approach gives "rich" graph embeddings for a range of different graph types and styles, and (b) our graph embeddings can be used as an extra, grounding input to powerful LLM without the need to
Supervised
GraphS min
5%
50.0
GraphSAGE-me
53
61.2
LGCN (65]
77.2
GAT 60
97.3
design/train specialized model, e.g., GNN 67 68 Transformer [69].
or a Graph
GCNII 66
99.5
4.2 Grounding LLMs with Graph prompting
We next investigate whether such an encoder can be successfully integrated with an LLM. Can an LLM understand graph structure? Our experimental evidence suggests that it can: the model learns to reason about graph properties and. in many cases, performs better than approaches relying on heavily specialized graph encoders. While this line of research is still in its early stages and requires further exploration, it closely parallels recent multimodal advances. Just as LLaVA [70] 7 72 acquires visual understanding and TimeLLM (73 develops temporal reasoning once their respective modalities are properly embedded, our findings indicate that LLMs can develop some graph reasoning capabilities when provided with rich structural representations such as those produced by FoGE.
Graph Understanding. In our first ex- periment, we examine whether LLM can answer questions about a graph's structure, such as the number nodes, the presence of cycles, and so on. We use GraphToken and conduct suite of six different experiments. Although our method's encodings are not specific to each underlying task, it performs com- petitively with specialized models, as shown in T Even when Graph'To- ken uses different embeddings for each node (node degree) or edge (edge exis- tence), our model still achieves compa- rable results using single embedding for the entire graph.
Table 5: GraphToken vs FoGE-LLM on GraphQA. Column / stands for a single embedding for the entire graph; O(n) stands for a single embedding per node. In all 6 tasks, although we use parameter-free, predetermined graph encoding, performance similar/better relative to a trainable graph encoder with a larger LLM.
ICL.
GraphToken
FoGE-LLM
Tokens
((n²)
1
O(n)
num of nodes
26.9%
99.6%
97.2%
num of edges
12.8%
42.6%
45.1%
cycle existence
83.2%
95.6%
97.9%
num triangles
16.2%
34.8%
37.7%
edge existence 54.4%
73.8%
74.3%
Advanced Graph Reasoning, Going beyond "simple" graph understanding tasks, we also examine our performance on more complicated weah-reesoning tasks, using the GraphReasoning dataset [17]. Graph not applicable here since each node is accompanied by a textual description which cannot be handled by that model. So, our main baseline is GraphLLM, which uses transformer combined with GNN to merge the graphical/textual information into one or more embedding vectors. Similar to GraphToken [16), GraphLLMM [17] also utilizes a different approach for each task, using multiple graph embeddings for each task. contrast, we achieve comparable performance using single graph embedding, showcasing the versatilityrrichness of the graph embeddings (Table 6).
8

---

Further, we see that using pretrained text encoder such as RoBERTa 47 reasonable, and results similar performance. This strong improvement over traditional symbolic methods, by allowing large set of "syetooss"veettors. Dealing with proteins is similar to advanced graph reasoning, since both datasets are graphs with additional node information.
to generate the vectors
Furthermore, in Table 7 we show the ac- euracy of FoGE-LLM for three protein- related tasks on Jaffe. Although the size of the protein graphs is more than 10x larger compared to the ones in GraphQA
Table 6: GraphLLM vs FoGE-LLM. Although we are using the same, predetermined graph embedding for each task, we enjoy performance similar to GraphLLLM which leverages 5 graph embeddings, specific to the task at hand. The vectors stands for the two approaches (a) randomly generated (almost) orthogonal vectors (ignoring the node's ' text description), and (b) using RoBERTa 47 and
follow in generating them:
and GraphReasoning, our model is able, up to some extent, to understand the pro- vided protein, as whole (number of amino acids and number of links) as well as at an individual-node level for the task type of amino acid (where we prompt the model to determine the type of a specific vertex in the protein).
utilizing all vertices' information.
GraphLLM
FoGE-LLM
model size question specific output graph embeddings vectors
100M
25M No
Yes
random
RoBERTa
Hypergraphs. Existing works focus on specific forms of graphs and rarely ap- differ-
substructure count max triplet sum shortest path hipartite match
99.9%
97.3%
95.6%
95.7%
94.6%
94.7%
plicable (or easily modifiable) ent graph types. One common family of graphs applications is hypergraphs.
97.2%
95.7%
95.8%
99.8%
98.1%
97.3%
Here each edge is subset of the nodes, of arbitrary size (Figure Our formulation can handle such a generalization of the typical graphs with only minor modifications to the encoding formulation (Equation (4). Here, we show that our setup can indeed answer questions about such complicated structures, using our encodings as an extra prefix (graph prompting). Using the HyperGraphQA dataset, we assess the performance of FoGE-LLM on four common tasks. Since Token as well as GraphLLM cannot handle such data, we compare our model's s performance against two of the most common prgine-nngineering methods: 1. zero-shot, where the model is given the graph in text form along with the corresponding question, and 2. few-shot, where the model is given pairs of textualized graphs with the corresponding question/answer pair and it is asked produce the answer to a new combination of graph/question. The results are presented in Table 7 Interestingly, though hypergraphs have much more complicated structure than "simple" graphs, model achieves a performance very close to basic graph understanding (Table 5), or even better at some tasks.
4.2.1 FoGE-LLM runtime
Besides the raw performance gains as presented above, FoGE-LLM offers hypergraphs and proteins. very low inference time, due to two rea- sons. First, always "reserve" only a single token for the given graph. In contrast, zero/few-shot approaches that textualize the graph require a large num- ber of tokens, and grows larger the graph grows. This leads to an increase inference time, due the number of input tokens. Second, FoGE-LLM uses one or more linear adapters, no special-
Table 7: FoGE-LLM performance against ICL techniques for
Zero-Shot Few-Shot
FoGE-LLM
HyperQA
num of nodes num of edges node degree edge existence
04.5%
16.8%
85.0% 0%
03.9%
27.0%
95. 15
02.1%
10.1%
53.9%
65.9%
79.1%
87.9%
num of amino-acids
03.9%
17.1%
99.3%
Jaffe
num of links amino-acid type
03.8%
06.1%
13.2%
01.4%
12.3%
37.7%
ized architectures like in 11 [6] are needed. Based on our experiments, this impacts inference time,
and gives FoGE-LLIM strong efficiency benefits. Table S we show the average inference time
required for each approach.
5 Related Work
Geometric Algebra in Machine Learning. There is growing interest in application of geometric algebra in machine learning, particularly for developing models that maintain geometric properties.

---

While these ideas have been leveraged in the context of equivariance/ymmeety transformations 78 the concept is finding interesting uses in recent works. For example, [79] recently proposed Clifford Neural I Layers to model dynamical systems in fields like fluid dynamics and [29] described Geometric Clifford Algebra Networks (GCANs), specifically designed to respect symmetry group transformations. Beyond classical machine learning. geometric algebra finds more direct applications in quantum computing as well: [80] leveraged the isomorphism between Pauli matrices and Clifford Algebra to represent multidimensional data, to define specialized transforms for machine learning tasks.
in deep learning 74 75 76
Graphs & LLMs. The body of work describing ways to infuse extra, graphical information into a frozen LLM is sizable and growing. As discussed earlier, initial ap- proaches focused on converting the underlying graph into natural language form, such as "node is connected to node 3, node 5 is connected node 4, ... [3 14 15] These works while far from perfect showed viability: that a frozen LLM has the capability to reason about the given graph and answer graph-related questions, such as "is there a cycle in the graph?". Practical difficulties involving the format of graph serialization is an important factor in the performance and the results tend to be only moderately better than random. The perspective taken in [6 [7] was
Table 8: Average inference time for each approach. FoGE-LLM is significantly lower than zero/few shot approaches since the num- ber of input tokens does not grow with the graph size, while it enjoys a 40% improve- ment over GraphLLM dues to simpler en- coder/adapter.
Model
Inference time (s) 4
zero-shot
0.175 (+0.05)
few-shot
0.541 (+0.10)
GraphLLM [17]
0.052 (+0.01)
FoGE-LLM
0.031 (10.01)
fresh and led to an alternative approach: infusing the graph
information directly at the embedding level, by encoding the graph using model such as Graph Neural Network (GNN) [67, 68, [6] or a Graph Transformer [69] 17). These works significantly improved the state of the art, showing that carefully crafted graph embeddings are key to a successful grounding of an LLM.
6 Conclusions
We have described novel scheme to encode graph into a vector form for direct downstream use or to augment prompts fed to LLMs. Our approach, motivated by Fock space operations, offers numerous advantages in practice demonstrated via experiments. We can obtain encodings of arbitrary graphs quickly, with no trainable parameters, that nicely captures the important information content in the underlying graph. To utilize these encodings, we introduced FoGE-LLM - a way to fuse the graph information for graph-prompting with pre-trained, frozen LLM, allowing it to "understand" and reason about graphs. Given the growing interest in grounding L.LM responses based on additional domain-specii specifc priors, we believe that this is interesting direction. Our model, accompanied with a simple-to-trinn open-source codebase, performs favorably relative to highly specialized models. It is also quite flexible and can handle classes of graphs where other alternatives fall short or need adjustments.
Impact & Limitations. A key strength of our method is its parameter-fiee approach for generating rich graph embeddings. Such an approach can be a great fit in less computationally rich environments or in cases where the dataset's s size is not big enough for the trainable approaches, without, as we demonstrated extensively, lacking in performance data-rich situations. Given the scarcity of the data in many real-life graph-related problems (like the protein-based questions we answered here), our approach can benefit multiple aspects of research. However, the unsupervised nature of FoGE also limits the ability to fine-tune performance if the embeddings are insufficient for specific applications. So, we believe that building representation learners on top of these embeddings, as in FoGE-LLM, is good strategy. Additionally, when dealing with infinitely large vector sets, random generation is impractical. While RoBERTa works well in our experiments, integration with other models may involve some tria--and-error identify sensible configurations.
Acknowledgments We would like to thank Tom Reps and Anthony Gitter for discussions and feedback. Additional experiments on the applicability of FoGE in other types of datasets were performed during a summer internship of S.P. Chytas at L.NNI, and will be disseminated in a separate paper after approvals.
10

---

References
[1] Hongwei Liu, Zilong Zheng, Yuxuan Qiao, Haodong Duan, Zhiwei Fei, Fengzhe Zhou, Wenwei Zhang, Songyang Zhang, Dahua Lin, and Kai Chen. Mathbench: Evaluating the theory and application proficiency of Ilms with a hierarchical mathematics benchmark. In Findings of the Association for Computational Linguistics ACL 2024, 2024.
[2] Moa Johansson. What can large language models do for theorem proving and formal methods? In Bernhard Steffen, editor, Bridging the Gap Between Al and Reality, Cham, 2024. Springer Nature Switzerland. ISBN --78----31-40002--
[3] Tu Vu, Mohit lyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, et al. Freshllms: Refreshing large language models with search engine augmentation. arXiv preprint arXiv:2310.03214, 2023.
[4] Seyed Mahed Mousavi, Simone Alghisi, and Giuseppe Riccardi. Is your Ilm outdated? benchmarking Ilms alignment algorithms for time-sensitive knowledge. arXiv preprint arXiv:2404.08700, 2024.
[5] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research. PMLR, 2020.
[6] Yujuan Ding, Wenqi Fan, Liangho Ning. Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A survey rag meets Ilms: Towards retrieval-augmented large language models. arXiv preprint arXiv:2405.062. 022400051111, 2024.
[7] Qingxiu Dong, Lei Li,, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning. arXiv preprint arXiv:2301.00 000344, 2022.
[8] Barret Zoph, Colin Raffel, Dale Schuurmans, Dani Yogatama, Denny Zhou, Don Metzler, Ed H. Chi, Jason Wei, Jeff Dean, Liam B. Fedus, Maarten Paul Bosma, Oriol Vinyals, Percy Liang, Sebastian Borgeaud, Tatsunori B. Hashimoto, and Yi Tay. Emergent abilities of large language models. TMLR, 2022.
[9] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics. doi: 10. 00:53//1/2022.emlp--moo 759.
[10] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advunces in neural information processing systems, 33, 2020.
[11) Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. Table meets Ilm: Can large language models understand structured table data? a benchmark and empirical study. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, WSDM '24 New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400703713. doi: 10. 00.114//36665553366 36357752.
[12] Weizheng Lu, Jiaming Zhang. Jing Zhang, and Yueguo Chen. Large language model for table processing: A survey, arXiv preprint arXiv:2442..05121 2024.
[13] Heng Wang, Shangbin Feng. Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov. Can language models solve graph problems in natural language? In A. Oh, T. Naumann,
A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36. Curran Associates, Inc., 2023.
[14] Jiayan Guo, Lun Du, and Hengyu Liu. Gpt4graph: Can large language models understand graph structured data? an empirical evaluation and benchmarking. arXiv preprint arXiv:2305. 15066, 2023.
11

---

[15] Bahare Fatemi, Jonathan Halcrow, and Bryan Perozzi. Talk like graph: Encoding graphs for large language models. The Twelfth International Conference on Learning Representations. 2024.
[16] Bryan Perozzi, Bahare Fatemi, Dustin Zelle, Anton Tsitsulin, Mchran Kazemi, Rami Al-Rfou, and Jonathan Halcrow. Let your graph do the talking: Encoding structured data for llms. arXiv preprint arXiv:2402.0586. 2024.
[17] Ziwei Chai, Tianjic Zhang, Liang Wu, Kaiqiao Han, Xiaohai Hu, Xuanwen Huang, and Yang Yang. Graphllm: Boosting graph reasoning ability of large language model. arXiv preprint arXiv:2310.05845, 2023.
[18] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, 2021. Association for Computational Linguistics. doi: 10.18653/v1// 12201 lall-long..333.
[19] Mingehen Sun, Kaixiong Zhou, Xin He, Ying Wang, and Xin Wang. Gppt: Graph pre-training and prompt tuning generalize graph neural networks. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '22, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450393850. doi: 10.1145/ 3534678.3539243
[20] Zemin Liu, Xingtong Yu, Yuan Fang, and Xinming Zhang- Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. In Proceedings of the ACM Web Conference 2023, WWW '23, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450394161. doi: 10. 14//344335077.35883
[21] Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin, and Chao Huang. Graphgpt: Graph instruction tuning for large language models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '24, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704314, doi: 10. 22.114//6666722.6577777
[22] Yijun Tian, Huan Song, Zichen Wang, Haozhu Wang, Ziqing Hu, Fang Wang. Nitesh V. Chawla. and Panpan Xu, Graph neural prompting with large language models. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17), 2024. doi:
[23] Mazda Moayeri, Keivan Rezaei, Maziar Sanjabi, and Soheil Feizi. Text-to-concept (and back) via cross-model model alignment. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research. PMLR, 2023.
[24] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research. PMLR, 2021.
[25] John Gough and Joachim Kupsch. fields and processes: a combinatorial approach. Cambridge University Press, 2018.
[26] Allen Hatcher. Algebraic Topology. Cambridge University Press, 2002.
[27] Miroslav Fiedler. Algebraic connectivity of graphs. Czechoslovak Mathematical Journal, 23,
1973.
[28] Miroslav Fiedler. Laplacian of graphs and algebraic connectivity. Banach Center Publications,
25, 1989.
12

---

[29] David Ruhe, Jayesh K. Gupta, Steven De Keninck, Max Welling, and Johannes Brandstetter. Geometric clifford algebra networks. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JorR.org, 2023.
[30] Siqi Chen, therre-Philippe Dechant, Yang-Hur Mui He, Elli Heyes, Edward Hirst, and Dmitrii Riabchenko. Machine learning clifford invariants of ade coxeter elements. Advances in Applied Clifford Algebras, 34(3), 2024.
[31] David Ruhe, Johannes Brandstetter, and Patrick Forré. Clifford group equivariant neural networks. In Thirty-seventh Conference on Neural Information P Processing Systems, 2023.
[32] Johann Brehmer, Pim de Haan, Sönke Behrends, and Taco S Cohen. Geometric algebra transformer. In Advances in Neural Information Processing Systems, 2023.
[33] Zbigniew Oziewicz. The dirac operator as graph and the clifford hopf-gebra. PITMAN RESEARCH NOTES IN MATHEMATICS SERIES, 1998.
[34] William E Baylis. Clifford (Geometric) Algebras: with applications to physics, mathematics, and engineering. Springer Science & Business Media, 2012.
[35] Leo Dorst, Daniel Fontijne, and Stephen Mann. Geometric algebra for computer science (revised edition). The Morgan Kaufmann Series in Computer Graphics. Morgan Kaufmann, 2009.
[36] Pertti Lounesto. Clifford Algebras and Spinors. London Mathematical Society Lecture Note Series. Cambridge University Press, edition, 2001.
[37] Beata Casiday, Ivan Contreras, Thomas Meyer, Sabrina Mi, and Ethan Spingarn, Laplace and dirac operators on graphs. Linear and Multilinear Algebra, 72(2), 2024.
[38] Lek-Heng Lim. Hodge laplacians on graphs. Siam Review, 62(3), 2020.
[39] Akhil Mathew. The dirac operator. URL.attrps:/mmthhuuuccccco wawpaagg...duaaaaaaaaaaaaaaa adu/"amathew/tiran
pdf]
[40] H Blaine Lawson and Marie-Louiss Michelsohn. Spin geometry, volume 38. Princeton university press, 2016.
[41] H. BLAINE LAWSON and MARIE-LOUISE MICHELSOHN, Spin Geometry (PMS-38). Princeton University Press, 1989. ISBN
[42] Mohammad Mahmudul Alam, Edward Raff, Stella Biderman, Tim Oates, and James Holt. Recasting self-attention with holographic reduced representations. Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, edi- tors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research. PMLR, 2023.
[43] Ashwinkumar Ganesan, Hang Gao, Sunil Gandhi, Edward Raff, Tim Oates, James Holt, and Mark McLean. Learning with holographic reduced representations. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34. Curran Associates, Inc., 2021.
[44] Matthias Wolff, Günther Wirsching, Markus Huber, Peter beim Graben, Ronald Römer, and Ingo Schmitt. fock space toolbox and some applications in computational cognition. In Alexey Karpov, Oliver Jokisch, and Rodmonga Potapova, editors. Speech and Computer, Cham, 2018. Springer International Publishing. ISBN 978-3-319-99579-3.
[45] Avrim Blum, John Hopcroft, and Ravindran Kannan. Foundations of Data Science. Cambridge University Press, 2020.
[46] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v 1/2999
13

---

[47] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv: /692, 2019.
[48] Kenny Schlegel, Peer Neubert, and Peter Protzel. A comparison of vector symbolic architectures. Artificial Intelligence Review, 55, 2022. doi: 10010077//0442--22--00111
[49] Sotirios Panagiotis Chytas, Miso Choi, Hyunwoo.. J. Kim, and Vikas Singh. Reco: Reminder com- position mitigates hallucinations in vision-language models. arXiv preprint wrXiv:2506.22636, 2025.
[50] T.A. Plate. Holographic reduced representations. IEEE Transactions on Neural Networks, 6(3), 1995. doi: 10.1109/72.377668.
[51] Sotirios Panagiotis Chytas, Rudrasis Chakraborty, and Vikas Singh. Geometric algebra based encoding for graph prompting. In ICML 2024 Workshop on Efficient and Accessible Foundation Models for Biological Discovery, 2024.
[52] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch fsdp: Experiences on scaling fully sharded data parallel. Proc. VLDB Endow. 16(12), 2023. ISSN 2150-8097. doi: 10. 10.74778/361 1540.361 1569.
[53] William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Proceedings of the 31 International Conference Neural Information Processing Systems, NIPS' 17, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781519861999 510660964.
[54] Renming Liu and Arjun Krishnan. Open biomedical network benchmark: A python toolkit for benchmarking datasets with biomedical networks. In David A. Knowles and Sara Mostafavi, editors, Proceedings of the 18th Machine Learning in Computational Biology meeting, volume 240 of Proceedings of Machine Learning Research. PMLR, 2024.
[55] Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science, 92),, 2018.
[56] James Dunbar, Konrad Krawezyk, Jinwoo Leem, Terry Baker, Angelika Fuchs, Guy Georges, Jiye Shi, and Charlotte M. Deane. SAbDab: the structural antibody database. Nucleic Acids Research, 42, 2013. ISSN 0305-1048. doi: 10. 10.109///a///k11104.
[57] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09285 2023.
[58) Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: datasets for machine learning on graphs. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS 20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
[59] Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann Lecun. Spectral networks and locally connected networks graphs. In International Conference Learning Representations (ICLR2014), CBLS, April 2014, 2014.
[60] Zhiyuan Liu and Jie Zhou. Graph Attention Networks. Springer International Publishing, Cham, 2020. ISBN 978-3-00 778-3-031-015877 doi:
[61] Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. Autogluon-tabular: Robust and accurate automl for structured data. arXiv preprint arXivv2003.06500, 2020.
[62] Seongjun Yun, Minbyul Jeong, Sungdong Yoo, Seunghun Lee, Sean S. Yi, Raehyun Kim, Jaewoo Kang, and Hyunwoo J. Kim. Graph transformer networks: Learning meta-path graphs to improve gnns. Neural Networks, 153, 2022. ISSN 0893-6080. doi: http:///oi.org//0.1011 neunet.202..05.026
14

---

[63] Petar Veličković, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and Devon Hjelm. Deep graph infomax. In ICLR 2019, 2019.
[64] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning, arXiv preprint arXiv:2006.00 201,,41,,,,,,,,,,,, 2020.
[65] Hongyang Gao, Zhengyang Wang, and Shuiwang Ji. Large-scale learnable graph convolutional networks. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, KDD '18, New York, NY, USA. 2018. Association for Computing Machinery. ISBN 9781 9781450355520. doi: 10.1 19814442299999921199
[66] Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research. PMLR, 2020.
[67] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20(1), 2009, doi: 10. 11090NN..0000.0000 2005600..
[68] Lingfei Wu, Peng Cui, Jian Pei, Liang Zhao, and Xiaojie Guo. Graph neural networks: Foundation, frontiers and applications. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '22, New York. NY, USA, 2022. Association for Computing Machinery. ISBN 9781450393850. doi:
04445/3534667.35426000
[69] Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. arXiv preprint arXiv:2012.009699 2020.
[70] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. 2023.
[71] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge. 2024.
[72] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, arXiv preprint 2110:2231. 2023.
[73] Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin- Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-LLM: Time series forecasting by reprogramming large language models. In International Conference on Learning Representations (ICLR), 2024.
[74] Taco Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge equivariant convo- lutional networks and the icosahedral CNN. In Kamalika Chaudhuri and Ruslan Salakhutdinov. editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research. PMLR, 2019.
[75] Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.134 2021.
[76] Monami Banerjee, Rudrasis Chakraborty, Jose Bouza, and Baba C. Vemuri. Volterranet: A higher order convolutional network with group equivariance for homogeneous manifolds. IEEE Trans. Pattern Anal. Mach. Intell., 44(2), 2022. ISSN 0162-8828. doi: 10.1109/TPAML2000 3035130. 130.
[77] Xuanyu Zhu, Yi Xu, Hongteng Xu, and Changjian Chen. Quaternion convolutional neural networks. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision - ECCV 2018, Cham, 2018. Springer International Publishing. ISBN 978-3- 030-01237-3.
[78] Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups arbitrary continuous data. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 1 119 of Proceedings of Machine Learning Research. PMLR, 2020.
15

---

[79] Maksim Zhdanov, David Ruhe, Maurice Weiler, Ana Lucic, Johannes Brandstetter, and Patrick Forré. Clifford-sterrbble convolutional neural networks. arXiv preprint arXiv:2402.14730, 2024.
[80] Marco A. S. Trindade, Vinícius N. A. Lula-Rocha, and S. Floquet. Clifford Algebras, Quantum Neural Networks and Generalized Quantum Fourier Transform. Adv. Appl. Clifford Algebras, 33(3), 2023. doi:
00007/000000022-012220
[81] P Erdös and A Rényi. On random graphs i. Publicationes Mathematicae Debrecen, 1959.
[82] Réka Albert and Albert-Lászle Barabási. Statistical mechanics of complex networks. Rev. Mod. Phys., 74, 2002. doi: 10. M03/Re//oddPyss.4..4.
[83] Fan Chung and Linyuan Lu. The average distances in random graphs with given expected degrees. Proceedings of the National Academy of Sciences, 99(25), 2002. doi: 10. 10.3033/pn 252631999.
[84] David B. Jaffe, Payam Shahi, Bruce A. Adams, Ashley M. Chrisman, Peter M. Finnegan, Nandhini Raman, Ariel E. Royall, Funien Tsai, Thomas Vollbrecht, Daniel S. Reyes, and Wyatt J. McDonnell. Functional antibodies exhibit light chain coherence. Nature, 611, 2022.
[85] Chris Stark, Bobby-Joe Breitkreutz, Teresa Reguly, Lorrie Boucher, Ashton Breitkreutz, and Mike Tyers. Biogrid: a general repository for interaction datasets. Nucleic acids research, 34 (Database issue), 2006. ISSN 0305-1048. doi: 10. (0.1093/nar/kk1009.
[86] Sohyun Hwang, Chan Yeong Kim, Summo Yang, Eiru Kim, Traver Hart, Edward M. Marcotte, and Insuk Lee. Humannet v2: Human gene networks for disease research. Nucleic acids research, 47(D1), 2019. ISSN 0305-1048. doi: 10.1093/mar/gky1113 126. Publisher Copyright: 2018 The Author(s).
[87] William Falcon and The PyTorch Lightning team. PyTorch Lightning, 2019.
[88] Lung-Hoon Han, Sarah Batey, Adrian A Nickson, Sarah A Teichmann, and Jane Clarke. The folding and evolution multidomain proteins. Nature reviews. Molecular cell biology, 2007. ISSN 1471-0072. doi: 10. 10.1038/nmm2144.
16

---

The code can be found in
A Dataset details
In our experiments, we used the following datasets:
1. GraphQA [5]:
tasks (number of nodes, number of edges, cycle existence, number of triangles, node degree, and edge existence) on different graph structures (Erdos-Renyi [8]] Scale-Free, Barabasi-Albert 82] Stochastic Block Model, Star, Path and Complete).
It includes 6 different
2. GraphReasoning [17
Recently introduced in [7 to better assess the model's graph understanding ability, it consists of 4 more advanced graph-understanding tasks (substructure count, maximum triples sum, shortest path, and bipartite graph matching). Each graph node is accompanied by extra information in the form of a text description, making this dataset suitable testbed for our AaBERTa-basee vector encoding.
3. HyperGraphQA: We extend GraphQA to Hypergraphs. Specifically, consider 4 different graph-understanding tasks (number of nodes, number of edges, node degree, and edge existence) on 2 different hypergraph structures (Erdos- (Erdos-Renyi [81], and Chung-Lu Lu [83]. The training dataset consists of only 2000 instances, making it hard for large models to avoid overfitting.
4. Jaffe [84] Jaffe is recent dataset consisting of approximately 1.6 million natively paired human antibody sequences from healthy donors. To our knowledge, this represents by far the largest publicly available dataset of its kind.
5. PPI [53: PPI consists of 24 proteins collected from human tissue, with each node associated with 121 binary labels. Compiled from experimental techniques like yeast two-hybrid screen- ing and mass spectrometry. as well as computational predictions, such a dataset provides critical insights into the functional organization of the proteome. By understanding how proteins interact, scientists can uncover the molecular underpinnings of cellular processes and develop targeted therapeutic strategies.
6. OBNB [54] OBNB (Open Biomedical Network Benchmark) is collection of 15 datasets (including well-known datasets such as BioGRID (85) and HumanNet [86)). Each dataset's sample consists of gene accompanied by 3 vectors (named DISEASES, DisGeNET, GOBP) of node-level binary labels.
7. SabDab [56]:
SabDab (Structural Antibody Database) collection of 919 publicly available, annotated antibody structures (proteins). Each structure is accompanied by multiple annotations, such as the heavy and light chain pairing.
8. mol-HIV [55):
The ogbg-molHIV dataset consists of molecular graphs (atoms as nodes, chemical bonds as edges) labeled for the binary classification task of predicting whether a molecule inhibits HIV replication or not. Each molecule is represented with 9-dimensional atom features (e.g. atomic number, chirality. ring membership. formal charge), and the dataset is evaluated using scaffold splits with ROC-AUC as the metric.
B FoGE-LLM
B.1 Training details
We train the LLM-based construction with batch size of 16 and a learning rate of 1e-3. The model required less than 10 epochs to convergence, in contrast to other works that require more training time due to the ellaborate graph encoders (e.g., (17). Our implementation is based on Pytorch Lightning [87. which allows us to split and train the model on multiple GPUs using FSDP, This implementation allows the user to train this, or any similar, model to conventional GPUs with less memory while, at the same time, speed up the process by preloading all the obtained lightweight graph embeddings to the GPUs The merging of the graph embedding with the LLM based on the idea of prefix tuning [88. i.e., pre-append the embedding to the input text embeddings and, in our case, this is happening with the use of a linear adapter. We experimented both with a single linear adapter on the input layer, as well as linear adapter per layer and the difference was only marginal in the final results.
17

---

B.2 Inference details
Besides the low training time, FoGE-LLM enjoys an extremely low inference time, due two reasons. First, we always "reserve" only a single token for the provided graph. In contrast, zero/few-shor approaches that textualize the graph require a large number of tokens, prohibitively large as the graph grows. This leads an explosion of the inference time, due to the transformer's quadratic dependency on the number of input tokens. Second, FoGE-LLMM employs one or more linear adapters and does not require any specialized architectures, like existing solutions 16]. This, as we observed in our experiments, impacts the inference time, casting FoGE-LLM one the fastest graph-augment
Language Models. In Table 2 we present the average inference time required for each approach.
Table Average inference time for each approach on Llama-7B. FoGE-LLM significantly lower than zero/few shot approaches since the number of input tokens does not grow with the graph size, while it enjoys a 40% improvement over GraphLLM dues to its simpler encoder/adapter.
Model
Inference time (s) 1
zero-shot
0.175 (+0005)
few-shot
0.541 (#0.10)
GraphLLMM [17
0.052 (+-0.01)
FoGE-LLM
0.031 (+0.01)
C ICL prompting for hypergraphs
In Table Table 7 we demonstrate FoGE's superiority over In-Context Learning approaches, like zero-shot and few-shot prompting. Here explain how created the textual descriptions of the hypergraphs, that were used in both zero- and few-shot prompting. Following similar works for graph textualization [] [5], we first assign a number each node and then, new line, we explain which nodes are part of each hyperedge. An example can be seen below.
G describes a hypergraph among 0, 1, 2, 3, 4, 5, 6, 7, and 8.
In this hypergraph:
Hyperedge connects nodes 2, 3, 6.
Hyperedge connects nodes 4, 5, 7. Hyperedge 3 connects nodes 2.
Hyperedge 4 connects nodes 5, 7,
After the hypergraph textualization, the question follows in the case of zero-shot, while both the question and the answer follow the case of few-shot.
D Lossless representations
One advantage of the obtained embeddings is that fact that the underlying structures are recoverable. This allows us obtain unbiased vector estimates of complicated structures, such as graphs with multiple edge and node attributes. Here, we show how this property manifests in our specific formulation as well as more generally for pairs of key-item.
D.1 Capacity
One of the typical ways to examine the performance of such a construction is by assuming a vector as being the bundling of multiple binded pairs, in the following equation
k Sv;
(5)
and then examine how accurately we can recover each vector v,, given the corresponding k, In theory, the vector v; can be easily recovered using the operation:
k
(6)
18

---

In Fig. 6 we examine the cosine similarity of the obtain vector V, with the correct one (v;) as well as with all the rest ([vz)zze) We observe that the results follow closely the theoretical results above, with perfect separation of up to 100 pairs, and small overlap for 200 300 pairs of vectors.
pairs:
poirs:
pairs:
pairs:
number pairs:
worst-case wrong CS
correct CS
ii= K; Vi. how correctly we recover all pairs of keys- values back, as the number of pairs (n) grows. Worst-case wrong CS corresponds to the maximum cosine similarity of the recovered value vector with all value vectors but the correct one, and correct CS corresponds to the cosine similarity with the correct value vector.
Figure 6: Given vector R4096
D.2 Graph reconstruction
In our specific application, we deal with graphs and, as we analyzed, the graph representations we obtain are, in theory, lossless, i.e., we can recover back the original graph from the vector representation using the inverse vectors. Here, we examine whether this claim holds in practice too. In Fig. you can observe the strength of each edge after reconstruction, for different vector dimensionalities. We can observe that, even for moderately large dimension, there is a clear separation between the true edge set and the rest of the edges.
Original
Reconstructions
R4
0.4
R77
64
A1
A3
04
6
dim: 16
dim: 32
dimm: 64
Figure 7: Lossless representations: even for small vector dimension, we can obtain back the true edge set. The numbers show the cosine similarity of the obtained vector with the true edge vector, and can be used estimate the true edge set.
E FoGE runtime
In Table 10, show the runtime of FoGE increase the number of edges, conventional consumer CPU. The graphs are randomly generated instances of Mitos--eniii-a graphs. From this experiment, three observations stand out:
1. The linear relationship between the number of edges and the runtime.
2. Fast encoding of graphs with even millions of edges.
3. The ability of FoGE to handle huge graphs without any substantial increase in memory consumption.
Additionally, given the properties of the aggregation we use, the operations can be parallelized on multiple CPU threads, speeding up even further the computation on larger graphs. For example, similarly to our experiments with OBNB, we can calculate in parallel single embedding per node, and then aggregate them together. This can lead to significant improvement in encoding, exploiting the fact that aggregation operations are lightweight and
on CPUs.
19

---

Table 10: Total encoding runtime for random instances of Erdos Erdos-Renyi 8
using FoGE.
number of edges
500 0.041 (=10.01)
50000 3. 7.7999 (=0)35)
1200000 91.011 (=1.01)
5000000
runtime (soc)
0.0006 (=0.000
378.00 178.018 (#1.55)
30
10
-10
3
20
4
-30
-20
-10
10
Figure 8: T-SNE plot of the SabDab embeddings. Although the dataset is very small, each one of the populated clades occupies different region and, interestingly, clades I and 7 very similar, just like in real life. The T-SNE plot was robust to different choices hyperparameters, with no significant differences beyond simple translations of the space.
Preservation of Clade information on SabDab
Given that the SabDab proteins [56] are annotated with the heavy/light chain pairing, we can extract the clades and visualize their embeddings with respect to that information. As brief reminder, the clades correspond to superfamilies
proteins that share a common ancestor [88] To extract clades used the V gene heavy chain and chose seven families. It is well known from biology that antibodics that belong to the same clade are more similar than antibodics across different clades, so, here, examine if this real-worth biological property is reflected our embeddings. Specifically, after obtaining each protein's 's embedding using FoGE (in an unsupervised fashion without using the apply T-SNE transformation the high-dimensional vectors that
clade annotations), able to plot them, with a significant amount of noise, in just two dimensions. Although we reduce the dimensionality significantly, and, even worse, deal with extremely small dataset of just 919 proteins (Table ITp. in Fig. g we can observe that the proteins of each clade cluster together. This is different, qualitative indicator, which shows that FoGE is able preserve all the information that is encapsulated in the inputted structures.
Table I:: Distribution of samples across the different clades. In total there are 919 samples, with clades 1,3, being the most frequent.
Clade
3
5
Total
Count 325 28
414
101
25
3 23
919
G Additional results on OBNB
OBNB (which stands for Open Biomedical Network Benchmark) a collection of multiple, real- world protein datasets, where each node (or amino-acid) of each protein is accompanied by multiple binary labels. detailed analysis of the datasets and their labels can be found (54 and the corresponding repository. In Table 12 present the results all 18 reported datasets of OBNB.
20

---

Table 12: FoGE vs multiple unsupervised and supervised methods. After obtaining our embeddings, we use Random Forest to predict the corresponding node's label. The evaluation is based the APOP metric [54] and we can observe that FoGE always comparable best methods, while almost half cases best onc.
Network
Model
DISEASES
DisGeNET
GOBP
LabelProp
1.210
0.931
1.858
LogReg
1.556
1.026
2.571
GCN+BoT
1.511
1.014
2,442
SAGE+BoT
1.486
1.031
2,402
BioGRID
GIN+BoT
1.410
1.007
2.386
GAT+BoT
1.609
1.037
2.624
GaledGCN+BoT
1.547
1.038
2.517
FoGE
1.599
1.062
2,433
LabelProp
3.728
3.098
3.806
LogReg
3.812
3.158
4.053
GCN+BoT
3.552
3.053
3,921
SAGE+BoT
3.052
3.816
HumanNet
GIN+BoT
3.513
3.054
3.861
GAT+BoT
3.761
3.100
3.809
Got
3.677
3.086
3.889
FoGE
3,853
3.254
3,916
LabelProp
1.352
2.076
LogReg
1.644
1,240
2,806
GCN+BoT
1.648
1.211
2.685
SAGE+BoT
1,694
1.210
2.629
COMPPIHumanInt
GIN+BoT
1.608
1.219
2.611
GAT+BoT
1.665
1.230
2.755
GatedGCN+BoT
1.672
1.218
2.735
FoGE
1.660
1.241
2.586
LabelProp
0.964
0.939
1.714
LogReg
1.358
0.939
2.587
GCN+BoT
1.324
0.911
2.553
SAGE+BoT
1.246
0.865
2.513
BioPlex
GIN+BoT
1.349
0.868
2.504
GAT+BoT
1.355
0.873
2.548
BotedGCN+BoT
1.301
0.859
2,590
FoGE
1.273
0.879
2.599
LabelProp
0.545
0.598
1.086
LogReg
0.650
0.656
1.084
GCN+BoT
0.634
0.693
1.129
SAGE+BoT
0.593
0.679
1.190
HuRI
GIN+BoT
0.583
0.702
1.143
GAT+BoT
0.667
0.687
1.174
GaledGCN+BoT
0.596
0.69S
1.195
FoGE
0.684
0.729
1.070
LabelProp
1.358
0.897
1.593
LogReg
1.542
1.093
2.125
GCN+BoT
1.577
1.068
2.071
SAGE+BoT
1.478
1.986
OmniPath
GIN+BoT
1.452
1.073
1.993
GAT+BoT
1.552
1,048
2.068
GotedGCN+BoT
1.516
L.049
2.071
FoGE
1.511
1.085
2.102
FoGE is one of the best-performing methods across all benchmarks, showcasing once more the capabilities of our obtained embeddings.
H Impact of vector dimension
One of few the hyperparameters of FoGE is the dimensionality of the vectors (i.e. graph embeddings). Using GraphQA, we perform an ablation study on the impact of the dimension on the final accuracy of the model (Fig. 9). Relative accuracy is calculated the actual accuracy for each dimensionality,
21

---

divided by the best one, for each task respectively, and it allows us to compare different tasks with completely different best performances (Table Table 5).
relative accuracy
0.8
0.4
node
cycle
0.2
edge
triangles
64
128
256
512
1024
2048
symbol dimension
Figure 9: Accuracy versus vectors dimensionality. Although there is positive trend between the two quantities, the dependency on the dimension is not equally strong always positive in all tasks.
From this study, few important remarks surface that observe to hold true for the other datasets too. First of all, a larger dimensionality does not always "translate" to better results. We observe that for some tasks (cycle existence), achieve the optimal performance with dimension significantly lower than the maximum we consider (2048), matching essentially Graph Token's 's performance with less than 20K trainable parameters, while in some 2048. Finally, as with most of the tunable hyperparameters predetermined best strategy for choosing the dimensionality. For instance, when consider cycle existence or the number of triangles we can have highly performing model with a dimensionality of less than 128, while for tasks such as edge and node count the performance drops significantly reduce the dimensionality.
there is small drop as we go from 1024 to
machine learning models, there is no
22

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes].
Justification:
All contributions are mentioned explicitly in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes].
Justification:
The limitations can be found on the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
23

---

Justification:
There are no proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes].
Justification: All models used and the corresponding hyperparameters are mentioned in the paper and appendix.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes].
Justification:
All models are publicly available and we have included our code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/dde/eeee Cooeeeoodeesssseeeeeee //nips.com htc/://bbb///gdddee//eeeeeuuisss ssiniiiiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes].
Justification:
All training hyperparameters are mentioned.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes].
Justification: We followed and reported the existing evaluation criteria, to avoid any discrep- ancies in the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
25

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes].
Justification:
All relevant information is provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://aaarr...cc/oaaaa/chccceee
Question:
Answer: [Yes].
Justification:
We have reviewed and adhere to the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes].
Justification: Impact is discussed on the conclusion.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification:
No new models or data are released.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes].
Justification: All used assets public and cited accordingly.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes].
Justification:
No new models datasets released. Our code documented appropriately,
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification: No crowdsourcing was involved.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification:
No study participants needed.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
28

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA].
Justification:
LLMs were used only for grammar checks.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neurd/ httppp///..///ooeeeeeee///////// for what should or should not be described.
29


Reviews:
[{"review_id": "jlcsf1cqz4", "reviewer": "Reviewer_dfsh", "summary": "The paper presents a novel parameter-free Fock space based-graph encoder to express graph-structured data in diverse applications. Existing works generally train graph Transformers or simply serialize the graph with triplets or paths. The proposed Fock space-inspired representation effectively encodes graph. The authors’s experiments, the proposed method demonstrates the effectiveness of the proposed method.", "questions": "- Could you provide the results on large-scale graphs?\n- Could you provide additional experimental results based on other large language models?\n- I think that the proposed method is also applicable to prompt tuning instead of prefix tuning. Could you have any experimental results based on prompt tuning?", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- The proposed method is clearly written to understand the paper.\n- The proposed method seems novel to me.\n- The paper demonstrates the effectiveness of the work theoretically and empirically. \n\n## Weaknesses\n\n- From my thoughts, the proposed method has a limitation un handling a large scale graph because it has to express both edges and vertexes. Could you discuss it?\n- It would be better if the authors included more results based on diverse large language models such as Qwen or Gemma.", "was_revised": true, "final_justification": "I have read all the reviews and the corresponding responses. Most of my concerns have been addressed and I decided to maintain my score.", "rebuttal": "We thank the reviewer for their time and effort. Below we address all of the concerns brought up in the review. \n\n## Question 1: Provide the results on large-scale graphs?\n\nThanks for the suggestion. We indeed experimented with large-scale graphs, this was described in Section 4.1. For example, OBNB consists of graphs with tens of thousands of nodes (e.g., BioGRID has 18951 nodes) and our implementation currently scales quite well. In these cases and to be consistent with existing works, we produce a single encoding per node (mentioned around line 323). We find that our encodings are a strong alternative and perform even better than trainable methods in many cases! We are happy to expand this text more.\n\nThe reviewer will also like that the complexity of the encoding (described around line 265), is *linear* in the size (number of edges) of the graph. The specific aggregation we use has a complexity of only $d\\log(d)$, where $d$ is the dimension of the vectors. So scalability is really a strong benefit.  \n\nTo fully address this concern, we also show the runtime of FoGE below as we increase the number of edges, on a conventional consumer CPU. The graphs are randomly generated instances of Erdos-Renyi graphs. \n\n| number of edges | 1 | 500 | 50000 | 1200000 | 5000000 |\n| --- | --- | --- | --- | --- | --- |\n| **FoGE runtime (sec)** | 0.0006 (0.0003) | 0.041 (0.0074) | 3.769 (0.3573) | 94.041 (1.0427) | 378.018 (4.5523) |\n\nFrom above discussion, three points are clear:\n\n1. Linear relationship between the number of edges and the runtime. \n2. Fast encoding of graphs with even millions of edges.\n3. The ability of FoGE to handle huge graphs without any large increase in memory consumption.\n\nAdditionally, given the properties of the aggregation we use, the operations can be easily parallelized on multiple CPU threads, speeding up the implementation even further on larger graphs. For example, similar to our experiments with OBNB, we can calculate in parallel a single embedding per node, and then aggregate them together. This can lead to significant improvement in encoding, since our aggregation operations are lightweight and can even run on CPUs.\n\n## Question 2: Additional results on other LLMs\n\nOur approach is backbone-agnostic by design: the graph encodings are converted to embeddings via simple linear adapters that can interface with any LLM. We chose LlaMA since it is widely used and was also the LLM backbone in one of our baselines (GraphLLM). While repeating the full set of experiments with multiple LLM backbones will be very expensive, we are happy to include experiments with one additional LLM if this is considered essential. We welcome suggestions. \n\n## Question 3: Prefix tuning vs prompt tuning\n\nWe appreciate this question, but we believe there may be some confusion about terminology. Prompt tuning typically refers to learning soft prompt tokens in the embedding space, while discrete prompting involves crafting text-based prompts. Our graph encodings are high-dimensional continuous representations that is converted to the LLM's embedding space via linear adapters - this is different from prompt tuning approaches that optimize token-level embeddings. The graph structural information we encode will be difficult to be meaningfully expressed through discrete text prompts or soft prompt tokens. We like the suggestion, and will include a discussion on prompt tuning versus prefix tuning in our work and include any specific experiment along this direction which will add value."}, {"review_id": "xIBhFCVKLV", "reviewer": "Reviewer_gsTE", "summary": "This paper proposes a graph encoding method for Large Language Models (LLMs) inspired by Fock Spaces, a concept from mathematical physics, with the goal of designing generalizable task-agnostic graph embeddings that can be adapted into LLM embedding space for graph-based tasks. Based on concepts from Clifford Algebras and Fock Spaces, a graph embedding is obtained for a given input graph by initializing a vector space for node embeddings and summing up circular convolutions of node embedding pairs based on the edges present in the graph. This creates a graph embedding which preserves the structure of the graph, which then is transformed by a learned linear layer into the embedding that is used by the LLM to represent the graph.", "questions": "Questions:\nFor the basic graph understanding tasks (Table 2, Section E.1) in the Appendix, it is unclear whether the performance on tasks is accuracy or MSE, making it difficult to gauge performance. For example, on GraphQA and HyperGraphQA, it is unclear for the num edges and num nodes task whether the performance indicates MSE or accuracy.\nThe authors reference results on nineteen real protein datasets from PPI and OBNB in Tables 1 and 2 in Results section 4.1, however it is unclear in the Table captions whether the performance is reported over nineteen datasets or rather the ones referenced in each caption. Clarifying the datasets in each Table and caption would help readability of the results.\nCan the authors provide analysis or discussion about the scalability of the method to larger graphs? For instance, on larger graphs of over 1 million nodes, how long would computing the graph embedding take? Additionally, would the dimensionality of the vector become a bottleneck in the ability to encode larger graphs?", "limitations": "The method lacks comprehensive comparisons to existing baselines in the GNN + LLM space. However, with more consistent baseline evaluation and more analysis of parameter efficiency versus baseline methods, the quality of the submission could be increased.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\nParameter Efficiency: The proposed method is parameter-free except for learned linear adapters that transform the graph embedding space into the LLM embedding space. The LLM is frozen in the experiments, giving the proposed method an advantage in terms of parameter-efficiency.\nOriginality: The method is novel in its formulation of graph encoding based on concepts from Fock space representations, and presents a new way of encoding graphs for LLMs for diverse graph domains. Additionally, the method allows for the encoding of node attributes, a limitation of some previous Graph Neural Network (GNN) + LLM works. The method is generalized up to hypergraphs as well, improving its applicability to various graph domains.\nEvaluation Datasets: The authors benchmark their method on six graph datasets, including graph reasoning datasets (GraphQA) and protein graphs (OBNB, PPI). This highlights the applicability of the proposed method.\n\nWeaknesses\nPerformance Versus Challenging Baselines: FoGE demonstrates similar, sometimes better performance to GNN baselines in the first phase of results (evaluating graph embedding quality) and GNN + LLM baselines in the second phase of results (grounding LLMs with graph prompting). Because the performance of the method is not consistently and significantly higher than baseline methods, the highlight of the method is its novelty and efficiency in graph encoding. This could be emphasized more by adding in trainable parameter counts in more of the results, as the authors did in Table 4.\nSignificance Values: Standard error in performance is not reported in any of the results tables, except for Table 3 in the appendix. Reporting standard error in more results would increase confidence in the performance gains FoGE does see over baseline methods (e.g. in Table 3 and 5 in the main text).\nBaseline Methods & Statistical Significance: Performance results versus other GNN + LLM works (Tables 3 through 5 in the main text) compare against different baseline models in each comparison, leaving a complete comparison on each dataset incomplete. Benchmarking against GraphToken, GraphLLM, and other prefix-tuning methods such as regular prompt tuning of the LLM with a textual graph description consistently on each dataset would give a more complete performance comparison.", "was_revised": true, "final_justification": "FoGE is a novel, parameter-efficient graph encoding method inspired by Fock spaces, applicable across diverse graph types including hypergraphs. Strengths include originality, scalability, and theoretical grounding. My earlier concerns on baseline coverage, statistical reporting, and evaluation breadth were largely addressed in the rebuttal. Some limitations remain, but the contribution is significant and timely, warranting a borderline accept.", "rebuttal": "We sincerely thank the reviewer for the time and effort in evaluating our work. Below, we address each of the concerns in detail. We hope our responses clarify any reservations and contribute to a stronger support of our submission.\n\n## Weakness 1: Performance Versus Challenging Baselines\n\nWe are delighted that the reviewer recognizes the multiple strengths of FoGE, beyond just the raw performance measures. The unsupervised nature of FoGE, its training-free approach, and the limited amount of trainable parameters in the LLM-related experiments are indeed some of the unique qualitative advantages of FoGE over GNN-based methods. \n\nWe much appreciate the reviewer’s suggestion, and we will include more information, similar to Table 4, about the adapters size, the number of graph embeddings, as well as other qualities of each approach for each experiment, to further highlight the strengths of FoGE.\n\n## Weakness 2: Significance Values\n\nWe appreciate the suggestion. Significance values were included for experiments where such information was available for all baselines or could be reasonably estimated, without a very large increase in the compute budget required to re-run the experiments. Based on this comment, we will include error bars/intervals for our method for all experiments.\n\n## Weakness 3: Baseline Methods (Performance results versus other GNN + LLM works on all benchmarks incomplete)\n\nWe thank the reviewer for this comment. You will see from the explanation below that our comparisons may appear incomplete because the baselines cannot adapt to all the different types of datasets we have used without very extensive changes. This is one of the main strengths of FoGE, which we highlight with these experiments, but we agree that it may give the wrong impression that method X was intentionally not run on dataset Y.  \n\n- Neither GraphLLM nor GraphToken are able to handle hypergraphs; so, in Table 5, we compare only with text-based approaches: zero and few-shot learning, as also noted in the review.\n- Similarly, GraphToken is not able to handle the datasets proposed by GraphLLM in Table 4, hence no comparison with GraphToken is meaningful in that case. We mention this a few times in the paper (e.g., lines 354, 389), but if suggested, we are happy to emphasize further.\n- Finally, comparison with text-based approaches of GraphToken and GraphLLM has already been established in those papers. Based on this suggestion, we will add these results too in the appendix.\n\nWe appreciate any suggestions or recommendations of additional baselines the reviewer believes are important to include.\n\n## Question 1: Performance metrics on Table 2 of Appendix\n\nAll results except the “has_cycle” are measured using MSE, since they correspond to regression tasks. The “has_cycle” task corresponds to accuracy (we used % sign but will say accuracy clearly). We apologize for any confusion. We will update the text so that the measurement details are more prominent. \n\n## Question 2: Results on PPI and OBNB\n\nThe complete set of results on the 18 datasets of OBNB are included in the appendix (in Table 5). There we can check the performance of FoGE on all datasets, compared with multiple strong supervised and unsupervised baselines, such as GCN, GAT, and GIN. Table 2 in the main paper demonstrates FoGE’s performance on PPI, showcasing that, although our setup is unsupervised, it is superior to most supervised methods, with an F1 score of 99.2. Finally, Table 1 included only a subset of the results on OBNB, due to space restrictions (the complete results are located in Table 5 of the appendix).\n\nWe are sorry if this was not clear, and we will update the manuscript to avoid any confusion.\n\n## Question 3: Scalability discussion and dimensionality dependence\n\nWe appreciate this nice suggestion, and we will emphasize it much more. The complexity of the encoding is provided around line 265: it is linear in the size (number of edges) of the graph, and the specific aggregation we employ only has a complexity of $d\\log(d)$, where $d$ is the dimension of the vectors. This is among the strongest advantages of FoGE, and we are happy to make this more prominent in the paper. \n\nBelow, we show the runtime of FoGE as we increase the number of edges, on a conventional consumer CPU. The graphs are randomly generated instances of Erdos-Renyi graphs.\n\n| number of edges | 1 | 500 | 50000 | 1200000 | 5000000 |\n| --- | --- | --- | --- | --- | --- |\n| **FoGE runtime (sec)** | 0.0006 (0.0003) | 0.041 (0.0074) | 3.769 (0.3573) | 94.041 (1.0427) | 378.018 (4.5523) |\n\nFrom above, three observations stand out:\n\n1. The linear relationship between the number of edges and the runtime. \n2. Fast encoding of graphs with even millions of edges.\n3. The ability of FoGE to handle huge graphs without any substantial increase in memory consumption.\n\nAdditionally, given the properties of the aggregation we use, the operations can be parallelized on multiple CPU threads (or GPU), speeding up even further the computation on larger graphs. For example, similarly to our experiments with OBNB, we can calculate in parallel a single embedding per node, and then aggregate them together. This can lead to significant improvement in encoding, exploiting the fact that our aggregation operations are lightweight and can even run on CPUs.\n\nRegarding the dimensionality becoming a bottleneck part of the question, we agree that the vector dimensionality can be a factor that can affect performance (and this is true for any encoder). For our construction specifically, we provided detailed studies of the vectors’ dimension impact on the quality of the results (Appendix D and H). Additionally, much more details on the generic encoding capabilities of the operations used are also discussed in [43, 44] in a different context."}, {"review_id": "3fEwfEPCfQ", "reviewer": "Reviewer_aZCB", "summary": "This paper introduces FoGE, a parameter-free graph representation framework inspired by Fock space in quantum mechanics. FoGE encodes graphs as unordered multisets of node features and constructs graph embeddings through a series of multi-linear operations. The method is general-purpose and can handle various graph structures, including simple graphs, hypergraphs, and attributed graphs. Furthermore, FoGE can be combined with a frozen language model via prefix-tuning using a lightweight linear adapter.", "questions": "See above.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n- The paper introduces a theoretically grounded and lossless graph representation framework inspired by Fock space. It is a convenient method that does not require trainable parameters in the embedding stage.\n- This method can generate graph embeddings that are not tailored to any specific downstream task. This suggests strong generalization across tasks and domains.\n\nWeakness:\n- The role of the large language model (LLM) in the overall framework is not well explained. It is unclear whether the LLM meaningfully contributes to understanding the graph structure or primarily serves as a text generation component. Further exploration of the interaction between the graph embeddings and the LLM would help clarify its impact and value in the architecture.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating our work and its multiple strengths. Below, we address their main concern regarding LLMs.\n\n## Weakness: Role of large language model (LLM) in the overall framework is not well explained. Unclear whether the LLM meaningfully contributes to understanding the graph structure or primarily serves as a text generation component. \n\nWe appreciate this question about the LLM's role. Briefly, we can read this as: why use an LLM at all and is it doing much? Our initial experiments (Section 4.1) show simple models can process FoGE embeddings quite well for traditional tasks. The rationale for LLMs was due to two advantages: (1)  language interface for flexible graph queries without pre-defining task types, and (2) easier integration with mature software ecosystems built around LLMs that reduce deployment overhead.\n\nSo does the LLM understand graph structure? Our experimental evidence suggests yes: the model does learn to reason about graph properties. For example, consider compositional reasoning: On complex tasks like bipartite matching, the LLM must understand by processing the encodings how local properties compose into global structures. Or for structural tasks, like max triplet sum or shortest paths, the LLM can also answer such questions successfully, indicating structural understanding.\n\nWhile this still needs more development, it mirrors multimodal success: just as LLaVA learns visual concepts and TimeLLM develops temporal reasoning once modalities are properly embedded, our results suggest LLMs can develop graph reasoning abilities when provided with rich structural encodings (like FoGE)."}, {"review_id": "loCC8uOPKJ", "reviewer": "Reviewer_nube", "summary": "This paper proposes FoGE, which applies Fock space concepts from quantum mechanics to create a parameter-free graph encoder for LLM prompting. The authors argue this approach can generate rich and informative graph representations that work well across different graph types, from simple networks to complex structures like proteins and hypergraphs.", "questions": "1. Can you provide more intuition or theoretical analysis for why Fock space concepts should be particularly well-suited for graph encoding?\n\n2. What would be the performance if FoGE were replaced with strong modern GNNs in the same LLM framework? Given that many advanced GNNs actually have relatively small parameter counts, the \"parameter-free\" advantage might not be as significant as claimed. This comparison would help isolate whether the benefits come from the Fock space encoding itself or simply from having a structured graph representation.", "limitations": "See Weakness", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. I appreciate the bold attempt to connect quantum mechanics with graph learning. It's refreshing to see researchers exploring mathematical frameworks from physics, even if the connection isn't immediately obvious.\n\n2. The method avoids the complexity of training additional parameters, which could be computationally efficient and reduce overfitting concerns compared to traditional GNN-based approaches.\n\n3. The authors show their method works across quite diverse graph types, which suggests it might be more general than domain-specific approaches.\n\n**Weaknesses**:\n\n1. I'm having trouble seeing why Fock space, which was designed for quantum particle systems, should naturally fit graph structures. The motivation seems to be \"we have this cool mathematical tool from physics, let's see if it works\" rather than \"here's a specific problem with current methods that quantum concepts can solve.\" This makes the whole approach feel somewhat unmotivated.\n\n2. While the paper introduces an intriguing approach, it would benefit from deeper theoretical examination. The authors claim the encoding provides rich and informative representations, but more analysis would help readers understand what specific graph properties are captured or preserved. Questions like how this encoding theoretically compares to existing methods, what makes it potentially advantageous, and under what conditions it should work well could strengthen the theoretical foundation significantly.\n\n3. The paper would benefit from a more systematic evaluation approach. Currently, the authors jump directly to LLM applications without first establishing how well the Fock space encoding works for graph representation learning itself.", "was_revised": true, "final_justification": "I think most of my concerns have been addressed; the only concern is that although the evaluation may seem expensive, it is not very comprehensive for both the LLM and GNN communities, such as only considering LLaMA2-7B and evaluating on a limited graph benchmark dataset. However, the innovation of this work is commendable, so I evaluate it as a borderline accepted manuscript and respect the views of other reviewers and AC.", "rebuttal": "We sincerely thank the reviewer for their time and effort in evaluating our work. Below, we address each of their concerns in detail. We hope our responses clarify any reservations and contribute to a stronger support of our paper.\n\n## Weakness/Question 1: Fit of Fock Space to graphs\n\nIf we view the title literally, this paper applies Fock space for graph encoding, which could be  intellectually interesting but indeed, in this case, your comment would be partly valid. But based on the explanation below, you will likely agree that Fock spaces are not our starting point: they come up as an important design choice in a sequence of modeling steps.\n\nConsider a graph that we want to encode. It is common to take the graph Laplacian to give the graph’s structural properties. Its square root (discrete Dirac operator) follows immediately from spectral graph theory. Now, Dirac operators naturally generate Clifford algebra representations - this observation has been used in several recent papers that apply Clifford algebras for graph-like structures, albeit for modeling dynamical systems or message passing on topological structures. The graph to Clifford link in existing works is nice, but for graph encoding specifically, we know that using Clifford algebra directly does not yield radical benefits due to the 2^n dimensional complexity. So, we are partially stuck.\n\nThis is exactly where Fock spaces enter the story. As we show in Section 2.2 (lines 145-161), invoking Fock spaces via the spinor representation reduces this 2^n complexity while providing a very useful insight: creation and annihilation operators. Why is this important? Given the already known connections (although from much older literature) between Fock spaces and Vector Symbolic Architectures, we can then use very effective practical options for creation/annihilation: this step makes VSA use theoretically grounded, and allows a unified treatment of different graph types with identical machinery and strong performance which all reviewers appreciated. We hope this answer was satisfactory, but please let us know and we will be very happy to engage more on this point!\n\n## Weakness 2: Deeper theoretical examination\n\nWe appreciate the suggestion to more clearly write down the theoretical benefits.\n\nFoGE preserves several key graph properties that existing methods often sacrifice:\n\na) Unlike spectral or neural methods that often compress graph information, our encoding is mathematically lossless (lines 217-224). Given an embedding, we can recover any edge $(i,j)$ by evaluating $p_j^T(p_i^{-1} \\otimes g)$, and graph size via $p_i^T(s^{-1} \\otimes g)$. This property is rare in graph encoding literature to our knowledge.\n\nb) The creation/annihilation operator framework, while instantiated heuristically, naturally handles diverse graph components (nodes, edges, hyperedges, attributes) through the same algebraic operations like $\\otimes$ and $\\oplus$. Most methods will require architectural changes for different graph types, but we can use the same machinery.\n\nWe can also enumerate theoretical advantages over some alternatives. Versus spectral methods, we preserve local structure, not just global connectivity patterns. Versus GNNs, there is no over-smoothing, nor architectural constraints on graph size or type. Versus graph transformers: we get linear complexity instead of quadratic attention.\n\nBy design, FoGE will work well when: (1) preserving complete structural information is important, (2) we want to handle diverse graph types easily, (3) computational efficiency is desirable. It may be less optimal when task-specific inductive biases are available and more important for the use case. In principle, we can encode multi-scale relationships through the hierarchy: single nodes (grade-1), pairwise interactions (grade-2), higher-order patterns (grade-k). This gives a principled and convenient multi-resolution view.\n\n## Weakness 3: Evaluation approach- *the authors jump directly to LLM applications without first establishing how well the Fock space encoding works for graph representation learning itself.*\n\nOne of our biggest goals/concerns when writing the paper was to provide a complete evaluation of FoGE in isolation and not merely as a module living inside an LLM. So, our experiments included a thorough examination of the encoding capabilities of FoGE, its resilience to noise, the effect of vectors’ dimension, and how well it preserves graph relationships. Below we list what we have shown.\n\n1. Tables 1 and 2 demonstrate that FoGE is one of the best approaches for biological datasets, such as BioGRID, HumanNet, and PPI. \n2. Appendix D demonstrates how well our operations function as we increase the size of the vectors and the size of the underlying graphs.\n3. Appendix E is dedicated to both simple and advanced graph understanding tasks, using 4 different datasets (GraphQA, Jaffe, HyperGraphQA, and mol-HIV). Our results show that FoGE is not lacking in performance in any case, and it is one of the best solutions for real-world datasets like mol-HIV.\n4. Appendix F shows how well FoGE encodings reflect the real graph relationships, by showcasing that these encodings preserve the clade information of the proteins, although no such information is available during encoding.\n5. Appendix G generalizes the results of the main paper (Table 1), showcasing FoGE’s capabilities across a collection of 18 different datasets which are part of the OBN Benchmark.\n6. Finally, Appendix H depicts the significance of the vector dimension across the tasks of GraphQA.\n\nThese experiments, we hope you agree, are quite extensive so we presented a summary of all the results in Section 4.1, while the details are pushed to the appendix. We would appreciate any further suggestions that will help us strengthen the results further and showcase even better the capabilities of FoGE.\n\n## Question 2: FoGE-LLM vs GNN-LLM\n\nThe differences between a GNN-based LLM and a FoGE-based LLM are indeed studied in the paper. GraphLLM uses the exact same backbone and overall training procedure as we do, making the comparison as fair as possible. Despite FoGE-LLM being problem-agnostic and with about 25% of GraphLLM’s trainable parameters, we observe no performance degradation at all (Table 4). Additionally, FoGE-LLM is applicable to scenarios where GNN-based LLMs fails, such as hypergraph understanding (Table 5). \n\nFinally, we fully agree that many modern GNNs have a relatively small parameter count. We examine the performance of such models (e.g., GIN, GCN, GAT) and compare it against FoGE on multiple different settings and datasets (Tables 1, 2 of the main paper, Tables 3, 5 of the Appendix). Our findings consistently show that FoGE offers multiple advantages over such models, beyond just a strict reduction in parameter count.\n\nWe welcome any suggestions on additional models that the reviewer feels should be part of our experiments. FoGE works quite well and we are confident that any additional experiments will be consistent with what is included in the paper."}]

Author rebuttal:

Meta-review: This paper addresses the problem of designing a graph embedding to be used as a prompt to a pretrained LLM. The paper proposes a graph encoding method inspired by Fock Spaces, a concept from mathematical physics, with the goal of designing generalizable task-agnostic graph embeddings that can be adapted into LLM embedding space for graph-based tasks. Based on concepts from Clifford Algebras and Fock Spaces, a graph embedding is obtained for a given input graph by initializing a vector space for node embeddings and summing up circular convolutions of node embedding pairs based on the edges present in the graph. This creates a predefined graph embedding (with no trainable parameters) which mostly preserves the graph information. This embedding is then transformed by a learned linear layer to define the embedding that is used as an input to the LLM. The LLM here is pretrained and frozen, namely, in training, only the linear layer is optimized. 

The reviewers were mostly positive about this submission, stating that it is timely, the graph embedding is novel, efficient, theoretically grounded, and applicable across diverse types of graphs.
Decision: Accept (poster)