Hierarchical Graph Tokenization for Molecule-Language Alignment
Authors: Yongqiang Chen
Quanming Yao
Juzheng Zhang
James Cheng
Yatao Bian
Venue: ICML 2025 poster (2025)
Keywords: molecular-language alignment
large language models
hierarchical graph neural networks
tokenization
biomolecular studies
molecule
Primary area: deep_learning->graph_neural_networks
TL;DR: We show that integrating the intrinsic hierarchical graph information is essential for molecule-language alignement.

Abstract: Recently, there has been a surge of interest in extending the success of large language models (LLMs) from texts to molecules. Most existing approaches adopt a graph neural network to represent a molecule as a series of node tokens for molecule-language alignment, which, however, have overlooked the inherent hierarchical structures in molecules. Notably, higher-order molecular structures contain rich semantics of functional groups, which encode crucial biochemical functionalities of the molecules. We show that neglecting the hierarchical information in tokenization will lead to subpar molecule-language alignment and severe hallucination. To address this limitation, we propose HIerarchical GrapH Tokenization (HIGHT). HIGHT employs a hierarchical graph tokenizer that encodes the hierarchy of atom, motif, and molecular levels of informative tokens to improve the molecular perception of LLMs. HIGHT also adopts an augmented instruction tuning dataset, enriched with the hierarchical graph information, to further enhance the molecule-language alignment. Extensive experiments on 14 real-world benchmarks verify the effectiveness of HIGHT in reducing hallucination by 40%, and significant improvements in various molecule-language downstream tasks. The project is available at https: //higraphllm.github.io/.

Hierarchical Graph Tokenization for Molecule-Language Alignment
Yonggiang Chen 123
Quamming Yao Juzheng Zhang S
James Cheng
Yatao Bian
Abstract
languages (Radford et al., 2019; OpenAl, 2022; Touvron et al., 2023a; Bubeck et al., 2023). Recently, there has been surge of interest in extending the capabilities of LLMs to graph-structured data (Jin et al., 2023; Li et al., 2023d; Wei et al., 2024; Mao et al., 2024; Fan et al., 2024), par- ticularly molecular graphs (Zhao et al., 2023; Cao et al., 2023). Inspired by the success of large vision-language models (Zhang et al., 2024b; Liu al., 2023a), recent ef- forts in developing large graph-language models (LGLMss) typically adopt graph neural network (GNN) (Xu et al., 2019) to tokenize molecules as a series of node embed- dings (or node tokens), and then leverage an adapter such as a Multi-layer perceptron (MLP) or a Q-former (Li et al., 2023a) to transform the node tokens into those compatible with LLMs (Fan al., 2024). To bridge the gap between the graph and language modalities, LGLMs will undergo molecule-language instruction tuning with the molecu- lar graph and the corresponding captions describing the molecules (Jin et al., 2023; Li et al., 2023d; Fan et al., 2024).
Recently there has been a surge of interest in extending the success of large language mod- els (L.LMs) from texts to molecules. Most ex- isting approaches adopt a graph neural network to represent a molecule as a series of node to- kens for molecule- Laneuull- alignment, which, however, have overlooked the inherent hierar- chical structures in molecules. Notably, higher- order molecular structures contain rich semanties of functional groups, which encode crucial bio- chemical functionalities of the molecules. We show that neglecting the hierarchical informa- tion in tokenization will lead to subpar molecule- language aligmment and severe hallucination. To address this limitation, we propose Herarchiccl GrapH Tokenization (HTGHT). HIGHT employs hierarchical graph tokenizer that encodes the hierarchy of atom, motif, and molecular levels of informative tokens to improve the molecu- lar perception of LLMs. HIGHT also adopts an augmented instruction tuning dataset, enriched with the hierarchical graph information, to fur- ther enhance the molecule-language alignment. Extensive experiments on 14 real-world bench- marks verify the effectiveness of HIGHT in re- ducing hallucination by 40%, and significant im- provements in various molecule-language down- stream tasks. The project available at https: //higraphl 1mm github. io/.
Despite recent progress, the tokenization in existing LGLMs neglects the essential hierarchical structures inherent in molecular graphs. In particular, in molecular graphs, the high-order substructures, such as motifs or functional groups, encode rich semantics of the biochemical function- alities of the molecules (Milo et al., 2002; Bohacek et al., 1996; Sterling & Irwin, 2015). For example, the presence of hydroxide functional group ("-OH") often indicates higher water solubility. Therefore, such substructural cues are essential for enabling LLMs reason about the molecules in chemically meaningful way. However, exist- ing LGLMs mostly tokenize molecules solely at the atom (node) level, and feed LLMs with only node-level tokens. Consequently, it requires LLMs to implicitly infer the un- derlying substructures during the instruction tuning stage. The absence of the critical substructures not only increases the unnecessary burdern on the LLMs, but also leads to misaligned representations and higher likelihood of hal- lucinations in downstream tasks. To quantify the issue, we introduce a diagnostic benchmark, called Moti MotifHal lu, which evaluates the perception ability of LGLMs about the existence of common functional groups. Surprisingly, we find that existing LGLMs often produce false-positive predictions (i.e., keep answering "Yes" for any functional groups), highlighting a critical limitation in current graph to-
1. Introduction
Large language models (LLMs) have demonstrated impres- sive capabilities in understanding and processing natural
Most of the works were done when Yongqiang Chen was PhD student at CUHK. 'BBZUAI 'Carnegie Mellon University "The Chinese University of Hong Kong 'Tsinghua University University of Maryland, College Park 'Department of Computer Science, National University of Singapore. Correspondence to: Yatao Bian <bianyt@comm edd.es..sg.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Hierarchical Graph Tokenization for Molecul--Language Alignment
The molesule a eybbheeadeenecarooyyy acid heving CeC bonds et the and S-poettions..
MusiHall
This molecute has 1 carboxylic adids group, and 2 chal-chain hydroxyts groups...
Classntition
Large Language Model
Property Regieston
Foward Prediction
HIGHT
"Can you toll mo more about this molocula?
Node
Mone
Text Tokenizer
Nolecuaar Caption
Reagert
(a) Overview of the HIGHT framework.
(b) Summary of performance.
Figure 1. (a) Illustration of HIGHT: Given a molecule (i.e., PubChem ID 3. 5,6 26-Diihydraxyeyclohexe 1,. 1.3-diene---eee -carboxylic acid). HIGHT detects the motif's and incorporates the "supernodes" for each motif (The whole graph is also considered as "super motif".) .). Then, HIGHT tokenizes the molecule into both node-level (i.e., atoms) and motif-level (i.e., functional groups) tokens. The hierarchical view enables LLMs align the molecular structures and the language descriptions of the molecule better. (b) Performanee Overview: HIGHT significantly reduces the hallucination of LGLMs and improves the downstream performance across various molecule-centric tasks. Due to the heterogeneity of the evaluation metrics each task, we perform some transformations on the numerical values. In MotifHallu, we report the macro Fl scores. For Property Classification and Molccular Caption, we report the averaged scores of all the subtasks or submetrics. For Property Regression, we normalize the values to the range between and 100. i.e. for a. the reported number is 0.5/a. a. For Chemical Reaction Prediction, we report the averaged values of BLEU, RDK, MACCS, and MORGAN.
kenization strategies (Sec. 3.2). This observation motivates the following research question:
also propose the first hallacination benchmark Mot i ifriffalllee lu, synthesized through question-answering based on common functional groups.
Is there a feasible approach to integrate the intrinsic hierarchical molecular information into LLMs?
We conduct extensive experiments with 14 real-world benchmarks. The results show that HIGHT significantly reduces the hallucination Mot i iffflllallu by up to 40% and consistently improves the performances on down- stream molecule-language tasks.
To tackle the problem, we propose a new pyolecule-language alignment strategy called Hlerarchical GrapH Tokenization (HIGHT). As illustrated in Fig. 1, HIGHT adopts a hierarchi- cal graph tokenizer and a hierarchical molecular instruction tuning dataset to facilitate better alignment of molecule and language modalities. Specifically, inspired by the suc- cess of hierarchical GNNs in molecular representation learn- ing (Zhang et al., 2021; Zang et al., 2023; Inae et al., 2023; Luong & Singh, 2023), HIGHT transforms the original molecular graph into a hierarchical graph with motif-level and molecule-level nodes added in. Then, HIGHT employs Vector Quantized- Variational AutoEncoder (VQVAE) to obtain atom-level, lotif- and molecule-level tokens separately with the self-supervised tasks (Zang et al., 2023).
Hence, HIGHT together with Motifllllu and Hi PubChem, lay the solid foundation for developing graph foundation models via graph-language alignment.
2. Preliminaries
Large Graph-Language Models. As LLMs have demon- strated great capabilities across wide range of natural language tasks, there has been increasing interest in extending LLMs to broader applications where the text data are associated with the structure information (i.c.. graphs) (Jin et al., 2023; Li et al., 2023d; Wei et al., 2024; Mao et al., 2024; Fan et al., 2024). A graph can be de- and
In addition, to further encourage the encoding and alignment of hierarchical information, HICHT augments the original molecular instruction tuning dataset with motif-level de- scriptions. Our contributions can be summarized as follows:
noted as g (V.E) with set of n nodes v set of m edges (u, v) E E. Each node has node at- tributes T. E R2d and each edge (u, v) has edge attributes Cup E Rdd, A number of LGLMss have been developed to process graph-text associated data D = {G. c}, where c = [c],..., 0,] is to the caption of the graph G. For node- centric tasks, C; will associate with the nodes (Tang et al., 2023), while in this paper we foeus on graph-centre tasks,
To the best of our knowledge, we are the first to incor- porate the hierarchical graph information into LGL,ss with the consideration of both the architecture-level and the instruction tuning data.
To facilitate the molecule-language alignment study,
2

---

Hierarchical Graph Tokenization for Molecule-Language Alignment
i.e., molecules and molecular captions (Liu et al., 2023c). Usually, an 1-layer GNN is employed encode a graph as:
language models with more sophisticated graph information fusion modules. For example, Liu et al. (2023c); Zhao et al. (2023) develop advanced cross-modal adapters and gener- alized position embeddings to promote better alignment based on encoder-deecnder-base
h = COM(n (t AGG(((h) (h "n, )u E ("))), (1)
molecular language mod- els. Liang et al. (2023); Cao et al. (2023); Li et al. (2024) develop cross-modal adapters for decoder only language models such as Llama (Touvron et al., 2023a). Orthogo- nal to the aforementioned works, we focus more on what information one shall extract from the molecules for the alignment. We choose to build our methods upon decoder- only language models, with the hope of building a versatile agent that can perceive molecules beyond the language, im- age, and audio modalities (Xi et al., 2023).
where he Re'h refers to the node embedding of node u after I layers of GNN, AGG(-) is the aggregation function (e.g., mean) among the information from neighbors of node u, and COM is the operator for combining information of node u with its neighbors (u) (c.g., concatenation). Then, after I message passing iterations, the graph-level embedding can be obtained as:
hg = READOUT (In
(2)
where READOUT(-) is a pooling operator (e.g., mean pooling) among all the node embeddings. With the rep- resentations of the nodes and graphs, LGLMs can fuse the graph and language information in various ways, such as transforming into natural languages describing the graphs (Fatemi et al., 2024), or neural prompts within the LLMS (Tian et al., 2024). In addition, the embeddings can also be leveraged to post-process the LLM outputs (Liu et al., 2024b). Orthogonal to different fusion mechanisms, in this work, we focus on transforming graph embeddings into imput tokens of L.MMs, which can be formulated as (Tang et al., 2023; Chen al., 2024a; Liu et al., 2023c; Zhao et al.,
In the meantime, existing works also try to enrich the pooeeule-langusze alignment with additional modalities, such as 2D (Liu et al., 2023c) and 3D (Li et al., 2024) infor- mation. In contrast, we focus on the intrinsic hierarchical information of the molecules, such as motifs.
Hierarchical Graph Representation Learning. The hi- erarchical nature has been widely incorporated in learning high-quality graph representations (Ying et al., 2018). Espe- cially in molecular graphs, the high-order structural informa- tion naturally captures the existence of motifs and functional groups. Therefore, the hierarchy of atom-motii-moleculle has been widely applied in self-supervised molecular rep- resentation learning (Zhang ct al., 2021; Zang et al., 2023; Inae et al., 2023; Luong & Singh, 2023). Nevertheless, how to properly incorporate the hicrarchical information in molecular instruction tuning of LGLMS remains unclear.
2023; Cao et al., 2023: Li et al., 2024):
po(alq, h)
po(a: (h)),
(3)
where the LGLM is required to approximate pe to out- put the desired answer a given the question q, and the graph tokens h adapted with adapter fn I A that projects the graph tokens to the embedding space of LLMs. One could also incorporate the 1D information such as SMILES (Weininger, 1988) into q and for alignment.
In addition, concurrent works by Park et al. (2024) and Hu & Li (2024) explored incorporating hierarchical graph in- formation into LLMs. Nevertheless, they mostly focus on the architecture-level incorporation, while we show that it is crucial to integrate the hierarchical information in the instruction tuning data. More importantly, we highlight the consequences of inadequate alignment due to the lack of hierarchical information, i.e., hallucination, and demon- strate the usefulness of the hicrarchical information in a wide range of downstream tasks.
Molecular Foundation Models. There is a separate line of works aiming to develop language models for molecules and proteins- the language of lives, from ID sequences such as SMILES (Irwin et al., 2022), 2D) molecular graphs (Rong et al., 2020; Wang et al., 2022; Zhang et al., 2024a), 3D ge- ometric conformations (Liu et al., 2022; Zhou et al., 2023), to scientific text (Beltagy et al., 2019) and multimodal molecule-text data (Liu et al., 2023b; Luo et al., 2023a; Christofidellis et al., 2023; Liu et al., 2024c; Su et al., 2022: Zeng et al., 2022; Srinivas& & Runkana, 2024). The adopted backbones range from encoder-decoder architectures such as MolT5 (Edwards et al., 2022) and Galactica (Taylor et al., 2022), to auto-regressive language modeling (Luo et al., 2023b; Liu et al., 2023c). Inspired by the success of large laiiual models (Li et al., 2023a; Zhu et al., 2023; Liu et al., 2023a), the community further seeks to develop molecular foundation models built upon existing molecular
3. Graph Tokenization in LGLMs
In this section, we analyze the limitations of node-centric tokenization, which is widely adopted in existing LGLMs.
3.1. Node-Centric Tokenization
Specifically, most existing LGLMs directly take the node tokens from GNNs inputs LLMs (Cao et al., 2023):
Po pa(alg q. h) = II
- pa (a: fo (hr),, ..., fo (hm),, aci),
(4)
3

---

Hierarchical Graph Tokenization for Molecule-Language Alignment
where h1 m.y hy are node embeddings from GNN typi- cally pretrained through self-superrised learning large- scale molecular datasets such as ZINC250K (Sterling & Irwin, 2015). fn is the adapter to project the node tokens to the LLM tokens. There are various options to tokenize a molecule (Liu et al., 2023d). In this work, we consider a state-of-the-art tokenizer (Xia et al., 2023) that pretrains VQVAE (van den Oord al., 2017) with masked atoms modeling and constructs codebook 2 to discretize atoms: Zu arg min,)/k hh... ei e2 where 2. E Σ is the quantized index of atom and e, is the codebook embedding of the i-th entry. The codebook is trained through reconstruction loss with respect to some attribute v, of atom i:
functional groups by LGLMs. Specifically, consider the 38 common functional groups in RDKit! and leverage RDKit (Landrum, 2016) to detect the existence. We adopt 3, 300 molecules from ChEBI-20 (Edwards et al., 2021) and query the existence of a functional group:
Is there <functional group name> in the molecule?
Then, we examine the outputs from LGLM meaning "Yes" or "No". For each molecule, we construct questions with positive answers for all kinds of functional groups detected in the molecule, and questions with negative answers for randomly sampled functional groups from the remaining. Hence Mot fHa llu consists of 23, 924 questions. While it is easy to scale up Mot iffalllllll lu with more molecules and functional groups, we find that the current scale is al- ready sufficient to demonstrate the issue (Table 2).
1
C
)7
+
|sg[
[h,]
ez.
|
|b
71
i=1
8 + 2
|sg] [e,.] - h.112.
(5) where sg]:] is the stop-gradient operator in straight-through estimator (Bengio al., 2013), 6; is the reconstructed at- tribute of atom i with a decoder, and 9 is hyperparamter. In Mole-BERT, the attribute is simply the type of atom. Mole-BERT also manually partitions the codebook into groups of common atoms such as carbon, nitrogen, and oxygen to avoid codebook conflicts (Xia et al., 2023).
4. Hierarchical Graph Tokenization
To improve the molecule-language alignment, we propose new strategy called Hlerarchical GrapH Tokenization (H (HT)H which contains a hierarchical graph tokenizer and a hierarchical molecular instruction tuning dataset to augment the inputs with hierarchical information.
Intuitively, the trained atom tokens encode some contextual information, such the neighbors of the atoms. However, node-centric tokenization makes the molecule-language alignment more challenging, as LLMs have to addition- ally relate the multiple nodes to align the corresponding texts during the instruction tuning process. Specifically, in molecules, motifs or functional groups usually capture rich semantics, and often share many common atoms such as carbon, nitrogen, and oxygen (Bohacek et al., 1996). As shown in Fig. 2, both the carboxylic acid ("R-COOH") and the hydroperoxide ("R-OOH") functional groups all contain two oxygen atoms and a hydrogen atom. For a molecule with hydroperoxide attached to a scaffold with carbon atoms, would be hard for LLMS to distinguish which functional group is present in the molecule. Furthermore, due to the loss of positional information in the node-centric tokeniza- tion (Liang al., 2023; Cao et al., 2023), the limited expres- sivity of GNNs (Xu et al., 2019) and the positional biases of auto-regressive LLMs (Lu et al., 2022), it is more challeng- ing for LLMs to relate the desired nodes in a motif, which will lead to subpar molecule- languagg alignment.
4.1. Hierarchical Graph Tokenizer
Inspired by the success of hierarchical GNNs (Zhang et al., 2021; Zang et al., 2023), we transform the original molec- ular graph 9 into a hierarchical graph G' with lotif-level and molecule-level nodes added in. Specifically, we lever- age the Breaking of Retrosynthetically Interesting Chemical Substructures (BRICS) algorithm (Degen et al,, 2008)² to detect and inject set of k + supernodes, denoted as M (M(1), ..... M(R) (B+11)), with k motifs and the original molecule Ml++1) = G. Furthermore, denoting the set of nodes and edges in M y! and ele respec- tively, we augment the original molecular graph G as g' with augmented nodes V and edges Ex:
V' = VU (ull em ((+1)), E ClI))
(6)
where un is the motif super nodes added to the original molecule, and Eli = U weven ((u, v(ll)) are the augmented edges connecting to the motif super node from nodes within the corresponding motif. We employ separate VQVAEs for atoms and motifs to learn meaningful code embeddings with
3.2. Motif Hallucination
'nttps:/ http::://ggttbbb com/ htt////kkt/////oloo./ master/Data /at.aaaaaaaaaiiiaaaaaa Punct ional Groups. txt
To understand the issue of node-centric tokenization more clearly, we construct a simple benchmark called Moti i fHal Lu, to concretize the hallucination of common
"Note that HI GHT possesses a high degree of extensibility and can be augmented by incorporating advanced motif extraction techniques (such as (Zhang et al., 2021)).
4

---

Hierarchical Graph Tokenization for Molecule-Language Alignment
No, the hydroxide group conslats of -OH, but...
HO
Yes, hydroxide group is present in the molacule
Large Language Model
Large Language Model
"Is there hydroxide
"Is there hydroxide in this molscule?"
N
14
in this molecule?"
N
10
-04
Tokens
Node Tokens
Motif Tokens
(a) Node-centric tokenization.
(b) HIGHT tokenization.
Figure Illustration of hallucination caused by node-centric tokenization. With only node-level tokens, LLMs have to relate the nodes within specific functional group to align useful molecular structures with the corresponding language descriptions. Yet, due to the arbitrary order of atoms and position biases in LLMs, it hard to recognize each functional group, leading to severe hallucinations.
several self-supervised learning tasks. The reconstructed attributes in Eq. 4 include atom rypes at the atom-level and the number of atoms at the motif-level (Zang et al., 2023).
To this end, we propose HiPubChem, which augments the molecular instruction tuning dataset with captions of the functional groups. We consider both the positive and negative appearances of motifs: For the positive case, we directly append the caption of all functional groups detected with RDKit. We also include brief introduction of the functional groups to provide fine-grainet information for molecule-language alignment. For the negative case, we randomly sample kineg motifs not appeared in the molecule to explicitly instruct LGLMs on the absence of the mo- tifs. Despite the simple augmentation strategy, we find that HiPubChen significantly reduces the hallucination issue and improves the alignment performance.
Merely feeding the motif tokens with node tokens to LLMs still can not help distinguish the motifs from atoms properly, hence we propose to further attach positional encodings p to all of the tokens. We choose to use Laplacian positional embeddings (Dwivedi et al., 2020) while one could also adopt other variants (Ying et al., 2021). Since different types of tokens contain distinct semantics, we adopt separate adapters for different types of tokens. Denoting the motif tokens as (4) for motif (i)), generation with HIGHT is:
po(alg qq h. 1m))
III po (h1)), Su(hn); fm (h ) fo (1tt)))) aci);
4.3. Hierarchical Graph Instruction Tuning
We use two-stage instruction tuning (Cao al., 2023).
(7)
where (-) and f.) are the adapters for BRICS motifs and the original molecules, respectively.
Stage 1 Alignment Pretraining. We curate new molecule- text paired dataset from PubChem following the pipeline of Liu et al. (2023b). We set the cutoff date by Jan. 2024, and filter out unmatched pairs and low-quality data, which results 295k molecule-text pairs. Furthermore, we con- struct the HiPubChem- 225k dataset. The first stage mainly warms up the adapter to properly project the graph tokens with the LLM embedding space. To avoid feature distortion, both the LLM and the GNN encoder are frozen.
4.2. Hierarchical Graph Instruction Tuning Dataset
Although HIGHT tokenizer properly extracts the hierarchi- cal information from the input graph modality, it remains challenging to properly align the language to the correspond- ing molecular information, without the appearance of the respective captions in the texts. For example, if the caption does not contain any information about the water solubility of the hydroxide functional group ("-OH"), LGLMs will never know that "-OH" motif corresponds to the water solu- bility of the molecule, despite that HIGHT tokenizer extracts the "-OH" token. In fact, the commonly used molecular in- struction tuning curated from PubChem (Kim et al., 2022) in existing LGLMs (Liu et al., 2023c; Cao et al., 2023; et al., 2024), contains surprisingly little information about motifs. Some samples are given Appendix C.2.
Stage 2 Task-specific Instruction Tunning, With prop- erly trained adapter, we further leverage the specifce instruction tuning datasets from Mo MoleculeNes (Wu et al., 2017), ChEBI-20 (Mendez et al., 2019), and Mol In--Instuuciion ions (Fang al., 2024). More details are given in Appendix C. In Stage 2. we still keep the GNN encoder frozen, while tuning both the adapter and the LLM (with low-rank adaptation, i.c., LoRA (Hu et al., 2022)).
5

---

Hierarchical Graph Tokenization for Molecule-Language Alignment
Table 1. Detailed results in motif ballucinations Met i fHa u.. Due the imbalance of samples from positive and negative classes, we incorporate diverse evaluation metrics to provide detailed comparison between different methods in terms of hallucination.
METHOD
Macro FI f
F1 (pos) f
Fl (neg)
Micro F1 f
AUROC +
Acc
Yes Ratio
4, 124
19, 800
GIMLET (Zhao et al., 2023)
50.0
0.1
99.9
0.05
49.9
82.6
0.2
Galactica-6.78 (Taylor et al., 2022)
56.6
17.5
95.7
12.9
50.7
77.6
8.5
InstructMol (Cao et al., 2023)
52.6
95.7
9.5
28.3
48.4
20.0
94.5
HIGHT
66.8
85.5
48.2
29.7
53.2
39.1
69.4
Table 2. Results of motif hallucinations on Mot i fHallu 1u.
not include the baselines with "-GS" for tasks other than Moti fa ll we find that incorporating the ID input does not always bring improvements in the experiments.
METHON
Fl (pos)
Fl (neg)
Acc
Yes Ratio
Node-centric Tokenization
InstructMl-GG
95.7
9.5
19.9
94.5
InstructMol-Gi (I. (lLaa22-7b.chat
99.6
18.3
98.7
InstruciMol-GS
97.1
10.6
20.9
94.4
For non-regression-bott based models, including the pretrained models such as KV-PLM (Zeng et al., 2022), GraphCL (You et al., 2020) and GraphMVP (Liu al., 2022), and molecular foundation models that are trained with tremen- dous molecule-centie datasets such as MolT5-based meth- ods (Edwards et al., 2022), Galactica (Taylor et al., 2022), MoMu (Su et al., 2022), MolFM (Luo et al., 2023a), Uni- Mol (Zhou et al., 2023), MolXPT (Liu et al., 2023e), GIT- Mol (Liu et al., 2024c), and BioMedGPT (Luo et al., 2023b). We adopt the results from the previous works Fang et al. (2024); Cao et al. (2023) if applicable.
Hienorehical Tokenization
HIGHT-G
85.5
48.2
39.1
74.7
HIGHT-G (LLama-2-7b-ctat)
55.1
65.2
46.6
49.3
HIGHT-GS
84.5
42.7
35.1
73.1
Ablation variants of HIGHT HIGHT-G G w/o BiPubChem
96.6
12.5
21.6
96.6
HIGHT-GS w/o HiPubCham
98.2
19.4
93.3
5. Experimental Evaluation
We conduct extensive experiments to compare HIGHT with previous node-centric tokenization across 14 real-world tasks, including property prediction, molecular description, and chemical reaction prediction. The details and examples regarding the datasets and tasks involved in the experiments are given in Appendix C. We briefly introduce the setups below and leave the details in Appendix D.
For regression- based LGLMs, we consider LLMs such as ChatGPT (OpenAI, 2022), Llama (Touvron et al., 2023a) as well as instruction tuned LLMs such as Alpaca (Dubois et al., 2023), Baize (Xu et al., 2023), ChatGL.M (Zeng et al., 2023) and Vicuna (Chiang et al., 2023). We also consider parameter-efficient finetuned LLMs using the backbone of llama2 (Touvron al., 2023b) as done by Mol-Instruction (Fang et al., 2024).
5.1. Experimental settings
Architecture. The GNN backbone is a 5-layer GIN (Xu et al., 2019) with hidden dimension of 300. The adapter is single-layer MIP. We consider base L.LMs of vieuna- v-1.3-7B (Chiang et al., 2023) for all the tasks and 1lama-2- 7B-chat (Touvron et al., 2023b) for ablation studies.
5.2. Motif Hallucination
We begin with proof-of-concept study with motif hallu- cination. We mainly compare L.LLMs with node-centric to that with HIGHT tokenization with Mot i iffal lu after stage I instruction tuning. For non-regression -bbased mod- els, we include two state-of-the-art I.GLMs GIMLET (Zhao et al., 2023) and Galactica (Taylor et al., 2022). We do not include the other regression-based models as we found they consistently answered "Yes", making nuanced Fl comparison less informative for them. To avoid the issue of format following, we compare the loss values by feeding the answers of "Yes" and "No" to the corresponding LLM, calculating the language modeling losses, and taking the one from "Yes" and "No" with a lower loss as the answer.
Baselines. Since the focus of this work lies in the tok- enization, our main comparison focuses on between HIGHT and node-centric tokenization. Nevertheless, we also in- clude a series of existing LGLMS based on non-regression LLMs and regression LLMs, provide overview of the performance achieved by HIGHT. We would like to note that there are existing differences in pretraining data and information used between H GHT and those baselines. For details, please refer to Table 7 in the Appendix.
For the node-centric based tokenization, we implement the baseline mainly based on InstructMol (Cao et al., 2023) with VQVAE tokenizer from Mole-BERR (Xia et al., 2023). HIGHT is implemented based on the same architecture with only the tokenizer replaced. We use the suffix "-G" to refer to LGLMs with only 2D graph input and "-GS" to refer to LGLMs with both 2D graph and ID selfies input (Krenn et al., 2019; Fang et al., 2024; Cao et al., 2023). We do
Reduction of hallucination. Due to the class imbalance issue in Mot i fHallu, we first report comprehensive met- ries in Table 1. It can be found that HIGHT maintains great balance for both positive and negative classes compared to baselines. Especially, in terms of macro Fl scores that are
6

---

Hierarchical Graph Tokenization for Molecule-Language Alignment
et al., 2014) via Mean Absolute Error (MAE).
Table 3. Results of molecular property prediction tasks (regression) on QM9. We report the result in MAE. t: few-shot in montott learning (ICL) results from (Fang et al., 2024). A refers to the
The results of molecular property prediction are given in Table 3 and Table 4 for regression and classification, re- spectively. We can find that HIGHT always significantly boosts the performance in both types of tasks. Remark- ably, in CYP450 (Zhao et al., 2023), HIGHT significantly outperforms the state-of-the-an model, demonstrating the
HOMO-LUMO energy gap.
METHOD
HOMO! 1 LUMO Art
1
Alpaca! (Dubois et al., 2023)
322.109
Baize' 2023)
261.543
LLamm2: (Touvron
20236) ICL)
0.6411
0.7510
Vicuna- 138 (Chinng et al. 2023) (S-shot ICT.
0.7135
3/807
1.5407
1,978
Mol-Imttmation (Fang al., 2024)
0.0210
0.0100
0.003
0.0210
advances of LGLM with hierarchical graph tokenization.
InmnacMMal-
0.0111
0.0133
0.0477
0.0130
Interestingly, Llama-2 (Touvron et al., 2023b) can match the
HIGHT-G
0,0078
0.0086
0.0095
0.0086
state-of-the-ar performance in HIV in few-shot setting, while performing significantly worse in other datasets, for which we suspect some data contamination might exist.
averaged across classes, respectively, HIGHT demonstrates significant improvements up to 14%.
Molecular description requires the LGLMS to generate a caption of the molecule. We adopt the widely used bench- mark ChEBI-20 (Edwards et al., 2021) which evaluates the linguistic distances of the generated molecule captions of molecular characteristics such as structure, properties, bi- ological activities etc.. We report the metrics of BLEU (Pap- ineni et al., 2002), ROUGE (Lin, 2004) and Meteor (Baner- jee & Lavie, 2005). The LGLMs are trained using the ChEBI-20 train split, selected according to the best train- ing loss, and evaluated using the test split.
The results of the tokenization-focnt given in Table 2. Following the practice in LVLMs, we present the F1 scores, accuracies, and the ratio that the model answers "Yes" (Li et al., 2023c). Given the imbal- ance of positive and negative samples, we separately report the Fl scores for different classes. It can be found that the LGLMs with node-centric tokenization consistently answer with "Yes" despite the absence of the corresponding func- tional groups. In contrast, HI GHT significantly reduces the worst class hallucination up to 40% in terms of Fl scores, and improves the accuracies up to 30%. The improvements are consistent and significant with both vicunna and Ilama2 LLM backbones.
comparison are
As shown in Table 5, HIGHT consistently brings significant improvements over LGLMs with node-centric tokenization. Nevertheless, compared to the molecular foundation models such as MoT5 (Edwards et al., 2022) pretrained on a signifi- cant amount of molecule-text related corpus, there remains gap for regression regres LGLMs even with HIGHT. The gap calls for future investigations how to incorporate HIGHT into the pretraining of the LGLMs properly.
Ablations with different inputs and LLM backbones. We also conduct simple ablation studies by additionally incorporating the ID sequence inputs with SELFIES (Fang et al., 2024; Cao et al., 2023). Contrary to previous results that additionally feeding the ID sequence always improves the performance of LGLMs, we find that the additional ID sequence may increase the degree of the hallucination. We suspect that it could be caused by the extremely long sequences of the SELFIES (Krenn et al., 2019) that may distract the attention signals of LLMs. Nevertheless, HIGHT still suffers less from the distraction and performs better.
Chemical reaction prediction requires the LGLMs to pre- dict the results of the chemical reaction analysis, which are crucial for Al-aided drug discovery (Fang et al., 2024). Reagent prediction aims to predict the suitable reagents for particular chemical reaction. Forward reaction prediction aims to predict the products of a chemical reaction, given the reactants and the reagents. Retrosynthesis prediction aims to predict the suitable reactants given a target product. The inputs and outputs for chemical reaction related tasks adopt the SELFIES (Krenn et al., 2019) as recommended by (Fang et al., 2024). We report both linguistic distance metrics such as BLEU (Papineni et al., 2002) and Levenshtein (Yujian & Bo, 2007), and molecular similarity measures such as similarity of the molecular fingerprints (Landrum, 2016).
In addition, when without Hip PuPubChem (or with the HIGHT architecture), LGLMs will still suffer the hallucination, due to the low quality of the instruction wning data, demonstrat- ing the necessity of both components of HIGHT.
5.3. Molecular-Centric Benchmarks
Molecular property prediction requires LGLMs to an- swer about particular properties given the molecule. We use datasets BACE, BBBP, HIV, SIDER, ClinTox, MUV, and Tox21 from MoleculeNet and CYP450 from GIM- LET (Zhao et al., 2023) to evaluate the classification perfor- mance with ROC-AUC. We also adopt the regression-basse property prediction datasets from (Fang et al., 2024), where we evaluate several quantum chemistry measures such as HUMO, LUMO, and HUMO-LUMO gap (Ramakrishnan
As shown in Table 6, across all tasks in chemical reaction prediction, LGLMS with HIGHT consistently and signif- icantly improve the performances compared to the node- centric tokenization. Meanwhile, LGLLMs with HIGHT achieve state-of-the-art results in several tasks and metrics, compared to other regression-based I. LGLMs that even incor- porate stronger LLM backbone such as
7

---

Hierarchical Graph Tokenization for Molecule-Language Alignment
Table ROC-AUC Results of molccular property prediction tasks (classification) on MoleculeNet (Wu al., 2017). Evaluation InstructMol and HIGHT adopt the likelihood of the tokens of "Yes" and "No". Most of the instruction tuning datasets are from GIMLET (Zhao et al., 2023). SIDER ClinTox are converted following the MoleculeNet task deseription.
METHOD
BACE
BBBP
HIV t
SIDER
ClinTox
MUV t
Tox21 7,831
CYP450 f
MOLECULES
1,513
2,039
41,127
1,427
1,478
93,087
16,896
TASKS
27
2
17
12
5
KV-PLM (Zeng et al. 2022)
78.5
70.5
71.8
59.8
84.3
61.7
49.2
59.2
GraphCL (You et al., 2020)
75.3
69.7
78.5
60.5
76.0
69.8
73.9
GraphMVP-C (Liu et al., 2022)
81.2
72.4
77.0
60.6
84.5
74.4
77.1
MoleculeSTM-GG (Tiu et al., 2023h)
80.8
70.0
76.9
61.0
92.5
73.4
76.9
MoMu (Su ct al., 2022)
76.7
70.5
75.9
79.9
60.5
57.8
58.0
MolFM (Luo
2023a)
83.9
72.9
79.7
76.0
77.2
Uni-Mol (Zhou al., 2023)
85.7
72.9
80.8
65.9
91.9
82.1
78.1
Galactica- 1.3B (Taylor al., 2022)
57.6
60.4
72.4
54.0
58.9
57.2
60.6
46.9
Galactica-6.78 (Taylor al., 2022)
58.4
53.5
72.2
55.9
78.4
63.9
Galactica-30B (Taylor et al., 2022)
72.7
59.6
75.9
61.3
82.2
68.5
Galactica-1200 120B (Taylor al., 2022)
61.7
66.1
74.5
63.2
82.6
68.9
GIMLET (Zhao et al. 2023)
69.6
59.4
66.2
64.4
61.2
71.3
LLama-2-7b-eee (4-shot) (Touvron al., 20236)
54.2
67.8
46.9
62.0
57.6
LLama-2-1 3b-chat (4-shot) (Touvron al., 2023b)
74.7
52.8
72.4
47.9
57.5
55.6
InstructMol-G
64.3
48.7
50.2
51.0
50.0
50.0
59.0
59.1
HIGHT-G
77.1
61.8
63.3
58.8
55.3
51.1
67.4
80.5
Table 5. Results of molecular description generation task on the test split ChEBI-20.
MODEL
BLEU-21
BLEU-47
ROUGE-It
ROUGE-21
ROUGE-L-
METEORT
MoTS-base (Edwards et al., 2022)
0.540
0.457
0.634
0.485
0.568
0.569
MoMu (MofTS-buse) (Su et al., 2022)
0,549
0.462
0.576
MolFM (MolTS-base) (Luo et al., 2023a)
0.585
0.498
0.653
0.508
0.594
0.607
MolXPT (Liu ct al., 2023c)
0.594
0.505
0.660
0.511
0.597
0.626
GrT-Mol-graph (Liu ul., 2024c)
0.290
0,210
0.540
0.445
0,512
0.491
GFT-MoI-SMILES (Liu et al., 2024c)
0.264
0.176
0.477
0.374
0.451
0.430
GIT-Mol-Graph--SMII
(Liu et 2024c)
0.352
0.263
0.575
0.485
0.560
0.430
Text+Chem T5-augm-bas (Christofidellis
2023)
0.625
0,542
0.682
0,543
0.622
0.648
GPT-3.5-trro (10-shot MolReGPT) (Li al., 2023b)
0.565
0.482
0.623
0.450
0.543
0.585
GPT-4-0314 (10-shot MolReGPT) (Li et al., 2023b)
0.607
0.525
0.634
0.476
0.562
0.610
GPT-3.5-turbo (zero-shot) (Li 2023b)
0.103
0.050
0.261
0.088
0.204
0.161
BioMedGPT-10B (Luo al., 20236)
0.234
0.141
0,386
0.206
0.332
0.308
Mul-Instruction (Fang al., 2024)
0.249
0.171
0.331
0.203
0.289
0.271
InstruciMo--G
0.481
0,381
0.554
0.379
0,488
0.503
HIGHT-G
0.504
0.405
0.570
0.397
0.502
0.524
and additional information of SELFIES.
found that, although HIGHT requires longer training time and relatively higher tunable parameters, the absolute values are not high. Moreover, during inference, LLM latency consumes most of the computation, HIGHT can even reduce the inference latency by generating more concise answers.
5.4. Empirical Analysis
Generalist capabilities. follow the previous practice in training and evaluating generalist models (Liu et al., 2023a) and consider the settings: a) As shown in Fig. 3(a), first train the model with all chemical reaction prediction data by 3 epochs to elicit the format following and the knowledge adaption capabilities of the LGLMs after stage The models are named with "-All"; b) As shown in Fig. 3(b), we train the model with retrosynthesis task data and evaluate the zero-shot transfer performance on forward reaction prediction. Under both settings, we can find that HIGHT boosts the generalist capabilities significantly.
A blation studies. To better understand the effectiveness of distinct components HI GHT, we conduct ablation stud- ies that train InstructMol (Cao al., 2023) with the lapla- cian positional encodings with HiP PuPuhcmm, given in Fig. 3(c). We can find that, merely incorporating positional encoding or hierarchical instruction tuning is not sufficient to achieve the same performance as HIGHT. On the contrary, without a proper architecture design as HIGHT, LGLMs with previous node-centric tokenization with HiPubChem will confuse LLMs and even lead to degenerated down- stream task performances. In addition, we also compare LGLMs with llama2 backbone. As shown in Fig. 3(a).
Computation overhead. In Appendix 1. we report the computation overhead of pretraining and inference as well as tunable parameters of HIGHT and InstructMol. It can
8

---

Hierarchical Graph Tokenization for Molccule-Language Alignment
Table Results chemical reaction tasks. These tasks encompass reagent prediction. forward reaction prediction, and retrosynthesis. ft few-shot ICL results from (Fang et al., 2024). *; use task task-specific instruction data to finetune.
MODEL
ExACtt
BLEUT
LEVENSHTEIN,
RDK FTST MACCS FTST
MORGAN FTS"
VALIDITY|
Reagen Prodiction
Alpaca (Dubois 2023)
0.000
0.026
29.037
0.029
0.016
0.001
0.186
Baize' (Xu et al., 2023)
0.000
0.051
30.628
0.022
0.018
0.004
0.099
ChatGLM (Zeng al., 2023)
0.000
0.019
29.169
0.017
0.006
0.002
0.074
Llama' (Touvron et al., 2023a)
0.000
0.003
28,040
0.037
0.001
0.001
0.001
Vicuna' (Chiang al., 2023)
0.000
0.010
27.948
0.038
0.002
0.001
0,007
Mol-Instruction (Fang et al., 2024)
0.044
0.224
23.167
0.237
0,364
0.213
1.000
Llama-7h (Touvron LoRA)
0.136
0.294
0.106
InstructMol-G
0.031
0.429
31.447
0.389
0.249
0.220
1.000
HIGHT-G
0.050
0.462
28.970
0.441
0.314
0.275
1.000
Forward Reaction Prediction
Alpucal (Dubois al., 2023)
0.065
41.989
0.004
0.024
0.008
0.138
Buize' (Xu et al., 2023)
0.000
0.044
41,500
0.004
0.025
0.009
0.097
CharGLM' (Zeng
2023)
0.100
Llama' ('ouvvon et al., 2023a)
0.001
Vicunal (Chiung al., 2023)
0.007
0.059
Mol- Iostrussttttionn (Fang et al., 2024)
0.045
0.654
27.262
0.313
0.509
0.262
1.000
Llama-7b' (Touvron et al., 202RA)(LoA))
0.012
0.804
29.947
0.499
0.649
0.407
1.000
InstructMol-G
0.853
24.790
0.512
0.362
0.303
0.993
HIGHT-G
0.037
0.869
23.759
0.590
0.394
0.340
0.993
Reimoyythhsis
Alpaca' (Dubois et al., 2023)
0.000
0.063
46.915
0.005
0.023
0.007
0.160
Baize (Xu et al., 2023)
0.000
0.095
44.714
0.025
0.050
0.023
0.112
ChatGLM (Zeng et al., 2023)
0.000
0.117
48.365
0.056
0.075
0.043
0.046
Llama! (Touvron et al., 2023a)
0.000
0.036
46.844
0.018
0.029
0.017
0.010
Vicuna" (Chiung et 2023)
0.000
0.057
46.877
0.025
0.030
0.021
0.017
Mol-Instruction (Fang et al., 2024)
0.705
31,227
0,283
0.487
0,230
Llamu-7b' (Touvron
LoRAA)(LoRA)
0.000
0.283
53.510
0.136
0.294
0.106
1.000
InstrucMoo-G
0.001
0.835
31,359
0.447
0.277
0.241
0.996
HIGHT-G
0.008
0.863
28.912
0.564
0.340
0.309
1.000
InstruchMol-lamas
HIGHTillmaz
35
Instruct#fai-G
InstructMoi-G
InstructMol-Ail
50
+HilPubChem
50
30-
30
10
10
10-
Reagent Pred
Forward Pred
Retrosyithesia
BLEU
MACCS
MORGAN
Reagent Pred
Forward Pred
Retrosymthesis
(a) Different training settings
(b) Zero-shot transfer
(c) Ablation variants of HIGHT
Figure 3. Ablation studies.
HIGHT still significantly boosts the performance. More ablation studies are provided in Appendix E.
HIGHT represents significant step forward in advancing graph comprehension capability of LLMs, and highlighting paths for future research in this direction.
6. Conclusions
Meanwhile, incorporating 3D information into the graph- language alignment is also promising future direction, es- pecially for broader scientific tasks such as single-cell mod- eling and understanding. For example, built upon HIGHT, one could design new 3D tokenizer to accommodate 3D properties of motifs, scale up 3D data to include amino acids in proteins and certain recurrent structures in RNA sequences, incorporate 3D positional encoding, and curate instruction tuning data with 3D descriptive captions.
This paper presents HI GIGHT, a novel hierarchical graph tok- enization technique. By incorporating the hierarchical graph information, HIGHT improves the molccule-kanguage align- ment performance, reducing hallucinations and boosting accuracy in molecular tasks. Nevertheless, the current focus on molecular graphs requires further verification for wider applicability to other forms of graph data, such as those originating from social networks. Despite the limitation,

---

Hierarchical Graph Tokenization for Molecule-Language Alignment
Acknowledgments
Chen, R., Zhao, T., Jaiswal, A., Shah, N., and Wang, Z. Llaga: Large language and graph assistant. arXiv preprint, arXiv:2402.081700 2024a. (Cited on page 3)
We thank the reviewers for their valuable comments. JC was supported by RGC Young Collaborative Research Grant No. C2005-24Y.
Chen, Y., Zhang, Y., Bian, Y., Yang. H., Ma, K., Xie, B., Liu, T., Han, B., and Cheng, J. Learning causally invariant representations for out-of-distribution generalization on graphs. In Advances in Neural Information Processing Systems, 2022. (Cited on page 17)
Impact Statement
This paper mainly focuses on how to best represent graph information for LLMs to better understand the graphs. We demonstrate the effectiveness of our method on molecule- centric tasks, which could facilitate the broader use of LLMs for tasks like Al-aided drug discovery and human-machine interactions in biomedicine. Besides, this paper does not raise any ethical concerns. This study does not involve any human subjects, practices, to data set releases, po- tentially harmful insights, methodologies and applications, potential conflicts of interest and sponsorship, discrimina- tion/bias/fairness concerns, privacy and security issues, legal compliance, and research integrity issues.
Chen, Y., Bian, Y., Zhou, K., Xie, B., Han, B., and Cheng, J. Does invariant graph learning via environment augmenta- tion learn invariance? In Thirty-seventh Conference on Neural Information Processing Systems, 2023. (Cited on page 17)
Chen, Y., Bian, Han, B., and Cheng, J. How interpretable are interpretable graph neural networks? In International Conference on Machine Learning. 2024b. (Cited on page 17)
Chiang, W.-I.., Li, 7.,, Lin, Z.,, Sheng, Y., Wu, Z.,, Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., Stoica, I., and Xing, E. P. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. URL https: :ptps/// / //mmyss....m/ org/blog/ 2023 03-30 300-viuua/.. (Cited on pages 6, 7, 9, 22, 26 and 27)
References
Banerjee, S. and Lavie, A. METEOR: An automatic met- ric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Work- shop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65-72, 2005. (Cited on pages and 24)
Christofidellis, D., Giannone, G., Born, J., Winther, O., Laino, T., and Manica, M. Unifying molecular and textual representations via multi-task language modelling. In International Conference on Machine Learning, volume 202, pp. 6140-6157, 2023, (Cited on pages 3, 8 and 17)
Beltagy, I., Lo, K., and Cohan, A. SciBERT: A pretrained language model for scientific text. In Conference on Empirical Methods in Natural Language Processing, pp- 3615-3620, 2019, (Cited pages 3 and 24)
Degen, J., Wegscheid-GGrllane C., Zaliani, A., and Rarey, M. On the art of compiling and using 'drug-like' chemical fragment spaces. ChemMealChem, 3:1503-1507, 2008
Bengio, Y., Léonard, N., and Courville, A. C. Estimating or propagating gradients through stochastic neurons for con- ditional computation. arXiv preprint, arXiv:1308.3432. 2013. (Cited on page 4)
(Cited on page 4)
Doreian, P. and Woodard, K. L. Defining and locating cores and boundaries of social networks. Social Networks, 16 (4):267-293, 1994. (Cited on page 17)
Bohacek, R. S., McMartin, C., and Guida, W. C. The art and practice of structure-based drug design: A molecular modeling perspective. Medicinal Research Reviews, 16 (1):3-50, 1996. (Cited on pages 1, 4 and 20)
Dubois, Y., Li, X., Taori, R., Zhang, T., Guirajani, I., Ba, J., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacafarm: A simulation framework for methods that learn from hu- man feedback, 2023. (Cited on pages 6, 7, 9, 26 and 27)
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz. E., Kamar, E., Lee, P., Lec, Y. T., Li, Y., Lundberg, S. M., Nori, H., Palangi, H., Ribeiro, M. T., and Zhang, Y. Sparks of artificial general intelli- gence: Early experiments with GPT-4. arXiv preprint, inXiv:2303.12713 2023. (Cited on page 1)
Durant, J. L., Leland, B. A., Henry, D. R., and Nourse, J. G. Reoptimization of mdl keys for use in drug discovery. Journal of chemical information and computer sciences, 42 6:1273 80, 2002. (Cited on page 24)
Cao, H., Liu, Z., Lu. X., Yao, Y., and Li, Y. Instructmol: Multi-modal integration for building a versatile and reli- able molecular assistant in drug discovery, 2023. (Cited
Dwivedi, V. P., Joshi, C. K., Luu, A. T., Laurent, T., Bengio, Y., and Bresson, X. Benchharkkng graph neural networks. arXiv preprint 2rXiv:2003.00982 2020. (Cited on page 5)
on pages 1, 3, 4, 5, 6, 7, 8 and 17)
10

---

Hierarchical Graph Tokenization for Molecule-Language Alignment
Edwards, C., Zhai, C., and Ji, H. Tex(2mol: Cross-modal molecule retrieval with natural language queries. In Pro- ceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 595-6077 2021. (Cited on pages 4, 7, 21 and 24)
Jin, B., Liu, G., Han, C., Jiang, M., H., and Han, J. Large language models on graphs: A comprehensive survey. arXiv preprint, arXiv:2312.027833, 2023. (Cited on pages 1 and 2)
Kim, S., Chen, J., Cheng, T., Gindulyte, A., He, J., He, S., Li. Q.. Shoemaker, B. A., Thiessen, P. A., Yu, B., Zaslavsky, L.,, Zhang, J., and Bolton, E. E. PubChem 2023 update. Nucleic Acids Research, 5 51(DI):D1373- D1380, 10 2022. (Cited on pages and 17)
Edwards, C., Lai, T. M., Ros, K., Honke, G., Cho, K., and Ji, H. Translation between molecules and natural lan- guage. In Conference on Empirical Methods in Natural Language Processing, pp. 375-413, 2022. (Cited on pages 3, 6, 7, 8 and 17)
Krenn, M., Hase, F., Nigam, A., Friederich, P., and Aspuru- Guzik, A. Self-referencing embedded strings (selfies): A 100% robust molecular string representation. Machine Learning: Science and Technology. 1, 2019. (Cited on pages 6, and 20)
Fan, W., Wang. S., Huang, J.. Chen, Z., Song, Y., Tang, W., Mao, H., Liu, H., Liu, X., Yin, D., and Li. Q. Graph machine learning in the era of large language models (Ilms.. arXiv preprint, N.Kv:22404.14222, 2024. (Cited on pages and 2)
Landrum, G. Rdkit: Open-source cheminformatics soft- ware, 2016. URL Releg//Reeeeeeeee 6_09_4. mekit/reeeasess htt////t////// (Cited on pages 4, 7, and 24)
com/ com/rákit/
Fang, Y., Liang, X., Zhang, N., Liu, K., Huang, R., Chen, Z. Fan, X., and Chen, H. Mol-instructions: A large- scale biomolecular instruction dataset for large language models. In International Conference on Learning Repre- sentations, 2024. (Cited pages 5, 6, 7, 8, 9, 20, 24, 26 and 27)
Li, J., Li, D., Savarese, S., and Hoi, S. BLIP-2: Bootstrap- ping language- image pre-training with frozen image en- coders and large language models. In International Con- ference on Machine Learning, pp. 19730-19742, 2023a. (Cited pages 1 and 3)
Fatemi, B., Halcrow, J., and Perozzi, B. Talk like graph: Encoding graphs for large language models. In Inter- national Conference on Learning Representations, 2024. (Cited on page 3)
Li, J., Liu, Y., Fan, W., Wei, X.-Y., Liu, H., Tang, J., and Li, Q. Empowering molecule discovery for molecule- caption translation with large language models: A chatgpt perspective. arXiv preprint arXiv:2306.06615, 2023b. (Cited on page 8)
Hastings, J., Owen, G., Dekker, A., Ennis, M., Kale, N., Muthukrishnan, V., Turner, S., Swainston, N., Mendes, P., and Steinbeck, C. Chebi in 2016: Improved services and expanding collection of metabolites. Nucleic Acids Research, 44:D1214- -11229,,1111, D1219, 2015. (Cited pages 18 and 21) Hu, C. and Li, H. Exploring hierarchical molecular graph representation in multimodal llms. arXiv preprint, arXiv:2411 1,04008 2024. (Cited on page 3)
Li, S., Liu, Z., Luo, Y., Wang. X., He, X., Kawaguchi, K., Chua, T.S.,, and Tian, Q. Towards 3d molecule- text interpretation in language models. In International Conference on Learning Representations, 2024. (Cited on pages 3 and 5)
Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., and Wen, J. Evaluating object hallucination in large vision-language models. arXiv preprint, arXiv:2305. 10355,, 2023c. (Cited on pages 7, 21, 22 and 24)
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. (Cited on page 5)
Li, Y., Li, Z., Wang, P., Li, J.. Sun, X., Cheng, H., and Yu, J. X. A survey of graph meets large language model: Progress and future directions. arXiv preprint, MrXiv:2311.1239) 2023d. (Cited on pages and 2)
Inae, F.,, Liu, G., and Jiang, M. Motif-aware attribute masking for molecular graph pre-training. In NewrIPS 2023 Workshop: New Frontiers in Graph Learning, 2023. (Cited on pages 2 and 3)
Liang, Y., Zhang, R., Zhang, I., and Xie, P. Drugchat: To- wards enabling chatgpt-like capabilities on drug molecule graphs. arXiv preprint, arXiv:2309.03907 2023. (Cited on pages 3 and 4)
Irwin, R., Dimitriadis, S., He, J., and Bjerrum, E. J. Chem- former: a pre-trained transformer for computational chemistry. Machine Learning Science Technology, 3(1): 15022, 2022. (Cited on page 3)
Lin, C.-Y. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, 2004. (Cited on pages and 24)

---

Hierarchical Graph Tokenization for Molecule-Language Alignment
Liu, C., Chen, Y., Liu, T., Gong, M., Cheng, J., Han, B., and Zhang, K. Discovery of the hidden world with large language models. In Advances in Neural Information Processing Systems, pp. 102307-102365, 2024a. (Cited on page 17)
Luo, Y., Yang, K., Hong, M., Liu, X. Y., and Nie, Z. Molfm: A multimodal molecular foundation model, 2023a. (Cited on pages 3, 6, 8 and 17)
Luo, Y., Zhang, J., Fan, S., Yang. K., Wu, Y., Qiao, M.. and Nie, Z. Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine, 2023b. (Cited on pages 3, 6 and 8)
Liu, H., Li, C., Wu, Q. and Lee, Y. J. Visual instruction tuning. In NeurlPS, 2023a. (Cited on pages 1, and 8)
Luong, K.-D. and Singh, A. Fragment-based pretraining and finetuning on molecular graphs. In Thirty- Conference Neural Information Processing Systems, 2023. (Cited on pages 2 and 3)
Liu, H., Feng. J., Kong. L., Liang, N., Tao, D., Chen, Y., and Zhang, M. One for all: Towards training one graph model for all classification tasks. In International Conference on Learning Representations, 2024b. (Cited on page 3)
Mao, H., Chen, Z., Tang, W., Zhao, J., Ma, Y., Zhao, T., Shah, N., Galkin, M., and Tang. J. Graph foundation models. arXiv preprint, arXiv:2402.02216, 2024. (Cited on pages 1 and 2)
Liu, P., Ren, Y., Tao, J., and Ren, Z. Git-mol: A multi-modal large language model for molecular science with graph, image, and text. Computers in Biology and Medicine, pp. 108073, 2024c. (Cited on pages 3, 6 and 8)
Mendez, D., Gaulton, A., Bento, A. P., Chambers, J., Veij, M. D., Felix, E., Magariños, M. P., Mosquera, J. F., Mutowo-Meullenet, P., Nowotka, M., Gordillo- Marañón, M., Hunter, F. M. I., Junco, L., Mugumbate, G., Rodríguez-Lóóez. M., Atkinson, F., Bosc, N., Radoux, C. J., Cegura-Cabrera, A., Hersey, A., and Leach, A. R. Chembl: towards direct deposition of bioassay data. Nu- cleic Acids Research,
Liu, S., Wang, H., Liu, W., Lasenby, J., Guo, H., and Tang, J. Pre-training molecular graph representation with 3d geometry. In International Conference on Learning Rep- resentations, 2022. (Cited on pages 3, 6, and 17)
Liu, S., Nie, W., Wang, C., Lu, J., Qiao, Z., Liu, L.,, Tang, J., Xiao, C., and Anandkumar, A. Multi-modal molecule structure-text model for text-based retrieval and editing- Nature Machine Intelligence, 5(12):1447-1457, 2023b. (Cited on pages 3, 5, 8 and 17)
(ssttabase-Issue):D999900 2019. (Cited on page 5)
Miao, S., Liu, M., and Li, P. Interpretable and generalizable graph learning via stochastic attention mechanism. Inter- national Conference on Machine Learning, 2022. (Cited on page 17)
Liu, Z., Li, S., Luo, Y., Fei, H., Cao, Y., Kawaguchi, K., Wang, X., and Chua, T.-S. MolCA: Molecular graph- language modeling with cross- projector and uni- modal adapter. Conference on Empirical Methods in Natural Language Processing, 2023c. (Cited on pages 3, and 17)
Milo, R., Shen-Orr, S. S., Itzkovitz, S., Kashtan, N., Chklovskii, D. B., and Alon, U. Network motifs: simple building blocks of complex networks. Science, 298 5594: 824-7, 2002. (Cited on page 1)
Liu, Z., Shi, Y., Zhang, A., Zhang, E., Kawaguchi, K., Wang, X., and Chua, T.-S. Rethinking tokenizer and decoder in masked graph modeling for molecules. In Advances in Neural Information Processing Systems, 2023d. (Cited on page 4)
OpenAL. Chatgpt. https: / http:/caat.tt.. http:/hhaaa..peai.coooooo. chat / 2022. (Cited on pages and 6)
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-. Bleu: a method for automatic evaluation of machine translation. In Annuul Meeting of the Association for Computational Linguistics, 2002. (Cited on pages 7 and 24)
Liu, Z., Zhang, W., Xia, Y., Wu, L., Xie, S., Qin, T., Zhang, M., and Liu, T.-Y. MolXPT: Wrapping molecules with
text for generative pre-training. In Annual Meeting of the Association for Computational Linguistics, pp. 1606- 1616. Association for Computational Linguistics, 2023e.
Park, J., Bae, M., Ko, D., and Kim, H. J. LLamo: Large lan- guage model-based molecular graph assistant. In Annual Conference on Neural Information Processing Systems, 2024. (Cited on page 3)
(Cited on pages 3, 6, 8 and 17)
Lu, Y., Bartolo, M., Moore, A., Riedel, S., and Stene- torp, P. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics, pp. 5086-8098, 2022.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z.,, Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An imperative style, high- high-performance deep learning library. In Advances in
(Cited on page 4)
12

---

Hierarchical Graph Tokenization for Molecule-Language Alignment
Neural Information Processing Systems, pp. 8024-8035, 2019. (Cited on page 24)
Tian, Y., Song, H., Wang. Z.., Wang, H., Hu, Z., Wang, F, Chawla, N. V., and Xu, P. Graph neural prompting with large language models. In Thirty-Eighth AAAI Confer- ence on Artificial Intelligence, pp. 19080-19088, 2024. (Cited on page 3)
Radford, A., Wu, J., Child, R.,. Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners, 2019. (Cited on page 1)
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models, arXiv preprint, (rXiv:2302.1397/ 2023a. (Cited on pages 3, 6, 9 and 26)
Ramakrishnan, R., Dral, O., Dral, P. O., Rupp, M., and von Lilienfeld, O. A. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1, 2014. (Cited on page 7)
Ribeiro, P., Paredes, P., Silva, M. E. P., Aparicio, D., and Silva, F. survey on subgraph counting: Concepts, algo- rithms, and applications network motif's and graphlets. ACM Computing Survey, 54(2), 2021. (Cited on page 17)
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhos- ale, S., Bikel, D., Blecher, L.., Canton-Ferrer, C.,, Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J.,, Fu, W., Fuller, B., Gao, C.,, Goswami, V., Goyal, N., Hartshor, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I. Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kam- badur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint, MxXiv:2307.09288 2023b. (Cited on pages 6, 7, 8 and 27)
Rong, Y., Bian, Y., Xu, T., Xie, W., WEI, Y., Huang, W., and Huang. J. Self-supervised graph transformer on large- scale molecular data. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp- 12559-12571. Curran Associates, Inc., 2020. (Cited on page 3)
Schneider, N., Sayle, R. A., and Landrum, G. A. Get your atoms in order an open-source implementation of a novel and robust molecular canonicalization algorithm. Journal of chemical information and modeling, 55 10: 2111-20, 2015. (Cited on page 24)
Srinivas, S. S. and Runkana, V. Crossing new frontiers: wowwleegg--ugmeeteed large language model prompting for zero-shot text- novo molecule design. arXiv preprint, arXiv:2408.1 1866, 2024. (Cited on page 3)
van den Oord, A., Vinyals, O., and Kavukcuoglu, K. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pp. 6306-6315, 2017. (Cited on page 4)
Sterling. T. and Irwin, J. Zine 15 1551is ligand discovery for ev- eryone. Journal of Chemical Information and Modeling, 53(11):232--2337 2015. (Cited on pages and 4)
Wang, Q., Lin, Y., Chen, Y., Schmidt, L., Han, B., and Zhang, T. Do CLIP models always generalize better than imagenet models? In Annual Conference on Neural Information Processing Systems, 2024. (Cited on page 17)
Su, B., Du, D., Yang, Z.Q Zhou, Y., Li, J., Rao, A., Sun, H., Lu, Z., and rong Wen, A molecular multimodal foundation model associating molecule graphs with nat- ural language. arXiv preprint arXiv:2209.05548) 2022. (Cited on pages 3, 6, 8 and 17)
Wang, Y., Wang. J., Cao, Z., and Farimani, A. B. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence, 4(3):279-287, 2022. (Cited on page 3)
Tang, J., Yang, Y., Wei, W., Shi, L.,, Su, I.,, Cheng, S., Yin, D., and Huang, C. Graphgpt: Graph instruction tuning for large language models, arXiv preprint, 1xXi:22310013023 2023. (Cited on pages 2 and 3)
Wei, L.,, Gao, J., Zhao H., and Yao, Q. Towards versa- tile graph learning approach: from the perspective of large language models. arXiv preprint, 1rXiv:2402.11641, 2024. (Cited on pages 1 and 2)
Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A. S., Saravia, E., Poulton, A., Kerkez, V., and Stojnic, R. Galactica: A large language model for science. arXiv preprint, arXiv:221 11022221109008 1.09085, 2022. (Cited on pages 3, 6, and 17)
Weininger, D. Smiles, chemical language and information system. 1. introduction to methodology and encoding rules. Journal of Chemical Information and Computer Sciences, 28:31-36 1988. (Cited on page 3)
13

---

Hierarchical Graph Tokenization for Molecule-Language Alignment
Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Ge- niesse, C., Pappu, A. S., Leswing, K., and Pande, V. S. Moleculenet: A benchmark for molecular machine learn- ing. arXiv preprint arXiv:1703.000:: 2017. (Cited on pages 5, 8, 20 and 24)
You, Y., Chen, T., Sui, Y., Chen, T., Wang, Z., and Shen, Y. Graph contrastive learning with augmentations. In Advances in Neural Information Processing Systems, pp. 5812-5823, 2020. (Cited on pages 6, 8 and 17)
Yu, J., Xu, T., Rong, Y., Bian, Y., Huang. J., and He, R. Graph information bottleneck for subgraph recognition. In International Conference on Learning Representations, 2021. (Cited on page 17)
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., Zheng. R., Fan, X. Wang, X., Xiong, L.,, Zhou, Y., Wang, W., Jiang, C., Zou, Y., Liu, X., Yin, Z., Dou, S., Weng, R., Cheng, W., Zhang, Q. Qin, W., Zheng, Y., Qiu, X., Huang, X., and Gui, T. The rise and potential of large language model based agents: A survey, 2023. (Cited on page 3)
Yujian, L. and Bo, L. A normalized levenshtein distance metric. IEEE Transactions Puttern Analysis and Ma- chine Intelligence, 29(6):1091-1095 2007. (Cited on pages 7 and 24)
Xia. J., Zhao, C., Hu, B., Gao, Z., Tan. C., Liu, Y., Li, S., and Li, S. Z. Mole-BERT: Rethinking pre-training graph neural networks for molecules. In International Conference on Learning Representations, 2023. (Cited
Zang, X., Zhao, X., and Tang, B. Hierarchical molecular graph self- self-supervised learning for property prediction. Communications Chemistry, 6(1):34, 2023. (Cited on pages 2, 3, 4, 5 and 22)
on pages 4, 6, 17 and 22)
Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., Xia, X., Tam, W. L., Ma, Z., Xue, Y., Zhai, J., Chen, W., Liu, Z.,, Zhang, P., Dong, Y., and Tang, J. GLM-130b: An open bilingual pre-trained model. In International Conference on Learning Representations, 2023. (Cited on pages 9 and 26)
Xu, C., Guo, D., Duan, N., and McAuley, J. Baize: An open-source chat model with parameter-efficient tuning on self-chat data, arXiv preprint, arXiv:2304.01196 2023. (Cited on pages 6, 7, 9, 26 and 27)
Xu, J., Chen, Y., Dong, X., Lan, M., Huang, T., Bian, Q- Cheng, J., and Ke, Y. Brainood: Out-of-distribution generalizable brain network analysis. arXiv preprint, wxXiv25502.01688, 2025. (Cited on page 17)
Zeng, Z., Yao, Y., Liu, Z., and Sun, M. A deep-Icarning system bridging molecule structure and biomedical text with comprehension comparable to human professionals. Nature communications, 13682)), 2022. (Cited on pages 3, 6, 8 and 17)
Xu, K., Hu, W., Leskovec, J., and Jegelka, S. How powerful are graph neural networks? In International Conference on Learning Representations, 2019. (Cited on pages 1,
Zhang. J.. Bian, Y., Chen, Y., and Yao, Q. Unimot: Uni- fied molecule-text language model with discrete token representation. arXiv preprini arXiv:2408. 00863, 2024a. (Cited on page 3)
4, and 22)
Yao, T., Chen, Y., Chen, Z., Hu, K., Shen, Z., and Zhang, K. Empowering graph invariance learning with deep spurious infomax. In Forty-first International Conference on Machine Learning, 2024. (Cited on page 17)
Zhang, J., Huang, J., Jin, S., and Lu, S. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024b. (Cited on page 1)
Yao, T. Chen, Y., Hu, K., Liu, T., Zhang, K., and Shen, Z. Learning graph invariance by harnessing spuriosity. In International Conference on Learning Representations, 2025. (Cited on page 17)
Zhang, Z., Liu, Q., Wang, H., Lu, C., and Lee, C.-K. Motif- based graph self-supervised learning for molecular prop- erty prediction. In Advances in Neural Information Pro- cessing Systems, pp. 15870-15882, 2021. (Cited on pages 2, 3 and 4)
Ying, C., Cai, T., Luo, S., Zheng, S., Ke, G., He, D., Shen, Y., and Liu, T.-Y. Do transformers really perform badly for graph representation? In Advances in Neural Information Processing Systems, 2021. (Cited on page 5)
Zhang, Z., Liu, Q., Hu, Q., and Lee, C.-K. Hierarchical graph transformer with adaptive node sampling. arXiv preprint arXiv:2210.039300 2022. (Cited on page 17)
Ying, Z., You, J., Morris, C., Ren, X., Hamilton, W., and Leskovec, J. Hicrarchical graph representation learning with differentiable pooling. In Advances in Neural In- formation Processing Systems, 2018 (Cited on pages 3 and 17)
Zhao, H., Liu, S., Ma, C.,, Xu, H., Fu, Deng, Z.-H., Kong, L., and Liu, Q. GIMLET: A unified graph-text model for instruction- molecule zero-shot learning. In Neural Information Processing Systems, 2023. (Cited on pages
1,3, 6, 7, 8, 17 and 19)
14

---

Hierarchical Graph Tokenization for Molecule-Language Alignment
Zhou, G., Gao, Z., Ding, Q. Zheng, H., Xu, H., Wei, Z Zhang, I.., and Ke, G. Uni-mol: A universal 3d molecu- lar representation learning framework. In The Eleventh International Conference on Learning Representations, 2023. (Cited on pages 3, 6, 8 and 17)
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023. (Cited on page 3)
15

---

Hierarchical Graph Tokenization for Molecule-Language Alignment
Appendix of HIGHT
Contents
More Future Works
17
Comparison between other LGLMs
17
Details of Instruction Tuning Datasets
17
C.I Details of the PubChem Dataset
17
C.2 Details of HiPubChem Dataset
18
C.3 Details of Property Prediction Dataset
20
C.4 Details of Reaction Prediction Dataset
20
C,5 Details of Molecular Description Dataset
21
C.6 Details of Mot i fHallu Dataset
Details of Experiments
22
More Ablation Studies
24
E.1 Computation overhead
24
E.2 Ablation studies with different setups of the tokenizers
24
16

---

Hierarchical Graph Tokenization for Molecule-Language Alignment
A. More Future Works
Built upon HIGHT, there are several promising future directions. For example, one could extend this study to more types of graphs, such as social networks and knowledge graphs, by exploring the crucial substructures therein:
Indeed, motifs generically exist in other types of graphs and are crucial for variety of tasks (Ribeiro et al., 2021). For example, cliques can define boundaries between groups of people in social networks (Doreian & Woodard, 1994). The idea of HIGHT could be seamlessly applied to other graphs where we have some prior knowledge about critical motifs.
Meanwhile, when we do not have prior knowledge about the motifs, the GNNs intrinsically model the hierarchical nature of graphs in different orders (Ying et al., 2018) and thus can be integrated into LGLMs to learn the hierarchical graph information. A similar idea has been verified successful in graph transformers (Zhang al., 2022).
Furthermore, one could also adopt interpretable GNNs to identify the critical subgraphs for the task (Yu et al., 2021; Miao al., 2022; et al., 2022; 2023; Yao et al., 2024; Liu et al., 2024a; Yao et al., 2025; Xu et al., 2025). It is also interesting to further investigate the hallucinations caused by the spurious correlations during the alignment (Wang et al., 2024).
Chen et al., 2024b) that capture the underlying causal information about the underlying tasks (Chen
B. Comparison between other LGLMs
Table Comparison between other LGLMs in terms of the backbone, instruction tuning, downstream usage for Molecular Property prediction, and capable tasks. It can be found that HIGHT is capable of various tasks, given limited pre-training data and information. Note that compared to the instruction tuning data for other LGLMs, such as KV-PLM (Zeng et al., 2022), which consists of papers with detailed information about molecules, the text descriptions in HIGHT contain relatively simple sentences.
Model
Rackhone
Information
Instraction Tuning
Downstream
Molltt/lss (Cao et al., 2023)
GNN+L.luma
2D Graph+ Text
HiPuhChem- 295k
LoRA
GraphCL (You et al., 2020)
GNN GNN
2D Graph
Downstream training data
Fineturing
GraphMPP (Liu et al., 2022)
graph
Finetuning
MoleculeSTM (Kin al,, 2023)
GNNASciBERT
2D Graph+3ID Geaph+Text
PubChems MubChemSTMM-22 TM-280K
Finetuning 6
KV-PLM (Zeng al., 2022)
BERT
ID SMILES+Test
$2ore acudemic papers
Finetuning 5
MolTS (Fdwaads al. 2022)
TS
ID SMIL Text
C4-7INC(100M)
Finetuning
Text=Chem T5 (Christoofdeellss et al., 2023)
TS
ID SMILES+Text
Multi 33.5
Fineturing
5
MoMu al. 2022)
GNN+BERT
Graph+Tes
Graph-Docuee 15in.15.6K
Finetuning
MolFM (Luo et al., 2023a)
GNN+BERT
Knowledge Graph++D+3D Graph+Text
KG- 4G-155-S22RC-37M
Finetuning
Uni-Mol (Zhou et al., 2023)
Transformer
3D Graph
malecule-20MM -0rculee20MM--oteiinn...
Finetuning
Golactica (Taylor al., 2022)
GPT
ID SEILES/Test
Decument-59M<chhmee prompts-2.5M
zero-shot 12
MolXPT (Liu et al., 2023e)
GPT2
ID SMILES+
Mixed text-68M
Finetuning 2
GIMLET (Zhao al., 2023)
GNN+T5
Graph Text
ChemBI-733K
zero-shoe
C. Details of Instruction Tuning Datasets
We provide a summary of the datasets for instruction tuning and evaluation in this paper as in Table Meanwhile, we also list the data sources and the corresponding licenses of the sources for each task and dataset. Then, will elaborate more on the details of the datasets in the following subsections.
C.1. Details of the PubChem Dataset
PubChem² is one of the largest public molecule database (Kim al., 2022), and has been widely adopted by the alignment training of LGLMs (Liu et al., 2023c;b; Cao et al., 2023). Our construction of PubChem predominantly follows Liu et al. (2023b). We will briefly describe the main steps and interested readers may refer the details to (Liu et al., 2023b):
We curate the data from PubChem using the official API and set the data cutoff date as 12 Jan. 2024. It downloads both the molecular structure (e.g., SMILES, 2D molecular graphs) in SDF format, and the text descriptions.
Then, we will filter out molecules that do not have descriptions or can not match via the PubChem ID. In the descriptions, the molecule names are replaced with "This molecule". in order to facilitate LLMs understand the instructions.
hhttps://pubchen nebi. . m1mm www.nihh... gov
17

---

Hierarchical Graph Tokenization for Molecule-Language Alignment
Table 8 Summary of datasets involved in our paper.
Datasets
Train
Test
Content
PubChem
295,228
N/A
Molecules and the associated descriptions from PubChem.
HiFubChem
295,228
N/A
Molecules and the associated descriptions from PubChem and about functional groups in the molecule.
MoleculeNet-HII
32,901
4,113
Question answering about the ability of the molecule to inhibit HIV replication.
MoleculeNet-BACE
1,210
152
Question answering about the ability of the molecule to bind the BACE1 protein
MoleculeNet-BBBB
1,631
204
Question answering about the ability of the molecule to diffuse across the brain blood barrier.
MoleculeNet-SIDER
1,141
143
Question answering about the ability of the side effects. Question answering about the toxicology.
MoleculeNeet-llnnTox
1,188
148
MoleculeNec-MUU
74,469
9,309
Question answering about PubChem bioAssay
MoleculeNet-Tox21
6,877
860
Question answering about Toxicology in the 21st century
CYP45-
13,516
1,690
Question answering about CYP PubChem BioAssay CYP 1A2, 2C9, 2C19, 2D6, 3A4 inhibition.
Property Prediction (Regres-
360,113
1,987
Question answering about the quantum mechanics proper- ties of the molecule.
sion)
Forward Reaction Prediction
124,384
1,000
Question answering about the products of chemical re- action, given specific reactants and reagents.
Reagent Prediction
124,384
1,000
Question answering about suitable catalysts, solvents, or ancillary substances required for specific chemical reac- tion.
Retrosynthesis Prediction
128,684
1,000
Question answering about the reactants and reagents of chemical reaction, given specific products.
ChEBI-20
26,407
3,300
Molecules and the associated Chemical Entities of Biolog- ical Interest (ChEBI) (Hastings al., 2015) annotations. Question answering about existing functional groups in the molecule.
MotifHalln
N/A
23,924
Finally, the curation generates 295k molecule-text pairs that we term as PubChem- 295k. PubChem-295k will be mainly used for the stage alignment training.
C.2. Details HiPubChem Dataset
HiPubChem augments the molecular instruction tuning dataset with captions the functional groups. We consider both the positive and negative appearances of motifs when augmenting the instructions. For the positive case, we directly append the caption of all functional groups detected with RDKit:
This molecule has <4> <functional group name> groups.
For the negative case, we randomly sample king that do appear in the molecule:
This molecule has no <functional group name> groups.
18

---

Hierarchical Graph Tokenization for Molccule-Language Alignment
Table
Summary of data resources and licenses of datasets involved in our paper.
Task>Datusets
Data Sources
License URL
License Note
PuhChem, R.FubChem
PuhChem
httpp: ptmps://wen /tw.///wnnnnnn www...lm.nih......
Works produced by the U.S. gov- ermment are not subject to copyright protection the United States. Any sach works found on National Li- beury of Medicine (NLM) Web sites may freely used reproduced without permission in the U.S.
hov//ee__policiee hhmm
Reaction Prediction
USPTO
htlps: htt////www.ee wswloo. learni saaantng-and-reoone
can be freely used, reused, and redistribtedd by anyone.
Property Prodiction
MoleculeNet
htttps://spensoureee cooeseemmtt
Permission bereby grantod. free of charge, any person obtaining copy of this wofeware and associ- sted documentation files (the "Soft- ware"), deal in the Software with- out restriction, including without limitation the rights to use. copy. modify, merge, publish, distribute. subliceuse, and/or sell copies of the Software, and to permit persons whom the Software is furnished do so.
Property Prediction
CYP450
https:// nih. gow/web_pon loiea homl
The data is from Zhao al. (2023) that curates PubChem BinAssay CYP 1A2, 2C9, 2C19, 2D6, 3A4 inhibition. Thus it shares the same license PubChem.
Molecular Notiflello
Description,
ChEBI
You are free to: Share - and redistribute the material in any modium format. Adapt remix, transform, and build upon the mate- any parpose, even commer. cially.
http/licenee//byy..0
Table 10. Summary of inputs and outputs of the tasks in experiments.
input
output
motif hallucination
molcculc and question about the existence of motif
yes or no yes or no property value
molecular property prediction (classification) molecular property prediction (regression)
molecule and question about the existence of the property
molecule and question about the value of the property molecule and question asking for the molecular caption molecules and question about the reaction
molecular caption
molecular caption molecular results
chemical reaction prediction
Table II. Examples of PubChem and HiPubChhhee datasets.
PubChem
HiPabChem
SMILES: -OC-CCCCCOOOOOO-----C------ This molecule an O-acykcamitine having acetyl acyl substiuent. It bas a role as human metabolite, functionally related to an acctic acid. conjugate
This molecule has carboxylic acids functional group. This molecule has no methyl amide, or amide, or nitro or thiols groups. This molecule an O-acylcamitine having acetyl acyl substitunt, has human metabolite. is functionally related acetic acid. It is conjugate base of an O-acctylcamitinium.
O-acety D-acetlcamiiinium.
SMILES: CCWCCICCOC--OOCCAeeeeeeeeeCCooo. This molecule is member maphthalenes.
This molecule has functional groups. This molecule member of naphthalenes. SMILES: Co buiellllweccciiiooooocCHACCCCCCO http:///wweccccccooooocccccccCCC CiCCiiiOOOoCCCCCaalaeeeeCCCCCCCe
This melecule is coproporpyrinngee. has role an Escherichia coli metabolite and mouse metaholite. is a conjugate acid copooporphyrinngen III(-.).
This moleccle has carboxylic acids functional groups. This molecule has no methyl amide, or diazo, or cyano thiols groups. This molecule copropopperrizogen. and mouse metabolite. It is conjugate acid of coproporphyrinogea III(4)).
Excherichia metaholite
Despite the simple augmentation strategy, we find that HiPubChen significantly reduces the hallucination issue, and improves the molecule-language alignment performance.
For comparison, we provide examples of PubChem and PubChem in Table 11.
19

---

Hierarchical Graph Tokenization for Molecule-Language Alignment
C.3. Details of Property Prediction Dataset
The task of molecular property prediction mainly aims predict certain biochemical or physical properties of molecules. Usually, these properties have close relation with the molecular substructures (i.e., functional groups) (Bohacek et al., 1996). In this work, consider the scenarios of both binary classification based and the regression based molecular property prediction, and the datasets are mainly derived from MoleculeNet (Wu et al., 2017).
For the classification, we consider three subtasks, HIV, BACE, and BBBP. The HIV subtask mainly evaluates whether the molecule is able to impede the replication of the HIV virus. The BACE subtask mainly evaluates the binding capability of a molecule to the BACEI protein. The BBBP subtask mainly evaluates the capability of a molecule to passively diffuse across the human brain blood barrier. For task-specific instruction tuning, we convert those classification based datasets into instructions. Examples are given in Table 12.
Table 12. Examples of the property prediction (classificationn datascts.
Dataset
Question
Answer
III
SMILES: b=DDDD//ckkekekk000000000000000-
Please help me evaluate whether the given molecule can imperle the replication the III vinus.
No
BACE
SMILES: (of/:=/CCccae2eoooeee2/CCCCCC
Can the given molecule hind to the BACEI protein?
Yes
BBBP
SMILES: CelcfulI+[[++ (uieuftt-jo--aCC(NnnnCCCCCa hatp///ccccccccnCCCCCCCCCCCCCCC Can the given molecule passivlyy ditfuse across the brain blood barrier?
Yos
Table 13, Examples of the property prediction (regression) datasets.
Question
Answer
SELFIES: Cot.r//oooncciccccciirreeeeeeeee
Can you give me the energy difference between the HOMO and LUMO orbitals of this molecule?
0,2756
SELFIES: [CCCC//==BBankkCCCC/C]ONNNNInBB HCCp:////wwna/IICCCCCCCCC-/aaaa [CJ [[]][=][[]
What the lowest unoecupied molecular orbital (LUMO) energy of this molecule?
-0.0064
SELFIES: [C][]]= = [[][==CC]==C]]]CCC]C]] [[Branch] J[C] ][ [[C][][[]
Please provide the highest uccupied molecular orbital (IIOMO) energy of this molecule.
-0,2132
For regression, adopt the instruction tuning data from - Dall-Inntructiion (Fang al., 2024). The regression based property prediction focuses predicting the quantum mechanics properties of the molecules. The 1D sequence information in this task is given by SELFIES (Krenn et al., 2019). The original data is sourced from the QM9 subset of the MolculeNet (Wu et al., 2017). There are three subtasks: (i) Highest occupied molecular orbital (HOMO) energy prediction; (ii) Lowest occupied molecular orbital (LUMO) energy prediction; (iii) and HUMO-LUMO gap energy prediction. Some examples of the regression based property prediction dataset are given in Table 13.
C.4. Details of Reaction Prediction Dataset
We adopt three chemical reaction related tasks from Mo 1ool-Inse Inst t ruct i ons (Fang et al., 2024): Forward reaction predic- tion, reagent prediction, and retrosynthesis prediction. The input and output contain ID sequence information given by SELFIES (Krenn et al., 2019). Some examples of the Mol-Ins wwnnsstuuctiioo ions datasets are given in Table 14, where the SELFIES represented molecules are denoted as ",SELFIES;" " for clarity.
The task of forward reaction prediction aims to predict the possible products of a chemical reaction. The input includes the SELFIES sequences of the reactant and reagent of the chemical reaction. And the model needs to predict the SELFIES of the products. The original data is sourced from USPTO 4, which consists of chemical reactions of organic molecules extracted from American patents and patent applications.
The task of reagent reaction prediction aims to predict the suitable catalysts, solvents, and ancillary substances with respect to chemical reaction. The input includes the SELFIES sequences of the chemical reaction. The original data sourced
hhttps://develppee uspto. uapto.gov/datt. gata
20

---

Hierarchical Graph Tokenization for Molecule-Language Alignment
Table 14. Examples of the chemical reaction datasets.
Task
Examples
Forward Reaction Prediction
Question: the provided reactants and reagents, propose potential produet,/SELFIESS
Answer: |SELFIES)
Reagent Prediction
Question: Please suggest some possible seagents that could have been used following chemical reaction. The reaction |SELFIES;
Answer: ;SELFIES;
Retrosynthesis Prediction
Question: Please suggest potential reactants the given product. The product is: ,SELFIES;
Answer: ;SELFIES;
from USPTO* S. as the other tasks.
The task of retrosynthesis prediction aims to reverse engineer a particular compound by predicting the potential reactants or reagents that required to synthesis the compound. The input includes the SELFIES sequences of the target product. The original data is sourced from USPTO similar to the other tasks.
C.5. Details of Molecular Description Dataset
For the molecular description task, we adopt a widely used dataset CHEBI 20 (Edwards al., 2021). Based on the molecules from PubChem, Edwards et al. (2021) collected the Chemical Entities of Biological Interest (ChEBI) (Hastings et al., 2015) annotations of the molecules, which are the descriptions of molecules. We transform the task into the instructions, and present some samples in Table 15. The authors collect 33, 010 molecule-text pairs and split them into training (80%), validation (10%), and testing (10%) subsets. We mainly adopt the original training split to tune the model and evaluate the tuned model on the original test split.
Table 15. Examples of the molccular descrioption datasets.
Question
Answer
SMILES: C[=CCCCCC-C//)AA//OOOO00
Could you give me brief overview this molecule?
The molecule the organoarsonic anion formed by loss a single proton from the arsunic acid grouping in phenylarsonic acid. It conjugate base of phenylar lar- sonic acid.
SMILES: CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
Could you provide description of this molccule?
The molecule is acyclic carboxylic anhydride resulting from the formal condensation of the carboxy groups two molecules of dodecanoic acid. derives from a dode- canoic acid.
SMILES: CCCCNC=O
Please give some details about this molecule.
The molecule is a member of the class of formamides that is formamide substituted by butyl group at the N atom. It has role as human metabolite. It derives from formamide.
C.6. Details of MotlfHlllu Dataset
The MotifHal 1 lu is mainly used to measure the hallucination of common functional groups by LGLMs. For the construction of Mot i EHallu, we consider the common functional groups in RDKit" as shown in Table 16. There are 39 common functional groups, while we neglect the one with the name of "???".
Then, we leverage RDKit (Landrum, 2016) to detect the existence of the left 38 valid functional groups within a molecule. We consider 300 molecules from ChEBI-20 test split (Edwards et al., 2021), and adopt the query style as for large lasion-language models (Li et al., 2023c) that queries the existence of specific functional group one by one:
"https://dewe ceeedeeeelper...oe cpt. .ieptto.ov/ddt...
"https://deve wwwpr..sptoogoo//eeeee
/https://gi com http://tttwa....oooototo////tt//
ona anal aoaaGGrups.... txt
21

---

Hierarchical Graph Tokenization for Molecule-Language Alignment
Is there <functional group name> in the molecule?
Examples of Mot fHa lu are given in Table 17.
During the evaluation, we detect whether the LGLM gives outputs meaning "Yes" or "No" following the practice in (Li et al., 2023c). For each molecule, we construct questions with positive answers for all kinds of functional groups detected the molecule, and questions with negative answers for randomly sampled functional groups from the 38 common functional groups in RDKit. The construction finally yields 23, 924 query answer pairs about the existence of functional groups the molecule. While is easy to scale up Moti i fHal by automatically considering more molecules and broader scope of functional groups, we find that the current scale is already sufficient to demonstrate the hallucination phenomena in LGLMs.
D. Details of Experiments
Implementation of graph tokenizer. We implement the GNN tokenizer/encoder based on the same GNN backbone, which is a 5-layer GIN (Xu et al., 2019). The hidden dimension is 300. For the node-centric tokenization, we employ the VQVAE GNN tokenizer from e-BRT (Xia et al., 2023) and adopt self-supervised learning tasks from the official Mole-BERT implementation. 8 For HIGHT, we train the VQVAE with the self-supervised learning tasks from (Zang et al., 2023) based on the official implementation. Meanwhile, those recommended by (Xia et al., 2023; Zang et al., 2023).
set the hyperparameters of GNN tokenizer training the same as
After training the tokenizer, we adopt the GNN encoder within the tokenizer instead of the codebook embeddings as we empirically find that the GNN embeddings perform better than that using the VQVAE codebook embeddings.
Implementation of LGLMs. For the cross-modal adapters, we implement it single-layer MLP with an input dimension of 300 our main focus is the tokenization. For HIGHT, we adopt three distinct adapters to handle the node- level, motif-level and graph-level embeddings. Meanwhile, we also adopt a Laplacian position encodings with respect to the supernode-augmenned graphs. The dimension of the Laplacian position encoding is set to 8, therefore the input dimensions of the adapters in HIGHT will be 308.
For the LoRA adapters, we use LoRA rank of 128 and scaling value a of 256 for molecular property prediction (classification) in order better fit with the task, and use LoRA rank of 64 and a scaling value of 16 for all the remaining methods and tasks.
For the base LLM, we mainly adopt vicuna-v-1. (Chiang et al., 2023). The overall scale of parameters is around 6.9B.
Implementation of instruction tuning. In stage instruction tuning, we train all methods based on PubChem-288 295k dataset. The training goes 5 epochs, with batch size of 64 (distributed to GPUs) by default. If there is OOM issue, will decrease the batch size a little bit to 40. The learning rate is set to 2 x 10 for all methods.
For classification- property prediction, the training goes 20 epochs, with a batch size of 128 (distributed to 4 GPUs) by default. If there is an OOM issue, we will decrease the batch size a little bit to 64. The learning rate is set to 8 x 10-5 for all methods.
For regression- based property prediction, the training goes epochs, with batch size of 64 (distributed to 4 GPUs) by default. The learning rate is set x 10-5 for all methods.
For molecular description, the training goes 50 epochs, with a batch size of 64 (distributed to 4 GPUs) by default. If there is an OOM issue, we will decrease the batch size a little bit to 32. The learning rate is set to 8 x 10-5 for all methods.
For forward reaction prediction, the training goes 5 epochs, with a batch size of 64 (distributed to 4 GPUs) by default. The learning rate set to 2 x 10-5 for all methods.
For reagent prediction, the training goes epochs, with a batch size of 64 (distributed 4 GPUs) by default. The learning rate is set to 2 x 10- for all methods.
Shttps: ://: thub. coom/uunxiag / 1 BERT httpp:: // thub. com www..///www.////taaooooe/
22

---

Hierarchical Graph Tokenization for Molecul--Language Alignment
Table 16. List of functional groups from RDKit used construct i FHall u. The functional group with the name "???" is neglected.
Chemical Representation
SMARTS
Name
NCC==OCCII
methyl amide
-C=000
+-C(=0JO:DD]
carboxylic acids
-(=0))0Me
carhonyl methyl ester
-C(=O)H
*-C[=O)CCDD]]
terminal aldchyde
-C=OON
*CC(=))N:DI]]
amnie
C(=0)C
carbonyl methyl
-N=C=0
* [ND2)-CCDD000011
isocyanate
-N=C=S
isothiocyanate
Nitrogen containng groups
-NO2
nitro
-N=0
*-[N:R0]=(O:DI]
nitroso
=N-O
"=[N:ROJ(OO:DI
oximes
NCCH
Imines
-N-H2
Imines
N=NCH3
terminal azo
-N=N
*- [N.D2)=[N:DI]
hydrazines
-N#N
* [N:D2]
diazo
-CHN
[C::D2)(NNDDI
cyano
containing groups
S02NH2
primary sulfonamide
NHSO2CH3
methyl sulfonamide
-SO31
* *[SDD4)=00==+400000
sulfonic acid
SO3CII3
methyl ester sulfonyl
S02CH3
methyl sulfonyl
-S22C
sulfonyl chloride
SOCH3
methyl sulfinyl
SC113
methylthio
-S
+=[::DI]
thiols
=S
*=[::D]]
thiocarbonyls
Miscellaneous frugments
-X
$1,555 DECDDMNCCDDDDNCCDDDDCCCCCC *-(C:D4](FXF)FF
halogens
-Bu
t-butyl triftuoromethyl
-CF3
-CHCH
+[CCDD2)[C::D...)
acctylenes
cPropyl
cyclopropyl
Teeny groups
-OBI
ethoxy
-OMe
methoxy side-chain hydroxyls
-0
*[[ODD]]
=0
*[O:DI]]
side-chain aldehydes or ketones
-N
*N[:DII
primary amines
=N
*NN:DIJ
???
#N
*NN:DIJ
nitriles
For retrosynthesis prediction, the training goes 5 epochs, with a batch size of 64 (distributed to 4 GPUs) by default. The learning rate is set to 2 x 10-5 for all methods.
Training and evaluation. Throughout the paper, we use a max token length of 2048 Mcanwhile, we adopt an Adam W optimizer with a warmup ratio of 3% for optimizing all models. We select the final model according to the best training loss.
23

---

Hierarchical Graph Tokenization for Molecul--Language Alignment
Table 17. Examples of the Mot fHall
dataset.
Question
Answer
SMILES: Tww.:///w.....aaa///////////////
Is there methyl sulfonyl group the molecule?
No
there carbonyl methyl ester group in the molecule?
SMILES: CWYCCCT=OXCCCCNNCCCCCCCC Xtt.:///Cw.C...////////////////.
there lerminal aldehyde group the melecule?
No
For the evaluation of classification-basedd property prediction, we adopt the ROC-AUC following the common practice (Wu et al., 2017).
For the evaluation of regression-based property prediction, we adopt the Mean Absolute Error (MAE) following the common practice (Fang et al., 2024).
For the evaluation of molecular description, we adopt BLEU-2, BLEU-4, ROUGE-I, ROUGE-2, ROUGE-I, and METEOR following the common practice (Papineni al., 2002; Lin, 2004; Edwards al., 2021). To improve the reliability of the evaluation, the metrics are computed based the tokenizer scibert..sciocab.uece wnnaaaaaiiiaaabaaaeeeeeeeeeeeeee of SciBERT (Beltagy et al., 2019).
We follow the common practice to evaluate models for the tasks of chemical reaction predictions (Fang et al., 2024). We adopt linguistic metrics such as BLEU (Papineni et al., 2002), ROUGE-L (Lin, 2004). METEOR (Banerjee Lavie, 2005) and Levenshtein scores (Yujian & Bo, 2007). Meanwhile, we also validate the validity the generated molecular sequences with RDKit (Landrum, 2016). addition, several molecular similarity measures are also leveraged. Specifically, we present the MAE of the RDKit, MACCS, and Morgan fingerprints assess the semantic similarity of the generated compounds and the ground truth ones (Durant et al., 2002; Schneider et al., 2015).
As for the Mot i fHallllllll lu, in order to avoid the drawbacks that LGLMs may output answers that do not follow the instructions, we compare the loss values by feeding the answers of "Yes" and "No", and take the one with a lower autoregressive language modeling loss as the answer. Following the practice in LVLMs, we present the Fl scores, accuracies, and the ratio that the model answers "Yes" (Li al., 2023c). Given the severe imbalance of positive and negative samples, we separately report the Fl scores for positive and negative classes.
Software and hardware. We implement our methods with PyTorch 11.3 (Paszke et al., 2019). We run experiments on Linux Servers with NVIDIA V100 and NVIDIA A100 (40G) graphics cards with CUDA 11.7.
E. More Ablation Studies
E.1. Computation overhead
Table 18. Training Computational Overhead. We count the average graph size of PubChem and HiPubChem, where HiPubChee adds 9 additional tokens average. The real preprocessing time and training time shown below, which are estimated based on 4 A100 40G GPUs. Although HIGHT requires more time to train, the absolute computational overhead of HIGHT is not high.
Graph Size
Preprocessing Time
Training Time
PubChem
34.39
16min 32sec 8hour 17min 59sec
HiPubChem
43.21
25min 35sec
15hour 36min 23sec
E.2. Ablation studies with different setups of the tokenizers
In Table 21, present results of the ablation studies with different setups of HIGHT and node-centric tokenizer.
24

---

Hierarchical Graph Tokenization for Molecule-Language Alignment
Table 19. Inference Computational Overhead. The inference computational overhead is estimated based A100 40G GPUs. During the inference, the LLM latency takes up the majority of time. A well-trained LGLM with HIGHT is able generate more concise and valid answers and thus may take less time during inference.
Property Prediction
MolCaption
Reagent Prediction
Forward Reaction
Retrosynthesis
InstructMol HIGHT
14min 54sec
6hour 22min 27sec
56min 56sec
Ihour 34min 28sec
Ihour 50min 47sec
15min 12sec
4hour 59min 50sec
50min 29sec
Ihour 22min 08sec
Thour 49min 42sec
Table 20. Number of Tunable Parameters during Training. When pretraining the GNN tokenizer, the number of tunable parameters is the number of parameters in GNN encoder; In stage 1, the number of tunable parameters is the number of parameters in the projector; In stage 2, the number of tunable parameters is the number of parameters in the projector and in LoRA.
graph token dimension
GNN encoder
params in projector
params in tokenizer
LoRA
InstructMol HIGHT
300d
1,860,905
1,232,896
3,093,801
$59,907,840
300d
1,865,105
3,796,992
5,662,097
159,907,844
25

---

Hicrarchical Graph Tokenization for Molccule-Languag, Alignment
Table 21. More results chemical reaction tasks with ablation studies. These tasks encompass reagent prediction, forward reaction
prediction, and retrosynthesis. few-shot ICL results from (Pang al., 2024). lask-specific instruction data finetune.
MODEL
EXACTК
BLEUt
LEVENSHTEING
RDK FTS1 MACCS PTS1
MORGAN FTSt
VALIDITYI
Reagent Prediction
Alpacal (Dubois 2023)
0,000
0.026
29.037
0.029
0,016
0.001
0.186
Baize! (Xu al., 2023)
0.000
0.051
30.628
0.022
0.018
0.004
0.099
ChatGLM! (Zenged 2023)
0.000
0.019
29.169
0.017
0,006
0.002
0.074
LLama (Touvron al., 2023a)
0.000
0.003
28.040
0.037
0.001
0.001
0.001
Vicuna (Chiang 202.3
0.000
0.010
27.948
0.038
0,002
0.001
0.007
Mol-Instruction (Fang et al., 2024)
0.044
0.224
23.167
0.237
0.364
0.213
1.000
LLama-7hs (Touvron et al., 2023aXLoRA)
0.000
0.283
53.510
0.136
0.294
0.106
1.000
InstructMol-G
0.031
0.429
31,447
0.389
0.249
0.220
L.00
+ Posiiionnl Encoding
0.009
0,423
30.833
0,370
0.231
0.197
0.986
+Fi PubChem
0.016
0.473
30,455
0.369
0.237
0.194
0.990
+Large Tokenizer
29.163
InstructMol-GS
0.057
0.439
29.757
0.437
0.314
0.271
0.999
0.016
0.459
29.238
0.359
0.225
0.189
0.988
HIGHT-G
0.050
0.462
28.970
0.441
0.314
0.275
1.000
HIGHT-GS
0.067
0.482
27.167
0.462
0.346
0,303
1.000
HIGHT +1.Lamu-2-7-cchut
0.057
0.495
26.591
0.453
0.333
0.293
1.000
Forward Reaction Prediction
Alpacal (Dubois 2023)
41.989
0.1124
Baizel (Xu al. 2023)
41.500
ChatGLM! (Zeng 2023)
0.000
0.183
40.008
0.050
0.100
0.044
0.108
LLama (Touvron
2023a)
0.000
0.020
42,002
0.002
0.039
Vicuna (Chiang 2023)
0.000
0.057
41,690
0.007
0.016
0.006
0.059
Mol-Instruction (Fang et al., 2024)
0.045
0,654
27.262
0.313
0.509
0.262
1.000
LLama-7b' (Touvron et al., 2023aXLoRA)
0.012
0.804
29.947
0.499
0.649
0.407
1.000
InstuutMol-Ci
+Positional Encoding
0.0102
0,829
26.622
0.419
0.328
0.268
0.981
+HiPubChem
0.011
0.819
26.010
0.396
0.315
0.264
0.975
+Large Tokenizer
0.040
24.051
0,380
0,328
0.996
InstructMo-GSS
0.252
0.926
17.773
0.755
0.599
0.543
1.000
InstructMolI.. ama-2-7h-chh
0.020
0.841
25.109
01426
0.339
0).884
0.998
HIGHT-G
0.037
0.869
23.759
0.590
0.394
0,340
0.993
HIGHT-GS
0,293
0.935
16.687
0.774
0.618
0,566
1.000
HIGHT +LL.ama---7<chaa
0.042
0.873
23.854
0.590
0.402
0.344
0.996
Retmssythhsis
Alpaca (Dubois al.. 2023)
0.000
0.063
46.915
0.005
0.023
0.007
Baizel (Xu et al., 2023)
0.000
44.714
0.050
0.023
ChaiGLM (Zeng ul., 2023)
0.000
0.117
48.365
0.075
LLama (Touvron et al., 2023a)
0.000
0.036
0.029
0.017
Vicuna (Chiang
20233
46.877
Mol-Instruction (Fang et al., 2024)
0.009
0.705
31.227
0.283
0.487
0.230
1.000
LLamm-7Tb (Touvron et al., 2023a)LoRA)
0.000
0.283
$3.510
0.136
0.294
0.106
1.000
InstructMo-GG
0.001
0.835
31.359
0.447
0.277
0.241
0.996
+Positional Encoding
0.000
0.793
33.859
0.295
0.218
0.192
0.983
+Ei PubChem
0.000
0.755
35.811
0.282
0.218
0.177
0.997
+Large Tokenizer
0.001
0.842
30.613
01459
0.287
0,263
0.999
InstnctMol-GG
0.172
20,300
0.765
0.615
0.568
InsteetMol-LLama---700
0.000
0.806
32.128
0.292
0.234
0.202
0.985
HIGHT-G
0.008
0.863
28.912
0.564
0.340
0.309
1.000
HIGHT-GS
0.202
0.914
20.194
0.772
0.623
0.577
0.999
HIGHT +L.Lam---7b-chatt
28.964
26

---

Hicrarchical Graph Tokenization for Molecule-Languaee Alignment
Table 22. Full results of motif hallucinations Mot fHalll with ablation studies.
METHOD
FI (pos) t
Fl (neg) f
Fl (avg)
Node-centric Tokenization
InstructMal-G
95.7
9.5
52.6
MaatuutMo--GLLLLaa-----
99.6
2.8
51.2
InstructMol-GS
97.1
10.6
53.8
Hierarchical Tokenization
HIGHT-G
85.5
48.2
66.9
BGGGT--G+LJama--7bbbee
55.1
65.2
HIGHT-GS
84.5
42.7
63.6
Ablation variants
InstructMal-G + Positional Encoding
96.4
19.8
58.1
InstructMo]-G HiP
96.6
12.5
54,6
HIGHT-G w/o HiP FuFuhhhem
96.6
12.5
54.6
HIGHT- w/o HiPubChen
52.4
Table 23. Results of molecular property prediction tasks (regression) QM9 with ablation studies. We report the result in MAE. f: few-shot in-context learning (ICL) results from (Fang et al., 2024). A refers to the HOMO-LUMO energy gap.
METHOD
HOMO + LUMO
Ar 4
AvG
LLM Based Generalist Models
Alpaca (Dubois et al., 2023)
322.109
Baize (Xu et al., 2023)
261.343
LLama2-7B (Touvron et al., 2023b) (S-shot ICL)
0.7367
0.8641
0.5152
0.7510
Vicuna-13B (Chiang al., 2023) (S-shot ICL)
0.7135
3,6807
1.5407
1.9783
Mol-Instruction (Fang et al., 2024)
0.0210
0.0210
0.0203
0.0210
InstructMol-G
0.0111
0,0133
0.0147
0,0130
+Positional Encodings
0.0300
0.0395
0.0357
0.0350
+HiPubChem
0.0305
4,4019
0.0494
1.1226
HIGHT-G
0.0078
0.0086
0.0095
0.0086
27


Reviews:
[{"review_id": "cGIKt6U2Et", "reviewer": "Reviewer_eiUy", "summary": "This paper aims to address the issue of tokenization in existing LGLMs (large graph-language models) that neglect the essential hierarchical structures inherent in molecules. The hierarchical structures are reflected as motifs or functional groups that are subgraphs within the larger molecular graph.\n\nThe proposed solution is to introduce \"motif tokens\" as input to LLMs, in addition to the atom tokens and text tokens as usual. The \"motif tokens\" are obtained by an existing VQVAE approach (Xia et al., 2023).\n\nThe evaluation of the proposed method is conducted across various downstream tasks, including property classification/regression, molecular captioning, and a custom-designed task focused on answering yes/no questions regarding the presence of functional groups.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer eiUy,\n\nThank you for your time and insightful suggestions for our paper. Please find our responses to your concerns below.\n\n> In molecular property prediction and chemical reaction tasks, HIGHT does not demonstrate substantial advantages.\n\nWe need to clarify that the performance gaps between HIGHT and the baselines lie in the differences of the base models. For example, Uni-Mol is built on a T5 architecture pretrained on large-scale 3D molecular data. GIMLET is built on a T5 pretrained on molecular property prediction-focused data.\n\nWe present a [detailed table](https://anonymous.4open.science/r/HIGHT-rebuttal-82B1/comparison_LGLMs.md) listing the information and pretrained data used by HIGHT and baselines. It can be found that some \"baselines\" are pretrained onto significantly larger datasets with additional information, **which are not fair baselines for comparison**. We report their performances in order to have an overview of the overall progress.\n\nTherefore, **our direct baseline is InstructMol, which uses the same architecture and pretraining data**. Compared to InstructMol, HIGHT demonstrates significant improvements.\n\n> how does the inclusion of motif tokens contribute meaningfully to other downstream tasks beyond functional group recognition?\n\nThe visualization of Figure 1b may introduce some perceptual biases about the improvements brought about by the different scales of evaluation metrics used in the method. Due to the heterogeneity of the evaluation metrics, we transform the numerical values a bit for better visualization. The new Figure 1b and the corresponding details are given [here](https://anonymous.4open.science/r/HIGHT-rebuttal-82B1/overview.md).\n\nComparing InstructMol and HIGHT, indeed, **incorporating motif-related information improves other downstream tasks**.\n\n> Limitatiions of tokenization by BRICS; Hierarchy in the tokenization process; Primary novelty of the work\n\nWe need to clarify that the **primary novelty of this work lies in the introduction of the hierarchical tokenization**. It is well known that motif information is critical to molecular understanding. Without that, LGLMs will exhibit high hallucination and perform subpar on molecule-language alignment. \n\nWe acknowledge the limitations of BRICS in tokenization, while it's sufficient to demonstrate the usefulness of hierarchical graph information. **Despite the simplicity of our methodology, it brings significant improvements compared to LGLMs without hierarchical graph information**. Meanwhile, our method possesses a high degree of extensibility and can be augmented by incorporating advanced motif extraction techniques (such as[1,2]), or learnable motif extraction, which could further boost the performance of HIGHT. \n\nIn addition, we would also kindly note a key technical contribution in HIGHT lies in not only the tokenization, but also in the **alignment tuning process via HiPubChem**. We feel the original title may introduce confusion, thus we would like to change our title to ``Hierarchical Molecule-Language Alignment'' in order to highlight the key focus of our work, i.e., the importance of hierarchical information for molecule-language alignment.\n\n> Metrics in MotifHallu\n\nWe have supplemented the details in our manuscript: \n\n- To calculate the F1 scores for positive and negative classes, we first divide the motif hallu dataset according to the class label. For the positive F1 score, the calculation is normal. For the negative F1 score, we reverse the labels and predictions. \n- Motif Hallu is quite imbalanced. There are 4,124 positive and 19,800 negative samples. \n- As Acc will be biased due to the imbalance, we updated the table to report classwise averaged F1 score, i.e., macro F1. To obtain an overview of the results, we present a detailed table [here](https://anonymous.4open.science/r/HIGHT-rebuttal-82B1/motif_hallu.md) with various metrics. It can be found that, HIGHT still obtains significant improvements up to 14 in terms of macro F1.\n\n**References**\n\n[1] Motif-based graph self-supervised learning for molecular property prediction, NeurIPS'21.\n\n[2] Molecular representation learning via heterogeneous motif graph neural networks, ICML'22.", "claims_and_evidence": "The paper claims that Hierarchical Graph Tokenization could improve LGLM understanding of molecular structures, thereby benefiting various downstream tasks.\n\nHowever, in molecular property prediction (classification), the proposed method performs significantly worse than other baselines (though slightly better than LLaMA-2 in some cases). A similar lack of supporting evidence is observed in chemical reaction prediction tasks, where the proposed method fails to demonstrate substantial advantages.\n\nThis weakens the argument that introducing motif tokens inherently improves molecular understanding across tasks.\nAs shown in Figure 1b, the proposed method significantly outperforms baselines only on MotifHallu, the benchmark dataset designed specifically for motif-related tasks. However, the proposed method has been specifically designed to enhance functional group-related understanding by introducing \"motif tokens\" as input to LLMs. By explicitly incorporating these motif tokens, the input itself becomes aware of functional groups, making it understandable that the model performs more accurately on motif-specific questions. But it cannot address the key question: how the inclusion of motif tokens contributes meaningfully to other downstream tasks beyond functional group recognition.", "methods_and_evaluation_criteria": "The concept is presented as “Hierarchical Graph Tokenization”, but in practice, its implementation is limited to adding motif embeddings as tokens alongside atom tokens in the input to LLMs. Notably, motifs are pre-identified using the Breaking of Retrosynthetically Interesting Chemical Substructures (BRICS) algorithm, rather than being learned within the model itself. \n\nFurthermore, atoms and motif tokens are obtained separately using distinct VQVAEs, meaning that hierarchy is not explicitly reflected in the tokenization process.\n\nAdditionally, the tokenization of both atoms and motifs relies on an existing VQVAE approach (Xia et al., 2023), making the primary novelty of this work the incorporation of “k” motif embeddings via VQVAE, rather than a fundamentally new tokenization strategy.\n\nIn the evaluation on MotifHallu, the metrics were not clearly defined.", "theoretical_claims": "There is no theoretical claim.", "experimental_designs_or_analyses": "The experimental evaluation was conducted on various tasks, which are sufficient.", "relation_to_broader_scientific_literature": "If the claim could be supported, the contribution of this work can be a valuable addition to the field of LLM models for chemical tasks.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "None", "questions_for_authors": "In “Given the imbalance of positive and negative samples, we separately report the F1 scores for different classes.”   How are the F1 scores for the positive and negative classes computed? In a binary classification problem, the F1 score is typically reported as a single value, considering both classes.\n“Given the imbalance of positive and negative samples”, what is the ratio of positive vs negative?\nIf it is super imbalanced, why the Acc is still reported? And Acc does not given a misleading high value?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aewTyjpEce", "reviewer": "Reviewer_brfq", "summary": "The paper introduces a new representation of graphs (specifically, molecules) for the purpose of tokenization for LLMs. The key aspect of the new representation is that it not entirely just node-based, rather it captures features in the graph at both the node and the motif level. It is not clear to me whether there are actually more than two models in the hierarchy, and how the different levels communicate.\n\nThe authors demonstrate impressive performance on several benchmark datasets. However, setting up a model of this type involves several components, including the LLM itself and fine tuning, so it is not entirely clear what fraction of the improvement comes from the hierarchical tokenization itself.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer brfq,\n\nThank you for acknowledging our performance improvements and constructive suggestions. Please find our explanations to your questions below:\n\n>  it is not entirely clear what fraction of the improvement comes from the hierarchical tokenization itself.\n\nWe kindly refer Reviewer brfq to Figure 3c and Table 16 in the appendix, where we conduct comprehensive ablation studies to analyze the contributions of each component in HIGHT. The differences between HIGHT and InstructMol lie in the positional encoding, hierarchical instruction tuning, and the hierarchical tokenizer. Thus, we study the performances of InstructMol plus those components, where each can also be considered as a variant of HIGHT without hierarchical tokenization and some components.\n\nIn experiments, one can find that, compared to HIGHT, merely incorporating positional encodings, using the hierarchical instruction tuning, or using a larger tokenizer in InstructMol can not bring improvements or even lead to decreased performance. It demonstrates that, **while each component in HIGHT is critical to the performance, the hierarchical tokenizer contributes most to the alignment performance**.\n\nIn addition, in Figure 3a, we also experiment with different base LLMs such as vicunna and Llama2, where HIGHT demonstrates consistent improvements.\n\n> Discussion between Hu et al. 2024.\n\nFirst, we would like to kindly refer Reviewr brfq to the ICML reviewer instruction, that ``Authors cannot expect to discuss other papers that have only been made publicly available within four months of the submission deadline.''. Even Hu et al., came out in November 2024, it is within four months of the ICML deadline.\n\nNevertheless, we will revise our manuscript to include a discussion with Hu et al. Although both of our works incorporate the hierarchical information into LGLMs, \n\n- Hu et al., does not consider the influence of the hierarchical information on the molecule-language alignment, i.e., hallucinations. \n- They also do not consider incorporating the hierarchical information into the alignment tuning phase. \n- They evaluate the usefulness of the hierarchical graph information in a limited number of tasks.\n\n> Questions related to hierarchical graph representation \n\nHierarchical graph representation refers to **aggregating the low-level (i.e., node-level) into meaningful higher-level abstractions, usually via subgraph aggregation.** For example, in a social network, the higher-level abstraction could be a small group or a community. Although it is not limited to a restricted number of abstraction layers, in molecules, the hierarchy of atom-motif-molecule is the usual schema of the abstraction[1]. Hence, **we simply follow the common practice in the community to obtain the hierarchical molecular representations, i.e., using BRICS to identify common functional groups[2,3]**. Nevertheless, one could also apply subgraph learning strategies to enable more flexible hierarchies.\n\nIn fact, when going beyond small molecules to biomolecules such as proteins, more abstract subgraphs, such as the 3D folded architecture, or combined 3D structure between small drugs and protein pockets, are also critical to the functionalities of proteins[4].\n\n\n\n> Other minor suggestions\n\nWe have revised our manuscript according to your suggestions.\n\nFor the caption of Fig 1: HIGHT brings improvements to all tasks. Due to the heterogeneity of the evaluation metrics, we transform the numerical values a bit for better visualization. The new Figure 1b and the corresponding details are given [here](https://anonymous.4open.science/r/HIGHT-rebuttal-82B1/overview.md).\n\n**References**\n\n[1] The art and practice of structure-based drug design: A molecular modeling perspective. Medicinal Research Reviews 1996.\n\n[2] Motif-based graph self-supervised learning for molecular property prediction, NeurIPS'21.\n\n[3] Molecular representation learning via heterogeneous motif graph neural networks, ICML'22.\n\n[4] Independent SE (3)-Equivariant Models for End-to-End Rigid Protein Docking, ICLR'22.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "Attempted to, to the extent that the experimental process of a complex pipeline involving LLMs can be verified based on just a paper.", "relation_to_broader_scientific_literature": "The authors survey the literature in some depth", "essential_references_not_discussed": "There is a recent paper titled \"Exploring Hierarchical Molecular Graph Representation in Multimodal LLMs\" by Hu et al that addresses the same problem but does not seem to be cited. This might just be because the paper only appeared on ArXiV in November of last year. Nonetheless, I would very much like the authors to compare the two approaches.", "other_strengths_and_weaknesses": "The hierarchical graph representation itself is not described in detail.", "other_comments_or_suggestions": "- The intro seems to have been a little hastily written and its English could be improved. \n- The caption of Fig 1 says \"all metrics have been improved a bit\". What does this mean??\n- Rather than \"high order molecular structures\" I would say \"higher order molecular structures\".\n- In line 200, \"concrete\" should not be used as a verb", "questions_for_authors": "1. How does the hierarchical graph representation relate to BRICS exactly?\n2. Does the representation have just two levels of hierarchy or more?\n3. How do the different levels communicate with each other?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6dMmtOsagk", "reviewer": "Reviewer_zQSo", "summary": "This paper proposes HIGHT, a novel molecular graph tokenization and post-training framework for applying large language models to  molecular graphs. The paper proposes a novel hierarchical tokenization method incorporating molecular motif information, and uses novel alignment pretraining strategy to train models before task-specific instruction tuning. Experiments and ablation studies show that this method achieves excellent performance in various molecular property prediction, molecular description and chemical reaction prediction benchmarks.\n\n## update after rebuttal\n\nMy concerns have been well addressed so I will keep my rate.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer zQSo,\n\nThank you for your time and insightful suggestions, as well as your acknowledgment of the value and convinceness of our work. Following your suggestions, we have revised our manuscript to include a discussion on future work regarding extending HIGHT to incorporate 3D information.\n\n- **Hierarchical 3D Tokenizer**: To incorporate the 3D hierarchical molecular information, we need to design a new tokenizer based on GNN backbones such as EGNN and SE(3) Transformers compatible with 3D information. For example, we can extend the hierarchical VQVAE in HIGHT, to accommodate 3D properties of motifs, such as SE(3) equivariance to the motif representations.\n- **Scaling 3D Data for Alignment**: When extending to 3D information, one could curate both the molecules, proteins and RNA data. The motifs in proteins then can be certain amino acids. In RNA, the motifs can be certain recurrent structures. \n\n- **3D Tokenizer Training**: To train the tokenizer to fully capture the 3D molecular information, we can design several self-supervised learning objectives such as 3D spatial position recovery, masked atom/motif prediction.\n\n- **3D Alignment Tuning**: To facilitate the alignment of molecule and language, we need to extend the instruction tuning dataset to include descriptions of 3D molecular properties, such as 3D positions. In addition to incorporating the 3D hierarchical information, we also need to incorporate language captions of the 3D hierarchical structures, such as properties of the amino acids. In addition, one could also incorporate pairwise hierarchical graph information, such as binding affinities of molecules to proteins.\n\nTherefore, it is promising to align 3D graphs with languages for broader scentific tasks, for which HIGHT provides the foundation for incorporating 3D hierarchical graph information for better alignment of graphs and languages.", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing experiment results.", "methods_and_evaluation_criteria": "Yes, proposed methods and evaluation criteria make sense for molecular property prediction, molecular description, and chemical reaction prediction problem.", "theoretical_claims": "There are no proofs or theoretical claims in the paper.", "experimental_designs_or_analyses": "Yes, I have checked experiment settings, and experiment results and analyses in motif hallucination, molecular property prediction, molecular description and chemical reaction prediction experiments. All experiments are sound and valid to me.", "relation_to_broader_scientific_literature": "The novelty contributions and empirical findings are very useful and valuable to broader chemical and biological science domains, as this paper proposes an effective method to make better use of the power of large language models on molecule data, potentially providing excellent technical tools for scientists in these domains for their domain-specific applications.", "essential_references_not_discussed": "There are no essential related works not discussed.", "other_strengths_and_weaknesses": "No other comments on strengths and weaknesses.", "other_comments_or_suggestions": "No other comments or suggestions.", "questions_for_authors": "While the proposed method makes valuable contributions in easing the application of large language models to molecules, one significant gap still exists --- the proposed method cannot encode 3D structure information of molecules into sequence representations, which may impacts molecular prediction performance (e.g., results in Table 2 are not as good as state-of-the-art 3D graph neural network models). I would like to hear author's opionion about how could the proposed the method be extend to encoding 3D molecular structures?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sYFHqcTLZv", "reviewer": "Reviewer_VbFG", "summary": "Previous LGLMs usually focus on the node level of molecules, ignoring the structure information in molecules. To address this, the paper proposes a novel strategy called Hierarchical Graph Tokenization (HIGHT), which uses a hierarchical graph tokenizer to encode the hierarchy of atom, motif, and molecular levels of informative tokens to improve the molecular perception of LLMs. The paper first uses VQ-VAE to obtain embeddings for atoms and motifs. Then, it concatenates the motif embeddings after the atom embeddings to feed into the models. The paper also uses a new augmented instruction tuning dataset, HiPubChem, enriched with the hierarchical graph information, to further enhance the molecule-language alignment. The paper also proposes the first hallucination benchmark MotifHallu based on common functional groups.  Extensive experiments show the effectiveness of HIGHT in reducing hallucination by 40%, and significant improvements in various molecule-language downstream tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer VbFG,\n\nThank you for your time and suggestions to our work. Please find our detailed responses to your questions below:\n\n> Related work section\n\nThank you for acknowledging our contribution as the first to incorporate the hierarchical graph information. As for the referred works, we have revised Sec 2 as the related work section, and included the discussion of existing attempts:\n\nExisting works also try to enrich the molecule-language alignment with additional modalities, such as 2D[2] and 3D[1] information. In contrast, we focus on the intrinsic hierarchical information of the molecules, such as motifs.\n\n> It is relatively more straightforward given the function groups in the inputs\n\nWe kindly refer Reviewer VbFG to Table 17 in the appendix, where we can find that, even with the functional groups in the inputs (i.e., HIGHT w/o HiPubChem), it remains challenging for LGLM to correctly recognize whether the functional groups exist in the input molecule.\n\nIn addition, it is not straightforward that the hierarchical alignment across atom-functional group-molecule is beneficial to broader downstream tasks.\n\n> Why choose VQVAE to obtain node embedding\n\nWe choose to use VQVAE because it is one of the simplest and standard architectures (e.g., Mole-Bert and other LGLMs as in [here](https://anonymous.4open.science/r/HIGHT-rebuttal-82B1/comparison_LGLMs.md)). We would like to kindly note that, when adding SMILES/SELFIES to LLMs, it does not directly output node embeddings. Nevertheless, in Table 16 and 17, we also consider adding SELFIES to LLMs, which slightly decreases the perception of functional groups, and increases the chemical reaction prediction performances.\n\n> Computational overhead\n\nWe have revised our manuscript to include the computational overhead report, including training and inference latency, as well as tunable parameters, which can be found [here](https://anonymous.4open.science/r/HIGHT-rebuttal-82B1/computation.md). Although HIGHT requires longer training time and relatively higher tunable parameters, the absolute values are not high. Moreover, during inference, as LLM latency consumes most of the computation, HIGHT can even reduce the inference latency by generating more concise answers.\n\n> Performance gain in Table 1\n\nWe kindly refer Reviewer VbFG to Table 17 in the appendix, where we can find that, merely using the HiPubChem for instruction tuning with InstructMol still suffers from severe hallucination. It means that both tokenizers and the instruction tuning in the alignment phase are necessary to reduce hallucination. In addition, in Table 16, we can find similar phenomena in downstream tasks of chemical reaction prediction, which further strengthens the necessity for the two components.\n\n> GIMLET in MotifHallu\n\nWe have evaluated two state-of-the-art SMILES based LGLMS, i.e., GIMLET and Galactica-6.7B in MotifHallu. The results are given [here](https://anonymous.4open.science/r/HIGHT-rebuttal-82B1/motif_hallu.md). Interestingly, those models demonstrate a high hallucination of negative classes. Therefore, we recommend reporting macro F1 scores (F1 scores averaged across classes). **HIGHT demonstrates significant improvements in terms of macro F1 scores over all baselines, up to 14**.\n\n> Comparison with the instruction data\n\nWe provided a comparison between the instruction tuning data used in HIGHT and other LGLMs [here](https://anonymous.4open.science/r/HIGHT-rebuttal-82B1/comparison_LGLMs.md). It can be found that HIGHT uses significantly less information and data to achieve relatively good capabilities across multiple tasks.  \n\n> code\n\nWe have provided AC an anonymous link to our code, and kindly asked AC to share to the reviewers according to the ICML policy.", "claims_and_evidence": "1. The paper claims they are the first to incorporate the hierarchical graph information into LGLMs. To my best knowledge, it seems to be true. However, the paper needs to include a related work section to compare itself against other previous models. Some other papers also try to capture hierarchical information in a similar way, such as [1,2]. A related work section can help authors future clarify their novelty.\n2. The hallucination benchmark MotifHallu seems to be problematic. Given that the proposed methods already contain those function groups in their input, it is relatively more straightforward for such a model to answer correctly. \n\n\n[1] Li, S., Liu, Z., Luo, Y., Wang, X., He, X., Kawaguchi, K., ... & Tian, Q. (2024). Towards 3d molecule-text interpretation in language models. ICLR 2024.\n\n[2] Liu, Z., Li, S., Luo, Y., Fei, H., Cao, Y., Kawaguchi, K., ... & Chua, T. S. (2023). Molca: Molecular graph-language modeling with cross-modal projector and uni-modal adapter. EMNLP 2023.", "methods_and_evaluation_criteria": "The method to obtain node/motif embedding is based on Hierarchical Molecular Graph Self-supervised Learning (HiMol) https://www.nature.com/articles/s42004-023-00825-5. While it looks fine to use VQ-VAE to obtain motif embedding, it is unclear why the paper chose the VQ-VAE to obtain the node embedding instead of the regular SMILES/SELFIES string. It might be better to include an ablation study to use SMILE/SELFIES string for atom embedding to make a fair comparison for results in Table 2. Additionally, using VQ-VAE to obtain the embeddings seems to use more computational resources. The paper needs to report its GPU hours, parameter number, and other computation-related information to make a fair comparison. Otherwise, the proposed method might not be scalable.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The paper seems to include relatively comprehensive baselines and downstream tasks to compare against. The paper also conducts ablation study and zero/few-shot testing. However, some problems still exist. \n1. In Table 1, the paper compares the InstructMol-? against HIGHT-?. However, based on the ablation in this table, it seems that the performance gain is mainly from the instruction tuning dataset instead of the actual hierarchical tokenization part, which decreases the novelty of the proposed model. GIMLET, a method the paper also used as baseline in Table 3, should be added as a SMIES-based LGLM for Motif hallucination experiments to show the contribution of the hierarchical tokenization. \n2. In Table 4, the paper claims that the paper fails to achieve SOTA because some of the previous methods used a large amount of pertaining data. However, the paper also used a large amount of instruction tuning data (MoleculeNet, CheBI-20, etc) in the second stage, which seems to be exactly the same as those methods. The paper needs to do some comparison between those two instruction tuning data. Otherwise, the previous methods seem to be better due to less parameters and simpler architecture.", "relation_to_broader_scientific_literature": "The paper provides a new representation learning strategy for molecules. However, the paper fails to include a related work section, which makes readers confusing.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Weaknesses:\n1. The paper did not provide any code for reviewers to check.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a graph tokenization approach to enable graph input with LLMs. The proposed approach claims to be specifically hierarchical, yet the paper is missing discussion of a very related work [1], which also focuses on considering multiple graph levels, and others the reviewers mention. In this respect, the claim that this paper is the first hierarchical tokenization seems wrong. Main reviewer criticism concerns the lack of comparison with baselines (some was provided in the rebuttal) and in how far specific components are critical for performance on specific types of data. The reviewers opinions generally are mixed/favorable towards the paper with ratings between weak reject and accept.

[1] LLaMo: Large Language Model-based Molecular Graph Assistant, Neurips'24
Decision: Accept (poster)